Opt Problem

The /opt Problem

There is not enough space on the rootfs to fit all applications, therefore an additional partition has to be used. But how?

There is a section in the Maemo 5 Developer Guide explaining the basics: Installing under /opt and /MyDocs

Current situation

• 256MB OneNAND chip: bootloader, kernel, root (ubifs - with compression)
• 32GB eMMC: /home ~2GB (ext3), /home/user/MyDocs ~29GB (vfat), swap 0.7GB
• symlink: /opt/ -> /home/opt/
• OneNAND is probably "much" faster than the eMMC (hard numbers? read? write? small blobs? big blobs?)
• file system benchmark results for different media
• OneNAND performance figures: Sustained read performance: 108MB/s. Sustained write performance: up to 17MB/s. See Samsung OneNAND specs.
• eMMC performance figures: read performance 37MB/s. Write performance 20MB/s. Write speed of 17MB/s has been achieved in this test at symbianbriefs.blogspot.com. My own read tests however show ~11MB/s (370MB file).
• Decisions were taken in a "what shall we do about /opt?" BOF at the Maemo Summit 2009. Minutes: http://lists.maemo.org/pipermail/maemo-developers/2009-October/021289.html

Some goals

• Firmware should be flashable.
• Should not break applications.
• One should be able to install more than 256mb worth of software.
• Standards compliance, less need to make incompatible quirks.
• A fix should be doable after shipping (before shipping, probably not?)

Current problems / developer and packaging issues

This should be a list of current problems developers are experiencing related to optification:

• Optification breaks package on upgrade from non-optified older version

• Symlinks take up space in the rootfs. This may become an issue if there are lots of them.

• List of non-optified packages and applications - Please feel free to add every piece of software which is using too much space in rootfs

Tool for detection of non-optified packages

Storage Usage provides a great visual way to detect non-optified packages.

Possible solutions, even the bad ones

Applications are made relocatable and installed to /opt

• Relocatable applications can move everywhere without breaking.
• No symlinks get scattered over the rootfs (symlinks take up space, too).
• Not all applications can easily be made relocatable.
• Requires more work by the programmer.
• May not be feasible for ports from an upstream source.
• Putting a startup script into /usr/bin setting up environment variables such as LD_LIBRARY_PATH can help.
• autotools-based packages can use --prefix=/opt/<packagename>.
• Python apps can make use of the __file__ module-scope variable to determine where they are installed (e.g. installdir = os.path.dirname(__file__)).
• Could be taken into account as an alternative to maemo-optify wherever feasible.

Have a 2-5gb partition on the emmc for addon packages mounted to /usr/local or /opt

• Most apps by default usually go there
• for /opt store according to the FHS standard /opt/<package> and symlink to /opt/maemo/{bin,lib,share,...} (actually the standard is /opt/<vendor>/whatever In this case using /opt/maemo directly would work).
• Having ld.so.conf and PATH set properly avoids all the symlink madness
• Don't do symlinks for /usr/local|/opt - make it a real partition
• Explain to people why they don't have full 32GB of storage - or use LVM on the eMMC to dynamicaly change the size as things get installed through some dpkg hooks or such - we do know how much it will take ahead of time so if there isn't enough storage do a resize(VFAT should be resizable with something like gparted maybe?) ext3/2 are both online resizeable (growth only!)
• Requires a bit more work when packaging - make some defaults pre-set in SDK
• All apps should use a normal usr layout underneath any path. Example: /opt/maemo/{bin,sbin,lib, etc...} same if it would be in /usr/local - works like GNU stow

store addon packages completely in /home/opt hierarchy

• see this page
• like previous solution but on the standard /home partition
• very clean solution as no files (except for /var/lib/dpkg which may be moved as well) are created or modified on the NAND.
• /etc and /var could be copied there as well and mounted to root using bind. This way OneNAND would only be changed be reflashes and if /home is not mounted (e.g., by a keypress during boot) a fallback system could boot from unmodified OneNAND.
• if OneNAND is significantly faster, mark some often used shared libraries as candidates for a /usr/lib/cache directory in which they are copied during installation until a certain cache limit is reached.

During packaging move all files >500kb to /opt and symlink to root

• Optify is basically this.
• Install is not slowed down, work is done when packaging.
• Some packages break when replaced with symlinks. How to prevent - hack on hack?
• A buildbot could force this to debian/rules automatically.

Use aufs/unionfs to overlay OneNAND root with eMMC filesystem.

• This could be partial overlay, for example only /lib and /usr.
• Could keep essential libraries on OneNAND so they would work faster.
• Has anyone measured the speed penalty for aufs/unionfs usage?
• Is it faster or slower that moving root to eMMC?
• Might be difficult to have new programs to benefit from faster OneNAND.

Hack with --datadir, --libdir etc. through debian/rules

• Could be forced by a buildbot.
• Should move most of the big lumps on larger media if there are some.
• Works at least with packages with configure.
• Requires at least additional $LIBDIR setup?
• Programs which have hardcoded paths (ugly code) break.
• Could additionally symlink files from opt to root.

Require that applications have --prefix /opt

• Could be forced by a buildbot.
• Should not break anything.
• Programs would not benefit from faster OneNAND.
• Requires at least additional $PATH setup?
• Programs which have hardcoded paths (ugly code) break.
• Could additionally symlink files from opt to root.
• Just reminds me of 770's /var/lib/install..
• Against standard

Move root from OneNAND to eMMC and combine with /home to create a new root

• How much faster OneNAND is? Numbers?
• Can flasher flash on eMMC?
• Leave only the bootable and rescue stuff in the rootfs
• Maybe use it uncompressed for speed and add a /fast/usr folder to OneNAND to store some system files that would benefit from the speed (example: move /usr/something to /fast/usr and symlink it or just place /fast/usr in the path)

Use mount -o bind (or -o rbind)

• Method: Do not use NAND for any additional data. Boot into standard image. Any additional changes made are only available after -o bind or -o rbind which are defined in /etc/fstab . Setup provides potential fail-over; if -o bind or -o rbind fails user gets into 'rescue shell' providing 1) recovery method 2) quick install to vanilla firmware without 3rd party utilities, networking, or computer (think: no network available, expensive roaming, no computer with fwflasher or Nokia Software Downloader available, ...)

Turn the problem around, pivot the root with /opt

• After pivot /opt would be root and old root would be /opt. Symlink the old root contents to new root would make those available.
• Can the base system dependencies handle this?
• Can Nokia created software handle this?
• Could this work? If so, then all additional packages would install directly on eMMC.

After install move all files >500kb to /opt and symlink to root

• Could be done at background after install or with post-install hook.
• There might not be enough space on / even for install.
• Install is already slow, could make it even slower.
• Some packages break when replaced with symlinks. How to prevent - hack on hack?
• Uninstall might be messy, some post-remove magic needed.

Chroot install in /opt, symlink to where it belongs

• A Bit like stow.
• Maybe somebody can see something good in this?
• Difficult hack.. would need all the tools etc to be visible in the chroot.

Make /usr another partition

You know, historically /usr exists entirely for the purpose of separating the application partition from the underlying system, which is exactly our problem.

• All the core nokia software and libraries should be relocated to / or maybe a /maemo.
• /usr and /opt should become symlinks to /home/usr and /home/opt.
• Maemo targeted GUI software would be encouraged to use /opt/[application].
• Linux libraries and CLI software could safely use /usr, minimizing the required modifications.
• Any frequently used accessed files in /home but not /home/user/MyDocs could be cached in /cache, which then gets overlaid with /home using unionfs or something more specialized.
• Another location like /alt could be used by packages that know they need the NAND's speed, but this is discouraged.