Setting up overlayFS on Raspberry Pi
Warning: Do this only on a new SD image that can be recreated, or make sure you can mount the SD card elsewhere to revert changes.
Recent changes
Now rewritten for systemd / systemctl services only. You can still use /etc/init.d/saveoverlays as a traditional service if you prefer, but this guide now describes using it with syncoverlayfs as the systemd service unit name.
There is now support for syncing high priority files before everything else. Refer to configuration section for details
Previous changes
Please note that overlays are now synced via an intermediate / staging directory to the underlay, i.e. /home is now synced to /home_stage before being synced to /home_org. The reason for this is that changing the underlay gives unpredictable results for the overlay view (per the design document on kernel.org).
- Before doing so, a list of processes / services are first killed if found. By default domoticz and mosquitto (change as you please). This is to ensure valid / stable data are being synced in case you do a service saveoverlays sync. For a normal restart, these services are stopped before the sync takes place.
- After sync, the system must be immediately rebooted without further syncs (keep root partition mounted read/write , then reboot)
- Do not copy files directly from the overlay to any of the underlays
- Remarks for Jessie also works for Stretch
... also, the source files for setting this up are now on github (finally)
Purpose
In this page, we will describe how to implement overlayfs using the normal SD card directory tree as the bottom (persistent) layer, and as many RAM / tmpfs areas you may want as the upper layer to provide the overlay mounts. In most cases this will do:
- /boot and the root filesystem are changed to be normally read only
- /var and /home is changed into read/write overlay filesystems
- The syncoverlayfs service will take care of syncing the overlay filesystems back to the bottom layer SD card upon system shutdown and restarts.
Using this, you can eliminate SD card writes almost entirely. As the read-write parts are then RAM based, this will also speed up Domoticz.
Tip: On Raspian / Raspberry pi, the ext4 filesystem write activity can be listed as follows:
cat /sys/fs/ext4/mmcblk0p2/lifetime_write_kbytes
Preparations
Requirements
OverlayFS is included in kernel versions 3.18 and later.
In case of errors, it is strongly recommended to be able to mount and edit the modified SD card contents via a card reader or similar on another system
Installation
The easiest way to get the needed scripts into place is this:
cd /tmp wget https://github.com/hansrune/domoticz-contrib/raw/master/utils/mount_overlay wget https://github.com/hansrune/domoticz-contrib/raw/master/init.d/saveoverlays-stretch wget https://github.com/hansrune/domoticz-contrib/raw/master/utils/rootro wget https://github.com/hansrune/domoticz-contrib/raw/master/init.d/syncoverlayfs.service mv saveoverlays-stretch saveoverlays chmod a+rx saveoverlays mount_overlay rootro sudo cp mount_overlay /usr/local/bin/ sudo cp saveoverlays /etc/init.d/ sudo cp rootro /usr/local/bin/ sudo ln -s rootro /usr/local/bin/rootrw sudo cp syncoverlayfs.service /lib/systemd/system/ # For raspian Jessie, stretch and later, also do sudo systemctl disable dphys-swapfile
Configuration
In /lib/systemd/system/syncoverlayfs.service, make sure that RequiresMountsFor contains all overlayfs mounts
Optionally, create /etc/default/saveoverlays with parameters like these defaults:
KILLPROCS="domoticz mosquitto" # processes to kill before sync, just in case still running PRIORITYFILES="/var/log/fake-hwclock.data /var/lib/domoticz/domoticz.db" # files to sync before everything else
Backup and recovery
In case of errors, it is a good idea to backup the below. You can then mount the modified SD card on another machine.
sudo cp -v /boot/cmdline.txt /boot/cmdline.txt-orig sudo cp -v /etc/fstab /etc/fstab-orig cd /var sudo tar --exclude=swap -cf /var-orig.tar .
Setup procedure
Stop using swap
In case you also use the SD card for swap, this should be turned off as follows:
sudo dphys-swapfile swapoff sudo dphys-swapfile uninstall sudo update-rc.d dphys-swapfile disable # For raspian Jessie and later, also do sudo systemctl disable dphys-swapfile
Move files you need to be able to update with a read-only root
Some files may need to be continously updated after all changes are done. For example, if you use the fake-hwclock service, the data file for that service need to be moved:
sudo systemctl disable fake-hwclock.service sudo systemctl stop fake-hwclock.service sudo mv /etc/fake-hwclock.data /var/log/fake-hwclock.data sudo ln -s /var/log/fake-hwclock.data /etc/fake-hwclock.data
The fake-hwclock actions are taken care of in the saveoverlays service described later. There is no harm in leaving it as is, but it will complain during start and do nothing.
Set up fuse and mount script
Install the fuse package. We need only /sbin/mount.fuse from there. lsof is essential to find open files in case you cannot remount a filesystem back to read-only
sudo apt-get install fuse lsof
A mount script or program specified in /etc/fstab to do the actual overlay mount. When the filesystem type is fuse, the first /etc/fstab field is used for the name of the program to do the mount operation. In our case we use /usr/local/bin/mount_overlay to do the mount.
Please note that if the root filesystem is read-write in /etc/fstab, the mount will be a loopback mount instead.
Set up the syncoverlayfs service
This syncoverlayfs service will take care of saving changes from the overlays back to the underlying SD card storage (now the _org part of the overlay). It also contains some logic to check if the fake clock data should be loaded. If you are not using the fake-hwclock service, you should comment that out (two places).
The overlays are automatically detected, so you should not need to change any of that.
Enable the service as follows:
sudo systemctl daemon-reload sudo systemctl enable syncoverlayfs.service
The syncoverlayfs service still uses /etc/init.d/saveoverlays script to do the actual syncing, The naming difference is to make sure they are not mixed up.
Note: If you have remounted root to be read-write, this service assumes that changes are not to be synced. Please remount back to read-only if you want changes to be synced on next reboot.
Change boot command line
Change /boot/cmdline.txt similar to this, i.e. add noswap fastboot ro. In Raspbian Stretch root=/dev/mmcblk0p2 is different and looks like: root=PARTUUID=badee776-02
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait noswap fastboot ro
Update /etc/fstab for read-only SD card
/etc/fstab need to be updated similar to this:
proc /proc proc defaults 0 0 /dev/mmcblk0p1 /boot vfat ro 0 2 /dev/mmcblk0p2 / ext4 ro,noatime 0 1 mount_overlay /var fuse nofail,defaults 0 0 mount_overlay /home fuse nofail,defaults 0 0 none /tmp tmpfs defaults 0 0
/etc/fstab need to be updated similar to this only for Jessie . Stretch works fine with the simpler fstab setup above (provided the service unit uses RequiresMountsFor
proc /proc proc defaults 0 0 /dev/mmcblk0p1 /boot vfat ro 0 2 /dev/mmcblk0p2 / ext4 ro,noatime 0 1 mount_overlay /var fuse nofail,defaults,x-systemd.automount,x-systemd.requires=/var,x-systemd.device-timeout=10s 0 0 mount_overlay /home fuse nofail,defaults,x-systemd.automount,x-systemd.requires=/home,x-systemd.device-timeout=10s 0 0 none /tmp tmpfs defaults 0 0
Stop services
These services may have files open or may otherwise disturb the changes to be made
for S in nginx domoticz fail2ban shellinabox mosquitto cron ntp nodered rsyslog influxdb; do sudo systemctl stop $S; done
Prepare the overlay directories
Then change into layered mount setups as follows (as user root):
sudo su -
for D in /home /var
do
mv -v ${D} ${D}_org
cd ${D}_org
find . | cpio -pdum ${D}_stage
mkdir -v ${D} ${D}_rw ${D}/.overlaysync ${D}_org/.overlaysync
done
exit
The .overlaysync subdirs are added and tested for during sync to make sure only relevant directories are being synced. If these are missing, you will see messages like
Skipping this step: /var/.overlaysync or /var_org/.overlaysync> or /var_stage not available.
Loopback mount /home and /var before boot
This is to make sure the next reboot does not run into trouble due to moved data (rsync with delete). When root is not read-only, this will result in /var_org being loopback mounted to /var, and /home_orig loopback mounted to /home
sudo mount /home sudo mount /var
Modify domoticz service
For some reason the domoticz executable is mistaken on startup directory when in an overlay filesystem.
To get around that, add the following after other DAEMON_ARGS lines in /etc/init.d/domoticz.sh or similar in the systemctl service specification
APPDIR=$( dirname $DAEMON ) DAEMON_ARGS="$DAEMON_ARGS -approot $APPDIR/ -wwwroot $APPDIR/www -dbase $APPDIR/domoticz.db"
The trailing slash for approot is needed.
Reboot
Now reboot. Since root is not mounted read-only at this time, the saveoverlays service will not attempt to sync anything.
It is recommended to have a console attached so that any error messages can be detected, and problems dealed with.
Tests
Check your mounts
After boot, the mount command should display similar to this, with relevant changes in red
/dev/root on / type ext4 (ro,noatime,data=ordered) devtmpfs on /dev type devtmpfs (rw,relatime,size=234492k,nr_inodes=58623,mode=755) tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=47756k,mode=755) tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=95500k) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) /dev/mmcblk0p1 on /boot type vfat (ro,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro) ramdisk on /var_rw type tmpfs (rw,relatime) overlay on /var type overlay (rw,relatime,lowerdir=/var_org,upperdir=/var_rw/upper,workdir=/var_rw/work) ramdisk on /home_rw type tmpfs (rw,relatime) overlay on /home type overlay (rw,relatime,lowerdir=/home_org,upperdir=/home_rw/upper,workdir=/home_rw/work) none on /tmp type tmpfs (rw,relatime)
Check that you can update the system
For most updates, the following will do
rootrw sudo apt-get update sudo apt-get upgrade rootro sudo init 6
In case there are more updates than space available in /var, upgrade the packages in steps by repeating this:
rootrw sudo apt-get update sudo apt-get upgrade -s # this will show what packages need to upgrade sudo apt-get install --only-upgrade <packagenames ...> # cut & paste package names rootro sudo init 6
If you cannot remount read-only
In case you cannot remount to read-only, you need to to a manual sync of the overlay filesystem. This can happen when updates restarts services.
sudo /etc/init.d/saveoverlays sync sudo reboot
Note: When the root partition is mounted read/write, a stop of the syncoverlayfs service does nothing.
Check that overlays are synced to SD card
These syncs are logged to /var/log/saveoverlays.log
References
There are many references for doing similar setups. The most similar one using unionfs is likely this:
Other material:
- OverlayFS on wikipedia
- OverlayFS design