Setting up overlayFS on Raspberry Pi

From Domoticz
Revision as of 14:10, 8 September 2018 by Hansrune (talk | contribs) (→‎Check that you can update the system)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Warning: Do this only on a new SD image that can be recreated, or make sure you can mount the SD card elsewhere to revert changes.

Recent changes

Now rewritten for systemd / systemctl services only. You can still use /etc/init.d/saveoverlays as a traditional service if you prefer, but this guide now describes using it with syncoverlayfs as the systemd service unit name.

There is now support for syncing high priority files before everything else. Refer to configuration section for details

Previous changes

Please note that overlays are now synced via an intermediate / staging directory to the underlay, i.e. /home is now synced to /home_stage before being synced to /home_org. The reason for this is that changing the underlay gives unpredictable results for the overlay view (per the design document on kernel.org).

  • Before doing so, a list of processes / services are first killed if found. By default domoticz and mosquitto (change as you please). This is to ensure valid / stable data are being synced in case you do a service saveoverlays sync. For a normal restart, these services are stopped before the sync takes place.
  • After sync, the system must be immediately rebooted without further syncs (keep root partition mounted read/write , then reboot)
  • Do not copy files directly from the overlay to any of the underlays
  • Remarks for Jessie also works for Stretch

... also, the source files for setting this up are now on github (finally)

Purpose

In this page, we will describe how to implement overlayfs using the normal SD card directory tree as the bottom (persistent) layer, and as many RAM / tmpfs areas you may want as the upper layer to provide the overlay mounts. In most cases this will do:

  • /boot and the root filesystem are changed to be normally read only
  • /var and /home is changed into read/write overlay filesystems
  • The syncoverlayfs service will take care of syncing the overlay filesystems back to the bottom layer SD card upon system shutdown and restarts.

Using this, you can eliminate SD card writes almost entirely. As the read-write parts are then RAM based, this will also speed up Domoticz.

Tip: On Raspian / Raspberry pi, the ext4 filesystem write activity can be listed as follows: 

cat /sys/fs/ext4/mmcblk0p2/lifetime_write_kbytes

Preparations

Requirements

OverlayFS is included in kernel versions 3.18 and later.

In case of errors, it is strongly recommended to be able to mount and edit the modified SD card contents via a card reader or similar on another system

Installation

The easiest way to get the needed scripts into place is this: 

cd /tmp
wget https://github.com/hansrune/domoticz-contrib/raw/master/utils/mount_overlay
wget https://github.com/hansrune/domoticz-contrib/raw/master/init.d/saveoverlays-stretch
wget https://github.com/hansrune/domoticz-contrib/raw/master/utils/rootro
wget https://github.com/hansrune/domoticz-contrib/raw/master/init.d/syncoverlayfs.service
mv saveoverlays-stretch saveoverlays
chmod a+rx saveoverlays mount_overlay rootro
sudo cp mount_overlay /usr/local/bin/
sudo cp saveoverlays /etc/init.d/
sudo cp rootro /usr/local/bin/
sudo ln -s rootro /usr/local/bin/rootrw
sudo cp syncoverlayfs.service /lib/systemd/system/
# For raspian Jessie, stretch and later, also do
sudo systemctl disable dphys-swapfile

Configuration

In /lib/systemd/system/syncoverlayfs.service, make sure that RequiresMountsFor contains all overlayfs mounts

Optionally, create /etc/default/saveoverlays with parameters like these defaults: 

 KILLPROCS="domoticz mosquitto" # processes to kill before sync, just in case still running
 PRIORITYFILES="/var/log/fake-hwclock.data /var/lib/domoticz/domoticz.db" # files to sync before everything else

Backup and recovery

In case of errors, it is a good idea to backup the below. You can then mount the modified SD card on another machine. 

sudo cp -v /boot/cmdline.txt /boot/cmdline.txt-orig
sudo cp -v /etc/fstab /etc/fstab-orig
cd /var
sudo tar --exclude=swap -cf /var-orig.tar .

Setup procedure

Stop using swap

In case you also use the SD card for swap, this should be turned off as follows: 

sudo dphys-swapfile swapoff
sudo dphys-swapfile uninstall 
sudo update-rc.d dphys-swapfile disable
# For raspian Jessie and later, also do
sudo systemctl disable dphys-swapfile

Move files you need to be able to update with a read-only root

Some files may need to be continously updated after all changes are done. For example, if you use the fake-hwclock service, the data file for that service need to be moved: 

sudo systemctl disable fake-hwclock.service
sudo systemctl stop fake-hwclock.service
sudo mv /etc/fake-hwclock.data /var/log/fake-hwclock.data
sudo ln -s /var/log/fake-hwclock.data /etc/fake-hwclock.data

The fake-hwclock actions are taken care of in the saveoverlays service described later. There is no harm in leaving it as is, but it will complain during start and do nothing.

Set up fuse and mount script

Install the fuse package. We need only /sbin/mount.fuse from there. lsof is essential to find open files in case you cannot remount a filesystem back to read-only 

sudo apt-get install fuse lsof

A mount script or program specified in /etc/fstab to do the actual overlay mount. When the filesystem type is fuse, the first /etc/fstab field is used for the name of the program to do the mount operation. In our case we use /usr/local/bin/mount_overlay to do the mount.

Please note that if the root filesystem is read-write in /etc/fstab, the mount will be a loopback mount instead.

Set up the syncoverlayfs service

This syncoverlayfs service will take care of saving changes from the overlays back to the underlying SD card storage (now the _org part of the overlay). It also contains some logic to check if the fake clock data should be loaded. If you are not using the fake-hwclock service, you should comment that out (two places).

The overlays are automatically detected, so you should not need to change any of that.

Enable the service as follows: 

sudo systemctl daemon-reload
sudo systemctl enable syncoverlayfs.service

The syncoverlayfs service still uses /etc/init.d/saveoverlays script to do the actual syncing, The naming difference is to make sure they are not mixed up.

Note: If you have remounted root to be read-write, this service assumes that changes are not to be synced. Please remount back to read-only if you want changes to be synced on next reboot.

Change boot command line

Change /boot/cmdline.txt similar to this, i.e. add noswap fastboot ro. In Raspbian Stretch root=/dev/mmcblk0p2 is different and looks like: root=PARTUUID=badee776-02 

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait noswap fastboot ro

Update /etc/fstab for read-only SD card

/etc/fstab need to be updated similar to this: 

proc            /proc           proc    defaults    0 0
/dev/mmcblk0p1  /boot           vfat    ro          0 2
/dev/mmcblk0p2  /               ext4    ro,noatime  0 1
mount_overlay   /var            fuse    nofail,defaults 0 0
mount_overlay   /home           fuse    nofail,defaults 0 0
none            /tmp            tmpfs   defaults    0 0


/etc/fstab need to be updated similar to this only for Jessie . Stretch works fine with the simpler fstab setup above (provided the service unit uses RequiresMountsFor 

proc            /proc           proc    defaults    0 0
/dev/mmcblk0p1  /boot           vfat    ro          0 2
/dev/mmcblk0p2  /               ext4    ro,noatime  0 1
mount_overlay   /var            fuse    nofail,defaults,x-systemd.automount,x-systemd.requires=/var,x-systemd.device-timeout=10s 0 0
mount_overlay   /home           fuse    nofail,defaults,x-systemd.automount,x-systemd.requires=/home,x-systemd.device-timeout=10s 0 0
none            /tmp            tmpfs   defaults    0 0


Stop services These services may have files open or may otherwise disturb the changes to be made 

for S in nginx domoticz fail2ban shellinabox mosquitto cron ntp nodered rsyslog influxdb; do sudo systemctl stop $S; done

Prepare the overlay directories

Then change into layered mount setups as follows (as user root): 

sudo su - 
for D in /home /var
do
  mv -v ${D} ${D}_org
  cd ${D}_org
  find . | cpio -pdum ${D}_stage
  mkdir -v ${D} ${D}_rw ${D}/.overlaysync ${D}_org/.overlaysync
done
exit

The .overlaysync subdirs are added and tested for during sync to make sure only relevant directories are being synced. If these are missing, you will see messages like 

 Skipping this step: /var/.overlaysync or /var_org/.overlaysync> or /var_stage not available.

Loopback mount /home and /var before boot

This is to make sure the next reboot does not run into trouble due to moved data (rsync with delete). When root is not read-only, this will result in /var_org being loopback mounted to /var, and /home_orig loopback mounted to /home 

sudo mount /home
sudo mount /var

Modify domoticz service

For some reason the domoticz executable is mistaken on startup directory when in an overlay filesystem.

To get around that, add the following after other DAEMON_ARGS lines in /etc/init.d/domoticz.sh or similar in the systemctl service specification 

APPDIR=$( dirname $DAEMON )
DAEMON_ARGS="$DAEMON_ARGS -approot $APPDIR/ -wwwroot $APPDIR/www  -dbase $APPDIR/domoticz.db"

The trailing slash for approot is needed.

Reboot

Now reboot. Since root is not mounted read-only at this time, the saveoverlays service will not attempt to sync anything.

It is recommended to have a console attached so that any error messages can be detected, and problems dealed with.

Tests

Check your mounts

After boot, the mount command should display similar to this, with relevant changes in red 

/dev/root on / type ext4 (ro,noatime,data=ordered) 
devtmpfs on /dev type devtmpfs (rw,relatime,size=234492k,nr_inodes=58623,mode=755)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=47756k,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) 
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=95500k)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
/dev/mmcblk0p1 on /boot type vfat (ro,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
ramdisk on /var_rw type tmpfs (rw,relatime)
overlay on /var type overlay (rw,relatime,lowerdir=/var_org,upperdir=/var_rw/upper,workdir=/var_rw/work)
ramdisk on /home_rw type tmpfs (rw,relatime) 
overlay on /home type overlay (rw,relatime,lowerdir=/home_org,upperdir=/home_rw/upper,workdir=/home_rw/work)
none on /tmp type tmpfs (rw,relatime)


Check that you can update the system

For most updates, the following will do 

rootrw
sudo apt-get update
sudo apt-get upgrade
rootro
sudo init 6

In case there are more updates than space available in /var, upgrade the packages in steps by repeating this: 

rootrw
sudo apt-get update
sudo apt-get upgrade -s # this will show what packages need to upgrade
sudo apt-get install --only-upgrade <packagenames ...> # cut & paste package names
rootro
sudo init 6

If you cannot remount read-only

In case you cannot remount to read-only, you need to to a manual sync of the overlay filesystem. This can happen when updates restarts services. 

sudo /etc/init.d/saveoverlays sync
sudo reboot

Note: When the root partition is mounted read/write, a stop of the syncoverlayfs service does nothing.

Check that overlays are synced to SD card

These syncs are logged to /var/log/saveoverlays.log

References

There are many references for doing similar setups. The most similar one using unionfs is likely this:

Other material:

Retrieved from "https://wiki.domoticz.com/index.php?title=Setting_up_overlayFS_on_Raspberry_Pi&oldid=11407"