Moving Log and Temporary Files To a RAM Drive

From Domoticz
Jump to navigation Jump to search

Objective:

Move most frequently written files to a ram drive to reduce SD Card wear. This article will focus on the domoticz log and some temporary files as well as the openzwave log, that we will have reside in a RAM drive mounted as a new directory /tmp/log.
With respect to the size of the RAM drive, 20MB is used as a standard trade off considering the usual size of the domoticz and openzwave temporary files, but this may have to be adjusted up or down based on the particular setup.
With respect to Domoticz, besides the log file, there are two files that can be moved to the RAM drive:

Please be aware of the following risk involved in moving the Write Ahead Log file to RAM.
SQLite normally stores all content in a single disk file. However, while performing a transaction, information necessary to recover the database following a crash or power failure is stored in auxiliary journal files. Such journal files are described as "hot". The journal files have the same name as the original database file with the addition of -journal or -wal suffix.
SQLite must see the journal files in order to recover from a crash or power failure. If the hot journal files are moved, deleted, or renamed after a crash or power failure, then automatic recovery will not work and the database may go corrupt.
Source: sqlite.org

/home/pi/domoticz/domoticz.db-wal  !! This will no longer work for domoticz version V4.11753 and newer
/home/pi/domoticz/domoticz.db-shm  !! due to an update of the internal sqlite version

These are SQLite temporary files

The write-ahead log or WAL file is used as a rollback journal and created when the first connection to the database is opened and is normally removed when the last connection to the database closes. However, if the last connection does not shutdown cleanly, the WAL file will remain in the filesystem for recovery purposes. The shared memory or SHM file is used as an index for the WAL file.

It might be possible to have Domoticz configured to change the location of these files, but for simplicity purposes, it is just possible to create symbolic links to the RAM drive mount point… Unfortunately, it seems that these links are deleted on Domoticz's shutdown, so a solution is to modify the domoticz.sh startup script to create these links each time Domoticz is started.
Requisite: SSH access to the RPi system and decent knowledge of the bash shell, including the nano editor (or any other of your choice). It is also assumed that you are user “pi” and domoticz is installed in /home/pi/domoticz

This article is based on several sources to be found on this wiki or the internet, including: Setting_up_a_RAM_drive_on_Raspberry_Pi

ALL THIS IS DONE AT YOUR OWN RISK AND ASSUMES YOU KNOW WHAT YOU ARE DOING. YOU MAY BRICK YOUR SYSTEM OR DOMOTICZ INSTALL.


STEP 1 – Create the RAM disk and mount it
Create the mount point: 

mkdir /tmp/log

Make a backup copy of the system file we are going to edit, in case anything bad happens with the edit 

sudo cp -p /etc/fstab  /etc/fstab.save

Edit the /etc/fstab file: 

sudo nano /etc/fstab

and add the following line: 

tmpfs	        /tmp/log        tmpfs   nodev,nosuid,size=20M 0 0

then save, quit the editor and mount the RAM drive: 

sudo mount /tmp/log

Check the file system 

df -h

You should now see among the mount points something similar to this line 

tmpfs               20M    3,9M   17M  20% /tmp/log


STEP 2 – change Domoticz log and move temporary files to the newly created mount point:

Stop domoticz: 

sudo /etc/init.d/domoticz.sh stop

or 

sudo systemctl stop domoticz.service

Edit the /etc/init.d/domoticz.sh script file: 

sudo nano /etc/init.d/domoticz.sh

and edit the domoticz deamon logging parameters (default at system setup being #DAEMON_ARGS="$DAEMON_ARGS -log /tmp/domoticz.txt", also make sure you uncomment that line by removing the leading # character!) to now show: 

DAEMON_ARGS="$DAEMON_ARGS -log /tmp/log/domoticz.log -loglevel normal"

Note that the –loglevel parameter above can be omitted or obviously changed to include more or less levels of information like 

 DAEMON_ARGS="$DAEMON_ARGS -log /tmp/log/domoticz.log -loglevel normal,status,error,debug"

or 

 DAEMON_ARGS="$DAEMON_ARGS -log /tmp/log/domoticz.log -loglevel normal,status,error"

(you get the picture) if you want…

Now in light of the above comments on the SQLite temporary files, you need to edit the “start” section of the script to add a few lines as per below in bold: 

#
# Function that starts the daemon/service
#
do_start()
{
       # edit to move SQLite temp files to RAM drive
       ln -sf /tmp/log/domoticz.db-shm /home/pi/domoticz
       ln -sf /tmp/log/domoticz.db-wal /home/pi/domoticz
       # end of edit
       # Return
       #   0 if daemon has been started
       #   1 if daemon was already running
       #   2 if daemon could not be started
       start-stop-daemon --chuid $USERNAME --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null \
               || return 1
       start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON -- \
               $DAEMON_ARGS \
               || return 2
}

Then save and quit the editor


STEP 3 – Only if you are using a zwave network and wish openzwave to log to the RAM drive.

Note that you will lose openwave logs in case of a system reboot if you do this. The reason I decided to do this is that I noted that when a few of my zwave devices recently failed, openwave logs a deluge of data when it polls the dead devices…

While domoticz is still stopped, 

sudo rm /home/pi/domoticz/Config/OZW_Log.txt
ln -sf /tmp/log/OZW_Log.txt /home/pi/domoticz/Config


STEP 4: Apply the changes

Restart Domoticz: 

sudo /etc/init.d/domoticz.sh start

or 

sudo systemctl start domoticz.service

if prompted to do a systemctl daemon-reload you should do it…

You should now see all the relevant files in your /tmp/log/ directory… Here is an example: 

ls /tmp/log -l

-rw-r--r-- 1 pi   pi     32768 mars   8 11:31 domoticz.db-shm
-rw-r--r-- 1 pi   pi   5236552 mars  13 10:59 domoticz.db-wal
-rwxrwxrwx 1 root root    1909 mars  13 10:19 domoticz.log
-rw-r----- 1 root root 1069621 mars  11 10:55 OZW_Log.txt

Last point… I did not address the size issues… overtime, these files may grow too big for the RAM drive… It is not an issue for me as I am rotating these logs daily as described at [1] (in French but may be an English version exists somewhere), but you could otherwise be faced with a disk full issue if you do not maintain the logs size somehow… One way of doing that is copy the current logfile to safe place and truncate the log back to zero with the command 

sudo truncate -s 0 /tmp/log/domoticz.log     # on Linux
Retrieved from "https://wiki.domoticz.com/index.php?title=Moving_Log_and_Temporary_Files_To_a_RAM_Drive&oldid=14028"