Monitoring domoticz
Software crashes, it's a fact of life. Luckily there are a number of ways to keep an eye on Domoticz, and automaticallty restart it if it crashes.
Generally speaking there are two ways of doing this: via software designed to do this (Monit), or through a custom script that you run periodically. Both have advantages and disadvantages.
- Monit is robust, but may not always detect an actual crash.
- A script can test if Domoticz is actually ok, but might create problems itself.
Using Monit
Monit is a helpful program that automatically monitors and manages server programs to ensure that they not only stay online consistently, but that the file size, checksum, or permissions are always correct. Additionally Monit comes with a basic web interface through which all of the processes can be set up. This tutorial will cover the most basic setup and configuration to ensure Domoticz running 24/7.
Install Monit
Monit is easy installed through apt-get:
sudo apt-get install monit
Once Monit has been installed, you can add programs and processes that you want to have checked, to the configuration file:
sudo nano /etc/monit/monitrc
Configure Monit
Monit is very easy to use nearly out of the box. By default, it is set up to check that services are running every 2 minutes and stores its log file in "/var/log/monit.log". (I have changed this to my ramdisk check Wiki)
sudo nano /etc/monit/monitrc
Modify the following parameters:
- set daemon 300 -> Check services at 300 seconds (5 minutes). Be sure you leave enough time to do automated database backups and updates.
- with start delay 300 -> Start checking the services after 300 seconds (this avoids bootloops if Domoticz has not started yet, give it some time to start fully)
- set logfile /var/log/monit.log -> It is preferred to change the location of the logfile to a mounted pen drive or HDD. Or store it in a RAM-drive.
Configure e-mail notifications
You can setup Monit to inform you about every status update by e-mail. I would recommend doing this, so you have a good view on stability of your system (how often does Monit needs to kick in actually?).
Gmail
set mailserver smtp.gmail.com port 587 username "[email protected]" password "MYPASSWORD" using tls with timeout 30 seconds set alert [email protected]
Note: For monit <= 5.16: use tlsv12 instead of tls
If you use Gmail with 2-step verification, you first have to create an App password
POP3
set mailserver smtp.yourprovider.com port 25
with timeout 30 seconds
set mail-format { from: [email protected] }
set alert [email protected]
To test if Monit can send you mails, issue a sudo monit reload and you should get a mail that the instance has changed.
Webservice
Monit comes with its own webserver, running on port 2812. To configure the webinterface, find and uncomment the section that begins with set httpd port 2812. Once the section is uncommented, fill in the IP-address of the machine where you have Domoticz running at, allow anyone to connect, and then create a monit user and password
set httpd port 2812 use address 0.0.0.0 # only accept connection from localhost (comment to connect from other hosts) allow 0.0.0.0/0.0.0.0 # allow anybody to connect to the server and allow admin:monit # require user 'admin' with password 'monit', change if your system is accessible from internet # allow 192.168.0.1/255.255.255.0 #This will only allow hosts coming from the local network (change 192.168.0.1 to your own LAN IP-range). # allow localhost #use in combination with local network to allow command line to work
Once this is configured, you should reload monit:
sudo service monit reload
You will then be able to access the monit web interface by going to “192.168.1.2:2812” (change to IP-adress of your Domoticz system) Login with the username and password you configured earlier in the configfile (admin/monit ?). Your screen should look something like this.
Configuring monitoring of programs
Once the web services are set up, you can begin to input the programs that you want monitored and protected into the "/etc/monit/monitrc" configuration file. To simply ensure that programs stay online, you can use the /etc/init.d commands to stop or start a program. Here are some example configurations:
Domoticz
check process domoticz with pidfile /var/run/domoticz.pid
start program = "/usr/bin/sudo /bin/systemctl start domoticz.service"
stop program = "/usr/bin/sudo /bin/systemctl stop domoticz.service"
# use below start/stop command for older systems not using systemctl
# start program = "/etc/init.d/domoticz.sh start"
# stop program = "/etc/init.d/domoticz.sh stop"
if failed
url http://127.0.0.1:8080/json.htm?type=command¶m=getversion
and content = '"status" : "OK"'
for 2 cycles
then restart
if 5 restarts within 5 cycles then exec "/sbin/reboot"
- Remark: if the service is not found, check if the response from Domoticz still exactly contains the string "state": "OK"
To check for CPU-usage, include the line if cpu usage > 70% for 3 cycles then restart for example (put it between the 'then restart' and the last 'if 5 restarts..' line).
Once you have set up the configuration, check the syntax:
sudo monit -t
After resolving any possible syntax errors, you can start running all of the monitored programs.
sudo monit start all
data collected Thu, 30 Aug 2012 18:35:00
Monit can be started with a command that then keeps it running in the background
sudo service monit start
Typing the command below displays monit’s status
sudo monit status
This shows us the details:
The Monit daemon 5.3.2 uptime: 1h 25m System 'myhost.mydomain.tld' status Running monitoring status Monitored load average [0.03] [0.14] [0.20] cpu 3.5%us 5.9%sy 0.0%wa memory usage 26100 kB [10.4%] swap usage 0 kB [0.0%] data collected Thu, 30 Aug 2012 18:35:00
Finish up
Once you have configured all of the above steps, Domoticz will be tracked constantly and the process will be restarted automatically if it would have stopped running because of <insert a reason>.
Notes
Pay attention when monitoring CPU-usage of process
When you want to monitor the CPU-usage of a single process (instead of the whole machine), be warned about this: http://www.brandonchecketts.com/archives/monit-cpu-usage-problem
Basically, monit sees the usage spreaded on all processorcores.
So if you want to restart a process when it consumes 100% CPU or more, you need to divide 100 by the amount of cores you have.
If you have a system with 4 cores (quadcore), a process is hogging up CPU when it is > 25% (100 / 4 = 25).
This also applies to the Raspberry Pi 2 i think, because it has a quadcore processor (Raspberry Pi 1 has a single core processor).
You can also run monit in 'verbose' mode, so you can see what it does. Start monit by monit -v.
See also the monit documentation: https://mmonit.com/monit/documentation/monit.html
Pausing Monit (for updating Domoticz by hand for example)
There are probably some situations where you want Monit not to kick in directly when Domoticz is stopped. Updating Domoticz by hand (Pulling latest code from Github & then compiling) for example.
In this scenario you first have to stop the monitoring by Monit. Otherwise if you stop Domoticz by hand, Monit will kick in directly and start Domoticz again.
First stop Monit:
sudo /etc/init.d/monit stop
Then stop Domoticz:
sudo /etc/init.d/domoticz.sh stop
When you are done modifying/updating/whatever, start Domoticz first:
sudo /etc/init.d/domoticz.sh start
Then start Monit again:
sudo /etc/init.d/monit start
Using a script
You can check it Domoticz is running though a simple bash script. However this one is not advised as it can cause a infinite loop of restarting Domoticz. Using Monit is preferred (also because it can check multiple processes). There are number of scripts you can try each with their own advantages.
The first script
#!/bin/sh
# check domoticz
status=`curl -m 60 -s -i -H "Accept: application/json" "http://192.nnn.nnn.nnn:pppp/json.htm?type=devices&rid=1" | grep "status"| awk -F: '{print $2}'|sed 's/,//'| sed 's/\"//g'`
if [ $status ]
then
echo "Domoticz has already been started"
else
sudo service domoticz.sh stop
sleep 5
sudo service domoticz.sh start
/home/pi/TelegramServices/Telegram.sh msg Firstname_Lastname "Domoticz was offline, attempting a restart."
fi
Source: http://gathering.tweakers.net/forum/list_message/43593482#43593482
The line with the 'Telegram.sh' is not required. You can use it to send a Telegram-notification if Domoticz is offline.
Copy the script to /home/pi/domoticz/scripts/
Execute chmod +x check_domoticz_online.sh so the script can be executed.
Open your crontab (Terminal: crontab -e) and add the following line:
*/1 * * * * sudo /home/pi/domoticz/scripts/check_domoticz_online.sh 2>&1 >> /dev/null
Exit the crontab editor by pressing CTRL-O (character 'O', not zero) and hitting Enter.
This will check every minute if the Domoticz service is running. If it isn't running, it will (re)start the Domoticz service.
Second Alternative script
There is a second alternative script to still use monit as primary monitoring service.
Create scripts to start domoticz via alternate script
Create script to execute from monitrc to start domoticz and notify you via telegram,let it execute and modify.
sudo touch /etc/monit/start_domoticz.sh sudo chmod +x /etc/monit/start_domoticz.sh sudo nano /etc/monit/start_domoticz.sh
#!/bin/sh /usr/bin/curl -s "https://api.telegram.org/bot0000000:KEYKEYKEYKEYKEYKEY/sendMessage?chat_id=12345678&text=$MONIT_SERVICE - $MONIT_DESCRIPTION" > /dev/null exec /etc/init.d/domoticz.sh start
Create script to execute from monitrc to notify before system is going for a reboot.
sudo touch /etc/monit/telegram_bot_notifications.sh sudo chmod +x /etc/monit/telegram_bot_notifications.sh sudo nano /etc/monit/telegram_bot_notifications.sh
/usr/bin/curl -s "https://api.telegram.org/bot0000000:KEYKEYKEYKEYKEYKEY/sendMessage?chat_id=12345678&text=$MONIT_SERVICE - $MONIT_DESCRIPTION" > /dev/null
Change monit service
sudo /etc/monit/monitrc
If you used above section "Configuring monitoring of programs" you must change te lines to read like below otherwise just add the lines below.
check process domoticz with pidfile /var/run/domoticz.pid start program = "/etc/monit/start_domoticz.sh" stop program = "/etc/init.d/domoticz.sh stop" if failed url http://127.0.0.1:8080/json.htm?type=command¶m=getversion and content = '"status" : "OK"' for 2 cycles then exec "/etc/monit/start_domoticz.sh" else if succeeded then exec "/etc/monit/telegram_bot_notifications.sh" if 5 restarts within 5 cycles then exec "/sbin/reboot"
Thanks to http://bartoz.no-ip.org/archives/3689