Monitoring domoticz

From Domoticz
Jump to navigation Jump to search

Software crashes, it's a fact of life. Luckily there are a number of ways to keep an eye on Domoticz, and automaticallty restart it if it crashes.

Generally speaking there are two ways of doing this: via software designed to do this (Monit), or through a custom script that you run periodically. Both have advantages and disadvantages.

- Monit is robust, but may not always detect an actual crash.
- A script can test if Domoticz is actually ok, but might create problems itself.

Using Monit

Monit is a helpful program that automatically monitors and manages server programs to ensure that they not only stay online consistently, but that the file size, checksum, or permissions are always correct. Additionally Monit comes with a basic web interface through which all of the processes can be set up. This tutorial will cover the most basic setup and configuration to ensure Domoticz running 24/7.

Install Monit

Monit is easy installed through apt-get: 

sudo apt-get install monit

Once Monit has been installed, you can add programs and processes that you want to have checked, to the configuration file: 

sudo nano /etc/monit/monitrc

Configure Monit

Monit is very easy to use nearly out of the box. By default, it is set up to check that services are running every 2 minutes and stores its log file in "/var/log/monit.log". (I have changed this to my ramdisk check Wiki) 

sudo nano /etc/monit/monitrc

Modify the following parameters:

  • set daemon 300 -> Check services at 300 seconds (5 minutes). Be sure you leave enough time to do automated database backups and updates.
  • with start delay 300 -> Start checking the services after 300 seconds (this avoids bootloops if Domoticz has not started yet, give it some time to start fully)
  • set logfile /var/log/monit.log -> It is preferred to change the location of the logfile to a mounted pen drive or HDD. Or store it in a RAM-drive.

Configure e-mail notifications

You can setup Monit to inform you about every status update by e-mail. I would recommend doing this, so you have a good view on stability of your system (how often does Monit needs to kick in actually?).

Gmail 

   set mailserver smtp.gmail.com port 587 
   username "[email protected]" password "MYPASSWORD"
   using tls
   with timeout 30 seconds
   set alert [email protected]

Note: For monit <= 5.16: use tlsv12 instead of tls

If you use Gmail with 2-step verification, you first have to create an App password

POP3 

  set mailserver smtp.yourprovider.com port 25
  with timeout 30 seconds
  set mail-format { from: [email protected] }
  set alert [email protected]

To test if Monit can send you mails, issue a sudo monit reload and you should get a mail that the instance has changed.

Webservice

Monit comes with its own webserver, running on port 2812. To configure the webinterface, find and uncomment the section that begins with set httpd port 2812. Once the section is uncommented, fill in the IP-address of the machine where you have Domoticz running at, allow anyone to connect, and then create a monit user and password 

set httpd port 2812
 use address 0.0.0.0  			# only accept connection from localhost (comment to connect from other hosts)
 allow 0.0.0.0/0.0.0.0     	# allow anybody to connect to the server and
 allow admin:monit         	# require user 'admin' with password 'monit', change if your system is accessible from internet
 # allow 192.168.0.1/255.255.255.0	#This will only allow hosts coming from the local network (change 192.168.0.1 to your own LAN IP-range).
 # allow localhost          		#use in combination with local network to allow command line to work

Once this is configured, you should reload monit: 

sudo service monit reload

You will then be able to access the monit web interface by going to “192.168.1.2:2812” (change to IP-adress of your Domoticz system) Login with the username and password you configured earlier in the configfile (admin/monit ?). Your screen should look something like this.

Configuring monitoring of programs

Once the web services are set up, you can begin to input the programs that you want monitored and protected into the "/etc/monit/monitrc" configuration file. To simply ensure that programs stay online, you can use the /etc/init.d commands to stop or start a program. Here are some example configurations:

Domoticz 

  check process domoticz with pidfile /var/run/domoticz.pid
  start program = "/usr/bin/sudo /bin/systemctl start domoticz.service"
  stop  program = "/usr/bin/sudo /bin/systemctl stop domoticz.service"
  # use below start/stop command for older systems not using systemctl
  # start program = "/etc/init.d/domoticz.sh start"
  # stop  program = "/etc/init.d/domoticz.sh stop"
  if failed
     url http://127.0.0.1:8080/json.htm?type=command&param=getversion
         and content = '"status" : "OK"'
     for 2 cycles
     then restart
  if 5 restarts within 5 cycles then exec "/sbin/reboot"
  • Remark: if the service is not found, check if the response from Domoticz still exactly contains the string "state": "OK"
  • Remark2: If running monit in a Docker container make sure you use the real IP of the Domoticz server in the check url instead of 127.0.0.1.

To check for CPU-usage, include the line if cpu usage > 70% for 3 cycles then restart for example (put it between the 'then restart' and the last 'if 5 restarts..' line).

Once you have set up the configuration, check the syntax: 

sudo monit -t

After resolving any possible syntax errors, you can start running all of the monitored programs. 

sudo monit start all

data collected                    Thu, 30 Aug 2012 18:35:00

Monit can be started with a command that then keeps it running in the background 

sudo service monit start

Typing the command below displays monit’s status 

sudo monit status

This shows us the details: 

The Monit daemon 5.3.2 uptime: 1h 25m 
System 'myhost.mydomain.tld'
 status                            Running
 monitoring status                 Monitored
 load average                      [0.03] [0.14] [0.20]
 cpu                               3.5%us 5.9%sy 0.0%wa
 memory usage                      26100 kB [10.4%]
 swap usage                        0 kB [0.0%]
 data collected                    Thu, 30 Aug 2012 18:35:00

Finish up

Once you have configured all of the above steps, Domoticz will be tracked constantly and the process will be restarted automatically if it would have stopped running because of <insert a reason>.

Notes

Pay attention when monitoring CPU-usage of process

When you want to monitor the CPU-usage of a single process (instead of the whole machine), be warned about this: http://www.brandonchecketts.com/archives/monit-cpu-usage-problem
Basically, monit sees the usage spreaded on all processorcores.
So if you want to restart a process when it consumes 100% CPU or more, you need to divide 100 by the amount of cores you have.
If you have a system with 4 cores (quadcore), a process is hogging up CPU when it is > 25% (100 / 4 = 25).
This also applies to the Raspberry Pi 2 i think, because it has a quadcore processor (Raspberry Pi 1 has a single core processor).

You can also run monit in 'verbose' mode, so you can see what it does. Start monit by monit -v.
See also the monit documentation: https://mmonit.com/monit/documentation/monit.html

Pausing Monit (for updating Domoticz by hand for example)

There are probably some situations where you want Monit not to kick in directly when Domoticz is stopped. Updating Domoticz by hand (Pulling latest code from Github & then compiling) for example.

In this scenario you first have to stop the monitoring by Monit. Otherwise if you stop Domoticz by hand, Monit will kick in directly and start Domoticz again.

First stop Monit: 

sudo /etc/init.d/monit stop

Then stop Domoticz: 

sudo /etc/init.d/domoticz.sh stop

When you are done modifying/updating/whatever, start Domoticz first: 

sudo /etc/init.d/domoticz.sh start

Then start Monit again: 

sudo /etc/init.d/monit start


Using a script

You can check it Domoticz is running though a simple bash script. However this one is not advised as it can cause a infinite loop of restarting Domoticz. Using Monit is preferred (also because it can check multiple processes). There are number of scripts you can try each with their own advantages.

The first script

#!/bin/sh
# check domoticz
status=`curl -m 60 -s -i -H "Accept: application/json" "http://192.nnn.nnn.nnn:pppp/json.htm?type=devices&rid=1" | grep "status"| awk -F: '{print $2}'|sed 's/,//'| sed 's/\"//g'`
if [ $status ]
then
echo "Domoticz has already been started"
else
sudo service domoticz.sh stop
sleep 5
sudo service domoticz.sh start
/home/pi/TelegramServices/Telegram.sh msg Firstname_Lastname "Domoticz was offline, attempting a restart."
fi

Source: http://gathering.tweakers.net/forum/list_message/43593482#43593482

The line with the 'Telegram.sh' is not required. You can use it to send a Telegram-notification if Domoticz is offline.

Copy the script to /home/pi/domoticz/scripts/
Execute chmod +x check_domoticz_online.sh so the script can be executed.
Open your crontab (Terminal: crontab -e) and add the following line: 

*/1 * * * * sudo /home/pi/domoticz/scripts/check_domoticz_online.sh 2>&1 >> /dev/null

Exit the crontab editor by pressing CTRL-O (character 'O', not zero) and hitting Enter.
This will check every minute if the Domoticz service is running. If it isn't running, it will (re)start the Domoticz service.

Second Alternative script

There is a second alternative script to still use monit as primary monitoring service.

Create scripts to start domoticz via alternate script

Create script to execute from monitrc to start domoticz and notify you via telegram,let it execute and modify. 

 sudo touch /etc/monit/start_domoticz.sh 
 sudo chmod +x /etc/monit/start_domoticz.sh
 sudo nano /etc/monit/start_domoticz.sh

 #!/bin/sh
 /usr/bin/curl -s  "https://api.telegram.org/bot0000000:KEYKEYKEYKEYKEYKEY/sendMessage?chat_id=12345678&text=$MONIT_SERVICE - $MONIT_DESCRIPTION" > /dev/null
 exec /etc/init.d/domoticz.sh start

Create script to execute from monitrc to notify before system is going for a reboot. 

 sudo touch /etc/monit/telegram_bot_notifications.sh
 sudo chmod +x /etc/monit/telegram_bot_notifications.sh
 sudo nano /etc/monit/telegram_bot_notifications.sh

 /usr/bin/curl -s  "https://api.telegram.org/bot0000000:KEYKEYKEYKEYKEYKEY/sendMessage?chat_id=12345678&text=$MONIT_SERVICE - $MONIT_DESCRIPTION" > /dev/null

Change monit service 

 sudo /etc/monit/monitrc

If you used above section "Configuring monitoring of programs" you must change te lines to read like below otherwise just add the lines below. 

 check process domoticz with pidfile /var/run/domoticz.pid
 start program = "/etc/monit/start_domoticz.sh"
 stop  program = "/etc/init.d/domoticz.sh stop"
 if failed
 url http://127.0.0.1:8080/json.htm?type=command&param=getversion
     and content = '"status" : "OK"'
 for 2 cycles
 then exec "/etc/monit/start_domoticz.sh" else if succeeded then exec "/etc/monit/telegram_bot_notifications.sh"
 if 5 restarts within 5 cycles then exec "/sbin/reboot"


Thanks to http://bartoz.no-ip.org/archives/3689

Retrieved from "https://wiki.domoticz.com/index.php?title=Monitoring_domoticz&oldid=18737"