MQTT

From Domoticz
Jump to navigation Jump to search

Introduction

Architecture

MQTT [1] is a machine-to-machine (M2M)/"Internet of Things" connectivity protocol. It was designed as an extremely lightweight publish/subscribe messaging transport. MQTT provides a publish/subscribe message pattern to provide one-to-many message distribution and decoupling of applications.

Domoticz supports a number of hardware devices natively (rfxtrx433, zwave, smartmeter etc.). There are zillions of other devices out there, with a lot of interfaces. Using it's native MQTT interface Domoticz can publish events from inside to the outside world. Domoticz can also respond to actions requested by anyone (and passed on by the MQTT-broker). Now something has to take care of creating and interpreting these messages.

The Node-RED tool provides an alternative way to creating little programs (flows) to interface with anything you want. But maybe your application can create MQTT-messages on it's own that can be understood by Domoticz. Or maybe you like to create programs in Node.js itself. So using Node-RED is not mandatory, it does however provide a fun and attractive way to handle messages.

  • Node-RED [2] is a tool running in the Node.js platform providing a browser-based flow editor that makes it easy to wire together "flows". Flows can be then deployed to the runtime in a single-click.
  • Node.js [3] is a platform built on Chrome's JavaScript runtime for easily building fast, scalable network applications.

You'll need

  • Domoticz running on a Raspberry Pi (Windows or Mac are supported but this tutorial focusses on a Pi)
  • A MQTT broker (we use mosquitto, we'll install that later)
  • Add hardware gateway to Domoticz: "Create a MQTT Client Gateway with LAN interface" or "MQTT Auto Discovery Client Gateway" (on localhost)

If the hardware MQTT is not supporting Domoticz MQTT protocol or Home Assistant Autodiscover protocol then you have to write your own transformation.

This can be done with Python Plugins, DzVents (with OpenURL to mosquitto_pub) or Node-Red

If you want to use Node-Red for writing your own interface you will need some extra software installed:

  • Node.JS (we'll install that later)
  • Node-RED (we'll install that later)
  • A few Node-RED example Flows

Installing software

Installing Mosquitto

By using Docker (Docker Compose) on Linux.

For this to work you need to:

- Install both Docker Engine and Docker Compose.

- Install Mosquitto using docker (compose) (from https://github.com/vvatelot/mosquitto-docker-compose) 

sudo su -
git clone https://github.com/vvatelot/mosquitto-docker-compose /opt/mosquitto
cd /opt/mosquitto
make setup-project
docker compose up -d

There is one additional step we need to be doing, we want the MQTT broker to restart when the system starts.

For this you have to edit the file docker-compose.yaml in /opt/mosquitto and add the following line underneath 'mosquitto:' 

    restart: unless-stopped

Make sure it aligns with spaces at the same start as the line below (build:) so it looks like: 

services:
  mosquitto:
    restart: unless-stopped
    build:
      context: .

Next restart the docker compose container with 

docker compose down
docker compose up -d

MQTT should now be available on port 1883. You can also use an application like 'MQTT Explorer' to connect to your new broker

Legacy installation method- on Linux

It is highly recommended that if you are using Debian or Rasbian that you use the additional Mosquitto repositories. This will ensure you get the latest stable version of Mosquitto as determined by the project itself, rather than an outdated version via the Debian or Raspbian repositories.

Rather than re-iterating the install procedure here, the guide given a thttps://randomnerdtutorials.com/how-to-install-mosquitto-broker-on-raspberry-pi/ is simple and easy to follow, and we encourage you to give their process a shot.

On Windows

Check this instruction: http://www.steves-internet-guide.com/install-mosquitto-broker/

General Warning config change for v2.0 and up

From Mosquitto v2.0.x you need extra configuration, else no connection is made with Domoticz see also https://www.domoticz.com/forum/viewtopic.php?f=28&t=35706

To see your installed version:

mosquitto -h | grep version

To fix this edit the /etc/mosquitto/mosquitto.conf file and add:

listener 1883

allow_anonymous true


Note that his will give access to the mqtt broker for all devices on your network

Eclipse has produced a document to support users to migrate from version 1.x to 2.0.

See: https://mosquitto.org/documentation/migrating-to-2-0/


If only a MQTT Broker is needed (for example Zigbee2MQTT) then continue with

Add hardware "MQTT Client Gateway" for general MQTT hardware or

Add hardware "MQTT Auto Discovery Client Gateway" for Hardware that supports HA Autodiscover protocol

When Using Node-RED for writing your own interface

Installing Node.JS
  1. NOTE: Do you need this, since node.red instalation later removes old node.js and npm and installs newer ones ?
  2. Please follow the instructions to install nodejs.
  3. After that, install some modules using the Node Package Manager:
sudo npm install -g mqtt url request

If you want to debug or auto-reload of the script, also install nodemon: 

sudo npm install -g nodemon

...

Installing Node-RED

Instructions can be found here: [4] 

Node-RED should now be available on HTTP port 1880

Add hardware "MQTT Client Gateway"

Domoticz needs to subscribe to the Mosquitto MQTT message broker that is now running locally.

Under Setup/Hardware add a device of type: "MQTT Client Gateway with LAN interface" with settings:

Main parameters

  • Data Timeout: Disabled
  • Remote Address: localhost, or the IP adress of the system where the mqtt broker server (eg mosquitto) is running if installed on another system.
  • Port: 1883, or the port of the mqtt broker server (eg mosquitto) if not 1883 (default)
  • Username: Leave empty when mqtt broker has no user accounts setup else give mqtt broker account name
  • Password: Leave empty when mqtt broker has no user accounts setup else give mqtt broker account password
  • Prevent Loop: True (default). This prevents looping Domoticz/in topics back to Domoticz/out resulting in unwanted reactions of devices writing and listening to domoticz/in and -out. Set to False if you want the result of Domoticz/in topics being published to domoticz/out.

Publish Topic

The publish topic is determined by this setting and/or on which floorplans / roomplans a device is includ

  • if the publish topic of the MQTT hardware is set to out (Flat) then the publish topic will be domoticz/out
  • if the publish topic of the MQTT hardware is set to / (Hierarchical) and a device is defined in Roomplan mqttDevices and this Roomplan is part of floorplan X then the publish topic will be domoticz/out/X/mqttDevices
  • if the publish topic of the MQTT hardware is set to out +/ (Combined) and a device is defined in Roomplan mqttDevices and this Roomplan is part of floorplan X then the publish topics will be domoticz/out AND domoticz/out/X/mqttDevices
  • if the publish topic of the MQTT hardware is set to out +/ (Combined) and a device is defined in multiple roomplans and these Roomplans are part of floorplan X then the publish topics will be domoticz/out AND domoticz/out/X/roomplan1 AND domoticz/out/X/roomplan2 .. domoticz/out/X/roomplanN
  • if the publish topic of the MQTT hardware is set to None then the publish topic will not be be published to domoticz/out only domoticz/in will be read
  • if the publish topic of the MQTT hardware is set to Index then the publish topic will be domoticz/out/$idx were $idx is the idx number of the device.
  • if the publish topic of the MQTT hardware is set to Index with retain then the publish topic will be domoticz/out/$idx were $idx is the idx number of the device and retained in MQTT broker.
  • if the publish topic of the MQTT hardware is set to Name then the publish topic will be domoticz/out/$name were $name is the name of the device.
  • if the publish topic of the MQTT hardware is set to Name with retain then the publish topic will be domoticz/out/$name were $name is the name of the device and retained in MQTT broker..
Extra options with retain are visible in Publish Topic pull down menu.

Other parameters

  • Topic in Prefix: Prefix for the topic in to listen. Prefix cannot contain a "#" symbol.
    • To prevent Domoticz from listening to topic domoticz/in (or custom prefix) leave the field empty.
    • If Domoticz needs to listen to more topics add a new MQTT Client Gateway for each topic
  • Topic out Prefix: Prefix for the topic out to send. Prefix cannot contain a "#" symbol.
    • To prevent Domoticz from writing to topic domoticz/out (or custom prefix) leave the field empty.
    • If Domoticz needs to send to more topics add a new MQTT Client Gateway for each topic

The CA-filename and TLS version are advanced settings if you also require a certificate to communicate with your broker.


To finish press 'Add'.


To test if Domoticz receives data, create a Dummy Virtual temperature sensor.

  • Under Setup/Hardware add a device of type: "Dummy (Does nothing, use for virtual switches only)" with the name of your choice ("Dummy" for example).
  • In the corresponding entry of your just created "Dummy" hardware, click the "Create Virtual Sensors" button that appear following his type. Then enter the name of your choice (Let's say "Fictive Temp").
  • Under Setup/Devices, get the "idx" number of your "Fictive Temp" device (Let's say it's 1).

Then publish a temperature measurement to the virtual sensor idx noted above (example idx 1) like: 

mosquitto_pub -h localhost -m '{ "idx" : 1, "nvalue" : 0, "svalue" : "25.0" }' -t 'domoticz/in'

If the mosquitto_pub command is not found install the mosquitto client: 

sudo apt-get install mosquitto-clients

Filter devices to send to Topic Out

From 2024.1 it is possible to select the devices that need to send data to the topic out (default domoticz/out). The rest of the devices will not send MQTT data.

Default the list is empty and all devices will send data.


To filter the devices go to menu Setup - Hardware and click on the Setup button of the MQTT Client gateway. A new dialog page will be shown were you can select the devices.

Add hardware "MQTT Auto Discovery Client Gateway"

From Stable 2022.1 Domoticz MQTT supports Home assistant MQTT Discovery

Autodiscovery allows domoticz to receive device definition and configuration information which is used to automatically create corresponding devices in domoticz and send/receive messages for these devices to the controlling system. Both zwave js ui (zwavejs2mqtt) and zigbee2mqtt default to 'homeassistant' as the discover prefix (in HomeAssistant section, and check if switched on) but both allow this to be changed. See also the specific Domoticz gateway pages eg Zigbee2MQTT or ZwaveJS2MQTT

For questions about MQTT Autodiscover go to the forum: https://domoticz.com/forum/viewforum.php?f=82

As Autodiscover has various implementations by supporting gateways please go to this link for AD debug tools if there are issues.

Main parameters

  • Data Timeout: Disabled
  • Remote Address: localhost, or the IP adress of the system where the mqtt broker server (eg mosquitto) is running if installed on another system.
  • Port: 1883, or the port of the mqtt broker server (eg mosquitto) if not 1883 (default)
  • Username: Leave empty when mqtt broker has no user accounts setup else give mqtt broker account name
  • Password: Leave empty when mqtt broker has no user accounts setup else give mqtt broker account password
  • Auto Discovery Prefix: Default homeassistant.
See also the specific Domoticz gateway pages eg Zigbee2MQTT or ZwaveJS2MQTT

Domoticz devices will be created once data is received. So especially for battery operated devices it can take some time to have this data received. Sometimes it is possible to activate these sensors manually by pushing a button to get them discovered. The new devices will show up in the device list (menu Setup -> Devices)

If you intend to integrate multiple different systems through autodiscovery then a best practice would be to configure separate MQTT hardware workers for each integration. That way, you can always delete one without impacting others. This would require changing the default autodiscovery topic to make it specific for that interface (e.g. use zwaveauto for zwavejs2mqtt, zigbeeauto for zigbee2mqtt and so on).

Currently supported AutoDiscover device types:

  • Alarm control panels
  • Binary sensors
  • Cameras
  • Covers
  • Device Trackers
  • Device Triggers
  • Fans
  • Humidifiers
  • HVACs
  • Lights
  • Locks
  • Numbers (through hardware mqtt Autodiscover setup)
  • Scenes
  • Selects
  • Sensors
  • Switches
  • Tag Scanners
  • Vacuums

List of projects supporting MQTT AutoDiscover:

Zwave JS UI (former Zwavejs2mqtt), Zigbee2mqtt, ESPHome, ecowitt2mqtt, EsPurna, IOTLink, MiFlora MQTT Daemon, OpenMQTTGateway, Teleinfo MQTT, Wyzesense2MQTT, Arduino HAMqttDevice, EMS-bus-Gateway etc

If you are a developer who wants to use this MQTT payload definition go to https://www.home-assistant.io/docs/mqtt/discovery/

MQTT Autodiscovery Setup

After creation of the MQTT Autodiscover hardware in menu Setup - Hardware you can click on the setup button to configure the discovered devices if they have that option using Autodiscovery Number devices..

Note: All other normal discovered devices like switches, lights, sensors etc can be found in the device list, see menu Setup -> Devices

A menu will be shown with the discovered configuration devices. You can select the device and change the value. The new value will be sent to the mqtt device controller (zigbee2mqtt, zwaveJS-UI, ESP, etc)


Domoticz MQTT communication

Communication from and to Domoticz works via JSON. Default MQTT topics of the Domoticz for incoming and outcoming messages are: 

  domoticz/in
  domoticz/out

From stable 2021.1: To see the MQTT messages in Domoticz logging you have to enable hardware debugging as they are moved from info to debug level.

Modify: /etc/init.d/domoticz.sh with:

DAEMON_ARGS="$DAEMON_ARGS -loglevel normal,status,error,debug" # debug enabled for now

DAEMON_ARGS="$DAEMON_ARGS -debuglevel hardware"

than: 

sudo systemctl daemon-reload
sudo service domoticz stop
sudo service domoticz start

Alternatively it is also possible to stop the Domoticz process and start Domoticz manually with the switch "-loglevel normal,status,error,debug -debuglevel hardware" (without the quotes)

Domoticz to MQTT

{
 "idx" : 5,
 "name" : "Internal Temperature",
 "id" : "00080A",
 "unit" : 1 
 "dtype" : "Temp",
 "stype" : "TFA 30.3133",
 "nvalue" : 0,
 "svalue1" : "41.2",
 "Battery" : 100,
 "RSSI" : 12,
}

{
 "idx" : 7,
 "name" : "S0 P1",
 "id" : "123456",
 "unit" : 0
 "dtype" : "Energy",
 "stype" : "CM119 / CM160",
 "nvalue" : 0,
 "svalue1" : "90",
 "svalue2" : "2975.00",
 "Battery" : 100,
 "RSSI" : 12,
}

The format is compatible as described in Domoticz_API/JSON_URL's wiki page, except the "svalue" (String Value) is split into multiple parts for easier parsing.

MQTT to Domoticz

The format is mostly compatible as described in the JSON wiki: Domoticz_API/JSON_URL's

For JSON the rule is that numeric values and booleans are given without quotes and string values between double quotes.

Domoticz parameter nvalue is always a number, svalue is always a string.
eg "idx" : 7, "nvalue" : 0, "svalue" : "90;2975.00", "parse": false

Update Sensors/Thermostat Setpoint

Note: If "command" is not set, it defaults to "udevice". udevice only updates the device in domoticz and will not trigger action scripts. So normally do not use udevice for switches, scenes and groups, only for sensors.

The two following messages are equivalent: 

{  "command": "udevice",  "idx" : 7,  "nvalue" : 0,  "svalue" : "90;2975.00" }
{  "idx" : 7,  "nvalue" : 0,  "svalue" : "90;2975.00" }

To update in Domoticz only and to prevent event triggers (like MQTT domoticz/out messages) use additional "parse": false 

{  "command": "udevice",  "idx" : 7,  "nvalue" : 0,  "svalue" : "90;2975.00", "parse": false }

Additional on update command you can update battery level ("Battery") and/or signal strength ("RSSI") but only as additional parameters on nvalue and svalue 

{  "command": "udevice",  "idx" : 7,  "nvalue" : 0,  "svalue" : "90;2975.00", "Battery" : 100, "RSSI" : 12 }


Tip: on the following page a user listed all types of sensor devices and its update commands: https://piandmore.wordpress.com/2019/02/04/mqtt-out-for-domoticz/

Sending a customEvent trigger to dzVents dzVents 3.0.0

{"command" : "customevent", "event" : "MyEvent" , "data" : "myData" }'

Sending a Switch Command

Note: Do not use "command": "udevice" for switching. udevice only updates the device in domoticz and will not trigger action scripts. So normally do not use udevice for switches, scenes and groups, only for sensors. 

{"command": "switchlight", "idx": 2450, "switchcmd": "On" }
{"command": "switchlight", "idx": 2450, "switchcmd": "Toggle" }
{"command": "switchlight", "idx": 2450, "switchcmd": "Set Level", "level": 50 }
{"command": "setcolbrightnessvalue", "idx": 2450, "hue": 274, "brightness": 40, "iswhite": false }
{"command": "setcolbrightnessvalue", "idx": 2450, "hex": "RRGGBB", "brightness": 100, "iswhite": false }
{"command": "setcolbrightnessvalue", "idx": 2450, "color": {"m":3,"t":0,"r":0,"g":0,"b":50,"cw":0,"ww":0}, "brightness": 40}

The options for setcolbrightnessvalue are explained on Domoticz_API/JSON_URL's#Set_a_light_to_a_certain_color_or_color_temperature

Sending a Group Command

Maybe a bit confusing at first but the type group belongs to the collection of scenes and can be switched and toggled (contrary to type scenes) 

{"command": "switchscene", "idx": 2, "switchcmd": "On" }
{"command": "switchscene", "idx": 2, "switchcmd": "Off" }
{"command": "switchscene", "idx": 2, "switchcmd": "Toggle" }

Sending a Scene Command

Please note that you can only activate a scene (by sending an On command). It can't be toggled or deactivated. This is because in a type Scene you put switches with the wanted state (On, Off, Dimlevel. etc) 

{"command": "switchscene", "idx": 24, "switchcmd": "On" }

Sending a Notification

The complete message format with all available parameters is as follows: 

{"command": "sendnotification",
"idx": idx, 
"name": "Some name", 
"subsystem": "target subsystems" 
"subject": "Message Subject", 
"body": "Message Body",
"extradata": "extradata" 
"priority": priority, 
"sound": "mysound",
"brfromnotification": "brfromnotificationdata" }

idx defaults to 0, subsystems defaults to NOTIFYALL(0), which sends the message on all enabled subsystems), priority default is 0 and brfromnotification is true by default.

A simple message to all users on all (enabled) subsystems could look as follows 

{"command": "sendnotification",
 "subject": "Urgent message",
 "body": "My first MQTT notification",    
 "priority": 0}

To send a message only via FCM only to a single mobile device, you can use 

{"command": "sendnotification",
 "subject": "Urgent message",
 "subsystems": "fcm",
 "extradata":"midx_42",
 "body": "My first MQTT message on FCM",
 "priority": 0 }

For fcm messages the extradata parameter should be used to specify the mobile device idx to send the message to. By default it is send to all devices. The device midx can be found in domoticz->"setup"->"more options" ->"mobile devices".

Set a User Variable

{"command": "setuservariable", "idx": 1, "value": "12.3" }

Request Device Info

{"command": "getdeviceinfo", "idx": 2450 }

Request Scene Info

{"command": "getsceneinfo", "idx": 1 }

Log message

{"command" : "addlogmessage", "message" : "MyMessage to log" }


Debugging MQTT

Use MQTT Explorer tool

Sometimes it is hard to see what messages are send and read by Domoticz or the MQTT devices in your home network.

Suggested is to use the tool MQTT Explorer, it displays all MQTT topics in an easy to read tree-view. This tool can also be used to remove retained messages from old MQTT devices.

  • Visualize topics and topic activity
  • Copy Topics and/or values to clipboard as text (see screenshot)
  • Delete retained topics
  • Search/filter topics
  • Delete topics recursively
  • Diff view of current and previous received messages
  • Publish topics
  • Plot numeric topics
  • Retain a history of each topic

This tool has also its own docker image for those already using docker: https://hub.docker.com/r/smeagolworms4/mqtt-explorer


Copy Icons for Topic name and payload value:

Delete retained topic

If a topic has a retained message, a red RETAINED button appears in MQTT Explorer. Click it and the retained message will be deleted. The topic will continue to be displayed in the tree-view and will only disappear after you disconnect/re-connect with MQTT Explorer.

If these messages were used to create Domoticz devices you will have to restart Domoticz or update (= restart) MQTT hardware gateway in menu Setup - Hardware in order to delete internal device tables.


Debugging MQTT AutoDiscover devices

When you experience issues with MQTT Auto Discovery (Like your device is not discovered, or not or partly working), we need some additional data to debug/simulate the issue.

Go to wiki page MQTT AD Report Problems for the instructions.

Useful MQTT possibilities

Owntracks

Owntracks ([owntracks.org]) is an app (both Android and iPhone) which periodicaly sends the location of a device to a mqtt-broker (e.g. mosquitto). A Node-RED script could subscribe to the owntracks-topic and republish the info in a Domoticz compatible message so Domoticz can use the location info.

MQTTMapper plugin

If your device is communicating through MQTT but not able to communicate with MQTT AutoDiscovery or the Domoticz MQTT protocol you can also install the MqttMapper Python plug-in, allowing to map MQTT topics directly to Domoticz devices.

For more information see forum topic https://www.domoticz.com/forum/viewtopic.php?t=39279 and github repository https://github.com/FlyingDomotic/domoticz-mqttmapper-plugin

More information & useful links


Native MQTT Libraries

For interacting directly with Domoticz via MQTT.

NodeJS

sudo npm install node-domoticz-mqtt

See: https://www.domoticz.com/forum/viewtopic.php?f=21&t=10190

Python

paho-mqtt

You may use *paho-mqtt* library to communicate with domoticz MQTT client

sudo pip install paho-mqtt

See: https://pypi.python.org/pypi/paho-mqtt/1.3.1

MQTTMapper plugin

If your device is communicating through MQTT but not able to communicate with MQTT AutoDiscovery or the Domoticz MQTT protocol you can also install the MqttMapper Python plug-in, allowing to map MQTT topics directly to Domoticz devices.

For more information see forum topic https://www.domoticz.com/forum/viewtopic.php?t=39279 and github repository https://github.com/FlyingDomotic/domoticz-mqttmapper-plugin

Arduino

Here is an example on the forum: https://www.domoticz.com/forum/viewtopic.php?t=38155

Here is a good starting point: https://github.com/knolleary/pubsubclient/blob/master/examples/mqtt_basic/mqtt_basic.ino
This project contains an Arduino sketch for a sonoff device with MQTT/Domoticz compatibility: https://github.com/arendst/Sonoff-MQTT-OTA-Arduino

Another example project: https://github.com/MikaPetteriLundgren/Arduino-Domoticz-Gateway

Tip for reading domoticz/out with a sketch:

Arduino PubClient has a default message size which is 128 bytes. As you probably know json message from domoticz on domiticz/out is much bigger. What you need to do is edit PubSubClient.h and change default size of message.

Just change

#define MQTT_MAX_PACKET_SIZE 128

to

#define MQTT_MAX_PACKET_SIZE 1024

Retrieved from "https://wiki.domoticz.com/index.php?title=MQTT&oldid=18684"