Using your XiaoMi Home Hardware without the Cloud

• ZigBee Communication without a proprietary Hub
  • Setting up the bridge
  • Running the bridge
    • Determine location of CC2531 USB sniffer and checking user permissions
    • Installation
    • Configuration
    • Running ZigBee2MQTT
    • Running ZigBee2MQTT as a Daemon
    • Update zigbee2mqtt to the latest version
  • Pairing Devices
• Using Zigbee Devices to control INSTAR IP Cameras in Node-RED
  • Add the MQTT input to an existing Flow

ZigBee Communication without a proprietary Hub

Zigbee is a specification for a suite of high-level communication protocols used to create personal area networks with small, low-power digital radios, such as for home automation, medical device data collection, and other low-power low-bandwidth needs, designed for small scale projects which need wireless connection. Hence, Zigbee is a low-power, low data rate, and close proximity wireless ad hoc network.

The Zigbee2MQTT bridge enables you to use selected supported ZigBee devices without the otherwise mandatory Hub and Vendor Cloud connection. Among the devices that you can control are smart (ZigBee) light bulbs from Belkin, IKEA, OSRAM and Philips as well as smart sensors from SmartThings and Xiaomi.

In this tutorial we are going to install the ZigBee2MQTT bridge together with Xiaomi smart sensors / buttons, on the hardware side, and Node-RED and Home Assistant, on the software side, to control our INSTAR IP Camera.

The ZigBee bridge consists of a ZigBee Sniffer Module and a controlling software written in Node.js. It bridges events and allows you to control your Zigbee devices via MQTT. In this way you can integrate your Zigbee devices with whatever smart home infrastructure you are using.

Needed Hardware:

• CC debugger
• CC2531 USB sniffer
• Downloader cable CC2531
• PC or Raspberry Pi to run bridge

The Debugger, Sniffer and connecting cable can be ordered directly through online shops like AliExpress.

The debugger and cable are only used once to flash the firmware onto the stick. There is alternative way to do that using an Arduino board instead.

Setting up the bridge

The following description shows you how to flash the hardware onto your CC2531 stick under Windows 10 - for Linux or macOS please follow this link.

1. Install SmartRF Flash programmer (NOT V2). This software is free but requires a Texas Instruments account in order to download.

2. Install the CC debugger driver on your PC (Windows only). Before continuing, verify that the CC Debugger driver has been installed correctly. In case the CC Debugger is not recognized correctly install the driver manually.

3. Connect CC debugger --> Downloader cable CC2531 --> CC2531 USB sniffer.

4. Connect BOTH the CC2531 USB sniffer and the CC debugger to your PC using USB.
5. If the light on the CC debugger is RED press set reset button on the CC debugger. The light on the CC debugger should now turn GREEN. If not use CC debugger user guide to troubleshoot your problem.
6. Download the firmware CC2531ZNP-Prod.hex
7. Start SmartRF Flash Programmer, setup as shown below and press Perform actions.

Running the bridge

After flashing your CC2531 USB sniffer with CC2531ZNP-Prod.hex firmware we can remove the CC Debugger and connecting cable and plug the Sniffer into our Linux computer you can use a Raspberry Pi 3 with Raspbian Stretch Lite, but will work on any Linux machine. In the example below we used the Core i3 Mini PC with Debian Stretch that we used to install the Software MotionEye on.

For alternative installation paths please follow this link.

NOTE: it is reported that Raspbian Jessie doesn't work.

Determine location of CC2531 USB sniffer and checking user permissions

We first need to determine the location of the CC2531 USB sniffer. Connect the CC2531 USB to your Raspberry Pi. Most of the times the location of the CC2531 is /dev/ttyACM0. This can be verified by:

ls -l /dev/ttyACM0
crw-rw---- 1 root dialout 166, 0 May 16 19:15 /dev/ttyACM0  # <-- CC2531 on /dev/ttyACM0

As an alternative, the device can also be mapped by an ID. This can be handy if you have multiple serial devices connected to your pi. In the example below the device location is: /dev/serial/by-id/usb-Texas_Instruments_TI_CC2531_USB_CDC___0X00124B0018E32D5B-if00

ls -l /dev/serial/by-id
total 0
lrwxrwxrwx 1 root root 13 Nov 24 18:59 usb-Texas_Instruments_TI_CC2531_USB_CDC___0X00124B0018E32D5B-if00 -> ../../ttyACM0

Installation

To use ZigBee2MQTT we first need to install Node.js and Git:

sudo apt update
sudo apt install git

You can confirm that you have installed Git and Node correctly by running the following commands:

git --version
node -v

Then clone ZigBee2MQTT from Github and change the owner of the /opt/zigbee2mqtt directory to your default Linux user - in our case this is nodeadmin:

sudo git clone https://github.com/Koenkk/zigbee2mqtt.git /opt/zigbee2mqtt
sudo chown -R nodeadmin:nodeadmin /opt/zigbee2mqtt

Then change into the directory and use npm to install all dependencies:

cd /opt/zigbee2mqtt
npm install

Configuration

Before we can start zigbee2mqtt we need to edit the configuration.yaml file. This file contains the configuration which will be used by zigbee2mqtt. Open the configuration file:

nano /opt/zigbee2mqtt/data/configuration.yaml

We are using a local Mosquitto MQTT server that we already configured with a client login which has to be added here. Also we are planning to use a local installation of Home Assistant that we want to be able to directly communicate with the ZigBee2MQTT bridge:

# Home Assistant integration (MQTT discovery)
homeassistant: true

# allow new devices to join
permit_join: true

# MQTT settings
mqtt:
  # MQTT base topic for zigbee2mqtt MQTT messages
  base_topic: zigbee2mqtt
  # MQTT server URL - change this to a local IP if you are running you server on a different device
  server: 'mqtt://localhost'
  # MQTT server authentication, uncomment if required:
  user: debian
  password: my_password

# Serial settings
serial:
  # Location of CC2531 USB sniffer
  port: /dev/ttyACM0

Make sure that permit_join is set to true, to allow new devices to be added. It can be set to false once all your Zigbee devices are integrated.

For the serial port you can use the location of CC2531 USB sniffer that we determined to be either /dev/ttyACM0 or /dev/serial/by-id/usb-Texas_Instruments_TI_CC2531_USB_CDC___0X00124B0018E32D5B-if00 - both can be used interchangeably (the location might differ in your case!).

Running ZigBee2MQTT

Now that we have setup everything correctly we can start zigbee2mqtt directly from our console to see if there are any error messages:

cd /opt/zigbee2mqtt
sudo npm start

You can see that the service is running, connected to the local MQTT server and published a message to zigbee2mqtt/bridge/state with the payload online. We can use the program MQTT.fx to verify that the message was received by our Mosquitto server:

Running ZigBee2MQTT as a Daemon

To run zigbee2mqtt as a service and start it automatically on boot we will run zigbee2mqtt with systemctl:

sudo nano /etc/systemd/system/zigbee2mqtt.service

Add the following to this file to automatically run the index.js file from the zigbee2mqtt bridge inside the Node runtime:

[Unit]
Description=zigbee2mqtt
After=network.target

[Service]
ExecStart=/usr/bin/node /opt/zigbee2mqtt/index.js
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=zigbee2mqtt
Restart=always
 # Restart service after 10 seconds if node service crashes
 RestartSec=10

[Install]
WantedBy=multi-user.target

Save the file, exit and verify that the configuration works:

sudo systemctl daemon-reload
sudo systemctl start zigbee2mqtt
systemctl status zigbee2mqtt.service

Now that everything works, we want systemctl to start zigbee2mqtt automatically on boot, this can be done by executing:

sudo systemctl enable zigbee2mqtt.service

Some tips that can be handy later:

# Stopping zigbee2mqtt
sudo systemctl stop zigbee2mqtt

# Starting zigbee2mqtt
sudo systemctl start zigbee2mqtt

# View the log of zigbee2mqtt
sudo journalctl -u zigbee2mqtt.service -f

Update zigbee2mqtt to the latest version

# Stop zigbee2mqtt and go to directory
sudo systemctl stop zigbee2mqtt
cd /opt/zigbee2mqtt

# Backup configuration
cp -R data data-backup

# Update
git checkout HEAD -- npm-shrinkwrap.json
git pull
rm -rf node_modules
npm install

# Restore configuration
cp -R data-backup/* data
rm -rf data-backup

# Start zigbee2mqtt
sudo systemctl start zigbee2mqtt

Pairing Devices

The number of devices that you can use with the zigbee2mqtt bridge is steadily growing (there are alternative pairing instructions available here). In this tutorial we are going to add 3 XiaoMi Mi Home sensors - a Temperature & Humidity Sensor, a Wireless Switch and a Home Occupancy Sensor.

IMPORTANT: Before you start, make sure that permit_join: true is set in your configuration.yaml. Otherwise new devices cannot join the network! Also verify that the ZigBee2MQTT Bridge is running.

We first start with the Temperature & Humidity Sensor that has a pairing button on its side. Press the button for about 5 seconds - the blue LED will start to blink: in my case it started to blink slowly in 3-time sequences. After a few seconds the blinking sped up and stopped and the device was paired with the ZigBee Bridge.

The pairing can be found inside a log file in /opt/zigbee2mqtt/data/log/:

info: Connecting with device...
info: MQTT publish, topic: 'zigbee2mqtt/bridge/log', payload: '{"type":"pairing","message":"connecting with device"}'
info: Device incoming...
info: MQTT publish, topic: 'zigbee2mqtt/bridge/log', payload: '{"type":"pairing","message":"device incoming"}'
info: New device with address 0x00158d00023a21c8 connected!
info: MQTT publish, topic: 'zigbee2mqtt/bridge/log', payload: '{"type":"device_connected","message":"0x00158d00023a21c8"}'
info: MQTT publish, topic: 'homeassistant/sensor/0x00158d00023a21c8/temperature/config', payload: '{"unit_of_measurement":"°C","device_class":"temperature","value_template":"{{ value_json.temperature }}","json_attributes":["linkquality","battery","voltage"],"state_topic":"zigbee2mqtt/0x00158d00023a21c8","availability_topic":"zigbee2mqtt/bridge/state","name":"0x00158d00023a21c8_temperature","unique_id":"0x00158d00023a21c8_temperature_zigbee2mqtt"}'
info: MQTT publish, topic: 'homeassistant/sensor/0x00158d00023a21c8/humidity/config', payload: '{"unit_of_measurement":"%","device_class":"humidity","value_template":"{{ value_json.humidity }}","json_attributes":["linkquality","battery","voltage"],"state_topic":"zigbee2mqtt/0x00158d00023a21c8","availability_topic":"zigbee2mqtt/bridge/state","name":"0x00158d00023a21c8_humidity","unique_id":"0x00158d00023a21c8_humidity_zigbee2mqtt"}'

Shortly afterwards the sensor data started coming in:

info: MQTT publish, topic: 'zigbee2mqtt/0x00158d00023a21c8', payload: '{"temperature":26.74,"linkquality":55,"humidity":54.1,"battery":100,"voltage":3035}'

So far so good. But the device ID was a bit hard to read and memorize: 0x00158d00023a21c8. Luckily this can be adjusted in /opt/zigbee2mqtt/data/configuration.yaml:

devices:
  '0x00158d00023a21c8':
    friendly_name: 'temp_hum_sensor'
    retain: false

All paired devices will be listed at the bottom of the configuration file - just change the friendly_name to something human-readable. Remember that this will also change the MQTT topic the Zigbee bridge is using to publish the sensor readouts:

info: MQTT publish, topic: 'zigbee2mqtt/temp_hum_sensor', payload: '{"temperature":26.33,"linkquality":78,"humidity":54.32,"battery":99,"voltage":3005}'

The topic changed from zigbee2mqtt/0x00158d00023a21c8 to zigbee2mqtt/temp_hum_sensor !

But pairing both the Wireless Switch and Home Occupancy Sensor turned out to be more difficult. Holding the pairing button - that here could only be reached with a metal pin - gave the same pattern. The blue LED started flashing - first slowly, then fast. But I kept getting timeouts for both devices - that identified as 0x00158d00023d229c and 0x00158d00020b3dff - for the pairing process inside the /opt/zigbee2mqtt/data/log/

error: Cannot get the Node Descriptor of the Device: 0x00158d00023d229c (Error: Timed out after 10000 ms)
error: Cannot get the Node Descriptor of the Device: 0x00158d00020b3dff (Error: Timed out after 10000 ms)

You can find an instruction that you need to press and hold the pairing button for 5 seconds (until the blue LED starts flashing), then release and keep pressing the button repeatedly (1s interval) afterwards - which feels a little bit counter-intuitive. But it actually works. Checking the log file shows that a Occupancy Sensor with the ID 0x00158d00020b3dff was connected:

info: New device with address 0x00158d00020b3dff connected!
info: MQTT publish, topic: 'zigbee2mqtt/bridge/log', payload: '{"type":"device_connected","message":"0x00158d00020b3dff"}'
info: MQTT publish, topic: 'homeassistant/binary_sensor/0x00158d00020b3dff/occupancy/config', payload: '{"payload_on":true,"payload_off":false,"value_template":"{{ value_json.occupancy }}","device_class":"motion","json_attributes":["linkquality","battery","voltage"],"state_topic":"zigbee2mqtt/0x00158d00020b3dff","availability_topic":"zigbee2mqtt/bridge/state","name":"0x00158d00020b3dff","unique_id":"0x00158d00020b3dff_occupancy_zigbee2mqtt"}'
info: MQTT publish, topic: 'zigbee2mqtt/0x00158d00020b3dff', payload: '{"occupancy":true,"linkquality":131}'
info: Device incoming...
info: MQTT publish, topic: 'zigbee2mqtt/bridge/log', payload: '{"type":"pairing","message":"device incoming"}'
info: MQTT publish, topic: 'zigbee2mqtt/0x00158d00020b3dff', payload: '{"occupancy":true,"linkquality":128}'

Repeating the same for the Wireless Switch and a new device with ID 0x00158d00023d229c showed up - excellent!

info: New device with address 0x00158d00023d229c connected!
info: MQTT publish, topic: 'zigbee2mqtt/bridge/log', payload: '{"type":"device_connected","message":"0x00158d00023d229c"}'
info: MQTT publish, topic: 'homeassistant/sensor/0x00158d00023d229c/click/config', payload: '{"icon":"mdi:toggle-switch","value_template":"{{ value_json.click }}","json_attributes":["linkquality","battery","voltage","action","duration"],"force_update":true,"state_topic":"zigbee2mqtt/0x00158d00023d229c","availability_topic":"zigbee2mqtt/bridge/state","name":"0x00158d00023d229c","unique_id":"0x00158d00023d229c_click_zigbee2mqtt"}'
info: Device incoming...
info: MQTT publish, topic: 'zigbee2mqtt/bridge/log', payload: '{"type":"pairing","message":"device incoming"}'
info: MQTT publish, topic: 'zigbee2mqtt/0x00158d00020b3dff', payload: '{"occupancy":false,"linkquality":128}'
info: MQTT publish, topic: 'zigbee2mqtt/0x00158d00023d229c', payload: '{"click":"single","linkquality":94}'
info: MQTT publish, topic: 'zigbee2mqtt/0x00158d00023d229c', payload: '{"click":"double","linkquality":120}'
info: MQTT publish, topic: 'zigbee2mqtt/0x00158d00023d229c', payload: '{"click":"triple","linkquality":120}'
info: MQTT publish, topic: 'zigbee2mqtt/0x00158d00023d229c', payload: '{"click":"quadruple","linkquality":115}'
info: MQTT publish, topic: 'zigbee2mqtt/0x00158d00023d229c', payload: '{"click":"long","linkquality":118}'
info: MQTT publish, topic: 'zigbee2mqtt/0x00158d00023d229c', payload: '{"click":"long_release","duration":1236,"linkquality":118}'

Again, we can adjust that a human-readable name is displayed instead of the ID in /opt/zigbee2mqtt/data/configuration.yaml:

devices:
  '0x00158d00023a21c8':
    friendly_name: 'temp_hum_sensor'
    retain: false
  '0x00158d00020b3dff':
    friendly_name: 'motion_pir_sensor'
    retain: false
  '0x00158d00023d229c':
    friendly_name: 'smart_button'
    retain: false

The new names will now be used for the corresponding MQTT topics:

info: MQTT publish, topic: 'zigbee2mqtt/motion_pir_sensor', payload: '{"occupancy":true,"linkquality":94,"battery":99,"voltage":3025}'
info: MQTT publish, topic: 'zigbee2mqtt/temp_hum_sensor', payload: '{"temperature":27.79,"linkquality":110,"humidity":55.92,"battery":99,"voltage":3005}'
info: MQTT publish, topic: 'zigbee2mqtt/smart_button', payload: '{"battery":100,"voltage":3042,"linkquality":68,"click":"single"}'
info: MQTT publish, topic: 'zigbee2mqtt/smart_button', payload: '{"battery":100,"voltage":3042,"linkquality":60,"click":"double"}'
info: MQTT publish, topic: 'zigbee2mqtt/smart_button', payload: '{"battery":100,"voltage":3042,"linkquality":63,"click":"triple"}'
info: MQTT publish, topic: 'zigbee2mqtt/smart_button', payload: '{"battery":100,"voltage":3042,"linkquality":57,"click":"quadruple"}'
info: MQTT publish, topic: 'zigbee2mqtt/smart_button', payload: '{"battery":100,"voltage":3042,"linkquality":57,"click":"long"}'
info: MQTT publish, topic: 'zigbee2mqtt/smart_button', payload: '{"battery":100,"voltage":3042,"linkquality":65,"click":"long_release","duration":1015}'

Note that the Occupancy Sensor sends a true or false payload depending on if motion was detected or not. The Temperature & Humidity Sensor just sends number values for temperature and humidity. And the Wireless Button sends click events and can discern between single, double, triple, quadruple and long / long_release clicks.

Using Zigbee Devices to control INSTAR IP Cameras in Node-RED

We now set up the Zigbee Bridge, connected Zigbee sensors and made sure that the messages are send to our Mosquitto MQTT server. We also found out the topics that our bridge uses for each sensors to publish its messages - zigbee2mqtt/motion_pir_sensor, zigbee2mqtt/temp_hum_sensor and zigbee2mqtt/smart_button - and what kind of payload to expect.

Now we can use the program MQTT.fx to verify that the message was received by our Mosquitto server:

Every time the state changes the sensor will send a Zigbee message to our Bridge, which then forwards its payload to our Mosquitto server - perfect!

We can now start our Node-RED - that we either installed under Windows 10, on a Raspberry Pi or a Debian Linux Server, add MQTT Nodes and subscribe to all 3 MQTT topics:

Connect all 3 of them to a Debug Node and deploy the flow. Click the button, warm up the temp sensor and trigger the motion detection - the MQTT messages will show up in the debug pane in Node-RED:

Add the MQTT input to an existing Flow

In the following sequence we are using the input for the wireless button to switch Alarm Areas on our cameras - we already used the same sequence for the Amazon Alexa Integration (the complete flow can be found in the Home Assistant Tutorial):

You can copy the following JSON code and import it into Node-RED (How do I import flows to Node-RED?):

[{"id":"ebde9a4c.90efe8","type":"link out","z":"17b43655.05bd3a","name":"","links":["57b22bf7.0e2874","5eb0d369.b3b15c","b5ce97db.b3f4f8","f335d7f4.4bbe18"],"x":1308,"y":100,"wires":[]},{"id":"accb31dd.a130c","type":"mqtt in","z":"17b43655.05bd3a","name":"Wireless Button","topic":"zigbee2mqtt/smart_button","qos":"1","broker":"40415d07.67dd94","x":780,"y":100,"wires":[["2537690e.6230d6"]]},{"id":"2537690e.6230d6","type":"json","z":"17b43655.05bd3a","name":"","property":"payload","action":"","pretty":false,"x":929,"y":100,"wires":[["17df313d.2e7aff"]]},{"id":"17df313d.2e7aff","type":"change","z":"17b43655.05bd3a","name":"single / double","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.click","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":1076,"y":100,"wires":[["1f97ee5b.dfc642"]]},{"id":"1f97ee5b.dfc642","type":"string","z":"17b43655.05bd3a","name":"","methods":[{"name":"replaceAll","params":[{"type":"str","value":"single"},{"type":"str","value":"{\"val\" : \"on\"}"}]},{"name":"replaceAll","params":[{"type":"str","value":"double"},{"type":"str","value":"{\"val\" : \"off\"}"}]}],"prop":"payload","propout":"payload","object":"msg","objectout":"msg","x":1226,"y":100,"wires":[["ebde9a4c.90efe8"]]},{"id":"40415d07.67dd94","type":"mqtt-broker","z":"","name":"Debian Mosquitto","broker":"localhost","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]

The MQTT Node outputs the message from our button as a string. We first have to turn it into a JavaScript Object with a JSON Node.

Next follows a Change Node that changes the Message Payload to the value of Click - as we learnt above, this value can either be single, double, triple, quadruple, long or release:

After that follows a String Node (this node type has to be installed manually - it is called node-red-contrib-string) that searches for the payload single or double and replaces them with something that we need to trigger the Flow below - in this case we need a JSON expression that either sets a value to on or off: {"val" : "on"} / {"val" : "off"}:

And last but not least we add a Link Node to plug it in to all 4 sequences for the for the 4 alarm areas. Now every time we single-click the Wireless Button all areas are activated. A double-click deactivates them:

We attached a Debug Node to the MQTT input to visualize the input given by the Wireless Button - Note that only the single and double click trigger the Alarm Area Sequences - the rest of the input options can be used for other functions.