Latency reduction / increase performance

[robertugo2]

Hi. Probably all of you are using Z2M in combination with some automation SW like HA / AppDaemon / NodeRed. Possibly some of you have some automations like "when button is pressed, then toggle a light".

I am one of such users and recently I was working on how latency between button press and the light status change can be minimalized. I want to share with you my findings and ask you, if you can propose something more or additional areas, where I can take a look.

So lets start with my findings:

Get rid of docker, if you are using it
Using docker adds additional delay in processing and adds a lot of new threads, that are needed to be managed by the task scheduler. That's of course introduce delay in processing. Use native installation methods to mitigate that problem.
EDIT
Problem is mainly with IO latency, creating new threads and context switching. Overall performance is not affected, but in terms of latency / reaction time there is a problem. References:
[1] https://www.researchgate.net/publication/341483813_Performance_Evaluation_of_Container_Runtimes
[2] https://www.phoronix.com/scan.php?page=news_item&px=AppArmor-Linux-5.5-72-Tests&fbclid=IwAR3u3nyP7B6XN4DgvPyD1Zrjw4oAbQfye5DRO95R1m6glgQ9_9EBAfdVDW4
[3] https://unix.stackexchange.com/questions/73515/does-apparmor-decrease-the-system-performance

Using Real time kernel
I have found out, that using the RT kernel like https://packages.debian.org/pl/linux-image-rt-amd64 can improve overall response time for serial / network events.

Use Round-robin scheduling policy
For mqtt broker, automation SW and Z2M use RR scheduling policy. In combination with RT kernel it will give those processes real time capabilities. Info https://man7.org/linux/man-pages/man7/sched.7.html
You can ensure, that RR scheduling policy is enforced by adding scheduling policy to systemd configuration file, for example:

...
CPUSchedulingPolicy=rr
CPUSchedulingPriority=80

Use dedicated automation SW
Using HA for automations is simple, but HA is quite big and automations are relatively slow. Using for example AppDaemon with MQTT addon gives a performance boost. Also thanks to that, you can put only automation SW into RR scheduling policy, leaving HA with default one.

Set "latency-performance" profile
I'm using "tuned" daemon to do that. See https://computingforgeeks.com/optimize-linux-system-performance-with-tuned-adm/ for more info.

Tune Z2M settings
It is needed to disable functions, that are not needed and disable some attributes from publishing.
General ones:

• "legacy_api": false,
• "homeassistant_legacy_triggers": false
  Per button/switch:
• Add "click" to filtered attributes

Other things
And that's all, what I found. I will be glad, if you can share your opinion and maybe add something to this topic.

[Gis70]

Hi, thx a lot for this feedback, I'm using docker for simple management of several things xx2mqtt. I have latency issue with buttons, sometime it's short, sometimes long not everytime the same latency.

[robertugo2]

I had similar problem. First reaction for a button click after some time was always longer than just after this first one. Like something needed to wake up, I don't know. But after moving everything to OS level it disappeared, so every request is processed with similar latency,

[tasmatto]

I had similar problem

[mcmatti]

I had the same issue, main problem was that I couldn't figure out how to identify the bottleneck in HA on Raspi. Post in HA forum

[Tediore]

Currently running HA and Zigbee2mqtt Docker containers on Debian 10, and HA automations related to Zigbee button presses are instantaneous... what sort of latency were you noticing before making these changes?

[Gis70]

Lucky you. I need to debug log to see where I spent time. I have more than 30 devices. Docker is on a virtual machine on a gen8 microserver. My home automation is Jeedom on another VM. When I click a button, in the z2m log UI, it took about a second to show the event. What do you trigg with your button ?

[Tediore]

A couple buttons trigger a light, one triggers a fan or space heater depending on season, one triggers a "good night" automation that does a bunch of stuff.

[atxbyea]

I have 100ish devices, with zero latency at all, I haven't done a single tuning command tbh, 5 door sensors, 6 motion sensors, 7 buttons of different flavours and all my lights.

Running on a raspberry pi 3b+ in docker, only running zigbee2mqtt and zwave2mqttjs, the rest of my environment runs between a r510 with 24 cores and a mini itx box with 4 cores, all in docker containers

[robertugo2]

After some time without using given button - latency goes even for 1s. After "waking up" (I don't know what was causing that) latency was about 300ms. Now I have 80ms between seeing mqtt message with "action" from button to see light status change on mqtt.

[m0wlheld]

Get rid of docker, if you are using it

Stopped reading ...

[robertugo2]

Please be aware, that I'm not talking about "performance" that is the same or even worse on RT system, but about "latency", so speed of reaction. Docker, as any other virtualization system, creates some additional layer between HW and app. So thanks to that we can speed up the response for serial data.

[m0wlheld]

This is where you are wrong. Docker is not a virtualization system, it is a process isolation system, you still use the same kernel and the same peripheral as the host.

I‘d say your performance issue is HA and the current slow-down issue with their MQTT add-on.

[robertugo2]

Is there any known issue with HA mqtt addon? My opinion is based on actual measurements, that I done before and after moving everything to OS level. I know, that Docker is not a virtualization system, but it abstracts IO level operations to protect host system.

[artkrz]

Why are you so sure that docker or actually any software is causing your latency problems? Perhaps it's your coordinator, network. I'm not questioning that steps you took helped but maybe Docker and underlying OS isn't a root of your problem?

[robertugo2]

To be honest - no. But when I moved everything to OS level, I observed big speed up on everything.
Thing, that could cause that, besides the docker:

• AppArmor - that is enabled for HA managed containers
• HA Supervisor / DNS / ... can cause additional overhead for that
  I don't have such knowledge to track everything down, but I can write, that I observed a change.

[artkrz]

The reason I'm asking is that it sounds really strange. I'm using Conbee2 with Home Assistant installed on Proxmox VM (which means I additionally pass USB from Hypervisor to Guest VM and then to docker z2m container) and when I turn a light bulb from a wall switch connected to shelly i3 (wifi) the bulb turns on instantly, no visible lag at all.

[robertugo2]

Possibly you have quite good setup. I'm running everything on Dell WYSE with AMD processor.

[netweaver1970]

I experience the same issue with the initial press of a Z2M managed button taking a lot long time to be end-to-end actioned by HA but subsequent presses have results very fast.
I'm using always the latest Z2M as a docker instance on a Raspi3B, connecting to a HASS OS + MQTT add-on on a Raspi4. But I saw the same behavior when all components were on the same Raspi3B, using older HA versions (supervised install).

While directly Zigbee bound switch-light connections are ALWAYS very fast. But then of course there's no Z2M, MQTT broker, HA automation in the middle to slow things down.

[robertugo2]

@netweaver1970 I had the same issue! First one was always very slow and subsequent ones were much faster! I was digging into it and the only solution, that I found, was to move everything to OS level. I have some suspicious, that mosquitto addon could cause some problems, but as I moved everything to OS level I don't have such problem anymore.

[Gis70]

What kind of coordinator do you have all ? I have slaesh’s cc2652rb with lastest firmware in zigbee 3.0.

[robertugo2]

cc2538+cc2592

[mailwash]

Hi. I'm using Aqara wireless wall switches + automations based on events form them. I wonder if there is a way to "disable" double clicks to make them "not wait" for second click. (in most cases i'm using only one click and double click).

[Semmu]

that would require a firmware change (so it immediately sends zigbee messages on the first click and does not wait if the second click happens or not), and unfortunately i doubt it can be done, as they are closed source.

[bartplessers]

Interested too. The latency caused by some "smart" bottons with single, double, tripple, quadrupple,long, double, ... click drives me mad. Right now, I try to avoid such devices in simple scenario's (i.e. wall switch in room). Of course they still are of use in more complex automations.

B

[timdonovanuk]

I have a fairly complex setup, including:

• zigbee stick running on a Pi using serial over tcp (ser2net)
• Proxmox virtual machines for all servers
• zigbee2mqtt is running in docker
• home assistant is running on another vm
• automations are handled by nodered running on home assistant, with some reasonably complex flows

I'd hate to follow the actual round trip of pushing a button - even if you ignore abstractions like virtualisation of networking and docker, it's probably at least 12+ hops from button push to lights on :)

If I understand my logs correctly, the delay between pushing a button and a light coming on (or at least z2m reporting it is on), is 300ms

2021-04-08T09:59:03.628766087+01:00 MQTT publish: topic 'zigbee2mqtt/bathroom_2_button_1', payload '{"action":"single","battery":100,"last_seen":"2021-04-08T09:59:03+01:00","linkquality":60,"voltage":3042}',
...
2021-04-08T09:59:03.971322241+01:00 MQTT publish: topic 'zigbee2mqtt/bathroom_2_ceiling_light_1', payload '{"brightness":191,"color":{"x":0.345100160725069,"y":0.35160700531884},"color_temp":200,"last_seen":"2021-04-08T05:28:25+01:00","state":"ON"}',

This is also using a CC2501, the slowest of the slow zigbee sticks, and using the old homeassistant_legacy_triggers, which generate a ton of noise and mqtt traffic.

I would think 300ms is fast enough to be considered instant - it can't out perform a physical light switch, as that operates at basically the speed of light.

[bartplessers]

Hi everyone,

Although I'm a huge fan of docker, I must say that I had similar problems (see #7108)
I have multiple instances of HA running on all kind of machines, but the only thing I changed now was moving my Mosquitto-broker from the HA addone to vanilla MQTT-broker on my rpi.

Latency seems to be dissapeared immediatly....

Don't know if this due to docker architecture, or due to the implementation of mqtt-broker in HA-addon....

kind regards,
Bart Plessers

[atxbyea]

Add-ons definitely have been known to do .... Interesting stuff, and is one of the reasons I'm not using that locked down appliance style with forced updates and no personal freedom.
I do however see how many users are struggling with HAOS issues that no other installation methods suffer from.

Easy != Good

[bartplessers]

Maybe I have to correct myself a little bit.

I moved from docker to native software (MQTT- broker, Z2M and mosquitto_sub on command line to publish images.
Altough things run slightly smooter, the latency (and even worse: errors) still exist. See also
#7163

Kind regards,
Bart

[gchrisak]

I have some xiaomi buttons that control smart lamps(wifi).
Since around two months ago im experiencing delays of around 1-2 seconds on the initial button press.
After the initial button press all other presses are instant. Then after a few minutes of no action, the initial press is delayed again.

Z2M logs that initial button press instantly so I know the issue is not there.
I also tested with ZHA and there is no delays.

So is this an issue with the MQTT broker? Anyone knows if they changed something the last few months that would cause this?

[atxbyea]

Which buttons? I use the Xiaomi rounded corner single button and wireless dual switch buttons, they have literally 0 delay, whenever pressed.

[Gis70]

Lucky you. can you describe your system ? (Docker ?), Broker in a docker ?

[gchrisak]

Which buttons? I use the Xiaomi rounded corner single button and wireless dual switch buttons, they have literally 0 delay, whenever pressed.

These WXKG01LM

[bartplessers]

Same issue here.
I tried to narrow this down and came to

#7163

Things are smoother since I run software natively, but publishing bulk messages induces latency and/or errors

[Wireheadbe]

One of the best latency reductions you can do..

• Put your devices in Z2mqtt groups
• Bind your switches directly to the groups in Z2mqtt
• Only use HA for "complex" non-realtime automations where a couple of msec don't make a difference.
• Bonus: your switches work without HA as well..

[Hagbard235]

Hi, I have a similar issue. Instead of or in addition to the mqtt message, I would like to react to an action with a rest/webhook call to avoid mqtt latency. For me, a relay (non zigbee) should be started when a key is pressed and stopped when it is released, and you can imagine that the detour via mqtt means latency. Does anyone have any ideas or tips? I would also consider self-customized legacy solutions.