Zigbee devices became unavailable in HA after restart or reboot of HA

[Motya-HA]

What happened?

After recent update to laters version Z2M (1.33.0-1) all Hue Zigbee devices are droping from network and become unavailable. Requires Z2M restart to bring them back

What did you expect to happen?

HUE zigbee devices will not drop from network and be stable

How to reproduce it (minimal and precise)

No response

Zigbee2MQTT version

1.33.0-1

Adapter firmware version

6.10.3.0 build 297

Adapter

ITEAD_SONOFF_Zigbee_3.0_USB_Dongle_Plus_V2

Debug log

No response

[Bcow7]

Same here. I don't have any error message in the logs. It works well, then one random command fails and the device is not connected anymore to the network.

I have a Conbee II adapter.

A reboot of the RPi solves the issue until next failure.

Can it be an incompatibility between some new frames in the last update and the firmware of the adapter?

[Motya-HA]

Correction. These are not just Hue devices.
Recent check showed that pretty much all Zigbee devices with exception to inovelli dimmers

[KoalaWerewolf]

Just a comment based on my setup. I had problems with that firmware on the ZBDongle-E, I'm now on 7.1.1 and it is working much better.

[Motya-HA]

@KoalaWerewolf
Thanks for suggestion.
I can't see 7.1.1 firmware in that site:
https://github.com/xsp1989/zigbeeFirmware/tree/master/firmware/Zigbee3.0_Dongle-NoSigned/EZSP

[KoalaWerewolf]

I used this way of flashing: https://darkxst.github.io/silabs-firmware-builder/
and the firmware from here: https://cloud.time-vortex.net/s/Jy2x6tj2GNx2QZn
(check this post and the thread above: Koenkk/zigbee-herdsman#319 (comment))

[Motya-HA]

@KoalaWerewolf
Thanks
I'll try when I'll be back home (traveling now)
Can I do it with Mac or Linux?
I do not have any Windows machine at home

Thanks

[KoalaWerewolf]

I’m sure you can, but I don’t know if that webflasher works.

[Grandma-Betty]

Same here. Almost all ZigBee devices are disonnecting. It's a mess.
I'm on the Sonoff ZigBee 3.0 USB dongle with latest custom firmware.

[Motya-HA]

Some more details - every time there is an update or home assistant restart, all Z2M devices become disconnected. Restart of Z2M add on fixing the issue.
Any plans to address this bug?

[stevehollaar]

I have this issue as well: my entities provided by Zigbee2MQTT (via Mosquitto broker) become "Unavailable" when I restart Home Assistant, until I also restart the Z2M add-on. I'm not sure how to debug this

[mundschenk-at]

Are they disconnected from Zigbee2MQTT/the ZigBee network or just "unavailable" in HA? The later would most likely be an issue with retained MQTT topics (or the lack thereof), the former would be more serious.

[Motya-HA]

They became "unavailable" in HA. In order to make them "available" I need to restart Z2M add-on. sometimes several times.
I've updated the topic subject

[stevehollaar]

This debugging thread on the MQTT side of things may be helpful: home-assistant/core#99936 (comment)
It seems Z2M may be dropping entity names sent to the config topic, leading to a conflict when it falls back to the device class. There's a bugfix in the upcoming 2023.9.2 release that may help, but I'm unsure if a fix is also needed in Z2M

[Motya-HA]

I think fix is needed in Z2M in a first place. All these issues have started after recent update for Z2M

[rossowl]

Same problem. Devices and groups are unavailable after HA restart until I manualy restart Z2M.
3 Z2M networks:
Sonoff (Z2M docker 1.33.1)
Sonoff (Z2M docker 1.33.1-dev)
zzh! (Z2M HA addon 1.33.0)

[cpuks]

@rossowl do you still encounter that as I had to downgrade to 2023.8.4 HA due to UI slow/laggy so want to confirm if I'll have to deal with that now - every HA restart == Z2M restart?

[Motya-HA]

I confirm I still have this issue:
Z2M HA addon 1.33.1-1
Home Assistant 2023.10.4
Supervisor 2023.10.0
Operating System 11.0
Frontend 20231005.0

[cpuks]

OK I didn't have that when running external mqtt/z2m now with fresh HA VM I've got mqtt broker from addons and it's an issue.

[Motya-HA]

now it requires 2 or even 3 restarts of Z2M add-on for some devices to become responsive :-(

[francisp2]

OK I didn't have that when running external mqtt/z2m now with fresh HA VM I've got mqtt broker from addons and it's an issue.

The mqtt add-on is very buggy at the moment

[Motya-HA]

The mqtt add-on is very buggy at the moment

I do not have any issues with Z-Wave JS UI (uses MQTT as well)

[mundschenk-at]

Z-Wave JS UI does not use MQTT when integrated with HA.

[Motya-HA]

"The Z-Wave JS UI add-on provides an additional control panel, allowing you to configure every aspect of your Z-Wave network. It provides a decoupled gateway which can communicate using Z-Wave JS WebSockets (used by the Home Assistant Z-Wave JS integration) and MQTT (even simultaneously)"

[mundschenk-at]

Yes, you can enable MQTT (disabled by default!). But the Z-Wave JS integration does not use it even if you do. So unless you have enabled the MQTT client in the Z-Wave JS UI settings and don't use the Z-Wave JS integration, your HA instance does not use MQTT to access your Z-Wave devices.

[Motya-HA]

I did enable it

[rossowl]

I can confirm that with external mqtt everything works.
Intel NUC:
Hass OS
MQTT addon disabled

RPi4 with docker:
eclipse-mosquitto
2x zigbee2mqtt

[Motya-HA]

Installed Zigbee2MQTT in Linux box (stoped add-on in HA). Rebuild whole Zigbee network. Same issue. After restart of HA (Zigbee2MQTT run on Linux box interrupted) all my Zigbee devices becaem unavailable. Need 2, 3 reboot of linux box with Zigbee2Mqtt to make all devices responsive again

[xekil]

Same problem, HA and ADD-ON up to date, have you made progress on this problem? thanks in advance

[popy2k14]

@xekil here was made some progress: #21500

[Motya-HA]

I completely solved the problem with moving to EMQX broker (from Mosquito)

https://www.emqx.io/

[popy2k14]

@Motya-HA thx for the hint. Is it just as easy as uninstall Mosquitto HA addon with EMQX HA addon?
Is there also a good integration of the addon like on mosqouitto (HA users, ...)?

[Motya-HA]

Stop Mosquito add-on. Download EMQX add on and follow instructions

https://github.com/hassio-addons/addon-emqx/blob/main/emqx/DOCS.md

Good luck

[Motya-HA]

this is good instructions:

https://smarthomescene.com/guides/setting-up-emqx-mqtt-broker-in-home-assistant/

[convicte]

I can confirm EMQX is very feature-rich and capable, having it set up for testing in my secondary HA instance.

My only big gripes are that if you are using any kind of MQTT bridges (in my case the SUPLA electricity meter) it loses connection during HA/EMQX reboots and even though it's enabled, it never comes back up. I have to manually go in, log into EMQX UI, turn it OFF and ON, to make it work. No setting I've been able to find resolves this issue, including clean reboot on start.
Not even the EMQX reboot fixes it. You have to do an OFF/ON for some reason from the UI.
There is also no API from HA that I could find to automate the process, so it's a good old manual labor when the readings go down.

Second, 'with great power comes great responsibility' and all the features are incredibly overwhelming to start with, especially when devs decide to change menu names and layout every few weeks/months. I've spent some time finding options I used to use, since they were moved and drowned in the jungle of switches and nobs.

TL;DR, if it resolves the above-mentioned issue, I guess it's a no-brainer.

[xekil]

@xekilici des progrès ont été réalisés : #21500

Thanks, but will this be fixed quickly?

[convicte]

See the latest comments.
The proposed fix doesn't work, so NO it won't be.

[xekil]

What reliable solution then?
Change MQTT Broker?
Make an MQTT MOSQUITTO VM without guarantee that it works?
THANK YOU

[convicte]

Manage your expectations - people invest their own time to provide something to you for free, so getting frustrated and demanding over a small bug is not going to get you anywhere.
If you seek guaranteed solutions, find a commercial home automation platform, pay for it and expect the provider to guarantee reliability and long term support.
Sorry for stating the obvious, but it felt necessary.

Potential 'solutions':

1. Wait patiently for someone to provide you an 'out of the box' fix, which may take days to months.
2. Invest your own time into testing a solution/workaround for yourself, e.g. a different broker, automation rebooting the broker when things go unavailable or anything else you find on the internet as a path worth following.
3. Lastly, if neither of the above seem appealing enough, continue to reboot HA/MQTT when things go offline, since it's not that much of a problem in the first place (been doing it for a few months now).

These are the pleasures of community driven, free project! Be a part of the community that makes things work together, or buy a product! ;)

[xekil]

I think we need to clarify things, I am not at all frustrated or demanding, far from it, and quite the contrary, I spend a lot of time helping others for free in several areas, and I am very grateful to everyone those who work to make home assistant the best home automation solution.

I am just surprised that this problem which seems major to me is not one of the priorities or rather that there is no official communication on this problem which does not seem to be an isolated case especially since the majority of the Home community Assistant is essentially equipped with Zigbee and uses the Zigbee2MQTT and MOSQUITO Add-ons.

I already have a workaround solution (restarting the add-on when starting home assistant with an automation) and it works, but it shouldn't work like this, I'm wondering, because I don't have a clear answer on this problem and whether a possible solution will be found or not, in which case, it is obvious that I will switch to another broker or another instance to try, etc... With Proxmox everything is easier :)

Sorry if my messages may have seemed frustrated and demanding, but that is not the case, I have been using open source software for several years and I know very well that you should not have the same requirements as for something of General public ;)

[Motya-HA]

What reliable solution then? Change MQTT Broker? Make an MQTT MOSQUITTO VM without guarantee that it works? THANK YOU

I moved to EMQX (I am running HAOS) and I have "0" issues. It took literally 10-20 min (I have 2 Zigbee2MQTT add-ons running with 2 different NET coordinators). I do not know if they are changing interface, or anything else. It just works.

[popy2k14]

this is good instructions:

https://smarthomescene.com/guides/setting-up-emqx-mqtt-broker-in-home-assistant/

Thanks a lot for this link.
Switched to EMQX now and restarted HA several times -> YESSS!! All Z2M devices where available!
With mosquitto, after 1-2 restarts, several devices where not available.

Will see how it goes in a few weeks.
Maybe (finally!) i can ditch my Z2M resatart automation.

[Motya-HA]

Will see how it goes in a few weeks. Maybe (finally!) i can ditch my Z2M resatart automation.

LOL...I had restart automation as well. Made it disabled, deleted after 1 month

Glad it worked for you as well

[xekil]

I really want to test this EMQX add-on but I'm afraid of having duplicate entities or ones that change their name from MOSQUITO to EMQX.

How does it work for MQTT entities that have been renamed in Home Assistant?

For example, a sensor which is reassembled with MOSQUITO to Home Assistant named sensor.0x25422548654 and which I then rename in Home Assistant to sensor.door_garage, will it remain named sensor.door_garage with EMQX as broker or will it reassembled in sensor.0x25422548654?

THANKS

[popy2k14]

@xekil i Have 145x MQTT Z2M devices (~1300 entities) and not a single one has issues after migrating!

Just stop mosquitto, install/start EMQX, configure HA & Z2M User. Configure Z2M to use the new broker and user. Press "Reconfigure" on the MQTT integration and fill in the new mqtt://a0d7b954-emqx:1883 path.
See here for more information: https://smarthomescene.com/guides/setting-up-emqx-mqtt-broker-in-home-assistant/

Good luck

[xekil]

@popy2k14 Thank you for this feedback, I was a little scared because I have 165 devices :)
I've already read this guide and it doesn't seem very complicated, but there was no mention of the possible renaming in home assistant, we'll see :)
I will make a Proxmox backup just before migrating or case :)
THANKS

[popy2k14]

@xekil you're welcome. A backup doesn't hurt 😉

[xekil]

Migration completed successfully, nothing to report and the entities have not changed their name, all my Add-ons are operational, relieved :)

I created users identical to those I had created in Home Assistant for the MOSQUITO Add-on and therefore for the MQTT integration I did not have to do anything, because I also created a user who had the same password that was automatically generated by the MQTT integration.

So, in case a rollback should happen, I have a minimum of configuration to do ;)

For the moment, following several restarts of Home Assistant, I no longer had any problems with Zigbee2MQTT, and I was finally able to deactivate my automatic restart of Zigbee2MQTT when starting Home Assistant!

Thank you all for your feedback and long live EMQX.

[popy2k14]

@xekil Nice to hear that you have successfully migrated to EMQX.
For me EMQX running stable (at least a week).
For a long term test, we will see :-)

[Motya-HA]

For me it runs for 3 months at least. I even forgot it's name - probably good think.

Just launch and forget...
This is the way it should be with everything :-)

[deviantintegral]

I came across this and am surprised. I have two separate HA installs using zigbee2mqtt plus the mosquito broker, and I've never had this issue. It makes me think the bug can only be reproduced by some configuration option somewhere?

[zen2]

I came across this and am surprised. I have two separate HA installs using zigbee2mqtt plus the mosquito broker, and I've never had this issue. It makes me think the bug can only be reproduced by some configuration option somewhere?

Do you use zigbee2mqtt and mosquitto HA add-ons or plain ones ?

It seems the problems is only with HA add-ons. I use zigbee2mqtt and mosquitto services installed next to HA and so no HA add-ons and didn't have any problems.

[deviantintegral]

I have one install where zigbee2mqtt is separate, in an LXC container, pointing to mosquito running as an HA addon. In the other, everything is in HAOS on a Raspberry Pi.

[mbendtsen]

I also tried to switching to EMQX, but ran into trouble with Node-Red and zigbee2mqtt nodes. Some times the devicelist would not update and just give an get devices timed out error.

And btw, my behavior of this problem is only with devices number 79 and up. Everything else works fine, but the newly added devices. Restarting z2m seems to work.