Zigbee2MQTT 2.0.0 breaking changes

[Koenkk]

This discussion contains a list of breaking changes for Zigbee2MQTT 2.0.0, the list of breaking changes will be updated gradually. Feedback and suggestions are welcome!

Most of the breaking changes are about removing legacy features that have been deprecated for years (replaced & improved since), and as such, automatically disabled for newer networks.

Breaking changes

• Improved adapter discovery
  • When upgrading from Zigbee2MQTT 1.x.x, this might result in USB adapter discovery error (No valid USB adapter found), see Z2M 2.0.0: `USB adapter discovery error (No valid USB adapter found)` #24364 how to fix this
• The MQTT legacy API has been removed (advanced.legacy_api setting), this includes the following MQTT topics:
  • zigbee2mqtt/bridge/log
  • zigbee2mqtt/bridge/(bind|unbind)
  • zigbee2mqtt/bridge/group/*
  • zigbee2mqtt/bridge/networkmap/*
  • zigbee2mqtt/bridge/ota_update
  • zigbee2mqtt/bridge/device/DEVICE_FRIENDLY_NAME/get_group_membership
  • zigbee2mqtt/bridge/config/*
  • zigbee2mqtt/bridge/configure
• Removed deprecated zigbee2mqtt/bridge/request/config MQTT API, this includes the following MQTT topics:
  • zigbee2mqtt/bridge/request/config/last_seen
  • zigbee2mqtt/bridge/request/config/elapsed
  • zigbee2mqtt/bridge/request/config/log_level
  • zigbee2mqtt/bridge/request/config/homeassistant
• Removed deprecated features:
  • Report feature (advanced.report setting)
  • Soft reset feature (advanced.soft_reset_timeout setting)
  • Retrieve state option (devices.IEEE_ADDR.options.retrieve_state setting)
• Removed several deprecated availability settings: availability_timeout, availability_blocklist, availability_passlist, availability_blacklist, availability_whitelist and advanced.legacy_availability_payload (see availability docs for the new settings)
  • Due to the removal of advanced.legacy_availability_payload, zigbee2mqtt/bridge/state will now always be a JSON object ({"state":"online"} or {"state":"offline"})
• Changes to permit join (to address security concern and sometimes unstable behavior):
  • The option to permit joining forever has been removed; it is now limited to a maximum of 254 seconds.
  • The permit_join setting has been removed, use the frontend or MQTT to permit joining
• The update_available property from the device state has been removed (use update.state instead)
• Renamed updateAvailable to update_available in zigbee2mqtt/bridge/response/device/ota_update/check
• Breaking for "bare-metal" installations:
  • For Git based installations: removed configuration.yaml from repo, this will result in the following error when updating:

    error: Your local changes to the following files would be overwritten by merge:
        data/configuration.yaml
    

    To fix this, update by executing:

    mv data/configuration.yaml data/configuration.yaml.bak
    ./update.sh # (update as you usually do)
    mv data/configuration.yaml.bak data/configuration.yaml

  • pnpm is now used instead of npm, the installation instructions and update.sh script has been updated accordingly. For existing installations you need to execute npm install -g pnpm before updating.
• Removed configuring group members through configuration.yaml (groups.devices setting). This will not impact current group members; however, you will no longer be able to add or remove devices from a group through the configuration.yaml.
• For zigbee2mqtt/bridge/request/device/bind, zigbee2mqtt/bridge/request/group/members/add, zigbee2mqtt/bridge/request/group/members/remove and bridge/request/device/configure_reporting, the endpoint has been moved to the endpoint property and cannot be set via the device/from property anymore.

TODO (not implemented yet)

• Remove Home Assistant legacy:
  • homeassistant>legacy_triggers - action/click sensor (already deprecated for a long time)
  • homeassistant>legacy_entity_attributes
  • device>xyz>legacy
  • remove duplicate select/sensors
  • remove update available and state sensors
• Remove all legacy options
• Remove legacy device option (mostly affects legacy.ts)
• Log unused options on z2m startup
• Check settings.schema.json for more option to deprecate

PRs

• feat!: Improved adapter discovery. zigbee-herdsman#1197
• Update docs for improved adapter discovery zigbee2mqtt.io#3061
• fix(ignore): Changes for improved adapter discovery #24147
• Update adapter discovery docs zigbee2mqtt/hassio-zigbee2mqtt#639
• feat!: The big cleanup #24200
• fix: HA permit join switch: disable automatically after 254 seconds #24245
• Remove retrieve_state zigbee2mqtt.io#3075
• Remove readAfterWriteTime zigbee-herdsman-converters#8089
• feat!: Improve permit join. zigbee-herdsman#1214
• fix: Improve permit join #24257
• Permit join MQTT API change. zigbee2mqtt.io#3092
• Docs changes for "the big cleanup" pr zigbee2mqtt.io#3094
• Switch to pnpm zigbee2mqtt/hassio-zigbee2mqtt#645
• fix(ignore): Switch to pnpm zigbee-herdsman-converters#8121
• fix(ignore): Switch to pnpm zigbee-herdsman#1215
• Remove configuring group members through configuration.yaml zigbee2mqtt.io#3104
• fix!: Remove configuring group members through configuration.yaml #24338
• Remove configuring group members through configuration.yaml zigbee2mqtt.io#3104
• fix!: Remove configuring group members through configuration.yaml #24338
• API change for endpoint specification for bind, groups, configure reporting zigbee2mqtt.io#3123
• fix!: Improve bind/bridge/groups parsing and resolving consistency #24432
• fix: match API change for z2m PR 24432 nurikk/zigbee2mqtt-frontend#2165

[mundschenk-at]

I'd like to remove the legacy* discovery options for Home Assistant to simplify the code flow.

[chris-1243]

@Koenkk

If my understanding is correct, right now and by setting:

legacy_entity_attributes: false
legacy_triggers: false

it should simulate how it will work in v2.0.0.

Any time frame or roadmap for the release of v2.0.0?

@mundschenk-at

Could you be a bit more explicit where and how we may implement this solution in a blueprint? It is going to be a huge breaking change for me. I know it is not a HA github as the change is done via Z2M, I am asking (sorry, my knowledge are not that high in that topic...still learning😉).

[mundschenk-at]

@chris-1243 That really depends on what blueprints your are using. They might need to be adapted by their creators (and you would need to upload the new version). Now if you are talking about scripts or automations, most likely you would just need to replace sensor.my_device_action with event.my_device_action in your script or automation, but of course again it somewhat depends on what you are currently doing (since HA is pretty flexible).

[chris-1243]

@mundschenk-at Actually, what I do not understand is how would you select such entity in a blueprint. I have made some blueprints (the community was really helpful) some months ago. The legacy API creates the sensor which might be select in HA. I am just wondering if you might have some tips on how to achieve this

[mundschenk-at]

@chris-1243 The HA event support currently only exists as PR, so you cannot find that entity in any released version of Z2M.

[chris-1243]

@mundschenk-at Ok.... Now I see why I am stuck... I wanted to anticipate this change. I did not see you PR. Many thanks for your time and answer. It will be a minor change

[Drafteed]

My suggestion is to remove sensor.*_update_state and binary_sensor.*_update_available HA entities in favor of update entity.

[Koenkk]

added!

[Drafteed]

I'd also like to remove the duplicate sensor for select (enum) entities.

[Koenkk]

Also added!

[mundschenk-at]

So what's the timeframe for 2.0.0 approximately? (I am asking if I should do a larger clean-up PR for Home Assistant discovery instead of adding event entities to 1.x?)

Also, wouldn't it be easier if there was a dev-2.0 branch instead of keeping everything as draft PRs?

[Koenkk]

Lets aim before EOY

[devbis]

It would be great to add to dev console in UI writing and reading numeric (non-standard) cluster attributes.
Probably use simple numeric input field instead of the drop-down list.

Maybe it's not for z2m but for the frontend repo.

[hacscred]

How about implementing a tool to translate a valid 1.xx configuration to a valid 2.0 configuration?

Possibly with some instructions for porting actually incompatible features, but only when present.

The thing I liked about this project was its stability (only since the last 10 or so 1.xy releases). Why would you ruin your (until this decision) great reputation?

Software can always be made to be backwards compatible and it's not like that's hard to accomplish (every design patterns book even explains how to do that). It's just not a "cool" thing. Software has utility when it does work and you don't need to track its development.

Every second people need to care about you making things "better", is lost productivity.

Why are you breaking APIs and existing configurations again (that would want to upgrade to version 2)?

[Koenkk]

Not every change can be done without breaking, some wrong decisions were made in the past which have to be corrected at some time.

[hacscred]

It's literally not a constructive comment. You are just saying something is a fact, without proof.

The only valid reason to some extent is that "You don't want to", since I have no rights. (Pretending to act as if you answered the question is just annoying. )

There's nothing to discuss anymore, though. If you had wanted to answer the question, you would have done so the first time.

Thanks for the time of stability for as long as it lasted.

[Drafteed]

@hacscred What kind of hysterics? All breaking changes will be described in the migration guide, legacy parameters can be removed automatically without user intervention. Some small things will have to be fixed manually, but this is the first time in several years for this project. Breaking changes always happen during development to improve the code base and further improve the product itself. Even Home Assistant breaks something in every release (look at any release notes at the end).

In the end, no one is stopping you from staying on 1.x.x and migate to a new version whenever it’s convenient for you.

[mundschenk-at]

@hacscred From GitHub history, I assume your are not a software developer yourself. That's alright, your concerns are valid nonetheless, and backwards compatibility is something most developers strife for, because it makes life easier for everyone, up to a point. But no, keeping backwards compatibility is not always "not like that's hard to accomplish", and it does come with a maintenance cost.

As @Koenkk wrote, sometimes decisions prove to be unwise in retrospect, sometimes the environment (e.g. Home Assistant) has evolved since the original implementation and things have to change. In a lot of these cases, the original functionality has been deprecated for a long time, which means that there is an alternative, preferred way to do things and that notice has been given that the original functionality will be removed at some point in the future. So it's not like this completely out of the blue.

As @Drafteed noted, you don't have to change immediately to the new version. Also the necessary changes to your configuration will likely not be as big as it might seem from the technical roadmap if you mainly use Z2M with Home Assistant and directly by MQTT.

One point I'd like to add, regarding "I have no rights": You are not a customer. You don't pay developers (@Koenkk and other contributors) to cater to your specific requirements. So yes, you don't have the right to demand a certain outcome.

[hacscred]

@mundschenk-at Everything you said is wrong and misguided. Please apologize.

@Drafteed Do you ever say something that's not redundant?

[starox]

I am in favor of breaking change for 2.0 and major updates. It is not always possible or desirable to keep old stuff.

I have a question about the 1.x z2m.
How the old branch will evolve ?

• frozen and not touched anymore
• maintenance for security fixes
• some small improvement allowed

Is there a policy for z2m ?

[Koenkk]

My idea is: frozen and not touched anymore (because of time reasons)

[Chartreusito]

Remove Home Assistant legacy:

homeassistant>legacy_triggers - action/click sensor (already deprecated for a long time)

What are those supposed to be replaced by ?

I read here that actions are replaced by event, but how would it work with devices like old Ikea remotes, as i don't see events implemented yet ?

[Drafteed]

You can use my guide in the HA community:
https://community.home-assistant.io/t/aqara-buttons-all-in-one-zigbee2mqtt/268717/45?u=drafted

[Koenkk]

By device triggers: https://www.zigbee2mqtt.io/guide/usage/integrations/home_assistant.html#via-mqtt-device-trigger-recommended

[Chartreusito]

Ok then i'm already ok, thanks. Since i doubted about that, i may not be the only one to do so, it would be good idea to mention the device trigger doc while talking about this "deprecation".

[RobLeighton67]

So with Z2M 2.0 , will it be true that I cannot enable permit join via MQTT, wait up to 5 mins for a device to join, interview and configure then disable permit join via MQTT ?

[mundschenk-at]

Yes, insofar as the new maximum inclusion period will be 254 seconds (slightly more than 4 minutes) in the draft PR.

[RobLeighton67]

Too proscriptive for me. I wonder how many others will discover just how short 4 minutes is when you're running around a building trying to activate a sensor and check the result!

[Nerivec]

What's removed is the permit_join setting in configuration.yaml (a.k.a. "forever permit join"). Triggering via MQTT, of course, remains available.
254 seconds is indeed now always enforced as the maximum time to avoid issues (stacks reacted differently to the "forever" mechanism that Z2M previously employed, which was a workaround, since the value itself always had the hard limit of [0-255], and could result in seemingly inexplicable failures.).

[RobLeighton67]

One more question on detail, will 'permit join timeout' kill an existing interview/configure if in progress ?

[Nerivec]

Interview and configure are extra steps specific to Z2M that are not part of the ZigBee join, hence not affected by the permit join status.

[Sineos]

I've been enjoying Zigbee2MQTT for years now. Many thanks for maintaining such a great product so diligently.
By the same token, many thanks for the clean-up and refactoring initiatives over the last few months (mainly @Koenkk / @Nerivec). Surely one of the most tedious and least rewarding tasks ever, but essential for healthy development and maintenance.

Hats off from me.

[Zolli]

I look at the proposed changelist and everything is really promising @Koenkk, but one thing i want to put up for discussion, and that is the removal of legacy_triggers, its make a lot of sense for clean Home Assistant use cases, where HA ise used as an automation platform as well as the integration.

I myself, and assume a lot of people use other automation platform, for example Node-RED. The HA-NR integration is greate, device triggers also added so basically this change should not be a prolem, right?

Sadly, it is, generally as device triggers work, you cannot target multiple device with one selector, in the past I wrote a lot of common subflow/function that use the generic state triggers and filters entities by regex, because I use consistent naming for my entities, not really need to deploy 100s of device triggers just use, for example this regex: sensor.cover_switch_.*_action|sensor.wall_switch_.*_action extend the conencted function node a bit and everything works.

Can we at least shift the deprecation of this action/click entities because i assume that i not the only one that has such a use-case.

[Koenkk]

@mundschenk-at let's try to do at least 1.X release with the event support before releasing 2.X

[Zolli]

It would be really nice, thank you. 🤲

[chris-1243]

@mundschenk-at let's try to do at least 1.X release with the event support before releasing 2.X

Yes, this would be a great idea. This would allow time to the community to see if there is any problem and anticipate the change👍

For sure, I would love to able to test already as some of my HA blueprints are a nightmare to change with a MQTT trigger without errors...🥴

[mundschenk-at]

@chris-1243 We are still hashing out some details, but I've opened a documentation PR where you can see how it will work.

[chris-1243]

@mundschenk-at

I had a look. Thanks for your effort.

I hope you may provide a dev branch or a way to try this feature when possible.

I will need to crash some automations to understand more precisely (some blueprint I made...🤫)

[deviantintegral]

I may have missed this elsewhere, but I think it's worth coming up with a plan for the latest docker tag. I'm imagining a new user who has used zigbee2mqtt for less than a month, following the docker instructions. They may not realize that they're going to get bumped major versions and be confused if things break.

What if:

• When 2.0.0 is tagged, a new 1.x release is created that shows a note in the UI and in the startup logs.
• For 3 (just suggesting) months latest sticks with the 1.x branch, and a latest-v2 is pushed for 2.0.0+.
• After that, latest is updated to 2.x.

[Drafteed]

This will also be a bad experience. First you will need to switch to latest-v2, then back to latest. These are unnecessary movements that will still lead to the fact that the latest tag will still be updated to 2.x in a few months.

Probably the best solution would be to display warnings (in logs, in release notes, etc.) about upcoming breaking changes when using legacy functionality, such as legacy sensors. Like in Home Assistant, which displays deprecated messages in logs for several months and generates a system issue.

[gr6803]

I have been using Zigbee2mqtt for years. Many thanks and my respect for your great work.
Because of communication limitations I use three different appliances (raspberry-based) within my building. Now I want to change my visualisation from Gira Homeserver to Home Assistant.
I'm just looking for a way to integrate my 3 Z2M systems into Home Assistant. It seems, HA will accept only one. Is there a way or could you provide one with the upcoming release?

[mundschenk-at]

You can already do that, just set a different base_topic for each installation.