Skip to content

EZSP v14+: getExtendedTimeout / lookupNodeIdByEui64 responses decoded with pre-v14 layouts, so set_extended_timeout() never short-circuits #747

Closed

Description

@SimonRubens

On this coordinator — EmberZNet 8.0.2.0, where bellows is using its EZSP v14-or-later command table — getExtendedTimeout answers with an sl_status_t, and lookupNodeIdByEui64 answers with an sl_status_t followed by the node id, exactly as Silicon Labs' own host code for EmberZNet 8.0 does. bellows decodes both with the EZSP v4 response layouts, which are inherited unchanged into the v14 table and from there into v16-v19. set_extended_timeout() therefore misreads the current state, never takes its early return, and runs lookupNodeIdByEui64 + setExtendedTimeout before every unicast to a known device.

I have not established in which EZSP version each of the two responses changed; what follows is what this adapter sends on v14.

Where

The v4 layouts:

"getExtendedTimeout": (
0x7F,
{
"remoteEui64": t.EUI64,
},
{
"extendedTimeout": t.Bool,
},
),
"replaceAddressTableEntry": (
0x82,
{
"addressTableIndex": t.uint8_t,
"newEui64": t.EUI64,
"newId": t.EmberNodeId,
"newExtendedTimeout": t.Bool,
},
{
"status": t.EmberStatus,
"oldEui64": t.EUI64,
"oldId": t.EmberNodeId,
"oldExtendedTimeout": t.Bool,
},
),
"lookupNodeIdByEui64": (
0x60,
{
"eui64": t.EUI64,
},
{
"nodeId": t.EmberNodeId,
},
),

"getExtendedTimeout": (0x7F, {"remoteEui64": t.EUI64}, {"extendedTimeout": t.Bool}),
...
"lookupNodeIdByEui64": (0x60, {"eui64": t.EUI64}, {"nodeId": t.EmberNodeId}),

v14 copies every v13 command that it does not override and only swaps EmberStatus/EzspStatus for sl_Status. Neither response contains those types, so both layouts pass through unchanged. setExtendedTimeout is overridden; these two are not:

for name, (command_id, tx_schema, rx_schema) in COMMANDS_v13.items():
if name in COMMANDS:
continue
if isinstance(tx_schema, dict):
tx_schema = {k: _REPLACEMENTS.get(v, v) for k, v in tx_schema.items()}
if isinstance(rx_schema, dict):
rx_schema = {k: _REPLACEMENTS.get(v, v) for k, v in rx_schema.items()}
COMMANDS[name] = (command_id, tx_schema, rx_schema)

v16, v17, v18 and v19 chain their tables from v14 and override neither command.

The consumer, called for every NWK unicast when the firmware lacks COMBINED_SEND:

async def set_extended_timeout(
self, nwk: t.NWK, ieee: t.EUI64, extended_timeout: bool = True
) -> None:
(curr_extended_timeout,) = await self.getExtendedTimeout(remoteEui64=ieee)
if curr_extended_timeout == extended_timeout:
return
(node_id,) = await self.lookupNodeIdByEui64(eui64=ieee)
# Check to see if we have an address table entry
if node_id != 0xFFFF:
await self.setExtendedTimeout(
remoteEui64=ieee, extendedTimeout=extended_timeout
)
return

if xncp_unicast is not None:
status, _ = await self._ezsp.xncp_send_unicast(xncp_unicast)
else:
if device is not None:
await self._ezsp.set_extended_timeout(
nwk=device.nwk,
ieee=device.ieee,
extended_timeout=extended_timeout,
)

What the NCP actually sends

Silicon Labs' EZSP host implementation for EmberZNet 8.0 (Simplicity SDK 2024.6):

https://github.com/SiliconLabs/simplicity_sdk/blob/82f49bac8d5573c52b1be9892ddd1c9e8ad7886a/protocol/zigbee/app/util/ezsp/command-functions.h#L2507-L2520
https://github.com/SiliconLabs/simplicity_sdk/blob/82f49bac8d5573c52b1be9892ddd1c9e8ad7886a/protocol/zigbee/app/util/ezsp/command-functions.h#L2549-L2564

sl_status_t sl_zigbee_ezsp_get_extended_timeout(sl_802154_long_addr_t remoteEui64)
  ...
    status = fetchInt32u();
    return status;

sl_status_t sl_zigbee_ezsp_lookup_node_id_by_eui64(sl_802154_long_addr_t eui64, sl_802154_short_addr_t *nodeId)
  ...
    status = fetchInt32u();
    *nodeId = fetchInt16u();

What the status means, per the EmberZNet 8.0 API reference (https://docs.silabs.com/d/zigbee-stack-api/8.0.0/message): sl_zigbee_get_extended_timeout returns SL_STATUS_OK if the retry interval will be extended, and SL_STATUS_FAIL if the normal interval is used.

zigbee-herdsman's ember driver decodes it the same way: a uint8 below v14, and a uint32 status from v14:
https://github.com/Koenkk/zigbee-herdsman/blob/8f461929aeca7fdcfdbf530f1922c5da0ee341d4/src/adapter/ember/ezsp/ezsp.ts#L5669-L5688

Observed

The adapter negotiated EZSP v14 or later: setExtendedTimeout replies with a status field, which exists only in the v14+ schema, and sendUnicast and messageSentHandler carry sl_Status values. The coordinator firmware is EmberZNet 8.0.2.0.

The unparsed remainder that bellows logs at DEBUG shows the misalignment byte for byte. Every reply below was followed by the Frame contains trailing data line shown:

Received command getExtendedTimeout: {'extendedTimeout': <Bool.true: 1>}   | Frame contains trailing data: b'\x00\x00\x00'
Received command lookupNodeIdByEui64: {'nodeId': 0x0000}                   | Frame contains trailing data: b'\x00\x00\xf5\xac'
Received command lookupNodeIdByEui64: {'nodeId': 0x0000}                   | Frame contains trailing data: b'\x00\x00K\x0b'
Received command lookupNodeIdByEui64: {'nodeId': 0x002D}                   | Frame contains trailing data: b'\x00\x00\xff\xff'
  • getExtendedTimeout: 01 00 00 00 is SL_STATUS_FAIL, meaning the normal interval is in use. bellows reads the first byte as True, which is the opposite.
  • lookupNodeIdByEui64 (OK): 00 00 00 00 | f5 ac is SL_STATUS_OK followed by the real node id 0xACF5. bellows reads 0x0000.
  • lookupNodeIdByEui64 (not found): 2d 00 00 00 | ff ff is SL_STATUS_NOT_FOUND followed by 0xFFFF. bellows reads 0x002D. All 109 0x002D replies in the window carried exactly this remainder.

Over a two-minute window of scene activations there were 256 unicasts. Each one produced exactly:

Sending command  getExtendedTimeout: () {'remoteEui64': <eui64>}
Received command getExtendedTimeout: {'extendedTimeout': <Bool.true: 1>}
Sending command  lookupNodeIdByEui64: () {'eui64': <eui64>}
Received command lookupNodeIdByEui64: {'nodeId': 0x0000}
Sending command  setExtendedTimeout: () {'remoteEui64': <eui64>, 'extendedTimeout': False}
Received command setExtendedTimeout: {'status': <sl_Status.OK: 0>}
Sending command  sendUnicast: ...

The next unicast to the same device repeats the whole sequence. Totals over the window:

  • 256 × getExtendedTimeout, all Bool.true
  • 256 × lookupNodeIdByEui64: 147 × 0x0000, 109 × 0x002D, never a real node id and never 0xFFFF
  • 256 × setExtendedTimeout, all OK
  • 256 × sendUnicast

Impact

  1. Two extra EZSP round trips for every unicast to a mains-powered device. On a networked coordinator, each EZSP round trip took a median 11–13 ms. The full per-unicast chain was a median 55 ms (p90 113 ms), against about 25 ms if the early return had worked. This is visible whenever HA sends a burst. A scene on 17 bulbs (34 unicasts: level, then colour) takes 2.5–2.9 s from the first lamp to the last, and the log shows the queue waiting on this serial path, not on the radio (the APS ack arrived a median 26 ms after sendUnicast).
  2. I can show the cost, not a functional consequence. In this network the wanted value is always False, because bellows clears extended_timeout whenever APS ACKs are requested (
    zigpy.types.TransmitOptions.ACK in packet.tx_options
    and packet.dst.addr_mode == zigpy.types.AddrMode.NWK
    ):
    aps_frame.options |= t.EmberApsOption.APS_OPTION_RETRY
    # We disable extended timeout if we enable ACKs
    extended_timeout = False
    ). setExtendedTimeout(..., True) does not appear once in seven days of logs (2026-09-06 to 2026-09-13), and the captured window contains no unicast to any of the four sleepy end devices. What the inverted reading does where True is wanted, I have not tested.
  3. The address-table branch in set_extended_timeout() is unreachable, because lookupNodeIdByEui64 can never return 0xFFFF.

Environment

  • Home Assistant OS 18.2, Home Assistant Core 2026.9.2. The library versions are the ones this HA release pins: the ZHA manifest requires zha==2.2.2, which pins bellows==1.0.1 and zigpy==2.2.0. I read the pins, not the installed dist-info.
  • Coordinator: SMLIGHT SLZB-06M (EFR32MG21), Ethernet, socket://…:6638. SMLIGHT release Zigbee firmware revision 20250220, published in their OTA list as slzb06m_zigbee_ncp_8.0.2.0_sw_flow_115200.gbl (EmberZNet 8.0.2.0). The firmware does not offer the XNCP combined send: bellows took the four-command path for every unicast.
  • Network: 40 devices, 35 of them routers.

Related

Suggested fix (untested sketch)

Override both responses for v14:

# bellows/ezsp/v14/commands.py
"getExtendedTimeout": (
    0x007F,
    {"remoteEui64": t.EUI64},
    {"status": t.sl_Status},
),
"lookupNodeIdByEui64": (
    0x0060,
    {"eui64": t.EUI64},
    {"status": t.sl_Status, "nodeId": t.EmberNodeId},
),

Then give EZSPv14 its own set_extended_timeout() that treats status == sl_Status.OK as "extended timeout is set" and takes the node id from the second field, with a non-OK status or 0xFFFF meaning "no entry".

Since v16-v19 chain from the v14 table, overriding it there covers them as well.

The same window also holds 48 Frame contains trailing data lines belonging to other commands (560 in total, 512 of them from these two), so other schemas may be out of date as well.

Other v4-era commands whose v14 response gained a leading sl_status_t without containing an EmberStatus field would be affected the same way; I have not audited them.

I can test a branch against this coordinator and provide full debug logs.

Written with the help of an AI assistant (Claude). Every number above comes from my own coordinator's journal and the linked sources, and I can re-run any of it on request.

Activity

  1. TheJulianJES commented on Sep 13, 2026

    @TheJulianJES
    Contributor

    @zigpy-review-bot Investigate, possibly also with second opinions from Copilot: GPT 6 and Fable 5.1 agent. Also consider firmware builder repo and memories there.

  2. zigpy-review-bot commented on Sep 13, 2026

    @zigpy-review-bot
    Collaborator

    I checked the report against bellows dev at 8f9d87b, which matches 1.0.1 for every file involved. I compared it with the Silicon Labs EZSP host and NCP sources (GSDK 4.4.6 = v13, SiSDK 2024.6.0 = v14, 2025.12.3 = v18, 2026.6.1 = v19) and read the closed stack functions from their LTO dumps. Copilot (GPT-6 Astra) and a Fable 5.1 agent gave second opinions, and all three agree on the core finding.

    The report is correct. Both responses changed at exactly EZSP v14, and bellows never overrode them:

    • getExtendedTimeout (0x7F): v13 reads one bool byte (fetchInt8u()). v14+ reads a 4-byte sl_status_t (fetchInt32u()), and the NCP sends it with appendInt32u(status).
    • lookupNodeIdByEui64 (0x60): v13 reads nodeId (fetchInt16u()). v14+ reads a 4-byte status followed by nodeId.
    • bellows only defines both in bellows/ezsp/v4/commands.py. The _REPLACEMENTS loop in v14/commands.py only swaps EmberStatus/EzspStatus, so both schemas pass through unchanged into v14, and through **COMMANDS_v14 into v16–v19. Decoding the reported bytes with bellows' own types gives exactly the logged values and trailing data. The proposed layouts decode the same bytes with nothing left over: FAIL; OK, 0xACF5; NOT_FOUND, 0xFFFF. bellows logs the leftover bytes only at DEBUG (protocol.py:180), so the misparse goes unnoticed.

    One correction to the impact section. SL_STATUS_OK (0) means "extended" and SL_STATUS_FAIL (1) means "normal", so the misread value is always the opposite of the real state. As a result, the early return in set_extended_timeout() fires exactly when the setting is wrong:

    • If bellows wants False and the setting is normal, it doesn't return early. It runs lookup plus setExtendedTimeout(False) on every unicast. This is the measured cost; the end state is correct.
    • If bellows wants True and the setting is normal, it returns early, so it never turns the extended timeout on.

    True is often wanted. zigpy sets extended_timeout=True for end devices and for devices with no node descriptor yet (zigpy/device.py:632-637). bellows only clears it when an APS ACK is requested (application.py:1010-1017), and zigpy requests one only for ask_for_ack=True or expect_reply=False. So the missing setExtendedTimeout(..., True) in the reporter's logs is caused by this bug, not evidence that True is never wanted.

    The practical effect is small, though. The flag only lengthens the gap between APS retries. bellows sets APS_OPTION_RETRY in two cases: with an ACK (where it forces the flag to False), and with FORCE_ROUTE_DISCOVERY, which zigpy uses for its own retry attempts. So the actual loss is that zigpy's retries of reply-expecting requests to sleepy end devices use normal retry spacing, unless the stack already extended the timeout itself (for direct children, or after an end-device announce from a sleepy node).

    Which firmware is affected

    • Third-party firmware on EmberZNet 8.0 or later without XNCP (like the SMLIGHT firmware in the report) is affected.
    • The current stable NabuCasa release, v2026.02.23, is built on GSDK 4.5.0, which is EZSP v13, so it decodes correctly.
    • The SiSDK-based v2026.08.18-beta1 builds have XNCP COMBINED_SEND (Consolidated unicast sending API NabuCasa/silabs-firmware-builder#209), so they skip this path entirely.

    Fix. The suggested fix is right, and both parts are needed:

    • Override both response schemas in bellows/ezsp/v14/commands.py. This covers v16–v19 through inheritance.
    • Override set_extended_timeout() on EZSPv14, which v16–v19 inherit. Extended means status == sl_Status.OK. An entry exists when status == OK and nodeId != 0xFFFF.
    • Add v14 copies of the set_extended_timeout tests in tests/test_ezsp_v4.py, which currently mock a t.Bool response.

    Two NCP details from the LTO dumps:

    • lookupNodeIdByEui64 searches the child, neighbor, and address tables, and returns NOT_FOUND with 0xFFFF only when all three miss. The "address table entry" check in set_extended_timeout() has therefore always meant "known to any table".
    • set_extended_timeout on the NCP always returns OK, even when it stored nothing.

    After the fix, unicasts to direct sleepy children that want False will still take the 3-command path every time. The NCP derives their extended state from the child table, and setExtendedTimeout(False) can't change that. A cache or a comment may be worth adding.

    Other outdated v14+ schemas. Fable ran a scripted comparison of every bellows v14–v19 command and callback against the SDK host decoders. Of the commands bellows actually calls, one more mismatch affects behavior:

    • leaveNetwork (0x20): since v14, the command takes a uint8 options argument, and the NCP reads it with options = fetchInt8u();. bellows sends no argument, so the NCP reads one byte past the end of the frame. The stack only acts on bit 0x20 (leave with rejoin). I haven't verified the real-world impact, but bellows should send the byte; zigbee-herdsman does for v14+.
    • setManufacturerCode gained an sl_status_t response. The 4 leftover bytes are harmless and likely explain some of the report's 48 other trailing-data lines.
    Outdated definitions bellows never calls (they would misparse if used)
    • Field order changed at v14 (status moved first, or a context field was added): exportTransientKeyByIndex/ByEui, exportLinkKeyByEui, getApsKeyInfo, importKey, checkKeyContext, findAndRejoinNetwork, setAddressTableRemoteEui64 (now setAddressTableInfo).
    • Widened by mistake by _REPLACEMENTS: the SDK keeps a 1-byte status, but bellows reads a 4-byte sl_Status. Affects sendPanIdUpdate, readAttribute, writeAttribute.
    • setPowerDescriptor and clearStoredBeacons gained an sl_status_t response.
    • Frame IDs removed or reused in v14 but still defined: proxyBroadcast 0x37, sendMulticastWithAlias 0x3A, getFirstBeacon/getNextBeacon, sendRawMessage 0x96, incomingSenderEui64Handler 0x62, 0xE3–0xE7/0xF0.
    • Callbacks whose v14 payload replaced lastHopLqi/lastHopRssi with sl_zigbee_rx_packet_info_t: macFilterMatchMessageHandler, incomingBootloadMessageHandler, macPassthroughMessageHandler, the ZLL handlers; also rawTransmitCompleteHandler.
    • getTokenCount/getTokenInfo widened from uint8 to uint32 at v18.
    • Green Power: on the wire, the GP address is 10 bytes (applicationId, an 8-byte union, endpoint), but bellows' EmberGpAddress is 14 bytes. bellows' own v17 gpepIncomingMessageHandler override uses that struct, so Fix gpepIncomingMessageHandler schema for EZSP v13+ #746, which fixes the v13 schema, doesn't cover v17 and later. Fix and forward gpepIncomingMessageHandler for Green Power support #713 changes the struct itself.

    Overlap: no existing PR touches 0x7F or 0x60. #631, which added v14 support, overrode the setExtendedTimeout response but not these two. #646 added the early return a month later, so on v14 and later it has never worked. Also, the report says zigbee-herdsman decodes both commands the same way, but that's only true for 0x7F. It still reads 0x60 as a bare uint16 (ezsp.ts:5753), so it has the same lookup bug.

    If you want a PR with the fix (optionally including the leaveNetwork byte), mention the bot again.

  3. TheJulianJES commented on Sep 13, 2026

    @TheJulianJES
    Contributor

    @zigpy-review-bot Open a PR, do review rounds before opening.

  4. zigpy-review-bot commented on Sep 13, 2026

    @zigpy-review-bot
    Collaborator

    Opened #748. It overrides both response schemas for v14 (v16–v19 inherit them) and moves the version-specific decoding in set_extended_timeout() into two helpers that EZSPv14 overrides. It also guards the address table fallback, which is now reachable on v14+, against a table size of zero. New tests decode the bytes captured in this issue through every v14+ table. They fail on dev, and the full suite passes.

    Not included: the leaveNetwork options byte and the other drift from the audit above. Mention the bot again if you want those as a follow-up PR. The branch is zigpy-bot/v14-extended-timeout-schemas, if the reporter wants to test it against the SLZB-06M.

    Review rounds
    • Round 1 (Fable 5.1 agent + Copilot GPT-5.6 Sol, 854ba31): Fable found no material issues and flagged two missing test cases. Copilot found that with CONFIG_ADDRESS_TABLE_SIZE: 0 (which the config schema allows) the newly reachable fallback calls random.randint(0, -1) and aborts the unicast. I confirmed this with a test that fails without the fix, then fixed it and added all three test cases.
    • Round 2 (delta up to 8a76760): Fable reported it ready. Its two-call probes on v13 and v14 showed that the restructured fallback behaves as before for a non-OK getConfigurationValue (last-ditch, not cached, re-queried) and for a cached size. Copilot raised one test nit (assert caching across a second call), which I folded in.
    • Wire layouts were checked against the Silicon Labs host decoders for v13 (GSDK 4.4.6), v14 (SiSDK 2024.6.0) and v19 (SiSDK 2026.6.0), including the getConfigurationValue and replaceAddressTableEntry schemas the fallback now uses.
  5. TheJulianJES commented on Sep 13, 2026

    @TheJulianJES
    Contributor

    @zigpy-review-bot Open PR(s) for the other issues as well. Do review rounds again.

  6. zigpy-review-bot commented on Sep 13, 2026

    @zigpy-review-bot
    Collaborator

    Opened two more PRs for the rest of the v14+ schema drift:

    • Fix EZSP v14+ leaveNetwork and setManufacturerCode schemas #749 covers the two remaining commands bellows calls. On v14+, leaveNetwork now sends its one-byte options argument (0x00, no option), which the NCP reads whether or not it's sent. The setManufacturerCode status is now parsed instead of being left over as trailing data.
    • Fix stale EZSP v14+ command and callback schemas #750 fixes the definitions bellows keeps but never calls. Each one was checked against the SDK host code for every version from v12 to v19. It covers the key export/import field order, the new findAndRejoinNetwork arguments, setAddressTableInfo, statuses that were added or widened by mistake, callbacks that now carry packetInfo, the getTokenCount/getTokenInfo widening, and deleting commands whose frame IDs the SDK removed or reused.

    Re-deriving the audit for #750 found some things my earlier comment missed or got wrong:

    • Four stale entries share a frame ID with commands bellows added for v17 and v18 (0x5D, 0x5F, 0x62, 0x3D). For example, calling getAddressTableRemoteNodeId() on v17+ would have sent gpClearProxyTable's frame ID.
    • checkKeyContext didn't change at v14. It has been defined with setPassiveAckConfig's arguments since v12.
    • The getTokenInfo size was widened in SiSDK 2025.12 without an EZSP version bump, so v18 firmware sends either width. Fix stale EZSP v14+ command and callback schemas #750 changes it for v19 only.

    Not included: the Green Power address struct, which #746 and #713 cover, and two harmless differences in called commands (importTransientKey still sends a flags byte the NCP ignores, and launchStandaloneBootloader reads only the low byte of the 4-byte status).

    The full suite passes on both PRs, and they merge cleanly with #748 and with each other.

    Review rounds
    • Fix EZSP v14+ leaveNetwork and setManufacturerCode schemas #749 at ddc25f0: a Fable 5.1 agent reported it ready after checking the layouts against the v13, v14 and v19 host and NCP code, every caller, and that the version gate always matches the protocol table in use. Copilot (GPT-5.6 Sol) had no findings.
    • Fix stale EZSP v14+ command and callback schemas #750 at b993b2f: a fresh Fable 5.1 agent re-derived the audit independently and reported it ready. It confirmed that 183 of the 210 new test cases fail on dev, and that nothing reachable on v14+ still uses a deleted or changed definition. Copilot (GPT-5.6 Sol) had no findings. One nit was not taken: readAttribute/writeAttribute keep typing their 1-byte ZCL status as EmberStatus, as in v13.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions