Skip to content

fix(mqtt/greengrass): release pooled subscriptions via linger eviction - #132

Merged
SBJakobsen merged 1 commit into
masterfrom
fix/greengrass-subscription-eviction
Jun 19, 2026
Merged

SBJakobsen merged 1 commit into
masterfrom
fix/greengrass-subscription-eviction

Conversation

@MathiasKoch

Copy link
Copy Markdown
Member

Summary

The Greengrass IPC pool never terminated subscription streams (src/mqtt/greengrass.rs), so every distinct topic ever subscribed stayed subscribed at the cloud broker for the life of the process. Each one is a cloud-side MQTT subscription counting toward AWS IoT's per-connection limit. Past 50 the nucleus opens a second connection under client ID <thing>#2, which a thing-policy-variable IoT policy rejects (NOT_AUTHORIZED), leaving the device online-but-unmanageable. This is the rustot-side hygiene work called out in factbird-edge-applications#191, section C.

This makes unsubscribe/Drop actually release subscriptions — bounded, and without reintroducing the resubscribe race the pool was originally built to avoid.

Lifecycle: linger eviction

Each pooled slot now has a lifecycle instead of being immortal:

  • When the last logical subscriber unsubscribes or is dropped, the slot goes idle and arms a linger timer instead of terminating immediately.
  • A generation counter (bumped on every claim, under the pool write lock) makes a resubscribe-within-linger a pure reuse: the pending timer sees the new generation and no-ops, so the live stream is reused with no cloud churn — the terminate is never sent, so the race can't occur for the common request/response cycle.
  • Only a slot idle for the full linger is evicted: removed from the pool and its forwarder aborted, which drops the StreamOperation and sends TERMINATE_STREAM, releasing the cloud subscription.

Settle guard

Because TERMINATE_STREAM is un-acked, a fresh subscribe to a just-terminated topic could overlap the in-flight teardown. The next subscribe to a topic terminated less than settle ago waits out the remainder, so the stale terminate is processed by the core first.

Knobs & observability

  • linger default 5s, settle default 500ms — overridable via with_linger/with_settle.
  • pooled_subscription_count() / pooled_topics() for monitoring the live subscription footprint.

Correctness notes

  • The pool→slot→pool reference cycle is broken with a Weak handle in the slot, so slots don't leak.
  • Eviction and claim both serialize on the pool write lock; eviction is Arc::ptr_eq-guarded so it can't remove a replacement slot.
  • subscribe_to_iot_core awaits the SubscribeToTopicResponse before returning, so a fresh resubscribe is live before any publish — no solicited response can be missed.

Scope

This is section C only — it bounds and reclaims subscriptions to buy headroom. A busy device can still legitimately exceed 50, so it does not replace the IoT policy fix (options A/B in the issue). The multi-shadow manager still subscribes per-shadow; moving those request/response paths onto the existing wildcard mechanism (making the count constant rather than merely bounded) is a possible follow-up.

One item to verify against a real core before fleet rollout: whether the GG core refcounts cloud subscriptions per-topic across streams. The settle guard keeps us safe under the pessimistic reading either way.

Pooled IPC subscription streams were never terminated, so every distinct
topic ever subscribed stayed subscribed at the cloud broker for the life of
the process. Each is a cloud-side MQTT subscription counting toward AWS IoT's
per-connection limit; past 50 the nucleus opens a `<thing>#2` connection that
a thing-policy-variable IoT policy rejects, leaving the device
online-but-unmanageable (factbird-edge-applications#191, section C).

Give each pooled slot a lifecycle instead of immortality:

- On the last subscriber's unsubscribe/drop, the slot goes idle and arms a
  `linger` timer rather than terminating immediately. Reuse within the window
  bumps a generation counter so the pending timer no-ops and the live stream
  is reused — no cloud churn, so the resubscribe race never arises for the
  common request/response pattern.
- A slot idle for the full `linger` is evicted: removed from the pool and its
  forwarder aborted, dropping the StreamOperation which sends TERMINATE_STREAM.
- Because terminate is un-acked, a `settle` guard delays the next subscribe to
  a just-terminated topic so the stale teardown is processed by the core first.

Defaults: linger 5s, settle 500ms, both overridable via with_linger/with_settle.
Adds pooled_subscription_count()/pooled_topics() for observability. The
pool->slot cycle is broken with a Weak handle; eviction and claim serialize on
the pool write lock and eviction is ptr_eq-guarded against replacement slots.
@SBJakobsen
SBJakobsen merged commit 5ac6240 into master Jun 19, 2026
5 checks passed
@SBJakobsen
SBJakobsen deleted the fix/greengrass-subscription-eviction branch June 19, 2026 08:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants