DynamoDB state-layer limits: 400KB guard, CAS visibility, dedup key scoping

[pequalsnp]

Four verified defects from the edge-case hunt. Reviewed independently and returned ship-with-notes; I additionally spot-checked the dedup scoping myself — neutralising claimKey back to a bare EventID fails both new tests.

Silent freeze on an oversized item

BytesStore had no pre-flight against DynamoDB's hard 400 KB item limit. An oversized sketch failed every write non-retryably, while Get kept serving the last value that fit as Present:true. A key that had silently stopped updating read as perfectly healthy.

The realistic trigger is key length, not K: examples/recently-interacted-topk/pipeline.go lifts an unbounded raw wire field into the sketch key, so two ~200 KB entity IDs overflow the row at the shipped K=32. (For calibration, I measured the live soak's actual rows at 624 B and 382 B — 0.16% of the limit — so this is a cliff you fall off, not a slope you notice.)

Now returns a typed ErrItemTooLarge carrying the key and measured size.

CAS contention was invisible

8 retries and ~17s of backoff, hardcoded with no setter, then a dead-letter — with no metric anywhere to say the writes were racing rather than failing. Now configurable via WithCASRetries / WithCASBackoff, counted under <pipeline>:cas_conflict, and ErrMaxRetriesExceeded carries table, key and attempt count.

Worth noting this is the metric that would tell you whether a hot key is contending — the thing the soak currently cannot measure.

Two pipelines sharing a dedup table starved each other

EventIDs are only unique within a source, and the claim key was the bare EventID. Two pipelines sharing one dedup table — the layout doc/design.md §13.4 recommends — meant whichever claimed an ID first made the other skip a merge that never ran.

The namespaced key "<pipeline>#<EventID>" is the format §13.4 already described. The code just never wrote it.

A lost response dropped a first delivery

A claim DynamoDB had committed but whose response was lost to a connection reset came back as a plain ConditionalCheckFailedException on the SDK's retry — indistinguishable from a peer's claim — and dropped a first delivery. Each MarkSeen now stamps a per-call claimant token and admits it in the condition. This works because the SDK's retry middleware sits after serialization, so the replay carries the identical token.

Rate is ~1e-5 of claims. Fixed for the invariant, not the magnitude.

Breaking, and what it means for the running soak

• NewDeduper(client, table, ttl) → NewDeduper(client, table, pipeline, ttl)
• NewBytesStore takes variadic options (existing 3-arg calls compile unchanged)

The dedup key format change gives one window of re-processing as previously-claimed IDs re-claim under their namespaced keys. No schema change — same partition-key attribute, different contents. At the soak's 1 event/sec that window is a handful of records; at production rates, drain in-flight records first or let the old rows age out via TTL.

Test plan

• TestBytesStore_MergeUpdate_RejectsOversizedItemBeforeWriting
• TestBytesStore_MergeUpdate_CASConflictExhaustsInjectedRetries
• TestDeduper_PipelineScopesShareATableWithoutColliding
• TestDeduper_SeparateInstancesShareATableWithoutColliding — the collision is in the table's key space, not the Go object
• TestDeduper_LostClaimResponseIsNotADuplicate — via the existing fakeTransport, no smithy middleware needed
• All five verified to fail pre-fix, by the implementer, by an independent reviewer, and (for the dedup pair) by me
• go build, go test -race, make test-unit, golangci-lint 0 issues
• CI green

Reviewer notes carried forward, not blocking

The size guard is slightly less conservative than its comment claims (numbers counted as decimal text where DDB packs tighter), and doc/design.md §7.4 still documents the old dedup schema. Both worth a follow-up.

🤖 Generated with Claude Code

https://claude.ai/code/session_01X2YMxeLgyRc9i5XPyEV75S