[STG-2749] Surface cache metadata on act/observe/extract results - #2543
Conversation
cacheStatus alone could not explain itself: a miss never said why, a hit never said how established the entry was, and MISS was stamped equally for a cold cache, a failed cache read, and a cached value that could not be replayed. The API already returns hitCount, threshold, ageMs, and missReason on every lookup — cacheService parsed them and passed them only to logger.debug. Adds an optional cacheMetadata object to StagehandResultMetadata carrying count, threshold, ageMs, missReason, and tokensSaved, populated on both the hit and miss paths. Hit and miss metadata are built separately so a result can never report both stories at once, and the two local failure modes get their own reasons (read_failed, replay_failed) instead of masquerading as a cold cache. cacheStatus is unchanged, so this is additive for all three SDKs. tokensSaved is plumbed through but stays undefined until the API's stateless get forwards the savings it already computes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 8ebf30a The changes in this PR will be included in the next version bump. This PR includes changesets to release 5 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
cacheStatus and cacheMetadata said the same thing at two altitudes. Collapse them into a single optional `cache` object with a required `status`, so it is present exactly when a lookup ran and always explains itself. Named `cache` rather than `cacheMetadata` to avoid metadata.cacheMetadata stutter, and it mirrors the `cache` input option. Drops ageMs: the entry's age is a server-side detail, still logged but not worth a field on every result. Updates the three SDK surfaces, examples, and v4 docs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…49-cache-metadata-v4 # Conflicts: # packages/sdk-go/internal/extensionassets/stagehand-extension.zip # packages/sdk-ts/examples/caching.ts
There was a problem hiding this comment.
cubic analysis
Review completed against the latest diff
Linked issue analysis
Linked issue: STG-2749: Surface cache metadata (count, miss reason, tokens saved) on v4
| Status | Acceptance criteria | Notes |
|---|---|---|
| ✅ | Add a cacheMetadata object to StagehandResultMetadataSchema carrying count, missReason, ageMs, threshold, and tokensSaved | Protocol and type schemas were added and wire formats updated; generated SDK models and docs were also updated to include cache metadata fields. |
| ✅ | Populate cacheMetadata in cacheService.withCache on the hit path (including count, threshold, ageMs, tokensSaved when present) | withCache maps server response fields into a hit metadata object and attaches it to result.metadata.cacheMetadata when present. |
| ✅ | Populate cacheMetadata in cacheService.withCache on the miss path, including read-failure and onHit-threw fallbacks, with distinguishable miss reasons | Miss handling builds cache metadata with missReason and includes different reasons for read failures and replay failures; miss metadata is attached when outcome is a MISS. |
| Forward tokensSaved from the stateless cache GET response into CacheGetResponse so tokensSaved can be surfaced | This PR adds client-side parsing and internal mapping for tokensSaved and maps it into cacheMetadata on hits, but the companion server/core change that actually forwards tokensSaved from handleStatelessCacheGet is not present here (the client accepts tokensSaved but the core change was noted in the issue as a separate required change). | |
| ✅ | Keep cacheStatus behavior unchanged (additive change) so existing clients/tests remain valid | cacheStatus is still set to HIT/MISS as before and new cacheMetadata is additive. |
| ✅ | Add tests for hit, miss, threshold miss, and cache-failure paths | Unit tests were added that cover hits (including token savings), basic miss and persistence, threshold progress misses, read failures, and replay failures, plus a test for no lookup when caching disabled. |
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Review feedback: - CacheTokenSavings fields default to 0, matching StagehandResultUsage, so a hit always reports all three counts. - metadata.cache is no longer optional. Making it required needs a representation for "no lookup ran", so CacheStatus gains DISABLED and services seed every result with it; withCache overwrites when it runs. Callers no longer need to null-check before reading status. - act now reports its aggregate llmUsage on cache writes. It was the one primitive that sent none, so act hits could never compute tokensSaved. - Adds cache-client tests for the tokensSaved wire shape and malformed payloads; cache-service tests mock past the parser, so a renamed field would previously only have surfaced in production. Merge: keeps upstream's per-operation usage aggregate alongside cache metadata, and rebuilds the Go-embedded extension, which was stale against the merged server build. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The service tests mock the cache client and the client test only parses payloads, so nothing proved that a real API response becomes the right metadata.cache. That seam is exactly where tokensSaved was being lost. Drives the real CacheClient over real HTTP against a local server speaking the API's captured response shapes, asserting the exact metadata for each outcome: cold miss, hit with count/threshold/savings, hit without recorded usage, threshold miss with progress, API failure (read_failed), unusable cached value (replay_failed), and caching off (DISABLED). Also pins that a write carries llmUsage and that a per-request threshold reaches both get and set. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
All reported issues were addressed across 1 file (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
afterAll fired server.close() without awaiting it, so the suite could finish while the fixture was still closing and leak the handle if a request were still in flight. Drop live connections, then await the close. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
All reported issues were addressed across 1 file (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
The repo pins pnpm 11.10.0 (devEngines, and CI sets it explicitly "to protect the lockfile"), but this branch was installed with pnpm 10, which rewrote the lockfile into the older format and dropped its configDependencies/packageManagerDependencies sections — 5k lines of diff and a format CI would not have produced. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…49-cache-metadata-v4
closeAllConnections() snapshots the tracked sockets, so calling it before close() leaves a window where a connection accepted afterwards keeps close() waiting. Close first, then drop — the order the SDK's integration closeServer helper already uses. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Nesting CacheMetadata inside StagehandResultMetadata pushed the literal past the line limit; ruff format wraps it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…49-cache-metadata-v4 # Conflicts: # packages/docs/v4/reference/stagehand.mdx
Renaming the field from CacheMetadata to Cache left the struct literal padded for the old, longer key. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
why
Adds cache observability to result metadata. Previously a result only told you
HITorMISSbut never why it missed, how close a key was to its threshold, or what a hit saved. The stateless cache API already returns all of it (hitCount,threshold,missReason); we parsed those fields and passed them straight tologger.debug.what changed
Before
After
cacheis present exactly when a lookup ran, sostatusbeing required means it always explains itself:Breaking:
metadata.cacheStatus→metadata.cache.statustest plan
Summary by cubic
Adds a required
metadata.cacheobject to all v4 act/observe/extract results to explain hits, misses, and token savings; reportsDISABLEDwhen no lookup ran. Completes Linear STG-2749 (v4 port of STG-2656) to improve debugging and cost visibility.New Features
metadata.cacheis required withstatus: "HIT" | "MISS" | "DISABLED", plus optionalcount,threshold,missReason, andtokensSaved(inputTokens,outputTokens,totalTokensdefault to 0); removedageMsfrom results (still logged).read_failedandreplay_failed.actnow writes aggregatellmUsageso hits can reporttokensSaved.packages/protocol/*,packages/sdk-ts,packages/sdk-python,packages/sdk-go), v4 docs, and tests, including end‑to‑end coverage and cache‑client parsing for thetokensSavedwire shape.Migration
result.metadata.cacheStatuswithresult.metadata.cache.status(TS),cache_status→cache.status(Python), and theCacheStatuspointer with a requiredCachestruct andCache.Status(Go).result.metadata.cache.status === "DISABLED"(no null checks).result.metadata.cache.tokensSaved.{inputTokens,outputTokens,totalTokens}on hits.Written for commit 8ebf30a. Summary will update on new commits.