feat: implement session metrics - #2533
Merged
Merged
Conversation
|
Contributor
There was a problem hiding this comment.
cubic analysis
All reported issues were addressed across 8 files
Linked issue analysis
Linked issue: STG-2670: figure out what we are doing with .metrics & token tracking
| Status | Acceptance criteria | Notes |
|---|---|---|
| ✅ | Record each returned operation's metadata.usage exactly once for act, observe, and extract | Controllers now capture the result and call runtime.metrics.record(...) for act/observe/extract rather than returning early; tests validate deltas against per-operation usage. |
| ✅ | Implement stagehand.metrics() JSON-RPC to return a detached, read-only snapshot (does not reset metrics) | metrics() now returns runtime.metrics.snapshot(); snapshot returns a shallow clone so callers cannot mutate internal state; RPC tests expect and verify an empty metrics response. |
| ✅ | Preserve V3-style per-method and overall totals (prompt, completion, reasoning, cached-input, inference-time) in the metrics shape | Accumulator maintains per-method buckets and total fields with the same names and semantics; docs updated to describe the shape/semantics. |
| ✅ | Do not increment counters for operations without usage (deterministic actions, cache hits, calls that throw before returning) | Accumulator's record method ignores undefined usage; tests verify ignoring undefined usage and that deterministic actions with no metadata.usage don't change totals. |
| ✅ | Avoid double-counting: each operation's usage is included exactly once in session totals | Controller records usage once after obtaining the result; smoke integration test verifies metrics deltas match the three operations' usage and compares emitted RPC snapshot wire form to the snapshot value. |
Architecture diagram
sequenceDiagram
participant Client as Client SDK
participant RPC as Transport/RPC
participant Ctrl as Stagehand Controller
participant Svc as Service (act/observe/extract)
participant RT as StagehandRuntime
participant Metrics as StagehandMetricsAccumulator
Note over Client,Metrics: NEW: Sessions now accumulate usage via metadata.usage
Client->>RPC: JSON-RPC request (e.g., act, observe, extract)
RPC->>Ctrl: route to controller method
Ctrl->>Svc: call service (act/observe/extract)
Svc->>Svc: perform LLM inference
Svc-->>Ctrl: result with metadata.usage
alt usage is defined and non-zero
Ctrl->>RT: runtime.metrics.record(method, usage)
RT->>Metrics: record(method, usage)
Metrics->>Metrics: add to per-method and total counters
else usage is undefined (deterministic action, cache hit, etc.)
Ctrl->>Ctrl: skip recording (counters unchanged)
end
Ctrl-->>RPC: return result
RPC-->>Client: JSON-RPC response
Note over Client,Metrics: NEW: metrics() returns detached snapshot
Client->>RPC: JSON-RPC request stagehand.metrics()
RPC->>Ctrl: metrics()
Ctrl->>RT: runtime.metrics.snapshot()
RT->>Metrics: snapshot()
Metrics-->>RT: copy of current counters
RT-->>Ctrl: detached copy
Ctrl-->>RPC: metrics object
RPC-->>Client: JSON-RPC response with metrics
Client->>Client: use results without mutating internal state
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
shrey150
force-pushed
the
shrey/stg-2670-stagehand-metrics
branch
from
July 31, 2026 20:03
f0d5324 to
5d27ff3
Compare
shrey150
force-pushed
the
shrey/stg-2670-stagehand-metrics
branch
from
July 31, 2026 20:18
5d27ff3 to
2837932
Compare
monadoid
approved these changes
Aug 3, 2026
# Conflicts: # packages/sdk-go/internal/extensionassets/stagehand-extension.zip
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
act,observe, andextractmetadata.usageexactly oncestagehand.metrics()JSON-RPC method as a detached session snapshotSemantics
Each Stagehand instance starts with zeroed metrics. Every successful operation returns a complete
metadata.usageobject. Deterministic actions and cache hits return zero-valued usage, so recording them leaves the counters unchanged. Calls that throw before returning a result are not recorded. Readingstagehand.metrics()does not mutate or reset the accumulated values.This PR is stacked on #2530, which introduces and populates the per-operation
metadata.usagevalues consumed here.E2E Test Matrix
corepack pnpm exec turbo run build --forcepnpm exec vitest run packages/sdk-ts/tests/browser-runtime/stagehand-launch-connect-smoke.test.tspnpm exec vitest run packages/server/tests/metrics.test.ts packages/server/tests/rpc-router.test.tspnpm --filter ./packages/server testpnpm checkgo -C packages/sdk-go run ./internal/extensionpack --checkScope
This preserves the existing public
StagehandMetricsresult shape and SDK methods; it only implements the previously stubbed server behavior.