Skip to content

feat: implement session metrics - #2533

Merged
shrey150 merged 14 commits into
v4-spikefrom
shrey/stg-2670-stagehand-metrics
Aug 3, 2026
Merged

feat: implement session metrics#2533
shrey150 merged 14 commits into
v4-spikefrom
shrey/stg-2670-stagehand-metrics

Conversation

@shrey150

@shrey150 shrey150 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add a runtime-scoped metrics accumulator for act, observe, and extract
  • record each returned operation's already-aggregated metadata.usage exactly once
  • implement the existing stagehand.metrics() JSON-RPC method as a detached session snapshot
  • preserve V3-style per-method and overall prompt, completion, reasoning, cached-input, and inference-time totals

Semantics

Each Stagehand instance starts with zeroed metrics. Every successful operation returns a complete metadata.usage object. Deterministic actions and cache hits return zero-valued usage, so recording them leaves the counters unchanged. Calls that throw before returning a result are not recorded. Reading stagehand.metrics() does not mutate or reset the accumulated values.

This PR is stacked on #2530, which introduces and populates the per-operation metadata.usage values consumed here.

E2E Test Matrix

Command / flow Observed output Confidence / sufficiency
corepack pnpm exec turbo run build --force 4 build tasks completed successfully with cache bypassed; the protocol, server extension, TypeScript SDK, and eval package outputs were rebuilt Proves the exact combined source builds without relying on a stale cached extension artifact.
pnpm exec vitest run packages/sdk-ts/tests/browser-runtime/stagehand-launch-connect-smoke.test.ts 1 browser-runtime file and 14 tests passed against headless Chrome Proves the public SDK reaches the merged server through the real extension/service-worker transport, deterministic actions return complete zero usage, inferred operations update session metrics exactly once, and repeated snapshots are read-only. The model adapter is deterministic rather than a live provider.
pnpm exec vitest run packages/server/tests/metrics.test.ts packages/server/tests/rpc-router.test.ts 2 files and 8 tests passed Proves zero initialization, per-method and total arithmetic, zero-valued usage leaves counters unchanged, snapshots are detached, and the metrics RPC routes successfully.
pnpm --filter ./packages/server test Server build passed; 24 files passed with 183 tests passing and 11 existing TODOs Covers the full server unit suite around the changed controller, runtime, and accumulator behavior.
pnpm check 8 tasks completed successfully Covers formatting, lint, type checking, package builds, and documentation validation.
go -C packages/sdk-go run ./internal/extensionpack --check Exited successfully with no output Proves the committed Go extension archive exactly matches the rebuilt server extension.

Scope

This preserves the existing public StagehandMetrics result shape and SDK methods; it only implements the previously stubbed server behavior.

@shrey150
shrey150 requested a review from a team as a code owner July 30, 2026 23:17
@changeset-bot

changeset-bot Bot commented Jul 30, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: c7207ac

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cubic analysis

All reported issues were addressed across 8 files

Linked issue analysis

Linked issue: STG-2670: figure out what we are doing with .metrics & token tracking

Status Acceptance criteria Notes
Record each returned operation's metadata.usage exactly once for act, observe, and extract Controllers now capture the result and call runtime.metrics.record(...) for act/observe/extract rather than returning early; tests validate deltas against per-operation usage.
Implement stagehand.metrics() JSON-RPC to return a detached, read-only snapshot (does not reset metrics) metrics() now returns runtime.metrics.snapshot(); snapshot returns a shallow clone so callers cannot mutate internal state; RPC tests expect and verify an empty metrics response.
Preserve V3-style per-method and overall totals (prompt, completion, reasoning, cached-input, inference-time) in the metrics shape Accumulator maintains per-method buckets and total fields with the same names and semantics; docs updated to describe the shape/semantics.
Do not increment counters for operations without usage (deterministic actions, cache hits, calls that throw before returning) Accumulator's record method ignores undefined usage; tests verify ignoring undefined usage and that deterministic actions with no metadata.usage don't change totals.
Avoid double-counting: each operation's usage is included exactly once in session totals Controller records usage once after obtaining the result; smoke integration test verifies metrics deltas match the three operations' usage and compares emitted RPC snapshot wire form to the snapshot value.
Architecture diagram
sequenceDiagram
    participant Client as Client SDK
    participant RPC as Transport/RPC
    participant Ctrl as Stagehand Controller
    participant Svc as Service (act/observe/extract)
    participant RT as StagehandRuntime
    participant Metrics as StagehandMetricsAccumulator

    Note over Client,Metrics: NEW: Sessions now accumulate usage via metadata.usage
    
    Client->>RPC: JSON-RPC request (e.g., act, observe, extract)
    RPC->>Ctrl: route to controller method
    Ctrl->>Svc: call service (act/observe/extract)
    Svc->>Svc: perform LLM inference
    Svc-->>Ctrl: result with metadata.usage
    
    alt usage is defined and non-zero
        Ctrl->>RT: runtime.metrics.record(method, usage)
        RT->>Metrics: record(method, usage)
        Metrics->>Metrics: add to per-method and total counters
    else usage is undefined (deterministic action, cache hit, etc.)
        Ctrl->>Ctrl: skip recording (counters unchanged)
    end
    
    Ctrl-->>RPC: return result
    RPC-->>Client: JSON-RPC response

    Note over Client,Metrics: NEW: metrics() returns detached snapshot

    Client->>RPC: JSON-RPC request stagehand.metrics()
    RPC->>Ctrl: metrics()
    Ctrl->>RT: runtime.metrics.snapshot()
    RT->>Metrics: snapshot()
    Metrics-->>RT: copy of current counters
    RT-->>Ctrl: detached copy
    Ctrl-->>RPC: metrics object
    RPC-->>Client: JSON-RPC response with metrics
    Client->>Client: use results without mutating internal state
Loading

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread packages/docs/v4/reference/stagehand.mdx Outdated
Comment thread packages/server/tests/rpc-router.test.ts Outdated
Comment thread packages/server/metrics.ts Outdated
@shrey150
shrey150 force-pushed the shrey/stg-2670-stagehand-metrics branch from f0d5324 to 5d27ff3 Compare July 31, 2026 20:03
@shrey150
shrey150 force-pushed the shrey/stg-2670-stagehand-metrics branch from 5d27ff3 to 2837932 Compare July 31, 2026 20:18
Base automatically changed from shrey/stg-2670-json-rpc-usage to v4-spike August 3, 2026 16:40
# Conflicts:
#	packages/sdk-go/internal/extensionassets/stagehand-extension.zip
@shrey150
shrey150 merged commit aceadd4 into v4-spike Aug 3, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants