Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,10 @@ This changelog is automatically generated by AI when releases are published.

## [Unreleased]

### Fixed

- **AI spend includes prompt caching for Droid, Continue CLI and Copilot in VS Code.** These agents' cache reads and writes did not reach the backend, so their spend was priced as if nothing was cached. Droid keeps its token totals only in the `<session>.settings.json` file next to the transcript, and Continue CLI keeps them at the top level of the session file, outside the history autter streams; autter now sends a snapshot of each total (input, output, cache reads, cache writes and, for Continue, its own cost) with the session's events. Copilot Chat's traces database stores cache writes only as a span attribute; autter now sends them as `cache_creation_tokens` next to `cached_tokens`. The other agents already carried their cache counts in the events autter forwards. [docs/agent-usage.md](docs/agent-usage.md) lists what each agent records and the convention the snapshot uses: disjoint counts, with input excluding cache reads and writes, and running session totals that are diffed, not summed.

## [3.9.2] - 2026-11-14

### Fixed
Expand Down
76 changes: 76 additions & 0 deletions docs/agent-usage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# Agent token usage and prompt caching

autter's background service streams each agent's session events to the
backend, which turns the token usage in them into the estimated AI spend shown
on the dashboard. Every estimate prices four token classes separately: input,
output, cache reads and cache writes. This page lists what each agent records
and how autter forwards it.

## How usage travels

Most agents put usage inside their own session events, and autter forwards
those events as written (secrets redacted, numbers untouched). The backend
reads each agent's shape and normalizes it there. autter adds data only where
the agent keeps usage outside the events it streams:

| Agent | Where usage comes from | Convention in the source | What autter adds |
| --- | --- | --- | --- |
| Claude Code | `message.usage` on assistant lines | Disjoint: `input_tokens` excludes `cache_read_input_tokens` and `cache_creation_input_tokens`; `cache_creation.ephemeral_5m_input_tokens` / `ephemeral_1h_input_tokens` split writes by TTL | Nothing |
| Codex | `token_count` events, `info.total_token_usage` (running total) | OpenAI: `input_tokens` includes `cached_input_tokens` and `cache_write_input_tokens`; `output_tokens` includes `reasoning_output_tokens` | Nothing |
| GitHub Copilot (VS Code, OTel) | Copilot Chat's OTel traces database | OpenAI: `input_tokens` includes `cached_tokens` (reads) and the cache writes | `span.cache_creation_tokens`, read from the `gen_ai.usage.cache_creation.input_tokens` span attribute, which the database does not copy into a column |
| GitHub Copilot (VS Code chat sessions) | `result.usage` | `promptTokens` / `completionTokens` only, no cache counts | Nothing (cache counts come from OTel) |
| Copilot CLI | `session.shutdown` event, `data.modelMetrics.<model>.usage` (per-model running total) | `inputTokens` includes `cacheReadTokens` and `cacheWriteTokens` | Nothing. Per-request `assistant.usage` events are not written to disk by Copilot CLI |
| Gemini CLI | `tokens` on `gemini` records | Google: `input` (`promptTokenCount`) includes `cached`; `thoughts` and `tool` are counted separately; no cache writes | Nothing |
| OpenCode | `tokens` and `cost` on assistant messages | Disjoint: `input`, `cache.read`, `cache.write`; `reasoning` separate from `output`; `cost` in USD | Nothing |
| Pi | `message.usage` on assistant messages | Disjoint: `input`, `cacheRead`, `cacheWrite` (`cacheWrite1h` is a subset); `output` includes `reasoning`; `cost.total` in USD | Nothing |
| Amp | `usage` on assistant messages | Disjoint (Anthropic style, camelCase): `inputTokens`, `cacheReadInputTokens`, `cacheCreationInputTokens` | Nothing |
| Continue CLI | Newer versions: `message.usage` on assistant history items. All versions: a session-level `usage` total | Per message: `prompt_tokens` with `prompt_tokens_details`; the session total is disjoint with `totalCost` | A usage snapshot of the session-level total (below) |
| Droid | `tokenUsage` in `<session>.settings.json`, next to the transcript (running total) | Disjoint: `inputTokens`, `cacheReadTokens`, `cacheCreationTokens`; `thinkingTokens` | A usage snapshot of the settings total (below) |
| Cursor, Windsurf, Firebender, Antigravity, Kiro | Not recorded in their transcripts or hook payloads | | Nothing; their sessions show as "no usage data" rather than $0 |

## Usage snapshot events

For Droid and Continue CLI, the reader appends one synthetic event to a batch
that read new records (only to the batch that reaches the end of the file, so
one per read-through). It does not count toward the stream's watermark.

```json
{
"type": "autter_usage_snapshot",
"timestamp": "2026-01-28T16:57:30.000Z",
"autter_usage_snapshot": {
"version": 1,
"source": "droid_settings",
"scope": "session_total",
"model": "claude-sonnet-4-6",
"input_tokens": 6743,
"output_tokens": 1450,
"cache_read_tokens": 42496,
"cache_write_tokens": 0,
"reasoning_tokens": 1088,
"cost_usd": null,
"input_includes_cache": false
}
}
```

- The counts are disjoint: `input_tokens` excludes cache reads and writes, and
`input_includes_cache` is always `false`.
- `output_tokens` includes reasoning; `reasoning_tokens`, when present, is a
subset of it.
- `scope` is `session_total`: each count is the session's running total when
the snapshot was taken. Consumers diff successive snapshots of a session and
never sum them.
- `cost_usd` is the cost the agent reported itself (Continue's `totalCost`),
or `null`.
- `source` is `droid_settings` or `continue_session`. `version` changes only
for a breaking change; new fields may be added.

## Limits

- Droid writes its settings file after a turn; usage from a session's last turn
arrives only when the transcript grows again.
- Codex records cache writes without their TTL (30-minute or 12-hour), so they
are priced at the 30-minute rate.
- Copilot reports cost in AI units (`copilot_usage_nano_aiu`), not dollars; it
is forwarded but not used as a dollar cost.
3 changes: 3 additions & 0 deletions docs/agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@ Every failed step prints a `fix:` line.
| JetBrains IDEs | IDE plugin (build 252 or newer) | Edits by GitHub Copilot for JetBrains and Junie | The model and prompts; other JetBrains AI plugins | `autter doctor --agent jetbrains` |
| Visual Studio (Windows) | IDE extension (VSIX, Visual Studio 2022) | Copilot edits | The model and prompts | `autter doctor --agent visual-studio` |

Token usage, including prompt-cache reads and writes, is described per agent in
[agent-usage.md](agent-usage.md).

Shell commands an agent runs (Bash, `run_in_terminal`, ...) are captured by
comparing the files before and after the command. Gitignored files and deletions
are not attributed; see [shell-edit-capture.md](shell-edit-capture.md).
Expand Down
25 changes: 25 additions & 0 deletions src/streams/agents/amp.rs
Original file line number Diff line number Diff line change
Expand Up @@ -447,4 +447,29 @@ mod tests {
assert_eq!(content[1]["type"], "text");
assert_eq!(content[2]["type"], "tool_use");
}

/// Amp assistant messages carry Anthropic-style disjoint usage in camelCase
/// (`inputTokens` excludes `cacheCreationInputTokens` / `cacheReadInputTokens`;
/// `totalInputTokens` is their sum). Real thread fixture, forwarded verbatim.
#[test]
fn test_assistant_usage_cache_fields_are_forwarded() {
let fixture = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"))
.join("tests/fixtures/amp-threads/T-019ca1ce-3ae2-7686-a41e-ccc078837f8a.json");
let (events, _) = drain_all(&AmpAgent::new(), &fixture);
let first = events
.iter()
.find(|e| e["role"] == "assistant")
.map(|e| crate::daemon::transcript_redaction::redact_json_secrets(e.clone()))
.unwrap();
let usage = &first["usage"];
assert_eq!(usage["model"], "claude-opus-4-6");
assert_eq!(usage["inputTokens"], 3);
assert_eq!(usage["cacheCreationInputTokens"], 1619);
assert_eq!(usage["cacheReadInputTokens"], 13248);
assert_eq!(
usage["totalInputTokens"].as_u64().unwrap(),
3 + 1619 + 13248,
"inputTokens is disjoint from the cache counts"
);
}
}
23 changes: 23 additions & 0 deletions src/streams/agents/codex.rs
Original file line number Diff line number Diff line change
Expand Up @@ -592,4 +592,27 @@ mod tests {
CodexAgent::detect_subagent_parent(Path::new("/nonexistent/path/rollout.jsonl"));
assert_eq!(result, None);
}

/// A Codex `token_count` rollout event (codex-rs/protocol `TokenUsage`):
/// OpenAI convention, `input_tokens` includes `cached_input_tokens` and
/// `cache_write_input_tokens`. Forwarded verbatim, cache fields intact.
#[test]
fn test_token_count_cache_fields_are_forwarded() {
use std::io::Write;
use tempfile::NamedTempFile;

let line = r#"{"timestamp":"2026-11-10T10:00:05.000Z","type":"event_msg","payload":{"type":"token_count","info":{"total_token_usage":{"input_tokens":24210,"cached_input_tokens":18944,"cache_write_input_tokens":4096,"output_tokens":812,"reasoning_output_tokens":448,"total_tokens":25022},"last_token_usage":{"input_tokens":12105,"cached_input_tokens":9472,"cache_write_input_tokens":2048,"output_tokens":406,"reasoning_output_tokens":224,"total_tokens":12511},"model_context_window":272000},"rate_limits":null}}"#;
let mut file = NamedTempFile::new().unwrap();
writeln!(file, "{}", line).unwrap();
file.flush().unwrap();

let (events, _) = drain_all(&CodexAgent::new(), file.path());
let event = crate::daemon::transcript_redaction::redact_json_secrets(events[0].clone());
let total = &event["payload"]["info"]["total_token_usage"];
assert_eq!(total["input_tokens"], 24210);
assert_eq!(total["cached_input_tokens"], 18944);
assert_eq!(total["cache_write_input_tokens"], 4096);
assert_eq!(total["output_tokens"], 812);
assert_eq!(total["reasoning_output_tokens"], 448);
}
}
85 changes: 81 additions & 4 deletions src/streams/agents/continue_cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,12 @@ impl Agent for ContinueAgent {
message: format!("Invalid JSON in {}: {}", path.display(), e),
})?;

// Continue keeps a running session total (with prompt-cache reads and
// writes and its own cost) at the top level, outside `history`.
let session_usage = parsed
.get("usage")
.and_then(crate::streams::usage_snapshot::from_continue_session_usage);

let history = match parsed.as_object_mut().and_then(|obj| obj.remove("history")) {
Some(serde_json::Value::Array(arr)) => arr,
_ => {
Expand All @@ -162,16 +168,25 @@ impl Agent for ContinueAgent {
};

let batch_limit = self.batch_size_hint();
let history_len = history.len() as u64;

let events: Vec<serde_json::Value> = history
let mut events: Vec<serde_json::Value> = history
.into_iter()
.skip(already_processed as usize)
.take(batch_limit)
.collect();

let new_watermark = Box::new(RecordIndexWatermark::new(
already_processed + events.len() as u64,
));
let next_index = already_processed + events.len() as u64;
let new_watermark = Box::new(RecordIndexWatermark::new(next_index));

// Append the session total to the last batch that read new history
// items. It is not a history record, so the watermark ignores it.
if !events.is_empty()
&& next_index >= history_len
&& let Some(snapshot) = session_usage.and_then(|usage| usage.to_event(None))
{
events.push(snapshot);
}

Ok(StreamBatch {
events,
Expand Down Expand Up @@ -401,4 +416,66 @@ mod tests {
assert_eq!(result.events[0]["message"]["content"], "Let me check");
assert!(result.events[0]["contextItems"].is_array());
}

fn snapshots(events: &[serde_json::Value]) -> Vec<&serde_json::Value> {
events
.iter()
.filter(|e| e["type"] == crate::streams::usage_snapshot::USAGE_SNAPSHOT_EVENT_TYPE)
.collect()
}

#[test]
fn test_session_usage_with_cache_tokens_is_forwarded_once() {
let fixture = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"))
.join("tests/fixtures/continue-cli-session-simple.json");
let history_len = {
let parsed: serde_json::Value =
serde_json::from_str(&std::fs::read_to_string(&fixture).unwrap()).unwrap();
parsed["history"].as_array().unwrap().len()
};

// A small batch size: the snapshot rides only on the batch that reaches
// the end of the history, so it is sent once per read-through.
let agent = ContinueAgent::with_batch_size(2);
let (events, wm) = drain_all(&agent, &fixture);
assert_eq!(events.len(), history_len + 1);
let snaps = snapshots(&events);
assert_eq!(snaps.len(), 1);
assert!(std::ptr::eq(snaps[0], events.last().unwrap()));

let usage = &snaps[0]["autter_usage_snapshot"];
assert_eq!(usage["source"], "continue_session");
assert_eq!(usage["scope"], "session_total");
assert_eq!(usage["input_includes_cache"], false);
assert_eq!(usage["input_tokens"], 1813);
assert_eq!(usage["output_tokens"], 305);
assert_eq!(usage["cache_read_tokens"], 4366);
assert_eq!(usage["cache_write_tokens"], 2183);
assert_eq!(usage["cost_usd"], 0.002423);

// The snapshot is not a history record: the watermark counts history only.
let wm = wm.as_any().downcast_ref::<RecordIndexWatermark>().unwrap();
assert_eq!(wm.0, history_len as u64);

// Nothing new: no events and no repeated snapshot.
let again = agent
.read_incremental(&fixture, Box::new(RecordIndexWatermark::new(wm.0)), "test")
.unwrap();
assert!(again.events.is_empty());
}

#[test]
fn test_session_without_usage_adds_no_snapshot() {
use std::io::Write;
use tempfile::NamedTempFile;

let mut file = NamedTempFile::new().unwrap();
write!(file, "{}", make_continue_json(2)).unwrap();
file.flush().unwrap();

let agent = ContinueAgent::new();
let (events, _) = drain_all(&agent, file.path());
assert_eq!(events.len(), 2);
assert!(snapshots(&events).is_empty());
}
}
22 changes: 22 additions & 0 deletions src/streams/agents/copilot_cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -310,4 +310,26 @@ mod tests {
assert_eq!(id, Some("evt-001".to_string()));
assert_eq!(parent_id, None);
}

/// Copilot CLI's persisted `session.shutdown` event (copilot-sdk
/// `ShutdownModelMetricUsage`): per-model totals where `inputTokens`
/// includes `cacheReadTokens` and `cacheWriteTokens`.
#[test]
fn test_shutdown_model_metrics_cache_fields_are_forwarded() {
use std::io::Write;
use tempfile::NamedTempFile;

let line = r#"{"type":"session.shutdown","data":{"shutdownType":"routine","totalPremiumRequests":3,"totalApiDurationMs":41250,"sessionStartTime":1794304800000,"currentModel":"claude-sonnet-4.6","modelMetrics":{"claude-sonnet-4.6":{"requests":{"count":7,"cost":3},"usage":{"inputTokens":148200,"outputTokens":3120,"cacheReadTokens":121600,"cacheWriteTokens":18400,"reasoningTokens":640}}}},"id":"e5f6a7b8-5555-6666-7777-888899990000","timestamp":"2026-11-10T10:05:00.000Z","parentId":"d4e5f6a7-4444-5555-6666-777788889999"}"#;
let mut file = NamedTempFile::new().unwrap();
writeln!(file, "{}", line).unwrap();
file.flush().unwrap();

let (events, _) = drain_all(&CopilotCliAgent::new(), file.path());
let event = crate::daemon::transcript_redaction::redact_json_secrets(events[0].clone());
let usage = &event["data"]["modelMetrics"]["claude-sonnet-4.6"]["usage"];
assert_eq!(usage["inputTokens"], 148200);
assert_eq!(usage["cacheReadTokens"], 121600);
assert_eq!(usage["cacheWriteTokens"], 18400);
assert_eq!(usage["reasoningTokens"], 640);
}
}
Loading
Loading