feat(usage): record usage in a local store and serve it to the panel

.gitignore

@@ -32,6 +32,7 @@ auths/*
 docs/*
 !docs/management-api-v8.md
 !docs/audio-transcription.md
+!docs/usage-statistics.md
 AGENTS.md
 CLAUDE.md
 GEMINI.md

README.md

@@ -247,7 +247,9 @@ see [MANAGEMENT_API.md](https://help.router-for.me/management/api)
 
 ## Usage Statistics
 
-Since v6.10.0, CLIProxyAPI and [CPAMC](https://github.com/router-for-me/Cli-Proxy-API-Management-Center) no longer ship built-in usage statistics. If you need usage statistics, use:
+This fork records usage in a local database and shows it on the management panel's Usage page: tokens and API-equivalent cost per day, value per subscription, and breakdowns by project and model. See [docs/usage-statistics.md](docs/usage-statistics.md).
+
+Upstream CLIProxyAPI and [CPAMC](https://github.com/router-for-me/Cli-Proxy-API-Management-Center) have not shipped built-in usage statistics since v6.10.0. These external tools remain an alternative:
 
 ### [CPA Usage Keeper](https://github.com/Willxup/cpa-usage-keeper)
 

config.example.yaml

@@ -1289,7 +1289,10 @@ observability:
     request-log: false
 
   usage:
-    # When false, disable in-memory usage statistics aggregation
+    # When true, record one row per upstream request (tokens, model, account, latency, project)
+    # in "<auth-dir>/usage.db" for the management usage dashboard, and feed the in-memory usage
+    # queue. No prompt or response content is stored. When false, new usage is discarded; rows
+    # already recorded stay in usage.db until the file is deleted.
     usage-statistics-enabled: false
 
     # How long (in seconds) usage queue items are retained in memory for the Management API.

docs/usage-statistics.md

@@ -0,0 +1,115 @@
+# Usage statistics
+
+The proxy can record one row per upstream request and serve aggregates for the
+management panel's **Usage** page: tokens and API-equivalent cost over time,
+what each subscription delivered against its price, and breakdowns by project,
+model, client key, and more.
+
+## Turning it on
+
+Set the flag in `config.yaml`, or use the "Turn on recording" button on the
+Usage page:
+
+```yaml
+observability:
+  usage:
+    usage-statistics-enabled: true
+```
+
+The change applies on config reload, with no restart. Recording is off in Home
+(cluster) mode, where usage is reported to Home instead.
+
+## What is stored
+
+Rows go to a SQLite database at `<auth-dir>/usage.db`. Each row holds:
+
+- Time, provider, model, endpoint, status code, latency, and time to first token.
+- Token counts in mutually exclusive buckets: uncached input, cache read, cache
+  write (with the 5 minute and 1 hour split), output, and reasoning.
+- The upstream account. OAuth accounts are stored by their email or project;
+  API-key credentials are stored masked (`sk-a...0001`).
+- The client API key, masked.
+- Client IP, user agent, and session id.
+- The working directory the coding agent declared, and the project derived
+  from it.
+
+No prompt or response content is stored, and no full API key.
+
+Rows are kept until you delete them, also after recording is turned off. To
+start over, stop the server and delete `usage.db` together with its `-wal` and
+`-shm` files. The database file is readable by its owner only.
+
+## Projects
+
+The proxy never sees a repository name. Coding agents state their working
+directory in each request (Claude Code in a system message or its system
+prompt, Codex in its environment context), and the project is the last
+directory of that path. Only those places are read, never user or assistant
+content, and only near the start of the conversation.
+Agent worktrees are folded into their repository:
+
+| Working directory | Project |
+| --- | --- |
+| `~/repos/app` | `app` |
+| `~/.t3/worktrees/app/t3-1234` | `app` |
+| `~/repos/app/.claude/worktrees/feature` | `app` |
+| `~/repos/app/.worktrees/feature` | `app` |
+| `~/.codex/worktrees/1a2b/app` | `app` |
+
+A request that declares no working directory inherits the one last seen in its
+session for the same client key, and is otherwise listed as unknown.
+
+## Cost
+
+Cost is computed at query time from the stored token counts and a price table,
+so it is an estimate of what the same usage would cost at API list prices. It is
+not a bill. Because nothing is priced at write time, changing a price re-prices
+all history.
+
+- Built-in prices live in `internal/usagestats/default_prices.json`, which
+  records its source and retrieval date. Refresh that file when providers
+  change their prices.
+- A model name matches its exact price first, then the longest priced name it
+  extends at a separator, so `claude-opus-4-5-20251101` uses the
+  `claude-opus-4-5` price.
+- Models without a price are listed on the Usage page, and their tokens are
+  left out of the cost. Add or override prices with the page's **Prices** button.
+- Service tiers, batch discounts, and long-context surcharges are not modelled.
+
+Subscription value compares that cost with the monthly plan price you enter for
+each account, prorated to the days recorded in the selected range. Each account
+also lists the models it served, with every model's API-equivalent cost and its
+share of the account's total.
+
+## Management API
+
+All endpoints are under `/v8/management/observability/usage` and need the
+management key. They answer `409` with `usage_statistics_disabled` while
+recording is off.
+
+| Endpoint | Purpose |
+| --- | --- |
+| `GET /stats` | Token, request, and cost totals grouped by `group_by` |
+| `GET /speed` | Latency and first-token percentiles and output tokens per second |
+| `GET /settings` | Price overrides and plan prices, plus the built-in price table |
+| `PUT /settings` | Replace price overrides and plan prices |
+
+Query parameters for `/stats` and `/speed`:
+
+- `from`, `to`: Unix milliseconds. Default is the last 30 days.
+- `utc_offset_minutes`: minutes east of UTC, used to bucket days and hours in
+  the viewer's calendar.
+- `group_by`: comma-separated dimensions. `/stats` accepts `day`, `hour`,
+  `model`, `alias`, `provider`, `account`, `auth_index`, `auth_type`,
+  `client_key`, `client_ip`, `user_agent`, `endpoint`, `project`, `session_id`,
+  `reasoning_effort`, `service_tier`, `status_code`, `stream`, and `failed`.
+  `/speed` accepts one of those, except `day` and `hour`.
+- `filter.<dimension>=<value>`: keep only rows with that exact value.
+- `limit`: maximum number of groups (default 2000). The largest groups are kept.
+
+Example:
+
+```bash
+curl -H "Authorization: Bearer $MANAGEMENT_KEY" \
+  "http://127.0.0.1:8317/v8/management/observability/usage/stats?group_by=day,project"
+```

go.mod

@@ -31,21 +31,24 @@ require (
 	github.com/sirupsen/logrus v1.9.3
 	github.com/skratchdot/open-golang v0.0.0-20200116055534-eef842397966
 	github.com/tiktoken-go/tokenizer v0.8.1
+	go.etcd.io/bbolt v1.5.0
 	golang.org/x/crypto v0.57.0
 	golang.org/x/net v0.60.0
 	golang.org/x/oauth2 v0.30.0
-	golang.org/x/sync v0.23.0
-	golang.org/x/sys v0.48.0
+	golang.org/x/sync v0.24.0
+	golang.org/x/sys v0.49.0
 	golang.org/x/text v0.42.0
 	gopkg.in/natefinch/lumberjack.v2 v2.2.1
 	gopkg.in/yaml.v3 v3.0.1
+	modernc.org/sqlite v1.62.0
 )
 
 replace github.com/MoeForge/moego v0.1.0-alpha.1 => github.com/router-for-me/moego v0.1.2
 
 require (
 	github.com/cespare/xxhash/v2 v2.3.0 // indirect
 	github.com/dlclark/regexp2/v2 v2.5.1 // indirect
+	github.com/ncruces/go-strftime v1.1.0 // indirect
 	github.com/pion/datachannel v1.6.2 // indirect
 	github.com/pion/dtls/v3 v3.1.5 // indirect
 	github.com/pion/logging v0.2.4 // indirect
@@ -56,11 +59,14 @@ require (
 	github.com/pion/srtp/v3 v3.0.12 // indirect
 	github.com/pion/transport/v4 v4.0.2 // indirect
 	github.com/pion/turn/v5 v5.0.12 // indirect
+	github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
 	github.com/rogpeppe/go-internal v1.15.0 // indirect
 	github.com/wlynxg/anet v0.0.5 // indirect
-	go.etcd.io/bbolt v1.5.0 // indirect
 	go.uber.org/atomic v1.11.0 // indirect
 	golang.org/x/time v0.14.0 // indirect
+	modernc.org/libc v1.80.1 // indirect
+	modernc.org/mathutil v1.7.1 // indirect
+	modernc.org/memory v1.12.1 // indirect
 )
 
 require (
@@ -80,7 +86,7 @@ require (
 	github.com/cloudflare/circl v1.6.3 // indirect
 	github.com/cloudwego/base64x v0.1.4 // indirect
 	github.com/cloudwego/iasm v0.2.0 // indirect
-	github.com/dustin/go-humanize v1.0.1 // indirect
+	github.com/dustin/go-humanize v1.1.0 // indirect
 	github.com/emirpasic/gods v1.18.1 // indirect
 	github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
 	github.com/gabriel-vasile/mimetype v1.4.3 // indirect
@@ -99,7 +105,7 @@ require (
 	github.com/klauspost/cpuid/v2 v2.3.0 // indirect
 	github.com/leodido/go-urn v1.4.0 // indirect
 	github.com/lucasb-eyer/go-colorful v1.3.0 // indirect
-	github.com/mattn/go-isatty v0.0.20 // indirect
+	github.com/mattn/go-isatty v0.0.24 // indirect
 	github.com/mattn/go-localereader v0.0.1 // indirect
 	github.com/mattn/go-runewidth v0.0.19 // indirect
 	github.com/minio/md5-simd v1.1.2 // indirect