Skip to content

Add coding-plan usage tracking and fix OpenCode backend issues - #157

Open
DavidSegun wants to merge 6 commits into
Quality-Max:mainfrom
DavidSegun:feat/plan-window-tracking
Open

Add coding-plan usage tracking and fix OpenCode backend issues#157
DavidSegun wants to merge 6 commits into
Quality-Max:mainfrom
DavidSegun:feat/plan-window-tracking

Conversation

@DavidSegun

Copy link
Copy Markdown

Adds rolling coding-plan usage tracking with a new /plan command, status bar updates, configurable window duration (plan_window_hours), and automatic handling of plan exhaustion.

Also fixes several OpenCode issues discovered during implementation:

  • Added token reporting for OpenCode and Codex.
  • Surface OpenCode errors instead of silently dropping them, including plan limit (429) responses.
  • Fixed two Windows compatibility issues (--auto handling and -- argument separator).
  • Suppressed benign OpenCode 1.0.105 trailing errors while gracefully handling missing token usage events.

Testing

  • Added unit and parser tests.
  • Verified end-to-end with OpenCode 1.0.105.
  • Build, go test, and go vet all pass.

Orchestration backends (Claude Code, Codex, OpenCode + Z.AI GLM) run on
subscription plans that meter usage over a rolling ~5-hour window, but
qmax-code never tracked it, so it was unclear how much was used or when the
plan would stop.

Add a time-based PlanWindow tracker (internal/api/planwindow.go) that opens on
the first orch turn, accumulates turns/tokens, rolls over after the window
elapses, and can be marked exhausted from a provider limit hit. Surface it in
the input status bar and a new /plan command (also summarized in /status and
/cost). Window length is configurable via plan_window_hours (default 5).

Also close two backend blind spots that fed the same problem:
- OpenCode and Codex now report per-turn token usage (previously CC-only),
  feeding both session cost and the plan window.
- OpenCode `{type:error}` events were being silently dropped; they are now
  surfaced, and a 429 / usage-limit refusal is detected as a plan-limit hit,
  reading the reset time from rate-limit headers when present.

Build + vet clean; new planwindow unit tests pass. Codex/OpenCode token-field
names are parsed defensively pending confirmation against a live successful run.
…er tests

Verified against live `opencode run --format json` output (opencode 1.0.105):
successful turns emit a trailing status-code-less UnknownError (an internal
schema-validation gripe), and this version emits no token-usage event in the
stream. Surfacing every error event would therefore spam a scary line after
every good turn, so only errors carrying an HTTP status code (auth/quota/5xx)
and detected usage-limit hits are shown by default; status-code-less events
appear only in verbose mode.

Adds opencode_stream_test.go driving the parser with the real captured streams:
success (renders answer, no false limit, no invented tokens), a 429 limit hit
(flags the window + reads Retry-After), a 403 subscription error (surfaced but
not a limit), and forward-compat token extraction for both field shapes.
… supports it

opencode 1.x removed the `run --auto` flag and now governs tool approvals
through the config `permission` block. qmax-code passed --auto unconditionally,
so on current opencode every turn failed: opencode printed its usage and exited
1 with no output ("opencode exited with error: exit status 1"), making the
OpenCode backend (incl. Z.AI GLM coding plans) unusable.

Probe `opencode run --help` once and only append --auto when the flag is still
advertised, preserving behavior on older opencode while working on 1.x. The
managed config's permission block already enforces the standard/unattended
policy, so nothing is auto-approved that was not before.

Verified against opencode 1.0.105: without --auto the run produces the answer
and the turn completes.
On Windows, npm installs opencode as a `.cmd` shim. Go's os/exec runs `.cmd`
files through cmd.exe, which swallows the `--` end-of-flags separator and drops
the positional message that follows it — opencode then aborts with "You must
provide a message or a command" and exits 1, so no OpenCode turn (including
Z.AI GLM / OpenRouter coding plans) can run on Windows.

Pass the message without `--` on Windows (sanitizeCCUserPrompt already strips
control bytes, so a lone positional is safely taken as the message); keep `--`
on other platforms where it protects a message starting with "-". Verified via
a Go exec repro: with `--` the message is dropped, without it the turn runs.
@DavidSegun
DavidSegun requested a review from Desperado as a code owner July 24, 2026 19:12
@sigilix

sigilix Bot commented Jul 24, 2026

Copy link
Copy Markdown

Sigilix Overview

Effort: 4/5 (large)

Quality gates

  • ⚠️ PR title follows convention — Title doesn't match repo convention: ^(?:feat|fix|chore|docs|refactor|test|perf|style|build|ci|revert)(?:\([^)]+\))?!?: .+
  • ✅ PR description is complete
  • ℹ️ PR is linked to an issue — No Closes #N / Closes SIG-N keyword found in PR body or commit messages.

Summary — latest push

Adds rolling 5-hour coding-plan usage tracking for subscription backends (Claude Code, Codex, OpenCode) with a new /plan command and status bar integration, while fixing several OpenCode issues: surfacing error events (including 429 plan-limit refusals), accumulating multi-step token usage, conditionally passing --auto, and stripping the -- argument separator on Windows. The plan window is keyed per backend and provider to keep independent quotas separate, and provider-reported reset times from rate-limit headers now override the local 5-hour estimate on exhaustion.

Important files

File Score Notes Next step
internal/repl/repl.go 5/5 Integrates PlanWindow into the REPL turn loop, status bar, and /plan, /status, /cost commands, and adds helpers for plan-backend detection, per-backend window keying, and turn recording. Verify that currentBackend(ag) returns the correct value during mid-session backend switches, ensuring planWindowFor doesn't allocate a stale or duplicate window.
internal/agent/opencode_agent.go 5/5 Adds token usage accumulation across multi-step turns, surfaces error events (including 429s), conditionally passes --auto, and strips the -- separator on Windows. Ensure the autoFlagOnce probe is invalidated or re-probed if the opencode binary is updated mid-session, otherwise a stale cached value could persist.
internal/api/planwindow.go 4/5 Defines the PlanWindow type that tracks rolling usage, rollover at provider-reported reset times, and exhaustion state for subscription quotas. Add a concurrency note to the doc comment explicitly stating PlanWindow must only be used from a single goroutine, since the REPL relies on this but the type looks generally useful.
internal/agent/planlimit.go 4/5 Introduces isPlanLimitMessage for broad heuristic matching of rate/quota errors and parseResetTime for extracting reset times from HTTP headers. Add unit tests for parseResetTime covering delta-seconds, HTTP-date, epoch-seconds, and small-delta heuristics to ensure no misparses.
internal/agent/codex_agent.go 4/5 Adds token usage extraction and plan-limit detection to the Codex backend stream, mirroring the OpenCode changes. Add parser tests for inspectCodexEvent covering the codexEvent struct variations (top-level vs nested usage) and plan-limit message matching.

Sequence diagram

sequenceDiagram
    participant User
    participant REPL
    participant PlanWindow
    participant OpenCodeAgent
    User->>REPL: /cc or /opencode turn
    REPL->>PlanWindow: planWindowFor(backend)
    REPL->>OpenCodeAgent: Run(userMsg)
    OpenCodeAgent-->>REPL: result / error
    OpenCodeAgent-->>REPL: LastTurnStats() / LastPlanLimit()
    REPL->>PlanWindow: Record(now, turnIn, turnOut)
    alt Plan limit hit (429)
        REPL->>PlanWindow: MarkExhausted(now, reset)
    end
    REPL->>User: /plan or status bar shows window state
Loading

Confidence: 4/5

The core plan-tracking logic is well-isolated in a pure data type with thorough tests, and the OpenCode fixes address specific known failure modes, though the single-goroutine assumption on PlanWindow and the cached --auto probe warrant a quick check.

  • In repl.go around line 1232, verify that currentBackend(ag) correctly resolves the backend after a mid-session switch so planWindowFor doesn't allocate a stale window for the old backend.
  • In opencode_agent.go, the autoFlagOnce cache means --auto support is probed only once per process; confirm this is acceptable if users upgrade opencode binaries without restarting.
  • In opencode_agent.go, the -- argument separator is stripped on Windows (runtime.GOOS == "windows") — confirm this doesn't break edge cases where the user prompt starts with a dash on that platform.
  • In planwindow.go, Record rolls over at ResetAt() which prefers ExhaustedReset over the local estimate; verify that a 429 response with a malformed or zero Retry-After doesn't cause an immediate rollover to time.Time{}.
  • In repl.go, recordPlanTurn is called even when runErr != nil to catch 429s; confirm that non-limit errors (e.g., network timeouts) don't incorrectly mark the window exhausted when the agent doesn't implement PlanLimitReporter.

Suggested labels: feature bug

Suggested reviewers: @Desperado


Posted · bc9cb5d · 0 findings — View review
Dismiss @sigilix dismiss <reason> (not-a-bug | bad-anchor | already-covered | too-minor | wrong-context) · Re-run /sigilix review
Sigilix · 0 of 50 reviews used in past 5h

@sigilix sigilix Bot added the enhancement New feature or request label Jul 24, 2026
@qualitymaxapp

qualitymaxapp Bot commented Jul 24, 2026

Copy link
Copy Markdown

⚠️ QualityMax Pipeline

Gate Result
🔍 AI diff review ✅ Clean · gemini-3.1-flash-lite · completed · 13 eligible / 3 reviewed · gemini-3.1-flash-lite
🔍 SAST completed · 16 eligible / 16 reviewed · gemini-3.1-pro-preview
🔍 Canonical PR review delivery completed · 0 eligible / 0 reviewed · exact-head review #4857181274 and overview #5182387349 confirmed
🧪 Repo Tests ✅ 579/579 passed (go)
🤖 AI Tests ⚠️ 51/56 passed

Powered by QualityMax — AI-Powered Test Automation

Copy link
Copy Markdown
Contributor

Review findings — request changes

I found four correctness issues in the new plan-tracking paths:

  1. BLOCKER — limit hits can bypass exhaustion tracking. LastPlanLimit() is read only inside if err == nil (internal/repl/repl.go:1209-1230). Both CLI agents return an error when a limit event produces no assistant output and the subprocess exits non-zero, so the usual 429/refusal path can set lastLimitHit but never call MarkExhausted. Please consume the limit state regardless of the run error, or return and handle a typed plan-limit error.

  2. WARNING — provider reset times do not roll the window over. PlanWindow.Record resets only after WindowLen. If the provider reports an earlier authoritative reset, the first successful turn after that time remains in the old exhausted window and the UI continues to show it as exhausted. Reset based on the effective ResetAt() when recording a new turn.

  3. WARNING — live backend switching mixes independent quotas. A single tracker is allocated once (internal/repl/repl.go:170-174), while /cc, /codex, and /opencode switch providers live. Turns and exhaustion state from separate subscription plans are therefore combined. Please keep trackers keyed by backend/provider, or otherwise start/select the correct window when the backend changes.

  4. WARNING — OpenCode undercounts multi-step/tool turns. The parser overwrites lastTurnIn/lastTurnOut for every usage-bearing event (internal/agent/opencode_agent.go:347-353), leaving only the final step's tokens. OpenCode emits usage on each step-finish, so tool loops can contain multiple token-bearing steps. Accumulate one canonical token payload per distinct step/event and add a two-step parser test.

Verification

  • go test ./... passed locally.
  • go test -count=3 ./... passed locally.
  • go vet ./... passed locally.
  • The GitHub QualityMax pipeline remains red at 529/530 repository tests; that single failure did not reproduce locally.

@Desperado Desperado left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @DavidSegun , please check the comments above.

Resolves the review on Quality-Max#157, which had been outstanding since 2026-07-25.

1. BLOCKER — limit hits could bypass exhaustion tracking. LastPlanLimit() was
   read only inside `if err == nil`, but a limit refusal that produces no
   assistant output makes the CLI subprocess exit non-zero, so the usual 429
   path set the agent's limit flag and never called MarkExhausted. The
   bookkeeping now runs whether or not the turn errored: a successful turn
   still advances the window, and the limit state is consumed either way.
   Extracted as recordPlanTurn so the behaviour is unit-testable rather than
   buried in the REPL closure.

2. Provider reset times now roll the window over. Record() rolled over only
   after WindowLen, so when a provider reported an earlier authoritative reset
   the first accepted turn after it landed in the old exhausted window and the
   UI kept showing a limit that no longer applied. Rollover now uses the
   window's effective ResetAt().

3. Live backend switching no longer mixes independent quotas. One tracker was
   allocated per session while /cc, /codex, and /opencode switch plans mid
   -session, combining turns and exhaustion state across separate
   subscriptions. Trackers are now keyed by quota, with OpenCode keyed per
   provider since its providers meter separately.

4. OpenCode no longer undercounts multi-step turns. The parser overwrote
   lastTurnIn/lastTurnOut on every usage-bearing event, keeping only the final
   step; opencode emits usage on each step-finish, so tool loops lost every
   earlier step. Usage now accumulates across steps, taking one canonical
   payload per event (the four shapes are the same numbers in different
   places) and skipping any step id already counted.

Tests: two-step and repeated-step parser tests, provider-reset rollover
boundary tests, failed-run exhaustion tests, and quota-keying tests.
go build, go test ./..., and go vet ./... all pass.
@sigilix sigilix Bot added the bug Something isn't working label Aug 4, 2026
@Desperado

Copy link
Copy Markdown
Contributor

@DavidSegun I've pushed fixes for the four review findings directly to this branch (your maintainer_can_modify was enabled, so no force-push — your four commits are untouched and mine sit on top). I also merged current main in, which resolves the conflict this PR had picked up since v1.22.1 landed.

Happy to hand any of these back if you'd rather do them your way.

1. BLOCKER — limit hits bypassing exhaustion tracking. The plan bookkeeping now runs whether or not the turn returned an error. A refusal that produces no assistant output exits non-zero, so reading LastPlanLimit() only inside if err == nil set the agent's limit flag and never called MarkExhausted — the window never showed the limit it exists to show. A successful turn still advances the window; a failed one only consumes the limit state. I extracted this as recordPlanTurn so it's unit-testable instead of buried in the REPL closure.

2. Provider reset times now roll the window over. Record used the window's effective ResetAt() rather than WindowLen alone. Previously, if the provider reported an earlier authoritative reset, the first accepted turn after it landed in the old exhausted window and the UI kept showing a stale limit.

3. Live backend switching no longer mixes quotas. Trackers are keyed per subscription via planWindowKey, since /cc, /codex, and /opencode switch plans mid-session. OpenCode is keyed per provider — zai and groq meter separately, so opencode/zai and opencode/groq get their own windows.

4. OpenCode multi-step undercount. Usage now accumulates across steps instead of being overwritten. Two details worth flagging: the four usage shapes (ev.Tokens, ev.Usage, ev.Part.Tokens, ev.Part.Usage) are the same payload in different places rather than separate counts, so exactly one canonical payload is taken per event; and steps are deduped by part id in case opencode re-emits one, since it does that for text parts.

Tests added

  • TestOpenCodeAccumulatesUsageAcrossSteps — the two-step parser test you were asked for: a tool-calling turn that costs 150/45 across two step-finish events reported 30/6 before.
  • TestOpenCodeDoesNotDoubleCountARepeatedStep and TestOpenCodeCountsOneCanonicalPayloadPerEvent — the two ways naive accumulation would overcount.
  • TestRecordRollsOverAtProviderReportedReset and TestRecordStillRollsOverOnWindowLengthWithoutAProviderReset — the rollover boundary.
  • TestRecordPlanTurnMarksExhaustedWhenTheRunFailed — the blocker, asserted directly.
  • TestPlanWindowKeySeparatesQuotas — table test over the backend/provider keys.

go build ./..., go test ./... (full suite, zero failures), and go vet ./... all pass.

@qualitymaxapp qualitymaxapp Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QualityMax Review — canonical overview updated; inline findings are attached to this review.

@qualitymaxapp

qualitymaxapp Bot commented Aug 4, 2026

Copy link
Copy Markdown

QualityMax Review

Verdict: COMMENT · Confidence: evidence-backed scan

Files eligible: 16 · Files reviewed: 16 · Files with findings: 0 · Findings: 0 · Inline cards: 0

Priority findings

priority location finding
No blocking findings

Review gates

gate status
AI diff review completed · eligible 13, reviewed 3 · LLM · served gemini-3.1-flash-lite
SAST completed · eligible 16, reviewed 16 · hybrid · served gemini-3.1-pro-preview
Overall review evidence clean
Inline evidence not needed

Important files

file risk note next step
No findings

Change diagram — Flow

flowchart TD
    subgraph RunLoop [CodexAgent.Run Loop]
    A[scanner.Scan] --> B[inspectCodexEvent]
    B --> C{Parse JSON Event}
    C --> D[Update Token Stats]
    C --> E{Check Error/Limit}
    E -- Limit Hit --> F[Set lastLimitHit = true]
    E -- No Limit --> G[Continue]
    A --> H[extractCodexMessage]
    H --> I[term.StreamText]
    end
Loading

Review lifecycle

Use the inline cards to inspect evidence and suggested remediation. Re-run the QualityMax review after pushing a fix; unchanged cards are identified by their stable finding marker. Dismiss with a reason through the existing QualityMax/GitHub review feedback flow. 0 prior card(s) are stale/resolved on this head. @qmax Q&A is tracked separately.

Proof legend: VERIFIED independently judged patch · REPRODUCED verified finding · GROUNDED deterministic evidence · MODEL-ONLY model judgment.

QualityMax project results are available in the configured project.

Receipt · commit bc9cb5d07025a95f350dec3fba1a669edfcb49d1 · run 2026-08-04T17:21:26+00:00 · model served gemini-3.1-pro-preview, gemini-3.1-flash-lite · model requested gemini-3.1-pro-preview, gemini-3.1-flash-lite · model review substantive — 1053 model output tokens · model source repository ai_review_preferences.preferred_model · re-review 1 · proof counts {}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working enhancement New feature or request qualitymax:reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants