Repository navigation
feat!: add a Decisions API judge and a pluggable overview step for clawhub - #148
Conversation
…awhub
The clawhub and clawhub-aig profiles now take their verdict from the OpenAI
Decisions API through clawscan-decisions-judge, a stdlib Python command shipped
in the runtime image. It builds one bounded request from the judge workspace
(exact artifact text plus an input_json part with run metadata, scanner
summaries, SARIF coverage gaps, and normalized findings), asks a verdict choice
plus dimension choices, risk-signal predicates, and per-finding predicates,
maps the typed answers onto the clawhub output schema, and records model,
usage, and latency. It enforces its own input limit; refusals, HTTP errors, and
malformed answers fail the judge with the real error.
Add an optional profile overview.command (--overview, --no-overview). It runs
after a completed verdict with the judge workspace plus judge.json and the
{{ judge.result }} prompt placeholder, records overview in the run artifact,
and never changes the verdict, gate, status, completeness, or exit status. The
clawhub overview runs Codex gpt-6-luna with low effort, routed through
OPENAI_BASE_URL when set.
BREAKING CHANGE: clawhub judge.result now holds typed fields only plus a
decision record; summary, dimension details, finding notes, and user_guidance
move to overview.result.
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
|
Codex review: needs maintainer review before merge. What this changesThis PR adds a Decisions API judge and an optional overview command that explains completed verdicts without affecting the gate. Example: Scan the daily-notes skill using the clawhub profile.
Review scores
ProductKind: Feature · Worth it: Yes · Fix scope: Complete Merge readiness✅ Ready for maintainer review Keep this maintainer-authored PR open: the prior findings are resolved, and review found no remaining actionable merge blocker. Priority: P2 Before mergeNone. FindingsNone. Agent review detailsHow this fits togetherThe runner passes artifact files and scanner evidence to an external judge, records its verdict, and optionally runs a separate explanatory command. flowchart LR
A[CLI and profile] --> B[Scanner adapters]
B --> C[Artifact and raw evidence]
C --> D[Decisions judge]
D --> E[Typed verdict]
E --> F[Optional overview]
E --> G[Run artifact and gate]
F --> G
Technical reviewBest possible solution: Land the external Decisions judge and advisory overview while retaining legacy artifact readability and the documented downstream deployment boundary. Do we have a high-confidence way to reproduce the issue? This is a feature PR; source inspection confirms the earlier limit and gap-preservation findings are addressed, without executing target code during this review. Is this the best way to solve the issue? The implementation fits the established external-judge boundary and reuses the shared harness for overview execution while keeping overview failures outside verdict completeness. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning medium; reviewed against fa01a95bcd18. Provenance checked
TestingProof path: shipped entry point. SecurityNone. EvidenceWhat I checked:
Likely related people:
LabelsLabel changes:
Label justifications:
Rating scale6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior. WorkflowClawSweeper edits this one comment on every review. Comment HistoryReview history (2 earlier review cycles)
Reviewed November 4, 2026, 4:13 AM ET / 09:13 UTC (Revision 3). |
…ictly - Exclude node_modules at every depth from plugin judge text, manifest, and budget, reporting only artifactCounts.excludedDependencies as a deliberate scope exclusion. Skill targets and the copied artifact are unchanged. - Project analysis_completeness lists (ledger exceptions, scope exclusions, references, limitations, incomplete analyzer statuses), component gaps, and bounded metadata provenance into bounded, path-aware records with counts and truthful omission accounting, for root reports and bundled children (with rootPath and prefixed paths). - Check byte, JSON value, and depth limits on the fully assembled evidence, including the manifest and counts, trimming with recorded omissions. - Keep finding text out of questions: finding predicates reference a stable evidence index. - Reject duplicate, missing, unexpected, mistyped, out-of-range, or non-finite answers; any refusal is terminal.
…able Add real stored artifacts from the ClawHub production pin (c3cfaf1) and pre-#148 main (fa01a95) and check they decode, summarize, and re-encode with judge.result unchanged and no overview. Run the exact pre-#148 clawhub profile as a project config end to end, and keep it valid against the config schema.
|
@clawsweeper re-review |
|
🦞👀 Re-review progress:
|
The fixture had no CompletedAt, so checkedAt fell back to the current millisecond and two renders could differ (pre-existing flake on main since #147).
findingCounts.omitted was added after the fit loop, so evidence measured at exactly 10,000 values was sent with 10,001. Initialize all accounting fields before measuring, re-check the exact final object, and add exact-boundary regressions (the old judge sends 10,001 values on the new test).
1,661 findings with context {} and an eight-character SKILL.md measure exactly 10,000 values untrimmed; the previous judge sends 10,001.
|
@clawsweeper re-review |
|
🦞👀 Re-review progress:
|
Summary
ClawHub's verdict now comes from the OpenAI Decisions API. Decisions returns typed answers only, so readable prose moves to a new, separate, pluggable overview step.
clawscan-decisions-judge(stdlib Python,docker/clawscan-runtime/judges/clawscan_decisions_judge.py, installed onPATHin the runtime image). It is an ordinary judge command: it reads the judge workspace and writes schema-valid JSON to{{ output }}. It owns its own checks (input limit, refusals, HTTP errors, schema validation). The runner stays judge-agnostic.overview.command(plus--overview <cmd>/--no-overview). It runs after a completed verdict in a fresh workspace with the same layout and placeholders plusjudge.jsonand the{{ judge.result }}prompt placeholder. Its result is recorded underoverviewin the run artifact. A failure, invalid output, or timeout (5 min, separate budget) is recorded and never changes the verdict, gate,status,complete, or exit code. It is skipped (status: "skipped") when the judge did not complete. Benchmarks skip the profile overview unless--overviewis passed.gpt-6-luna), then a Codexgpt-6-lunaoverview with low reasoning effort and Standard tier, routed throughOPENAI_BASE_URL(for example ClawRouter) when it is set.Judge request shape
POST {DECISIONS_BASE_URL}/v1/decisions:{ "model": "gpt-6-luna", "input": [ {"role": "developer", "content": "<clawhub/decisions.md policy>"}, {"role": "user", "content": [ {"type": "input_text", "text": "<exact artifact text files between random per-run boundary markers>"}, {"type": "input_text", "text": "Scanner and run evidence follows as JSON ..."}, {"type": "input_json", "data": { "metadata": {"target": {"kind": "...", "id": "..."}, "scannerStatus": {...}, "runContext": {...}}, "scanners": [{"scanner": "aig", "format": "sarif", "invocations": [...], "findingCount": 1}, ...], "findings": [{"scanner": "...", "ruleId": "...", "severity": "...", "title": "...", "file": "...", "line": 3}], "findingCounts": {...}, "artifactManifest": [{"path": "...", "bytes": 1, "sha256": "...", "status": "included|excerpted|omitted"}], "artifactCounts": {...} }} ]} ], "questions": [ {"type": "choice", "name": "verdict", "instructions": "...", "choices": [ {"value": "benign", "description": "..."}, {"value": "suspicious", "description": "..."}, {"value": "malicious", "description": "..."}]}, {"type": "choice", "name": "dimension.<purpose_capability|instruction_scope|install_mechanism|environment_proportionality|persistence_privilege>", "choices": [ok, note, concern]}, {"type": "predicate", "name": "signal.<exfiltration|tool_poisoning|backdoor|remote_code_execution|persistence|credential_access|obfuscation|destructive_actions>"}, {"type": "predicate", "name": "finding.<n>"} // "expected for the stated purpose?", up to 25 highest-severity findings ] }Verdict instructions: malicious = intentional harmful or deceptive behavior (exfiltration, tool poisoning, backdoors, RCE, persistence, destructive actions, deliberate misdirection). suspicious = real security weaknesses or undeclared risky behavior without clear malicious intent. benign otherwise. The policy states that scanner severity is never evidence of malice by itself, and that scanner failures and coverage gaps (metadata statuses and errors, SARIF
invocations[].executionSuccessfuland notifications) are uncertainty, not findings.Bounds (owned by the judge): text files are sent whole up to 120k chars each and 600k in total. Larger files are head/tail excerpted. Binary and non-UTF-8 files appear only in the manifest. Findings are capped at 200 and trimmed to keep
input_jsonunder 200 KiB (the API limit is 256 KiB, 10k values, depth 32). The total must fit--max-input-chars(default 1,000,000, envDECISIONS_MAX_INPUT_CHARS). Otherwise the judge exits 2 withdecisions judge input too large: N chars exceeds the L-char limit (...)and sends no request.Failures: any
refusalanswer, HTTP error (message/type/code preserved, e.g.Decisions API HTTP 401: Incorrect API key provided. (type=invalid_request_error, code=invalid_api_key)), network error, missing or unexpected answer, or schema mismatch exits 1 with the real error on stderr. ClawScan records a failed judge with that error and does not retry. HTTP 429/5xx get up to 2 internal retries within the judge timeout, honoringRetry-After.Config (env only):
DECISIONS_API_KEY(falls back toOPENAI_API_KEY),DECISIONS_BASE_URL(falls back toOPENAI_BASE_URL, thenhttps://api.openai.com/v1;/v1is appended when missing),DECISIONS_MODEL(defaultgpt-6-luna; ClawRouter needsopenai/gpt-6-luna),DECISIONS_MAX_INPUT_CHARS. The overview usesCODEX_API_KEY(falls back toOPENAI_API_KEY),OVERVIEW_MODEL(defaultgpt-6-luna), andOPENAI_BASE_URL. All are in the profilesandbox.envallowlist.Verdict mapping (
internal/profiles/clawhub/output.schema.json)verdictverdictchoiceconfidencehigh≥ 0.8,medium≥ 0.5, elselowdimensions.<name>{status, confidence}fromdimension.<name>scan_findings_in_context[]{ruleId (rule + location), scanner, expected_for_purpose (p ≥ 0.5), probability}decisionapi,endpointHost,requestedModel, API-reportedmodel,latencyMs,httpAttempts,usage, input accounting, verdict{choice, confidence, probabilities},signals{...}judge.resultfor clawhub/clawhub-aig no longer has prose:summary,dimensions.*.detail,scan_findings_in_context[].note, anduser_guidanceare gone. Dimensions are{status, confidence}. Findings gainscannerandprobability. There is a new requireddecisionobject.verdictandconfidencekeep their enums.overview(omitted when no overview is configured):{status: completed|failed|skipped|running|not_started, commandStarted, configuredModel, promptSha256, outputSchemaSha256, error, durationMs, result}. For clawhub,overview.result={summary, dimensions.<name> (string), scan_findings_in_context[{ruleId, note}], user_guidance}(internal/profiles/clawhub/overview.schema.json). Consumers must treat the overview as optional: show prose whenoverview.status == "completed", and never gate on it.judge.configuredModelis now absent for the clawhub judge (the Codex-only parser doesn't recognize it). Usejudge.result.decision.model/requestedModel.--sandbox off, the ClawHub worker must putclawscan-decisions-judgeonPATH, e.g.ln -s "$CLAWSCAN_SOURCE_DIR/docker/clawscan-runtime/judges/clawscan_decisions_judge.py" "$RUNNER_TEMP/clawscan-bin/clawscan-decisions-judge"(needs onlypython3), plusDECISIONS_*/OVERVIEW_MODELenv as desired. The Docker runtime image already ships it.cmd/verify-clawhub-promptstill compares the embeddedoutput.schema.jsonwith ClawHub's production Codex schema, so it will report a schema mismatch until ClawHub adopts the new schema.clawhub/prompt.mdstays embedded for that tool and for--judgecomparisons, but the profiles no longer use it.scanners/(for example a single plugin-wideaig-mcpSARIF) is ingested the same way, with invocation failures preserved as coverage gaps. No scanner adapters are touched here.Tests
cmd/clawscan/decisions_judge_test.go: tests at theclawscan <target> --profile clawhubseam with the real judge command, anhttptestDecisions stub, a stub SkillSpector subprocess, and a stubcodexoverview:input_text+input_json, verdict choices as objects with descriptions), auth header, model/usage/latency recorded, no key in the artifact; the overview completes, its prompt andjudge.jsoncarry the verdictDecisions API refused question(s): verdict, runfailed/complete:false/gateerror, no retry, overviewskippedDECISIONS_MAX_INPUT_CHARS=4000) → the judge's owndecisions judge input too large ... exceeds the 4000-char limit, zero requests sentfailedwith the real error, while verdict, gate,status,complete, and exit status are identical to a--no-overviewruninternal/runner/overview_test.go:judge.json+{{ judge.result }}rendering, overview timeout recorded (not returned), skip without a completed verdict, placeholder fails without a verdict.docker/clawscan-runtime/tests/test_decisions_judge.py(CI lint job + inside the runtime image build): request shape, mapping, refusal, HTTP error, 429 retry, oversized input, excerpting,input_jsonbounding, SARIF coverage gaps + scanner failures preserved, schema validation.internal/profiles: profile wiring, overview Codex args (luna, low effort, key alias,OPENAI_BASE_URLprovider routing),--overview/--no-overview, benchmark skip.go test -count=1 ./...,go vet ./..., gofmt, runtime-image unit tests,make docs-site, all green.Live smoke (ClawRouter, 2026-11-03)
Locally built
clawscanwith the realclawscan-decisions-judgeand Codex CLI 0.157.0,--profile clawhub --sandbox off, scanner fixtures (clean SkillSpector, empty A.I.G SARIF),DECISIONS_BASE_URL=https://clawrouter.openclaw.ai,DECISIONS_MODEL=openai/gpt-6-luna,OPENAI_BASE_URL=https://clawrouter.openclaw.ai/v1,OVERVIEW_MODEL=openai/gpt-6-luna, key fromMODEL_CLAWROUTER_API_KEY(never printed; 0 occurrences in either artifact):daily-notesSKILL.mdbenign/ high (p=1.0), all dimensionsok, all signals 0gpt-6-luna, 237 ms, 3,649 input tokens, 1 HTTP attempt./notes.md; its instructions are limited to that task…"completed,complete:true, gatepass, exit 0tar ~/.ssh ~/.aws | curl POSTandcurl … | shinto~/.zshrcmalicious/ high (p=0.99), all dimensionsconcern; exfiltration, RCE, persistence, tool_poisoning, credential_access = 1.0gpt-6-luna, 647 ms, 3,697 input tokenscompleted,complete:true, exit 0Note: through ClawRouter the Codex overview took about 170–200 s, well inside its 5-minute budget but not fast. The verdict itself took under 1 s.
Review fixes (15d86a2 + c82a2fe, runtime qualified at the final head)
node_modulesat any depth are excluded from the judge text, manifest, and budget and reported only asartifactCounts.excludedDependencies {files, bytes, reason}, a deliberate scope exclusion rather than missing coverage. The copied artifact is untouched, and skill targets keepnode_modulesin scope. The parent'sTestIntegratedPluginAIGDecisionsOverviewfails with the old judge and passes with this one.analysis_completenesslists (ledger exceptions, scope exclusions, references, limitations, incomplete analyzer statuses), component gaps (limitations, incomplete inspection, executable code), and boundedmetadataprovenance become{total, countsBy, counts, records, omitted}with group-balanced, path-aware records. Bundled children keeprootPath, prefixed paths, their own gaps, provenance, and findings (rootPathon each child finding). Covered by tests on the existingskillspector-usable-partial.jsonandskillspector-bundled-partial.jsonfixtures, plus a 3,000-record bounding test.omitted/manifestOmittedaccounting and a hard failure if nothing is left to trim. Final stats are recorded indecision.input.evidenceJson. The reviewer's repro (2,000 findings + 450 empty files) is a test.--max-input-charsis documented as a character budget, not a token guarantee.findingCounts.omittedwas added after the fit loop, so evidence measured at exactly 10,000 values went out with 10,001. All accounting fields are now initialized before the first measurement, and the exact final object is re-checked after assembly (assert_evidence_fits). The newtest_fit_evidence_exact_value_boundarypads evidence to exactly the limit; the previous judge sends 10,001 values there and the fixed judge stays within the limit. Also added: ClawSweeper's exact e1781d2 repro (1,661 findings,context: {}, 8-char SKILL.md: 10,001 values on the old judge), the reviewer's 1,200 findings + 463 empty files case, and a 455–471-file sweep asserting bytes, values, and depth on what is actually sent.indexin the evidence JSON. No scanner or artifact text is interpolated intoquestions[].instructions(hostile-title test).Overview latency investigation
Measured locally through ClawRouter with a captured overview workspace: the model turn itself takes 7–9 s (about 57k input tokens, mostly cached, two tool calls). The 172–204 s seen earlier is Codex CLI process overhead before
thread.started(9–227 s) and afterturn.completed(up to ~68 s), which varies run to run on this machine. It persists with an emptyCODEX_HOMEand withanalytics.enabled=false,feedback.enabled=false,check_for_update_on_startup=false, andsupports_websockets=false. ClawRouter/v1/modelsanswers in under 1 s. This looks environment-specific (this host routes Codex through a local egress proxy). This is not proof that the Codex CLI is fast elsewhere. Latency must be measured on the actual worker against the 900 s whole-plugin budget. The overview stays bounded (5 min) and non-gating, and no profile change is made for it here.Data-model compatibility (e1781d2; f905a55 fixes a pre-existing #147 test flake)
Proof in this repo:
cmd/clawscan/compat_test.go›TestLegacyStoredArtifactsRemainReadableuses real artifacts generated by the old binaries. Both come from--profile clawhub --sandbox off, with scanner fixtures and a stub judge writing the legacy prose result; host paths are scrubbed.testdata/legacy-clawhub-artifact-c3cfaf1.jsoncomes from the ClawHub production pin, with the legacyartifact_inspectionreceipt, receipt fields onjudge.attempts, and the retiredclawscan-staticscanner.testdata/legacy-clawhub-artifact-fa01a95.jsoncomes from pre-feat!: add a Decisions API judge and a pluggable overview step for clawhub #148 main.runner.Artifactand summarizes correctly (judge_completed: 1,clean: 1, no overview lines).overviewstays absent (omitempty) andjudge.resultis deep-equal to the stored value.TestPreOverviewProfileConfigRunsUnchangedruns the exact pre-feat!: add a Decisions API judge and a pluggable overview step for clawhub #148 clawhub profile (schemas/testdata/clawhub-profile-fa01a95.yml, with its original prompt and output schema) as a project config.codexstub stands in for the CLI.judge.resultis recorded unchanged, andjudge.configuredModelis stillgpt-6.1-sol.schemas/clawscan.schema_test.goalso keeps that pre-overview config valid against the updated config schema.overviewis optional at every level.overview.commandis configured, and--no-overviewremoves it.overview.statusother thancompleted, as "no prose". It never affects verdict or gate.Consumer side (openclaw/clawhub#4411 at
fba9211, draft, not activated), positive tests that keep legacy and old-pin behavior:scripts/security/clawscan-receipt.test.ts› "continues accepting legacy v1 and rejecting its extra fields"c3cfaf1) package scanner contract without fabricated AIG evidence:convex/packageSecurityScan.runtime.test.ts› "preserves the old pinned package scanner contract without fabricated AIG evidence"convex/packageSecurityScan.runtime.test.ts› "preserves prior optional scanner results when an older worker omits them"scripts/security/clawscan-decisions.test.ts› "takes prose only from a successful optional overview"Breaking boundary: the new
clawhub/clawhub-aigprofile projection is the break.judge.resultloses its prose fields and gainsdecision, prose moves tooverview.result, the clawhuboutput.schema.jsonchanges, andclawscan-decisions-judgemust be onPATH.clawscan-run-v1), custom/project profiles, and--judgeoverrides.c3cfaf1until #4411 and the paired runtime are independently qualified.Runtime qualification
000cda8refreshesscripts/runtime-image/production-contract.jsonand the production-contract workflow to the reviewed runtime candidateca84550. The Dockerfile now copiesjudges/, installsclawscan-decisions-judge, and runs its tests during the image build. The publish smoke also runsclawscan-decisions-judge --help. The locked-SDK qualification and both native image builds pass.Not done / follow-ups
.agents/skills/autoreviewcould not run locally. The Codex engine rejected the ClawRouter provider config, and the Claude engine's OAuth session had expired. Changes were self-reviewed.OPENAI_API_KEYand was not dispatched from this PR.