[Patrick-Erichsen]

Why

ClawHub's final verdict should come from the scanners' evidence alone. The current clawhub judge (Decisions API) and its overview read the artifact itself. That lets the judge reach its own conclusions about code the scanners already examined, and it puts untrusted artifact text directly into the judge's prompt.

What changes

• ClawHub profiles: clawhub (skills and plugins) and clawhub-aig now judge with native Codex gpt-6.1-sol (high reasoning) from scanner results only. The overview (gpt-6-luna) also works from evidence only.
  • Skills are judged from SkillSpector and A.I.G reports.
  • Plugins are judged from aig-mcp reports only.
• evidenceOnly setting: new optional judge.evidenceOnly / overview.evidenceOnly profile setting, off by default. When on:
  • The command's workspace has no artifact/ copy.
  • metadata.json keeps only the target kind and id; there is no file manifest, target input or host path.
  • {{ target.files }} and the artifact-reading ClawHub parity prompt fail before the command starts.
  • A --judge / --overview override keeps the general workspace.
• New prompt placeholders:
  • {{ scanner_results }} carries every scanner's status, error and unchanged raw report, so failed or partial coverage stays visible.
  • {{ run_context }} carries the target kind and id plus the --context JSON.
• Untrusted scanner text: the prompts treat everything inside scanner reports as untrusted. Missing or partial coverage counts as uncertainty, never as evidence of safety.
• Publisher trust: only a publisher verified by ClawHub in the job context can make an @openclaw package trusted by default. A package name, README or manifest cannot, and neither can a claim quoted in scanner results.
• Decisions judge: the generic Decisions harness stays available through --judge; its tests pass that harness explicitly.
• Budgets: the judge keeps 20 minutes and the overview 5 minutes. There are no new CLI flags.
• Runtime image: unchanged. The runtime tree is the same, and the image already ships codex-cli 0.164.0.

Breaking: judge.result keeps its typed fields but no longer carries the Decisions decision record.

Codex tools: verified status and known limitation

Checked against codex-cli 0.164.0's real Responses requests, using a loopback stub:

• Removed: the shell/exec tools, image viewing, web search and the goal tools. A forced exec_command, shell or local_shell call is rejected as unsupported call and runs nothing.
• Offered but inert: functions.exec (code mode). With code_mode_host disabled it returns code-mode host is disabled. Even when enabled, apply_patch is rejected by the read-only sandbox, no MCP server is configured, and the JavaScript isolate has no require, Deno or process.
• Known limitation — still offered: collaboration.* (spawning other agents), wait and request_user_input. No supported 0.164 setting removes them:
  • agents.max_threads=0 is rejected at startup ("must be at least 1").
  • agents.max_depth=0 is accepted, but a forced spawn_agent call still starts an agent.
  • A spawned agent gets the same restricted tools and the same evidence-only workspace: no shell, file, web or image tools and no artifact text. It can only add model calls within the judge's 20-minute budget.
  • The prompt forbids spawning or asking the user, but that is an instruction, not enforcement.
  • Monitoring caveat: Codex's --json event stream does not show spawn_agent calls, so "no tool calls in events" does not prove no agent was spawned.
  • Removing these tools entirely would take a small direct Responses API helper with an empty tool list. That is left as a follow-up.

Before / after

Test	main e2d64e1	this PR
TestClawHubProfilesJudgeSkillsFromScannerEvidenceOnly (clawhub, clawhub-aig)	FAIL	PASS
TestClawHubProfileJudgesPluginsFromAIGMCPEvidenceOnly	FAIL	PASS
TestClawHubEvidenceJudgeNeverStartsOnScannerFailure	PASS	PASS
TestEvidenceOnlyWorkspaceHoldsNoArtifact (absolute target input omitted)	n/a	PASS (FAIL before 9163076)
Publisher claim stays inside the untrusted block; trust needs verified context	n/a	PASS (FAIL with the old prompt line)

The new tests check:

• the actual prompt, arguments and workspace contain no artifact/ directory and no artifact text;
• each scanner report is identical to the original JSON;
• hostile scanner text and a fake publisher claim appear only inside the untrusted block;
• the plugin prompt contains only the aig-mcp report.

Real CLI against the stub, with real Codex 0.164 (each request's tools/additional_tools list was captured; the stub records requests only, so it has no attachable artifact beyond these fixtures):

• Before: main's Decisions judge request contained the artifact text, and main's overview prompt told the model to read artifact/.
• After: for skills and plugins, the judge and overview requests contain no artifact text, scanner results are present, and both steps complete.

Validation

• Rebased once onto bc69d8f (feat: give plugin scanner commands a 30-minute budget #151, plugin scanner 30-minute budget); the only conflict was the CHANGELOG, and git range-diff shows no other change.
• go vet ./... passes.
• go test -p 1 -count=1 ./... passes all 10 packages.
• make docs-site passes.
• Autoreview (--engine codex --model gpt-6.1-sol --thinking high --mode branch --base origin/main) on 9163076: scoped-clean, "patch is correct (0.86)". The rebased head f720ac4 has the same patch apart from CHANGELOG context.
• ClawSweeper review of f720ac4 raised two P3 items, fixed in 7349f04: the partial-report docs and AGENTS.md now separate artifact-capable judges from evidence-only judges, and both evidence-only rejection errors name the fix. Autoreview on 7349f04: scoped-clean, "patch is correct (0.83)". The full suite with -p 1, go vet and make docs-site pass.
• An earlier unconstrained go test ./... run under heavy host load failed TestDefaultCommandCancellationIsNotSuccess and TestBenchmarkCLIEmitsCanceledPriorCaseEvidence. Both pass alone, and the benchmark test fails the same way on main.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: needs maintainer review before merge.

What this changes

This PR makes the bundled ClawHub judge and overview consume scanner evidence without receiving a copy of the scanned artifact.

Example: Scan a plugin named probe-plugin with the clawhub profile.

• Before: The Decisions judge receives artifact contents alongside aig-mcp evidence.
• After: Codex receives aig-mcp evidence and run context, with no artifact copy in its workspace.

Review scores

Measure	Result	What it means
Overall readiness	🦐 gold shrimp (3/6)	The scoped implementation is correct by inspection and both earlier findings are fixed; independently inspectable runtime evidence remains limited.
Proof confidence	🦐 gold shrimp (3/6)	The ordinary contributor proof gate does not apply to this collaborator-authored PR; its real-Codex loopback verification is described in the body, while inspectable tests use a replacement Codex executable.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Product

Kind: Feature · Worth it: Yes · Fix scope: Complete
User problem: ClawHub adjudication currently receives artifact contents and can independently inspect code beyond the scanner evidence.
Reason: The feature owner explicitly chose scanner-only adjudication, and the patch implements that boundary without adding ClawHub-specific public CLI flags.

Merge readiness

✅ Ready for maintainer review

Keep open: this maintainer-authored PR adds behavior absent from main, resolves both prior findings, and has no remaining actionable code findings.

Priority: P2
Reviewed head: 7349f04e8e3b0ce563094af1ef5af85cbee24e1c

Before merge

None.

Findings

None.

Tests

• Missing end-to-end proof: No attached request trace independently demonstrates the reported real-Codex tool rejection and evidence-only requests through the shipped CLI.

Agent review details

How this fits together

Profiles configure scanner adapters and external judge commands; the shared runner prepares their workspaces, renders evidence, and records the verdict and optional advisory overview.

flowchart TD
  A[Skill or plugin] --> B[Scanner adapters]
  B --> C[Raw scanner reports]
  C --> D[Evidence only workspace]
  D --> E[Codex judge]
  E --> F[Recorded verdict]
  F --> G[Advisory overview]
  C --> G

Loading

Technical review

Best possible solution:

Land the scoped evidence-only profile change with the documented result-contract break and general judge overrides preserved.

Do we have a high-confidence way to reproduce the issue?

This is an intentional capability change: current-main source copies the artifact into judge workspaces, while the proposed mode omits it.

Is this the best way to solve the issue?

The implementation uses the existing shared harness and profile boundary, preserves raw evidence, and keeps artifact-capable overrides available.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against bc69d8fb4aa0.

Provenance checked

• Bundled judge and overview changes intended behavior with a stated reason (22d2f69: The Decisions judge supplied typed verdicts while a separate non-gating Codex overview supplied prose.)
• Shared judge workspace keeps the original intent (fa01a95: All profiles received one general workspace layout rather than profile-name-specific runner behavior.)
• Partial SkillSpector evidence changes intended behavior with a stated reason (a5301cc: Usable partial reports reached an independent judge with raw findings, gaps, provenance, and artifact access rather than failing a coverage gate.)
• Codex sandbox selection changes intended behavior with a stated reason (18d9157: The outer Docker boundary allowed artifact inspection when a nested read-only sandbox prevented it.)

Testing

Proof path: in-process harness.

Security

None.

Evidence

What I checked:

• Repository policy applied: Read the complete root AGENTS.md; no nested AGENTS.md, maintainer notes, or pr-simplify skill were present. The review followed the raw-evidence, judge-agnostic runner, partial-report, and advisory-overview contracts. No helper, second reviewer, build, or test was run. (AGENTS.md:65, 7349f04e8e3b)
• Main still uses artifact-backed judging: The pinned main profile invokes clawscan-decisions-judge, and its workspace preparation copies the target into artifact/. The introduced diff adds the evidence-only mode through the existing shared harness. (internal/runner/runner.go:2036, 7349f04e8e3b)
• Prior findings resolved: Compared the earlier reviewed head with this head: partial-report documentation and AGENTS.md now distinguish artifact-capable judges from evidence-only judges, and both rejection messages provide a repair. (docs/judge.md:324, 7349f04e8e3b)
• Overrides preserve the general interface: Evidence-only settings apply to the profile's own commands; explicit judge and overview overrides retain their existing workspace behavior. (internal/profiles/resolver.go:941, 7349f04e8e3b)
• Accepted direction and compatibility tradeoff: The collaborator-authored body explicitly chooses scanner-only adjudication, documents removal of the Decisions decision record, retains the Decisions harness through --judge, and acknowledges the remaining restricted collaboration tools. (7349f04e8e3b)
• Compatibility boundary: The added configuration booleans default to false, existing project commands remain authoritative, and stored judge results remain opaque interface values. The change does not rewrite saved configuration or historical artifacts; existing compatibility fixtures cover legacy result preservation. (cmd/clawscan/compat_test.go:28, 7349f04e8e3b)

Likely related people:

• Patrick Erichsen: Raw commit 22d2f69 adds internal/runner/overview.go:65 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: 22d2f694bb81; files: internal/runner/overview.go)

Preflight source

Source unknown. Source comparison at the observed PR snapshot only. Completion is self-reported; body, policy, tests, execution, and report bytes are not verified.

Labels

Label changes:

• add merge-risk: 🚨 compatibility: The built-in profiles change judge implementation and omit the Decisions decision record; the maintainer-authored body explicitly accepts and documents this consumer-facing break.
• add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.
• remove status: ⏳ waiting on author: Current PR status label is status: 👀 ready for maintainer look.

Label justifications:

• P2: This is a deliberate, bounded change to ClawHub adjudication rather than an urgent runtime repair.
• merge-risk: 🚨 compatibility: The built-in profiles change judge implementation and omit the Decisions decision record; the maintainer-authored body explicitly accepts and documents this consumer-facing break.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Optionally attach redacted request and rejection traces from the reported real-Codex skill and plugin runs.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

History

Review history (1 earlier review cycle)

• reviewed 2026-11-06T20:01:36.512Z sha f720ac4 :: needs changes before merge. :: [P3] Update the partial-report contract for evidence-only profiles | [P3] Give an actionable repair in evidence-only rejection messages

Reviewed November 6, 2026, 3:27 PM ET / 20:27 UTC (Revision 2).

[Patrick-Erichsen]

@clawsweeper re-review

[clawsweeper]

🦞👀
Exact review queued.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/42288740650
• Updated: 2026-11-06T20:29:33.315Z