Skip to content

fix(review): central pipeline reliability — hypothesis coverage toolchain + Strix submodule materialization - #656

Merged
seonghobae merged 9 commits into
mainfrom
claude/contextualwisdomlab-audit-governance-fb7470
Jul 30, 2026
Merged

fix(review): central pipeline reliability — hypothesis coverage toolchain + Strix submodule materialization#656
seonghobae merged 9 commits into
mainfrom
claude/contextualwisdomlab-audit-governance-fb7470

Conversation

@seonghobae

@seonghobae seonghobae commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Two central review-pipeline reliability fixes, both verified against the full .github suite.

1. Add hypothesis to the coverage-evidence trusted toolchain

The central coverage-evidence sandbox (the OpenCode approval gate) installs a fixed offline Python test toolchain from requirements-opencode-review-ci-hashes.txt and runs each target repo's pytest tests. Repos whose suites include always-on Hypothesis property tests fail at collection:

ERROR collecting tests/fuzz/test_fuzz_properties.py
E   ModuleNotFoundError: No module named 'hypothesis'

coverage-evidence failure → OpenCode posts REQUEST_CHANGES → those PRs can never be approved. This adds hypothesis (MPL-2.0, permissive; already a declared test dep across the org) to requirements-opencode-review-ci.txt and regenerates the hash lock. main does not carry it yet (verified). The 3→7 free-model expansion originally on this branch was dropped as redundant when main gained it via a sibling PR.

2. Skip submodule gitlinks in full-tree PR-head materialization (Strix)

build_pull_request_head_tree_scope_dir in strix_quick_gate.sh enumerates the whole PR-head tree with git ls-tree -r --full-tree and fails closed on any non-blob entry. A git submodule pointer (gitlink) lists as 160000 commit, so any repository containing a submodule fails Strix whenever a PR takes the full-head-context path (e.g. a Dockerfile/compose change makes pull_request_changes_require_full_head_context true) — even though the submodule is unchanged and carries no scannable blob content here.

Fix: skip gitlink entries (mode 160000 / type commit) during full-tree materialization, mirroring the exclusion the changed-file scope path already applies to submodule pointers. Observed on ContextualWisdomLab/gyeot#13 (changed server/Dockerfile in a repo with services/contextual-orchestrator and services/fast-mlsirm submodules).

New regression run_full_head_scope_skips_gitlink_case: a Dockerfile-change PR (full-head context) in a repo with an unchanged submodule gitlink — fails closed (exit 2) without the fix, passes with it.

Safety / scope

  • Both are review-pipeline-only; neither changes scripts/ci domain logic, so the 100% coverage/docstring gates on scripts/ci are unaffected.
  • The gitlink skip only adds a 160000/commit short-circuit before the existing non-blob fail-closed; every other tree entry is handled exactly as before.
  • Hash-pinned install discipline preserved (regenerated with the recorded compile command).

Verification

  • Full .github pytest suite: 710 passed.
  • Full strix_quick_gate shell suite: green (exit 0), including the new gitlink regression case.

Impact

Unblocks OpenCode approval for every hypothesis-using repo's PR queue, and stops Strix from spuriously failing closed on every submodule-bearing repo whose PR touches a container manifest.


Generated by Claude Code

The central coverage-evidence sandbox installs a trusted offline Python test
toolchain (coverage, interrogate, pytest, pytest-cov) and runs each target
repo's 'pytest tests'. Repos with always-on Hypothesis property tests
(contextual-orchestrator tests/fuzz/, fast-mlsirm, codec-carver, ...) fail
collection with 'ModuleNotFoundError: No module named hypothesis' -> pytest
exit 2 -> coverage-evidence FAIL -> OpenCode posts REQUEST_CHANGES and can
never approve, permanently blocking those repos' PR queues.

Add hypothesis (MPL-2.0, permissive test tool) to the toolchain so the sandbox
can run those suites. .github's own tests do not import hypothesis, so its
coverage-evidence and 100%-coverage gate on scripts/ci are unaffected.
Regenerated the hash lock via ./scripts/ci/compile_opencode_review_lock.sh
(hypothesis==6.163.0 + attrs, sortedcontainers).

Evidence: contextual-orchestrator PR #75 coverage-evidence run 30470910566
attempts 1 and 2 both failed at 'ImportError: No module named hypothesis' while
the repo's own 'Full unit and contract suite' passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wtuYFp4E22QnEU1bFMhsr
@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@seonghobae, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 3 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 7a1a7de5-6696-4659-bc37-022704ad1f01

📥 Commits

Reviewing files that changed from the base of the PR and between ea79ee9 and 9f19e08.

📒 Files selected for processing (4)
  • requirements-opencode-review-ci-hashes.txt
  • requirements-opencode-review-ci.txt
  • scripts/ci/strix_quick_gate.sh
  • scripts/ci/test_strix_quick_gate.sh

Comment @coderabbitai help to get the list of available commands.

Copy link
Copy Markdown
Contributor Author

Status: code-complete, blocked only on transient reviewer capacity

All of this PR's own required checks are green: coverage-evidence ✅, noema-review ✅, strix ✅, CodeQL (python/actions) ✅, Bandit ✅, pip-audit ✅, osv-scan ✅, Semgrep OSS ✅. The fix (adding hash-pinned hypothesis to the coverage-evidence trusted toolchain) is validated: I traced the sandbox build — it installs requirements-opencode-review-ci plus each repo's own hash-pinned requirements, so hypothesis (a cross-repo CI-only property-test tool) belongs in the always-installed central toolchain, while genuine runtime deps (e.g. fast-mlsirm's numpy) keep flowing through each repo's own requirements/ci.txt. Scope is correct and complete.

The only thing gating merge is opencode-review reporting "did not produce approval evidence" — the shared GitHub Models pool is in a transient capacity-exhaustion window (amplified by many concurrent org PRs). This is an uncontrollable external factor, not a code defect, and it self-heals on a later capacity window. A monitoring backstop will re-run the review cycle on a fresh window rather than compete for exhausted capacity now.

Coordination notes for other governance actors:

  • Do not force-merge / credential-bypass this PR. The maintainer explicitly chose "self-heal, no bypass."
  • Do not re-implement — the change is correct and complete as scoped above.
  • This PR is the keystone: once merged it unblocks coverage-evidence for every hypothesis-using repo (e.g. contextual-orchestrator#75, whose coverage-evidence currently fails deterministically on ModuleNotFoundError: No module named 'hypothesis').

Generated by Claude Code

Copy link
Copy Markdown
Contributor Author

Fix validated on the updated head — only transient model-pool capacity remains

Updated this branch onto the recovered main (now carrying the free-Zen reviewers from #655/#659 and the content-based coverage-lock materializer from #657). On the new head b7ec1bb:

  • coverage-evidence = success ✅ — the exact failure this PR fixes is resolved. The materializer now installs the base locks and the sandbox imports hypothesis, so hypothesis-using suites run instead of erroring at collection.
  • opencode-review = failure only because OPENCODE_MODEL_POOL_OUTCOME=exhausted (run 30504230876) — i.e. the review model pool, including the newly-added free-Zen fallbacks, is transiently saturated by the org-wide concurrent re-review load after recovery. This is an uncontrollable external capacity factor, not a code defect and not a coverage-evidence gap.

So the code is complete and its own gate is green; the remaining wait is purely a transient model-capacity window that self-heals as the re-review backlog drains. No force-merge / bypass (per the maintainer's "self-heal, no bypass" decision); a monitoring backstop will re-check for a fresh capacity window rather than thrash re-triggers and deepen the exhaustion.


Generated by Claude Code

seonghobae and others added 4 commits July 30, 2026 10:36
The org-wide merge queue keeps stalling on opencode-review
OPENCODE_MODEL_POOL_OUTCOME=exhausted: the free anonymous Zen pool for
public repos was only 3 models (nemotron-3-ultra-free,
deepseek-v4-flash-free, north-mini-code-free), so concurrent org PRs
saturate it and reviews fall through to the rate-limited paid providers.

Double the free public-repo pool by adding the other 4 models the Zen
catalog currently promotes as free-tier, tried after the 3 proven ones and
before any paid github-models/openai/openrouter fallback:

  big-pickle, laguna-s-2.1-free, ling-3.0-flash-free, mimo-v2.5-free

Exact ids verified against four independent sources that agree on the
slugs and the free-tier set: the OpenCode Zen docs source
(anomalyco/opencode packages/web/src/content/docs/zen.mdx), openclaw's
provider catalog (which lists exactly these seven as the "currently
promoted free-tier rows"), NousResearch/hermes-agent, and CodeWhale. All
share the same https://opencode.ai/zen/v1 endpoint and @ai-sdk/openai-
compatible driver already used by the pool. Each is added with
tool_call: true (required by the review agent) and the same 256k/64k
limit as deepseek/north; #662's free-provider 600s failover means a
weaker candidate is skipped rather than stalling the run.

Kept private repositories on paid providers only (the free anonymous
models' data terms still forbid private code), and updated the
test_opencode_agent_contract.py pool + tool_call assertions in lockstep.

Verification: 115 opencode tests pass
(tests/test_opencode_agent_contract.py 37, test_opencode_workflow_shell_syntax.py 5,
plus model-pool/reasoning-effort/security/context/docker suites 73);
workflow YAML re-parses.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wtuYFp4E22QnEU1bFMhsr
@seonghobae seonghobae changed the title fix(review): add hypothesis to coverage-evidence toolchain so property-test repos can be approved fix(review): expand free Zen reviewer pool (3→7) + add hypothesis to coverage-evidence toolchain Jul 30, 2026
…omlab-audit-governance-fb7470

# Conflicts:
#	.github/workflows/opencode-review-dispatch.yml
#	tests/test_opencode_agent_contract.py
@seonghobae seonghobae changed the title fix(review): expand free Zen reviewer pool (3→7) + add hypothesis to coverage-evidence toolchain fix(review): add hypothesis to the coverage-evidence trusted toolchain Jul 30, 2026
build_pull_request_head_tree_scope_dir enumerates the entire PR-head tree with
`git ls-tree -r --full-tree` and fails closed on any entry whose object type is
not `blob`. A git submodule pointer (gitlink) lists as `160000 commit`, so any
repository containing a submodule fails Strix whenever a PR takes the
full-head-context path (e.g. a Dockerfile/compose change makes
pull_request_changes_require_full_head_context true), even though the submodule
is unchanged and carries no scannable blob content in this repository.

Skip gitlink entries (mode 160000 / type commit) during full-tree
materialization, mirroring the exclusion the changed-file scope path already
applies to submodule pointers, instead of failing closed on a legitimately
non-blob tree entry.

Regression: run_full_head_scope_skips_gitlink_case exercises a Dockerfile-change
PR (full-head context) in a repo with an unchanged submodule gitlink -- it fails
closed (exit 2) without the skip and passes with it. Full strix gate suite green.

Observed on gyeot#13 (changed server/Dockerfile in a repo with the
services/contextual-orchestrator and services/fast-mlsirm submodules).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wtuYFp4E22QnEU1bFMhsr
@seonghobae seonghobae changed the title fix(review): add hypothesis to the coverage-evidence trusted toolchain fix(review): central pipeline reliability — hypothesis coverage toolchain + Strix submodule materialization Jul 30, 2026
@seonghobae
seonghobae merged commit 2cc4bf9 into main Jul 30, 2026
46 of 47 checks passed
@seonghobae
seonghobae deleted the claude/contextualwisdomlab-audit-governance-fb7470 branch July 30, 2026 07:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants