Skip to content

feat: add isolated US liquidity scanner v2 lab - #2

Draft
Vikkrantpol wants to merge 100 commits into
mainfrom
experiment/us-liquidity-scanner-v2
Draft

Vikkrantpol wants to merge 100 commits into
mainfrom
experiment/us-liquidity-scanner-v2

Conversation

@Vikkrantpol

@Vikkrantpol Vikkrantpol commented Jul 31, 2026 •

Copy link
Copy Markdown
Owner

US v2 upgrade validation COMPLETE — awaiting merge decision

Final HEAD: 4d1e6c5 (branch experiment/us-liquidity-scanner-v2, base main). No further development phase is planned; this PR is merge-ready pending human decision. Draft retained; not merged; not marked ready.

Validation evidence

  • Session 1 (completed session 2026-08-06): run 20260807-084027-783789-3dfb — 205 scanned / 118 qualified / 87 rejected / 0 errors, provenance_valid=true, benchmark ok.
  • Session 2 (completed session 2026-08-07): run 20260807-235426-316691-9e78 — 205 scanned / 118 qualified / 87 rejected / 0 errors, provenance_valid=true, benchmark ok.
  • Candidate-set digest identical across both sessions: ba041a2e9063121a.
  • Full-universe shadow (isolated runtime runtime/us_v2_full_shadow/, run 20260808-042934-560490-4d60): 7121 universe records / 3561 scanned / 1594 qualified / 1937 rejected / 30 expected data-availability warnings; benchmark_status ok.
  • Exact warning categories (30): 29 insufficient_history (young listings, completed sessions < 120) + 1 no_data (no daily bars — delisted). No schema / authorization / provenance / classification-contract / database / benchmark / unexpected-exception failures.
  • Final offline suite: 1344 passed / 0 failed under US_V2_OFFLINE_TEST=1 in an isolated temp runtime root; live runtime state byte-identical pre/post (sha256-verified); India isolation manifest unchanged; git diff --check clean.
  • Final merge-readiness audit: focused offline smoke — compileall OK; 120 tests passed (test_scan_plan / test_phase9_gating / test_cli); India isolation OK; git diff --check clean.

Normal shell operator workflow (documented in us_v2/docs/runbook.md)

./start_us_v2.sh build-universe
./start_us_v2.sh scan
./start_us_v2.sh summary --top 30
# optional:
./start_us_v2.sh report --formats json,csv,md
./start_us_v2.sh export --format csv

A plan-less scan is the normal daily mode (require_scan_plan=false); --plan PATH --yes scans are only for the exact-500 Phase 9 validation runs. Runtime lives under runtime/us_v2/; fresh --yes resets it; US_V2_RUNTIME_ROOT points at an isolated shadow root. Validation manifests/control baskets are upgrade-validation assets, not ordinary daily requirements.

Not performed

  • No Optimus migration was performed.
  • No deployment was performed.
  • No GitHub Actions were used or queried; no hosted CI rerun.

Phase 9 — Session 2 validation (completed session 2026-08-07)

What changed (us_v2/config.py, us_v2/config_schema.py, us_v2/scan_plan.py, us_v2/cli_actions.py)

  • Reject-list reconciliation (universe.reconcile_reject_list, default false) — closed the Session-1 divergence where the manifest reject list carried 6 symbols (CBU/CYH/GEL/NECB/UPS/UTHR) that the authoritative universe classifies as qualified candidates. When enabled, a symbol the universe qualifies is deterministically dropped from the manifest reject list at plan creation (scan_plan.py), so the plan reject list and the live universe can never disagree. Enabled in the private phase9_config.json; default config stays false.
  • No other production code changed during Session 2; the exact-500 chain, thresholds, and scoring are identical to Session 1.

Session 1 vs Session 2 comparison (same 500-symbol chain, same private config)

  • Session 1: run 20260807-084027-783789-3dfb, completed session 2026-08-06, scanned 205 / qualified 118 / rejected 87 / errors 0, benchmark_status ok, plan ff5b98c7 (single-use, consumed once), provider success rate 1.0.
  • Session 2: run 20260807-235426-316691-9e78, completed session 2026-08-07, scanned 205 / qualified 118 / rejected 87 / errors 0, benchmark_status ok, plan 319a7322 (single-use stage5-phase9-session2, bound to HEAD f5cabaa, consumed once), provider success rate 1.0.
  • Candidate set: identical — candidate_set_digest=ba041a2e9063121a in both; universe rebuild 500 records / 205 qualified / 270 rejected (10 via the session-history gate) / 25 classification_unknown; reject list reconciled to 156 symbols with zero qualified overlap.
  • Rejection set: identical (87/87). Scores move with market data between sessions (e.g. WRBY RS3M 87.8→32.7, QDEL price 16.32→12.28, INSM/ABNB rallied) — no anomalies; 111 rank changes are price-driven.
  • Doctor (--persisted-scan-readiness): 83/86 pass, 3 warn, 0 fail, ready=true; all plan/binding/control/exact-500/relaibility checks pass.

Full-universe shadow scan (isolated root runtime/us_v2_full_shadow/)

  • Fresh full-NYSE/NASDAQ/AMEX universe build: 7121 records (qualified 3561 / rejected 2738 / classification_unknown 822), provider reliability 1.0, errors 0, cache checkpointed/resumable.
  • Persisted shadow run 20260808-042934-560490-4d60: scanned 3561 / qualified 1594 / rejected 1937 / errors 30 (completed_with_warnings — all 30 are expected insufficient_history / 1 no_data on recent/delisted listings, no data corruption). Top: CCK 84.71, DELL 84.63, ALAB 84.11. Summary, report (json/csv/md), export csv, and history all captured inside the shadow root.

Final offline suite (run once, isolated temp runtime root)

  • 1344 passed / 0 failed under US_V2_OFFLINE_TEST=1 (639.55s) with US_V2_RUNTIME_ROOT pointed at a fresh temp root; live runtime state byte-identical pre/post (sha256-verified), no tracked runtime mutations.

Status

  • Head 9478369, base main, pushed to experiment/us-liquidity-scanner-v2. Draft retained; no merge; no ready.
  • One non-code commit in this phase: chore: gitignore the isolated full-universe shadow runtime root (the new shadow root was outside the existing runtime/us_v2/ ignore rule).

Phase 8O — discovery-integrity & live-accounting closure

What changed (us_v2/control_resolution.py, us_v2/cli_actions.py)

  • Immutable official contract: DISCOVERY_BUDGET_CONTRACT_V1 is now validated by exact equality, so an internally-consistent-but-mutated contract (e.g. ADR cap 24→25 / foreign 64→63 with the total held at 88 and digests recomputed) fails closed (resolution_budget_contract_immutability_mismatch).
  • Official vs effective budgets. The clamped per-category budget the resolution actually ran with is recorded separately as effective_discovery_budgets (+ effective_discovery_budgets_digest); both new fields are bound into _resolution_digest. The validator checks the effective digest, increasing caps ending at the effective hard cap (never above the official cap), and that only the terminal override may be a non-official cap.
  • Candidate-probe cross-validation. candidate_probe_counts hard cap / progressive caps / budget version must match the effective budget (resolution_budget_probe_{cap,progressive_caps,version}_mismatch); progressive_caps_seen is validated for membership / duplicates / ordering / magnitude (resolution_budget_progressive_caps_seen_{membership,duplicate,out_of_order,beyond_logical}).
  • Exhaust flags derived exclusively from stop_reason (resolution_budget_flag_stop_reason_mismatch): quota_reached, budget_exhausted, pool_exhausted and both-flags cases are mutually consistent by construction.
  • Non-checkpoint overrides now emit valid terminal checkpoints: 20 → [8,16,20], 10 → [8,10], 3 → [3], 32 → [8,16,32] (the override is no longer silently dropped from the cap list).
  • Exactly-cap-sized pool: a pool whose size equals the hard cap that misses quota now deterministically reports budget_exhausted (previously fell through to quota_reached).
  • Truthful live accounting: retries derive as max(0, attempts-1) from a single authoritative attempt count, and zero-attempt rate-limited / circuit-rejected results stay at zero provider attempts (no forged max(1, attempts)). The resolution CLI probe now returns structured attempt/retry/rate/circuit results taken from ReliableProvider.call().
  • Fail-closed persistence: a resolution shortfall persists the global ProviderReliability outcomes before returning, so provider health on the failure path stays auditable.

Validation

  • New tests/us_v2/test_phase8o_discovery_integrity.py (27 tests): each of the fifteen gaps was proven by a test that failed against the Phase 8N head and passes after this closure; effective-budget artifact embedding + digest binding included. High-risk resolution/accounting tests re-run 20x with zero flakes.
  • Full suite: 1286 passed (1259 baseline + 27) under US_V2_OFFLINE_TEST=1 on py3.11 (845.90s) and py3.12 (556.99s); Phase 8N/8I/8K/8M/8L/8G/8H and all Phase 9x/CLI/provider-reliability suites green; py_compile and git diff --check clean; runtime evidence gitignored/untracked.
  • Hosted CI us_v2_ci success on head d332ff6 for all four jobs (py3.11 + py3.12, two runs each, ~8m30s) — nothing cancelled/superseded.

Status

  • Head d332ff6, base main, pushed to experiment/us-liquidity-scanner-v2. Draft retained; no merge; no ready.
  • Phase 9A history is accurate: it was started and each attempt stopped fail-closed at a different gate — attempt 1 at control verification, attempt 2 at static strong capacity, attempt 3 at the foreign 8-probe boundary; no Phase 9A session persisted a successful live resolution/plan.
  • Stop marker PHASE8O_DISCOVERY_INTEGRITY_CLOSURE_PUSHED_AWAITING_GITHUB_AUDIT.

---## Phase 8N — category-specific progressive ADR/foreign discovery budgets

What changed

  • Budget contract (us_v2/control_resolution.py): the under-calibrated shared --max-probes 8 default that caused Phase 9A attempt 3 to fail closed (need 2 have 1, probed 8 of pool 5306) is replaced by control-discovery-budget-v1: ADR quota 3 / caps [8,16,24] / hard cap 24; foreign equity quota 2 / caps [8,16,32,64] / hard cap 64; total cap 88.
  • Per-category progressive probing (CandidateProber.resolve): each category gets one deterministic tier-aware ordering (_tiered_ordered: tier 0 = already-hinted records first, tier 1 = ordinary common equity; priority only, never exclusion or auto-acceptance), probed once left-to-right. A candidate is never probed twice; progressive caps are checkpoints, not restarts; retries never count as new logical candidates; budget never bleeds between categories. Exact stop reasons quota_reached / budget_exhausted / pool_exhausted with budget/pool flags.
  • CLI (us_v2/cli_actions.py): --adr-max-probes (default 24) and --foreign-max-probes (default 64); deprecated shared --max-probes alias maps min(v,24)/min(v,64) and is rejected when combined with either category flag; overrides below quota, above the per-category contract cap, or above the 88 total are rejected. A fail-closed resolution now writes a privacy-safe control_resolution_failure_diagnostic.json (aggregates only — no symbols/names/countries/raw metadata).
  • Fail-closed validator (_validate_discovery_budget): re-verifies the bound contract digest, official quota/caps/total equality, strictly-increasing caps, final cap == hard cap, cap >= quota, and per-category logical/selected/attempt/retry/stop-reason/pool consistency; combined logical <= total cap. The budget fields are bound into _resolution_digest, and the operation snapshot logical_request_count must equal the summed category logical candidates.
  • Attempt-3 diagnosis: forensic baseline (phase8n/discovery_budget_baseline.json) proves the failure was a budget-boundary artifact, not provider unavailability — foreign quota 2, pool 5306, probed 8, accepted 1, 7 not_foreign_equity, pool_remaining_after_budget 5298. Offline reproduction (phase8n/attempt3_aggregate_reproduction.json) replays the identical head state and shows the new budget continues the same ordering past probe 8 (no claim about any fixed post-probe-8 position).

Validation

  • New tests/us_v2/test_phase8n_discovery_budget.py (32 tests): budget defaults/clamps/rejections, tiering reorders but never excludes, progressive success across caps, budget exhaustion, no-duplicate probes, category-local accounting, retry/logical separation, pool exhaustion, artifact embedding + digest binding, validator mutations, exact foreign acceptance rules (EQUITY + non-US + no ADR only).
  • Full suite: 1259 passed under US_V2_OFFLINE_TEST=1 (1227 baseline + 32); Phase 8N high-risk tests re-run 20x with zero flakes; targeted 199-test bundle green; compileall and git diff --check clean; all runtime evidence gitignored/untracked.
  • Hosted CI us_v2_ci success on the current PR head (py3.11 + py3.12, re-run after the initial push-run was superseded/cancelled).

Status

  • Head 96475f5, base main, pushed to experiment/us-liquidity-scanner-v2. Draft retained; no merge; no ready.
  • Attempt history correction: attempt 1 stopped fail-closed at verification; attempt 2 at static strong capacity; attempt 3 at the 8-probe foreign discovery boundary (this Phase 8N closure).
  • Phase 9A remains NOT started; prior attempts stay fail-closed records. Stop marker PHASE8N_DISCOVERY_BUDGET_CLOSURE_PUSHED_AWAITING_GITHUB_AUDIT.

Phase 8M — corroborated compact unit/right evidence + real-chain re-root

What changed

  • Unit/right corroboration (us_v2/listing.py): enrich_cross_row_instrument_evidence now also promotes a compact U- or R-ending canonical symbol to a high-confidence corroborated_compact_unit / corroborated_compact_right hint when the same listing set contains a canonical common/class-share record for the base symbol, the name is anchored (Unit(s)/Right(s) with a bounded tail: period, expiring <date>, each consisting of, to receive, to purchase), the name carries no fund/ADR/ADS conflict markers, and the normalized issuer stem matches the base exactly or via a bounded prefix of enumerated class letters (a-e; plus Tangible Equity for units only). Same deterministic/idempotent/never-demote contract as 8L; weak name_keyword unit/right hints are never authorized.
  • Validator rederivation (us_v2/listing.py rederive_corroborated_warrant_evidence → generalized, us_v2/control_listing.py _cross_row_evidence_errors): both validators independently rederive warrant/unit/right assignments from immutable source fields with stable codes control_listing_corroborated_{warrant,unit,right}_rule (should-carry but wrong/missing) and _drift (claims rule without corroborating base).
  • Classifier/resolution authorization (us_v2/universe.py, us_v2/control_resolution.py): _STRONG_LISTING_RULES_BY_KIND and ALLOWED_HINT_RULES_BY_CATEGORY now authorize the corroborated compact rules for unit/right; static control_capacity_preflight covers all four strong categories.
  • Real-chain re-root (Part 4 decision): the on-disk P9 had been overwritten by a synthetic test fixture; the intact frozen CLA (7075 real stock rows, 3667 ETF rows) was preserved byte-identical under runtime/us_v2/validation/stage5/evidence/frozen_phase9a_attempt2/ and the evidence chain was regenerated from those records under current code via the offline CLI path — snapshot e914d1cbb3a21877 (self-authenticates), stock listing 0d6b6cb62c23ccf5 / canonical b67033f1000cad3e, ETF listing 8df78650063f539a / canonical 4ddffc4a03c760e8; load_phase9_stock_listing_audit_v1 ok and CLA valid=true usable=true, both snapshot-bound. Decision recorded in phase8m/reroot_digest_scheme_decision.json.
  • Evidence sufficiency (phase8m/unit_right_evidence_audit.json, phase8m/evidence_sufficiency_decision.json): capacity on the regenerated real chain = unit 104 (>= 3) via corroborated_compact_unit (base_U only; base_UN/base_UT rejected), right 55 (>= 2) via corroborated_compact_right (base_R only; base_RT rejected). Strong pools now preferred 381 / warrant 240 / unit 104 / right 55 — quota satisfied from real listing evidence alone.

Validation

  • New/updated tests (334 passed locally across the targeted suites): corroborated unit/right positives/negatives, bounded-prefix issuer matching, digest sensitivity, validator drift codes, classifier/resolution authorization, capacity gating.
  • compileall clean, git diff --check clean, all runtime evidence confirmed gitignored and untracked.
  • Hosted CI us_v2_ci success on push run and PR run for new head (py3.11 + py3.12).

Status

  • Head 193fe69, base main, pushed to experiment/us-liquidity-scanner-v2. Draft retained; no merge.
  • Phase 9A remains NOT started; the prior attempt stays PHASE9A_NO_GO fail-closed.

Phase 8L — corroborated compact-warrant evidence (warrant quota fix)

What changed

  • Cross-row enrichment stage (us_v2/listing.py): enrich_cross_row_instrument_evidence (wired into ingest_listing) promotes a compact W-ending canonical symbol (pure alphanumeric, e.g. AACIW) to a high-confidence corroborated_compact_warrant warrant hint only when the same listing set also contains a canonical common-equity or class-share record for the base symbol whose normalized issuer stem matches. Deterministic (plain symbol map, list-order visited), idempotent, never demotes an existing rule, and only upgrades weak name_keyword/default evidence. Never promotes WS/WT/delimited .W forms (those stay symbol_suffix).
  • Independent rederivation for validators (us_v2/listing.py::rederive_corroborated_warrant_evidence): computes the expected corroborated assignment purely from canonical_symbol/raw_name/instrument_hint, never from the persisted instrument_hint_rule, so validators check artifacts against a rederived expectation.
  • Validator wiring (us_v2/control_listing.py): both validate_control_listing_audit_v1 and validate_phase9_stock_listing_audit_v1 call _corroborated_warrant_errors, which emits stable failure codes control_listing_corroborated_warrant_rule (should-carry but wrong/missing) and control_listing_corroborated_warrant_drift (claims the rule without corroborating base evidence).
  • Classifier rule authorization (us_v2/universe.py): _strong_listing_kind now requires the hint rule to be explicitly authorized for the category (_STRONG_LISTING_RULES_BY_KIND: preferred→preferred_symbol, warrant→symbol_suffix+corroborated_compact_warrant, unit/right→symbol_suffix, class_share→class_share_slash, index→unsupported). A name_keyword warrant hint can never be strong listing evidence even if marked high-confidence.
  • Resolution rule authorization (us_v2/control_resolution.py): STRONG_SYMBOL_RULES replaced by ALLOWED_HINT_RULES_BY_CATEGORY; _strong_candidates only accepts authorized rules per category; validate_resolved_controls_v1 gained per-row resolution_unauthorized_rule.
  • Static control-capacity preflight (us_v2/control_resolution.py, us_v2/cli_actions.py): control_capacity_preflight (called at the start of resolve_controls and by the CLI before build_provider) computes the deterministic provider-free candidate pool per strong category and fails closed with ControlResolutionError before any provider budget is spent when a category cannot reach its quota (ADR/foreign excluded — provider-backed).
  • Frozen attempt-2 evidence: forensic audit of control_listing_audit.json shows all 296 warrant-hint records are name_keyword/low; 288 compact-W; fully corroborated capacity 240 (grammar 287 ok, base present 243, base common 241 / adr 2, issuer exact 111 / prefix 129). Rejections: 44 base absent, 2 ADR base, 1 issuer mismatch, 1 grammar. Audit cached at runtime/us_v2/validation/stage5/phase8l/warrant_evidence_audit.json (aggregate only, no symbols).
  • Offline verification: post-enrichment frozen data yields exactly 240 corroborated_compact_warrant records (rule distribution corroborated_compact_warrant: 240), warrant strong pool = 240, rederived expectation == actual corroborated set; the preflight fails closed on the frozen data for unit/right (static control-capacity shortfall: right need 2 have 0, unit need 3 have 0) because those remain name_keyword — only warrant is remediated and other strong sources are unchanged.

Validation

  • New suite tests/us_v2/test_phase8l_warrant_remediation.py (26 tests): corroboration predicate positives/negatives (anchored vs non-anchored names, base present/absent, class-share base, issuer prefix match, no base, ADR base, existing rule never demoted, existing name_keyword untouched, delimited WS untouched, unknown base not promoted), validator rederivation (rule-must-carry, drift, tamper), classifier authorization (authorized rule accepted, unjustified name_keyword fails closed, unknown rule never strong), resolution category rules, and static control-capacity preflight (passes with full quota fixtures, rejects provider-backed paid categories).
  • Two pre-existing tests updated for the tighter contract: test_cli.py::test_build_universe_listing_evidence_from_manifest now supplies the authorized preferred_symbol rule (the old test relied on a rule-less high-confidence hint being strong — exactly what 8L closes), and test_control_verification.py::test_verdict_stays_provider_independent supplies preferred_symbol for the strong-listing path (mirroring its sibling test_final_classifier_conflict_surfaced).
  • Full offline suite: 1216 passed (1195 main + 9 stress + 12 lock-subprocess) — local run green; compileall, pip check-equivalent gates, India isolation manifest check, runtime isolation guard, and git diff --check all clean.
  • Hosted CI us_v2_ci success on both the push run and the PR run for the new head (py3.11 + py3.12).

Status

  • Head: 8eef47d, base main, pushed to experiment/us-liquidity-scanner-v2. Draft retained; no merge.
  • Phase 9A remains NOT started (prior attempt stopped fail-closed at control verification, PHASE9A_NO_GO); no live Phase 9 work has run against the private config.

Phase 8K — source-producer closure (authorization chain complete)

What changed

  • Single shared provider + immutable source snapshot (us_v2/cli_actions.py, us_v2/provider_reliability.py): control-listing-audit now fetches the ordinary stock screener AND the authoritative ETF/ETN/ETP screener through one ReliableProvider (single circuit breaker, --max-attempts / --rate-limit) and binds a single immutable source-operation snapshot (control_listing_provider_snapshot.json). Both stock-listing-audit-v1 and control-listing-audit-v1 bind that snapshot's digest. Fail-closed: a source failure or empty source writes no canonical artifact.
  • Resolution provider snapshot + atomic revalidation (us_v2/control_resolution.py, us_v2/cli_actions.py): resolve-controls persists control_resolution_provider_snapshot.json (attempt/retry/success/empty/rate-limited/circuit-breaker/terminal counts), binds its digest via finalize_resolution_artifact after the snapshot exists, then atomically writes + revalidates. CandidateProber records category-local accounting (logical_candidates_attempted, provider_attempt_count, retry_count, metadata_success_count, provider_unavailable_count, selected_count, rejections).
  • Cross-source validator (us_v2/control_resolution.py): validate_resolved_controls_v1 now takes the stock + control listing audits and the resolution operation snapshot and verifies source-digest equality (resolution_source_digest_mismatch), snapshot-digest binding (resolution_operation_snapshot_mismatch), source membership of every selected row (stock vs authoritative ETF section), and anchor collisions.
  • Fail-closed chain loader (us_v2/phase9_loaders.py): load_phase9_resolved_controls_v1 requires the immutable control-listing audit, the resolution operation snapshot (operation_type=resolve-controls / operation_family=control_resolution), and the stock listing audit, then runs the full cross-source validation. Distinct error codes: phase9_operation_snapshot_missing / _invalid / _binding / phase9_resolution_invalid. load_phase9_control_report_v3 checks verification snapshot operation_type/operation_family/operation_id identity.
  • 30-symbol reconciliation (us_v2/control_verification_v3.py): validate_control_verification_report_v3 reconciles the exact 30-symbol set across the manifest control_symbols, the resolved basket, and the report rows (CONTROL_REPORT_ERROR_MISSING_CONTROL / CONTROL_REPORT_ERROR_COUNT).
  • Atomic write + reload revalidation (us_v2/cli_actions.py): sample_universe writes tickers_500.json + sample_manifest.json and verify-controls writes the verification report through _write_atomic_revalidated, re-validating on reload.
  • Offline CI guarantee (tests/us_v2/conftest.py, .github/workflows/us_v2_ci.yml): autouse network guard blocks socket, requests, urllib and yfinance entrypoints when US_V2_OFFLINE_TEST=1; the hosted CI job is relabeled "Offline US-v2 tests" and sets the guard on the pytest step.

Validation

  • Full offline suite: 1190 passed (1172 pre-8K + 18 new focused tests) under US_V2_OFFLINE_TEST=1 on hosted CI — py3.11 440s, py3.12 468s. Local guarded run 584s, unguarded 606s.
  • New suite tests/us_v2/test_phase8k_source_chain.py (18 tests): cross-source validation positives/negatives, loader fail-closed cases (missing/tampered/updated snapshot, drifted listing), 30-symbol report reconciliation, offline producer CLI E2E.
  • Stress: test_phase8k_source_chain 20x, test_phase8i_v3_chain 20x, test_control_verification_v3_negatives 20x, doctor suites 10x, test_phase9_stress 10x — all green, zero flakes.
  • Pipeline gates clean on hosted CI (py3.11 + py3.12): pip check, compileall, pytest exit 0, India isolation manifest check, runtime isolation guard, whitespace/conflict-marker check. Local git diff --check clean.

Status

  • Head: 8eef47d (Phase 8L on top), base main, pushed to experiment/us-liquidity-scanner-v2. Hosted CI us_v2_ci success on both the push run and the PR run for the new head.
  • Phase 8J (this PR's prior section) = consumer migration; Phase 8K (this section) = source-producer closure. Together the Phase 9 authorization chain is complete end to end.
  • Phase 9A (500-symbol live validation run) is NOT started: the one prior attempt stopped fail-closed at control verification (no successful session, PHASE9A_NO_GO — see the Phase 9A section below). No live Phase 9 work has ever run; no runtime state or scan has run against the private config. Draft retained; no merge.

Phase 8J — Phase 9 consumers migrated to the v3 chain (official-path hardening)

What changed

  • Canonical loader layer (us_v2/phase9_loaders.py, new): load_phase9_manifest_v3, load_phase9_resolved_controls_v1, load_phase9_control_listing_v1, and load_phase9_control_report_v3 — the single official path every Phase 9 consumer now routes through, with stable PHASE9_* error codes. The v3 report loader additionally binds the immutable provider-operation snapshot: it now fails closed on operation_id mismatch (report vs on-disk snapshot) and on logical_request_count != requested_control_count, so bounded retries can never inflate the logical control count.
  • Validator hardening (us_v2/control_verification_v3.py): added manifest-expectation cross-checks per control row (control_category, listing_source_kind, listing_hint + confidence/rule/evidence, expected_security_type, expected_classification_outcome), nonempty binding checks for every sealed binding field (control_report_missing_binding), and a timezone-aware generated_at check (control_report_invalid_timestamp). Both validate_control_verification_report_v3 and evaluate_control_verification_report_v3 enforce the new rules.
  • Wrapper result contract (us_v2/cli_actions.py): load_phase9_control_report now returns manifest / resolved_controls in its result (populated on both the fail and success paths), and _control_report_digests / _phase9_live_values / the strict doctor gate all consume the v3 digests only.
  • Test migration to v3: test_phase8g_remediation.py, test_control_verification_cli_v2.py, test_phase8h_pre_live_gate.py, test_doctor_readiness.py (build_phase9_ready_runtime binds a real reliability digest), test_phase8i_v3_chain.py (untrusted-control test no longer mutates expectations), and test_cli.py create-plan case now use stage5-manifest-v3 / resolved-controls-v1 / control-verification-v3 with live digests.
  • New negative suite (tests/us_v2/test_control_verification_v3_negatives.py, 34 tests): unknown statuses, incorrect top-level/trusted/all-trusted counts, all rederived count fields, edited row semantics with recomputed digests, manifest-expectation mismatch per field, duplicate/missing/extra rows, binding + timestamp negatives, and loader-level operation_id / logical_request_count binding failures.

Validation

  • Full offline suite: 1172 passed (1138 pre-migration + 34 new negative tests).
  • Focused critical suites run 20 consecutive iterations green (4020 runs, 0 failures).
  • Pipeline gates clean: compileall, git diff --check, no forbidden-path changes, no secrets staged.

Status

  • Head: ea93a38, pushed to experiment/us-liquidity-scanner-v2.
  • Hosted CI us_v2_ci success on both the push run and the PR run for the new head.
  • Phase 9A remains NOT started; no runtime state or scan has run against the private config. Draft retained; no merge.

Phase 8I — v3 control chain (production-aligned, tamper-evident)

What changed

  • New v3 modules: us_v2/control_listing.py (control-listing-audit-v1, combined control-source digest, authoritative ETF/ETN/ETP hint rule), us_v2/control_oracle.py (trust matrix judging the reconciled production classification, not the coarse provider label), us_v2/control_resolution.py (resolved-controls-v1, exact-30 quota contract, bounded fail-closed ADR/foreign discovery), and us_v2/control_verification_v3.py (self-authenticating control-verification-v3 report with valid vs Phase 9 ready).
  • Sampler v4 + manifest v3: sample_stage5_universe_v4 consumes the resolved controls as the authoritative cohort B; build_validation_manifest_v3 / validate_validation_manifest_v3 bind both listing sections, the combined control-source digest, the resolution digest, and re-derive every digest and the exact quota contract.
  • New CLI: control-listing-audit (with --stock-offline-file / --etf-offline-file offline repro) and resolve-controls, wired into cli.py help, the command map, and start_us_v2.sh.
  • Consumer parity verified: strict doctor readiness, plan-bound scan, plan-authorized dry run, persisted-scan preflight, create-scan-plan, validate-scan-plan, and explain-run all consume the canonical Phase 9 validator / fail-closed load_phase9_control_report.

Validation

  • Full offline suite: 1138 passed (1095 baseline + 43 new Phase 8I tests) — local 499s, hosted CI green on py3.11 and py3.12.
  • New suite test_phase8i_v3_chain.py (+ shared phase8i_fixtures.py): oracle status matrix, listing digest determinism/sensitivity, resolution determinism/tamper/shortfall, sampler v4 cohorts and guards, manifest v3 validate/tamper, verification v3 build/validate/evaluate/digest self-authentication, offline CLI, and end-to-end chain readiness.
  • Pipeline gates clean: pip check, compileall, pytest exit 0, India isolation manifest check, runtime isolation guard, git diff --check.

Status

  • Head: 20b1dbd, 6 commits added on top of the Phase 8H baseline.
  • Phase 9A remains NOT started after the fail-closed live session (see below); no runtime state or scan has run against the private config. Draft retained; no merge.

Phase 8H — pre-live gate correction (merged toward Phase 9A)

What changed

  • Canonical manifest gate before any provider work (us_v2/cli_actions.py): verify-controls now resolves the stage5 v2 manifest and runs the canonical validate_validation_manifest before telemetry().reset(), operation snapshot, build_provider, provider reliability load, and runtime evidence mutation. Stable failure codes: manifest_not_found, manifest_unreadable, manifest_not_object, manifest_invalid, manifest_ineligible, control_count_mismatch, control_membership_invalid. No absolute paths are printed; a failed gate creates no report, no operation snapshot, and no reliability file.
  • Single-use semantics corrected (us_v2/docs/runbook.md, us_v2/docs/stage5_phase8g_closure_report.md, --single-use help): the plan is reserved + consumed after canonical authorization and lock acquisition and before mutable persisted scan work; a preflight failure never consumes it, a dry run never consumes it, and once reservation succeeds a later scan failure does not restore or reuse the authorization. The earlier "consume after its first successful scan" wording is inaccurate and is removed.
  • New focused gate tests (tests/us_v2/test_phase8h_pre_live_gate.py, 12 tests): invalid-but-eligible, duplicate-control, semantic/digest-error, missing-manifest, and ineligible manifests all fail before build_provider (provider spy records zero calls, zero info fetches); no report/op-snapshot/reliability state is created on failure; corrected help wording is asserted; the valid-manifest path stays green (30/30 controls, report + snapshot + reliability persisted). Run 20 consecutive iterations green.

Validation

  • Full offline suite: 1095 passed (py3.11: 403s, py3.12: 460s) on hosted CI.
  • Pipeline gates all clean on both interpreters: pip check, compileall, pytest exit 0, India isolation manifest check (hashes unchanged), runtime isolation guard (no runtime/us_v2 files tracked), whitespace/conflict-marker check, set -eo pipefail throughout.
  • Local run identical: 1095 passed in 552s; git diff --check clean; india_isolation_manifest.py --check OK.

Status

  • Head: 49e782c, base 61031603, 7 commits ahead of the Phase 8F baseline db639dab.
  • Phase 9A (live listing audit, exact-500 sample, real control verification, universe build, reliability gate, single-use plan, strict doctor, plan-authorized dry run, one persisted 500-symbol scan, session-1 audit) has not been started yet — no runtime state or scan has been run against the private config.
  • Remains a draft; no merge.

Phase 9A — live session 1 attempt (fail-closed, NOT completed)

Phase 9A was started per the operating plan and stopped at the live control-verification gate:

  • Fresh runtime after backup (/tmp/us_v2_pre_phase9a_20260804_160958.tar.gz, sha256 e08d0d17fc49de68e1c160d9f54740b5c2fda54ff3cc2df39c5c57f4f7080273).
  • Private config runtime/us_v2/private/phase9_config.json (mode 600, provider=yfinance, strict_error_accounting=true, require_benchmark=true, bounded retries + 2 rps pacing): validate-config OK, doctor --scope config 6/6.
  • Live listing audit: canonical rows 7106 (≥500), listing_digest=a8822c46b61ade70, canonical_listing_digest=b442cebcb9e04f33.
  • Deterministic sample: --profile stage5-500 --size 500 --seed 20260801 produced byte-identical tickers_500.json across 3 runs (sha256 fc0fb306…), manifest stage5-manifest-v2 with 500 symbols (20 anchors / 30 controls / 450 random), validation_eligible=true, validate-sample-manifest = valid, listing digest verified.
  • Live verify-controls (real provider): 11 verified, 19 untrusted — 11 rejected_candidate, 8 conflicting_evidence, 0 provider_unavailable. all_controls_trusted=false, phase9_ready=false, report_valid=true, error_code=control_report_not_ready.
  • Per the Phase 9A plan, a bounded retry is permitted only when every untrusted control is provider_unavailable with no rejections/conflicts and no edits. Rejected/conflicting controls are present and the control set may not be silently replaced, so no retry was taken.
  • Result: PHASE9A_NO_GO — fail-closed at Part 10. No universe build, no plan, no scan, no dry run ran. Evidence preserved under runtime/us_v2/validation/stage5/phase9/ (all gitignored).
  • Post-attempt isolation: working tree clean, zero tracked files under runtime/us_v2.

Next action requires an operator decision (control set redesign / new deterministic manifest) before any Phase 9A session can resume.

Single shared provider fetches both control sources bound to one immutable
source snapshot; resolution binds its provider snapshot digest; cross-source
validator checks source digests, snapshot binding, membership and anchor
collisions; load_phase9_* loaders reject missing/invalid/mismatched snapshots
and drift; report reconciles the 30-symbol set; sample/verify write atomically
and revalidate.
18 new focused tests cover cross-source validation, loader fail-closed cases,
30-symbol report reconciliation and offline producer CLI E2E. conftest blocks
socket/requests/urllib/yfinance when US_V2_OFFLINE_TEST=1 and CI relabels the
job to Offline US-v2 tests with the guard set on the pytest step.
Cross-row enrichment promotes a compact W-ending canonical symbol to a
high-confidence warrant hint only when a canonical common/class-share
base record with a matching issuer stem exists in the same listing set
(rule corroborated_compact_warrant). Validators independently rederive
the assignment with stable failure codes. The universe classifier and
control resolution accept strong listing evidence only under explicitly
authorized rules, and resolve-controls runs a static control-capacity
preflight that fails closed before any provider budget is spent.

Frozen attempt-2 evidence: 240 compact-W warrants fully corroborated
(name_keyword/low -> corroborated_compact_warrant/high), sufficient to
fill the warrant quota without lowering the strong quota contract.
…universe on session history

Phase 9 exact-500 readiness fixes:
- universe: apply_session_history gate demotes qualified records with fewer
  completed sessions than min_history_sessions (indeterminate stays qualified)
- scan-plan: with universe.reconcile_reject_list the manifest reject list drops
  symbols the authoritative universe classifies as qualified, so weak
  name_keyword hints (e.g. 'unit' inside 'United'/'Community') no longer cause
  deterministic reject-list scan failures
- config: add universe.enforce_history_gate and universe.reconcile_reject_list
  keys + schema; cli exposes --no-history-gate
- doctor: classify ETFs from authoritative screener hint readiness fixes
- tests: history gate core + integration, reject reconciliation, doctor readiness
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant