Skip to content

fix: enforce security and experiment-validity gates - #4

Draft
aliengineering-byte wants to merge 1 commit into
mainfrom
fix/experiment-validity-gates
Draft

fix: enforce security and experiment-validity gates#4
aliengineering-byte wants to merge 1 commit into
mainfrom
fix/experiment-validity-gates

Conversation

@aliengineering-byte

Copy link
Copy Markdown
Owner

Summary

Hardens the restricted AST interpreter and closes experimental-validity leaks so the repository can support a bounded real-model pilot after the remaining contract blockers are resolved. This starts from audited commit ae50191591ed84f70f595efd7fc3a6bc97f1791b.

No paid model calls, empirical model results, release/tag, package publication, or paper-readiness claim is included.

Security fix

  • Routes BinOp and AugAssign through one bounded implementation.
  • Preflights exponentiation/integer growth, sequence and string concatenation/repetition, and allocation-prone string formatting.
  • Applies integer/string/container/nesting/item bounds on inputs, literals, assignments, unary/subscript/call outputs, and final results.
  • Adds quick-fail regressions for x **= 5000, oversized augmented sequences, alternative binary paths, and generated bounded-power equivalence tests.

Experimental validity

  • Removes _FaultAwareController, _known_corruption, and all injected_faults packet labels.
  • Keeps fault truth/events/schedules only in evaluator sidecars; prompt-injection text still reaches the model as untrusted evidence.
  • Adds immutable, hash-verified schedules keyed only by global seed, example, verifier, condition, and step.
  • Caches one initial candidate and generates one maximal trajectory per condition; direct, no-feedback, verify-once/repair-once, fixed, accepted-first, verifier-best-trajectory, and VCER replay it offline.
  • Makes status-only feedback primary and counterexamples an explicit bandwidth ablation.
  • Marks headline work BLOCKED: VRR-Guard and VRR-Stop are required contract baselines but are not approximated or implemented here.

Accounting, outputs, and analysis

  • Adds normalized per-call usage with raw provider usage, cached/reasoning tokens, estimated/provider flags, optional cost, complete-request estimates, verifier runtime, total-token budgets, and honest one-call overshoot.
  • Returns answer=null for every unverified terminal while retaining the last candidate.
  • Uses null for undefined rates and unavailable ECE/AURC; adds normalized cost per correct, raw transitions, task-clustered paired bootstrap, and Holm adjustment.
  • Requires explicit non-zero selector accounting for Best-of-N and candidate-independent, charged upfront evidence.

Executable/docs alignment

  • Renames strict-JSON .yaml examples/configs to .json; canonical counts are 12 math + 8 code.
  • Adds OpenAI-compatible CLI config plus a real localhost fake-server integration test.
  • Pins all third-party Actions to reviewed full commit SHAs and enforces measured 75% coverage.
  • Adds the two specified 2026 preprints to prior art/references, retires the broad paper title, and adds a blocked draft v0.2 amendment with the exact provisional pilot criterion.

Verification

  • ruff format --check .: pass
  • ruff check .: pass
  • strict mypy src: pass
  • 144 tests: pass
  • branch coverage: 76.82% (75% gate)
  • wheel + sdist build: pass
  • isolated wheel + sdist install and demo: pass
  • deterministic two-run canonical smoke comparison: pass
  • all-nine-fault smoke: 1,540 rows / 180 schedules / zero serialized evidence-label matches
  • clean-clone lint/type/test/coverage gate: pass
  • Gitleaks 8.30.1, all 9 commits: no leaks

Hosted CI and the hosted secret scan must be green before any release. Dependabot PRs #1-#3 were inspected; their major upgrades are deliberately not folded into this security/validity change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant