Skip to content

fix(tooling): harden project regression checks - #17

Merged
meaningforger merged 1 commit into
mainfrom
codex/regression-hardening
Sep 26, 2026
Merged

meaningforger merged 1 commit into
mainfrom
codex/regression-hardening

Conversation

@meaningforger

Copy link
Copy Markdown
Contributor

Summary

Harden the existing RFC-0090 compile and metric-result tooling without expanding its scope:

  • Reject numeric filters that lose precision through the current public float64 decoder, including scalar/array YAML and JSON inputs. Check original tokens before YAML decoding; reject ambiguous YAML numeric forms. Public REST/MCP decoding remains unchanged.
  • Reject more than nine fractional-second digits instead of silently truncating expected or observed values.
  • Prevent JSON/JUnit outputs from aliasing suite/config inputs. Explicit overwrite now requires a recognizable Metis report of the requested format; arbitrary models, source files, malformed reports, and symlinks are not overwrite targets.
  • Preserve registered stable execution error codes and caller actions, with distinct safe categories for timeout, cancellation, limits, busy, schema, and access failures. Keep raw messages, details, endpoints, and result values out of JSON and JUnit.
  • Add regression coverage and update the shipped contract and RFC safety requirements.

Validation

  • make check — passed
  • make docs-check, make fmt-check, git diff --check — passed
  • CGO_ENABLED=1 go test -tags duckdb ./cmd/metis ./app/tooling/regression -count=1 — passed
  • Focused race tests for regression tooling and project CLI — passed
  • GORACE=atexit_sleep_ms=0 CGO_ENABLED=1 go test -race ./app/tooling/regression ./cmd/metis -count=1 — passed

The initial full-package race run with default exit sleep hit the existing stdio subprocess test's one-second termination window (signal: terminated). It did not report a data race. Disabling the detector's exit sleep lets that full run pass; no stdio code or test was changed.

Compatibility

No new commands, analytical assertions, public query types, or benchmark changes. Previously accepted lossy/ambiguous numeric inputs now fail before execution. --overwrite remains available for existing Metis reports but no longer authorizes replacing unrelated files. Error-report fields gain safe diagnostic values without changing schema version or outcome counting.

@meaningforger
meaningforger merged commit 5446a84 into main Sep 26, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant