Skip to content

feat(tooling): runtime metric regression suites and JUnit reports - #15

Merged
meaningforger merged 1 commit into
mainfrom
codex/project-runtime-regression
Sep 26, 2026
Merged

meaningforger merged 1 commit into
mainfrom
codex/project-runtime-regression

Conversation

@meaningforger

Copy link
Copy Markdown
Contributor

Summary

  • Continue RFC-0090 after feat: add offline project compile regression suites #14 with metis project test --mode runtime, calling the production query_metrics service against deployment-owned placement and backends.
  • Require declared external/live fixtures and complete schema/row expectations. Compare exact numeric strings without float conversion, preserve duplicate multiplicity, distinguish null, and support deterministic keyed tolerances and ordered assertions.
  • Tighten private execution ceilings without mutating or relaxing deployment policy. Incomplete results, limits, timeouts, and execution outages cannot pass as expected semantic failures.
  • Add private atomic JSON/JUnit output, bounded value-free diagnostics, fixture provenance, signal cancellation, and conflicting-output checks.
  • Update README, shipped contract, RFC status, and independently reviewed Doris/ClickHouse fixture examples.

Validation

  • make check
  • make fmt-check
  • CGO_ENABLED=1 go test -tags duckdb ./cmd/metis ./app/tooling/regression ./app/bootstrap -count=1
  • Default tests cover strict schemas, exact numbers, duplicate/null semantics, tolerances and key uniqueness, report privacy, output safety, and private execution ceilings.
  • Production DuckDB CLI integration covers success, mismatch, expected/unexpected semantic errors, and byte-limit failure. Doris/ClickHouse examples are bootstrap/schema-validated offline; no external warehouse fixture was provisioned during local verification.

Boundaries

This is the standalone local-unrestricted metric-result slice only. Comparison/attribution suites and host-managed policy assertions remain proposed; no fixture DDL, baseline approval, raw query override, or benchmark changes are included.

@meaningforger
meaningforger merged commit c6efa55 into main Sep 26, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant