Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions .github/workflows/regressions.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
name: SQLite regressions

on:
push:
pull_request:

permissions:
contents: read

jobs:
sqlite-regressions:
runs-on: ubuntu-24.04
timeout-minutes: 10
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
with:
fetch-depth: 2
persist-credentials: false
- uses: astral-sh/setup-uv@eb1897b8dc4b5d5bfe39a428a8f2304605e0983c # v7.0.0
with:
version: "0.11.19"
checksum: "7035608168e106375b36d0c818d537a889c51a8625fe7f8f7cad5e62b947c368"
enable-cache: false
- name: Install pinned Python
run: uv python install 3.13.7
- name: Check whitespace
run: git diff --check HEAD^ HEAD
- name: Check Python syntax
run: uv run --python 3.13.7 --no-project --no-sync --offline python -m compileall -q scripts tests
- name: Run SQLite regression suite
run: uv run --python 3.13.7 --no-project --no-sync --offline python -m unittest discover -s tests -v
23 changes: 23 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,29 @@ Only `raw` and non-gold tool-assisted runs should be considered for fair ranking

The July 3 result is a pilot evaluation, not a leaderboard claim.

## Regression tests and CI

The `SQLite regressions` workflow runs on every push and pull request with
Python 3.13.7 and uv 0.11.19. Actions are pinned to commit SHAs, and the Linux
uv archive is verified against its release SHA-256. After runtime setup, checks
run offline with no third-party Python packages or benchmark downloads.

Run the same suite from the repository root with the pinned runtime installed:

```sh
uv run --python 3.13.7 --no-project --no-sync --offline python -m compileall -q scripts tests
uv run --python 3.13.7 --no-project --no-sync --offline python -m unittest discover -s tests -v
```

The 12 tests execute the real SQLite evaluator and comparator against temporary
fixture databases. They cover denied mutations and unchanged database contents,
safe literals/comments/identifiers, instruction limits and connection cleanup,
ordered versus unordered results, duplicate counts, types, NULLs, and the
versioned evaluation/report contract. A failed test exits nonzero and fails CI.
These are local fixture regressions, not measured model accuracy or acceptance
of downloaded benchmarks. CI also checks Python syntax and commit whitespace;
the repository currently has no configured formatter, linter, or type checker.

## Documentation

- `docs/HARNESS.md` explains the run and evaluation flow.
Expand Down