Evalgate

An evaluator that hasn't been evaluated is decoration.

Evalgate is a quality gate for LLM applications: a proxy that accounts for tokens and cost, and a statistical comparison that decides in CI whether a change makes the system worse. Unlike other gates, it publishes its own error rates, measured.

Status. The proxy, accounting, telemetry, pricing, evaluation engine and gate are done and verified (phases F0 to F6, see docs/ROADMAP.md). The meta-evaluation (F7) is measured and published with three of its four goals below threshold; no threshold was lowered. See Results.

What it does

• Proxy compatible with OpenAI's /v1/chat/completions, streaming and non-streaming. Adopting it means changing one environment variable. It records tokens, cost and one OpenTelemetry trace per call, including when the client drops the connection.
• Evaluation engine that scores a system case by case over a fixed set, with LLM judgements recorded and reused.
• Gate that compares two reports with a paired bootstrap (and Holm correction when there are more than three blocking metrics) and returns an exit code for CI.

It is not an observability platform: no UI, no annotation, no users (ADR-004). The gate can be used on its own, with reports produced by any tool.

Installation

Requires Python 3.12 and uv. Not published on PyPI.

git clone https://github.com/samuvm/evalgate.git
cd evalgate
uv sync --locked
uv run evalgate --help

Usage

Proxy

uv run evalgate serve --port 8080 --upstream http://localhost:11434/v1
OPENAI_BASE_URL=http://127.0.0.1:8080/v1 python your_app.py

The application does not change. With --clickhouse <DSN> traces are stored in ClickHouse (uv run evalgate migrate creates the schema); if the store goes down, requests keep being served.

Gate in CI

uv run evalgate eval compare base.json head.json --gate gate.yaml --out result.json

Exit code	Meaning
0	The change can go in
1	Blocked: some blocking metric got significantly worse
2	The reports cannot be compared

Inputs:

• Two reports in the eval-report.schema.json format. Each metric points with raw_path to its per-case values. Both reports must contain the same cases.
• gate.yaml with the metrics and their role:

  metrics:
    - { id: answer_correct, blocking: true }
    - { id: error_rate, blocking: true, higher_is_better: false }
    - { id: length, blocking: false }

• --goals (optional): a YAML file with the comparacion block (resamples, seed, confidence level). Defaults to docs/GOALS.yaml.

Bundled example

# The gate blocking a regression, with sample reports
uv run evalgate eval compare evals/suites/compare-fixture/base.json \
  evals/suites/compare-fixture/head-regresion.json \
  --gate evals/suites/compare-fixture/gate.yaml --out /tmp/result.json

# This repository going through its own gate, without calling any model
uv run evalgate eval self-gate

# The example RAG application behind the proxy (needs Ollama on localhost:11434)
make up
python examples/rag-app/generate_corpus.py
OPENAI_BASE_URL=http://127.0.0.1:8080/v1 python examples/rag-app/rag_app.py --limit 3
make down

Metrics

Metric	What it counts	Blocks
answer_correct	The answer contains the reference and none of the excluded answers	yes
context_recall	Share of the relevant documents that were retrieved	yes
abstencion_incorrecta	The system said "not in the documents" when the answer was there	yes
faithfulness	An LLM judge says the answer is grounded in the context	no (retired)
juez_ilegible	The judge's verdict could not be parsed	no

faithfulness is computed and published, but it does not decide: its judge does not reach the required agreement with human labels. The engine ships RAG metrics today; for other systems, the gate accepts any report in the contract format.

Results

Every number in this section is generated by make report from evals/reports/. The full detail, in Spanish, is in docs/NUMEROS.md.

The gate, measured

Measured by running the system many times with noise and comparing each run against a fixed baseline:

Goal	Measured	Threshold	Met?	Detail
G-JUDGE-KAPPA	0.47	>= 0.60	no	95% CI [0.27, 0.65] · 150 human labels · raw agreement 0.86
G-GATE-FP	0.102	<= 0.05	no	Wilson 95% [0.044, 0.218] · 5 blocks in 49 decisions against base-00
G-GATE-MDE	10.50	<= 8.00	no	percentage points, on answer_correct · 200 cases · 2450 pairs from 50 runs
G-GATE-POWER	1.00	>= 0.90	yes	minimum over degradations of size ≥ 15.75 pp · minimum over all 4: 0.02

In practice: comparing one run against another with that much noise, the gate blocks too often and cannot tell small drops apart, but it always blocks the two large degradations. With deterministic or recorded outputs that noise does not exist.

The rest of the system

Last phase closed in green: F6, 2026-10-08 (evals/reports/gate-F6.json).

Goal	What it measures	Value
G-ADOPTION	Adoption through one environment variable, no code changes (diff lines)	lineas_de_diff=0, n_lineas=8
G-CACHE-HIT	Marginal cost of re-running the suite	0.00
G-CACHE-INVAL	The judge cache key invalidates exactly what it should	ratio=1, n=63
G-DETERMINISM	Determinism of eval compare	ratio=1, n=3
G-DISCONNECT	Accounting when the client disconnects	ratio=1, n=217
G-FUNC-COV	Public functions without a unit test (plus property tests in pricing/ and stats/)	0
G-LAT-PROXY	p95 proxy overhead, non-streaming (ms)	6.97
G-LAT-TTFT	p95 time-to-first-token overhead, streaming (ms)	3.80
G-LINE-COV	Line coverage of the testable packages (%)	99.35
G-MUTATION	Killed mutants in the TDD packages (%)	88.58
G-OPENAI-CONTRACT	Compatibility with OpenAI's published schema	ratio=1.00, n=7
G-OTEL-CONTRACT	Spans conforming to the pinned OTel GenAI contract	ratio=1, n=25
G-PRICE-REPRO	Reproducibility of historical cost reports (EUR difference)	0.00
G-TOKENS-EXACT	Exact token accounting, non-streaming	ratio=1, n=5
G-TOKENS-STREAM	Token accounting, streaming	delta_con_usage=0
G-TRACE-0	Zero spans lost with a healthy store	perdidos=0, rps=50, servidas=6000
G-TRACE-DEGRADE	Graceful degradation with the store down (p95 delta, ms)	0.08

Design

• A trace never brings down or slows down a request. A bounded queue that drops, and counts, when full.
• Accounting always closes. Client cut off, tokens paid, tokens counted.
• No price in the code. Dated, immutable price tables; fixing a price means publishing a new one.
• No threshold in the gate. Every decision parameter comes from docs/GOALS.yaml.
• Paired comparison or an error. Reports with different cases are not compared.
• No metric calls a model, and the judge is never the model that generates.

Each rule has an automatic checker (docs/RULES.md); design decisions are recorded in docs/adr/. Project documentation is in Spanish.

Limitations

• Tested only against Ollama locally. No paid provider, no Bedrock.
• No reconciliation against a real invoice: cost is what a public price table declares.
• The evaluation corpus is synthetic by construction: it measures whether the gate detects degradations, not the quality of a real domain.
• The CI workflow (.github/workflows/gate.yml) is written but not enabled.

Development

make lint typecheck   # ruff and mypy --strict
make test-fast        # unit, property and contract tests
make test-int         # integration against ClickHouse (needs Docker)
make report           # regenerates the tables in this README

Path	Contents
src/evalgate/proxy/	Proxy and stream accounting
src/evalgate/pricing/, telemetry/	Dated prices; OTel traces and ClickHouse
src/evalgate/evals/, gate/, stats/	Metrics, cached judges, decision and statistics
examples/rag-app/	Example application; imports nothing from the package
evals/	Cases, suites and the reports every number comes from
docs/	Roadmap, goals, rules, ADRs and gate evidence

make gate-fast and make done are the gate each phase closes with. They read local state (.gate/) and a held-out test suite that is kept private so it cannot be tuned against, so they are not reproducible outside the development machine; the tests and the rule checkers are.

License

Apache-2.0.