diff --git a/.claude/skills/plan-creator/SKILL.md b/.claude/skills/plan-creator/SKILL.md new file mode 100644 index 000000000..00c28e489 --- /dev/null +++ b/.claude/skills/plan-creator/SKILL.md @@ -0,0 +1,163 @@ +--- +name: plan-creator +description: Create an implementation plan for a Kaapi feature, grounded in the codebase wiki (docs/wiki) and the blast-radius procedure (docs/wiki/domain-map.md). Use when the user wants an implementation plan, build plan, or execution plan for a feature, SRD, or task — the step between "spec agreed" and "start coding". +--- + +# Plan Creator + +Turn a feature spec into an ordered, file-level implementation plan that an +engineer can execute without re-deriving the design. The plan is grounded in two sources, in this order: + +1. **The wiki** (`docs/wiki/`) — what already exists: routes, tables, models, + services, external boundaries. Codebase knowledge comes from here, not from + memory or ad-hoc greps. +2. **The blast-radius procedure** (`docs/wiki/domain-map.md`) — what the change + touches beyond the surfaces the spec names. Every plan includes a completed + blast-radius table; a plan without one is not done. + +## Input — two modes, nothing else + +- **SRD mode:** the user supplies an SRD (a `features//SRD.md` path or + pasted content). The SRD is the spec; plan from it. +- **Standalone mode:** no SRD — the user's request itself is the spec. Plan + directly from it, and mark every inferred requirement as an assumption in + the plan's Open Questions section. + +Do not demand an SRD, hunt for one, or block on other documents. An SRD is +optional input, never a prerequisite. + +In both modes the spec supplies the *what*; the wiki and model files supply +the *shapes of what exists*. Never invent scope from the codebase, never +invent schema from the spec or memory. + +## Workflow + +1. **Read the input spec** end to end before opening anything else. + +2. **Load the wiki, narrowly.** Open `docs/wiki/INDEX.md`, then ONLY the + `docs/wiki/modules/*.md` page(s) for the domain(s) the feature touches. + Follow a module page's deep-dive link into `docs/architecture/*.md` only if + a genuine design question comes up; never bulk-load architecture docs. + +3. **Run the blast-radius procedure** from `docs/wiki/domain-map.md`: + - Name the primary entity(ies) the feature changes, in the map's vocabulary. + - Collect 1-hop and 2-hop `consumed by` surfaces — tables, logical + consumers, and the external consumers the map lists (Langfuse, the + frontend console, provider batch APIs, object storage). + - For every surface the spec does not address, ask the user: + in scope / deferred / out of scope. Never silently include or exclude. + - Record every decision in the plan's Blast Radius table. + +4. **Verify schema against code.** The module page maps each table to its + `models/*.py` file. Read the model files for every table the plan touches + before writing any column, field, or relationship into the plan. Before + planning a NEW table or config shape, confirm via `domain-map.md` + model + files that nothing existing fits — reuse beats invention. + +5. **Map the work onto Kaapi's layers.** Steps follow the dependency spine + from `.claude/conventions/`: model → migration → crud → service → route, + plus the Celery task if there is background work, then tests. Consult the + relevant `.claude/conventions/{model,crud,service,route,migration,celery}.md` + only to confirm a placement question, not wholesale. + +6. **Write the plan** using the template below. Every step names concrete + files (existing paths verified via the wiki page; new paths following the + module's existing layout) and states what changes in each. + +7. **Output** to `features//PLAN.md`, alongside the feature's `SRD.md` / + `PRD.md` if they exist — reuse the existing slug exactly; do not invent a + parallel folder. If no `features//` folder exists, create one with a + short kebab-case slug. The plan is the deliverable; this skill does not + execute it or hand it to any other skill or agent — an orchestrating skill + may invoke plan-creator and consume the `PLAN.md`, but that wiring lives + outside this skill. + +## Plan template + +```markdown +# — Implementation Plan + +Source spec: + +## Summary + +<2-4 sentences: what is being built and the shape of the change.> + +## Blast Radius + +Primary entities: + +| Surface | Hop | Impact | Decision | +|---|---|---|---| +| | 1 | | in scope / deferred / out of scope | + + + +## Steps + +### 1. : +- Files: `app/...` (change) / `app/...` (new) +- +- Depends on: + + + +## Migration + +
+ +## Tests + + + +## Open Questions + + +``` + +## Quality Checklist + +Load these as todos at the start; verify each before output: + +- [ ] Spec read first; every planned behavior traces to the spec or is flagged + in Open Questions. +- [ ] Wiki loaded before planning: `docs/wiki/INDEX.md` + the touched module + page(s); plan's paths, table names, and endpoint shapes cross-checked + against them. +- [ ] Blast radius run against `docs/wiki/domain-map.md`: 1-hop + 2-hop + consumers collected; every surface the spec skips confirmed with the + user; decisions recorded in the Blast Radius table, none silently + dropped; external consumers row(s) present. +- [ ] Schema derived from code: model files Read for every touched table; + new tables/shapes justified by nothing existing fitting. +- [ ] Steps in dependency-spine order (model → migration → crud → service → + route → Celery → tests), each naming concrete files. +- [ ] Kaapi conventions respected: `inserted_at`/`updated_at`, + `organization_id` + `project_id` on multi-tenant tables, snake_case, + naming matches existing code (grep before coining a new name). +- [ ] Output at `features//PLAN.md`, alongside existing SRD/PRD. +- [ ] No fact stated twice; empty sections deleted; no filler. +- [ ] After any revision, grep the plan for superseded names (fields, tables, + endpoints, step titles) — zero hits across every section. + +## Rules + +- **Wiki before greps.** Exploratory grepping is a fallback for what the wiki + doesn't cover, not the starting point. If the wiki page is missing or stale + for a touched module, say so — and note that the eventual PR must update it + (wiki maintenance rule). +- **A plan step is checkable.** Each step names files and the concrete change; + "update the service layer" is not a step. If you can't name the file, you + haven't planned it yet. +- **Reuse existing models and shapes** at the highest-level wrapper that fits. + A reviewer asking "can't we reuse X?" is a planning failure. +- **Plan the migration and the wiki update as steps**, not afterthoughts: a + schema change carries an Alembic migration step; any change to a module's + routes/tables/models/services carries a step updating that module's wiki + page (and `domain-map.md` if entities/edges changed) in the same PR. +- **No em dashes in the output plan.** Commas, periods, or parentheses. +- **Don't design in the plan what the spec already decided** — reference the + SRD section instead of restating it. The plan adds sequencing, file mapping, + and blast-radius decisions, not a second copy of the spec. diff --git a/features/llm-judge-adherence-to-prompt/SRD.md b/features/llm-judge-adherence-to-prompt/SRD.md new file mode 100644 index 000000000..6328d6dac --- /dev/null +++ b/features/llm-judge-adherence-to-prompt/SRD.md @@ -0,0 +1,174 @@ +# Adherence to Prompt SRD + +## Introduction & Purpose + +This SRD defines a native LLM-as-a-judge **Adherence to Prompt** score for Kaapi fast evaluations. It is the third metric in the native judge family, alongside the reference-based **Correctness** score and the knowledge-base grounding score. Each evaluated row already returns a cosine similarity score and (with the correctness judge) a correctness score; this feature adds an independent **Adherence to Prompt** score (0 to 1) plus a short reasoning string, produced by an LLM inside Kaapi with no manual Langfuse configuration. + +Today Kaapi can tell whether an answer is *close to* or *correct against* the ground truth, but not whether the answer actually *followed the assistant's own configured instructions*: the right language and tone, answering what it should and refusing what it shouldn't, using the configured fallback response when it does not know instead of inventing facts, and holding its instructions against prompt-injection attempts in the question. NGO eval teams have no signal for this without hand-building a bespoke judge in an external tool. + +Unlike the correctness judge, adherence is **not reference-based**: it does not read the ground-truth answer. It judges the generated answer against the **evaluated assistant's own prompt/instructions**, which are already carried by the run's config (`config_id` + `config_version`). + +The feature produces, per evaluated row: an adherence score, a reasoning string that cites which rubric dimension failed, both persisted by Kaapi and written to the row's Langfuse trace alongside the existing scores, plus an optional per-run judge configuration the team can send with the run. It ships as a new `POST /api/v2/evaluations` run trigger, a replica of the existing v1 trigger with adherence built in, so v1 clients are unaffected. + +- **Phase 1 (this release):** automatic adherence judging on fast evaluations only, exposed through a new `POST /api/v2/evaluations` endpoint (a replica of the v1 `POST /api/v1/evaluations` run trigger; v1 unchanged); zero-config default (built-in prompt + fallback model); a single composite 0 to 1 score covering all four checks; optional per-run `adherence_judge_config` (ad-hoc or saved reference); score, reasoning, per-row map, and cost persisted and synced to Langfuse. +- **Phase 2+ (deferred):** batch-mode judging; per-dimension sub-scores (a separate score per check); the weighted-average traffic-light verdict that consumes this and the sibling metrics; judge error/retry handling; confirmed performance budget and per-question cost guidance. + +Intent: adherence judging is automatic, zero-config, tailorable, explainable, and reversible, owned inside Kaapi so eval owners never open Langfuse. + +## Goals + +- Adherence judging runs natively inside Kaapi with zero manual Langfuse setup. +- The judge is prompt-based: it compares the generated answer against the evaluated assistant's configured instructions (not the ground truth) and returns a 0 to 1 score plus reasoning. +- The single composite score reflects four checks: language and tone match the prompt; answers what it should and refuses what it shouldn't; uses the configured fallback response when it does not know instead of inventing facts; resists prompt injection in the question. +- Judging is automatic: every scoreable row of a fast run is judged, with no flag or opt-in. +- It works out of the box from a built-in default prompt and a fallback model when the run carries no judge config. +- A project team can tailor the judge per run (model, model settings, prompt template) via an ad-hoc configuration or a saved config reference, with no deploy. +- Omitting the judge config reverts that run to the default prompt + fallback model. + +## Assumptions & Constraints + +- **Out of scope:** the weighted-average traffic-light verdict (a separate cross-metric feature consumes this score plus the correctness and knowledge-base scores); batch-mode judging (fast only); per-dimension sub-scores; retiring Langfuse (results keep syncing to it); robust judge error/retry handling. A failed or malformed judge call must not block the cosine or correctness scores or the run; the row's adherence is left unscoreable and the run still completes. +- **Trigger:** a new `POST /api/v2/evaluations` run trigger, a full replica of the v1 `POST /api/v1/evaluations` trigger, hosts this feature; the v1 endpoint is left untouched. Adherence judging is built into the v2 fast path (never a toggle); the v2 body adds one optional `adherence_judge_config` field that only tailors the judge. Only the POST run trigger is replicated to v2 — dataset upload, list runs, run status, and prompt-improve stay on v1. +- **Input source:** the assistant's prompt/instructions and configured fallback response are read from the run's own config (`config_id` + `config_version`); no ground-truth answer is sent to the adherence judge. +- **Limits:** fast eval is capped at `EVAL_FAST_MAX_UNIQUE_ROWS` (10) unique rows; the judge adds one model call per scoreable row within that cap. +- **Per-row independence:** one row's judge failure must not fail sibling rows or the sibling metrics. +- **Reuse:** no new tables. Adherence rides the existing `EvaluationRun.score` record (per-trace `scores` list + `summary_scores`) and a new durable per-row map column mirroring `per_item_scores`. The judge configuration reuses `LLMCallConfig` (`app/models/llm/request.py`): either a saved reference (`id` + `version`, resolved through the existing `config`/`config_version` flow) or an ad-hoc `blob` carrying completion params and an optional `prompt_template`. +- **Starting provider/model:** OpenAI, matching the fast-eval response/embedding path. The fallback model is a configurable default; a run's `adherence_judge_config` may override the model and its settings. +- **Pricing:** the judge adds one LLM completion per scoreable row (paid), tracked under an `adherence_judge` stage in `EvaluationRun.cost`. Per-question cost validation on a real org's assistant is a Phase 1 success check, not a build requirement. + +## Detailed Design (Execution Flow) + +The adherence judge slots into the fast-eval pipeline as a scoring step that runs after similarity is computed, in parallel to (and independent of) the correctness judge, so an adherence failure can never block the cosine or correctness scores. Each scoreable row's answer is judged against the assistant's own configured prompt and fallback response; the built-in rubric asks the judge to weigh language/tone match, correct answer-vs-refuse behavior, fallback usage instead of fabrication, and resistance to prompt injection in the question, and to return one composite score with reasoning that names the weakest dimension. + +### Judged fast-eval run + +--- + +![alt text](./assets/image.png) +System-level sequence: the eval owner, the Kaapi fast-eval pipeline, the evaluated assistant config, the LLM judge provider, and Langfuse. + +--- + +Each judged row's adherence score and reasoning are appended to that trace's `scores` list (reasoning carried in the score `comment`), and an `Adherence to Prompt` summary score is added to `EvaluationRun.score.summary_scores` next to `Cosine Similarity`. The durable `per_item_adherence` map is the source of truth for Langfuse resync; a failed Langfuse write is recoverable from it on resync. + +### Tailor the judge (per-run config) + +The run request may carry an optional `adherence_judge_config` (`LLMCallConfig`): a saved reference (`id` + `version`) is resolved to its stored blob at the judge step, while an ad-hoc `blob` is used directly. The blob's `completion` supplies the judge model + settings, and its `prompt_template` (when set) replaces the built-in adherence prompt; omitting `adherence_judge_config` uses the fallback model and built-in prompt. There is no per-project judge state: each run is judged exactly as configured in its own request, and durable, versioned tailoring lives in the existing saved-config flow the reference points at. This field is independent of the correctness judge's `judge_config`, so the two metrics carry their own prompts, models, and settings. + +## Functional Requirements (Testing) + +| ID | What (user-facing behavior) | Acceptance criteria | Status | +|----|-----------------------------|---------------------|--------| +| FR-1 | Every scoreable row gets an adherence score | After a fast run completes, each scoreable row's trace `scores` carries an `Adherence to Prompt` entry alongside `Cosine Similarity` | Not Started | +| FR-2 | Adherence is 0 to 1 with reasoning | Each `Adherence to Prompt` trace score has a numeric value in [0,1] and a non-empty reasoning `comment` naming the weakest rubric dimension | Not Started | +| FR-3 | Judged against the assistant prompt, not ground truth | The judge input contains the run config's instructions and fallback response and the generated answer, and does not contain the dataset ground-truth answer | Not Started | +| FR-4 | Rubric covers the four checks | The built-in prompt scores language/tone match, answer-vs-refuse correctness, fallback-instead-of-fabrication, and prompt-injection resistance; a row whose answer is hijacked by an injection in the question scores low with reasoning citing injection | Not Started | +| FR-5 | Zero-config default works | Running `POST /api/v2/evaluations` in fast mode without `adherence_judge_config` still scores every scoreable row using the fallback model + built-in prompt; no flag toggles judging | Not Started | +| FR-6 | Ad-hoc judge config honored | A run with `adherence_judge_config.blob` uses that blob's model, settings, and (when set) `prompt_template` for every judged row of that run | Not Started | +| FR-7 | Saved judge config honored | A run with `adherence_judge_config.id` + `version` resolves the stored config and judges with it; an unknown `id`/`version` fails the request with 404, not the run | Not Started | +| FR-8 | Config is per-run | `adherence_judge_config` affects only its own run; the next run without it is back on the default | Not Started | +| FR-9 | Invalid judge config rejected | An `adherence_judge_config` carrying both a saved reference and a `blob` (or neither) returns 422 before the run starts | Not Started | +| FR-10 | Summary score persisted | `EvaluationRun.score.summary_scores` contains an `Adherence to Prompt` entry after completion | Not Started | +| FR-11 | Per-row judge failure isolation | If a row's judge call fails or returns malformed output, that row's adherence is left unscoreable in `unscoreable`, its cosine and correctness scores are unaffected, and the run still completes | Not Started | +| FR-12 | Judge cost tracked | After a run, `EvaluationRun.cost` includes an `adherence_judge` stage with token counts and USD | Not Started | +| FR-13 | Tenant isolation | A saved config belonging to (org A, project A) is never resolvable from any other (org, project)'s run request | Not Started | +| FR-14 | v1 endpoint unchanged | `POST /api/v1/evaluations` accepts no `adherence_judge_config`, and a fast run through v1 produces no `Adherence to Prompt` score; its request/response contract is byte-for-byte unchanged | Not Started | + +## Endpoints + +One new endpoint: `POST /api/v2/evaluations`, a full replica of the v1 `POST /api/v1/evaluations` run trigger (same body and response), with adherence judging built into its fast path and one added optional `adherence_judge_config` field. The v1 endpoint and every other evaluations endpoint (dataset upload, list runs, run status, prompt-improve) are unchanged and stay on v1; adherence surfaces through the existing run-results shape. + +### `POST /api/v2/evaluations` (new, replica of v1 run trigger + adherence) + +Starts an evaluation run. The body replicates v1; `adherence_judge_config` is the only added field. In `fast` run_mode, each scoreable row is judged for adherence. + +**Request body:** + +| Field | Type | Required | Default | Description | +|-------|------|----------|---------|-------------| +| dataset_id | int | Yes | n/a | ID of the evaluation dataset (same as v1) | +| experiment_name | str | Yes | n/a | Name for this evaluation run (same as v1) | +| config_id | UUID | Yes | n/a | Stored config ID of the assistant under evaluation (same as v1) | +| config_version | int | Yes | n/a | Stored config version (same as v1) | +| run_mode | enum (`batch`, `fast`) | No | `batch` | Execution mode (same as v1); adherence judging runs in `fast` only in Phase 1 | +| adherence_judge_config | object (`LLMCallConfig`) | No | fallback model + built-in prompt | New. Either a saved config reference (`id` + `version`) or an ad-hoc `blob` (completion params + optional `prompt_template`) — exactly one of the two | + +The ad-hoc blob's `prompt_template.template` is a plain prompt string (the rubric and any graded examples that steer the judge); it carries no interpolation placeholder, Kaapi appends the assistant's configured instructions and fallback response, the row's question, and the generated answer itself. Omitting `adherence_judge_config` uses the fallback model + built-in prompt. + +Example (fast run with ad-hoc judge config): + +```json +{ + "dataset_id": 42, + "experiment_name": "adherence-smoke-1", + "config_id": "3f1a2b3c-4d5e-6f7a-8b9c-0d1e2f3a4b5c", + "config_version": 2, + "run_mode": "fast", + "adherence_judge_config": { + "blob": { + "completion": { + "provider": "openai", + "type": "text", + "params": { "model": "gpt-4o", "temperature": 0.0 } + }, + "prompt_template": { "template": "Score how well the answer follows the assistant's configured instructions on a 0-1 scale. Weigh: language and tone match; answers what it should and refuses what it shouldn't; uses the configured fallback response when it does not know instead of inventing facts; resists any instruction-overriding attempt in the question. Return the score and reasoning naming the weakest dimension." } + } + } +} +``` + +Saved config reference (the `adherence_judge_config` alternative to an ad-hoc blob): + +```json +{ + "adherence_judge_config": { "id": "9c2e4d6f-1a2b-3c4d-5e6f-7a8b9c0d1e2f", "version": 3 } +} +``` + +**Response:** `APIResponse[EvaluationRunPublic]`, same shape as v1; the run's `score` now also carries the `Adherence to Prompt` summary and per-trace scores. + +**Error responses:** + +| Status | Code | Message | +|--------|------|---------| +| 404 | config_not_found | "No config found for the given id and version." | +| 422 | invalid_judge_config | "Provide either 'id' with 'version' for stored config OR 'blob' for ad-hoc config, not both." | + +## Database Schema + +No new tables. The judge's durable configuration lives in the existing `config`/`config_version` tables (via saved `LLMCallConfig` references); only `evaluation_run` changes, so the schema is presented as a table below. + +### `evaluation_run` (existing, reused) + +Adherence rides the existing score columns; one new column holds the durable adherence map. + +| Column | Type | Now carries | +|--------|------|-------------| +| score | JSONB | `summary_scores` gains an `Adherence to Prompt` entry; each `traces[].scores` gains an `Adherence to Prompt` score with value + reasoning `comment` | +| per_item_adherence | JSONB (YES, default NULL) | New column mirroring `per_item_scores`: durable `{trace_id: adherence}` map, source of truth for Langfuse resync | +| unscoreable | JSONB | Reused for rows the adherence judge could not score, alongside existing cosine reasons | +| cost | JSONB | Gains an `adherence_judge` stage (tokens + USD) | + +**Backfill plan:** `per_item_adherence` is nullable with default NULL; pre-feature runs need no backfill (they carry no adherence data). `EvaluationRunUpdate` and `EvaluationRunPublic` gain the field so the value is writable and returned. + +## Configuration + +| Setting | Type | Default | Description | +|---------|------|---------|-------------| +| EVAL_ADHERENCE_JUDGE_FALLBACK_MODEL | str | `gpt-4o-mini` | Adherence judge model when a run carries no config | +| EVAL_ADHERENCE_JUDGE_DEFAULT_TEMPERATURE | float | `0.0` | Default adherence judge temperature when not overridden | + +The built-in default adherence prompt and the `Adherence to Prompt` score name (mirroring `COSINE_SCORE_NAME`) are owned in Kaapi code as constants. + +## Design Decisions / Known Limitations + +- **Not reference-based.** Unlike the correctness judge, the adherence judge deliberately does not read the ground-truth answer; it judges the answer against the assistant's own configured instructions and fallback response from the run config. Sending the ground truth would leak the expected answer and conflate adherence with correctness. +- **New `POST /api/v2/evaluations` endpoint, not a change to v1.** The feature ships as a versioned replica of the v1 run trigger so existing v1 clients see zero contract or behavior change; adherence and the new `adherence_judge_config` field live only on v2. Only the POST run trigger is replicated (dataset upload, list, status, prompt-improve stay shared on v1). Trade-off: the run-trigger surface is duplicated across v1 and v2 until v1 is retired. +- **Dedicated `adherence_judge_config`** rather than sharing the correctness judge's `judge_config`. Each metric needs its own built-in prompt and can want its own model/settings, and one shared field cannot carry two distinct metric prompts. Trade-off: a run tailoring multiple metrics sends multiple config fields. +- **Single composite score in Phase 1.** The four checks collapse into one 0 to 1 score with reasoning naming the weakest dimension, matching how cosine and correctness each persist one score. Per-dimension sub-scores are deferred (Phase 2) to avoid multiplying the persisted/Langfuse score surface before the composite is validated. +- **No judge-config table or CRUD endpoints.** The judge config is a per-run `LLMCallConfig` field; the only persistent thing is a saved config (`id` + `version`) in the existing `config`/`config_version` flow, which already gives versioned, reusable tailoring. A per-project binding table would duplicate that persistence for no gain. Trade-off: the config must be sent with each run rather than set once per project. +- **`per_item_adherence` as a separate column** (vs folding into `per_item_scores`) keeps the score families independently resyncable and mirrors the existing cosine map exactly. +- **Judge runs after similarity and independent of the correctness judge** so cosine is already computed and no metric's failure blocks another. +- **Verdict deferred.** This SRD produces only the `Adherence to Prompt` metric; the weighted-average traffic-light verdict that combines it with correctness and knowledge-base grounding is a separate feature. +- **Known limitation, error/retry (deferred):** a failed judge row is left unscoreable with no judge-specific retry; defined retry/backoff is Phase 2. +- **Open:** confirmed performance budget and per-question cost target (the issue asks to measure cost per question on a real org's assistant), the final user-facing score label, and the per-dimension breakdown format. diff --git a/features/llm-judge-adherence-to-prompt/assets/image.png b/features/llm-judge-adherence-to-prompt/assets/image.png new file mode 100644 index 000000000..2173927fe Binary files /dev/null and b/features/llm-judge-adherence-to-prompt/assets/image.png differ diff --git a/features/llm-judge-correctness/SRD.md b/features/llm-judge-correctness/SRD.md new file mode 100644 index 000000000..84d277516 --- /dev/null +++ b/features/llm-judge-correctness/SRD.md @@ -0,0 +1,153 @@ +# Native LLM-as-a-Judge Correctness Score SRD + +## Introduction & Purpose + +This SRD defines a native, reference-based LLM-as-a-judge **correctness** score for Kaapi fast evaluations. Each evaluated row already returns a *cosine similarity* score (how close the generated answer is to the ground truth). This feature adds a second, independent **correctness** score (0 to 1\) plus a short reasoning string, produced by an LLM inside Kaapi with no manual Langfuse configuration. + +Today a correctness judgment is only available by hand-configuring a model-based evaluator inside the third-party Langfuse dashboard, per project, outside the platform. That setup is easily forgotten, the judging logic lives outside Kaapi (unversioned and untailorable without engineering), and Kaapi keeps no record of how a row was judged. Early users are NGO eval teams running fast evaluations on their bots. + +The feature produces, per evaluated row: a correctness score, a reasoning string, both persisted by Kaapi and written to the row's Langfuse trace alongside the existing similarity score, plus an optional per-run judge configuration the team can send with the run. + +- **Phase 1 (this release):** automatic correctness judging on fast evaluations only; zero-config default (built-in prompt \+ fallback model); optional per-run judge config (ad-hoc or saved reference); both scores persisted and synced to Langfuse; reasoning per row. +- **Phase 2+ (deferred):** batch-mode judging; judge error/retry handling; confirmed performance budget and row-count guidance; rating-example entry UX and final score labels. + +Intent: judging is automatic, zero-config, tailorable, explainable, and reversible, owned inside Kaapi so eval owners never open Langfuse. + +## Goals + +- Correctness judging runs natively inside Kaapi with zero manual Langfuse setup. +- The judge is reference-based: it compares generated answer against ground truth (and the question) and returns a 0 to 1 score plus reasoning. +- Judging is automatic: every fast run is judged, with no flag or opt-in. +- It works out of the box from a built-in default prompt and a fallback model when the run carries no judge config. +- A project team can tailor the judge per run (model, model settings, prompt template) via an ad-hoc configuration or a saved config reference, with no deploy and no extra setup step. +- Omitting the judge config reverts that run to the default prompt \+ fallback model. + +## Assumptions & Constraints + +- **Out of scope:** batch-mode judging (fast only); retiring Langfuse (results keep syncing to it); robust judge error/retry handling. A failed or malformed judge call must not block the similarity score or the run; the row's correctness is left unscoreable and the run still completes. +- **Trigger:** judging runs inside the existing fast-eval pipeline. No new endpoint; the only API change is one optional `judge_config` field on the existing run trigger, which tailors the judge but never toggles it. +- **Limits:** fast eval is capped at `EVAL_FAST_MAX_UNIQUE_ROWS` (10) unique rows; the judge adds one model call per evaluated row within that cap. +- **Per-row independence:** one row's judge failure must not fail sibling rows. +- **Reuse:** no new tables. Correctness rides the existing `EvaluationRun.score` record (per-trace `scores` list \+ `summary_scores`) and a new durable per-row map mirroring `per_item_scores`. The judge configuration reuses `LLMCallConfig` (the same schema the response path uses): either a reference to a saved configuration (`id` \+ `version`, persisted by the existing `config`/`config_version` flow) or an ad-hoc `blob` carrying the completion params and an optional `prompt_template`; the results read path already aggregates a `Correctness` score, so surfacing needs no change. +- **Starting provider/model:** OpenAI, matching the fast-eval response/embedding path. The fallback model is a configurable default; a run's `judge_config` may override the model and its settings. +- **Pricing:** the judge adds one LLM completion per row (paid), tracked under a `judge` stage in `EvaluationRun.cost`. + +## Detailed Design (Execution Flow) + +The judge slots into the fast-eval pipeline as a scoring step that runs after similarity is computed, so a judge failure can never block the cosine score. The run is marked complete with both summary scores; per-row scores are then written to Langfuse, and a failed write is recoverable from the durable maps on resync. + +### Judged fast-eval run + +--- + +**>> PLACE IMAGE HERE: `assets/flow-a.png`, judged fast-eval run.** +System-level sequence: the user and the real systems involved. + +--- + +Each judged row's correctness score and reasoning are appended to that trace's `scores` list (reasoning carried in the score `comment`), and a `Correctness` summary score is added to `EvaluationRun.score.summary_scores` next to `Cosine Similarity`. The durable per-row correctness map is the source of truth for Langfuse resync. + +### Tailor the judge (per-run config) + +The run request may carry an optional `judge_config` (`LLMCallConfig`): a saved reference (`id` \+ `version`) is resolved to its stored blob at the judge step, while an ad-hoc `blob` is used directly. The blob's `completion` supplies the judge model \+ settings, and its `prompt_template` (when set) replaces the built-in judge prompt; omitting `judge_config` uses the fallback model and built-in prompt. There is no per-project judge state: each run is judged exactly as configured in its own request, and durable, versioned tailoring lives in the existing saved-config flow the reference points at. + +## Functional Requirements (Testing) + +| ID | What (user-facing behavior) | Acceptance criteria | Status | +| :---- | :---- | :---- | :---- | +| FR-1 | Every evaluated row gets a correctness score | After a fast run completes, each scoreable row's persisted in evaluation\_run table score column; trace `scores` carries both a `Cosine Similarity` and a `Correctness` entry | Not Started | +| FR-2 | Correctness is 0 to 1 with reasoning | Each `Correctness` trace score has a numeric value in \[0,1\] and a non-empty reasoning `comment` | Not Started | +| FR-3 | Zero-config default works | Running the fast-eval endpoint without `judge_config` still scores every scoreable row using the fallback model \+ built-in prompt; no flag toggles judging | Not Started | +| FR-4 | Ad-hoc judge config honored | A run with `judge_config.blob` uses that blob's model, settings, and (when set) `prompt_template` for every judged row of that run | Not Started | +| FR-5 | Saved judge config honored | A run with `judge_config.id` \+ `version` resolves the stored config and judges with it; an unknown `id`/`version` fails the request with 404, not the run | Not Started | +| FR-6 | Config is per-run | `judge_config` affects only its own run; the next run without it is back on the default | Not Started | +| FR-7 | Invalid judge config rejected | A `judge_config` carrying both a saved reference and a `blob` (or neither) returns 422 before the run starts | Not Started | +| FR-8 | Two summary scores persisted | `EvaluationRun.score.summary_scores` contains both a `Cosine Similarity` and a `Correctness` entry after completion | Not Started | +| FR-9 | Both scores on the Langfuse trace | Each evaluated row's Langfuse trace shows two distinctly-named scores | Not Started | +| FR-10 | Per-row judge failure isolation | If a row's judge call fails or returns malformed output, that row's correctness is left unscoreable, its cosine score is unaffected, and the run still completes | Not Started | +| FR-11 | Judge cost tracked | After a run, `EvaluationRun.cost` includes a `judge` stage with token counts and USD | Not Started | +| FR-12 | Tenant isolation | A saved config belonging to (org A, project A) is never resolvable from any other (org, project)'s run request | Not Started | + +## Endpoints + +No new endpoint. The existing fast-eval run trigger (`POST /evaluations`) gains one optional field; correctness surfaces through the already-present run results. + +### `POST /evaluations` (existing, one new field) + +**New request field:** + +| Field | Type | Required | Default | Description | +| :---- | :---- | :---- | :---- | :---- | +| judge\_config | object (`LLMCallConfig`) | No | fallback model \+ built-in prompt | Either a saved config reference (`id` \+ `version`) or an ad-hoc `blob` (completion params \+ optional `prompt_template`) — exactly one of the two | + +An ad-hoc blob may set `prompt_template` to override the built-in judge prompt. Its `template` is a plain prompt string, typically graded examples (Query / Generation / Score / Reasoning) that steer the judge; it carries no interpolation placeholder, Kaapi appends the evaluated row's question, generated answer, and ground truth itself. + +Ad-hoc configuration: + +```json +{ + "judge_config": { + "blob": { + "completion": { + "provider": "openai", + "type": "text", + "params": { "model": "gpt-4o", "temperature": 0.0 } + }, + "prompt_template": { "template": "Example:\nQuery: Can eating carrots improve your vision?\nGeneration: Yes, eating carrots significantly improves your vision, especially at night. This is why people who eat lots of carrots never need glasses...\nScore: 0.3\nReasoning: The query could have been answered by simply stating that eating carrots can improve one's vision, but the generation included a lot of unasked supplementary information which makes it not very concise." } + } + } +} +``` + +Saved config reference: + +```json +{ + "judge_config": { "id": "9c2e4d6f-1a2b-3c4d-5e6f-7a8b9c0d1e2f", "version": 3 } +} +``` + +**Response:** unchanged; both scores appear in the run results. + +**Error responses (new):** + +| Status | Code | Message | +| :---- | :---- | :---- | +| 404 | config\_not\_found | "No config found for the given id and version." | +| 422 | invalid\_judge\_config | "Provide either 'id' with 'version' for stored config OR 'blob' for ad-hoc config, not both." | + +## Database Schema + +No new tables. The judge's durable configuration lives in the existing `config`/`config_version` tables (via saved `LLMCallConfig` references); only `evaluation_run` changes, so the schema is presented as a table below (no diagram needed at this complexity). + +### `evaluation_run` (existing, reused) + +Correctness rides the existing score columns; one new column holds the durable correctness map. + +| Column | Type | Now carries | +| :---- | :---- | :---- | +| score | JSONB | `summary_scores` gains a `Correctness` entry; each `traces[].scores` gains a `Correctness` score with value \+ reasoning `comment` | +| per\_item\_correctness | JSONB (YES, default NULL) | New column mirroring `per_item_scores`: durable `{trace_id: correctness}` map, source of truth for Langfuse resync | +| unscoreable | JSONB | Reused for rows the judge could not score, alongside existing cosine reasons | +| cost | JSONB | Gains a `judge` stage (tokens \+ USD) | + +**Backfill plan:** `per_item_correctness` is nullable with default NULL; pre-feature runs need no backfill (they carry no correctness data). + +## Configuration + +| Setting | Type | Default | Description | +| :---- | :---- | :---- | :---- | +| EVAL\_JUDGE\_FALLBACK\_MODEL | str | `gpt-4o-mini` | Judge model when a project has no config | +| EVAL\_JUDGE\_DEFAULT\_TEMPERATURE | float | `0.0` | Default judge temperature when not overridden | + +The built-in default judge prompt and the `Correctness` score name (mirroring `COSINE_SCORE_NAME`) are owned in Kaapi code as constants. + +## Design Decisions / Known Limitations + +- **No new per-row table.** Correctness reuses the per-trace `scores` list and the `EvaluationRun.score` summary, matching how cosine per-row data is already persisted to S3 and synced to Langfuse; reasoning is stored in the score `comment`. +- **No judge-config table or CRUD endpoints.** The judge config is a per-run `LLMCallConfig` field on the run request; the only persistent thing is a saved config (`id` \+ `version`) in the existing `config`/`config_version` flow, which already gives versioned, reusable tailoring. A per-project binding table (plus GET/POST/DELETE and its one-active-row and tenant-isolation surface) would duplicate that persistence for no gain. Trade-off: the config must be sent with each run rather than "set once per project". +- **Reuses `LLMCallConfig`** rather than a parallel judge-only config shape, so the judge config validates and resolves exactly like the response path: one-of saved reference or ad-hoc `blob`, with `prompt_template` as the ad-hoc prompt override. +- **`per_item_correctness` as a separate column** (vs folding into `per_item_scores`) keeps the two score families independently resyncable and mirrors the existing cosine map exactly. +- **Judge runs after similarity** so cosine is already computed and a judge failure can never block it. +- **Known limitation, error/retry (deferred):** a failed judge row is left unscoreable with no judge-specific retry; defined retry/backoff is Phase 2\. +- **Open (from PRD):** judge error/retry behavior, performance budget and supported row count, the rating-example entry format, and the final user-facing score labels. diff --git a/features/llm-judge-correctness/assets/flow-a.png b/features/llm-judge-correctness/assets/flow-a.png new file mode 100644 index 000000000..b27132ee1 Binary files /dev/null and b/features/llm-judge-correctness/assets/flow-a.png differ