Compare LLM judges in the LLM Scorer, with support for TypeSafe's Jev

[ianarawjo]

Makes the LLM Scorer a way to compare LLM judges, and adds TypeSafe's Jev (a "decision" model on OpenRouter) as one of them. Also adds an example flow for the Jev launch.

What it does

1. Several judges in one LLM Scorer. "Add judge +" adds a model; each response is scored once per judge, with scores keyed by judge name. Multi-Eval keeps one judge.

2. Explicit answer formats. Categorical scorers list their categories (optionally with descriptions), and numeric ones their scale's levels. Judges are told exactly what to answer. Answers outside the format are kept as written and counted as invalid, instead of turning the whole run's scores into strings.

3. Ground truth and a Judges tab. "Compare to" picks a column as the true label. A new Judges tab in the Inspector shows:

   • agreement with the label and between judges (descriptive, with n), with bars in each judge's color;
   • cost and speed per judge (total cost, median time per answer, tokens);
   • the responses where judges disagree;
   • for judges that give probabilities (Jev), how often they're right at each confidence.

4. Jev support. Jev (~typesafe/jev-latest) is in the OpenRouter menu. As a judge it gets the response as its state, and one typed question built from the rubric and format:

   • true/false → yes/no
   • categorical → choice
   • numeric → scale

   It goes through OpenRouter's /api/alpha/decisions. Its probability for each answer is kept with the score. Formats Jev can't answer fail with a clear message before any request, and using it in a Prompt Node says to use it as a judge.

5. Cost in response stats. Builds on Record each response's latency, token counts and speed #459: cost is read from OpenRouter's usage.cost, for any model it serves, and exposed as stat_cost_usd.

6. Data straight into the scorer. Tabular Data, Text Fields, Items and Media nodes can connect directly to an LLM Scorer. A table row's other columns become variables.

7. Smaller scorer improvements:

   • The node is compact: format and options share one row.
   • Run's tooltip says what a run will send ("Will load scores from cache", "Will send 120 requests per judge").
   • Right-click → "Clear cached scores".

8. Example flow: "🧑‍⚖️ Compare LLM judges, with Jev" (jev-judges.cforge), run live against human labels:

   	Triage 120 bank questions (Banking77)	Flag scams in 100 texts (SMS Spam Collection, paraphrased first)
   Jev	99%, $0.0025	98%, $0.00135, 253 ms median
   Claude Sonnet 5	98%, $0.08	98%, $0.03, 1.8 s
   GPT-5.4 Mini	93%, $0.02	98%, $0.009, 823 ms

   Both datasets are CC BY 4.0 and credited in the flow. Samples were hand-picked to leave out crude or sensitive items.

Fixes found along the way

• queryLLM's cache index dropped earlier runs' cache files. An inner variable shadowed the one it saved, so changing a model's settings and changing them back re-queried everything. This affected Prompt Nodes too.
• Table View lost scores for responses that shared the same vars (e.g. from a Split Node). Each response now gets its own row when scores have their own columns.
• Categorical answers with "Reason before scoring" are now read from the SCORE: line. Before, only numbers were extracted.

Behavior changes for existing users

• Binary and numeric scores are read one at a time, not all-or-nothing. A single stray answer no longer turns every score into a string. Stray answers are kept as written and flagged.
• The Inspector modal's trigger() can open on a named tab. Other nodes' Inspectors are unchanged; the Judges tab only appears for LLM Scorers.

Testing

• jest: 1,219 passing, including new tests for:
  • answer parsing and formats
  • multiple judges
  • agreement, disagreements and reliability
  • Jev requests and answers (mocked)
  • caching across rubric changes
  • data inputs
  • cost formatting
• pytest: response-stats tests pass (with_stats_metavars gained cost_usd).
• tsc, eslint and prettier are clean on changed files.
• Live, in the dev server, against OpenRouter:
  • Jev alone and beside GPT-5.4 Mini and Claude Sonnet 5
  • the decisions endpoint's reply format
  • cost reporting for both Jev and chat models
  • the example flow's runs

Not verified

• Multi-Eval in the browser. It gets the categories and scale fields, but only its prop wiring changed.
• A production build, and the hosted web build's Example Flows (it needs the usual deploy step to include the new file).
• YAML export still writes only the first judge.
• Jev's endpoint is in alpha (/api/alpha/decisions), so its path may change. It's one constant in utils.ts.

🤖 Generated with Claude Code