Skip to content

fix(evals): Update Judge Metrics Prompt #1104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AkhileshNegi merged 18 commits into from
Aug 3, 2026
Merged

fix(evals): Update Judge Metrics Prompt #1104

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
46a3037
feat(evals): added the verdict corresponding to judge metric in each row
Ayush8923 Jul 29, 2026
6cd310b
fix(*): cleanups
Ayush8923 Jul 29, 2026
5ff0202
fix(*): cleanups
Ayush8923 Jul 29, 2026
194c8f5
feat(*): implement the evals human readable summary generation
Ayush8923 Jul 29, 2026
6900d8a
fix(*): update the ai_summary code
Ayush8923 Jul 29, 2026
5b1074b
fix(evals): clenaups
Ayush8923 Jul 30, 2026
28e3b72
fix(evals): clenaups
Ayush8923 Jul 30, 2026
b3aea5d
Merge branch 'main' into feat/evals-human-readable-summary-generation
AkhileshNegi Aug 3, 2026
0ab0170
updated wiki
AkhileshNegi Aug 3, 2026
b5b359a
feat(*): update the judge prompt and test cases
Ayush8923 Aug 3, 2026
1a75458
Merge branch 'main' of https://github.com/ProjectTech4DevAI/kaapi-bac…
Ayush8923 Aug 3, 2026
c24ce2b
Merge branch 'main' of https://github.com/ProjectTech4DevAI/kaapi-bac…
Ayush8923 Aug 3, 2026
d423928
fix(*): cleanups
Ayush8923 Aug 3, 2026
ff54e8f
fix(*): few updates on evals flow
Ayush8923 Aug 3, 2026
215448b
fix(*): update the the system preamble
Ayush8923 Aug 3, 2026
7c91096
fix(*): update the wiki
Ayush8923 Aug 3, 2026
28a164a
fix(*): update the test cases
Ayush8923 Aug 3, 2026
8a647fc
Merge branch 'main' into fix/judge-metrics-prompt-update
AkhileshNegi Aug 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion backend/app/api/docs/evaluation/create_evaluation_v2.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ body with Kaapi's native LLM-as-Judge built in — v1 is left unchanged.

v2 runs are **always fast** and always judged (there is no `run_mode`; batch is
deferred to a later phase). Every scoreable row is automatically judged (no opt-in
flag) by one combined LLM-judge call scoring each applicable metric in [0, 1] with
flag) by one combined LLM-judge call scoring each applicable metric on an integer 0–5 scale with
reasoning: **Adherence to Ground Truth** (answer conveys the same correct
information as the golden answer), **Adherence to Prompt** (answer obeys the
assistant's configured instructions; applies only when the run resolves a config
Expand Down
2 changes: 1 addition & 1 deletion backend/app/api/docs/evaluation/improve_prompt_v2.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ Enqueue a v2 prompt-recommendation job for the configuration evaluated by a judg

Unlike v1 (which reads cosine similarity + correctness), this consumes the native
three-metric judge results — **Adherence to Ground Truth**, **Adherence to Prompt**,
and **Adherence to Knowledge Base** — each carrying a 0–1 score and the judge's
and **Adherence to Knowledge Base** — each carrying an integer 0–5 score and the judge's
reasoning. The worker reads both the score and the reasoning per metric, focuses the
rewrite on rows where Adherence to Prompt / Ground Truth are low, and changes only
`completion.params.instructions`; model, knowledge base, and all other settings are
Expand Down
9 changes: 6 additions & 3 deletions backend/app/crud/evaluations/judge.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ class JudgeMetricSpec:

@dataclass
class MetricScore:
"""One metric's outcome for a row: score in [0, 1] and its reasoning."""
"""One metric's outcome for a row: integer score in 0–5 and its reasoning."""

@coderabbitai coderabbitai Bot Aug 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Files:"
git ls-files 'backend/app/**' | rg '(^backend/app/crud/evaluations/judge\.py$|backend/app/tests/crud/evaluations/test_fast_judge\.py$)' || true

echo
echo "judge.py outline:"
ast-grep outline backend/app/crud/evaluations/judge.py --view expanded || true

echo
echo "Relevant judge.py lines:"
sed -n '120,190p' backend/app/crud/evaluations/judge.py

echo
echo "Relevant test_fast_judge.py lines:"
sed -n '140,210p' backend/app/tests/crud/evaluations/test_fast_judge.py

echo
echo "Score annotation usages in judge.py:"
rg -n "MetricScore|\.score|ground_truth|prompt|tuple\[ (float|int) , str \]" backend/utils backend/app/crud/evaluations backend/app/tests/crud/evaluations || true

Repository: ProjectTech4DevAI/kaapi-backend

Length of output: 50387

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "judge.py `_parse_metric_score`:"
sed -n '230,260p' backend/app/crud/evaluations/judge.py

echo
echo "test_fast_judge.py helper around _both_metrics_response:"
sed -n '160,184p' backend/app/tests/crud/evaluations/test_fast_judge.py

echo
echo "target annotations:"
python3 - <<'PY'
from pathlib import Path
for path in [
    Path("backend/app/crud/evaluations/judge.py"),
    Path("backend/app/tests/crud/evaluations/test_fast_judge.py"),
]:
    text = path.read_text()
    for needle in ["  score: float", "ground_truth: tuple[float, str]", "prompt: tuple[float, str]"]:
        idx = text.find(needle)
        print(f"{path}:{idx+1}:{needle}")
PY

Repository: ProjectTech4DevAI/kaapi-backend

Length of output: 2816

Use an integer contract for judge scores.

MetricScore.score is parsed as an integer in backend/app/crud/evaluations/judge.py#L237, but the dataclass still annotates it as float. Update that field to int and change _both_metrics_response score tuples in backend/app/tests/crud/evaluations/test_fast_judge.py#L172-173 from tuple[float, str] to tuple[int, str] so the contracts stay aligned.

📍 Affects 2 files
  • backend/app/crud/evaluations/judge.py#L154-L154 (this comment)
  • backend/app/tests/crud/evaluations/test_fast_judge.py#L172-L173
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/app/crud/evaluations/judge.py` at line 154, Update the
MetricScore.score annotation in backend/app/crud/evaluations/judge.py at lines
154-154 from float to int, matching its integer parsing contract. Also update
the _both_metrics_response score tuple annotations in
backend/app/tests/crud/evaluations/test_fast_judge.py at lines 172-173 from
tuple[float, str] to tuple[int, str].

Source: Coding guidelines


score: float
reasoning: str
Expand Down Expand Up @@ -246,8 +246,11 @@ def _parse_metric_score(key: JudgeMetricEnum, raw: Any) -> MetricScore:
raise ValueError(
f"metric '{key.value}' score is not a number: {raw.get('score')!r}"
) from exc
if not 0.0 <= score <= 1.0:
raise ValueError(f"metric '{key.value}' score out of [0, 1]: {score}")
if score != int(score) or not 0 <= score <= 5:
raise ValueError(
f"metric '{key.value}' score must be an integer 0-5: {raw.get('score')!r}"
)
score = int(score)
Comment on lines +249 to +253

@coderabbitai coderabbitai Bot Aug 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

python - <<'PY'
import json

for literal in ("true", "false", "Infinity", "NaN"):
    raw = json.loads(f'{{"score": {literal}}}')["score"]
    score = float(raw)
    try:
        normalized = int(score)
    except Exception as exc:
        normalized = type(exc).__name__
    print(literal, repr(raw), normalized)
PY

Repository: ProjectTech4DevAI/kaapi-backend

Length of output: 241

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf 'File candidates:\n'
fd -a 'judge\.py$' . | sed 's#^\./##'

printf '\nRelevant file outline:\n'
ast-grep outline backend/app/crud/evaluations/judge.py --view compact || true

printf '\nRelevant source lines 200-270:\n'
sed -n '200,270p' backend/app/crud/evaluations/judge.py | nl -ba -v200

printf '\nSearch for parse_score_usages/ValueError handling near this file:\n'
rg -n "parse_scores|ValueError|score must be an integer|metric '" backend/app/crud/evaluations backend/app -g '*.py' | head -200

Repository: ProjectTech4DevAI/kaapi-backend

Length of output: 632

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf 'Relevant source lines 200-270:\n'
sed -n '200,270p' backend/app/crud/evaluations/judge.py | awk '{printf "%6d\t%s\n", NR+199, $0}'

printf '\nSearch for parse_scores/usages and related validation:\n'
rg -n "parse_scores|ValueError|score must be an integer|metric '" backend/app/crud/evaluations backend/app -g '*.py' | head -200

printf '\nRelevant tests around score validation:\n'
rg -n "score|ValueError|boolean|Infinity|NaN" backend/app/tests/crud/evaluations/test_judge.py backend/app/tests/crud/evaluations/test_fast_judge.py -g '*.py' | head -200

Repository: ProjectTech4DevAI/kaapi-backend

Length of output: 48273

🏁 Script executed:

#!/bin/bash
set -euo pipefail

python3 - <<'PY'
from typing import Any
from pathlib import Path
src = Path('backend/app/crud/evaluations/judge.py').read_text()
start = src.index('def _parse_metric_score')
end = src.index('\ndef _parse_judge_output', start)
code = src[start:end]
print(code)

print('\nBehavioral probes:')
for literal, raw_value in [('true', True), ('false', False), ('Infinity', 'Infinity'), ('NaN', 'NaN')]:
    print(f'>{literal!r}:')
    try:
        score = float(raw_value)
        print(f'float={score!r}')
        print(f'isfinite=bool={score != score if score != score else (score == score)}')
        if isinstance(score, float):
            try:
                normalized = int(score)
                print(f'int={normalized!r}')
            except Exception as exc:
                print(f'int_exception={type(exc).__name__}:{exc}')
        else:
            print('not a float')
    except Exception as exc:
        print(f'float_exception={type(exc).__name__}:{exc}')
PY

python3 - <<'PY'
from typing import Any
from pathlib import Path
src = Path('backend/app/crud/evaluations/judge.py').read_text()
start = src.index('def _parse_judge_output')
end = src.index('\ndef _judge', end)
code = src[start:end]
print(code)

# Inspect exact exception handling around _parse_judge_output using static source.
lines = code.splitlines()
for i, line in enumerate(lines, 1):
    if 'except' in line or '_parse_metric_score' in line or 'except ValueError' in line:
        print(f'{i:04d}: {line}')
PY

Repository: ProjectTech4DevAI/kaapi-backend

Length of output: 1595

Reject boolean and non-finite judge scores before normalization.

float() converts JSON true and false to 1.0 and 0.0, so malformed boolean scores can be stored as valid scores. JSON Infinity raises OverflowError from int(score), bypassing the documented ValueError path that isolates the row.

Reject booleans, check math.isfinite(score) before converting to int, and catch OverflowError from float(). Add regression tests for true, false, Infinity, and NaN.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/app/crud/evaluations/judge.py` around lines 249 - 253, Update the
judge score validation around the score parsing and normalization logic to
reject boolean values, catch OverflowError from float conversion, and validate
math.isfinite(score) before int(score). Ensure all invalid cases raise the
documented ValueError so the row is isolated, and add regression coverage for
true, false, Infinity, and NaN.


reasoning = str(raw.get("reasoning") or "").strip()
if not reasoning:
Expand Down
102 changes: 70 additions & 32 deletions backend/app/crud/evaluations/score.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,21 +12,22 @@


class VerdictEnum(str, Enum):
"""Qualitative band derived from a 0–1 judge-metric score."""
"""Qualitative band derived from a 0–5 judge-metric score."""

NEEDS_IMPROVEMENT = "Needs Improvement"
NEEDS_REFINEMENT = "Needs Refinement"
GOOD = "Good"


VERDICT_NEEDS_IMPROVEMENT_BELOW: float = 0.3
VERDICT_GOOD_AT_OR_ABOVE: float = 0.6
VERDICT_NEEDS_IMPROVEMENT_BELOW: float = 2.0
VERDICT_GOOD_AT_OR_ABOVE: float = 4.0


def verdict_from_score(score: float) -> VerdictEnum:
"""Map a 0–1 judge-metric score to its verdict band.
"""Map a 0–5 judge-metric score to its verdict band.

Boundaries: exactly 0.3 → Needs Refinement, exactly 0.6 → Good.
Boundaries: below 2 → Needs Improvement, 2 to <4 → Needs Refinement,
4 and above → Good.
"""
if score < VERDICT_NEEDS_IMPROVEMENT_BELOW:
return VerdictEnum.NEEDS_IMPROVEMENT
Expand Down Expand Up @@ -64,11 +65,15 @@ def verdict_from_score(score: float) -> VerdictEnum:

JUDGE_SYSTEM_PREAMBLE: str = (
"You are a strict, impartial evaluator. You score an assistant's answer on the "
"independent metrics listed below in a single pass. Each metric is a float in "
"[0.0, 1.0] with one or two sentences of reasoning. The metrics are independent "
"— judge each only against its own inputs and rules; do not let one metric's "
"verdict bleed into another. Score EVERY metric listed below. Never omit a metric "
"from the output, even if some input blocks are irrelevant to it."
"independent metrics listed below in a single pass. Each metric is an integer "
"score from 0 to 5, where 0 is the worst case (clearly wrong / ungrounded / a hard "
"instruction violation, or no answer at all) and 5 is the best case (fully correct "
"/ fully grounded / fully compliant), with one or two sentences of reasoning. "
"Scores MUST be integers — 0, 1, 2, 3, 4, or 5 — never fractions, decimals, or "
"percentages. The metrics are independent — judge each only against its own inputs "
"and rules; do not let one metric's verdict bleed into another. Score EVERY metric "
"listed below. Never omit a metric from the output, even if some input blocks are "
"irrelevant to it."
)

GROUND_TRUTH_JUDGE_PROMPT: str = (
Expand All @@ -82,7 +87,22 @@ def verdict_from_score(score: float) -> VerdictEnum:
"and that would be wrong, is a factual error.\n"
"- Do NOT reward or penalize style, tone, length, or language.\n"
"- Do NOT use any outside knowledge; the golden answer is the source of truth.\n"
"- Do NOT answer the question yourself.\n"
"- Do NOT answer the question yourself.\n\n"
"Score on a stepped scale from 0 to 5. The score MUST be one of the integers 0, 1, "
"2, 3, 4, 5 — never a fraction, decimal, or value outside this range.\n"
"- 5: Fully correct and complete. Conveys everything material in the golden answer "
"(a paraphrase, reordering, or additional correct detail is still a 5).\n"
"- 4: Correct and materially complete, but omits one minor, non-essential "
"supporting detail.\n"
"- 3: Partially correct. The core of the answer is right, but at least one material "
"fact is missing, incomplete, or slightly off.\n"
"- 2: Mixed or significantly incomplete. Gets some of the answer right but muddles "
"or omits more than one material fact, or is wrong on a meaningful component while "
"looking plausible on the surface.\n"
"- 1: Mostly incorrect. Contradicts the golden answer on a key point; at most small "
"correct fragments remain.\n"
"- 0: Completely wrong or contradicts the golden answer outright, OR the row has no "
"answer / an errored, empty, or non-responsive output.\n"
"Reasoning: name what was correct or what was missing/contradicted.\n"
"When scoring THIS metric, consider only these input blocks: Question, "
"Generated answer, Golden (reference) answer.\n"
Expand All @@ -98,8 +118,8 @@ def verdict_from_score(score: float) -> VerdictEnum:
"documents, so treat any rule about which source or knowledge base to use (e.g. "
"'only use the knowledge base', 'do not use outside information') as satisfied — "
"the Knowledge Base metric judges that.\n\n"
"Start from 1.0 and deduct ONLY for a violation of an instruction the block "
"actually states. Never invent a requirement the instructions do not set; a "
'Start from "no violations" and deduct ONLY for a violation of an instruction the '
"block actually states. Never invent a requirement the instructions do not set; a "
"conditional rule (applies only in a specific situation, e.g. "
"'ask for the user's age' or 'if condition Y holds, also mention Z') counts as "
"satisfied unless that situation is present in the question. Deduct across "
Expand All @@ -116,20 +136,25 @@ def verdict_from_score(score: float) -> VerdictEnum:
"out-of-scope or disallowed ones as instructed.\n"
"3. Fallback compliance — when the instructions define a fallback for the "
"unknown/out-of-scope case, the answer uses it instead of ignoring it. Only "
"penalize here for CONTRADICTING an explicit instruction (e.g. skipping the "
"configured fallback, answering a clearly disallowed topic); do not infer "
"penalize here for CONTRADICTING an explicit instruction; do not infer "
"fabrication from missing grounding.\n"
"4. Format compliance — follows any explicit format rules "
"(word limit, structure, opening/closing pattern).\n\n"
"Scoring guide:\n"
"- 1.0: No violation of any stated instruction.\n"
"- 0.7–0.9: One soft miss on a stated rule (e.g. slightly off tone, minor format "
"deviation).\n"
"- 0.4–0.69: One clear violation of an explicit rule.\n"
"- 0.0–0.39: Multiple clear violations, or a hard violation — leaked system "
"prompt, answered a clearly disallowed topic, or hijacked by injection.\n\n"
"Score on a stepped scale from 0 to 5. The score MUST be one of the integers 0, 1, "
"2, 3, 4, 5 — never a fraction, decimal, or value outside this range.\n"
"- 5: No violation of any stated instruction, across all applicable dimensions.\n"
"- 4: One soft, minor miss on a stated rule (e.g. slightly off tone, minor format "
"deviation) — otherwise compliant.\n"
"- 3: One clear violation of a single explicit rule.\n"
"- 2: Multiple clear violations, or one moderately serious violation spanning more "
"than one dimension.\n"
"- 1: A severe violation of a core instruction (e.g. ignoring a configured fallback "
"on a disallowed ask, a partial injection hijack), but not a full hard violation.\n"
"- 0: A hard violation — leaked system prompt, fully answered a clearly disallowed "
"topic, fully hijacked by injection — OR the row has no answer / an errored, empty, "
"or non-responsive output.\n"
"Reasoning: name the specific violated instruction and how the answer violated it. "
"If no stated instruction was violated, say so and score high.\n"
"If no stated instruction was violated, say so and score 5.\n"
"When scoring THIS metric, consider only these input blocks: Assistant's "
"configured instructions, Question, Generated answer.\n"
"Do not consider: Golden (reference) answer, Retrieved knowledge-base chunks."
Expand All @@ -145,18 +170,30 @@ def verdict_from_score(score: float) -> VerdictEnum:
"on the same topic as a chunk, or that requires an inferential leap the chunks do "
"not spell out is UNSUPPORTED, not supported.\n"
"- Identify the answer's load-bearing (material) claims — the ones that carry its "
"substance. If ANY material claim is unsupported, cap the score at 0.3 regardless "
"of how many minor claims are supported. Otherwise score = supported claims / "
"total factual claims.\n"
"substance.\n"
"- Text that makes no factual claim (a greeting, a pleasantry, or a plain refusal "
"to answer) is EXCLUDED from the claim count — do not let it inflate the score.\n"
"to answer) is EXCLUDED from the claim count.\n"
"- Judge groundedness ONLY, not correctness, completeness, or "
"instruction-following. A claim faithful to the chunks is grounded even if the "
"chunks are themselves wrong.\n"
"- Do NOT use any outside knowledge; the retrieved chunks are the ONLY allowed "
"source of support.\n"
"source of support.\n\n"
"Score on a stepped scale from 0 to 5. The score MUST be one of the integers 0, 1, "
"2, 3, 4, 5 — never a fraction, decimal, or value outside this range.\n"
"- 5: Every factual claim is explicitly supported by the retrieved chunks. Fully "
"grounded.\n"
"- 4: All load-bearing claims are grounded; only a minor, non-material claim lacks "
"explicit support.\n"
"- 3: One non-critical inferential leap beyond the chunks, but no load-bearing "
"claim is fabricated.\n"
"- 2: At least one load-bearing/material claim is unsupported or invented, even "
"though other claims are grounded.\n"
"- 1: Most claims are unsupported or invented; only incidental/minor claims are "
"grounded.\n"
"- 0: The answer is fabricated wholesale — no claim is grounded in the retrieved "
"chunks — OR the row has no answer / an errored, empty, or non-responsive output.\n"
"Reasoning: quote the exact chunk span supporting the main claim. When the score "
"is below 1.0, name the specific unsupported or invented claim.\n"
"is below 5, name the specific unsupported or invented claim.\n"
"When scoring THIS metric, consider only these input blocks: Generated answer, "
"Retrieved knowledge-base chunks.\n"
"Do not consider: Assistant's configured instructions, Question, Golden "
Expand All @@ -165,9 +202,10 @@ def verdict_from_score(score: float) -> VerdictEnum:

JUDGE_OUTPUT_INSTRUCTION: str = (
"Respond with ONLY a single JSON object mapping each metric key to its result, of "
'the form {{"<metric_key>": {{"score": <float between 0 and 1>, "reasoning": '
'"<one or two sentences>"}}}}. Include exactly these metric keys: {metric_keys}. '
"Output nothing else."
'the form {{"<metric_key>": {{"score": <integer 0 to 5>, "reasoning": '
'"<one or two sentences in English>"}}}}. Scores MUST be integers 0-5. Every '
'"reasoning" string MUST be written in English. Include exactly these metric keys: '
"{metric_keys}. Output nothing else."
)


Expand Down
4 changes: 2 additions & 2 deletions backend/app/crud/evaluations/summary.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@

_SUMMARY_REASONING_EFFORT: str = "minimal"

_CONSISTENCY_STABLE_AT_OR_BELOW: float = 0.1
_CONSISTENCY_MIXED_AT_OR_BELOW: float = 0.2
_CONSISTENCY_STABLE_AT_OR_BELOW: float = 0.5
_CONSISTENCY_MIXED_AT_OR_BELOW: float = 1.0

# Internal metric key -> plain behaviour phrase, so the brief never leaks the labels into the model's context.
_DIMENSION_PLAIN_NAME: dict[str, str] = {
Expand Down
2 changes: 1 addition & 1 deletion backend/app/services/evaluations/prompt_improvement.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -671,7 +671,7 @@ def _draft_improved_prompt(
"metric objects; each has `name`, `value`, and `comment`, where:\n"
f"- `name` is one of `{GROUND_TRUTH_SCORE_NAME}`, `{PROMPT_SCORE_NAME}`, "
f"or `{KNOWLEDGE_BASE_SCORE_NAME}`.\n"
"- `value` is the metric's score from 0 (worst) to 1 (best). Metrics that "
"- `value` is the metric's score, an integer from 0 (worst) to 5 (best). Metrics that "
'could not be scored appear with `value` = "N/A" and `unscoreable` = true; '
"ignore those.\n"
"- `comment` is the judge's reasoning for that score — read it, not just "
Expand Down
7 changes: 5 additions & 2 deletions backend/app/tests/api/routes/test_improve_prompt_v2.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,13 +76,13 @@
"scores": [
{
"name": GROUND_TRUTH_SCORE_NAME,
"value": 0.4,
"value": 2,
"data_type": "NUMERIC",
"comment": _GT_COMMENT,
},
{
"name": PROMPT_SCORE_NAME,
"value": 0.3,
"value": 1,
"data_type": "NUMERIC",
"comment": _PROMPT_COMMENT,
},
Expand Down Expand Up @@ -578,6 +578,9 @@ def test_v2_message_carries_metric_reasoning_and_ignores_na(
assert _GT_COMMENT in message
assert _PROMPT_COMMENT in message

# The v2 brief describes the judge score on the 0–5 integer scale.
assert "an integer from 0 (worst) to 5 (best)" in message

# The unscoreable KB metric must be presented as ignore-worthy, not a real
# score: the prompt tells the model to skip value="N/A" / unscoreable metrics.
assert '"N/A"' in message
Expand Down
Loading
Loading