feat(evals): Added verdict categorization system

[Ayush8923]

Issue

Closes #1092

Summary

• Adds a per-metric verdict band to each row of an Evaluation v2 (judge) run, so results are scannable at a glance (pass / borderline / fail) instead of forcing users to interpret raw 0–1 scores.
• Each numeric judge-metric score in a row's scores (Adherence to Ground Truth / Prompt / Knowledge Base) now carries a verdict derived from that score:

1. 0 ≤ score < 0.3 --> Needs Improvement
2. 0.3 ≤ score < 0.6 --> Needs Refinement
3. 0.6 ≤ score ≤ 1 --> Good

Checklist

Before submitting a pull request, please ensure that you mark these task.

• Ran fastapi run --reload app/main.py or docker compose up in the repository root and test.
• If you've fixed a bug or added code that is tested and has test cases.

Summary by CodeRabbit

• New Features

  • Added qualitative verdict labels to numeric judge metric scores: Needs Improvement, Needs Refinement, or Good.
  • Verdicts now appear on applicable trace scores in v2 evaluations.

• Documentation

  • Updated evaluation documentation to explain verdict bands and clarify which score types do not receive verdicts.

• Tests

  • Added coverage for verdict thresholds, labels, and trace-score behavior across supported evaluation types.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 0bb3b875-0683-48b2-8258-ce5e80184da2

📥 Commits

Reviewing files that changed from the base of the PR and between 6cd310b and b3a4e5d.

📒 Files selected for processing (1)

• backend/app/crud/evaluations/score.py

💤 Files with no reviewable changes (1)

• backend/app/crud/evaluations/score.py

---

📝 Walkthrough

Walkthrough

Numeric v2 judge metric trace scores now include verdict bands derived from rounded scores. The score module defines thresholds and enum values, fast evaluation populates verdicts, tests cover boundaries and exclusions, and evaluation documentation describes the contract.

Changes

Evaluation verdict bands

Layer / File(s)	Summary
Verdict mapping and trace schema backend/app/crud/evaluations/score.py, backend/app/tests/crud/evaluations/test_score.py	Defines VerdictEnum, 0.3/0.6 thresholds, score mapping behavior, and an optional TraceScore.verdict field with boundary and serialization tests.
Trace verdict population and validation backend/app/crud/evaluations/fast.py, backend/app/tests/crud/evaluations/test_fast_judge.py, docs/wiki/modules/evaluations.md	Adds verdicts to numeric v2 trace scores, verifies exclusions for cosine and N/A entries, and documents the resulting trace behavior.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• ProjectTech4DevAI/kaapi-backend#1004: Adds native LLM judge numeric scores to fast evaluation trace scores, which this change extends with verdict bands.
• ProjectTech4DevAI/kaapi-backend#1058: Adds the prompt-adherence judge metric in the same fast evaluation scoring pipeline.
• ProjectTech4DevAI/kaapi-backend#1095: Also modifies the score and fast evaluation modules to add verdict bands to v2 judge metric trace scores.

Suggested labels: enhancement

Suggested reviewers: akhileshnegi

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Linked Issues check	⚠️ Warning	The PR adds and tests verdict mapping for v2 scores, but it does not show verdicts in evaluation run views as required by issue [#1092].	Update every evaluation run row view or serializer that displays score and reason to also display the computed verdict.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the main change: adding verdict categorization for evaluations.
Out of Scope Changes check	✅ Passed	The backend logic, tests, and documentation directly support the verdict field and threshold requirements.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/evals-verdict-field

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

OpenAPI changes   ⚪ No API surface changes

Note

This PR does not modify the API contract.

_{main ↔ a172d73c · generated by oasdiff}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

backend/app/crud/evaluations/score.py (1)

14-14: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use StrEnum to satisfy Ruff UP042.

Line 14 triggers the configured Ruff rule; StrEnum preserves the existing string serialization and is available with the required Python 3.11+ baseline.

Proposed fix

-from enum import Enum
+from enum import StrEnum

-class VerdictEnum(str, Enum):
+class VerdictEnum(StrEnum):

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/app/crud/evaluations/score.py` at line 14, Update VerdictEnum to
inherit from the standard library StrEnum instead of combining str with Enum,
preserving its existing members and string serialization behavior while
satisfying Ruff UP042.

Sources: Coding guidelines, Linters/SAST tools

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@backend/app/tests/crud/evaluations/test_fast_judge.py`:
- Around line 1091-1185: Add return annotations to the three test methods in
backend/app/tests/crud/evaluations/test_fast_judge.py lines 1091-1185, and
annotate the nested _judge(params) callback parameter and return type. In
backend/app/tests/crud/evaluations/test_score.py lines 25-42, add self:
TestVerdictFromScore to each TestVerdictFromScore test method.

---

Nitpick comments:
In `@backend/app/crud/evaluations/score.py`:
- Line 14: Update VerdictEnum to inherit from the standard library StrEnum
instead of combining str with Enum, preserving its existing members and string
serialization behavior while satisfying Ruff UP042.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 16cc1f72-0090-4fdf-b809-aab74f6fb049

📥 Commits

Reviewing files that changed from the base of the PR and between 89132d7 and 6cd310b.

📒 Files selected for processing (5)

• backend/app/crud/evaluations/fast.py
• backend/app/crud/evaluations/score.py
• backend/app/tests/crud/evaluations/test_fast_judge.py
• backend/app/tests/crud/evaluations/test_score.py
• docs/wiki/modules/evaluations.md

[codecov]

Codecov Report

❌ Patch coverage is 98.36066% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...kend/app/tests/crud/evaluations/test_fast_judge.py	97.14%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[github-actions]

🎉 This PR is included in version 1.4.0-main.5 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀