A hands-on companion to Anthropic's engineering guide "Demystifying evals for AI agents", practiced on a real, human-labeled dataset and mirrored in LangSmith.
The agent is deliberately tiny. The point of this repo is not the agent, it is the evaluation methods: how to tell whether an LLM agent actually works, how much to trust that verdict, and whether the eval itself is any good.
A two-step financial-news sentiment agent (Claude Haiku 4.5):
classify_sentiment: label a sentence asnegative/neutral/positive(structured output, so the label is always one of the three).explain_reasoning: explain why, pointing at the words that drove the decision. This step exists so there is a real transcript to grade, not just a bare label.
graph.png is generated by make_graph.py with langchain_core's Graph,
wiring in the real functions from agent.py / graders.py.
Dataset: Financial PhraseBank (Malo et al., 2014): 4,846 financial-news sentences labeled by humans (604 negative / 2,879 neutral / 1,363 positive). It ships ground truth, so it works as a ready-made golden set.
Two graders that see different things, neither enough on its own:
| Grader | Type | Needs reference? | Catches |
|---|---|---|---|
accuracy_grader |
code-based | yes (reference-based) | wrong label |
reasoning_quality_grader |
LLM-as-judge (Claude Sonnet 5) | no (reference-free) | reasoning that doesn't support the label, invented sentiment, self-contradiction |
The judge is a stronger model than the agent it grades (Sonnet judging Haiku), and it deliberately never sees the reference label, so the two graders don't overlap. Being reference-free also means the judge could run as an online evaluator on real traffic, where no ground truth exists.
| Script | Question it answers | Concept |
|---|---|---|
run_eval.py |
How accurate is the agent, and where does it get confused? | grader stack, confusion matrix |
run_pass_k.py |
Ask the same thing 5 times: same answer? | pass@k vs pass^k |
run_calibration.py → run_calibration_compare.py |
Can I trust the LLM judge? | grader calibration / evaluator alignment |
run_module6_langsmith.py |
Can the eval tell a good agent from a bad one? | eval-suite health, baseline experiment |
explore_dataset.py |
What does the data look like? | — |
Run on 20 Oct 2026. All samples are small (8–20 sentences), see Limitations.
| Metric | Result |
|---|---|
| Label accuracy (code grader) | 10/20 = 50% |
| Reasoning supports label (LLM judge) | 11/20 = 55% |
reference \ predicted negative neutral positive
negative 2 0 0
neutral 4 4 5
positive 0 1 4
- 9 of the 10 errors are on neutral sentences. The agent reads sentiment into purely factual statements (e.g. a plant's production capacity → "positive").
- The judge caught what the code grader can't: in 3 cases the label said
negativewhile the agent's own reasoning said the sentence was neutral or factual. An exact-match grader only sees the label, so it misses self-contradiction entirely.
| pass@5 | pass^5 |
|---|---|
| 40% | 40% |
- pass@k = at least one of k trials correct (optimistic, "can it?").
- pass^k = all k trials correct (the reliability a single real call gets).
The gap between them measures instability. Here the gap is zero: every sentence was either right 5/5 or wrong 5/5. The errors are systematic, not random, so retries would not help; a clearer definition of "neutral" in the prompt would.
A human graded 8 transcripts before seeing the judge's verdicts (to avoid anchoring), answering the same question the judge answers. The judge then graded the exact same frozen transcripts (re-running the agent would produce different text and silently break the comparison).
Agreement: 8/8 = 100%. That includes both cases the human rejected, which matters more than the headline number: with 6 "yes" vs 2 "no", a judge that always says "yes" would still score 75%.
Same dataset, same evaluators, two LangSmith experiments:
| Target | Accuracy | Reasoning supported |
|---|---|---|
| Real agent | 10/12 = 83% | 12/12 = 100% |
Majority-class baseline (always neutral) |
5/12 = 42% | 0/12 = 0% |
A clear gap means the eval discriminates between a good and a bad agent. If the two had scored about the same, the eval would be broken or saturated, and none of its other numbers could be trusted. The baseline gets some labels right by luck (the dataset is ~59% neutral), but it can never justify them, which the judge sees at once.
- Small samples. Accuracy was 50% on one sample and 83% on another. The agent didn't change, the sentences did. With 12–20 examples that much variance is expected; these numbers illustrate the methods, they are not a benchmark.
- Random, not stratified, sampling. The 20-sentence sample had only 2 negative sentences, so negatives are barely tested.
- The reasoning is post-hoc.
explain_reasoningreceives the already chosen label, so the judge grades consistency between label and explanation, not the model's actual decision process. - Calibration n=8 is a good sign, not proof.
The two sources name the same ideas differently:
| Anthropic guide | LangSmith | In this repo |
|---|---|---|
| task | example (inputs + reference outputs) | one PhraseBank sentence + label |
| set of tasks / golden set | dataset | PhraseBank sample |
| trial | run (several = repetitions) | one run_agent() call |
| transcript | trace / run outputs | AgentTranscript |
| outcome | (what the evaluator inspects) | the final label |
| grader | evaluator | graders.py |
| grader verdict | feedback (score + comment) | True/False + judge comment |
| evaluation harness | evaluate() → experiment |
the run_*.py scripts |
| grader calibration | evaluator alignment (annotation queues) | run_calibration*.py |
| eval-suite health | baseline experiment + comparison view | run_module6_langsmith.py |
Requires Python 3.12 and uv.
uv sync
cp .env.example .env # add your ANTHROPIC_API_KEY and LANGCHAIN_API_KEY
uv run python run_eval.py
uv run python run_pass_k.py
uv run python run_calibration.py
uv run python run_calibration_compare.py
uv run python run_module6_langsmith.py # creates a small dataset + 2 experiments in LangSmith
uv run python make_graph.py # regenerates graph.pngEvery script is read-only apart from LLM API calls (and, for the Module 6 script, writing a dataset and experiments to your LangSmith account).
- Anthropic, "Demystifying evals for AI agents".
- Malo, P., Sinha, A., Korhonen, P., Wallenius, J., Takala, P. (2014). Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology. Financial PhraseBank is released under CC BY-NC-SA 3.0.
