Skip to content

About

Hands-on agent evaluation lab (Anthropic's 'Demystifying evals for AI agents' + LangSmith): code + LLM-judge grader stack, pass@k vs pass^k, judge calibration against human labels, and an eval-health check vs a baseline — on a Haiku financial-sentiment agent.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Agent Eval Lab — Financial Sentiment

A hands-on companion to Anthropic's engineering guide "Demystifying evals for AI agents", practiced on a real, human-labeled dataset and mirrored in LangSmith.

The agent is deliberately tiny. The point of this repo is not the agent, it is the evaluation methods: how to tell whether an LLM agent actually works, how much to trust that verdict, and whether the eval itself is any good.

The agent under test

A two-step financial-news sentiment agent (Claude Haiku 4.5):

  1. classify_sentiment: label a sentence as negative / neutral / positive (structured output, so the label is always one of the three).
  2. explain_reasoning: explain why, pointing at the words that drove the decision. This step exists so there is a real transcript to grade, not just a bare label.

Agent and grader stack

graph.png is generated by make_graph.py with langchain_core's Graph, wiring in the real functions from agent.py / graders.py.

Dataset: Financial PhraseBank (Malo et al., 2014): 4,846 financial-news sentences labeled by humans (604 negative / 2,879 neutral / 1,363 positive). It ships ground truth, so it works as a ready-made golden set.

Grader stack

Two graders that see different things, neither enough on its own:

Grader Type Needs reference? Catches
accuracy_grader code-based yes (reference-based) wrong label
reasoning_quality_grader LLM-as-judge (Claude Sonnet 5) no (reference-free) reasoning that doesn't support the label, invented sentiment, self-contradiction

The judge is a stronger model than the agent it grades (Sonnet judging Haiku), and it deliberately never sees the reference label, so the two graders don't overlap. Being reference-free also means the judge could run as an online evaluator on real traffic, where no ground truth exists.

One script per question

Script Question it answers Concept
run_eval.py How accurate is the agent, and where does it get confused? grader stack, confusion matrix
run_pass_k.py Ask the same thing 5 times: same answer? pass@k vs pass^k
run_calibration.py → run_calibration_compare.py Can I trust the LLM judge? grader calibration / evaluator alignment
run_module6_langsmith.py Can the eval tell a good agent from a bad one? eval-suite health, baseline experiment
explore_dataset.py What does the data look like? —

Results

Run on 20 Oct 2026. All samples are small (8–20 sentences), see Limitations.

1. Accuracy and failure pattern (run_eval.py, n=20)

Metric Result
Label accuracy (code grader) 10/20 = 50%
Reasoning supports label (LLM judge) 11/20 = 55%
reference \ predicted   negative   neutral  positive
negative                       2         0         0
neutral                        4         4         5
positive                       0         1         4
  • 9 of the 10 errors are on neutral sentences. The agent reads sentiment into purely factual statements (e.g. a plant's production capacity → "positive").
  • The judge caught what the code grader can't: in 3 cases the label said negative while the agent's own reasoning said the sentence was neutral or factual. An exact-match grader only sees the label, so it misses self-contradiction entirely.

2. Consistency (run_pass_k.py, 10 sentences × 5 trials)

pass@5 pass^5
40% 40%
  • pass@k = at least one of k trials correct (optimistic, "can it?").
  • pass^k = all k trials correct (the reliability a single real call gets).

The gap between them measures instability. Here the gap is zero: every sentence was either right 5/5 or wrong 5/5. The errors are systematic, not random, so retries would not help; a clearer definition of "neutral" in the prompt would.

3. Can we trust the judge? (run_calibration*.py, n=8)

A human graded 8 transcripts before seeing the judge's verdicts (to avoid anchoring), answering the same question the judge answers. The judge then graded the exact same frozen transcripts (re-running the agent would produce different text and silently break the comparison).

Agreement: 8/8 = 100%. That includes both cases the human rejected, which matters more than the headline number: with 6 "yes" vs 2 "no", a judge that always says "yes" would still score 75%.

4. Is the eval itself healthy? (run_module6_langsmith.py, n=12)

Same dataset, same evaluators, two LangSmith experiments:

Target Accuracy Reasoning supported
Real agent 10/12 = 83% 12/12 = 100%
Majority-class baseline (always neutral) 5/12 = 42% 0/12 = 0%

A clear gap means the eval discriminates between a good and a bad agent. If the two had scored about the same, the eval would be broken or saturated, and none of its other numbers could be trusted. The baseline gets some labels right by luck (the dataset is ~59% neutral), but it can never justify them, which the judge sees at once.

Limitations

  • Small samples. Accuracy was 50% on one sample and 83% on another. The agent didn't change, the sentences did. With 12–20 examples that much variance is expected; these numbers illustrate the methods, they are not a benchmark.
  • Random, not stratified, sampling. The 20-sentence sample had only 2 negative sentences, so negatives are barely tested.
  • The reasoning is post-hoc. explain_reasoning receives the already chosen label, so the judge grades consistency between label and explanation, not the model's actual decision process.
  • Calibration n=8 is a good sign, not proof.

Terminology: Anthropic guide ↔ LangSmith

The two sources name the same ideas differently:

Anthropic guide LangSmith In this repo
task example (inputs + reference outputs) one PhraseBank sentence + label
set of tasks / golden set dataset PhraseBank sample
trial run (several = repetitions) one run_agent() call
transcript trace / run outputs AgentTranscript
outcome (what the evaluator inspects) the final label
grader evaluator graders.py
grader verdict feedback (score + comment) True/False + judge comment
evaluation harness evaluate() → experiment the run_*.py scripts
grader calibration evaluator alignment (annotation queues) run_calibration*.py
eval-suite health baseline experiment + comparison view run_module6_langsmith.py

Running it

Requires Python 3.12 and uv.

uv sync
cp .env.example .env      # add your ANTHROPIC_API_KEY and LANGCHAIN_API_KEY
uv run python run_eval.py
uv run python run_pass_k.py
uv run python run_calibration.py
uv run python run_calibration_compare.py
uv run python run_module6_langsmith.py   # creates a small dataset + 2 experiments in LangSmith
uv run python make_graph.py              # regenerates graph.png

Every script is read-only apart from LLM API calls (and, for the Module 6 script, writing a dataset and experiments to your LangSmith account).

Credits

  • Anthropic, "Demystifying evals for AI agents".
  • Malo, P., Sinha, A., Korhonen, P., Wallenius, J., Takala, P. (2014). Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology. Financial PhraseBank is released under CC BY-NC-SA 3.0.

About

Hands-on agent evaluation lab (Anthropic's 'Demystifying evals for AI agents' + LangSmith): code + LLM-judge grader stack, pass@k vs pass^k, judge calibration against human labels, and an eval-health check vs a baseline — on a Haiku financial-sentiment agent.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages