evals-engineering

Use cultivar to test your Agent Skills and Docs by running them in sandboxes, and across different agents.

Optimize and run consistent AI Functions.

A local-first personal assistant that shows the four pillars behind every serious agent: Harness · Loop · Memory · Eval/LLM-Ops. No frameworks hiding the critical parts.

Pithom Labs Blog

Salary benchmarks, CI/CD regression testing pipelines, OpenTelemetry standards, and enterprise rubrics for LLM Evals Engineers in 2026.

Local AI server for refit (OpenAI, Anthropic, Gemini).

Awesome list for eval engineering (evals engineering): tools that automate LLM evals and agent evals, turn traces into test cases, and calibrate LLM-as-a-judge.

Typed, unit-test-style evaluations for LLMs, agents, and other ML applications—with flexible datasets, scoring, and local reports.