Evaluations (evals) are tests for AI. They're automated quality checks, assessing how well AI performs its intended tasks. They have all the same benefits are regular software testing - confidence, trust, accuracy, reliability, etc.
Since AI is [[20260326100028-ai-non-deterministic]], evals are really important. It's how you can take an unpredictable behemoth, and at least get to probability. This also makes writing evals a bit harder, as there's so simple input/output test, there's a lot more grey. Qualitative rather than quantative.
There are three different types of evals. Different types of AI need different types of evals, much like any other software tests. Over time, your evals need to improve with your AI tooling. Much like [[20260610023222-test-driven-development-tdd]], there's methodologies around eval-driven development: https://vercel.com/blog/eval-driven-development-build-better-ai-faster
These are simple assertions, pass/fail checks like unit tests.
This is using human judgement to get an understanding if things are working properly. Not very automated, and not very fast. Expected to be subjective.
Passing the results of your prompts to another AI tool, and using that as a judge. This can be written in an assertion style, giving the second AI tool a 'ground truth' to evaluate against.
https://www.thoughtworks.com/en-au/insights/decoder/a/ai-evals https://www.aihero.dev/what-are-evals https://hamel.dev/blog/posts/evals/ https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
[[20250811104331-model-context-protocol-mcp]] [[20260306121850-ai-agent-skills]]