Evaluating LLM applications: metrics, LLM-as-judge and benchmarks
When you ship an LLM application, how do you know it is getting better, not worse? Unlike a classifier where accuracy tells you everything, LLM applications produce open-ended outputs that require multi-dimensional evaluation. This lesson covers the full evaluation stack: from automatic metrics you can run in CI, to LLM-as-judge that scales human-level assessment, to the public benchmarks that let you compare your model against the field.