Build an evaluation suite for an LLM feature in two hours

Most teams shipping language model features test them by trying a few prompts and deciding the answers look fine. That works until the model changes, the prompt changes or users do something new. Then nobody can say whether the feature got better or worse.

Elif and Kenji run evaluation for two very different products: Tidewater’s analytical assistant and Corvid’s research agent. In this workshop they guide you through building a small, honest evaluation suite for a feature of your own, or for a sample feature we provide. You will collect and clean real test cases, write graders that check what matters rather than surface wording, calibrate a model-based grader against your own judgement, and run the suite against two models to see where they differ.

By the end you will have a suite that runs in minutes, a score you trust, and a clear idea of what to add next.

What you will take away

• A working evaluation suite for one feature, built on cases you collected yourself
• How to write and calibrate graders, including when not to use a model as the judge
• A routine for growing the suite from real user sessions after launch

Before you come

• A laptop with Python 3.12 or later and a code editor
• An API key for at least one hosted language model, with a little credit on it
• Optional: twenty to fifty real inputs from a feature you work on, with anything sensitive removed
• We email a starter repository a week before the conference; please clone it before you arrive