Your evals are lying to you: measuring models on real work
Public benchmarks say models are nearly perfect at analysis. A benchmark built from real company data says otherwise. Elif Demir on what it takes to measure models honestly.
- Thursday 10 June
- Day 2
- 11:40–12:20
- 40 minutes
- Studio
- First floor, river side
- Talk
- Advanced
Ask a language model to answer a question from a clean, well-documented public dataset and it will usually get it right. Ask it the same question of a real company’s tables, with cryptic column names, duplicated customers and a revenue field that changed meaning in 2023, and its accuracy roughly halves.
Elif built the benchmark Tidewater uses to test models against this kind of real, messy work. She explains how it was put together, from anonymised customer schemas to questions written by working analysts, and what it revealed: models that look interchangeable on public benchmarks differ widely here, and some popular prompting techniques make things worse.
She also covers the measurement problems that affect anyone evaluating a model: leakage, grader bias, answers that are right for the wrong reasons, and why a single accuracy figure hides most of what you need to know.
What you will take away
- How to build an evaluation set from your own domain that resists leakage
- The most common ways evaluation scores mislead, and how to check for each
- How to report model quality so decisions rest on the right numbers
