Two million users, one agent: the failures we did not test for

Corvid’s research agent reads a user’s sources, plans a search and writes a cited answer. Before launch it scored well on every evaluation the team had. Within a month of reaching two million people it had found a dozen ways to go wrong that nobody had imagined: citing a document the user had deleted, looping on a single paywalled page, and answering a third of its Portuguese speakers confidently in Spanish.

Kenji walks through the incidents in order, what each one looked like in the logs, and what the team changed afterwards. The answer was rarely a better prompt. It was usually better plumbing: a trace for every step the agent takes, staged rollouts for every model change, and a review queue fed by real sessions rather than invented ones.

The talk ends with the rollout system Corvid uses today, which lets the team switch models weekly with a clear view of what got better and what got worse.

What you will take away

• A catalogue of agent failures that only appear at scale, and the early signals for each
• How to turn real user sessions into an evaluation set without breaking privacy promises
• A staged rollout pattern for model changes that a small team can run every week