Small models next to the data: running inference in every customer's region

When Halden’s largest European customers asked for every byte of their support data to stay in their own region, sending it all to one large hosted model was no longer an option. The team had a choice: offer a lesser product in some regions, or rethink how its AI features ran.

Lucas led the second option. Halden now runs a set of small, fine-tuned models in six regions, beside the data they work on, with a larger model used only where it measurably helps and the customer allows it. Lucas walks through the serving stack, the routing that picks a model for each request, the fine-tuning pipeline, and the evaluation that decides whether a small model is good enough for a given task.

The results were better than expected: lower cost per reply, faster responses, and quality within a couple of points on most tasks. Some tasks still need the large model, and he explains which.

What you will take away

• When a small, fine-tuned model can replace a large hosted one, and how to prove it
• A routing design that picks the right model for each request
• The operational cost of running inference in many regions, with real figures