Trace a real request end to end: observability from zero

Most teams have logs, some have metrics and a few have traces, but very few can follow one slow request from the user’s click through every service and query it touched. That is the question you most need answered at three in the morning.

Lucas teaches the tracing course every new Halden engineer takes, compressed into two hours. You start with a small but realistic app, a web front end, an API, a job queue and a database, with no instrumentation at all. Step by step you add distributed tracing with OpenTelemetry, structured logs that join up with the traces, and a handful of metrics that are actually worth alerting on.

Then the app breaks. Three bugs have been planted in it, including a slow query that only appears under load and a retry storm in the queue. You find each one using only what you have built.

What you will take away

• A working, instrumented app to use as a reference for your own systems
• How to connect traces, logs and metrics so that each one leads to the others
• A debugging routine that starts from a symptom and ends at a line of code

Before you come

• A laptop with Docker, or another container runtime, installed and running
• Git and a code editor; examples are in TypeScript and Go, and you can follow in either
• At least 8 GB of free memory for the local stack
• Clone the workshop repository and run its setup script before the day, so nobody is downloading images in the room