Description
The service runs. It returns 200s in the demo, the deploy is green, and everyone moves on, and then it is 3am and the checkout is broken and you open the dashboard and every panel is green. The dashboard was measuring whether the servers were up, not whether anything worked, and those quietly became different things. You go to the logs and they are a wall of routine info-level chatter with the actual error buried somewhere you cannot grep for, printed as a plain string with no request id, so you cannot follow the one failing request through it. The request touched three services and you have no way to see where in that chain the time went or the error started. There is an alert, and it fired, but it fired because a disk crossed 80 percent, which nobody cares about, and it never fired for the thing that is actually down. An agent will hand you a service exactly like this: it runs, and it is invisible. The tooling stays silent about the difference, because a service with no logs, no metrics, no traces, and a disk-space alert deploys identically to one you can see into. What you lack is not a tool. It is the judgment to look at a running service and know whether production is visible, and to make it visible before the night you need it.
This book trains you to see what your software does in production, and to own that visibility the way a reviewer owns a diff. It starts from where most engineers actually are: the code ships, an agent may have written it, it runs, and nobody checked whether you could see it run. So the book teaches the three pillars from the ground up, using one service you instrument end to end across every chapter, a notes API called Quill with a summarize endpoint that calls out to a model, so the lessons compound instead of resetting. You learn structured logging that a person can search under pressure, with levels that mean something and a correlation id that lets you follow one request through the noise. You learn the metrics that actually catch outages, RED and USE, counters and gauges and histograms, and why a p99 latency graph tells you what an average hides. You learn distributed tracing, following a single request across services to find where the time and the failure live. You learn to build dashboards that turn red when users are hurting instead of staying green through an outage, and alerts that page a human for a broken symptom rather than a full disk, without burning the on-call rotation out on noise. You learn error tracking that groups and contextualizes real failures instead of drowning you in duplicates, on-call and incident visibility, when to reach for a log versus a metric versus a trace, and what observability costs so the bill and the cardinality do not surprise you. The through-line is a reviewer’s question asked in daylight, calmly, on a schedule: if production broke right now, could you tell what was wrong, and how fast. For engineers who are going to ship services either way and want to be the one who can see them when it counts.
Written for you if
This book is for: engineers who ship code that now runs in production, often code an AI agent wrote, and who own the question nobody answered while it was being built, which is whether they can actually see what the software is doing once real users hit it. You can write a service and get it deployed. What you cannot yet do is tell, at 3am with the checkout broken, what is failing and why, because the service logs nothing you can search, exposes no metric tied to whether it works, carries no trace across the three services a request touches, and pages you when a disk is 80 percent full while staying dead silent when payments stop. This book assumes you can read code and deploy a service; it does not assume you have ever instrumented one. It teaches the reviewer’s posture toward production visibility: not how to install a tool, but how to judge whether you could see an outage coming and see it clearly once it started.
What you’ll get inside
- Chapter 1: The service runs. Can you see what it is doing?
- Chapter 2: Your logs are a wall of noise at the exact moment you need to grep them
- Chapter 3: The dashboard was green while the servers were fine and nothing worked
- Chapter 4: The request touched three services and you have no idea where it broke
- Chapter 5: The green dashboard during the outage, and how to build one that turns red
- Chapter 6: The pager went off for a full disk and stayed silent for the outage
- Chapter 7: The same error happened ten thousand times and you saw it once
- Chapter 8: It is your night on call. What do you actually reach for first?
- Chapter 9: A log, a metric, or a trace: which one answers the question you have?
- Chapter 10: The observability bill that cost more than the thing it was watching
- Chapter 11: Wiring one real service so you can see everything it does
- Chapter 12: From the engineer who shipped it to the one who can see it
