System Observability, Telemetry, and Site Reliability

Four signals and the question each one answers

Monitoring asks known questions. Is the error rate above one percent? Is the disk above 90 percent? Observability asks unknown ones. Why do requests from this one tenant time out only when they hit the cache warm path?

The difference is not the tool, it is whether the data carries enough dimensions to answer a question that nobody wrote down in advance. Four signal types together carry those dimensions, and each answers a different class of question.

Metrics are numbers aggregated over time. They answer "how much" and "how often" cheaply, at constant storage cost per series, for any time range. They cannot answer "which request" because aggregation destroyed the individual.

Traces follow one request across services. They answer "where did the time go" and "what called what". Their cost grows with request volume, which is why sampling exists.

Logs record discrete events with arbitrary detail. They answer "what exactly happened at this instant". They are the most expensive signal per unit of insight and the most complete.

Profiles attribute resource use to code. They answer "which function spends the CPU" and "which allocation site holds the memory", continuously and in production. Continuous profiling entered public alpha as an OpenTelemetry signal in March 2026, and eBPF-based collectors run it below one percent CPU overhead.

Metrics and the cardinality trap

A metric series is one time series, identified by a name and a set of label values. Storage and query cost scale with the number of distinct series, not with the number of data points, because each series carries its own index entry and its own chunk.

Cardinality is the product of the distinct values of every label. A metric with 10 services, 20 endpoints, 5 status codes, and 50 instances has 50,000 series, which is fine. Add a user_id label with a million values and it has 50 billion, which destroys the monitoring system.

The failure mode is specific and worth naming. A high-cardinality label appears in a pull request, passes review because the label is obviously useful, deploys, and the metrics backend runs out of memory within an hour. The service is healthy and the monitoring for every other service is down. Put a cardinality limit in the ingestion path, and alert on series growth rate.

The rule is to keep unbounded identifiers out of labels and put them in traces and logs, where the storage model tolerates them. Exemplars bridge the gap: a histogram bucket carries a sample trace identifier, so a click on the slow bucket in a dashboard opens an actual slow trace.

Four metric types cover almost everything. A counter only increases, and the query layer computes a rate from it. A gauge goes up and down. A histogram counts observations into buckets, which is what makes percentiles possible. A summary computes quantiles at the client, which makes them impossible to aggregate correctly.

Percentiles, honestly

An average latency is close to useless. A service with 99 percent of requests at 10 milliseconds and 1 percent at 5 seconds has an average of 60 milliseconds, which describes no request that ever happened. Users experience the distribution, and the tail of the distribution is what they complain about.

Percentiles do not average. The p99 of two servers is not the average of their two p99 values, and no arithmetic on the two numbers recovers the true value. The only correct aggregation merges the underlying histograms and then computes the percentile from the merged buckets. This is why a system that reports precomputed quantiles per instance cannot produce a correct service-level percentile, and why histogram-based metrics are the right choice.

Bucket boundaries determine accuracy. A histogram with buckets at 10, 100, and 1,000 milliseconds reports a p99 somewhere between 100 and 1,000 and cannot do better. Use exponential buckets with a fixed relative error, which is what native histograms in Prometheus and the exponential histogram in OpenTelemetry provide. They give a bounded relative error, typically a few percent, at a fixed number of buckets, and they merge exactly.

Chapter 4 showed that tail latency amplifies across a fan-out. A page that calls one hundred services in parallel, each with a 100 millisecond p99, gives a user-visible p50 of 100 milliseconds. For any service behind a fan-out, the p99.9 is the number that matters and the p50 is decoration.

Distributed tracing

A trace is a tree of spans. Each span has a name, a start time, a duration, a parent, and a set of attributes. The trace identifier and the current span identifier propagate with every outbound call, which is how the tree gets built across process boundaries.

The propagation format is standardized. W3C Trace Context defines a traceparent header carrying the version, the 16-byte trace identifier, the 8-byte parent span identifier, and flags, plus a tracestate header for vendor data. Every modern library emits and accepts it, which means that cross-organization tracing works when both sides agree to pass the headers through.

  trace 4bf92f3577b34da6a3ce929d0e0e4736                total 412 ms
  |
  +-- api-gateway  POST /checkout                       [============] 412
      |
      +-- auth  ValidateToken                           [=]            8
      |
      +-- cart  GetCart                                 [==]          21
      |   +-- redis  GET cart:4471                      [=]            2
      |
      +-- pricing  Quote                                [=====]       67
      |   +-- catalog  BatchGet (x40, sequential)       [====]        61  <--
      |
      +-- payments  Authorize                           [==========] 298  <--
          +-- psp-adapter  HTTP POST                    [=========] 287
              +-- dns  resolve psp.example.com          [====]       118  <--

  Three findings in one picture:
    payments dominates, and almost all of it is one external call
    DNS resolution takes 118 ms, which means no caching or a bad resolver
    catalog is called 40 times in sequence: an N+1 that a batch call fixes

Sampling controls cost. Head-based sampling decides at the root, before anything runs, usually with a fixed probability. It is cheap and it discards errors and slow requests at the same rate as everything else, which is exactly backward.

Tail-based sampling buffers the complete trace and decides after it finishes. It keeps every trace with an error, every trace above a latency threshold, and a small percentage of the rest. The result is a set of traces that is far more useful per byte stored. The cost is that the collector must buffer spans until the trace completes, which needs memory and needs every span of one trace to reach the same collector instance.

OpenTelemetry is the vendor-neutral standard for all of this. The API is what application code calls. The SDK implements it and handles batching and sampling. The OTLP protocol carries data over gRPC or HTTP. The Collector receives, processes, and exports to any backend, which is where tail sampling, attribute redaction, and cardinality limiting belong.

Semantic conventions are the part that makes data portable. They define the attribute names for HTTP, for databases, for messaging, and for other domains, so that a dashboard works across services written by different teams. Their maturity varies by domain: HTTP and several core areas are stable, Kubernetes attributes reached release candidate in 2026, and the generative AI conventions were still in development through mid-2026.

Logs and profiles

Structured logging emits key-value pairs rather than formatted sentences. A line that carries event=payment_failed order_id=4471 psp=stripe code=card_declined duration_ms=287 trace_id=4bf92f... is queryable. The same information in prose needs a regular expression that breaks whenever someone edits the message.

Include the trace identifier in every log line. That one field turns logs and traces into one dataset: find the slow trace, pivot to its logs, read the exact error. Without it, correlation is a manual search by timestamp and service name, which fails under load because many requests share a timestamp.

Log volume is the largest observability cost in most organizations, and most of it is never read. Three controls work. Sample high-volume informational logs at a low rate and keep all warnings and errors. Set retention by value: days for debug, weeks for informational, months for audit. Move the audit subset to a separate, cheaper, append-only store, because it has a different access pattern and a different regulatory requirement.

Continuous profiling closes the loop between a symptom and a line of code. An eBPF-based profiler samples stacks across every process on a node with no code changes and no restart, at roughly one percent CPU. It resolves symbols cleanly for compiled languages, and interpreted and JIT runtimes need extra work for readable stacks. Parca and Pyroscope both implement this, both support OTLP, and both understand trace identifiers, which allows a profile to be filtered to the requests in one trace.

eBPF also gives protocol-level visibility with no instrumentation. Tools in this family parse HTTP, gRPC, DNS, and database protocols at the kernel boundary and produce request-level telemetry for services that were never instrumented. It is the fastest way to get visibility into a system that you inherited. It does not replace application tracing, because the kernel cannot see business context such as a tenant identifier or a feature flag.

Service level objectives and error budgets

A service level indicator, or SLI, is a measurement of one aspect of service quality, expressed as good events divided by valid events. A service level objective, or SLO, is a target for that indicator over a window. An error budget is the allowed shortfall, which is one minus the objective.

The budget reframes reliability as a quantity to spend. A 99.9 percent availability objective over 30 days gives 43.2 minutes of budget. Within budget, the team ships. Outside budget, the team fixes reliability before shipping features. The rule turns an argument about priorities into an arithmetic fact, which is the entire point.

Choose the objective from what users need, not from how many nines sound impressive. Each additional nine costs roughly an order of magnitude more engineering, and it is wasted when the network between the user and the service is less reliable than the objective. A mobile application on a cellular network cannot deliver five nines to its users whatever the backend does.

Burn rate alerting is the correct way to page on an objective. Burn rate is the multiple of budget consumption that exhausts the budget exactly at the end of the window. A burn rate of one uses the budget evenly. A burn rate of fourteen point four consumes two percent of a 30-day budget in one hour.

Use several windows at once. A fast burn over a short window catches an outage. A slow burn over a long window catches a steady degradation that no single moment triggers. Pair each long window with a short one so that the alert resolves quickly after the problem stops.

Alerting discipline

Alert on symptoms, not on causes. A symptom is something a user experiences: more errors, higher latency, missing data. A cause is an internal state: high CPU, a full queue, a restarted pod. Causes produce false pages, because a system can be healthy with high CPU and broken with low CPU. Symptoms produce true pages, because a symptom is by definition a user-visible problem.

Every page must be actionable, urgent, and novel. Actionable means that a human can do something now, so an alert whose runbook says "wait and see" belongs in a ticket. Urgent means that it cannot wait until morning. Novel means that the same alert has not fired ten times this week, because a repeated alert trains people to ignore it.

Alert fatigue is a measurable failure. Track the number of pages per on-call shift and the fraction that led to action. More than two pages per shift is unsustainable, and an actionable fraction below half means that the alerting is training people to ignore it. Both numbers belong in a regular review with the same seriousness as latency.

Every alert needs a runbook, and the runbook needs three sections: what this alert means in plain terms, what to look at first, and what actions are safe. Write it when you create the alert, because nobody writes it at three in the morning.

Incident response and what follows

An incident has roles. The incident commander coordinates and decides, and does not debug. The operations lead makes changes to the system. The communications lead updates stakeholders. The scribe records the timeline. One person can hold several roles in a small incident. Name the roles out loud regardless, because an incident with no named commander has five people debugging and nobody deciding.

Mitigate before you diagnose. Roll back, fail over, shed load, or disable the feature flag. Understanding can wait, and a user-visible outage cannot. Teams that debug first routinely extend an outage by an hour, to learn something that the same data gives them afterward.

A postmortem is blameless or it is worthless. A culture that assigns blame gets less information, because people omit the parts that make them look bad, and those parts are usually where the systemic weakness lives. The output is a set of action items with owners and dates, and a review of whether the previous incident's action items were completed.

Four measures describe incident performance. Time to detect runs from the start of impact to an alert firing. Time to acknowledge runs from the alert to a human responding. Time to mitigate runs to the restoration of user-visible function. Time to resolve runs to the permanent fix. Time to detect is the one that observability work improves directly, and it is usually the largest.

Sources and further reading

Betsy Beyer and others, "Site Reliability Engineering" and "The Site Reliability Workbook", the second of which contains the multi-window burn rate alerting material. Alex Hidalgo, "Implementing Service Level Objectives". Charity Majors, Liz Fong-Jones, and George Miranda, "Observability Engineering". Gil Tene's talks on coordinated omission and the HdrHistogram documentation. The OpenTelemetry specification and semantic conventions. The Parca and Pyroscope documentation for continuous profiling at scale.