Telemetry, Observability 3.0, and Reliability Engineering

8.1 OpenTelemetry Standards and Context Propagation

Monitoring answers questions somebody wrote down in advance. Is the error rate above one percent? Is the disk above 90 percent? Observability answers questions nobody predicted, such as why requests from one tenant time out only on the cache warm path.

The difference is not the tool. It is whether the telemetry carries enough dimensions to answer a question that was never anticipated, and whether those dimensions survive to query time.

The wide event

A wide event is one structured record per unit of work. It carries every dimension known at the end of that work. That means the tenant, the endpoint, the outcome, the duration, the cache state, the feature flags, the build identifier, the queue wait, the database time, and the trace identifier.

The argument for it is about when context gets assembled. In the three-pillar model, context is reconstructed at read time by correlating a metric with a log with a trace. That requires the correlating keys to exist in all three, and it fails when one of them dropped the dimension. In the wide event model, context is injected at write time, so the record is already complete when it arrives.

Take a question such as "which tenants on build 9f3a saw a cache miss and a slow database read on the checkout endpoint". With wide events that is one query over one dataset. With three pillars it is a join across three systems that do not share a schema.

The cost is storage and the tooling to query high-cardinality data. A metrics backend cannot hold a tenant identifier as a label, because cardinality is the product of label values. A columnar event store can, because it stores each field as a column and scans rather than indexes. That difference in storage model is the actual technical content behind the version numbers.

Trace context propagation

A trace is a tree of spans. Each span has a name, a start time, a duration, a parent, and attributes. The trace identifier and the current span identifier propagate with every outbound call, which is how the tree spans processes.

W3C Trace Context standardizes the format. The traceparent header carries four fields separated by hyphens: a version, a 16-byte trace identifier, an 8-byte parent span identifier, and one flags byte whose low bit records the sampling decision. A tracestate header carries vendor-specific key-value data and must be forwarded unchanged even by a system that does not understand it.

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             |  |                                |                |
             |  |                                |                +- flags: sampled
             |  |                                +- parent span id, 8 bytes
             |  +- trace id, 16 bytes, unique for the whole trace
             +- version

tracestate: vendorA=t61rcWkgMzE,vendorB=00f067aa0ba902b7
            forward this unchanged, even if you do not parse it

Propagation breaks at three specific places, and each break produces a trace that ends where the interesting part begins.

An asynchronous boundary loses context unless it is carried explicitly. A message published to a queue must carry traceparent in its headers. The consumer extracts it and starts a span with a link rather than a parent, because the consumer's work is causally related but not nested.

A thread pool or a coroutine scheduler loses context stored in thread-local state. Runtimes provide a context-propagating wrapper, and code that submits a raw task to a raw executor bypasses it.

A third-party library that builds its own HTTP client bypasses the instrumented one. The result is a span for the call with no child spans for what it did.

The collector pipeline

The OpenTelemetry Collector is a separate process that receives, processes, and exports telemetry. Its pipeline has four stages: receivers, processors, exporters, and connectors that route one pipeline's output into another.

The collector is where the operational controls belong, because it is the one place that sees all telemetry and can be changed without redeploying applications. Four processors do most of the work.

The batch processor groups records to cut export overhead, and it is required rather than optional. The memory limiter processor refuses data when the collector approaches a memory ceiling, which stops the collector from becoming the thing that dies. The attributes processor adds, redacts, and hashes fields, which is where personal data gets removed before it leaves the trust boundary. The tail sampling processor decides which traces to keep after they complete.

  APPLICATION           AGENT (daemonset, per node)      GATEWAY (deployment)
  +------------+        +--------------------------+     +------------------+
  | SDK        |  OTLP  | receivers: otlp, host     |     | receivers: otlp  |
  | spans      |------->| processors:               |OTLP | processors:      |
  | metrics    |        |   memory_limiter          |---->|   memory_limiter |
  | logs       |        |   resource (k8s metadata) |     |   tail_sampling  |
  | profiles   |        |   batch                   |     |   attributes     |
  +------------+        | exporters: otlp -> gateway|     |   batch          |
                        +--------------------------+     | exporters:       |
                                                         |   backend, S3    |
   Agent adds node and pod metadata cheaply.              +------------------+
   Gateway does tail sampling, because it needs every span of a trace
   in one place. Route by trace id to a gateway instance, or tail
   sampling drops spans it never saw.

Tail sampling is the control that changes the economics. Head sampling decides at the root, before anything runs, usually with a fixed probability, so it discards errors and slow requests at exactly the same rate as everything else. Tail sampling buffers the complete trace and decides after it finishes. It keeps every trace with an error, every trace above a latency threshold, a sample of the rest, and every trace for a tenant under investigation.

The deployment constraint is the one that catches people. Every span of one trace must reach the same collector instance for the decision to see the whole trace. That requires routing by trace identifier, which the load balancing exporter provides, and it makes the gateway tier stateful in a way the agent tier is not.

Semantic conventions

Semantic conventions define the attribute names, so that a dashboard works across services written by different teams in different languages. Without them, one service emits http.status, another status_code, and a third response.code, and no query spans all three.

Their maturity varies by domain, and the honest 2026 picture is mixed. HTTP and several core areas are stable. Kubernetes attributes reached release candidate in 2026. Database conventions are partly stable. Messaging remains in development. The generative AI conventions were still marked development through mid-2026, every attribute of them, which matters for anyone building agent telemetry on the assumption that the names are settled.

Treat unstable conventions as a versioning problem. Record which convention version a service emits, and keep a translation layer in the collector so that a rename does not break every dashboard at once.

8.2 eBPF-Based Low-Overhead Kernel Tracing

Application instrumentation sees what the application knows. It does not see the time a thread spent waiting for a lock, blocked on a disk read, descheduled, or stalled on a page fault. eBPF sees all of it, with no code change and no restart.

Volume I covered the eBPF verifier, map types, and attach points. This section covers what to do with them when a service is slow and nobody knows why.

Continuous profiling

A profiler samples stacks at a fixed frequency, typically 99 hertz to avoid aliasing with timer interrupts, and aggregates them into a flame graph. An eBPF profiler does this for every process on a node at once, from outside the processes, with no agent inside them.

The cost is about one percent of CPU and a few hundred megabytes of agent memory. That is low enough to run permanently in production, which is the point: a profile taken during an incident is worth far more than one taken afterward on a healthy system.

Symbolization is where the practical difficulty lives. The eBPF path resolves stacks cleanly for compiled languages with frame pointers, which means Go, Rust, C, and C++ built appropriately. Interpreted and just-in-time runtimes need extra work. The JVM and Node.js need a perf map or an equivalent, so that generated code addresses map to method names. Python and Ruby need an unwinder that walks the interpreter's own frame structures. Check which of your runtimes the tool actually symbolizes before you commit to it.

Grafana Pyroscope 2.0, released in April 2026, rearchitected its storage to write profiles directly to object storage with no in-memory ingester tier and supports OTLP. Parca's agent became the basis for the OpenTelemetry eBPF profiler. Both understand trace identifiers, so a profile can be filtered to the requests in one trace, which connects a slow trace to the exact function that consumed the time.

Off-CPU analysis

A flame graph from a CPU profiler shows where a thread spent CPU. It shows nothing about where a thread spent time not running, which for most services is where the latency is.

Off-CPU analysis instruments the scheduler. A program on the sched_switch tracepoint records the stack and the timestamp when a thread leaves the CPU and computes the duration when it returns. Aggregating by stack produces an off-CPU flame graph, which names the exact call site that blocked and the reason.

# Off-CPU time by stack, for one process, over 30 seconds.
# Anything over a few milliseconds is a candidate.
offcputime-bpfcc -p $(pgrep -f myservice) -f 30 > offcpu.folded
flamegraph.pl --title "Off-CPU" --countname us offcpu.folded > offcpu.svg

# Which syscalls block, and for how long, as a histogram.
funclatency-bpfcc -m 'vfs_read'
syscount-bpfcc -L -p $(pgrep -f myservice)

# Run-queue latency: time from runnable to running. This is the
# number that a CPU quota or a noisy neighbor makes large.
runqlat-bpfcc -m 10 1

# Lock contention, user-space futex path.
funclatency-bpfcc -u -m '/usr/lib/libc.so.6:pthread_mutex_lock'

The combination of an on-CPU profile and an off-CPU profile accounts for all wall-clock time. Together they answer the question that a single profiler cannot: whether a slow request spent its time computing, waiting on a dependency, waiting for a lock, or waiting for a CPU to become available.

Zero-instrumentation request tracking

An eBPF program attached at the socket layer sees every byte a process sends and receives. With a protocol parser it reconstructs HTTP requests, gRPC calls, DNS queries, and database statements, producing request-level telemetry for a service that was never instrumented.

That is the fastest way to get visibility into a system you inherited. It also has two limits that make it a complement to application tracing rather than a replacement.

It cannot see business context. The kernel knows the bytes on the socket and not the tenant identifier, the feature flag, or the campaign that caused the request. Those exist only in the application.

It cannot see inside TLS without help. Traffic encrypted before it reaches the socket is opaque at that layer. The workaround is a uprobe on the library's encrypt and decrypt functions, which reads the plaintext buffer before encryption. That works, it depends on the library version and symbol availability, and it is a decision with a security dimension that deserves an explicit review rather than a default.

8.3 Tail Latency Mitigation

The mean latency of a service describes no request that ever happened. Users experience the distribution, and they complain about its tail.

Why the tail dominates at scale

A request that fans out to several backends completes when the slowest one completes. Take a backend with a p99 of 100 milliseconds. A request touching 100 such backends in parallel hits at least one slow response with probability 1 minus 0.99 to the power 100, which is 63 percent.

The user-visible p50 therefore equals the backend p99. Adding more backends makes it worse. For any service behind a fan-out, the p99.9 is the number that matters and the p50 is decoration.

Read the table as a design constraint. A 100-way fan-out needs a backend p99.9 to deliver a usable p99, which is two orders of magnitude harder than tuning a p99.

Hedged and tied requests

A hedged request sends a second copy to another replica after a delay, and takes whichever answer arrives first. Set the delay at the p95, so that 95 percent of requests never hedge and the extra load is about 5 percent. The effect on the tail is large, because the second copy usually lands on a replica that is not experiencing whatever caused the first to be slow.

A tied request goes further. Send the request to two replicas at once, and tell each one about the other. Whichever replica starts work first sends a cancellation to the other. The extra work is bounded by the time it takes the cancellation to arrive rather than by the full request duration.

  NO HEDGE
    request ------------------------------------------> 480 ms  (p99.9)
    one unlucky replica, one slow response

  HEDGE AT p95 = 40 ms
    request -----X (still running)
             40ms|
                 +-- hedge --------> 52 ms      take the first answer
    extra load: 5 percent of requests send a second copy
    p99.9 falls from 480 ms to roughly 60 ms

  TIED REQUEST
    request --> replica A (queued behind 3 requests)
            --> replica B (idle, starts immediately)
                  B sends "I have started" to A, A drops its copy
    extra load: bounded by the cancellation round trip

Three conditions must hold before hedging is safe. The operation must be idempotent, because both copies can execute. There must be a retry budget, or a slow dependency turns hedging into a doubling of load exactly when it can least afford it. And the hedge must go to a different replica, since a hedge to the same instance reproduces whatever made the first copy slow.

The specific causes, and what each one looks like

Tail latency has a small number of causes, and each has a signature.

A garbage collection pause shows as a latency spike affecting every in-flight request on one instance at once, correlated across endpoints, with no corresponding rise in dependency latency. Collector logs confirm it in seconds. Volume I Chapter 5 covers the remedies: size the heap for three to four times the live set, move to a concurrent collector for large heaps, and reduce allocation in the hot path.

CPU quota throttling shows as a periodic stall with a period near the cgroup period, affecting one container while the node has idle CPU. cpu.stat confirms it with a rising nr_throttled.

Run-queue delay shows as a gap between the time a request arrived and the time the handler started running. runqlat measures it directly. It is caused by oversubscription, by a noisy neighbor, or by more runnable threads than the quota allows.

A micro-burst is the one that hides from every dashboard. A service that averages 40 percent CPU over a minute can be at 100 percent for 200 milliseconds inside that minute, and the queue that forms during those 200 milliseconds produces the tail. A one-minute average cannot show it, and a one-second histogram can. Sample at a resolution finer than the phenomenon, or measure queue depth, which integrates the burst and does not need the resolution.

A lock convoy shows as a latency floor that rises with concurrency while CPU stays flat. An off-CPU profile names the lock.

And a cold cache, a JIT recompilation, a connection pool refill, or a leader election each produce a burst of slow requests right after a deployment or a failover. Comparing latency before and after a deploy therefore needs a warm-up exclusion window.

8.4 Reliability Metrics and Error Budgets

A service level indicator is a measurement of one aspect of quality, expressed as good events divided by valid events. A service level objective is a target for that indicator over a window. An error budget is the allowed shortfall.

Defining an indicator that means something

Three properties make an indicator useful. It must measure something a user experiences, which rules out CPU and disk. It must be a ratio of good to valid, which makes it comparable across traffic levels. And it must be measurable at the point closest to the user that is still under your control, which usually means the edge rather than the service.

Four indicator shapes cover most services. Availability is successful requests over valid requests, where a 400 is valid and not your fault, and a 500 is neither. Latency is requests faster than a threshold over valid requests, which is a ratio rather than a percentile and therefore composes correctly. Freshness is records processed within a time bound over total records, which suits a pipeline. Correctness is records without a detected defect over total records, which needs a checker.

The threshold in a latency indicator is a product decision, not a technical one. "99 percent of requests faster than 300 milliseconds" states both the target and the tolerance. It is far more useful than a p99 number that drifts with the traffic mix.

Choose the objective from what users need, not from how many nines sound impressive. Each additional nine costs roughly an order of magnitude more engineering, and it is wasted when the network between the user and the service is less reliable than the objective. A mobile application on a cellular network cannot deliver five nines to its users whatever the backend does.

Burn rate alerting

Alerting on "the objective is breached" is useless, because by then the budget is gone. Alerting on any error is useless, because errors are normal. Burn rate solves both.

Burn rate is the multiple of budget consumption that exhausts the budget exactly at the end of the window. A burn rate of 1 uses the budget evenly. A burn rate of 14.4 consumes 2 percent of a 30-day budget in one hour.

Use several windows at once. A fast burn over a short window catches an outage. A slow burn over a long window catches a steady degradation that no single moment triggers. Pair each long window with a short one at roughly one twelfth its length, so that the alert resolves quickly once the problem stops rather than staying lit for hours.

The budget also functions as a decision rule, which is its real purpose. Within budget, the team ships features. Outside budget, the team fixes reliability before shipping. That converts an argument about priorities into an arithmetic fact, and it gives the team a defensible reason to say no.

Alerting discipline

Alert on symptoms, not on causes. A symptom is something a user experiences: more errors, higher latency, missing data. A cause is an internal state: high CPU, a full queue, a restarted pod. Causes produce false pages, because a system can be healthy with high CPU and broken with low CPU.

Every page must be actionable, urgent, and novel. Actionable means a human can do something now, so an alert whose runbook says "wait and see" belongs in a ticket. Urgent means it cannot wait until morning. Novel means the same alert has not fired ten times this week, because a repeated alert trains people to ignore it.

Track two numbers with the same seriousness as latency: pages per on-call shift, and the fraction of pages that led to an action. More than two pages per shift is unsustainable. An actionable fraction below half means the alerting is training people to ignore it.

Every alert needs a runbook with three sections: what this alert means in plain terms, what to look at first, and what actions are safe. Write it when you create the alert, because nobody writes it at three in the morning.

Chaos engineering

A resilience configuration that has never been exercised does not work. Timeouts, retry budgets, circuit breakers, bulkheads, and failover paths all sit idle during normal operation and rot silently.

Chaos engineering runs the experiment on purpose. The method has four steps. State a hypothesis about steady-state behavior in measurable terms, such as "checkout success stays above 99.5 percent". Inject a specific fault. Observe whether the hypothesis holds. And stop immediately if it does not.

Progress through a maturity order rather than starting at the end. Begin in a staging environment with a single pod termination. Move to production during a low-traffic window with a narrow blast radius. Add latency injection into one dependency, then error injection, then a zone failure. Reach continuous automated experiments only once the earlier stages stopped finding new failures.

Two controls are mandatory and they are what separates an experiment from an outage. A blast radius limit confines the fault to a small share of traffic or to one cell. And an automatic abort stops the experiment when the steady-state metric leaves its bound, without waiting for a human.

The experiments worth running first are the ones that reproduce the incidents you have already had. Restart the cache tier under load and observe whether the database tier survives a zero percent hit ratio. Add 500 milliseconds to one dependency and observe whether the caller's timeout fires before its thread pool fills. Fail an availability zone and measure how long failover actually takes, rather than how long the design document says it takes.

Sources and further reading

The OpenTelemetry specification, the collector documentation, and the semantic conventions registry with its stability markers. The W3C Trace Context recommendation. Charity Majors, Liz Fong-Jones, and George Miranda, "Observability Engineering", and Charity Majors' writing on wide events. Brendan Gregg, "Systems Performance", second edition, and "BPF Performance Tools", for off-CPU analysis and the tooling. Jeff Dean and Luiz Barroso, "The Tail at Scale", CACM 2013, for hedged and tied requests. Gil Tene's talks on coordinated omission and the HdrHistogram documentation. Betsy Beyer and others, "The Site Reliability Workbook", for multi-window burn rate alerting. The Parca and Pyroscope documentation for continuous profiling at scale.