Production Readiness Checklists and the Failure Catalog

How to use these lists

Read the checklist first and the failure catalog second. Every checklist line has an entry in the catalog that names the incident it prevents. A team that knows the failure applies the control correctly. A team that only knows the control applies it as written and misses the variant.

Nothing here is universal. Cross out what does not apply to your service and write down why, because a deliberate omission is an engineering decision and a forgotten one is a future incident.

Checklist: transport and protocol

• Every request-response socket sets TCP_NODELAY, and each message is written in one call.
• Socket buffers are sized from the bandwidth-delay product, not left at the default.
• The listener uses SO_REUSEPORT with one socket per worker thread.
• The accept queue depth and the application backlog argument are set together, and the overflow counter is monitored.
• HTTP/3 is advertised through Alt-Svc or an HTTPS DNS record, with HTTP/2 kept as the fallback.
• QUIC flow control windows exceed the bandwidth-delay product for the longest path served.
• The load balancer routes QUIC by connection identifier, not by four-tuple hash.
• 0-RTT early data carries only idempotent requests, and the server tracks recently seen tickets.
• The certificate chain sends the leaf and the intermediate only, with certificate compression enabled.
• TLS session ticket keys rotate on a schedule, are distributed through the secret manager, and the newest key's age is alerted on.
• Hybrid post-quantum key agreement is enabled on every public endpoint.
• Every gRPC call sets a deadline, and the deadline propagates to every downstream call.
• Connection maximum age is set with jitter, so that connections rebalance after a scale-out.
• Protobuf field numbers are never reused, removed numbers are marked reserved, and negative integer fields use sint.

Checklist: consensus and state

• The consensus group size and placement match the failure domain the service must survive.
• Election timeout, heartbeat interval, and broadcast time are each an order of magnitude apart.
• The read path is documented as ReadIndex, lease read, or through the log, and the team knows which.
• Membership changes use joint consensus or single-server steps, and new members join as non-voting learners first.
• The snapshot interval keeps restart replay under a few seconds.
• Pre-vote is enabled, so a partitioned node cannot force elections.
• Consistency level is chosen per operation, and the choice is recorded next to the code that makes it.
• Conflict resolution is chosen per field, and no field silently uses last-write-wins unless losing a concurrent update is harmless.
• No cross-service two-phase commit exists. Multi-service changes use a saga with compensating actions.
• Every state change that must produce an event uses a transactional outbox or change data capture.
• Every externally triggered mutation accepts an idempotency key, and the key is stored with the result in the same transaction as the effect.

Checklist: storage

• The partition key distributes evenly, keeps co-read data together, and appears in almost every query.
• Logical partitions outnumber physical nodes by a large factor, so growth is a reassignment rather than a rehash.
• Every node holding a primary copy uses a drive with power loss protection, or the replication factor tolerates a cache loss.
• Group commit is enabled, and commit latency is measured rather than assumed.
• Compaction has a reserved bandwidth share, and level zero file count, pending compaction bytes, and stall duration are all monitored.
• Bloom filters are enabled on every LSM column family that serves point reads.
• idle_in_transaction_session_timeout and statement_timeout are set.
• The age of the oldest transaction is alerted on, not the query count.
• Autovacuum thresholds are tuned per table, with the hottest tables set far more aggressively than the default.
• Long analytical reads are routed to a replica with its own cleanup horizon.
• Retention, archival, and deletion paths exist, and the team exercised each one, including the erasure path for personal data.

Checklist: event streaming

• Replication factor is 3 with min.insync.replicas set to 2, and unclean leader election is disabled.
• linger.ms is set above zero, and batch size and compression are tuned together.
• Partition count is chosen for three years of growth, because increasing it breaks per-key ordering.
• Per-partition rates are monitored, not only topic totals, so that a hot key is visible.
• max.poll.interval.ms exceeds the worst-case batch processing time, and max.poll.records is set accordingly.
• Cooperative sticky assignment and static group membership are enabled.
• Consumer lag is measured in seconds, and rebalance rate is alerted on separately.
• Deserialization failures route straight to the dead letter topic, separately from processing failures.
• Every dead letter record carries the original topic, partition, offset, exception, and attempt history.
• A replay tool for the dead letter topic exists, and someone ran it at least once.
• Schema compatibility is set to full or full transitive, and the upgrade order is documented.
• Change data capture replication slot lag has a paging alert.

Checklist: service communication

• Every call has a timeout derived from the p99.9 of the healthy distribution.
• Retries happen at exactly one layer, with a budget capped at 10 to 20 percent and full jitter backoff.
• Circuit breakers use a sliding window with a minimum request volume, not a consecutive failure count.
• Each downstream dependency has its own connection pool and its own concurrency limit, sized by Little's law.
• Load is shed by priority class, so degradation removes the least important traffic first.
• Every queue in the service has a maximum size and a documented policy for a full queue.
• Queue depth and rejection count are exported, and the alert is on the rejection rate.
• Liveness probes are local and cheap, and they never call a downstream service.
• Readiness probes report dependency health, and failing one removes the pod from endpoints without restarting it.
• Mutual TLS uses short-lived workload identities, and authorization policy matches on identity rather than on an address.
• Fault injection ran against this service, and the timeout, breaker, and bulkhead each demonstrably fired.

Checklist: edge and API

• Rate limits are layered: global, per customer, per endpoint per customer, and per address for unauthenticated paths.
• The limiter updates atomically, and expensive operations cost more tokens than cheap ones.
• Rejections return 429 with limit, remaining, and reset headers, and the reset instant is jittered across callers.
• JWT validation rejects none, uses an algorithm allowlist, and never lets the token choose its own verification path.
• Expiry, not-before, audience, and issuer are all validated, and the key set is cached with a bounded refresh.
• External credentials are translated into short-lived internal tokens at the edge.
• API keys are stored as a fast hash, structured with a scannable prefix, and support two active keys for rotation.
• The firewall rule set ran in detection mode for a full business cycle before blocking, with per-rule metrics.
• Cache keys are normalized, Vary is limited to headers with low cardinality, and surrogate keys support targeted purging.
• stale-while-revalidate and request coalescing are both configured.
• Query depth limits, complexity budgets, and persisted queries are enforced on any GraphQL endpoint.
• Partial-failure semantics are defined and documented for every aggregating endpoint.

Checklist: runtime and orchestration

• Containers run as a non-root user with no_new_privs, a dropped capability set, and a seccomp profile.
• Untrusted workloads run behind a hypervisor boundary, not on a shared kernel.
• CPU requests are set from measured use, memory request equals memory limit, and CPU limits are omitted unless a hard cap is required.
• CPU throttling counters are monitored.
• The runtime's thread count derives from the cgroup quota, not from the host core count.
• The image runs a real init as PID 1, or the application reaps children and handles SIGTERM.
• Frequently written files live on a volume, not in an image layer.
• Deployment is by image digest, never by a mutable tag.
• Every namespace starts with a default-deny network policy for ingress and egress.
• Pod address ranges are sized for the cluster expected in three years.
• Pod MTU is set explicitly and tested with a large payload where an overlay is in use.
• Storage classes use WaitForFirstConsumer for any zonal storage.
• Admission webhooks exclude control plane namespaces, set a short timeout, and use in-process policy where possible.
• etcd runs on local NVMe, with fsync duration alerted on and a defragmentation schedule.
• Pod disruption budgets exist for every service that preemption or a drain can affect.

Checklist: observability

• Every service emits one wide structured event per request, carrying tenant, endpoint, outcome, duration, build, and trace identifier.
• traceparent is propagated across every boundary, including message queues and thread pools.
• Tail sampling keeps every error and slow trace, with trace-aware routing to the collector.
• Metric cardinality has an enforced budget, and series growth rate is alerted on.
• Latency objectives are expressed as a ratio under a threshold, not as a percentile.
• Burn rate alerts use paired long and short windows.
• Every alert is a symptom, is actionable, and has a runbook written when the alert was created.
• Pages per on-call shift and the actionable fraction are reviewed regularly.
• Continuous profiling runs on every node, and the profiler symbolizes the runtimes actually in use.
• Load tests send at a fixed rate and record from the intended send time.
• Chaos experiments have a stated hypothesis, a blast radius limit, and an automatic abort.

Checklist: AI infrastructure

• Prefix caching is enabled.
• The maximum running batch is set from the inter-token latency target, not from free memory.
• Time to first token, inter-token latency, and goodput are all measured separately.
• Autoscaling uses queue depth and time to first token, never accelerator utilization.
• Quantization was measured on the team's own evaluation set, not accepted from a benchmark.
• Vector index rebuild cadence is planned, and tombstone accumulation is monitored.
• Authorization is enforced by partition or during traversal, never by post-filtering.
• Retrieval uses hybrid search with rank fusion and a reranking stage.
• Agent runs execute under a durable workflow engine, and workflow code is deterministic.
• Every agent has a step limit, a wall-clock limit, and a token budget enforced in the state machine.
• Irreversible actions require an explicit approval state.
• Tool credentials are scoped to the current user, and tool arguments are schema-validated before execution.
• Client disconnect cancels generation, and a test covers the cancellation.
• Cost is tracked per run, per user, and per tenant, with alerting on rate of change.

The failure catalog

Each entry names a failure, its signature, and the checklist line that prevents it.

Transport

Forty millisecond stalls on small request-response traffic. Nagle's algorithm and delayed acknowledgement interact. The signature is a bimodal latency histogram with a mode near 40 milliseconds. Set TCP_NODELAY.

Throughput plateaus far below the link rate with no loss and no CPU saturation. The receive window is below the bandwidth-delay product. The signature is a flat throughput ceiling that scales with round-trip time. Size the buffers.

QUIC connections break when a mobile client changes network. The load balancer hashes the four-tuple and sends migrated packets to a backend with no state. The signature is mobile-only errors that never reproduce on wifi. Route by connection identifier.

A resumed session replays a state-changing request. Early data carried a POST. The signature is a duplicated action with no client retry. Restrict 0-RTT to idempotent methods.

Months of recorded traffic become decryptable after one key leak. A session ticket key never rotated. The signature is absent until the breach. Rotate on a schedule.

Consensus and state

A cluster elects a leader and serves stale reads for an unbounded period. The new leader never committed an entry in its own term. Append a no-operation entry on election.

Two leaders in one term after a membership change. The change created two disjoint majorities. Use joint consensus or single-server steps.

A healthy cluster suffers repeated elections. One node is partitioned from the leader but not from the voters, and it keeps incrementing the term. Enable pre-vote.

A write returns success and then vanishes. Last-write-wins discarded it inside the clock skew window. The signature is silent, with no error and no metric. Use a conflict strategy that keeps both versions.

An invariant breaks with no failing transaction. Write skew under snapshot isolation. The signature is a state that no single transaction created. Use serializable isolation or lock the rows the decision reads.

A multi-service operation leaves the system half-changed. Two-phase commit blocked, or a dual write failed between its two parts. Use a saga and an outbox.

Storage

A service runs fine for hours and then shows multi-second write latency with no change in load. Compaction fell behind and write stalls engaged. Watch level zero file count and reserve bandwidth.

Database size grows while the live data set does not. MVCC bloat, held back by a long-running transaction. Alert on the age of the oldest transaction.

The database reports success and a power cut loses the last minute. A consumer SSD acknowledged writes in a volatile cache. Use power loss protection.

A shard becomes hot while the rest idle. The partition key correlates with traffic, often because it is a timestamp or one large tenant. Measure per-partition rates.

Growing the cluster requires moving nearly all the data. The key space was hashed modulo the node count. Pre-split into many logical partitions.

Streaming

A consumer group stops making progress and restarts repeatedly. Processing exceeds max.poll.interval.ms, so members are ejected and rebalance. Raise the interval and cut the batch size.

A consumer crash-loops on one record forever. A poison pill fails deserialization on every attempt. Route deserialization failures to the dead letter topic.

The primary database fills its disk with no change in write volume. A stopped change data capture connector retains write-ahead log segments. Page on replication slot lag.

Read-committed consumers stop advancing while the partition keeps accepting writes. A transaction is open and holds back the last stable offset. Bound transaction.timeout.ms.

A schema change breaks consumers that were never notified. Compatibility mode was set to none, or the upgrade order was reversed. Use full compatibility.

Service communication

One slow dependency stops the service answering unrelated requests. A shared thread pool filled. Give each dependency its own bulkhead.

A brief dependency failure becomes a total outage. Retries amplified along a chain. Retry at one layer with a budget and jitter.

A dependency blip restarts every pod in the deployment at once. A liveness probe called that dependency. Keep liveness local.

The system does not recover after the trigger is gone. Retry load alone now exceeds capacity, which is a metastable failure. Shed load down to well below capacity and restore gradually.

A newly added backend receives no traffic. Long-lived multiplexed connections were pinned by a layer 4 balancer. Balance at layer 7, balance in the client, or age connections out.

Edge

Every limited client retries at the same millisecond. The reset instant was not jittered. Jitter it.

An attacker forges a token accepted as valid. The verifier trusted the token's own alg header. Use an allowlist and reject none.

Cache hit ratio collapses after a marketing campaign. Tracking parameters created one cache entry per campaign. Normalize the cache key.

Every request after expiry reaches the origin at once. No stale-while-revalidate and no request coalescing. Configure both.

A firewall deployment blocks legitimate traffic on day one. Blocking mode was enabled without a detection period. Run detection first.

Runtime

p99 latency shows a periodic cliff of tens of milliseconds while the node has idle CPU. A CPU quota throttles the container each period. Match thread count to quota.

A pod keeps running in a broken state after an out-of-memory kill. The kernel killed a worker rather than the main process. Set memory.oom.group.

Small requests succeed and large ones hang. Pod MTU was not reduced for the overlay. Set it explicitly and test with a large payload.

A cluster stops accepting writes for one resource. An admission webhook became unavailable with failurePolicy: Fail. Scope it, time it out, or use in-process policy.

Every controller lists the full state at once and the control plane saturates. etcd compaction passed a watcher's revision. Monitor compaction and watcher lag.

Failover of a stateful pod takes eight minutes. Volume detach waited out the unreachable timeout. Replicate in the application if that is too slow.

Observability

The monitoring system runs out of memory and takes down visibility for everything. One unbounded label was added to a metric. Enforce a cardinality budget in the collector.

Traces end exactly where the interesting part begins. Context was lost at an asynchronous boundary or a raw thread pool. Propagate explicitly.

The load test shows a healthy p99 that production contradicts. Coordinated omission removed the samples from the worst moment. Send at a fixed rate.

A service-level p99 is computed from per-instance p99 values. Percentiles do not average. Merge histograms instead.

A service averaging 40 percent CPU shows a bad tail. A micro-burst saturates it for 200 milliseconds inside each minute. Measure queue depth.

AI infrastructure

Throughput looks excellent and users complain. Time to first token grew while the batch grew. Optimize goodput, not throughput.

Accelerators look 30 percent busy and the service is saturated. Decode is memory bandwidth bound and utilization does not show it. Autoscale on queue depth.

Retrieval quality degrades slowly over weeks. Tombstones accumulated in the graph index. Schedule rebuilds.

A user sees no results while a colleague sees ten for the same query. Authorization was applied by post-filtering. Partition or filter during traversal.

An agent charges a card twice after a deployment. The loop restarted with no journal, and the tool had side effects. Use durable execution and idempotent tools.

An agent run costs a thousand times the expected amount. No step limit and no token budget. Enforce both in the state machine.

A retrieved document changes the agent's behavior. Indirect prompt injection. Scope the tools, scope the credentials, validate the arguments, and gate the irreversible actions.