Scalability Trade-Offs and Case Studies

Capacity arithmetic before architecture

Every scaling decision starts with an estimate that takes ten minutes and is accurate within a factor of three. The estimate prevents most bad architectures, because most bad architectures come from never computing the number at all.

Four quantities describe any request-serving system. Request rate, in requests per second, at the peak rather than the average. Payload size, for the request and the response separately. Work per request, as CPU milliseconds and as downstream calls. Data volume, as the total set and as the working set.

From those four, everything else follows. Server count is request rate divided by per-server capacity, times a headroom factor. Chapter 4 fixed the headroom factor: target 60 to 70 percent utilization, because queueing delay at 90 percent is nine service times and at 99 percent is ninety-nine. Network need is request rate times payload. Memory need is working set plus per-connection overhead times connection count. Connection count comes from Little's law: concurrency equals arrival rate times latency.

Two rows carry the whole design. The cache hit ratio turns one million requests per second into fifty thousand, which is the difference between a database tier of seven replicas and a database tier that cannot exist. And the in-flight count of four thousand sets every pool size in the system, from connection pools to thread pools to the concurrency limit in the load shedder.

Case study one: a read-heavy API at one million requests per second

The target is one million requests per second, a p99 under 50 milliseconds, and survival of a full availability zone loss. The workload is read-heavy with a 100 to 1 read-to-write ratio.

The architecture is a cache hierarchy, and each layer exists to remove load from the layer below.

  clients                                       1,000,000 rps
     |
     v
  ANYCAST EDGE / CDN            hit 60%            ---> 600,000 served
     |  TLS 1.3 + HTTP/3, 0-RTT resume
     v                                             400,000 rps
  LOAD BALANCER (L4, eBPF, DSR)  no cache
     |  consistent hashing on the cache key
     v                                             400,000 rps
  API SERVERS  (126 instances)
     |  local in-process cache, hit 50%      ---> 200,000 served
     v                                             200,000 rps
  DISTRIBUTED CACHE (Redis / Memcached cluster)
     |  hit 75%                              ---> 150,000 served
     v                                              50,000 rps
  READ REPLICAS  (7 instances)
     |  miss to primary only on write path
     v
  PRIMARY DATABASE                                   10,000 wps

  Combined hit ratio: 95%. Each layer must survive the full load of the
  layer above, because a cold cache delivers 0% for its first minutes.

The edge layer terminates TLS close to the user. Chapter 9 gave the reason: the handshake costs a round trip, and moving the endpoint from 80 milliseconds away to 10 milliseconds away saves 70 milliseconds on every new connection. Anycast routing does the placement with no client logic.

The in-process cache on each API server is the cheapest layer and the one that teams skip. A hash map lookup costs 25 nanoseconds against 200 microseconds for a network round trip to a cache cluster, which is a factor of eight thousand. It holds the hottest few thousand keys, and it is where a distribution with a heavy head gets flattened.

The distributed cache uses consistent hashing so that a node loss remaps only that node's keys. Chapter 9 gave the reason: a modulo hash remaps almost every key and empties the whole cache at once, which delivers the full one million requests per second to the database tier.

Three cache failure modes need explicit handling, and all three killed production systems in the past.

A thundering herd happens when a popular key expires and a thousand concurrent requests all miss and all query the database. Fix it by letting one request recompute while the others serve the stale value, which is request coalescing plus stale-while-revalidate.

A cache stampede happens when many keys expire together, because they were all written together. Fix it by adding jitter to every time to live, so that expiry spreads over a window instead of landing on one second.

Cache penetration happens when requests ask for keys that do not exist anywhere, so every one reaches the database. Fix it by caching the negative result with a short time to live, and by putting a Bloom filter in front. At 10 bits per key, Chapter 3 showed that a filter over a billion keys costs 1.2 gigabytes and rejects 99 percent of the nonexistent lookups.

Case study two: sharding across regions

Sharding splits data across independent groups so that each group holds a fraction. It is the answer when one machine cannot hold the data, cannot serve the write rate, or cannot meet a latency target from one location.

The shard key is the decision that everything else follows from, and it is close to irreversible. Three properties matter. It must distribute load evenly, so no shard becomes hot. It must keep together the data that queries read together, so that most queries touch one shard. It must be present in almost every query, because a query without the shard key must fan out to every shard.

Hash sharding distributes well and destroys range queries. Range sharding preserves range queries and creates hot spots when the range is a timestamp, because every write lands on the newest shard. Composite keys give both: hash the high-order component and range-order the low-order one, which is what a partition key plus sort key gives.

The last row is usually the right answer for this domain, and the "fails at" column is the point. Global analytics queries must not run on the transactional store at all. Send changes to a column store through change data capture, which Chapter 12 described, and let the analytical engine answer them.

Resharding is the operation that teams underestimate. Moving from 8 shards to 16 remaps half the keys under a naive hash. Two techniques avoid the pain. Consistent hashing with virtual nodes moves only the keys of the shards that change. Pre-splitting into many more logical shards than physical nodes, for example 1,024 logical shards on 8 nodes, turns resharding into a reassignment of logical shards to nodes with no rehashing at all. Choose the second at design time and the problem disappears.

Multi-region placement is a separate decision from sharding, and PACELC from Chapter 8 governs it. Three patterns cover almost every real system.

A single writer region with read replicas elsewhere gives strong consistency for writes and low-latency local reads. Writes from a distant region pay the round trip, which Table 8.1 puts at 70 to 90 milliseconds across the Atlantic. This is the correct default, and most systems that believe they need more do not.

Regional sharding puts each customer's data in one home region and routes their requests there. Latency is local for the owner, consistency is strong within a region, and cross-region access is slow but correct. Data residency requirements often force this pattern anyway.

Global synchronous replication gives external consistency everywhere and pays a consensus round trip on every write. Spanner and Aurora DSQL provide it. Choose it when a write in one region must be immediately visible in another, and be certain that requirement is real, because it costs 100 milliseconds per write forever.

Case study three: a schema migration with no downtime

The goal is to change a column, a table, or a whole storage system while serving traffic, with the ability to stop at any point.

The pattern is expand and contract. Its rule is that every intermediate state must be valid for the old code and the new code at the same time, because a deployment is never instantaneous.

Splitting a full_name column into first_name and last_name runs in six phases.

1. Expand. Add the new columns as nullable. No code reads them. The old code is unaffected because it does not know they exist.
2. Dual write. Deploy code that writes both the old column and the new ones on every create and update. Reads still use the old column. The system now produces correct new data.
3. Backfill. Run a background job that fills the new columns for existing rows, in batches, with a rate limit, and with progress stored so that it resumes. Compare a sample for correctness as it runs.
4. Shadow read. Deploy code that reads the new columns, compares them against the old, logs any mismatch, and returns the old value. Run until the mismatch rate is zero over a full business cycle.
5. Switch. Deploy code that reads the new columns and returns them. Keep writing both. This step is a one-line change and it is instantly revertible.
6. Contract. After a soak period long enough to roll back, stop writing the old column, then drop it.

-- Phase 1: expand. Adding a nullable column is metadata-only on
-- PostgreSQL 11+ and MySQL 8, so it takes no table lock.
ALTER TABLE users ADD COLUMN first_name text;
ALTER TABLE users ADD COLUMN last_name  text;

-- Phase 3: backfill in bounded batches. Never one statement over the
-- whole table: that holds one transaction open, which Chapter 7 showed
-- blocks vacuum and grows the undo log across the whole database.
WITH batch AS (
    SELECT id FROM users
     WHERE first_name IS NULL AND full_name IS NOT NULL
     ORDER BY id
     LIMIT 5000
     FOR UPDATE SKIP LOCKED
)
UPDATE users u
   SET first_name = split_part(u.full_name, ' ', 1),
       last_name  = substr(u.full_name,
                           length(split_part(u.full_name, ' ', 1)) + 2)
  FROM batch
 WHERE u.id = batch.id;
-- Repeat until zero rows affected. Sleep between batches and watch
-- replication lag, which is the signal that the backfill is too fast.

Three rules make the difference between a boring migration and an incident.

Never take a long lock. A statement that rewrites a large table blocks every other transaction. In PostgreSQL, an ALTER TABLE that needs an exclusive lock queues behind every running query, and it then blocks everything that arrives after it. Use online schema change tooling for the operations that need a rewrite.

Always keep a rollback path. Each phase is revertible on its own. The only irreversible step is dropping the old column, and it happens last, after a soak period, when nothing has read it for days.

Always watch replication lag during a backfill. A backfill that outruns replication makes every read replica stale, and a read-after-write path that uses a replica then returns old data to users. Lag is the rate limiter.

The same six phases apply to larger changes. Moving a table to a new database runs dual writes, a backfill, shadow reads that compare both stores, then a switch. Moving from a monolith's table to a new service's store runs the same sequence with the service in front. The phase structure does not change with the size of the change.

Case study four: the failure that does not stop when the cause does

Metastable failure is the class that causes the longest outages, and it is underrepresented in design reviews because it does not appear in any single component's behavior.

The pattern has three parts. The system has a stable state under normal load. A trigger pushes it into a second state that also sustains itself. The second state persists after the trigger is gone, because the state itself generates the load that keeps it there. Removing the trigger does not recover the system, and only removing the load does.

A concrete sequence. A cache node restarts, so the hit ratio drops. The database sees ten times its normal load and slows down. Request latency rises past the client timeout. Clients retry, so the database sees more load. Each retry consumes a connection and a thread for the full timeout, so the server's concurrency saturates. Queue depth grows, which raises latency further, which causes more timeouts and more retries. The cache node comes back and refills, and the system does not recover, because the retry load alone now exceeds the database's capacity.

     +--------------------------------------------------------+
     |                                                        |
     v                                                        |
  cache miss rate up  -->  database load up  -->  latency up  |
                                                        |     |
                                  +---------------------+     |
                                  v                           |
                          client timeout  -->  retry  --------+
                                  |
                                  v
                       more concurrency held, more queueing

  The loop has gain greater than one: each pass adds more load than
  the last. The trigger is gone by the third pass and the loop runs on.
  Recovery requires breaking the loop: shed load, or drop retries,
  or return errors fast enough that clients stop waiting.

Prevention is cheaper than recovery, and every mechanism comes from an earlier chapter. Retry budgets from Chapter 11 cap retries at 10 to 20 percent of requests, which keeps the loop gain below one. Circuit breakers stop sending to a failing dependency, which removes the load instead of adding to it. Load shedding rejects excess work quickly rather than queueing it. Bounded queues everywhere, from Chapter 11, stop the queue growth that converts overload into memory exhaustion. Deadline propagation from Chapter 9 stops work whose result nobody will read.

Recovery, when prevention failed, has one reliable procedure. Shed load aggressively, down to a level well below capacity. Let the system reach a healthy steady state. Then restore traffic gradually, in steps, watching the latency at each step. Restoring all at once returns the system to the failure state immediately.

The numbers, on one page

Every design decision in this book reduces to a comparison between two numbers on this scale.

Read the table as a decision procedure. Any design that turns an operation on one row into an operation two rows below it made the system a hundred times slower at that step. The design must earn that with something else. Any design that moves an operation up the table is worth doing if it does not cost correctness.

A review checklist

The following questions catch most of what goes wrong. They are ordered by how often the answer is missing.

• What is the peak request rate, and what is the utilization target that sets the instance count?
• Where is the unbounded queue? Every queue needs a maximum and a policy for a full queue.
• What is the retry policy at each layer, and what is the total amplification across the chain?
• What happens when the cache tier is cold? Does the layer below survive the full rate?
• Which operations are idempotent, and how does a retry carry the idempotency key?
• What is the consistency requirement per operation, not per system?
• What is the shard key, and which queries cannot use it?
• Which transaction can stay open longest, and what does it block while it does?
• What does a full availability zone loss cost, and has that been tested?
• What is the rollback path for the next deployment and for the next migration phase?
• Which alerts page, and is each one a symptom that a user experiences?
• What is the error budget, and who decides when it is spent?

Closing

The chapters of this book are one argument. Hardware sets the constants. The kernel decides what an operation costs to request. Data structures decide how many of those operations an algorithm needs. Concurrency decides how many can happen at once and what coordination costs. Language runtimes decide the shape of the pauses. Design boundaries decide which operations cross a network. Storage engines decide what durability costs. Consensus decides what agreement costs. Networking, containers, meshes, and streams decide how the pieces reach each other. Inference, telemetry, and security decide what the newest and the oldest requirements cost. Scale is what happens when every one of those decisions compounds.

An engineer who knows the numbers in Table 16.3 and the mechanisms behind them can predict a system's behavior before building it, and can explain a system's behavior after it surprises everyone. That prediction and that explanation are the same skill, and it is the one this book set out to build.

Sources and further reading

Jeff Dean's "Numbers Everyone Should Know", which Table 16.3 updates for 2026 hardware. Martin Kleppmann, "Designing Data-Intensive Applications". Nathan Bronson and others, "Metastable Failures in Distributed Systems", HotOS 2021. Marc Brooker's writing at the AWS Builders' Library on timeouts, retries, and static stability. Michael Nygard, "Release It!", second edition. The Google SRE books for error budgets and for incident practice. Sam Newman, "Monolith to Microservices", for extraction and migration sequencing.