Chapter 1 · Part I: High-Throughput Networking and Distributed State
Network Transport Protocol Internals
Every distributed system rests on a transport that can lose packets, reorder them, and stall. This chapter takes apart HTTP/3 stream multiplexing, the three ways to remove the kernel from the data path, the encoding that a remote procedure call puts on the wire, and the cryptographic cost of a TLS 1.3 handshake.
In this chapter
1.1 HTTP/2 Head-of-Line Blocking and HTTP/3 QUIC Streams
HTTP/1.1 allowed one request at a time per connection. Browsers worked around that limit by opening six connections per origin, which multiplied the handshake cost and gave each connection its own congestion window that started from nothing.
HTTP/2 replaced the workaround with multiplexing. One connection carries many streams, each stream carries one request and one response, and frames from different streams interleave on the wire. The application-level blocking disappeared. A lower one took its place, and it is a property of TCP rather than of HTTP.
Why TCP blocks every stream at once
TCP delivers a byte stream in order. The receiving kernel holds any segment that arrives after a gap and delivers nothing to the application until the missing segment arrives. HTTP/2 runs many logical streams inside that single ordered byte stream, so one lost segment stalls every stream whose data sits behind the gap.
The cost is a full retransmission timeout or a fast retransmit round trip. On a path with 2 percent loss and an 80 millisecond round trip, a page that loads forty resources over one HTTP/2 connection can stall repeatedly, and each stall blocks resources that had already arrived in full.
HTTP/2 over TCP: one ordered byte stream
wire: [s1][s3][s5][XX][s1][s3][s7][s1][s5][s3]
^ lost segment
kernel receive buffer holds everything after the gap.
Streams 1, 3, 5 and 7 all wait, even though only one segment is missing.
Application sees nothing until the retransmission arrives: +1 RTT.
HTTP/3 over QUIC: one ordered byte stream PER STREAM
wire: [s1][s3][s5][XX][s1][s3][s7][s1][s5][s3]
^ lost packet, carried stream 5 data only
QUIC delivers stream 1, 3 and 7 data immediately.
Only stream 5 waits for the retransmission.
Figure 1.1 One lost segment stalls every HTTP/2 stream behind it.
QUIC moves the ordering guarantee down to the stream. Each stream has its own sequence of offsets and its own delivery state. A lost packet blocks only the streams whose data that packet carried. Everything else reaches the application at once.
Streams, frames, and the four stream types
A QUIC stream is identified by a 62-bit variable-length integer. The two low bits encode the type. Bit 0 says who initiated the stream, where 0 means the client and 1 means the server. Bit 1 says whether the stream is bidirectional, where 0 means bidirectional and 1 means unidirectional.
That encoding gives four types with no negotiation. Stream 0 is the first client-initiated bidirectional stream, which HTTP/3 uses for the first request. Streams 2 and 3 are the first client and server unidirectional streams, which HTTP/3 uses for control and for the header compression encoder and decoder.
Data travels in frames inside packets. A STREAM frame carries a stream
identifier, a byte offset, and a payload. Because the frame carries an
explicit offset, a receiver reassembles a stream from packets that arrive in
any order. The design is why QUIC needs no equivalent of the TCP sequence
number for ordering across streams.
Table 1.1: the QUIC frames that carry most of the behavior
| Frame | Purpose | Notes |
|---|---|---|
STREAM |
stream data with an explicit offset | carries the FIN bit |
ACK |
acknowledges packet numbers, with ranges | carries ECN counts |
CRYPTO |
handshake bytes, outside any stream | one per packet number space |
MAX_DATA, MAX_STREAM_DATA |
flow control credit | connection and stream level |
MAX_STREAMS |
how many streams the peer can open | prevents stream exhaustion |
RESET_STREAM, STOP_SENDING |
abort one stream | the rest of the connection continues |
NEW_CONNECTION_ID, RETIRE_CONNECTION_ID |
rotate the connection identifier | enables migration and unlinkability |
PATH_CHALLENGE, PATH_RESPONSE |
prove that a new path works | required before full migration |
NEW_TOKEN |
address validation token for a later connection | removes a round trip |
CONNECTION_CLOSE |
terminate, with an error code | application or transport level |
Packet number spaces and loss detection
QUIC numbers packets separately from stream offsets, and it never reuses a packet number. That single rule removes the retransmission ambiguity that TCP has, where an acknowledgement for a retransmitted segment does not say whether it acknowledges the original or the copy. A QUIC acknowledgement names exactly one transmission, so the round-trip time estimate stays clean.
Three packet number spaces exist: Initial, Handshake, and Application. Each has its own numbering and its own encryption keys. The separation lets the handshake and the application data progress independently, and it stops a lost handshake packet from affecting application loss detection.
Loss detection uses a packet threshold and a time threshold. A packet counts as lost when a packet with a higher number is acknowledged and the gap exceeds the threshold, which defaults to 3. It also counts as lost when a time threshold passes, which is 9 eighths of the larger of the smoothed round-trip time and the latest sample. A probe timeout sends new data when no acknowledgement arrives at all.
Flow control at two levels
QUIC applies flow control per stream and per connection. The receiver
advertises a maximum offset per stream through MAX_STREAM_DATA and a maximum
total across all streams through MAX_DATA. A sender that reaches either
limit stops and sends DATA_BLOCKED or STREAM_DATA_BLOCKED.
Two levels are necessary because one level fails in each direction. Per-stream control alone lets a peer open ten thousand streams and consume unbounded memory. Connection-level control alone lets one stream consume the whole window and starve the rest. The combination bounds both.
The tuning rule follows the bandwidth-delay product. The connection window must exceed bandwidth multiplied by round-trip time, or the sender idles waiting for credit. A 1 gigabit path with an 80 millisecond round trip needs 10 megabytes of connection window. Many default configurations ship with 1 megabyte, which caps that path at about one eighth of its capacity.
A small QUIC flow control window looks like a slow server
The symptom is throughput that plateaus far below the link rate, with
no loss and no CPU saturation. The cause is a receive window that the
bandwidth-delay product exceeds. Raise initial_max_data and
initial_max_stream_data_* transport parameters together, and enable
window auto-tuning where the implementation offers it. Measure with a
single large download over a long path, because a short path hides the
problem completely.
Connection identifiers and migration
A TCP connection is identified by the four-tuple of source address, source port, destination address, and destination port. Change any element and the connection breaks. A phone that moves from wifi to cellular changes its source address, so every TCP connection dies and every TLS session restarts.
QUIC identifies a connection by an explicit connection identifier that travels
in the packet header. The four-tuple can change while the identifier stays the
same, so the connection survives the move. The client sends packets from the
new address, the server validates the new path with PATH_CHALLENGE and
PATH_RESPONSE, and traffic continues with no new handshake and no new
session key.
Path validation exists to stop an amplification attack. Without it, an attacker spoofs a victim address, claims a path change, and directs the server's traffic at the victim. The challenge and response prove that whoever claims the new address can receive at it. Until validation completes, the server limits its send rate on the new path to three times the bytes received.
Each endpoint issues several connection identifiers through
NEW_CONNECTION_ID and retires old ones. Using a fresh identifier after a
path change stops an observer from linking the two paths to one user, which is
a privacy property that TCP cannot offer.
Connection migration needs a load balancer that understands it
A layer 4 load balancer that hashes the four-tuple sends the migrated packets to a different backend, which has no state for that connection. The connection then fails in a way that looks like a server bug. The fix is connection-identifier-aware routing: the server encodes a routing key in the identifier it issues, and the load balancer parses that key instead of hashing the tuple. Every serious QUIC deployment does this, and it is the single most common reason that a first QUIC rollout fails on mobile networks.
Zero round-trip time resumption, and its one hazard
A full QUIC handshake costs one round trip, because the transport handshake and the cryptographic handshake run together. A resumed connection costs zero, because the client sends application data in its first flight using a key derived from a pre-shared key that the previous connection established.
The saving is real and the hazard is specific. Zero round-trip data carries no proof of freshness, so an attacker who records it can replay it. The server has no general way to tell a replay from a legitimate retransmission.
Three controls limit the damage. The application sends only idempotent requests in early data, which in HTTP terms means GET and HEAD and nothing else. The server tracks recently seen tickets and rejects duplicates within a window, which catches most replays at the cost of state. And the session ticket carries a short lifetime, which bounds the window in which a replay is possible at all.
Table 1.2: connection setup cost, first visit and repeat visit
| Stack | New connection | Resumed | Survives IP change |
|---|---|---|---|
| HTTP/1.1 over TCP and TLS 1.2 | 3 RTT | 2 RTT | no |
| HTTP/2 over TCP and TLS 1.3 | 2 RTT | 1 RTT | no |
| HTTP/2 over TCP with TLS 1.3 and TCP Fast Open | 1 RTT | 0 RTT | no |
| HTTP/3 over QUIC | 1 RTT | 0 RTT | yes |
Where HTTP/3 loses
QUIC runs in user space in most deployments, so it does not get the decades of kernel optimization that TCP has. Early measurements put its CPU cost at two to three times TCP per byte. Generic segmentation offload for UDP, better implementations, and partial hardware support narrowed the gap, and it has not closed.
Three other constraints matter for a rollout. Some corporate and mobile networks block UDP on port 443, so a QUIC-only deployment fails for those users with no diagnosis. Middleboxes cannot inspect QUIC, which removes a class of network troubleshooting that operations teams rely on. And on a stable low-latency wired path, the measured benefit against HTTP/2 is small and sometimes negative, because the loss that QUIC handles better does not occur.
Advertise HTTP/3 with an Alt-Svc header or an HTTPS DNS record, keep HTTP/2
on TCP as the fallback, and measure the two paths separately in real user
monitoring. An aggregate number hides the case where QUIC fails and the client
falls back after a timeout, which is worse than never offering it.
1.2 Kernel-Bypass and Zero-Copy Networking
A byte that leaves an application traverses a fixed path. The application writes to a socket, the kernel copies into a socket buffer, the protocol stack adds headers, the driver builds a descriptor, and the network card reads the buffer by direct memory access. Each step costs, and at high packet rates the total dominates everything the application does.
The techniques in this section remove steps from that path. They fall into three groups by how much of the kernel they remove, and the right choice depends on which cost actually binds.
Table 1.3: data path cost per packet, one direction
| Path | Per packet | Packets per second, one core | Removes |
|---|---|---|---|
Ordinary socket, write |
1.5 to 3 us | 300,000 to 600,000 | nothing |
Socket with MSG_ZEROCOPY |
1.2 to 2.5 us | 400,000 to 800,000 | the payload copy |
sendfile or splice |
0.8 to 1.5 us | 600,000 to 1,200,000 | two copies, two crossings |
Kernel TLS with sendfile |
0.9 to 1.6 us | 600,000 to 1,100,000 | the user-space encrypt copy |
| AF_XDP, zero-copy mode | 150 to 400 ns | 10,000,000 to 25,000,000 | the socket and the stack |
| XDP, drop in the driver | 40 to 80 ns | 15,000,000 to 30,000,000 | everything above the driver |
| DPDK poll mode | 50 to 150 ns | 10,000,000 to 15,000,000 | the kernel entirely |
| RDMA write, end to end | 1 to 2 us | limited by the fabric | the remote CPU |
Removing copies inside the kernel
sendfile transfers bytes from a file descriptor to a socket without them
entering user space. The kernel reads pages from the page cache and hands them
to the network stack directly. A static file server that uses read followed
by write performs four copies and four boundary crossings per chunk. With
sendfile it performs two copies and one crossing, and on hardware that
supports scatter-gather it performs one.
splice generalizes the idea to any pair of descriptors where one side is a
pipe. It moves pages by reference through a kernel pipe buffer rather than by
copying. A proxy that reads from one socket and writes to another uses two
splice calls through one pipe and never touches the bytes.
/* Proxy bytes from one socket to another without copying to user space. */
static int pipefd[2];
ssize_t proxy_chunk(int in_fd, int out_fd, size_t max) {
ssize_t n = splice(in_fd, NULL, pipefd[1], NULL, max,
SPLICE_F_MOVE | SPLICE_F_NONBLOCK);
if (n <= 0) return n;
ssize_t written = 0;
while (written < n) {
ssize_t w = splice(pipefd[0], NULL, out_fd, NULL,
n - written, SPLICE_F_MOVE | SPLICE_F_NONBLOCK);
if (w <= 0) return w;
written += w;
}
return written;
}
MSG_ZEROCOPY applies to ordinary socket writes. The kernel pins the user
pages and transmits from them, then notifies the application through the error
queue once the pages are free for reuse. The application must not modify the
buffer until that notification arrives, which makes the interface harder to
use than sendfile. The gain appears above about 10 kilobytes per write and
is negative below it, because pinning pages costs more than copying a small
buffer.
Kernel TLS moves the record encryption into the kernel. Without it, a server
that uses sendfile for plain HTTP must fall back to reading into user space,
encrypting, and writing, which restores every copy. With kernel TLS the
handshake still runs in user space, the resulting keys are installed into the
socket, and sendfile works again on an encrypted connection. Hardware that
supports TLS offload pushes the encryption into the network card.
XDP: a program in the driver
The eXpress Data Path runs an eBPF program at the earliest point in the receive path, inside the network driver, before the kernel allocates a socket buffer. The program returns one of five verdicts.
XDP_DROPdiscards the packet, at a few tens of nanoseconds.XDP_PASSsends it up the normal stack.XDP_TXbounces it back out the same interface.XDP_REDIRECTsends it to another interface or into an AF_XDP socket.XDP_ABORTEDdrops it and raises a tracepoint, for a program error.
XDP runs in one of three modes. Native mode runs inside the driver and gives the full performance. Generic mode runs in the stack after socket buffer allocation, which works on any driver and gives a fraction of the speed. Offload mode runs on the network card itself, which few cards support.
The performance difference against older tooling is large enough to change
what is possible. An iptables drop costs several microseconds and walks a
rule chain whose cost grows with the rule count. An XDP drop costs tens of
nanoseconds and performs a hash map lookup whose cost does not. One core
therefore absorbs a volumetric denial of service attack that otherwise
saturates the machine.
AF_XDP: a fast lane that keeps the kernel
AF_XDP gives an application a socket that receives packets directly from an XDP program, through memory shared with the kernel. The application registers a memory region called a UMEM and four rings: fill and completion rings that it shares with the kernel, and receive and transmit rings that carry descriptors.
user space kernel / driver
+---------------------------+
| UMEM: one big mmap'd |<---- packets land here by DMA
| region split into frames | (zero-copy mode: the driver
+---------------------------+ DMAs straight into the UMEM)
^ |
| v
+-----------+ +-----------+ +------------------------------+
| FILL ring | | RX ring |<----| XDP program returns |
| app gives | | kernel | | XDP_REDIRECT into the socket |
| empty | | gives full| +------------------------------+
| frames | | frames |
+-----------+ +-----------+
+-----------+ +-----------+
| TX ring | | COMPLETION| Descriptors are 64-bit offsets
| app gives | | kernel | into the UMEM. No packet bytes
| full | | returns | ever cross the boundary.
| frames | | free ones |
+-----------+ +-----------+
Figure 1.2 AF_XDP rings. The application and the kernel share one memory region.
Two modes exist. Copy mode works on any driver and copies each packet into the UMEM. Zero-copy mode requires driver support and has the card write straight into the UMEM, which removes the copy entirely.
AF_XDP is the pragmatic choice for most teams that need high packet rates in 2026. The card stays under kernel control, other traffic follows the normal path, the machine remains manageable, and the redirected traffic reaches tens of millions of packets per second.
DPDK: no kernel at all
The Data Plane Development Kit binds the network card to a user-space poll mode driver. There are no interrupts, no system calls, and no kernel data path. A core spins on the receive ring and processes packets as they arrive.
The costs are total and they need to be stated plainly. The card belongs to the application and the host cannot use it for anything else. Cores run at 100 percent whether or not traffic flows, which shows on a power bill and confuses every capacity metric. The application must implement whatever it needs from the protocol stack, including address resolution, fragmentation, and checksums. Debugging tools that rely on the kernel stack stop working.
DPDK earns those costs in a narrow set of workloads: line-rate packet processing, network functions, firewalls, packet brokers, and market data handlers. It does not earn them in a web service.
RDMA: removing the remote CPU
Remote Direct Memory Access lets one machine read or write another machine's memory with no involvement from the remote CPU. It reaches 1 to 2 microseconds end to end, which is close to the wire time.
The programming model is queue pairs. Each endpoint creates a send queue and a receive queue, registers memory regions with the adapter, and posts work requests. The adapter performs the transfer and posts a completion. Memory registration pins the pages and gives the adapter a key, which is why RDMA applications register large buffers once at start-up rather than per operation.
Operations divide into two families. Two-sided operations, SEND and
RECEIVE, need the remote side to post a receive buffer, so both CPUs
participate. One-sided operations, READ and WRITE, name a remote address
and a key, and the remote CPU never learns that they happened. One-sided
operations give the lowest latency and require the initiator to know the
remote layout.
Two transports carry it. InfiniBand uses its own link layer and its own switches. RoCEv2 carries RDMA over UDP on an ordinary Ethernet fabric, which is cheaper and which demands a near-lossless network. Priority flow control and explicit congestion notification with DCQCN provide that, and both need switch configuration that a general-purpose datacenter network does not have by default.
RoCE without a configured fabric fails in a way that looks like a bug
RDMA over converged Ethernet assumes that the fabric almost never drops a packet, because its loss recovery is expensive. A fabric without priority flow control drops under congestion, the transport retries whole sequences, and throughput collapses to a fraction of the link rate. Priority flow control that is misconfigured produces the opposite failure: pause frames spread backward and create a congestion tree that stalls unrelated traffic. Treat the fabric configuration as part of the application, and test it under a congestion pattern that matches production.
Choosing
The decision follows from which cost binds, and the honest answer for most services is that none of these does.
| Binding cost | Technique | Note |
|---|---|---|
| Copying large static payloads | sendfile, kernel TLS |
a few lines of change |
| Proxying bytes between sockets | splice through a pipe |
no user-space touch |
| Large writes from application memory | MSG_ZEROCOPY |
above 10 KB only |
| Dropping unwanted packets at high rate | XDP in native mode | tens of nanoseconds |
| Processing millions of packets per second | AF_XDP zero-copy | keeps the kernel usable |
| Line-rate packet processing, dedicated box | DPDK | the card belongs to you |
| Microsecond remote memory access | RDMA, RoCEv2 or InfiniBand | needs a designed fabric |
Before you adopt any of them, measure whether the network stack is the bottleneck at all. In almost every service it is serialization, a database round trip, a lock, or a garbage collector. Volume I Chapter 14 covers the measurement that settles the question in under an hour.
1.3 High-Performance Remote Procedure Calls and Serialization
A remote procedure call has four costs: encoding the request, sending it, decoding it, and the same three in reverse. Encoding and decoding are the parts that the application controls, and at high request rates they dominate everything except the network round trip.
gRPC on HTTP/2
gRPC maps a procedure call onto HTTP/2 streams. One call is one stream. The request headers carry the method path, the content type, and the deadline. The request body carries a sequence of length-prefixed messages. The response carries headers, then messages, then trailers that hold the status code.
Putting the status in trailers rather than headers is what makes streaming work. The server sends headers and begins streaming responses before it knows whether the call will succeed, and it reports the outcome at the end. A protocol that put the status in the response headers must buffer the whole response first.
Four call patterns follow from the stream model. Unary is one message each way. Server streaming sends many responses to one request. Client streaming sends many requests and gets one response. Bidirectional streaming runs both directions independently over one stream, with no coupling between the two.
Each message carries a five-byte prefix: one compression flag byte and a four-byte big-endian length. That framing is why a gRPC message has a hard size limit in every implementation, defaulting to 4 mebibytes, and why a request that exceeds it fails with a clear error rather than corrupting the stream.
Protocol Buffers on the wire
A protobuf message is a sequence of key and value pairs with no schema on the wire. The key is a varint holding the field number shifted left by three, combined with a three-bit wire type. Nothing names the field, so the field number is the contract.
message Reserve { string sku = 1; int32 quantity = 2; bool rush = 3; }
Reserve { sku: "ABC", quantity: 300, rush: true }
field 1, wire type 2 (length-delimited)
key = (1 << 3) | 2 = 0x0A
len = 3
bytes = 41 42 43 -> 0A 03 41 42 43
field 2, wire type 0 (varint)
key = (2 << 3) | 0 = 0x10
300 = 0b100101100 -> varint AC 02 -> 10 AC 02
field 3, wire type 0 (varint)
key = (3 << 3) | 0 = 0x18
true = 01 -> 18 01
total 10 bytes. The same value as JSON:
{"sku":"ABC","quantity":300,"rush":true} = 40 bytes
Varints use 7 bits per byte with a continuation bit, so small
numbers cost one byte and large ones cost up to ten.
Figure 1.3 Protobuf wire encoding of one small message.
Five wire types exist. Type 0 is a varint, used for integers and booleans and enumerations. Type 1 is a fixed 64-bit value. Type 2 is length-delimited, used for strings, bytes, embedded messages, and packed repeated fields. Type 5 is a fixed 32-bit value. Types 3 and 4 were groups and are deprecated.
Two encoding details cost real bandwidth when ignored. A plain int32 encodes
a negative number as a ten-byte varint, because the encoder sign-extends it to
64 bits. Use sint32 or sint64 for values that go negative, because they
apply zigzag encoding that keeps small magnitudes small. And a repeated
numeric field must be packed, which proto3 does by default, or each element
carries its own key byte.
The compatibility rules follow from the wire format. Adding a field with a new
number is safe in both directions, because an old reader skips an unknown
field using the wire type. Removing a field is safe if the number is never
reused, which is what reserved declares. Changing a field's type is safe
only within a compatible wire type group, and changing a field number is never
safe.
FlatBuffers and the cost of parsing at all
Protobuf decoding allocates objects and copies bytes. For a message with many fields where the reader touches two of them, that work is waste.
FlatBuffers stores data in a layout that is already the in-memory representation. A reader accesses a field by following an offset from a vtable and reads the value in place. There is no parse step, no allocation, and no copy. Access to any field costs a pointer dereference.
The trade is size and mutability. A FlatBuffer is typically 1.2 to 1.6 times the size of the equivalent protobuf, because it stores alignment padding and offset tables. It is also immutable in practice, so producing one means building a new buffer rather than editing an existing one.
Table 1.4: encoding cost for a 1 KB record with 20 fields
| Format | Encoded size | Encode | Decode, all fields | Access 2 fields |
|---|---|---|---|---|
| JSON, general library | 1,050 B | 2,400 ns | 3,800 ns | 3,800 ns |
| JSON, SIMD parser | 1,050 B | 1,900 ns | 600 ns | 600 ns |
| MessagePack | 720 B | 900 ns | 1,100 ns | 1,100 ns |
| Protocol Buffers | 480 B | 420 ns | 560 ns | 560 ns |
| Cap'n Proto | 620 B | 300 ns | 0 ns | 25 ns |
| FlatBuffers | 640 B | 380 ns | 0 ns | 20 ns |
| Apache Arrow, columnar batch | varies | amortized | 0 ns | 5 ns per value |
The last three rows share one property that the first four do not: access cost is independent of message size. That property decides the choice for a service that receives a large record and reads a small part of it, which describes most routing, filtering, and indexing work.
Connection pooling under multiplexing
HTTP/1.1 pools connections because each connection carries one request at a time. HTTP/2 and gRPC multiplex, so one connection carries many, and the pooling rules change completely.
Three limits bind instead. SETTINGS_MAX_CONCURRENT_STREAMS caps in-flight
calls per connection, and it commonly defaults to 100 on a server and to a
much higher value in gRPC implementations. The connection-level flow control
window caps unacknowledged bytes across all streams. And one connection maps
to one TCP connection, so it inherits one congestion window and one receive
buffer.
The load balancing consequence is the one that surprises teams. A layer 4 load balancer assigns a connection to a backend once. With long-lived multiplexed connections, every request on that connection goes to the same backend forever, so load spreads by connection count rather than by request count. Adding a backend attracts no traffic until clients reconnect.
Three fixes exist, and the right one depends on the deployment. Balance at layer 7, where the proxy terminates HTTP/2 and distributes individual streams. Use client-side balancing, where the client resolves every backend address and picks per call. Or force periodic reconnection with a maximum connection age and a jittered grace period, which redistributes load without a proxy.
Four gRPC client options that decide behavior under load
Set a deadline on every call, and propagate it, because a call with no
deadline holds a stream and a server thread until the connection
breaks. Set GRPC_ARG_MAX_CONNECTION_AGE_MS with a jitter so that
connections rebalance after a scale-out. Set keepalive with a period
above the server minimum, because a client that pings too often gets
ENHANCE_YOUR_CALM and a closed connection. And size the connection
pool from the concurrent stream limit rather than from the request
rate: one connection carries hundreds of calls, so a pool of ten
connections is usually plenty.
Compression, and when it costs more than it saves
gRPC supports per-message compression. Gzip cuts a text-heavy payload by 60 to 80 percent and costs 10 to 40 microseconds per message. Zstandard reaches similar ratios at a quarter of the CPU. Both lose to no compression on a payload under about 1 kilobyte, because the header and the setup dominate.
Inside a datacenter with 25 gigabit links, compression usually costs more than it saves for small messages, because bandwidth is cheap and latency is not. Across a wide area link or to a mobile client, compression wins clearly. Configure it per route rather than globally.
1.4 The TLS 1.3 Handshake and Its Cryptographic Cost
TLS 1.3 removed a round trip and removed most of the ways to configure it wrongly. The cipher suite list went from hundreds to five. RSA key transport is gone, so every handshake gives forward secrecy. CBC mode, compression, and renegotiation are gone, which removed several whole vulnerability classes.
The handshake in one round trip
The client sends ClientHello with its supported groups and a key share for
the group it expects the server to choose. Guessing correctly is what saves
the round trip. If the server wants a different group, it responds with
HelloRetryRequest and the handshake costs two round trips instead of one.
The server replies with ServerHello carrying its own key share. Both sides
now derive the handshake secret, and everything after that point is encrypted,
including the certificate and the extensions. The server sends
EncryptedExtensions, Certificate, CertificateVerify, and Finished. The
client validates, sends its own Finished, and sends application data.
0
|
v
HKDF-Extract <-- PSK (or zeros if no pre-shared key)
|
+--> Derive-Secret(., "ext binder" | "res binder") binder_key
+--> Derive-Secret(., "c e traffic") client_early_traffic_secret
| <-- 0-RTT data uses this
v
Early Secret
|
v
HKDF-Extract <-- (EC)DHE shared secret
|
+--> Derive-Secret(., "c hs traffic") client_handshake_traffic
+--> Derive-Secret(., "s hs traffic") server_handshake_traffic
v
Handshake Secret
|
v
HKDF-Extract <-- 0
|
+--> Derive-Secret(., "c ap traffic") client_application_traffic
+--> Derive-Secret(., "s ap traffic") server_application_traffic
+--> Derive-Secret(., "res master") resumption_master_secret
v
Master Secret
Compromise of an application traffic secret reveals nothing about the
handshake secret, because HKDF-Extract is one-way.
Figure 1.4 TLS 1.3 key schedule. Every secret derives from the one above.
Forward secrecy and what it costs
Forward secrecy means that recording today's traffic and stealing the server's private key tomorrow does not decrypt the recording. TLS 1.3 gives it on every handshake, because the session key comes from an ephemeral Diffie-Hellman exchange and the certificate key only signs.
The cost is one ephemeral key generation and one scalar multiplication per handshake on each side. On X25519 that is roughly 30 to 50 microseconds per operation on a modern core. The signature is the other half: an ECDSA P-256 signature costs about 25 microseconds to produce and about 70 to check, while an RSA-2048 signature costs about 700 microseconds to produce and about 25 to check.
That asymmetry decides certificate choice for a server under load. The server signs on every full handshake and the client checks. An RSA certificate puts the expensive operation on the server. An ECDSA certificate puts it on the client. A busy server that serves full handshakes runs several times more of them with ECDSA.
Table 1.5: cryptographic cost on one modern core
| Operation | Cost | Handshakes per second per core |
|---|---|---|
| X25519 key generation plus agreement | 30 to 50 us | |
| ECDSA P-256 sign | ~25 us | |
| ECDSA P-256 verify | ~70 us | |
| RSA-2048 sign | ~700 us | |
| RSA-2048 verify | ~25 us | |
| Full handshake, ECDSA P-256, server side | 80 to 150 us | 7,000 to 12,000 |
| Full handshake, RSA-2048, server side | 750 to 900 us | 1,100 to 1,300 |
| Resumption from a session ticket | 10 to 25 us | 40,000 to 100,000 |
| AES-128-GCM bulk, AES-NI | 4 to 8 GB/s | |
| AES-128-GCM bulk, VAES and AVX-512 | 12 to 20 GB/s | |
| ChaCha20-Poly1305 bulk, no AES hardware | 1.5 to 3 GB/s | |
| ChaCha20-Poly1305 bulk, with AVX2 | 3 to 5 GB/s |
Hardware acceleration for the bulk data
Handshakes are a small part of the total cryptographic cost on a busy server. Bulk encryption is the rest, and hardware handles it.
AES-NI provides single-instruction AES rounds and carry-less multiplication for the GHASH authentication. With it, AES-128-GCM reaches 4 to 8 gigabytes per second per core. Vector AES with AVX-512 processes four blocks per instruction and roughly triples that.
ChaCha20-Poly1305 exists for hardware without AES instructions, which in practice means older mobile processors. It uses only additions, rotations, and exclusive-or, so it runs at a predictable speed everywhere and it is naturally constant time. On a server with AES-NI it is slower than AES-GCM, and the correct configuration offers both and lets the client choose.
The cipher preference rule follows: put the client's preference first for these two suites, because the client knows what its own hardware accelerates. Most modern stacks do this by default.
Session tickets and key rotation
A session ticket is the server's state, encrypted with a key that only the server holds, and handed to the client to store. The client presents it to resume, and the server decrypts it to recover the session secret. The scheme costs the server no storage, which is why it scales.
It also creates a forward secrecy problem. The session ticket encryption key, or STEK, encrypts every ticket. An attacker who obtains the key and recorded the traffic can decrypt every resumed session that key protected. The ephemeral exchange in the original handshake gave forward secrecy, and the ticket takes it back.
Rotation bounds the exposure. Generate a new ticket key on a schedule, continue accepting the previous one until every outstanding ticket expires, and then destroy it. A common configuration rotates every few hours and keeps two or three keys valid.
Shared ticket keys across a fleet need a distribution plan
A load balancer fleet must share ticket keys, or a client that reaches a different server cannot resume and pays a full handshake. Teams solve this by distributing keys from a central source. Two failures follow. A key that never rotates gives an attacker one target that decrypts months of recorded traffic. And a key distributed through a configuration management system often lands in a repository or a log. Distribute ticket keys through the secret manager, rotate on a schedule, and alert if the age of the newest key exceeds the rotation period.
Certificate chains and handshake size
The server sends its certificate chain in the handshake, and the chain size affects latency more than most teams expect. A chain that exceeds the initial congestion window, which is commonly ten packets or about 14 kilobytes, needs an extra round trip before the client can complete the handshake.
Three controls keep it small. Send the leaf and the intermediate and not the root, because the client already has the root. Use ECDSA certificates, whose keys and signatures are a fraction of RSA's size. And enable certificate compression, which TLS 1.3 supports and which cuts a typical chain by half.
OCSP stapling belongs in the same discussion. Without it, a client that checks revocation makes a separate connection to the certificate authority, which adds a full round trip and leaks the browsing target. With stapling, the server attaches a signed, time-limited revocation status to the handshake. The operational requirement is that a failure to fetch the staple must not fail the handshake, and that the staple gets refreshed well before it expires.
Post-quantum key agreement
The threat is harvest now and decrypt later. An adversary records encrypted traffic today and decrypts it once a sufficiently large quantum computer exists. Any data whose confidentiality must outlive the next decade is already at risk, and a key exchange upgrade is the control that addresses it.
Hybrid key agreement combines a classical exchange with a post-quantum one and derives the session key from both, so the session stays secure unless both break. X25519MLKEM768 is the deployed combination, pairing X25519 with the NIST-standardized ML-KEM-768. Browsers shipped it in 2024, major content delivery networks now enable it by default, and measurement studies through 2026 put roughly half of surveyed domains on a hybrid exchange.
The cost is bytes rather than CPU. ML-KEM-768 adds about 1.2 kilobytes to the client hello and about 1.1 kilobytes to the server hello. That pushes some handshakes past the initial congestion window, which is another reason to keep the certificate chain small. The lattice operations themselves are fast, and on many measurements ML-KEM is quicker than the elliptic curve operation next to it.
Signatures are the harder half and they will migrate over years. An ML-DSA-65 public key is about 1,952 bytes and a signature about 3,309, against 32 and 64 for an elliptic curve. A chain carrying several of those grows the handshake by several kilobytes. Plan the key exchange migration now and track the IETF work on smaller chains and trust anchor negotiation for the signature half.
Sources and further reading
RFC 9000 for QUIC transport, RFC 9001 for its use of TLS, RFC 9002 for loss
detection and congestion control, and RFC 9114 for HTTP/3. RFC 8446 for TLS
1.3 and the IETF drafts on ML-KEM hybrid key agreement. The Linux kernel
documentation for XDP and AF_XDP, and the sendfile(2), splice(2), and
msg_zerocopy pages. The DPDK programmer's guide. The RDMA Aware Networks
Programming User Manual for queue pairs and memory registration. The Protocol
Buffers encoding documentation and the FlatBuffers internals page. The gRPC
core documentation on HTTP/2 mapping and keepalive.
What this chapter established
- HTTP/2 removed application head-of-line blocking and exposed TCP's. One lost segment stalls every stream behind the gap, and the stall costs a round trip.
- QUIC moves ordering down to the stream, so a lost packet blocks only the streams whose data it carried.
- A QUIC packet number is never reused, which removes TCP's retransmission ambiguity and keeps the round-trip estimate clean.
- QUIC applies flow control per stream and per connection. A connection window below the bandwidth-delay product caps throughput with no loss and no CPU saturation.
- A connection identifier separates the connection from the four-tuple, so a network change costs no handshake. A load balancer that hashes the tuple breaks this.
- Zero round-trip data is replayable. Send only idempotent requests in it, track recently seen tickets, and keep ticket lifetimes short.
sendfileandspliceremove copies inside the kernel.MSG_ZEROCOPYpays above about 10 kilobytes and costs more below it.- XDP drops a packet in tens of nanoseconds against microseconds for
iptables. AF_XDP gives tens of millions of packets per second and keeps the machine manageable. - DPDK removes the kernel entirely and takes the card, the cores, and the tooling with it. It earns that in packet processing and nowhere else.
- RoCE assumes a near-lossless fabric. Without priority flow control it collapses under congestion, and with it misconfigured it spreads pause frames into unrelated traffic.
- Protobuf field numbers are the contract. Never reuse a number, mark removed ones reserved, and use
sint32for values that go negative. - FlatBuffers and Cap'n Proto make field access independent of message size, which wins when a service reads a small part of a large record.
- A layer 4 load balancer plus long-lived multiplexed connections spreads load by connection, not by request. Balance at layer 7, balance in the client, or age connections out.
- ECDSA puts the expensive handshake operation on the client and RSA puts it on the server. A busy server runs several times more full handshakes with ECDSA.
- A session ticket key takes back the forward secrecy that the handshake gave. Rotate it on a schedule and distribute it through the secret manager.
- Hybrid post-quantum key agreement is deployed now and costs bytes rather than CPU. Keep the certificate chain small so that the larger hello still fits the initial window.