High-Performance Networking and Protocols

The path a packet takes and what each step costs

A byte that leaves an application traverses a fixed sequence. The application writes to a socket, which copies into a kernel buffer. TCP segments it, adds headers, and hands it to the IP layer. The driver builds a descriptor and rings a doorbell. The network card reads the buffer by direct memory access and puts bits on the wire. The receiver reverses the sequence.

Each step costs. The socket write costs a system call and a copy. The protocol processing costs a few hundred nanoseconds. The driver and the card cost a descriptor write and an interrupt or a poll. The wire costs the distance. On the receive side, Chapter 2 described NAPI polling, softirqs, and the steering that decides which core does the work.

The table gives the whole case for kernel bypass in one column. Kernel processing costs 1.5 to 3 microseconds per packet in each direction. At 64 byte packets that caps one core at roughly 300,000 to 600,000 packets per second. A 100 gigabit link carries 148 million small packets per second. Ordinary sockets reach a fraction of a line rate, and that is fine for a web service and fatal for a packet processor.

TCP mechanics that decide latency

A TCP connection opens with a three-way handshake, which costs one round trip before any data moves. TCP Fast Open lets a client send data in the SYN on a repeat connection, which removes the round trip and carries a replay risk that the application must tolerate.

Congestion control decides how fast a sender can send. The classical loss-based algorithms, Reno and CUBIC, grow the window until a loss occurs and then back off. They interpret loss as congestion, which is correct on a link with small buffers and wrong on a link with large ones. A deep buffer absorbs the excess and adds delay instead of dropping, so the sender keeps growing and the queue keeps filling. That is bufferbloat, and it produces hundreds of milliseconds of added latency on a link that reports no loss at all.

BBR takes a different signal. It estimates the bottleneck bandwidth and the minimum round-trip time, and it paces the send rate to the product of the two. It therefore keeps the bottleneck busy without filling its queue. BBR version 1 was aggressive against CUBIC flows and starved them in some conditions. Version 3 addresses the fairness and the loss response, and it is the version to deploy in 2026.

Flow control is separate from congestion control. The receiver advertises a window, and the sender never has more unacknowledged data in flight than that window. The maximum useful in-flight data equals the bandwidth multiplied by the round-trip time. A 10 gigabit link with an 80 millisecond round trip needs 100 megabytes in flight, and a default socket buffer of a few megabytes caps the throughput at a fraction of the link.

# Bandwidth-delay product sets the buffer you need.
#   10 Gb/s * 80 ms = 100 MB in flight.
sysctl -w net.ipv4.tcp_rmem="4096 131072 134217728"
sysctl -w net.ipv4.tcp_wmem="4096 65536 134217728"
sysctl -w net.core.rmem_max=134217728
sysctl -w net.core.wmem_max=134217728

# Modern congestion control and queue discipline.
sysctl -w net.ipv4.tcp_congestion_control=bbr
sysctl -w net.core.default_qdisc=fq

# Accept queue depth. A full queue drops SYNs silently.
sysctl -w net.core.somaxconn=8192
sysctl -w net.ipv4.tcp_max_syn_backlog=8192

Three small behaviors produce latency bugs that look mysterious.

Nagle's algorithm delays a small write until the previous data is acknowledged, so that a stream of tiny writes becomes fewer packets. Delayed acknowledgement makes the receiver wait up to 40 milliseconds before acknowledging, so that it can combine the acknowledgement with a response. Together they produce a 40 millisecond stall on a request-response protocol that writes a header and a body separately. Set TCP_NODELAY on every request-response socket, and write the whole message in one call.

TIME_WAIT keeps a closed connection's tuple reserved for twice the maximum segment lifetime, which is 60 seconds on Linux. A server that opens many short outbound connections exhausts its ephemeral ports. The correct fixes are connection reuse and keep-alive, not tcp_tw_recycle, which breaks behind network address translation and no longer exists in current kernels.

The accept queue is finite. A listening socket holds completed connections until the application calls accept. When the queue fills, Linux drops the new connection silently by default, and the client sees a timeout rather than a refusal. Watch the overflow counter in netstat -s and size somaxconn together with the application backlog argument.

TLS 1.3

TLS 1.3 removed every round trip that a handshake can remove. The client sends its hello with a key share for the group it expects the server to pick. The server replies with its own key share, its certificate, and a finished message, all encrypted after the key exchange. The client sends its finished message and application data. One round trip, and everything after the server hello is encrypted.

The protocol also removed the legacy that caused most TLS vulnerabilities. It has no RSA key transport, so every handshake gives forward secrecy. It has no CBC mode and no non-authenticated encryption, so padding oracle attacks disappear. It has no renegotiation and no compression. The cipher suite list went from hundreds to five.

Session resumption uses a pre-shared key from a previous connection. With 0-RTT, the client sends application data in its first flight, before the handshake completes. The saving is a full round trip, and the cost is that 0-RTT data is replayable, because the server has no way to tell a legitimate retransmission from a replay. Send only idempotent requests in 0-RTT data, and never a state-changing one.

Post-quantum key exchange is now mainstream. The concern is harvest now and decrypt later, where an adversary records encrypted traffic today and decrypts it once a quantum computer exists. Hybrid key agreement answers it by combining a classical exchange with a post-quantum one, so the session stays secure unless both break. X25519MLKEM768 is the deployed combination, pairing X25519 with the NIST-standardized ML-KEM-768. Browsers shipped it in 2024, and major content delivery networks now enable it by default. Measurement studies through 2026 put roughly half of surveyed domains on a hybrid exchange and roughly half still fully classical.

HTTP/1.1, HTTP/2, and HTTP/3

HTTP/1.1 sends one request per connection at a time. Pipelining existed and failed in practice because intermediaries handled it badly and because a slow response blocks everything behind it. Browsers worked around the limit by opening six connections per origin, which multiplies the handshake cost and the congestion control state.

HTTP/2 multiplexes many streams over one TCP connection. It frames everything, compresses headers with HPACK, and supports priorities and server push. Multiplexing removed the application-level head-of-line blocking and exposed a lower one: TCP delivers bytes in order, so one lost segment stalls every stream on that connection until the retransmission arrives. On a lossy mobile link, HTTP/2 over TCP can perform worse than six separate HTTP/1.1 connections.

HTTP/3 runs over QUIC, which runs over UDP and implements its own reliability, ordering, and congestion control per stream. A lost packet stalls only the streams whose data it carried. QUIC also merges the transport and the cryptographic handshake, so a new connection costs one round trip in total rather than one for TCP plus one for TLS.

  HTTP/1.1 or HTTP/2 over TCP + TLS 1.3
    client            server
      |---- SYN ------->|
      |<--- SYN/ACK ----|          RTT 1  (TCP)
      |---- ACK+CH ---->|
      |<--- SH+FIN -----|          RTT 2  (TLS)
      |---- FIN+DATA -->|
                                   first byte of request: after 2 RTT

  HTTP/3 over QUIC
      |---- Initial(CH) ->|
      |<--- Initial+HS ---|        RTT 1  (transport + crypto together)
      |---- HS+DATA ----->|
                                   first byte of request: after 1 RTT

  Repeat visit, session resumption with 0-RTT
      |---- Initial(CH)+DATA ->|   request travels in the first flight
                                   first byte of request: after 0 RTT
                                   replayable: idempotent requests only

QUIC adds two properties that TCP cannot have. A connection identifier separates the connection from the IP and port tuple, so a phone that moves from wifi to cellular keeps its connection with no new handshake. Every packet is encrypted including most of the header, which stops middleboxes from inspecting and from ossifying the protocol.

QUIC costs CPU. It runs in user space, it has no decades of kernel optimization, and it processes one UDP datagram at a time unless the stack uses segmentation offload. Early deployments reported two to three times the CPU per byte against TCP. The gap narrowed with generic segmentation offload for UDP, with better implementations, and with partial hardware support.

Adoption in 2026 sits around 35 percent of the top ten million sites on the server side, with roughly 93 percent of browser users on a client that supports it. The performance benefit is real and it is conditional. On a lossy, high-latency, or mobile path, HTTP/3 wins clearly. On a stable low-latency wired path, the difference is small and sometimes negative.

gRPC and the shape of a service call

gRPC runs protocol buffers over HTTP/2. The interface definition file generates client stubs and server skeletons in every supported language, so the contract is a build artifact rather than a document.

Four call patterns exist. Unary is one request and one response. Server streaming sends many responses to one request. Client streaming sends many requests and gets one response. Bidirectional streaming runs both directions independently over one stream. The streaming modes are the reason to pick gRPC over ordinary HTTP with JSON, because they are ordinary in gRPC and awkward otherwise.

Protocol buffers encode fields by number with a compact varint format. The encoding is typically three to ten times smaller than the equivalent JSON and parses several times faster, because it needs no text scanning and no allocation per field name. The field numbers are the contract: never reuse a number, never change a type, and mark removed numbers as reserved.

Deadlines are the feature that teams underuse. A gRPC deadline propagates automatically through the call chain, so a 200 millisecond budget at the edge becomes a shrinking budget at every hop. A service that receives an already-expired deadline fails immediately, instead of doing work that nobody will read. Set a deadline on every call. A call with no deadline holds resources until the connection breaks.

syntax = "proto3";
package inventory.v1;

service Inventory {
  rpc Reserve(ReserveRequest) returns (ReserveResponse);
  rpc WatchStock(WatchRequest) returns (stream StockEvent);
}

message ReserveRequest {
  string sku = 1;
  int32 quantity = 2;
  string idempotency_key = 3;   // the client generates this, not the server
  reserved 4, 5;                // numbers of two removed fields
}

The idempotency key in that message is the answer to the problem that opened Chapter 8. A retry carries the same key, the server recognizes it, and the second call returns the first call's result without performing the action again. Store the key with the result for at least as long as the client retries, and make the store part of the same transaction as the effect.

Load balancing for gRPC differs from HTTP/1.1 because connections are long-lived and multiplexed. A layer 4 load balancer pins a connection to one backend, and every request on that connection goes to the same place, so the load spreads by connection and not by request. Balance at layer 7, or use client-side balancing with a resolver that returns every backend address, which is what gRPC calls a round_robin policy over a dns or xds resolver. Chapter 11 puts this in a service mesh.

Kernel bypass

Three technologies remove the kernel from the data path, at three different points on the cost and complexity curve.

DPDK removes it entirely. It binds the network card to a user-space driver, maps the descriptor rings into the process, and polls them. There are no interrupts, no system calls, and no copies, and one core reaches 10 to 15 million packets per second. The costs are total: the card belongs to the application, the core spins at 100 percent, and the application must implement whatever it needs from the protocol stack.

AF_XDP keeps the kernel and gives it a fast lane. An eBPF program in the driver redirects selected packets into a memory region shared with user space, and everything else follows the normal path. The application reaches 10 to 20 million packets per second for the redirected traffic while the machine keeps working normally for everything else. This is the pragmatic choice for most teams in 2026.

RDMA lets one machine read and write another machine's memory with no CPU involvement on the remote side. InfiniBand and RoCE, which is RDMA over converged Ethernet, both provide it. End-to-end latency runs 1 to 2 microseconds. The constraint is that RoCE needs a lossless fabric with priority flow control, which means a network designed for it rather than a general datacenter network. Storage systems, high-performance computing, and GPU collective communication all depend on it, and Chapter 13 returns to the last one.

Load balancing

A layer 4 load balancer forwards packets and reads only the transport header. It is fast, it preserves end-to-end TLS, and it cannot route on a path or a header. A layer 7 load balancer terminates the connection, parses the request, and routes on anything. It is slower and far more capable.

Consistent hashing keeps a client on the same backend across membership changes. A plain modulo hash over the backend count remaps almost every key when the count changes, which empties every cache at once. Consistent hashing remaps only the keys that belonged to the departed backend. The ring implementation uses virtual nodes to spread load evenly, and Google's Maglev uses a lookup table that gives better balance and near-minimal disruption.

Direct server return has the load balancer forward the request and the backend answer the client directly. Response traffic, which is usually much larger than request traffic, never crosses the balancer. Cilium, Maglev, and most high-throughput layer 4 balancers support it. It needs the backend to own the virtual address and it complicates health tracking.

Anycast announces one address from many locations and lets routing pick the nearest. It gives geographic distribution with no client logic and no DNS delay, and it is the basis of every large content delivery network and public resolver. The caveat is that a routing change moves a client mid-connection, which a stateless protocol tolerates and a long TCP connection does not. QUIC connection migration helps here, because the connection identifier survives the path change.

Tuning that pays

A short list covers most of the available gain.

• Set TCP_NODELAY on every request-response socket, and write each message with one call.
• Use SO_REUSEPORT with one listener per worker thread.
• Size socket buffers from the bandwidth-delay product, not from the default.
• Use BBRv3 with the fq queue discipline for paths with any distance.
• Turn on TSO, GSO, and GRO so that the card and the stack handle large segments.
• Use jumbo frames inside a datacenter when every hop supports 9000 bytes.
• Keep connections alive and pool them. A new connection costs a handshake and a slow start.
• Set a deadline on every call, and propagate it.
• Measure at the p99 and the p99.9, because the mean hides every queueing effect from Chapter 4.

Sources and further reading

W. Richard Stevens and others, "TCP/IP Illustrated", volume 1, third edition. Neal Cardwell and others on BBR, and the BBRv3 material from the IETF congestion control working group. RFC 8446 for TLS 1.3, RFC 9000 for QUIC, and RFC 9114 for HTTP/3. The IETF drafts on ML-KEM hybrid key agreement for TLS 1.3. Daniel Eisenbud and others, "Maglev: A Fast and Reliable Software Network Load Balancer", NSDI 2016. The DPDK and AF_XDP documentation, and the Cilium material on eBPF-based load balancing.