Modern AI Infrastructure and Agentic System Design

9.1 High-Throughput Model Serving Architectures

A transformer forward pass costs about two floating point operations per parameter per token. Generating one token requires reading every weight that participates. For a dense model at batch size one, that is every weight, once, per token.

The generation rate is therefore bounded by memory bandwidth divided by model size in bytes, and not by the arithmetic rate at all. A 70 billion parameter model in BF16 occupies 140 gigabytes. On an accelerator with 8 terabytes per second of memory bandwidth, that caps single-stream generation near 57 tokens per second, whatever the arithmetic units can do.

The table explains four industry behaviors at once. Quantization is popular because halving the weight bytes doubles the token rate. Mixture-of-experts architectures are popular because only the routed experts get read. Larger batches are essential because they amortize one weight read across many sequences. And each accelerator generation advertises bandwidth as prominently as it advertises flops.

Two workloads wearing one name

Inference has two phases with opposite resource profiles, and treating them as one workload is the most common architectural error in this area.

Prefill processes the prompt. Every prompt token goes through the model at once, as a matrix multiplication with a large inner dimension. Arithmetic intensity is high, the accelerator runs near its floating point peak, and the phase is compute bound.

Decode produces one token at a time. Each step reads all the weights and performs a matrix-vector product per sequence. Arithmetic intensity is close to one operation per byte, far below the ridge point of any accelerator, so the phase is memory bound and the arithmetic units idle.

Two conclusions follow. Batching helps decode enormously and helps prefill very little, because prefill already saturates the arithmetic units. And running both phases on one accelerator makes each worse: a long prefill blocks every decode step behind it, which appears as a latency spike on requests that were already in progress.

Chunked prefill splits a long prompt into pieces and interleaves them with decode steps, so no single prefill monopolizes the device. It cuts tail latency and slightly reduces prefill throughput.

Disaggregated serving puts the two phases on different machines. Prefill nodes are sized for compute, decode nodes for bandwidth and memory capacity, and the key and value cache transfers between them over a fast interconnect. Each phase then scales independently to its own demand. TensorRT-LLM, SGLang, and vLLM all support the pattern.

The key and value cache

Attention needs the keys and values of every previous token. Recomputing them at each step makes decode quadratic, so the system caches them. That cache is the dominant memory consumer in a production serving system, and its size has a closed form.

Bytes per token equals two, for the keys and the values, multiplied by four terms: the layer count, the number of key and value heads, the head dimension, and the bytes per element.

Take a 70 billion parameter model with 80 layers, 8 key and value heads under grouped query attention, a head dimension of 128, and BF16 elements. Per token that is 2 times 80 times 8 times 128 times 2, which is 327,680 bytes, or 320 kibibytes. A 32,000 token context costs 10 gibibytes for one sequence.

Every recent large model uses grouped query attention, multi-query attention, or a latent compression scheme, and Table 9.2 is the reason.

PagedAttention manages the cache the way an operating system manages memory. Instead of reserving a contiguous block for each sequence's maximum length, it allocates fixed-size blocks of a few tokens and keeps a per-sequence block table mapping logical positions to physical blocks. Fragmentation falls from 60 to 80 percent waste to a few percent, and the freed memory becomes additional batch capacity.

  physical KV blocks (16 tokens each)
  +----+----+----+----+----+----+----+----+----+----+
  | b0 | b1 | b2 | b3 | b4 | b5 | b6 | b7 | b8 | b9 |
  +----+----+----+----+----+----+----+----+----+----+
    ^    ^    ^         ^         ^
    |    |    |         |         |
  shared system prompt  |         |
  refcount = 3          |         |

  seq A block table: [b0, b1, b2, b3, b7]
  seq B block table: [b0, b1, b2, b4, b8]
  seq C block table: [b0, b1, b2, b5]

  Three sequences share the 48-token system prompt at zero extra cost.
  A write to a shared block triggers copy-on-write into a fresh block,
  exactly as a page fault would.

  Prefix caching turns a repeated 2,000-token system prompt from a
  per-request prefill cost into a one-time cost. For chat and agent
  traffic this is the single largest available win.

Batching and the scheduler

Static batching waits for a fixed number of requests, runs them together, and returns when the longest finishes. Every short request waits for the longest, and the accelerator idles on finished sequences.

Continuous batching runs one decode step for the whole batch, removes finished sequences, and admits waiting requests into the freed slots. No request waits for an unrelated one. Combined with PagedAttention and chunked prefill, this is the design that carries every serious inference server, and together the three report three to five times the throughput of a naive loop on the same hardware.

The scheduler has one tuning axis that matters. A larger running batch raises throughput and raises the time between tokens for every sequence in it, because each step does more work. Set the maximum running batch from the inter-token latency target, not from the memory that happens to be free.

Preemption handles the case where the cache fills. A sequence can be evicted and either recomputed later or swapped to host memory. Recomputation costs a prefill and no bandwidth over PCIe. Swapping costs a transfer and no compute. Under memory pressure, the choice between them is a real tuning decision, and recomputation usually wins for short sequences while swapping wins for long ones.

Optimize goodput, which is throughput subject to the latency constraint. A server that reaches high throughput by letting time to first token reach ten seconds produced tokens that no user waited for.

Quantization and speculative decoding

Reducing numeric precision cuts memory and, because decode is bandwidth bound, cuts latency proportionally. FP8 is the production default on supporting hardware, with quality loss within measurement noise for most models. FP4 doubles the rate again with per-block scaling factors and careful calibration. Weight-only integer methods such as GPTQ and AWQ keep activations in higher precision and store weights at 4 bits, which matches the memory-bound decode phase exactly.

The rule for a production decision is short. Quantize weights first, because that is where the bandwidth goes. Quantize the key and value cache second, because that is where the capacity goes. Quantize activations last, because that is where the quality goes. Measure on your own evaluation set, since quantization damage is task dependent.

Speculative decoding spends the idle arithmetic. A small draft model proposes k tokens, and the large model evaluates all k in one forward pass, which costs about the same as generating one token because the pass is bandwidth bound either way. A rejection sampling step accepts the longest correct prefix and guarantees that the output distribution matches the large model alone. Acceptance rates of 60 to 80 percent give speedups of two to three times with no quality change.

9.2 Multi-GPU Parallelism Strategies

A model that does not fit one accelerator must split across several. Five strategies exist, and the right combination depends on the interconnect far more than on the model.

The five strategies and their communication

Data parallelism replicates the model and splits the batch. Communication is one gradient all-reduce per training step and none at inference. It scales well and requires the whole model to fit on one device.

Tensor parallelism splits each weight matrix across devices. Megatron-LM established the standard layout: split the first linear layer of the feed-forward block by columns and the second by rows, so that only one all-reduce is needed per block rather than two. Attention splits by head, so each device holds whole heads. Every transformer layer then needs two all-reduce operations in the forward pass, which is frequent and latency sensitive, so tensor parallelism belongs inside one high-bandwidth domain and degrades badly across ordinary network links.

Pipeline parallelism assigns whole layers to different devices and passes activations between stages. Communication is point-to-point and small, so it tolerates a slower link. The cost is the pipeline bubble: with p stages and m microbatches, the idle fraction is p minus 1 divided by m plus p minus 1.

  NAIVE, 4 stages, 4 microbatches: bubble = 3/7 = 43 percent idle
  stage 0  F1 F2 F3 F4 .  .  .  B4 B3 B2 B1
  stage 1  .  F1 F2 F3 F4 .  B4 B3 B2 B1 .
  stage 2  .  .  F1 F2 F3 F4 B4 B3 B2 B1 . .
  stage 3  .  .  .  F1 F2 F3 B3 B2 B1 .  . . .
           <-- bubble -->        <-- bubble -->

  1F1B (one forward, one backward), 16 microbatches:
  bubble = 3/19 = 16 percent, and peak activation memory is bounded
  because a backward pass frees activations as soon as it can run

  Interleaved 1F1B with 2 virtual stages per device:
  bubble = 3/(2*16 + 3) = 8.6 percent, at the cost of more
  point-to-point messages

  Rule: raise the microbatch count until the bubble is acceptable,
  then stop, because activation memory grows with microbatches in flight.

Expert parallelism places different experts of a mixture-of-experts layer on different devices. Each token routes to its experts, so communication is an all-to-all exchange twice per such layer. Load balance is the problem. A routing function that favors a few experts leaves most devices idle. Training adds an auxiliary loss that encourages balance, and serving adds capacity factors and token dropping to bound the imbalance.

Sequence or context parallelism splits one long sequence across devices, which is what makes very long contexts feasible when one sequence's activations and cache exceed a single device. Ring attention passes key and value blocks around a ring so that each device computes attention against every block without materializing the full matrix.

The practical recipe follows the hardware hierarchy. Use tensor parallelism inside a node, where NVLink gives 1.8 terabytes per second per GPU. Use pipeline or expert parallelism across nodes, where the fabric gives a fraction of that. Use data parallelism across replicas. The ratio between NVLink and PCIe is roughly 28 to 1, and this recipe is that ratio applied to a model.

ZeRO and memory sharding

Training memory per parameter is the number that surprises people. With Adam in mixed precision, each parameter costs 2 bytes for the BF16 weight, 2 for the gradient, 4 for the FP32 master copy, 4 for the first moment, and 4 for the second moment. That is 16 bytes per parameter before any activation memory, so a 70 billion parameter model needs about 1.1 terabytes of optimizer state alone.

ZeRO shards that state across the data-parallel group rather than replicating it. Stage one shards the optimizer state, which cuts memory by about four times. Stage two adds gradients, cutting by about eight. Stage three adds the parameters themselves, cutting linearly with the group size.

Stage three works by gathering each layer's parameters just before use and releasing them after, which trades communication for memory. PyTorch's Fully Sharded Data Parallel implements the same idea natively. The practical consequence is that a model far larger than one device's memory trains on ordinary data-parallel hardware, at the cost of an all-gather per layer per step.

Offloading extends it further by moving optimizer state, and optionally parameters, to host memory or to NVMe. It makes very large models trainable on small clusters and it is slow, because the transfer is bounded by PCIe rather than by HBM. It is a capability of last resort rather than a performance technique.

Activation checkpointing trades compute for memory in the other direction. It discards intermediate activations during the forward pass and recomputes them during the backward pass, which cuts activation memory by a large factor at roughly 30 percent more compute. Selective checkpointing recomputes only the cheap layers and keeps the expensive ones, which recovers most of the memory at a fraction of the cost.

Failure handling at cluster scale

A ten thousand accelerator job encounters hardware failures on a timescale of hours. Failure handling is therefore not an operational detail, it is the difference between a training run that finishes and one that does not.

Four mechanisms carry it. Asynchronous checkpointing copies state to host memory quickly and writes to durable storage in the background, so the job stalls for a fraction of a second rather than for minutes. Fast failure detection uses collective operation timeouts rather than waiting for a process to exit, because a hung accelerator does not exit. Elastic training lets the job continue at a reduced world size rather than halting. And spare nodes let a failed rank be replaced without rebuilding the whole job.

The metric to track is goodput here as well: the fraction of wall-clock time spent doing useful computation rather than checkpointing, restarting, or waiting for a straggler. A large job with poor failure handling routinely loses 20 to 40 percent of its wall clock to those three.

9.3 Distributed Retrieval Infrastructure and Vector Databases

Retrieval-augmented generation puts relevant documents into the prompt. The infrastructure question is how to find those documents from a corpus of hundreds of millions of vectors in a few milliseconds.

Why exact search stops working

Above about 20 dimensions, every tree structure degrades to a full scan. The distance between the nearest and the farthest point converges, so pruning stops working. That is the curse of dimensionality, and embeddings have 384 to 4,096 dimensions.

Approximate nearest neighbor search replaced exact search. The measure is recall at k: of the true k nearest neighbors, how many did the search return? Production systems target 0.95 to 0.99, and the last few percent cost disproportionately.

The three index families

HNSW builds a layered proximity graph. The top layer holds few nodes with long edges, and each lower layer adds nodes and shorter edges. A search starts at the top, greedily walks toward the query, drops a layer, and repeats. It is the default for most production workloads in 2026: high recall out of the box, and it absorbs inserts without a rebuild.

Three parameters govern it. M is the number of connections per node per layer, typically 16 to 64, and it sets both memory and recall. efConstruction is the candidate list size during build, typically 100 to 500, and it trades build time for graph quality. efSearch is the candidate list size at query time, and it is the runtime dial between latency and recall.

IVF partitions the space with k-means into nlist cells, assigns each vector to its nearest centroid, and searches only the nprobe nearest cells. It uses far less memory than a graph and gives lower recall at the same latency. Product quantization compresses each vector by splitting it into sub-vectors and replacing each with a codebook index, which cuts memory by 10 to 50 times and costs 2 to 5 percent of recall.

DiskANN keeps the graph on NVMe rather than in memory, with a compressed representation cached in RAM for the distance computations and full vectors read from disk only for the candidates. Query latency is higher, at roughly 5 to 20 milliseconds at the median, and memory falls by orders of magnitude. For billion-scale corpora it is the practical path.

Deletion is the operational weakness of graph indexes. Removing a node breaks paths through it, so implementations mark it deleted, filter it from results, and rebuild periodically. A collection with heavy churn accumulates tombstones that degrade both recall and latency until the rebuild runs, and the rebuild is expensive. Plan the rebuild cadence at design time rather than discovering it.

Sharding and replication

Two sharding strategies exist and they suit different query patterns.

Random sharding spreads vectors evenly and requires every query to fan out to every shard, with results merged by score. Recall is preserved and latency is the maximum over shards, so Section 8.3's fan-out arithmetic applies directly: with 20 shards, the query p50 tracks the shard p95.

Semantic or tenant sharding routes by an attribute, so a query touches one shard. Latency is a single shard's latency and load can be uneven. For a multi-tenant application it is almost always the right choice, because a tenant's query never needs another tenant's vectors and the isolation is useful for deletion and for compliance.

Replication serves read scale and availability. Because the index is mostly read-only, replicas are cheap: build once and copy the index file. The practical pattern separates a writer that builds and a set of read replicas that mount the result, which makes index building an offline job rather than a capacity problem on the serving path.

Filtered search

Almost every real query has a filter: this tenant, this date range, this document type, these permissions. Combining a filter with a vector search is harder than either alone.

Pre-filtering computes the matching set first and searches only within it. It is exact and it defeats the index, because a graph index cannot be traversed over an arbitrary subset without losing connectivity. At low selectivity it degenerates to a scan.

Post-filtering searches first and discards non-matching results. It keeps the index fast and it returns too few results when the filter is selective: searching for 10 and filtering to 0.1 percent returns nothing. The workaround is over-fetching, which multiplies the search cost and still gives no guarantee.

Filtered graph traversal is the modern answer. The index evaluates the predicate during traversal and continues through non-matching nodes for connectivity while collecting only matching ones. Systems that implement it switch strategy based on estimated selectivity: post-filter for a broad filter, filtered traversal in the middle, and exact scan when the filter is narrow enough that the matching set is small.

Hybrid search and fusion

Dense vector search finds semantic similarity and misses exact terms. A query for an error code, a product identifier, or a rare proper noun retrieves plausible neighbors and not the document that contains the string. Lexical search with BM25 finds exact terms and misses paraphrase.

Hybrid search runs both and fuses the results. Reciprocal rank fusion is the usual method because it needs no score normalization: for each document, sum 1 divided by k plus the rank, over each result list, with k typically 60. Score normalization plus a weighted sum also works and requires calibrating two score distributions that are not comparable, which is why rank fusion is the common default.

def reciprocal_rank_fusion(result_lists, k=60, top_n=20):
    """Fuse several ranked lists without normalizing their scores.
    Each list is an ordered sequence of document ids, best first."""
    scores = {}
    for results in result_lists:
        for rank, doc_id in enumerate(results, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)[:top_n]


def retrieve(query, filters, top_n=20):
    dense  = vector_index.search(embed(query), filters, k=100)
    sparse = bm25_index.search(query, filters, k=100)
    fused  = reciprocal_rank_fusion([dense, sparse], top_n=top_n * 3)
    # A cross-encoder scores query and document together, which is far
    # more accurate and far too slow to run over the whole corpus.
    return cross_encoder.rerank(query, fused)[:top_n]

Reranking is the third stage and it produces the largest quality gain per unit of work. A bi-encoder embeds the query and the document separately, which is what makes the index possible and what loses the interaction between them. A cross-encoder processes the pair together and scores it, which is far more accurate and costs a model forward pass per candidate. Running it over 50 to 200 candidates from the first stage is affordable, and it commonly moves recall at 10 by several points.

Chunking decides what any of this can retrieve. A chunk too large dilutes the embedding across several topics. A chunk too small loses the context that makes the passage meaningful. Overlapping windows of 200 to 500 tokens with 10 to 20 percent overlap is a reasonable default, and structure-aware chunking that respects headings, paragraphs, and code blocks beats fixed windows on most corpora. Storing a small chunk for retrieval and returning its larger parent for generation gets both properties.

9.4 Production Agentic Workflows

An agent is a loop: observe, decide, act, repeat. It calls a model, the model requests a tool, the system runs the tool, the result goes back, and the loop continues until the model produces an answer or a limit is reached.

That loop is a distributed workflow with unreliable steps, long duration, and non-deterministic decisions. Every problem in this book applies to it.

Why a plain loop fails in production

A loop held in process memory loses everything when the process restarts. A deployment, an out-of-memory kill, a node drain, or a spot instance reclamation ends the agent halfway through, with three tools already executed and no record of what they did.

The failure is worse than a lost request, because the tools had side effects. The agent charged a card, sent an email, and created a ticket, and the retry does all three again.

Four properties are needed and none of them comes free. The state must survive a process restart. Each step must be individually retryable. Side effects must be exactly-once or idempotent. And a human must be able to inspect and intervene in a run that is still in progress.

Durable execution

Durable execution journals every step so that a workflow resumes from where it stopped, whatever crashed. The engine records each step's result in a persistent log. On recovery it replays the workflow code, and every step that already has a journal entry returns its recorded result instead of executing again. Execution continues from the first step with no entry.

The requirement that makes it work is determinism. Workflow code must produce the same sequence of steps from the same inputs and the same journal. It therefore cannot read the clock, generate a random number, iterate an unordered map, or call an external service directly. Everything non-deterministic becomes a recorded step, which the engine calls an activity.

  FIRST EXECUTION                        journal
    start workflow                       [ WorkflowStarted ]
    activity: fetch_context()  -> ctx    [ ActivityCompleted #1, result ]
    activity: llm_call(ctx)    -> plan   [ ActivityCompleted #2, result ]
    activity: charge_card(...)  -> ok    [ ActivityCompleted #3, result ]
    activity: send_email(...)   -> CRASH [ ActivityScheduled #4 ]

  REPLAY AFTER RESTART
    start workflow                       replayed from the journal
    activity: fetch_context()  -> ctx    returned from entry #1, NOT called
    activity: llm_call(ctx)    -> plan   returned from entry #2, NOT called
    activity: charge_card(...)  -> ok    returned from entry #3, NOT called
    activity: send_email(...)            no entry: EXECUTES NOW

  The card is charged exactly once across the crash. The model is
  called once, which also means the cost is paid once.

  This only holds if the workflow function is deterministic. A call to
  time.time() or random() in workflow code returns a different value on
  replay, the code takes a different branch, and the journal no longer
  matches the execution. That is a non-determinism error, and engines
  detect and fail on it rather than corrupting the run.

Durable execution became a headline theme in 2026, with Temporal the most visible implementation alongside Restate, Inngest, and several others. One pattern settled in practice. A durable engine runs the outer workflow, which handles retries, multi-day waits, compensation, and cross-service calls. A graph-based agent framework runs the inner reasoning loop, where the model calls and the tool use happen.

Explicit state machines

Durable execution gives recoverability. It does not give a bounded set of states, and an agent that can reach any state from any state is impossible to reason about, to test, or to constrain.

An explicit state machine names the states and the legal transitions: planning, retrieving, calling a tool, waiting for approval, summarizing, failed, done. Each transition has a guard. Each state has a timeout. Each terminal state has a defined outcome.

Three benefits follow immediately. A run that is stuck names the state it is stuck in, rather than presenting an opaque loop. A state can be marked as requiring human approval, which is the only workable control for actions with irreversible consequences. And a test can drive the machine through each transition with a stubbed model, which gives a test suite that does not depend on model output.

The step limit, the wall-clock limit, and the token budget belong at the machine level. An agent without them will, eventually, loop: the model calls a search tool, gets an unhelpful result, decides to search again with a slightly different query, and repeats until something stops it. A budget is the thing that stops it, and it is cheaper than the incident.

Memory

An agent's memory is several stores with different lifetimes, and conflating them is a common design error.

Working memory is the current context window: the system prompt, the recent turns, the tool results in flight. It is bounded by the context length and by the inter-token latency that a long context costs.

Episodic memory holds past interactions for this user or this session. It is retrieved rather than always present, usually through the vector infrastructure of Section 9.3, and it needs recency weighting so that an old fact does not outrank a recent correction.

Semantic memory holds durable facts extracted from interactions: a preference, a constraint, an entity relationship. It needs a write path with conflict resolution, because the user's stated preference changes and the store must not keep both.

Procedural memory holds learned workflows and tool usage patterns, which in practice is prompt or policy state rather than a separate store.

Two mechanisms keep working memory viable over a long run. Summarization compresses old turns into a shorter form when the context approaches its limit, which loses detail and is usually the right trade. Offloading writes detail to an external store and keeps a reference in context, which preserves detail and requires the agent to retrieve it deliberately.

Streaming and backpressure

An agent streams tokens to a user, and the user's connection is slower and less reliable than the generator.

Server-sent events is the usual transport, because it is a plain HTTP response with a simple framing and it reconnects with a Last-Event-ID header. A WebSocket suits bidirectional interaction such as interruption. Either way the generator produces tokens faster than a slow client consumes them, and the buffer between them needs a policy.

Section 5.4's rule applies unchanged: bound the buffer and decide what happens when it fills. For token streaming the right policy is usually to block the reader from the model rather than to drop tokens, because dropped tokens produce a corrupted answer. Blocking then propagates back into the serving scheduler, which frees the batch slot for another request.

Two agent-specific concerns sit on top. A user who cancels must actually stop the generation, or the cost continues after nobody is listening. Wire the HTTP disconnect to a cancellation that reaches the inference server and aborts the sequence, and confirm it works, because a framework that swallows the disconnect is common and the cost is invisible.

And a resumable stream needs the output persisted as it is produced. A user who reconnects after a network drop reads from the store and continues, instead of paying for the whole generation again. That makes the output store part of the streaming path rather than an afterthought.

Observing an agent

The telemetry unit for an agent is the step, and the trace is the run. Every step needs the model and version, the prompt token count, the completion token count, the latency, the cost, the tool name and arguments, the tool result size, and the outcome.

Four failure classes only appear at this granularity. A wrong tool choice looks like a successful run with an unhelpful answer. A retrieval that returned nothing relevant looks the same. A plan that drifted from the objective over several steps is invisible in any single step. And a loop shows as a rising step count rather than as an error.

Cost is a first-class metric here in a way it is not elsewhere, because a single agent run can cost a thousand times a plain request. Track cost per run, per user, and per tenant, and alert on the rate of change rather than the absolute. Enforce a per-run budget in the state machine, rather than discovering the overrun in a monthly bill.

Evaluation closes the loop. Unit tests with a stubbed model cover the state machine. Offline evaluation over a fixed set of cases, scored by a rubric, covers quality and catches regression when a prompt or a model version changes. Online evaluation samples production runs and scores them, which is the only thing that catches a distribution shift in real traffic. All three are needed, and a team that has only the first believes the system works until a user says otherwise.

Sources and further reading

Woosuk Kwon and others, "Efficient Memory Management for Large Language Model Serving with PagedAttention", SOSP 2023. Yaniv Leviathan and others, "Fast Inference from Transformers via Speculative Decoding", ICML 2023. Noam Shazeer, "Fast Transformer Decoding: One Write-Head is All You Need". Mohammad Shoeybi and others, "Megatron-LM". Samyam Rajbhandari and others, "ZeRO", SC 2020, and the PyTorch FSDP documentation. Yu Malkov and Dmitry Yashunin on HNSW. Suhas Jayaram Subramanya and others, "DiskANN", NeurIPS 2019. Herve Jegou and others on product quantization. Gordon Cormack and others on reciprocal rank fusion. The vLLM, SGLang, and TensorRT-LLM documentation. The Temporal documentation on durable execution and workflow determinism. The OWASP Top 10 for Large Language Model Applications, for prompt injection and the agentic threat model.