Low-Level Systems and Computer Architecture

The contract that the hardware offers

An instruction set architecture, or ISA, is the contract between a compiler and a chip. The ISA names the registers, the instructions, the memory ordering rules, and the exception model. Three ISAs carry almost all server work in 2026: x86-64 from Intel and AMD, AArch64 from Arm, and RV64GC from RISC-V. A microarchitecture is one implementation of an ISA. Zen 5, Granite Rapids, and Neoverse V3 implement different ISAs or the same ISA in different ways, and they run the same binary at very different speeds.

The distinction matters because the ISA hides the machine. A load instruction in x86-64 says "read eight bytes from this address". It does not say whether the read costs four cycles or four hundred. The same instruction costs 1 nanosecond when the data sits in the level one cache. It costs 80 nanoseconds when the data sits in main memory. It costs 100 microseconds when the page lives on an NVMe drive. The instruction is identical in all three cases.

Most engineers carry a mental model of the machine that comes from the C abstract machine: memory is a flat array, instructions run one after another, and every access costs the same. Every part of that model is false on real hardware. Memory is a hierarchy with five levels. Instructions run out of order, in parallel, and speculatively. Access cost varies by five orders of magnitude. Performance work is the work of replacing the false model with the true one.

  PROGRAMMER MODEL                 REAL MACHINE
  ----------------                 ------------
                                   4-8 instructions decoded per cycle
  one instruction at a time   -->  500+ instructions in flight
                                   out-of-order, speculative execution

                                   L1 (1 ns)  L2 (4 ns)  L3 (20 ns)
  flat, uniform memory        -->  DRAM local (80 ns)  remote (150 ns)
                                   CXL (370 ns)  NVMe (80,000 ns)

  reads and writes in         -->  store buffers, write combining,
  program order                    coherence protocol, memory barriers

A modern server core decodes four to eight instructions per cycle. It keeps more than five hundred instructions in flight at once. It runs at about 3 to 4 gigahertz under an all-core load. One cycle at 3.5 gigahertz lasts about 0.29 nanoseconds. Every number in this chapter converts between cycles and nanoseconds at roughly that rate.

The memory hierarchy and the cost of distance

The memory hierarchy exists because fast memory is expensive and large memory is slow. Static RAM holds a bit in six transistors and responds in one nanosecond. Dynamic RAM holds a bit in one transistor and one capacitor, and it responds in eighty. A cache is a small pool of fast memory that holds the data the core used most recently.

Table 1.1 gives the cost of one access at each level. The numbers come from a 2026 server class core at about 3.5 gigahertz. Treat the ratios as stable and the absolutes as approximate.

Read the table as a set of ratios. L1 is about 80 times faster than DRAM. DRAM is about 1,000 times faster than an NVMe drive. An NVMe drive is about 500 times faster than a cross-region network round trip. Those ratios have held for twenty years and they will hold for the next ten, because they follow from physics and from cost rather than from a product cycle.

The ratios drive one rule that runs through the whole book. Move work to the data, or move the data close to the work, but do not move data across a level boundary more often than the algorithm requires. Chapter 3 applies the rule to data structures. Chapter 7 applies it to storage engines. Chapter 13 applies it to the key and value cache of a language model.

Cache lines, sets, and ways

A cache does not hold bytes. It holds cache lines. A cache line is 64 bytes on x86-64 and on almost all Arm server cores. Apple silicon uses 128 bytes. When a core reads one byte that is not in the cache, the hardware fetches the whole 64 byte line that contains it. The other 63 bytes arrive free of extra latency, and they stay in the cache until something evicts them.

That single fact explains most of the performance difference between a good data layout and a bad one. If your next access falls inside the line that you just loaded, it costs about 1 nanosecond. If it falls outside, it costs about 80. The difference is a factor of eighty, and no compiler flag recovers it.

A cache is set associative. The hardware splits the physical address into three fields: an offset inside the line, an index that selects a set, and a tag that identifies the line inside the set. Consider a 48 KiB, 12 way L1 data cache with 64 byte lines. It holds 768 lines in 64 sets of 12 ways each. Bits 0 to 5 give the offset. Bits 6 to 11 give the index. The remaining bits give the tag.

   63                        12 11        6 5         0
  +----------------------------+------------+-----------+
  |            TAG             |   INDEX    |  OFFSET   |
  +----------------------------+------------+-----------+
               52 bits            6 bits       6 bits
                                  |
                                  v
                          set 0 .. set 63
                          each set holds 12 lines (ways)

  Two addresses that share INDEX bits compete for the same 12 ways.
  Thirteen hot addresses with the same INDEX cause conflict misses,
  even when the cache is almost empty.

Conflict misses follow directly from this layout. Allocate an array of structures whose stride is exactly 4096 bytes, then walk one field of each structure. Every access lands in the same set. The cache holds 12 of them and evicts the rest, and a 48 KiB cache behaves like a 768 byte cache. The classical fix adds a small pad to the stride so that consecutive elements spread across sets.

Caches classify misses into four kinds. A compulsory miss happens on the first touch of a line. A capacity miss happens because the working set exceeds the cache. A conflict miss happens because of the index collision above. A coherence miss happens because another core wrote the line. The first three respond to data layout. The fourth responds to thread placement, and the next section covers it.

Caches on x86-64 keep coherence with a protocol in the MESI family. Every line in every cache carries a state: Modified, Exclusive, Shared, or Invalid. AMD adds Owned and Intel adds Forward. The rule that matters for software is simple. A core that writes a line must hold it in Modified state, and it must first invalidate every other copy in every other cache. That invalidation travels over the interconnect and costs tens to hundreds of cycles.

Alignment, padding, and false sharing

Natural alignment means that an object of size N sits at an address that divides by N. An aligned 8 byte load touches one cache line. An 8 byte load at an address that ends in 0x3D touches two lines, and the hardware splits it into two accesses. On modern x86-64 the penalty for a split load is small, about one extra cycle, unless the load also crosses a 4 KiB page boundary. A page crossing load costs tens of cycles because it needs a second address translation.

False sharing is the expensive case, and it appears in almost every multi-threaded program that a team writes without thinking about layout. Two threads write two different variables. The variables sit in the same 64 byte line. The line now ping-pongs between the two cores. Every write invalidates the other core's copy, and every subsequent read pulls the line back. Neither thread shares data with the other, and yet both run at the speed of the interconnect rather than the speed of L1.

/* Bad: both counters live in one cache line. */
struct counters_bad {
    uint64_t requests;   /* thread A writes this */
    uint64_t errors;     /* thread B writes this */
};

/* Good: each counter owns its own line. */
struct counters_good {
    alignas(64) uint64_t requests;
    alignas(64) uint64_t errors;
};

The measured cost of the bad version runs between five and fifty times the cost of the good version. The exact factor depends on the write rate and on whether the two threads share an L3 cache or sit on different sockets. Two threads on different sockets pay the worst price, because the invalidation crosses the socket interconnect.

Padding to 128 bytes instead of 64 helps on Intel cores that prefetch adjacent line pairs. The adjacent line prefetcher pulls the neighbor of every line that it fetches. A 64 byte pad therefore still leaves two hot variables in one 128 byte prefetch unit. Java recognizes the problem in the standard library and pads with 128 bytes under @Contended. Rust programs use #[repr(align(128))] for the same reason. The cost of the extra padding is memory, and memory is the cheap resource here.

Access patterns and the prefetchers

Hardware prefetchers are small predictors that fetch a line before the program asks for it. A next-line prefetcher fetches line N+1 after a read of line N. A stride prefetcher detects a constant stride and continues it. A streaming prefetcher tracks several independent streams at once. Together they hide most of the DRAM latency for sequential and strided access.

Prefetchers have one hard limit that shapes data structure design. They do not cross a 4 KiB page boundary, because a page boundary can lead to a different physical page and a speculative fetch across it can be wrong. A sequential walk through a large array therefore takes a full latency hit once per page, which is once every 64 lines. Huge pages remove that stall, and the section on virtual memory returns to it.

Three access patterns matter in practice. Sequential access costs almost nothing beyond the bandwidth. Strided access costs almost nothing while the stride stays under a page and the prefetcher tracks it. Random access costs the full latency on every touch, because nothing can predict it.

The last row of Table 1.2 carries a lesson that surprises most engineers. Pointer chasing is slow because each load depends on the previous one. The core cannot start the second load until the first returns. Run eight independent chains in one loop and the core issues eight loads at once. The total time falls by a factor of seven, and no cache behavior changed. The term for this effect is memory level parallelism. A core supports between ten and sixteen outstanding line fills at a time, and a well written loop uses all of them.

Data layout controls how many of the 64 bytes in each line the program actually uses. An array of structures, or AoS, stores each record in one block. A structure of arrays, or SoA, stores each field in its own array.

/* AoS: one particle per record. Touching x loads y, z, and mass too. */
struct particle { float x, y, z, vx, vy, vz, mass, pad; };
struct particle parts[N];               /* 32 bytes per record */

/* SoA: one array per field. Touching x loads only x values. */
struct particles {
    float x[N], y[N], z[N];
    float vx[N], vy[N], vz[N];
    float mass[N];
};

A loop that reads only x and vx touches 8 of the 32 bytes per record in the AoS layout, which wastes 75 percent of every line. The SoA layout packs 16 x values into one line and wastes nothing. The SoA layout also vectorizes, because 16 consecutive x values fit one 512 bit register. The AoS layout needs a gather, and a gather costs about ten times a contiguous load.

The trade-off is real and it runs the other way for some workloads. A loop that reads every field of one record prefers AoS, because AoS puts the whole record in one or two lines. SoA scatters that record across seven arrays and seven lines. Choose the layout that matches the dominant access pattern, and measure before you rewrite.

Virtual memory in the hardware

Every address that a program uses is virtual. The memory management unit, or MMU, translates a virtual address into a physical one on every access. On x86-64 with four level paging, the translation walks four tables, and each table read is itself a memory access. A translation that misses every cache therefore costs four DRAM accesses, or about 320 nanoseconds, before the real load even starts.

The translation lookaside buffer, or TLB, caches recent translations. A typical 2026 core holds about 64 entries in the first level data TLB and about 2,048 to 4,096 entries in the shared second level TLB. Page walk caches hold the upper levels of the page table and cut a full walk to one or two accesses in the common case.

TLB reach is the amount of memory that the TLB can address without a walk. With 4 KiB pages and 2,048 second level entries, the reach is 8 MiB. Any working set larger than 8 MiB takes TLB misses on a random access pattern, whatever the cache does. That is a small number by 2026 standards, and it is the reason that huge pages matter.

  Virtual address (48-bit canonical)
  +--------+--------+--------+--------+-------------+
  | PML4   | PDPT   |  PD    |  PT    |   offset    |   4 KiB page
  | 9 bits | 9 bits | 9 bits | 9 bits |   12 bits   |   4 memory reads
  +--------+--------+--------+--------+-------------+

  +--------+--------+--------+----------------------+
  | PML4   | PDPT   |  PD    |       offset         |   2 MiB page
  | 9 bits | 9 bits | 9 bits |       21 bits        |   3 memory reads
  +--------+--------+--------+----------------------+

  +--------+--------+-------------------------------+
  | PML4   | PDPT   |            offset             |   1 GiB page
  | 9 bits | 9 bits |            30 bits            |   2 memory reads
  +--------+--------+-------------------------------+

  TLB reach with 2048 second-level entries:
    4 KiB pages ->    8 MiB      2 MiB pages ->   4 GiB
    1 GiB pages -> 2048 GiB

A 2 MiB huge page raises the reach from 8 MiB to 4 GiB, a factor of 512. It also removes one level from the walk. Databases, language runtimes with large heaps, and machine learning frameworks all gain from huge pages, and the gain runs between 5 and 30 percent on memory bound workloads.

Linux offers huge pages in two forms. Explicit huge pages come from a preallocated pool and from mmap with MAP_HUGETLB. Transparent huge pages, or THP, promote ordinary 4 KiB pages into 2 MiB pages in the background. THP is easier to adopt and it carries a tail latency risk, because the kernel can stall an allocating thread while it compacts memory to build a free 2 MiB block. Many latency sensitive services set THP to madvise so that only code that opts in receives huge pages.

NUMA and the price of a remote line

A non-uniform memory access system, or NUMA system, attaches memory to sockets rather than to the machine. Each socket carries its own memory controllers and its own DRAM channels. A core reads its own socket's memory across a short path. It reads the other socket's memory across the interconnect, which adds latency and cuts bandwidth.

A two socket 2026 server shows about 80 nanoseconds for a local DRAM read and about 140 nanoseconds for a remote one. Remote bandwidth runs at 60 to 70 percent of local bandwidth, and it saturates sooner because the interconnect carries coherence traffic as well as data. The ratio grows on four socket and eight socket machines.

Large single socket processors now split themselves into NUMA nodes as well. Intel calls the feature sub-NUMA clustering. AMD calls it nodes per socket. Both divide one physical socket into two or four logical nodes so that each group of cores uses the nearest memory controller. The effect is smaller than a socket crossing, about 10 to 20 nanoseconds, and it is large enough to matter for a database.

Linux assigns physical pages with a first touch policy. The page goes to the node of the thread that writes it first, not to the node of the thread that allocated it. A common failure follows: one initialization thread touches the whole heap, every page lands on node 0, and all worker threads on node 1 then read remotely for the life of the process.

# Inspect the topology and the per-node free memory.
numactl --hardware
lscpu | grep -i numa

# Bind a process to node 1 for both CPU and memory.
numactl --cpunodebind=1 --membind=1 ./my-service

# Interleave pages across all nodes for a bandwidth-bound workload.
numactl --interleave=all ./analytics-job

# Per-node statistics. numa_miss counts allocations that missed the
# preferred node.
numastat -c

Two placement strategies cover most cases. A latency sensitive service binds each worker thread and its memory to one node, and it runs one process per node behind a load balancer. A bandwidth bound batch job interleaves pages across all nodes so that it uses every memory controller at once. The wrong choice costs between 20 and 50 percent of throughput.

Instruction level parallelism and the cost of a wrong guess

A modern core is a scheduler, not a sequencer. It fetches and decodes several instructions per cycle, renames their registers to remove false dependencies, issues them out of order to eight or more execution ports, and retires them in program order. The structure that holds instructions in flight is the reorder buffer, and it holds 500 to 600 entries on a 2026 core.

Out-of-order execution hides latency. A load that misses L1 costs 14 cycles, and the core keeps running the instructions that do not depend on it. The technique works only when independent work exists. A chain of dependent operations defeats it completely, because every instruction waits for the previous one.

Two numbers describe every instruction: latency and throughput. Latency is the cycles from issue to result. Throughput is the cycles between two independent issues of the same instruction. A 64 bit integer add has latency 1 and throughput 0.25, because four ports can run it. A double precision divide has latency 13 to 15 and throughput 4. A floating point add has latency 3 and throughput 0.5.

/* Latency bound: each add waits 3 cycles for the previous one.
   4 elements per 12 cycles. */
float sum_slow(const float *a, size_t n) {
    float s = 0.0f;
    for (size_t i = 0; i < n; i++) s += a[i];
    return s;
}

/* Throughput bound: four independent chains fill the pipeline.
   4 elements per 3 cycles, about 4x faster. */
float sum_fast(const float *a, size_t n) {
    float s0 = 0, s1 = 0, s2 = 0, s3 = 0;
    size_t i = 0;
    for (; i + 4 <= n; i += 4) {
        s0 += a[i];     s1 += a[i + 1];
        s2 += a[i + 2]; s3 += a[i + 3];
    }
    float s = (s0 + s1) + (s2 + s3);
    for (; i < n; i++) s += a[i];
    return s;
}

A C compiler will not perform that transformation on its own, because floating point addition is not associative and the transformation changes the result. Pass -ffast-math and the compiler does it. Write the accumulators by hand and the compiler does not have to.

Branch prediction feeds the front end. A modern predictor uses a TAGE design, which indexes several tables with different lengths of branch history and picks the longest match. Accuracy on ordinary code runs above 99 percent. A misprediction costs 15 to 20 cycles, because the core discards every speculative instruction and refills the pipeline.

The practical consequence appears in sorting and in filtering. A branch that depends on unsorted data mispredicts about half the time. The same branch on sorted data predicts almost perfectly. A filter loop over sorted data can run five times faster than the same loop over shuffled data, with identical instruction counts. A branchless rewrite that uses a conditional move removes the difference and pays a small fixed cost instead.

SIMD: one instruction, many lanes

Single instruction multiple data, or SIMD, applies one operation to several values at once. A 256 bit AVX2 register holds eight 32 bit floats. A 512 bit AVX-512 register holds sixteen. One vaddps instruction adds sixteen pairs in the latency of one add.

x86-64 carries four generations. SSE2 gives 128 bits and every x86-64 chip supports it. AVX and AVX2 give 256 bits and cover almost every server since 2015. AVX-512 gives 512 bits, a set of mask registers, and useful instructions for gather, scatter, and conflict detection. AVX10 unifies the AVX-512 instruction set across performance cores and efficiency cores, which removes the reason that Intel disabled AVX-512 on hybrid consumer parts. Intel states that the Nova Lake generation supports AVX10.2 and the Advanced Performance Extensions, or APX. APX doubles the general purpose registers from 16 to 32 and cuts compiled loads by about 10 percent and stores by about 20 percent.

Arm takes a different path. NEON gives a fixed 128 bit width and every AArch64 core supports it. The Scalable Vector Extension, or SVE, and SVE2 hide the width from the program. A binary compiled for SVE runs on a 128 bit implementation and on a 2048 bit implementation without a recompile, because the loop asks the hardware how many lanes it has. Neoverse V3 implements Armv9.2 with two SVE2 units per core.

#include <immintrin.h>

/* Sum n floats with AVX2. Handles the tail with a scalar loop. */
float sum_avx2(const float *a, size_t n) {
    __m256 acc0 = _mm256_setzero_ps();
    __m256 acc1 = _mm256_setzero_ps();
    size_t i = 0;
    for (; i + 16 <= n; i += 16) {
        acc0 = _mm256_add_ps(acc0, _mm256_loadu_ps(a + i));
        acc1 = _mm256_add_ps(acc1, _mm256_loadu_ps(a + i + 8));
    }
    __m256 acc = _mm256_add_ps(acc0, acc1);
    __attribute__((aligned(32))) float lanes[8];
    _mm256_store_ps(lanes, acc);
    float s = 0.0f;
    for (int k = 0; k < 8; k++) s += lanes[k];
    for (; i < n; i++) s += a[i];
    return s;
}

Automatic vectorization works when the compiler can prove three facts. The trip count must be computable before the loop starts. The pointers must not overlap, which restrict asserts in C. The loop body must have no early exit and no call that the compiler cannot inline. Pass -O3 -march=native -fopt-info-vec-missed to GCC or -Rpass-analysis=loop-vectorize to Clang and the compiler reports the exact reason for each failure.

Three cautions apply. First, AVX-512 lowered the clock frequency on Skylake-SP and Cascade Lake by up to 25 percent under heavy use, which erased the gain for mixed workloads. Ice Lake and later cut that penalty sharply, and Zen 4 and Zen 5 implement AVX-512 with little or no frequency drop. Second, gather and scatter instructions cost about ten times a contiguous load, so SIMD rewards SoA layouts and punishes AoS layouts. Third, a vectorized loop that misses the cache gains nothing, because the bottleneck moved to memory.

Matrix work now has its own units. Intel Advanced Matrix Extensions, or AMX, add 8 KiB tile registers and a tile multiply instruction. Arm Scalable Matrix Extension, or SME, does the same for AArch64. Both target the same operation that dominates machine learning inference, and Chapter 13 measures what they deliver next to a GPU.

Measuring instead of guessing

Hardware performance counters give the ground truth. On Linux, perf reads them. Start with a small set that separates the four possible bottlenecks.

# Baseline: how many instructions per cycle, and where the stalls come from.
perf stat -e cycles,instructions,branches,branch-misses,\
cache-references,cache-misses,dTLB-load-misses ./workload

# Top-down analysis: the four top-level buckets.
perf stat --topdown -a --per-core sleep 10

# Which line of which function takes the cache misses.
perf record -e cache-misses:pp -g ./workload && perf report --stdio

# Which cache line two cores fight over.
perf c2c record ./workload && perf c2c report --stdio

Instructions per cycle, or IPC, is the first number to read. A value above 2.0 means that the core runs well and that any further gain must come from doing less work. A value below 0.5 means that the core stalls, and the counters say why. Top-down analysis sorts every cycle into four buckets: retiring, bad speculation, front end bound, and back end bound. Back end bound splits again into core bound and memory bound.

The roofline model turns the counters into a target. Compute the arithmetic intensity of a kernel, which is floating point operations divided by bytes moved from DRAM. Plot it against the machine peak. A kernel below the diagonal is memory bound and needs a layout change. A kernel below the horizontal line is compute bound and needs vectorization or a better algorithm. The model takes ten minutes and it prevents weeks of work on the wrong bottleneck.

Sources and further reading

Ulrich Drepper, "What Every Programmer Should Know About Memory", 2007, still the best single treatment of cache mechanics. John Hennessy and David Patterson, "Computer Architecture: A Quantitative Approach", sixth edition, for the microarchitecture. The Intel 64 and IA-32 Architectures Optimization Reference Manual and the Arm Neoverse V3 Software Optimization Guide for instruction latencies. Agner Fog's instruction tables for measured latency and throughput per microarchitecture. The Intel Architecture Instruction Set Extensions manual for AVX10 and APX. Huaicheng Li and others, "Pond: CXL-Based Memory Pooling Systems for Cloud Platforms", ASPLOS 2023, for disaggregated memory numbers.