Chapter 15 · Part IV: AI Infrastructure, Security, and High Availability
Systems Security, Cryptography, and Zero Trust
Security is a property of a whole system and not a feature that a library provides. This chapter covers the primitives and what each one is for, workload identity that replaces long-lived secrets, the post-quantum migration, and the memory safety problem that accounts for most of the vulnerabilities in systems code.
In this chapter
Start with a threat model, not with a control
A control with no threat model is a guess. The exercise that avoids the guess takes an afternoon and has four steps. List the assets, which are the things that have value: customer data, credentials, money movement, availability. Draw the trust boundaries, which are the places where data crosses from one level of trust to another. Enumerate the threats at each boundary. Decide, for each threat, whether to mitigate, transfer, accept, or eliminate.
STRIDE gives the enumeration a checklist. Spoofing attacks identity, and authentication answers it. Tampering attacks integrity, and signatures or message authentication codes answer it. Repudiation attacks accountability, and audit logs answer it. Information disclosure attacks confidentiality, and encryption and access control answer it. Denial of service attacks availability, and rate limits, quotas, and capacity answer it. Elevation of privilege attacks authorization, and least privilege and isolation answer it.
Two questions do most of the work in practice. What happens if this component is fully compromised? And what is the smallest set of permissions that lets it do its job? The answers produce a blast radius and a least-privilege configuration, which together matter more than any individual control.
Primitives, and what each one is for
Cryptography fails through misuse far more often than through broken math. The short table below maps a need to the correct primitive, and the rules after it cover the misuse that actually happens.
Table 15.1: cryptographic primitive by purpose
| Need | Primitive | Current choice | Not this |
|---|---|---|---|
| Content fingerprint | Hash | SHA-256, SHA-3, BLAKE3 | MD5, SHA-1 |
| Password storage | Password hash | Argon2id, scrypt, bcrypt | any plain hash |
| Message integrity, shared key | MAC | HMAC-SHA-256, KMAC | a hash of key plus message |
| Confidentiality plus integrity | AEAD | AES-256-GCM, ChaCha20-Poly1305 | AES-CBC with no MAC |
| Key from a password | KDF | Argon2id, PBKDF2 with high iterations | a single hash |
| Key from a key | KDF | HKDF | using the key directly |
| Proof of origin | Signature | Ed25519, ECDSA P-256, ML-DSA | encryption "in reverse" |
| Key agreement | KEM or DH | X25519, ML-KEM, hybrid of both | RSA key transport |
| Random values | CSPRNG | getrandom, /dev/urandom |
rand(), Math.random |
Five rules cover the misuse that causes real incidents.
Never invent a construction. Use an authenticated encryption mode, which binds confidentiality and integrity together, rather than combining a cipher and a MAC yourself. The order matters, the details matter, and the published attacks on hand-rolled combinations are numerous.
Never reuse a nonce with the same key. AES-GCM loses all confidentiality for the two messages and leaks the authentication key when a nonce repeats. Use a counter, or use a random 96-bit nonce with a hard limit on messages per key, or use a nonce-misuse-resistant mode such as AES-GCM-SIV where the risk is real.
Compare secrets in constant time. A comparison that returns early on the first differing byte leaks the secret one byte at a time to an attacker who measures timing. Every language has a constant-time comparison function, and it costs nothing to use it.
Encrypt with a purpose-derived key. Derive a distinct key per use from one root with HKDF and a context string. A compromise of one use then does not spread, and two uses never produce a nonce collision.
Store passwords with a memory-hard hash. Argon2id with a memory parameter of at least 19 mebibytes and two iterations is the current baseline, tuned upward until a hash takes a few hundred milliseconds on your hardware. Memory hardness is what defeats the graphics processor and the custom hardware attack, which plain iteration counts do not.
Key management
Key management is where most cryptographic designs fail, because the math is in a library and the key handling is in your code.
Envelope encryption is the pattern that scales. Generate a fresh data encryption key per object and encrypt the data with it. Then encrypt that key with a key encryption key. The key encryption key lives in a key management service or in a hardware security module. Store the wrapped key next to the ciphertext. Rotating the key encryption key rewraps the small keys and never touches the data, which turns a rotation from a full re-encryption into a metadata update.
A hardware security module holds keys in tamper-resistant hardware and performs operations without ever exporting the private key. A cloud key management service gives the same property as a service. The value is not that the hardware is unbreakable, it is that a compromised application server can request signatures under audit rather than walk away with the key.
Rotation needs a plan before it needs a schedule. Support two active keys at once, identify which key produced each artifact, decrypt with either, and encrypt with the newer one. Without a key identifier in the ciphertext, rotation requires a flag day and a full re-encryption. With it, rotation is continuous and boring.
The secret in the environment variable is in your logs
Environment variables appear in process listings, in crash dumps, in container inspection output, in continuous integration logs, and in error reports that frameworks generate automatically. They are convenient and they leak. Mount secrets as files with restrictive permissions, or fetch them at start-up from a secret manager with a workload identity, and prefer short-lived credentials over any stored secret. When a secret must exist in the environment, make it short-lived so that a leak expires.
Post-quantum cryptography
A sufficiently large quantum computer breaks RSA and elliptic curve cryptography through Shor's algorithm. No such machine exists. The reason to act now is harvest now and decrypt later: an adversary records encrypted traffic today and decrypts it when the machine arrives. Any data whose confidentiality must outlive the next decade is already at risk.
NIST standardized three algorithms in August 2024. FIPS 203 defines ML-KEM, which is a key encapsulation mechanism based on module lattices and derived from Kyber. FIPS 204 defines ML-DSA, a lattice signature derived from Dilithium. FIPS 205 defines SLH-DSA, a stateless hash-based signature derived from SPHINCS+, which rests on hash security alone and therefore serves as a conservative backup.
Key exchange migrated first because it is cheap and because it is where the harvest risk lives. Hybrid key agreement combines a classical exchange with a post-quantum one and derives the session key from both, so the session stays secure unless both break. X25519MLKEM768 is the deployed combination. Browsers shipped it in 2024, and major content delivery networks now enable it by default. Measurement studies through 2026 put roughly half of surveyed domains on a hybrid exchange.
Signatures are the harder half, for a size reason. An Ed25519 public key is 32 bytes and a signature is 64. An ML-DSA-65 public key is about 1,952 bytes and a signature about 3,309. A certificate chain carries several of each, so a handshake grows by several kilobytes and can exceed the initial congestion window, which adds a round trip. Work on smaller chains and on trust anchor negotiation is active, and the signature migration will run longer than the key exchange migration.
A post-quantum migration plan that is achievable
Build an inventory first: every place the organization uses public-key cryptography, with the algorithm, the key lifetime, and the confidentiality lifetime of the protected data. Enable hybrid key exchange on every TLS endpoint now, because it is a configuration change on modern stacks. Make the code crypto-agile: no hard-coded algorithm identifiers, no fixed-size buffers for keys or signatures, and an algorithm identifier stored with every artifact. Prioritize by confidentiality lifetime, so long-lived archives and root keys come first and short-lived session data comes last. Plan signatures over several years and track the IETF work.
Identity for workloads
A service that authenticates with a shared secret has three problems. The secret is long-lived, it is copied into configuration, and a leak gives indefinite access. Workload identity replaces all three.
SPIFFE defines a naming scheme and a document format. A SPIFFE ID is a URI
such as spiffe://example.com/ns/payments/sa/charge-service. A SPIFFE
Verifiable Identity Document, or SVID, carries that ID in an X.509 certificate
or a JWT. SPIRE is the reference implementation, and both are CNCF projects.
Attestation is the interesting part. A SPIRE agent on each node proves the node's identity to the server through a platform mechanism, such as a cloud instance identity document or a TPM. It then attests each workload locally by inspecting the process: its namespace, its service account, its container image digest, its parent process. Identity therefore derives from what the workload is, not from a secret that it holds.
1. NODE ATTESTATION 2. WORKLOAD ATTESTATION
+----------------+ +--------------------------+
| SPIRE agent | instance identity | agent inspects the caller |
| on each node |--- document, TPM -->| pid -> cgroup -> ns |
+----------------+ or k8s PSAT | -> service account |
| | -> image digest |
| attested node +--------------------------+
v |
+----------------+ v
| SPIRE server | issues an X.509-SVID for
| registry +CA | spiffe://example.com/ns/payments/sa/charge
+----------------+ TTL 1 hour, rotated at ~30 minutes
3. USE: mutual TLS between two workloads
charge-service --- presents SVID ---> ledger-service
<-- presents SVID ----
both validate against the same trust bundle
policy matches on the SPIFFE ID, never on an IP address
No shared secret exists anywhere in this picture. A leaked SVID
expires within the hour with no revocation infrastructure.
Figure 15.1 Workload identity: attestation, issuance, and use.
SVIDs are short-lived. A one-hour certificate with rotation at roughly the half-life is the common configuration. A leaked credential expires within the hour with no revocation infrastructure, which collapses the blast radius from indefinite to bounded. That property is the whole argument for the approach.
Mutual TLS then gives both authentication and encryption in one step. Each side presents its SVID, each validates the other against a common trust bundle, and the authorization policy matches on the SPIFFE ID rather than on an IP address. Chapter 11 showed the mesh that automates this.
Authentication and authorization for people
OAuth 2.0 delegates authorization. OpenID Connect adds authentication on top, returning an ID token that says who the user is. The security best current practice, which OAuth 2.1 consolidates, removed the parts that produced incidents.
The authorization code flow with PKCE is the only flow for new applications. The implicit flow is gone, because it put tokens in URLs where browser history and referrer headers leaked them. The resource owner password flow is gone, because it trains users to type their password into third-party applications. PKCE is required for every client, not only public ones, because it binds the authorization code to the client that requested it and defeats code interception.
Token choice is a real trade. A JWT is self-contained and a resource server validates it with a signature check and no network call, which scales. It also cannot be revoked before expiry, because nothing consults a server. An opaque token is a random string that the resource server introspects against the issuer, which allows immediate revocation and costs a round trip. Use short-lived JWTs, five to fifteen minutes, with refresh tokens that are revocable, and accept that a compromised access token stays valid for its lifetime.
Bearer tokens have one structural weakness: whoever holds the token can use it. Sender-constrained tokens fix it. They bind the token to a key that the client proves it holds, either through mutual TLS client certificates or through DPoP, which is a proof-of-possession header signed per request. A stolen bound token is useless without the private key, and high-value APIs are moving to this.
Authorization models
Role-based access control assigns permissions to roles and roles to users. It is simple and it does not express relationships, so it degrades into role explosion as the product grows: one role per combination of department, region, and resource type.
Attribute-based access control evaluates a policy over attributes of the subject, the resource, the action, and the environment. It expresses rules such as "a manager can approve an expense under 10,000 in their own cost center during business hours". Open Policy Agent and AWS Cedar are the common engines, and they externalize policy from application code.
Relationship-based access control models authorization as a graph. Google Zanzibar introduced the design, and SpiceDB, OpenFGA, and several products implement it. A check is a reachability question: does a path exist from this user to this resource through relations such as owner, editor, member, and parent? It naturally expresses shared documents, nested folders, and group membership, which is exactly where ABAC becomes awkward.
The rule for choosing is short. Use RBAC when permissions follow job function and the resource count is small. Use ABAC when context such as time, location, or data classification enters the decision. Use ReBAC when users share resources with each other and permissions inherit through containment.
Enforce authorization at the resource, not at the gateway alone. A gateway check is a good first filter and it does not know whether this user owns this specific record. Broken object-level authorization, which is a gateway check plus a missing per-record check, remains the most common serious API vulnerability.
Memory safety
Roughly 70 percent of serious vulnerabilities in large C and C++ code bases are memory safety defects. The categories are few and well understood: buffer overflow, use after free, double free, type confusion, uninitialized read, and integer overflow that leads to one of the others.
Hardware and compiler mitigations raise the cost of exploitation without removing the bugs. Address space layout randomization moves code and data to unpredictable addresses. Stack canaries detect a linear stack overflow before a return. Non-executable memory stops shellcode in a data page. Control flow integrity restricts indirect branches to valid targets. Shadow stacks protect return addresses in hardware. Arm Memory Tagging Extension tags allocations and pointers and traps on a mismatch, which catches both spatial and temporal errors at a few percent of overhead. CHERI goes further with hardware capabilities that carry bounds and permissions in the pointer itself.
Every one of these is a mitigation and not a fix. The bug remains, and attackers develop bypasses. The fix is a language where the class does not exist, and that is why the policy landscape changed. CISA and the FBI asked manufacturers serving critical infrastructure to publish a memory safety roadmap by January 1, 2026, and CISA and NSA published joint adoption guidance in June 2025. Rust in the Linux kernel lost its experimental status in December 2025 and now ships in Android drivers, in file systems, and in NVIDIA's Nova GPU driver.
The realistic strategy for existing code is neither a rewrite nor nothing. Write new components in a memory-safe language. Rewrite the parsers and the network-facing code first, because that is where untrusted input arrives. Run sanitizers and fuzzers continuously on what remains, because a fuzzer finds memory bugs faster than any review.
Supply chain
An attacker who compromises a dependency, a build system, or a package registry reaches every downstream consumer at once. Several high-profile incidents made this the fastest-growing attack class.
Four controls, applied together, cover most of it. A software bill of materials lists every component and version, so that a new advisory turns into a query rather than an investigation. Signed provenance, which Sigstore provides with keyless signing and a transparency log, ties an artifact to the source commit and the build that produced it. SLSA defines levels of build integrity, where the higher levels require a hardened, non-falsifiable build service. A reproducible build lets an independent party rebuild the artifact and get identical bytes, which detects a compromised builder.
Dependency hygiene is the unglamorous half. Pin versions by digest rather than by tag. Use a lock file and commit it. Mirror the registry so that a deleted or replaced upstream package does not break or poison a build. Guard against dependency confusion by configuring private scopes so that an internal package name cannot be resolved from a public registry.
Runtime security
The last layer assumes that something got in. Runtime controls reduce what an attacker can do next.
A seccomp profile restricts a container to the system calls it needs, which removes a large part of the kernel attack surface. A mandatory access control policy under AppArmor or SELinux restricts file and network access. An eBPF LSM program enforces custom policy at kernel security hooks. Falco and similar tools watch system call patterns and alert on behavior such as a shell spawning inside a container or an unexpected outbound connection.
Network policy limits lateral movement. Chapter 10 described default-deny policies per namespace. That single control turns one compromised pod from a foothold on the whole cluster into a foothold on one namespace. It is the highest-value control per hour of work in a Kubernetes environment.
Sources and further reading
NIST SP 800-207 for the zero trust architecture definition. FIPS 203, 204, and 205 for the post-quantum standards, and the NIST migration guidance that accompanies them. The OAuth 2.0 Security Best Current Practice and the OAuth 2.1 draft. The SPIFFE and SPIRE documentation. Ruoyu Zhou and others and the Zanzibar paper, "Zanzibar: Google's Consistent, Global Authorization System", USENIX ATC 2019. The CISA and NSA joint guidance on memory-safe languages. The SLSA framework and the Sigstore documentation. The OWASP API Security Top 10 for the authorization failures that appear most often.
What this chapter established
- A control with no threat model is a guess. Two questions do most of the work: what if this component is fully compromised, and what is the smallest permission set that lets it work?
- Cryptography fails through misuse. Use AEAD rather than a hand-built combination, never reuse a nonce with a key, compare secrets in constant time, and derive a distinct key per purpose.
- Envelope encryption turns key rotation from a full re-encryption into a metadata update, provided every ciphertext records which key produced it.
- Secrets in environment variables reach process listings, crash dumps, and continuous integration logs. Prefer short-lived credentials from a workload identity.
- Post-quantum key exchange is deployed now through hybrid X25519MLKEM768. Signatures are larger and will migrate over years.
- SPIFFE identities derive from what a workload is, attested by the platform. One-hour SVIDs collapse a leaked credential's blast radius from indefinite to bounded.
- Use the authorization code flow with PKCE, short-lived JWTs with revocable refresh tokens, and sender-constrained tokens for high-value APIs.
- Choose RBAC for job functions, ABAC for contextual rules, and ReBAC for resources that users share with each other. Enforce at the resource, not at the gateway alone.
- Roughly 70 percent of serious vulnerabilities in large C and C++ code bases are memory safety defects. Mitigations raise the cost and the language change removes the class.
- Supply chain security needs four controls together: a bill of materials, signed provenance, build integrity levels, and reproducible builds.
- A default-deny network policy per namespace is the highest-value runtime control per hour of work in a Kubernetes environment.