Containerization and Cloud-Native Runtime Internals

A container is a process, not a machine

There is no container object in the Linux kernel. A container is a process that runs with a restricted view of the system, and the restriction comes from several independent kernel features that a runtime applies together.

Namespaces restrict what the process sees. Control groups restrict what it consumes. Capabilities, seccomp filters, and a mandatory access control module restrict what it can do. A union file system gives it a root directory assembled from read-only layers. A container runtime is the program that combines all of these and then calls execve.

The consequence matters for security reasoning. A container shares one kernel with the host and with every other container. A kernel vulnerability that a container can reach is a host compromise. A virtual machine has a second boundary, the hypervisor, and that is the difference that the sandboxed runtimes later in this chapter try to recover.

Namespaces

Linux provides eight namespace types. Each one virtualizes one global resource, so that processes in different namespaces see different values.

The mount namespace gives a private mount table, which is what makes a container see its own root file system. The PID namespace renumbers processes, so that the container's first process is PID 1 and it cannot see host processes. The network namespace gives a private set of interfaces, routing tables, and firewall rules. The IPC namespace separates System V objects and POSIX message queues. The UTS namespace gives a private hostname. The cgroup namespace hides the position in the control group tree. The time namespace offsets the monotonic and boot clocks. The user namespace maps user and group identifiers.

The user namespace is the one that changes the security model. It maps identifiers so that root inside the container maps to an unprivileged identifier outside. A process that is UID 0 inside and UID 100000 outside can do everything to its own files and nothing to the host's. Rootless containers build on this: an ordinary user creates a user namespace, becomes root inside it, and creates every other namespace from there, with no set-user-ID binary and no daemon running as root.

# Build a namespace set by hand, to see that nothing magic happens.
unshare --mount --pid --net --uts --ipc --user --map-root-user --fork \
        /bin/sh

# Inside: a private hostname, PID 1, and one loopback interface.
hostname sandbox
echo $$                     # 1
ip link                     # lo only, DOWN

# From the host: the same process, with a different view.
ls -l /proc/<pid>/ns/       # symlinks naming each namespace instance
nsenter -t <pid> -n ip addr # join that network namespace

PID 1 inside the container inherits a responsibility that application authors forget. PID 1 reaps orphaned children and receives signals with default handlers disabled. An application that runs as PID 1 and does not reap accumulates zombie processes. One that does not handle SIGTERM gets killed by SIGKILL after the grace period, instead of shutting down cleanly. Run a small init such as tini, or use the runtime's own init option.

Control groups version 2

Chapter 2 introduced cgroup v2 from the kernel side. The container view adds delegation and the specific controllers that a platform sets.

The cpu controller offers two knobs. cpu.weight sets a proportional share that applies only under contention. cpu.max sets a hard quota as a pair of microseconds allowed per period. The quota is the one that produces the throttling cliff from Chapter 2, and the cliff is the single most common cause of unexplained tail latency in a Kubernetes cluster.

The memory controller offers memory.max as a hard limit, memory.high as a throttling threshold, memory.min as a protected reserve, and memory.low as a best-effort reserve. Kubernetes maps a memory limit onto memory.max and does not set memory.high by default, which means that a container that reaches its limit is killed rather than slowed. Setting memory.high about 10 percent below the limit converts many kills into slowdowns.

The io controller offers io.max for hard throughput limits and io.latency for a target latency that the kernel protects by throttling other groups. io.latency is the better tool for a mixed workload, because it protects the latency-sensitive group without capping it.

Images and layers

An image is an ordered list of layers plus a configuration document. Each layer is a tar archive of file system changes, identified by the digest of its content. The configuration names the entry point, the environment, and the layer order. A manifest ties them together, and a manifest list covers several architectures.

Content addressing gives deduplication and integrity for free. Two images that share a base layer store it once. A digest identifies exactly one byte sequence, so image@sha256:... is reproducible in a way that image:latest is not. Deploy by digest.

OverlayFS assembles the layers at run time. It stacks read-only lower directories under one writable upper directory. A read finds the file in the topmost layer that holds it. A write triggers a copy-up: the whole file copies into the upper layer first, and then the write applies. That behavior has a sharp consequence. Writing one byte to a two gigabyte file inside a container copies two gigabytes. Any file that a container writes repeatedly belongs on a volume, not in the image layers.

  container writable layer (upper)   +-------------------------------+
                                     | /app/cache.db  (copied up)    |
                                     | /tmp/session                  |
                                     +-------------------------------+
  image layer 3  (app code)          | /app/server, /app/cache.db    |
  image layer 2  (dependencies)      | /usr/lib/python3.14/...       |
  image layer 1  (base distro)       | /bin, /lib, /etc              |
                                     +-------------------------------+
                merged view: the topmost copy of each path wins

  Write one byte to /app/cache.db  -> copy the whole file to the upper
  layer, then apply the write. A 2 GiB file costs a 2 GiB copy.
  A whiteout file in the upper layer marks a deletion.

Build order controls the cache. Each instruction produces a layer, and a change invalidates that layer and every layer after it. Put the instructions that change rarely first and the ones that change on every commit last. Copy the dependency manifest and install dependencies before copying the source, so that a source change does not reinstall the dependency tree.

Multi-stage builds separate the build environment from the run environment. The build stage holds compilers and test tools. The final stage copies only the artifact into a minimal base. A distroless base contains the runtime libraries and nothing else, with no shell and no package manager. The result is a smaller image, a smaller attack surface, and a much shorter vulnerability scan report.

Pull time dominates cold start for large images. Measurements across production registries have long put the pull at the large majority of container start time, while the container reads only a small fraction of the image. Lazy pulling fixes it: eStargz, SOCI, and Nydus all make an image mountable so that the container starts immediately and fetches file content on demand. For a multi-gigabyte machine learning image, the difference is minutes against seconds.

Runtimes and the interfaces between them

The stack has three layers with two standard interfaces between them, and the separation is what lets pieces be replaced.

The kubelet talks to a high-level runtime over the Container Runtime Interface, or CRI, which is a gRPC service. containerd and CRI-O both implement it. The high-level runtime manages images, snapshots, and pod lifecycle.

The high-level runtime talks to a low-level runtime over the OCI runtime specification, which is a command line contract plus a JSON configuration file. runc is the reference implementation in Go. crun is a C implementation that starts faster and uses less memory. youki is a Rust implementation. All three read the same config.json and set up the same namespaces, cgroups, and mounts.

A shim process sits between them so that the container survives a restart of the high-level runtime. The shim owns the container's standard streams and its exit status, and the runtime reattaches to it. That is why upgrading containerd does not restart running containers.

Sandboxed runtimes replace the low-level runtime while keeping the same interfaces. gVisor puts a user-space kernel, called the Sentry, between the application and the host kernel, and it intercepts system calls. The host kernel surface shrinks to a small allowed set. The cost is 20 to 50 percent on I/O-heavy workloads, because every system call takes a longer path, and much less on CPU-bound work.

Kata Containers runs each pod in a lightweight virtual machine with its own kernel. Firecracker is a minimal virtual machine monitor written in Rust, and it implements a small device model. It boots a microVM to user space in about 125 milliseconds, with about 5 mebibytes of memory overhead. Both give a hardware isolation boundary. The choice between them is operational: Kata integrates with Kubernetes through the runtime class mechanism, and Firecracker underpins serverless platforms that manage their own scheduling.

Kubernetes runtime behavior

A pod is a group of containers that share a network namespace, an IPC namespace, and optionally a PID namespace. They share an IP address and can reach each other on localhost. A pause container holds the namespaces so that application containers can restart without losing them.

Quality of service class follows from the resource specification, and it decides who dies first under pressure. A pod with equal requests and limits on every container and every resource is Guaranteed. A pod with at least one request is Burstable. A pod with no requests and no limits is BestEffort. The kubelet evicts BestEffort first, then Burstable pods that exceed their requests, and Guaranteed last.

Requests and limits do different jobs and teams routinely confuse them. The request is what the scheduler uses to place the pod, and it reserves capacity. The limit is what the runtime enforces at run time. A CPU request becomes a cgroup weight. A CPU limit becomes a quota, with the throttling cliff. A memory request is scheduling only. A memory limit is a hard kill.

Probes control traffic and restarts, and they are three different things. A liveness probe that fails restarts the container, so it must test only that the process is stuck, never that a dependency is reachable. A readiness probe that fails removes the pod from service endpoints without restarting it, and it is the correct place to report a missing dependency. A startup probe disables the other two until the application finishes booting, which stops a slow-starting application from restarting forever.

The most damaging misconfiguration in Kubernetes is a liveness probe that calls a downstream service. The downstream service has a bad minute, every pod's liveness probe fails, every pod restarts at once, and a recoverable degradation becomes a total outage. Make the liveness probe local and cheap.

Container networking

Every pod gets its own network namespace and its own IP address. A Container Network Interface plugin creates the interface, assigns the address, and sets up routing. The interface is a small contract: an executable that receives JSON on standard input and returns JSON.

The classical datapath uses a virtual ethernet pair. One end sits in the pod namespace and the other in the host namespace, attached to a bridge. Traffic between pods on one node crosses the bridge. Traffic between nodes goes through either a route that the host knows, in a native routing mode, or an encapsulation such as VXLAN or Geneve, in an overlay mode.

Encapsulation costs bytes and CPU. A VXLAN header adds 50 bytes, which drops the usable payload below the interface MTU and causes fragmentation when the MTU is not adjusted. Native routing avoids that and requires the underlying network to carry pod routes, which a cloud provider's route table or BGP provides.

Service load balancing is where the datapath choice shows. kube-proxy in iptables mode writes a chain of rules per service, and packet processing walks the chain. The cost grows linearly with the number of services, and rule programming time grows with it too, so a cluster with thousands of services takes seconds to converge after a change. IPVS mode replaces the chain walk with a hash table and improves the lookup while keeping the update cost.

An eBPF datapath replaces the whole thing. Cilium compiles eBPF programs that perform service lookup in a hash map, which is a constant-time operation independent of the service count. It attaches them at the socket layer for local traffic, and at the driver layer with XDP for external traffic. It also supports direct server return, so responses skip the load balancing node. Reported improvements on large clusters are large, including service latency at the 99th percentile falling from tens of milliseconds to low single digits during deployment churn.

  KUBE-PROXY, iptables mode
    pod --> veth --> bridge --> iptables PREROUTING chain
                                  KUBE-SERVICES     (walk N services)
                                  KUBE-SVC-XXXX     (random pick of M pods)
                                  KUBE-SEP-YYYY     (DNAT to pod IP)
                              --> routing --> encap --> wire
    cost: O(N) rule walk, O(N*M) rules total, slow convergence

  CILIUM, eBPF datapath
    pod --> [ bpf_sock program at connect() time ]
              map lookup: service key -> backend list   (O(1))
              rewrite the destination in the socket, before any packet
          --> veth --> [ bpf at tc/XDP ] --> wire, or direct routing
    cost: O(1) lookup, no NAT on the packet path for local connections,
          optional XDP acceleration and direct server return

Network policy is the other half. By default every pod reaches every pod. A NetworkPolicy object selects pods and allows specific ingress and egress, and the CNI plugin enforces it. Policies are additive and default-deny per direction once any policy selects a pod. Start every namespace with a default-deny policy and add the allowed flows, because the reverse order never finishes.

Storage

The Container Storage Interface, or CSI, does for storage what CNI does for networking. A driver implements a gRPC service with controller and node plugins, and Kubernetes calls it to provision, attach, mount, expand, and snapshot volumes.

Two facts govern the design. A block volume attaches to one node at a time, which means that a pod using one cannot move without a detach and attach cycle, and that cycle takes tens of seconds. A file volume, over NFS or a similar protocol, mounts on many nodes and gives weaker consistency and lower performance.

Stateful workloads use a StatefulSet, which gives each pod a stable identity, a stable hostname, and its own persistent volume claim. The ordering guarantees matter for clustered databases, where a replica must not start before the primary exists.

Sources and further reading

The Linux kernel documentation under Documentation/admin-guide/cgroup-v2.rst and the namespaces(7) and capabilities(7) manual pages. The OCI image and runtime specifications. The Kubernetes documentation on quality of service, eviction, and probes. Harter and others, "Slacker: Fast Distribution with Lazy Docker Containers", FAST 2016, for the pull time result. Agache and others, "Firecracker: Lightweight Virtualization for Serverless Applications", NSDI 2020. The Cilium documentation on the eBPF datapath and kube-proxy replacement.