Container Engines and Orchestration Internals

7.1 Container Isolation Primitives

There is no container object in the Linux kernel. A container is a process that runs with a restricted view of the system, and the restriction comes from several independent kernel features that a runtime applies together.

Namespaces restrict what the process sees. Control groups restrict what it consumes. Capabilities, seccomp filters, and a mandatory access control module restrict what it can do. A union file system gives it a root directory assembled from read-only layers. A container runtime combines all of these and then calls execve.

Namespaces and the order of operations

Linux provides eight namespace types, and a runtime creates them with clone or unshare using one flag per type: CLONE_NEWNS for mounts, CLONE_NEWPID for processes, CLONE_NEWNET for the network, CLONE_NEWIPC, CLONE_NEWUTS, CLONE_NEWUSER, CLONE_NEWCGROUP, and CLONE_NEWTIME.

The order matters. The user namespace must be created first when running rootless, because creating the others requires privilege, and inside a new user namespace an unprivileged process holds a full capability set over that namespace. That single property is what makes rootless containers possible: an ordinary user creates a user namespace, becomes root inside it, and creates every other namespace from there, with no set-user-ID binary and no daemon running as root.

The PID namespace has a detail that surprises implementers. Creating it does not move the calling process into it. The first process in the new namespace is the child that clone creates, and that child becomes PID 1. A process that calls unshare(CLONE_NEWPID) and then execve without forking finds itself still in the old namespace.

PID 1 inherits responsibilities that application authors forget. It reaps orphaned children, and it receives signals with default handlers disabled. An application that runs as PID 1 and does not reap accumulates zombie processes. One that does not handle SIGTERM gets killed by SIGKILL after the grace period instead of shutting down cleanly. Run a small init such as tini, or use the runtime's init option.

Why pivot_root rather than chroot

chroot changes the apparent root directory and leaves the old root reachable. A process with CAP_SYS_CHROOT escapes it in a few lines: create a directory, chroot into it, and then walk up with .. from a file descriptor opened before the call. chroot was never a security boundary and its manual page says so.

pivot_root moves the root mount itself. The runtime mounts the new root, calls pivot_root(new_root, put_old) which makes the new root current and relocates the old one, then unmounts the old root and removes the mount point. After that the old root has no mount entry and no path, so there is nothing to walk back to.

/* The sequence a runtime performs inside a new mount namespace. */
/* 1. Make the whole tree private, or mount changes propagate to the host. */
mount(NULL, "/", NULL, MS_REC | MS_PRIVATE, NULL);

/* 2. Bind the new root onto itself, because pivot_root requires that
      new_root is a mount point and not the same mount as the current root. */
mount(new_root, new_root, NULL, MS_BIND | MS_REC, NULL);

/* 3. Prepare a place to park the old root inside the new one. */
mkdir(put_old, 0700);

/* 4. Swap them. */
syscall(SYS_pivot_root, new_root, put_old);
chdir("/");

/* 5. Detach the old root. MNT_DETACH unmounts lazily, so busy mounts
      below it go away once their last reference closes. */
umount2("/.pivot_old", MNT_DETACH);
rmdir("/.pivot_old");

Step 1 is the one that gets omitted and that causes the subtle failure. Mount namespaces share propagation by default, so a mount inside the container appears on the host and a host unmount removes a container mount. Marking the tree private stops the propagation in both directions.

Control groups version 2

The unified hierarchy replaced the many separate hierarchies of version 1. Every controller attaches to one tree, so a cgroup has one path and one set of limits rather than one path per resource.

Four controllers carry most of the weight. The cpu controller sets cpu.max as a quota and period pair and cpu.weight as a proportional share. The memory controller sets memory.max as a hard limit, memory.high as a throttling threshold, and memory.min as a protected reserve. The io controller sets io.max for hard limits and io.latency for a target that the kernel protects by throttling other groups. The pids controller caps the process count.

Two rules govern the hierarchy and both cause confusion on first contact. The no-internal-process rule says that a cgroup with children cannot hold processes directly, except for the root, so a tree has processes only in its leaves. And a controller must be enabled in a parent's cgroup.subtree_control before a child can use it, which is what makes delegation explicit.

The memory controller's two limits do different jobs, and the difference matters more than any other cgroup detail. A cgroup that crosses memory.high is throttled and reclaimed, and it keeps running. A cgroup that crosses memory.max triggers the out-of-memory killer inside that cgroup. Kubernetes maps a memory limit onto memory.max and does not set memory.high by default, which means a container that reaches its limit is killed rather than slowed.

When the kernel kills, it picks a process by an out-of-memory score that favors large resident sets. In a pod with several containers, or a container that forks workers, the victim is often not the main process, so the pod keeps running in a broken state with no restart. Set memory.oom.group so that the whole cgroup dies together, and read memory.events for oom_kill counts rather than relying on an exit code.

Pressure Stall Information is the signal to alert on. /proc/pressure/memory, /proc/pressure/io, and /proc/pressure/cpu, and their per-cgroup equivalents, report the percentage of time that at least one task waited and the percentage of time that every task waited. Free memory is useless on Linux, because the kernel uses all of it for the page cache. Pressure is not.

Seccomp, capabilities, and mandatory access control

A seccomp filter is a BPF program evaluated on every system call. It receives the call number and the architecture and returns an action: allow, error with a chosen errno, kill the thread, kill the process, trap, log, or notify a supervisor.

The architecture check is not optional and is the classic bug. On x86-64 a process can issue 32-bit system calls, where the numbers are completely different, so a filter that checks numbers without first checking arch blocks nothing on that path. Every correct filter starts by comparing the architecture and killing on a mismatch.

Default profiles block around 40 to 60 of roughly 350 system calls, including mount, kexec_load, ptrace, bpf, and the keyring calls. That removes a large part of the kernel attack surface for a cost of 1 to 5 percent.

Capabilities split root's power into about 40 units. The ones that matter for a container are CAP_SYS_ADMIN, which is so broad that granting it is close to granting root, CAP_NET_ADMIN, CAP_SYS_PTRACE, CAP_DAC_OVERRIDE, and CAP_NET_RAW. A container runtime drops all but a small default set. Dropping everything and adding back only what is needed is the correct posture, and CAP_NET_BIND_SERVICE for a port below 1024 is usually the only addition a normal service needs.

no_new_privs closes the remaining path. With it set, execve cannot grant privileges, so a set-user-ID binary inside the container gains nothing. The flag is inherited and cannot be unset, which makes it safe to apply early.

7.2 Kubernetes Control Plane Mechanics

Kubernetes is a set of control loops over a consistent store. Every component reads desired state, observes actual state, and acts to close the gap. Nothing issues commands to anything else.

etcd and the watch mechanism

etcd is a Raft-replicated key-value store with multi-version concurrency control. Every write increments a cluster-wide revision, and every key version is retained until compaction. A read at a revision returns exactly the state at that point.

Watches are what make the control loops work. A client watches a key range from a revision and receives every change after it, in order, with no polling. A client that disconnects reconnects from its last revision and receives the gap, provided compaction has not passed it.

That last clause is the failure mode that produces a cluster-wide incident. If compaction removes revisions that a watcher still needs, the watch is cancelled with a compaction error, and every informer must resynchronize with a full list. A full list from every controller at once is the load spike that takes a large cluster down.

etcd requires a fast disk more than it requires CPU. Raft commits an entry only after a majority fsyncs it, so the disk's fsync latency directly sets the write latency. A slow disk produces leader elections, because heartbeats compete with commits. Put etcd on local NVMe, not on network block storage, and alert on etcd_disk_wal_fsync_duration_seconds at the 99th percentile.

Defragmentation is the operational task that teams discover during an incident. etcd's storage does not return freed space to the file system automatically. A cluster that churns objects therefore grows toward the size limit even when the live data set is small. Defragmentation reclaims it and blocks that member while it runs, so it must be done one member at a time, starting with the followers.

The API server and the request path

The API server is the only component that talks to etcd. Everything else talks to the API server, which gives one place for authentication, authorization, admission, and validation.

A write request passes through a fixed pipeline. Authentication identifies the caller from a certificate, a bearer token, or a webhook. Authorization asks whether that caller can perform that verb on that resource, usually through role-based access control. Mutating admission webhooks and controllers can change the object. Schema validation applies. Validating admission webhooks and policies can reject it. Then the object is written to etcd.

  client
    |  TLS, client cert / bearer token / OIDC
    v
  [ authentication ]      -> who is this?
    v
  [ authorization ]       -> RBAC: can they do this verb on this resource?
    v
  [ mutating admission ]  -> webhooks, then built-in plugins
    |                        injects sidecars, defaults, labels
    v
  [ object schema validation ]
    v
  [ validating admission ] -> webhooks and ValidatingAdmissionPolicy (CEL)
    |                         rejects what violates policy
    v
  [ etcd write, revision N+1 ]
    |
    +--> every watcher on that resource receives the event

  A mutating webhook is on the critical path of every matching write.
  failurePolicy: Fail plus an unavailable webhook blocks all such writes,
  which is how an admission controller takes a cluster down.

Admission webhooks are the most common self-inflicted control plane outage. A webhook with failurePolicy: Fail that becomes unavailable blocks every write it matches. If its own pods cannot be scheduled because the webhook blocks pod creation, the cluster cannot recover on its own. Three controls prevent it: scope the webhook with namespaceSelector and objectSelector so that it never matches control plane namespaces, set a short timeoutSeconds, and prefer ValidatingAdmissionPolicy with CEL expressions, which runs in-process and cannot be unavailable.

The API server protects itself with API Priority and Fairness, which replaced the older global inflight limits. Requests are classified into flow schemas and assigned to priority levels with concurrency shares, so a misbehaving controller cannot starve the scheduler or the kubelet. Watch apiserver_flowcontrol_rejected_requests_total to see which client is being throttled and why.

Controllers and the informer pattern

A controller watches resources and reconciles. Reading directly from the API server per event overwhelms it, so every controller uses an informer.

An informer performs one initial list, then watches from that revision, and maintains a local indexed cache. Reads come from the cache and never touch the API server. Changes push keys into a rate-limited work queue, and worker goroutines pop keys and reconcile.

The queue is where the important properties live. It deduplicates, so ten events for one object produce one reconcile. It rate limits with exponential backoff per key, so a persistently failing object does not spin. And it guarantees that one key is processed by one worker at a time, which removes the need for locking inside the reconciler.

// A reconciler is level-driven: it reads the world and acts, and it
// never trusts the event that woke it.
func (c *Controller) reconcile(key string) error {
    ns, name, _ := cache.SplitMetaNamespaceKey(key)

    desired, err := c.lister.Widgets(ns).Get(name)     // from the cache
    if apierrors.IsNotFound(err) {
        return c.cleanup(ns, name)                     // deleted: tidy up
    } else if err != nil {
        return err                                     // requeue with backoff
    }

    actual, err := c.findActual(desired)
    if err != nil {
        return err
    }

    if !equality.Semantic.DeepEqual(desired.Spec, actual.Spec) {
        if err := c.converge(desired, actual); err != nil {
            return err
        }
    }
    // Status is a separate write, and it must be idempotent, because
    // this function will run again for reasons that have nothing to do
    // with this object changing.
    return c.updateStatus(desired, actual)
}

Level-driven reconciliation is the design principle, and it is what makes the system tolerant of disorder. The reconciler reads the current state and acts, rather than processing a delta. A missed event, a duplicate event, or an out-of-order event all resolve on the next pass, and a periodic resync guarantees that a pass happens even if no event arrives.

Custom resource definitions extend the API with new types that get the same storage, validation, watch, and role-based access control machinery. An operator is a controller for a custom resource that encodes operational knowledge: how to take a backup, how to run a rolling upgrade, how to fail over. The pattern works because the extension mechanism is the same one the built-in controllers use.

The scheduler

The scheduler assigns pods to nodes in two phases. Filtering removes nodes that cannot run the pod, considering resource requests, node selectors, taints and tolerations, volume topology, and pod affinity rules. Scoring ranks the survivors, considering resource balance, image locality, spreading, and affinity preferences. The highest score wins, with ties broken at random.

The scheduler framework exposes the phases as plugin extension points: PreFilter, Filter, PostFilter, PreScore, Score, NormalizeScore, Reserve, Permit, PreBind, Bind, and PostBind. A custom scheduling policy is a plugin at the right point rather than a fork, and several schedulers can run in one cluster with pods selecting one by name.

Two behaviors matter operationally. The scheduler stops scoring once it evaluates a set percentage of nodes, and that percentage falls as the cluster grows. The effect trades placement quality for latency on large clusters. And preemption evicts lower-priority pods to make room for a higher-priority one. It works, and it needs pod disruption budgets to stop it from taking an entire service down at once.

Scheduling operates on requests and never on actual use. A node whose pods request 4 CPUs and use 0.2 is full from the scheduler's point of view. That gap between requested and used capacity is the largest source of waste in most clusters, and closing it needs request values derived from measured use rather than from a first guess that nobody revisited.

7.3 Container Network Interface Internals

Every pod gets its own network namespace and its own address, and it must reach every other pod without network address translation. A CNI plugin makes that true.

The plugin contract

CNI is a small contract: an executable that reads JSON on standard input and writes JSON on standard output. The runtime calls it with a command in the environment, ADD to attach a pod, DEL to detach, CHECK to confirm, and VERSION.

On ADD the plugin receives the container identifier, the network namespace path, and the network configuration. It creates the interface, moves one end into the namespace, assigns an address from the configured address management plugin, sets routes, and returns the result. Plugins chain: one handles the interface, another handles addressing, a third adds port mappings or bandwidth limits.

The classical datapath uses a virtual ethernet pair. One end sits in the pod namespace as eth0 and the other in the host namespace attached to a bridge. Traffic between pods on one node crosses the bridge. Traffic between nodes goes through either a route the host knows or an encapsulation.

Overlay against native routing

An overlay encapsulates a pod packet inside a host packet. VXLAN wraps the original ethernet frame in UDP with an 8-byte VXLAN header carrying a 24-bit network identifier, and the tunnel endpoint on each node decapsulates. Geneve does the same with extensible options.

The overlay works on any underlying network, because the underlay only sees ordinary host-to-host UDP. The costs are 50 bytes of header per packet, an encapsulation and decapsulation step, and a hard-to-diagnose failure when the MTU is not adjusted.

Native routing puts pod addresses in the underlying network's routing table. BGP is the usual mechanism: each node peers with the fabric and advertises its pod address range. There is no encapsulation, no overhead, and no MTU problem, and it requires the network to accept those routes, which a cloud route table or a datacenter fabric provides and a flat corporate network often does not.

  VXLAN OVERLAY
   pod A 10.244.1.5 -> pod B 10.244.2.9
   [eth 14][ip 20: 10.244.1.5 -> 10.244.2.9][tcp 20][payload]
      wrapped by the VTEP on node 1:
   [eth 14][ip 20: 192.168.0.11 -> 192.168.0.12][udp 8][vxlan 8][original]
      50 bytes of overhead. Pod MTU must be 1450 if the host MTU is 1500.

  NATIVE ROUTING WITH BGP
   node 1 advertises 10.244.1.0/24, node 2 advertises 10.244.2.0/24
   [eth 14][ip 20: 10.244.1.5 -> 10.244.2.9][tcp 20][payload]
      the fabric routes it. No overhead, full MTU, and every hop can
      see the real addresses, which makes a packet capture readable.

The eBPF datapath

Service load balancing is where the datapath choice shows most clearly. kube-proxy in iptables mode writes a chain of rules per service, and packet processing walks the chain. The cost grows linearly with the service count, and the time to program the rules grows with it too, so a cluster with thousands of services takes seconds to converge after a change. IPVS mode replaces the chain walk with a hash table and improves the lookup while keeping much of the update cost.

An eBPF datapath replaces the mechanism. Cilium compiles eBPF programs that perform service lookup in a hash map, which is constant time in the service count, and attaches them at several layers. At the socket layer, a program rewrites the destination at connect time, so a local connection never produces a translated packet at all. At the traffic control layer it handles pod traffic. At the driver layer, XDP accelerates external traffic to NodePort and LoadBalancer services and supports direct server return, so responses skip the load balancing node.

Identity-based policy is the other change. An IP-based policy in a cluster where addresses recycle every few minutes is a policy about nothing. Cilium assigns a numeric identity to each unique set of pod labels, distributes the label-to-identity mapping, and carries the identity in the packet or resolves it from a map. A policy decision is then a lookup on identity pairs, which is constant time and which stays correct when an address is reused.

Address management

The address management plugin allocates pod addresses. Three models exist.

Host-local allocation gives each node a fixed range, usually a /24 carved from a cluster range, and the node allocates from it with no coordination. It is simple and it wastes addresses, because a node running 10 pods holds 254.

Cluster-wide allocation draws from one pool with a coordinating store, which uses addresses efficiently and adds a dependency on the request path for pod creation.

Cloud-provider allocation attaches real network addresses from the cloud's address space to the node and assigns them to pods. Pods then have addresses the wider network routes natively, security groups apply to them directly, and the node's address count becomes a scaling limit that the instance type sets.

Address exhaustion is the failure that arrives at the worst moment. A cluster sized for 50 nodes with a /16 and a /24 per node supports 256 nodes, and scaling past it requires renumbering, which means recreating the cluster. Size the ranges for the cluster you will have in three years, since address space costs nothing and renumbering costs a migration.

Network policy

By default every pod reaches every pod. A NetworkPolicy selects pods and allows specific ingress and egress. Policies are additive, and once any policy selects a pod, that pod is default-deny for the direction the policy covers.

The enforcement point differs by implementation. An iptables-based implementation writes rules per policy per pod. An eBPF implementation evaluates a map lookup on identity pairs. At a few hundred policies the difference is not visible, and at a few thousand it is the difference between a working cluster and one where rule programming never converges.

Start every namespace with a default-deny policy for ingress and egress, and add the allowed flows. The reverse order, which is to run open and tighten later, never finishes, because nobody can enumerate what is currently allowed. Egress policy matters as much as ingress: it is what stops a compromised pod from reaching the metadata service, an internal database, or the internet.

7.4 Container Storage Interface and Persistent Volumes

CSI does for storage what CNI does for networking. A driver implements a gRPC service, and the orchestrator calls it to provision, attach, mount, expand, and snapshot volumes.

The plugin architecture

A driver has two components. The controller plugin runs as a deployment, usually with one active replica elected by lease. It performs the operations that talk to the storage system's API: create a volume, delete it, attach it to a node, take a snapshot, expand it. The node plugin runs as a daemonset on every node and performs operations that touch that node: format the device, mount it, bind-mount it into the pod directory.

Three identity calls describe the driver: GetPluginInfo, GetPluginCapabilities, and Probe. The capability response is what tells the orchestrator which of the optional calls the driver supports, so a driver that cannot snapshot does not advertise it.

The volume lifecycle

The sequence from claim to running pod has six steps and each one has its own failure mode.

  1. PVC created, storageClass names a provisioner
       external-provisioner sidecar sees it
       -> CreateVolume(name, capacity, params, topology)
       -> storage system creates the volume, returns a volume handle
       -> a PersistentVolume object is created and bound to the claim

  2. Pod scheduled to node N
       volume topology constrains which nodes are eligible;
       WaitForFirstConsumer defers step 1 until the pod is scheduled,
       so that the volume is created in the right zone

  3. external-attacher sees the VolumeAttachment
       -> ControllerPublishVolume(volumeHandle, nodeId)
       -> cloud attaches the block device to the instance: 5 to 60 s

  4. kubelet calls the node plugin
       -> NodeStageVolume(volumeHandle, stagingPath)
       -> format if needed, mount once per node at a global path

  5. kubelet calls again, once per pod
       -> NodePublishVolume(stagingPath, targetPath)
       -> bind-mount the staged path into the pod's directory

  6. teardown reverses it:
       NodeUnpublishVolume -> NodeUnstageVolume
       -> ControllerUnpublishVolume -> DeleteVolume (if reclaim is Delete)

The split between stage and publish exists so that one volume can serve several pods on a node without formatting or mounting more than once. Stage happens once per node, publish happens once per pod.

Access modes constrain the topology, and they are properties of the storage rather than wishes. ReadWriteOnce means one node at a time, which is what a block device gives. ReadWriteOncePod restricts it further to one pod, which is what a database that cannot tolerate two writers needs. ReadWriteMany means many nodes at once, which needs a file protocol such as NFS rather than a block device.

Volume binding mode decides when provisioning happens. Immediate creates the volume as soon as the claim exists, which can place it in a zone where the pod cannot be scheduled. WaitForFirstConsumer defers creation until the pod is scheduled, so the volume lands in the right zone. For any zonal storage the second is the only correct choice, and the first is the default in older storage classes, which is a standing trap.

Stateful workloads

A StatefulSet gives each pod a stable identity, a stable hostname, and its own claim generated from a template. Pods are created in order and deleted in reverse order, which matters for a clustered database where a replica must not start before the primary exists.

Two behaviors surprise teams. Deleting a StatefulSet does not delete the claims, by design, so that data survives an accidental deletion. And scaling down leaves the claims behind, so scaling back up reattaches the original volumes to the original identities.

Snapshots are a CSI feature and they are not backups. A snapshot usually lives in the same storage system and shares its failure domain. A snapshot of a running database captures whatever was on disk at that instant, which for an engine with a write-ahead log is a crash-consistent image that needs recovery on restore. Application-consistent snapshots need the application to quiesce first, which is what a database-aware operator does by calling a freeze hook before triggering the snapshot.

The last consideration is where the data belongs at all. A distributed database that already replicates across nodes gains little from replicated block storage underneath and pays twice for durability. Local NVMe with application-level replication is faster and cheaper for that case. Network block storage with its own replication suits an application that cannot replicate itself, which is most single-writer workloads.

Sources and further reading

The Linux manual pages for namespaces(7), user_namespaces(7), capabilities(7), seccomp(2), pivot_root(2), and clone(2). The kernel documentation at Documentation/admin-guide/cgroup-v2.rst. The OCI runtime specification. The etcd documentation on the data model, watches, and maintenance. The Kubernetes documentation on API Priority and Fairness, the scheduler framework, and admission control. The CNI specification and the Cilium documentation on the eBPF datapath and identity-based policy. The CSI specification and the Kubernetes CSI developer documentation.