Stream Processing and Real-Time Event Infrastructures

The log as a primitive

A log is an append-only sequence of records, each with a monotonically increasing offset. That is the whole data structure. Its power comes from what it does not do: it does not delete, it does not update in place, and it does not track per-consumer state.

A traditional message queue deletes a message once a consumer acknowledges it. The broker therefore tracks per-message state, delivery becomes a coordination problem, and replay is impossible. A log keeps every record for a retention period and stores only one number per consumer group, which is the offset. Replay is a matter of resetting that number.

Three properties follow, and they are why the log became the backbone of event-driven architecture. Many consumers read the same records independently, at their own pace, with no effect on each other. A new consumer reads history from the beginning, which is how a new service builds its state or a new projection gets populated. A bug that corrupted a downstream store is correctable by fixing the code and replaying.

Order is per partition, not per topic. A topic splits into partitions for parallelism, and records with the same key go to the same partition, so records for one key stay ordered. Choosing the key is therefore the most consequential schema decision in a streaming system: it decides both the ordering guarantee and the parallelism.

  topic: orders,  3 partitions,  key = customer_id

  P0  [0][1][2][3][4][5][6][7]--> append here
  P1  [0][1][2][3][4]--------> append here
  P2  [0][1][2][3][4][5][6][7][8][9]--> append here

  group "billing"      reads P0@6, P1@4, P2@9     (near the head)
  group "analytics"    reads P0@2, P1@0, P2@3     (lagging, fine)
  group "new-search"   reads P0@0, P1@0, P2@0     (replaying history)

  Ordering is guaranteed within a partition only.
  All records for customer 4471 hash to one partition, so that
  customer's events stay ordered. Across customers, no order exists.

Kafka internals

A partition is a directory of segment files on disk. Each segment holds records plus an index from offset to file position and a second index from timestamp to offset. Writes append to the active segment. When it reaches a size or an age limit, the broker closes it and starts a new one. Retention deletes or compacts whole segments, which is cheap because it is a file unlink.

Two mechanisms give Kafka its throughput, and both come from the operating system rather than from clever data structures.

Writes go to the page cache and the operating system flushes them. Kafka does not fsync on every write by default. Durability comes from replication instead: a record is durable when enough replicas hold it, and the cluster tolerates the loss of any machine's page cache. Chapter 7 priced an fsync, and this design avoids paying it per record.

Reads use sendfile, which copies from the page cache to a socket inside the kernel with no copy into user space. A consumer that reads recent data reads from memory and the bytes never enter the broker process. This is why one broker sustains gigabytes per second with modest CPU.

Replication uses in-sync replicas. Each partition has a leader and a set of followers. A follower that keeps up stays in the in-sync replica set. The producer's acks setting decides when a write is acknowledged: acks=0 returns immediately, acks=1 returns when the leader has it, and acks=all returns when every in-sync replica has it. Combine acks=all with min.insync.replicas=2 on a three-replica topic, and a write survives one broker failure and the cluster still accepts writes.

KRaft replaced ZooKeeper for metadata. The controller quorum runs Raft over an internal metadata topic, which removes an external dependency, cuts failover time, and raises the supported partition count substantially. ZooKeeper mode was removed in Kafka 4.0.

Tiered storage moves closed segments to object storage and keeps recent segments local. The broker's local disk then sizes to the hot window rather than to the retention period. That makes a long retention affordable, and it makes rebalancing fast, because a moved partition copies only the local segments.

Diskless topics go further. KIP-1150, accepted in March 2026, proposes writing partition data directly to object storage rather than to broker disks. Follow-up proposals for the diskless core and coordinator were still under discussion later in 2026, so the feature is a direction rather than a shipped capability. Several commercial systems already implement the pattern, and it targets the cross-zone replication traffic that dominates the cost of a cloud Kafka deployment.

Redpanda takes a different implementation path with the same protocol. It is written in C++ on the Seastar framework with a thread-per-core, share-nothing design. It manages its own I/O rather than relying on the page cache, and it runs Raft per partition instead of the in-sync replica scheme. The reported benefit is lower and more stable tail latency with fewer tuning knobs.

Delivery semantics

Three levels exist, and the names hide how much work the third one takes.

At-most-once means that a record is delivered zero or one times. The consumer commits its offset before processing. A crash after the commit loses the record. This is correct for telemetry where a gap is acceptable.

At-least-once means one or more times. The consumer commits after processing. A crash before the commit reprocesses the record. This is the default in almost every system, and it requires the downstream effect to be idempotent.

Exactly-once means that the effect happens once, whatever fails. It does not mean that the message is delivered once, because that is impossible in an asynchronous network. It means that the visible result is as if it were.

Kafka builds it from three parts. The idempotent producer attaches a producer identifier and a sequence number to every batch, so the broker discards a duplicate that a retry created. Transactions let a producer write to several partitions and commit the consumer offsets atomically, so the read, the process, and the write form one unit. The consumer sets isolation.level= read_committed so that it never sees records from an aborted transaction.

// Read-process-write as one atomic unit.
producer.initTransactions();

while (running) {
    ConsumerRecords<String, String> records = consumer.poll(POLL_TIMEOUT);
    if (records.isEmpty()) continue;

    producer.beginTransaction();
    try {
        for (ConsumerRecord<String, String> record : records) {
            for (ProducerRecord<String, String> out : transform(record)) {
                producer.send(out);
            }
        }
        // The offsets commit inside the same transaction as the output.
        producer.sendOffsetsToTransaction(
                offsetsOf(records), consumer.groupMetadata());
        producer.commitTransaction();
    } catch (ProducerFencedException | OutOfOrderSequenceException e) {
        producer.close();          // another instance took over; stop
        throw e;
    } catch (KafkaException e) {
        producer.abortTransaction();   // safe to retry the whole batch
    }
}

The guarantee holds inside Kafka. It does not extend to an external system unless that system participates. A pipeline that reads Kafka, writes PostgreSQL, and commits an offset has two systems and no shared transaction. Two options close it. Make the database write idempotent with a natural key or a deduplication table keyed by the record's offset. Or use a sink connector that implements two-phase commit against the external system, which Flink's TwoPhaseCommitSinkFunction provides.

The cost of exactly-once in Kafka is a throughput reduction in the range of 10 to 30 percent and added latency from the transaction commit, which is why it is not the default. Idempotent consumers with at-least-once delivery are simpler and cheaper, and they reach the same outcome for most workloads.

Consumer groups and rebalancing

A consumer group divides the partitions of a topic among its members. Each partition goes to exactly one member, so parallelism caps at the partition count. A group with more members than partitions leaves the extras idle.

A rebalance reassigns partitions when membership changes. The classical protocol stops the world: every member revokes every partition, the group coordinator computes a new assignment, and every member starts again. Processing stops for the whole group during that window.

Cooperative sticky assignment fixes most of it. Members keep the partitions they will retain and revoke only the ones that move, which happens over two short rounds instead of one long stop. Static group membership goes further, giving each member a stable identifier so that a rolling restart does not trigger a rebalance at all as long as the member returns within the session timeout.

Consumer lag is the primary health metric. It is the difference between the partition's latest offset and the group's committed offset, and it converts to time with the record rate. Alert on lag in seconds rather than in records, because a thousand-record lag means one second on a busy topic and one hour on a quiet one.

Event time, processing time, and watermarks

A stream carries two clocks. Event time is when the event happened, and it lives in the record. Processing time is when the system sees it. They differ by network delay, by buffering, by a device that was offline, and by a replay that processes a week of history in ten minutes.

Any aggregation over time must use event time, or the answer changes depending on when the job ran. A replay under processing time puts a week of events into one window and produces a different result from the original run, which destroys reproducibility.

Event time creates a new problem: the system cannot know that it has seen every event for a window, because a late event can always arrive. A watermark is the answer. It is an assertion that no event with a timestamp earlier than T will arrive from now on. When the watermark passes the end of a window, the system emits the result.

Watermark generation is a heuristic, and it is a direct trade between latency and completeness. A watermark that lags the maximum observed timestamp by five seconds emits results five seconds late and captures events that arrive up to five seconds out of order. A watermark that lags by one hour is more complete and useless for a real-time dashboard.

Allowed lateness handles what the watermark missed. A window can stay open for a grace period after the watermark passes, updating its result when a late event arrives. Events later than the grace period go to a side output so that nothing is silently dropped. A late-event stream that nobody monitors is a data quality incident waiting to be discovered by a customer.

  TUMBLING, 1 minute: fixed, non-overlapping. One event, one window.
    |--10:00--|--10:01--|--10:02--|--10:03--|

  SLIDING, 1 minute size, 20 second slide: overlapping. One event, N windows.
    |--------|
        |--------|
            |--------|

  SESSION, 30 second gap: bounded by inactivity, per key.
    e  e e    e            (gap > 30s)        e   e
    |--------------|                          |------|

  WATERMARK
    events arrive:  10:00:58, 10:01:03, 10:00:59 (late), 10:01:10
    watermark = max_seen - 5s
    when the watermark passes 10:01:00, the 10:00 window emits.
    the 10:00:59 event arrives before that, so it is counted.
    an event at 10:00:30 arriving at 10:01:20 is late:
      within allowed lateness -> window updates and re-emits
      beyond it               -> side output, never silently dropped

State, checkpoints, and recovery

A stateless operator, such as a filter or a map, needs no recovery beyond replaying the input. A stateful operator, such as an aggregation, a join, or a deduplication, holds state that must survive a failure.

Flink keeps working state in memory or in an embedded RocksDB instance and checkpoints it to durable storage. Kafka Streams keeps state in RocksDB and backs it up to a compacted changelog topic in Kafka, so a restarted instance rebuilds its store by replaying the changelog.

The checkpoint protocol is the interesting part, because it must capture a consistent snapshot of a distributed dataflow without stopping it. Flink uses asynchronous barrier snapshotting, which is a variant of the Chandy-Lamport algorithm. The source injects a numbered barrier into the stream. Each operator, on receiving barrier N on all of its inputs, snapshots its state and forwards the barrier. When every operator acknowledges barrier N, the checkpoint is complete and consistent.

Recovery restores the last complete checkpoint and rewinds the sources to the offsets that the checkpoint recorded. Processing resumes, and the result is exactly-once state even though records were reprocessed, because the state went back too.

A savepoint is a checkpoint that an operator triggers on purpose and that survives job cancellation. It is how a job upgrade works: take a savepoint, stop the job, deploy new code, and restart from the savepoint. State schema evolution has rules, and an operator that changes its state type without a migration cannot restore.

Change data capture

Change data capture reads a database's write-ahead log and emits one event per row change. It gives a stream of every change with no application code and with no polling load on the database. Debezium is the common implementation, and it reads PostgreSQL logical replication, MySQL binary logs, MongoDB change streams, and others.

Chapter 6 introduced the outbox pattern to solve the dual write problem. Change data capture is the efficient way to implement the relay: the outbox row appears in the write-ahead log, the connector publishes it, and no polling query runs. The application writes state and outbox row in one transaction, and the log guarantees that both or neither appear.

Two operational facts matter. A logical replication slot in PostgreSQL retains write-ahead log segments until the consumer reads them, so a stopped connector fills the primary's disk. Monitor replication slot lag as a first-class alert. And an initial snapshot of a large table produces a burst that can overwhelm the downstream, so use an incremental snapshot mechanism that interleaves snapshot chunks with live changes.

Streaming into a lakehouse

The endpoint of most streaming pipelines is analytical storage. Open table formats made that endpoint uniform. Apache Iceberg is the format that every major engine now reads and writes, with a REST catalog for multi-engine access.

Iceberg version 2 added row-level deletes, which made streaming upserts practical. Version 3 introduced deletion vectors, which replace positional delete files with a compact bitmap per data file and make merge-on-read much cheaper, plus row lineage for tracking changes. Those two features are what turn a streaming sink from an append-only dump into a table that supports updates.

The small file problem is the operational reality of streaming into a table. A job that commits every thirty seconds produces 2,880 files per day per partition, and query planning then spends more time listing files than reading data. Every deployment needs a compaction job that rewrites small files into large ones and expires old snapshots. Treat it as a required component, not as an optimization.

Sizing and operations

Partition count is the parameter that is hardest to change later. Increasing it changes the key-to-partition mapping, which breaks per-key ordering for existing keys and invalidates any consumer state that assumed the old mapping. Decreasing it is not supported.

Choose the count from the target throughput divided by the per-partition throughput a single consumer achieves, then add headroom for growth. More partitions cost memory and file handles on the broker, longer rebalances, and more metadata. Fewer partitions cap parallelism. A common range is 12 to 100 partitions for a busy topic, and thousands per broker at the cluster level.

Hot partitions come from a skewed key. One customer that generates a third of the events puts a third of the load on one partition, and that partition's consumer becomes the bottleneck while the others idle. Detect it by measuring per-partition rates, not topic totals. Two fixes work. Add a salt to the key for the hot entities and handle the fan-in downstream. Or use a custom partitioner that splits the hot key across several partitions, where ordering permits.

Sources and further reading

Jay Kreps, "The Log: What every software engineer should know about real-time data's unifying abstraction", 2013. Martin Kleppmann, "Making Sense of Stream Processing". Tyler Akidau and others, "The Dataflow Model", VLDB 2015, and the "Streaming Systems" book that followed, for windowing and watermarks. Paris Carbone and others, "Lightweight Asynchronous Snapshots for Distributed Dataflows", for the Flink checkpoint protocol. The Kafka improvement proposals KIP-98 for transactions, KIP-500 for KRaft, KIP-405 for tiered storage, and KIP-1150 for diskless topics. The Apache Iceberg specification.