Kafka Storage and Consumption Model

Tue, 25 Aug 2026 00:00:00 +0000

Kafka’s high performance relies on “partitioned log + sequential writes + batched pulls”. Understanding the storage and consumption model is essential to configure parallelism correctly and avoid duplicate consumption.

Topics and Partitions

A topic is a logical subject; a partition is the physical unit of parallelism, and messages are only appended to the end of a partition.
Each partition is an ordered, immutable log composed of multiple segment files, maintaining an offset (unique and monotonically increasing within the partition).
The partition count determines the upper bound of consumption parallelism: at any moment a partition is consumed by exactly one consumer within a group.

Offset and Consumption Position

The position a consumer commits after processing messages is the offset, which Kafka stores in the internal topic __consumer_offsets:

Kafka Reliability and Exactly-Once

Tue, 25 Aug 2026 00:00:00 +0000

Kafka’s reliability requires coordinated configuration across the producer, broker and consumer. We cover it through three goals: no loss, no duplication, ordered.

Producer-Side Reliability

Control write durability level via acks:

acks=0: considered successful as soon as sent; may lose messages; highest throughput.
acks=1: success once the leader writes; may lose if the leader crashes.
acks=all: success only after all ISR replicas sync; safest, combined with min.insync.replicas to avoid single points.

Enabling retries (retries) handles transient failures but introduces duplicates: network jitter causes the producer to retry without receiving the ack, writing the same message twice.

Kafka on Zhuifeng Notes

Kafka Storage and Consumption Model

Topics and Partitions

Offset and Consumption Position

Kafka Reliability and Exactly-Once

Producer-Side Reliability