Kafka Architecture In-Depth: Beyond the Basics

What you'll learn

• Understand how Kafka's storage engine leverages immutable segments and sparse indexes
• Explain the mechanics of zero-copy data transfer and the OS Page Cache
• Reason about partition replication, In-Sync Replicas (ISR), and the High Watermark
• Understand the architectural shift from ZooKeeper to KRaft (Kafka Raft)
• Navigate producer/consumer mechanics, including cooperative rebalancing

The Mental Model: The Distributed Commit Log#

To understand Kafka, you must first forget traditional message queues. Traditional MQs (like RabbitMQ) track the state of every message. When a consumer reads a message, the broker removes it or marks it as deleted. This per-message state tracking requires complex B-Trees and random disk I/O, which scales poorly.

Kafka inverts this model. Data is written to an append-only log. Messages are never deleted individually; they are only purged in bulk based on a retention policy (e.g., time or size). Consumers track their own reading progress by maintaining an offset (an integer pointer) into this log.

Architecture— The Kafka Macro Architecture

Because the broker doesn't care who has read what (it just blindly appends data and serves sequential reads), Kafka can scale to handle millions of messages per second on commodity hardware.

Real-World Example: Event-Driven E-Commerce#

To see why this decoupling matters, consider a real-world microservices architecture. When a user clicks "Checkout", multiple downstream systems need to react immediately. If they communicated via synchronous REST APIs, a slow downstream service would bottleneck the entire checkout process.

Architecture— Decoupled E-Commerce Flow

The Anatomy of the Flow:

1. The Fast Path: The Order Service receives a request, saves it locally as PENDING, and appends an OrderCreated event to the orders-created topic. It immediately returns a 200 OK to the client. The user isn't kept waiting.
2. Parallel Processing: The Payment Service, Inventory Service, and Notification Service belong to completely different Consumer Groups. They read from orders-created simultaneously.
3. True Decoupling: The Order Service has zero knowledge of the downstream consumers. If the Notification Service crashes, it doesn't impact the checkout. The crashed service simply stops advancing its Kafka offset; once rebooted, it resumes exactly where it left off, catching up on missed emails.
4. Choreography over Orchestration: When the Payment Service successfully charges the card, it doesn't issue an RPC call to shipping. It simply produces a new event to payments-approved. The Fulfillment Service reacts to that event to begin packing the physical box.

The Storage Subsystem: Segments and Indexes#

A Kafka Topic is a logical concept. A Partition is the unit of scale and concurrency. But on disk, a partition is just a directory, and the actual data is split into Segments.

If a partition log grew indefinitely, finding a message or purging old data would be impossible. Therefore, Kafka splits partitions into segment files (default 1GB).

• orders-topic-0
  • 00000000000000000000.logActual message payloads
  • 00000000000000000000.indexMaps offsets to byte positions
  • 00000000000000000000.timeindexMaps timestamps to offsets
  • 00000000000000034512.logThe active segment
  • 00000000000000034512.index
• orders-topic-1

Sparse Indexing#

When a consumer requests to read from offset = 34515, Kafka needs to find where that message lives on disk. It doesn't scan the .log file. Instead, it uses the .index file.

However, the .index file is sparse. It does not contain an entry for every single message (which would waste RAM and disk). By default, Kafka writes an index entry roughly every 4KB of data.

To find offset = 34515:

1. Kafka finds the segment file whose name is less than or equal to 34515 (e.g., 00000000000000034512.log).
2. It binary-searches the corresponding .index file to find the closest offset before 34515 (say, offset 34513 at byte 10240).
3. It goes to the .log file at byte 10240 and sequentially scans forward to find 34515.

Zero-Copy and the OS Page Cache#

Why is Kafka so fast? Because it cheats. It offloads memory management to the OS and bypasses the CPU when sending data to consumers.

In a traditional application, sending data from disk to a network socket involves four context switches and four data copies:

1. Disk -> OS Read Buffer
2. OS Read Buffer -> Application Buffer (JVM Heap)
3. Application Buffer -> OS Socket Buffer
4. OS Socket Buffer -> NIC Buffer

Kafka uses the sendfile() system call, achieving Zero-Copy. The data goes from the Disk to the OS Page Cache, and then directly to the NIC Buffer. The JVM never touches the message payload during consumption.

Replication and Consistency#

To guarantee durability, Kafka replicates partitions across multiple brokers. For every partition, one broker is elected the Leader, and others are Followers.

• Producers only write to the Leader.
• Consumers (by default) only read from the Leader (though Kafka 2.4+ allows reading from followers to save cross-AZ traffic).

The In-Sync Replica (ISR) List#

Kafka maintains a dynamic list of followers that are caught up with the leader, known as the ISR (In-Sync Replicas). If a follower falls behind (determined by the replica.lag.time.max.ms configuration), it is kicked out of the ISR.

Watermarks and Visibility#

When a producer writes a message, it is not immediately visible to consumers. Kafka uses the concept of a High Watermark (HW).

1. Producer appends Message A to the Leader (Leader's Log End Offset moves).
2. Followers fetch Message A.
3. Followers acknowledge they have Message A.
4. The Leader advances the High Watermark to include Message A.
5. Only now can a Consumer read Message A.

Sequence— The Write Path (acks=all)

Producer Config	Durability	Latency	Use Case
acks=0	None (Fire & Forget)	Extremely Low	Metrics, clickstreams (loss tolerable)
acks=1	Medium (Leader only)	Low	Standard application logging
acks=all	High (Leader + ISR)	Higher	Financial transactions, billing data

KRaft: The Post-ZooKeeper Era#

Historically, Kafka relied on Apache ZooKeeper to manage cluster metadata, leader elections, and topic configurations. ZooKeeper was a separate distributed system, requiring its own operational overhead, JVM tuning, and security configurations.

As of Kafka 3.3+, ZooKeeper is deprecated in favor of KRaft (Kafka Raft).

Instead of an external system, a few Kafka brokers are designated as Controllers. These controllers form a quorum and use an event-sourced Raft consensus protocol to manage the metadata log.

Producer and Consumer Mechanics#

Idempotence and Exactly-Once#

Network errors happen. If a producer sends a batch, and the broker writes it but crashes before returning the ACK, the producer will retry. This causes duplicate messages.

By setting enable.idempotence=true (the default since Kafka 3.0), the producer assigns a Sequence Number and a Producer ID (PID) to every message. The broker caches the largest sequence number seen per PID and partition. If a retry arrives with an old sequence number, the broker silently drops the duplicate, ensuring Exactly-Once Semantics (EOS) for single-partition writes.

Consumer Groups and Rebalancing#

When multiple consumers read from a topic, they form a Consumer Group. Kafka assigns partitions to consumers to parallelize the read load. Crucially, a partition can only be consumed by one consumer within a group at a time.

Common mistakes

Having more consumers than partitions.

The extra consumers will sit completely idle. Ensure Partition Count >= Consumer Count.

Ignoring the max.poll.interval.ms configuration.

If your message processing takes longer than this interval, Kafka assumes the consumer is dead and triggers a rebalance.

If a consumer crashes, Kafka must reassign its partitions to the remaining healthy consumers. Historically, this caused a "Stop The World" event (Eager Rebalancing) where all consumers dropped all partitions, and then they were reassigned from scratch.

Modern Kafka uses Cooperative Sticky Rebalancing. Consumers only yield the specific partitions that need to be migrated, allowing other consumers to continue processing without interruption.

In Production#

Running Kafka in production at scale requires a deep understanding of its bottlenecks. It is rarely CPU-bound; it is almost always Disk I/O or Network bound.

Production readiness

0/5

• Set log.retention.bytes and log.retention.hours appropriately to prevent disk exhaustion.
• Configure min.insync.replicas=2 (with replication.factor=3) to prevent data loss on acks=all writes.
• Monitor Under Replicated Partitions (URP). A non-zero URP means a broker is dead or network is degraded.
• Use a robust filesystem. XFS or ext4 are standard; tune them for fast metadata operations.
• Set swappiness to 1. Do not let the OS swap the JVM heap, as it will cause massive GC pauses.

Test your understanding#

Test your knowledge

0/1 answered

1. 1.If you have a topic with 10 partitions, and a consumer group with 12 consumers, what happens?

   Each consumer gets 0.8 partitions on average.10 consumers get 1 partition each, and 2 consumers sit idle.Kafka throws an error and rejects the consumer group.Messages are round-robined to all 12 consumers.

Interview questions#

Interview questions

How does Kafka achieve high throughput with low latency despite persisting data to disk?

What is the High Watermark and why is it important?

Key takeaways

• 1Kafka is a distributed append-only log, not a traditional message queue.
• 2It achieves extreme performance via sequential I/O, OS Page Cache, and Zero-Copy.
• 3Durability is managed via the replication factor and the In-Sync Replicas (ISR) list.
• 4KRaft replaces ZooKeeper, bringing metadata management natively into the Kafka cluster.
• 5Exactly-once semantics are achieved using producer IDs and sequence numbers to deduplicate retries.

Continue reading

• Battle of the Brokers: Kafka vs. RabbitMQ vs. SQS in Production

  A deep dive into the architectural trade-offs between event streaming and message queuing, and how to combine them for hybrid resilience.