Kafka, Explained Simply (Part 2): The Internals Everyone Asks About
This is Part 2 of a two-part introduction to Kafka. Part 1 covered where Kafka actually sits inside a real app, with producers, topics, partitions, and consumer groups. This part goes a little deeper into the internals and the settings you'll actually have to touch once you start using it for real.
This is the part that actually gets asked about in interviews and shows up in real production incidents. Each of these is simple on its own, there's just more vocabulary.
The Technical Details Everyone Asks About
Broker vs. cluster
A broker is one single Kafka server. A cluster is a group of brokers working together. This distinction matters because it tells you how Kafka scales: scale one broker up (a bigger machine) and you're scaling vertically; add more brokers to the cluster and you're scaling horizontally, which is the usual approach at real traffic volumes.
Why Kafka is so fast: the append-only log
Every partition is physically stored as an append-only log file, new messages only ever get added at the end, nothing in the middle is ever changed. This single design choice is where most of Kafka's speed comes from:
- No one is ever searching the disk for a specific entry to edit, so there's no expensive lookup happening on every write.
- Messages are immutable, once written, a message cannot be changed. You can only eventually delete old messages based on a retention policy you configure (for example, "keep everything for 7 days").
This is also why people call Kafka a "distributed commit log", that's a fancy way of describing exactly what's above.
Leader, follower, and what happens when a broker dies
Click to view full screen
Every partition has one leader and some number of followers, usually living on different brokers. All writes go to the leader. Followers just continuously copy whatever the leader has, so they're always close to up to date. In newer Kafka versions, consumers are allowed to read from a follower too, not only the leader, which helps spread out read load.
If the broker holding the leader goes down, one of the in-sync followers gets promoted to be the new leader. Deciding which follower gets promoted used to be handled by a separate tool called Zookeeper. Modern Kafka doesn't need Zookeeper anymore, it handles this itself with a built-in consensus protocol called K-Raft. You don't need to know how K-Raft works internally, just that it's the thing quietly deciding who the new leader is.
Offsets: how a consumer remembers its place
Every message inside a partition gets a sequential number called an offset, basically "this is the 47th message in this partition." A consumer keeps track of the highest offset it has successfully processed, and that number gets saved back to Kafka.
Why this matters: if a consumer crashes and comes back up, it doesn't start from zero and it doesn't lose its place. It just resumes right after the last offset it had committed. This one idea is what makes Kafka consumers resilient to restarts and crashes.
There's a real production gotcha here. The setting auto.offset.reset decides what a brand new consumer group does when it has no saved offset yet: earliest reads from the very first message ever stored, latest only reads new messages from now on. If a topic has months of history and you forget to set this to latest, a new consumer can get flooded with old messages the moment it starts up and crash under the load. This has happened to real teams, not just in theory.
Kafka pulls, it doesn't push
Unlike some messaging systems that push data to you, Kafka consumers pull. A consumer repeatedly asks the broker, "anything new for me?" at a pace it controls. A few settings worth knowing exist (you don't need to memorize the exact names, just understand what each one is for):
| What you're tuning | What it does |
|---|---|
max.poll.records | Caps how many messages you get back in one pull. Lower it if each message takes a while to process. |
fetch.min.bytes | Broker waits until it has at least this much data before responding. Raise it to pull less often. |
fetch.max.wait.ms | A timeout partner to the above: send whatever's available once this much time has passed, even if the minimum bytes weren't reached. |
max.poll.interval.ms | Longest gap allowed between two pulls before Kafka assumes the consumer is dead and reassigns its partitions. |
At least once, and why idempotency matters
Kafka guarantees at-least-once delivery: within a single consumer group, a message will always be delivered, and it may occasionally be delivered more than once, for example, if a consumer crashes right after processing a message but right before telling Kafka it finished.
That means your own processing logic has to be safe to run twice on the same message without causing a double effect, like charging a card twice. Making an operation safe to repeat is called making it idempotent. Kafka handles its internal bookkeeping idempotently on its own, but whether your consumer logic is idempotent is entirely on you.
Retries and the dead letter queue
Kafka producers can retry automatically if a write fails. Consumers don't get this for free, if you want retry logic on the consuming side, you write it yourself.
If a message keeps failing no matter how many times you retry it, don't retry forever. The standard pattern is a dead letter queue (DLQ), basically a separate holding area where "messages we gave up on for now" get moved. An engineer can look at what piled up there later and figure out why those specific messages keep failing, usually a real bug, not bad luck.
Scaling: hot partitions
A hot partition is a partition that, because of how your partitioning key works out, ends up getting far more traffic than the others, which defeats the whole point of having multiple partitions. A few common fixes:
- Use a compound key (combine two fields instead of one) so traffic spreads out more evenly.
- Add a bit of randomness ("salting") to the key.
- Apply back-pressure, slow the producer down if consumers genuinely can't keep up.
Quick recap
| Term | What it actually means |
|---|---|
| Topic | A named stream for one kind of event |
| Partition | An ordered, append-only slice of a topic |
| Producer | Whatever writes messages into a topic |
| Consumer group | A set of consumers sharing the work of reading a topic |
| Broker | One Kafka server |
| Cluster | A group of brokers working together |
| Leader / Follower | The partition replica that takes writes, and the replicas that copy it |
| Offset | A message's position number inside a partition |
| Idempotent | Safe to run more than once without a double effect |
| DLQ | Where messages go after failing too many retries |
That's genuinely most of what you need to use Kafka with confidence, and to follow (or lead) a real conversation about it. Everything past this point is detail on top of these same ideas.