How Kafka Quietly Runs Flipkart, Netflix, and Half the Internet

You tap Buy Now on Flipkart. In the next second, six things happen. Your order is recorded. The seller is notified. Inventory is decremented. A confirmation email starts moving. An SMS is queued. Your recommendation profile is updated. And somehow, the page just says "Order placed" without making you wait for any of it.

That fan-out is not magic. It is almost always Kafka.

This post is a plain walkthrough of what Kafka actually is, how the pieces fit together, and why it ends up at the center of so many large systems.

You do not need distributed systems experience to follow it. If you have ever written code that calls another service, you already have enough.

Kafka in one sentence

Kafka is a notice board that never forgets.

One service pins a note to it ("order 4417 was placed"). Kafka writes that note down, in order, and keeps it for days. Any number of other services read the board at their own speed. Nobody has to be online at the same time. Nobody has to know who else is reading.

That is genuinely the whole idea. The rest of this post is about why "write it down and let people read it later" turns out to be the right shape for systems that handle millions of events a second.

The problem Kafka is solving

The natural way to wire services together is direct calls. Order service calls inventory, calls email, calls SMS, calls recommendations. It looks fine on a whiteboard. Then production happens.

Every new consumer of "an order was placed" forces a change in the order service. If the email provider is slow, checkout slows down with it. If recommendations crash, you cannot place an order. Retries and timeouts pile up. One service hiccup turns into a checkout outage.

What you actually want is for the order service to do one thing: announce that an order happened. Whoever cares can listen. Whoever is slow can catch up later. Whoever is down can replay from where they left off when they come back.

That is the job Kafka does.

The shape of Kafka in one paragraph

A Kafka cluster is a set of servers called brokers. They store streams of records organised into topics. Each topic is split into partitions, which are append-only logs persisted on disk. Producers write records to a topic. Consumers read them. A consumer group is a set of consumers that cooperate to read a topic in parallel, with each partition assigned to exactly one consumer in the group.

Everything else is detail.

If that paragraph was dense, here is the same thing as a picture. One topic, orders.placed, split into three partitions, read by two independent teams:

                     TOPIC: orders.placed
      ┌──────────────────────────────────────────────┐
      │  partition 0 │ o0 │ o1 │ o2 │ o3 │        →  │
      │  partition 1 │ o0 │ o1 │ o2 │             →  │
      │  partition 2 │ o0 │ o1 │ o2 │ o3 │ o4 │   →  │
      └──────────────────────────────────────────────┘
            ▲                              │
            │ writes                       │ reads
            │                              │
   ┌────────┴───────┐            ┌─────────┴─────────┐
   │ order-service  │            ▼                   ▼
   │   (producer)   │     ┌─────────────┐    ┌─────────────┐
   └────────────────┘     │   group:    │    │   group:    │
                          │  email-svc  │    │  analytics  │
                          └─────────────┘    └─────────────┘
                           reads at its       reads at its
                            own speed          own speed

Three things to notice, because they are the three things beginners usually get wrong:

1. Each partition is a list that only grows at the right end. Nothing is ever edited or deleted in place.
2. Reading does not remove anything. email-svc reading order o2 does not stop analytics from reading it too.
3. Each group tracks its own position in each partition. They move at different speeds and never block each other.

The four words you need to keep straight

Topic. A named stream of events. For example, orders.placed or payments.failed. You can think of it as the name of a folder of log files.

Partition. A topic is split into N partitions. A partition is a strictly ordered, append-only log — a file you are only ever allowed to add to at the end. Records inside a partition get an offset, which is just a line number that always counts up: 0, 1, 2, 3. Ordering is guaranteed inside a partition, not across partitions. This is the single most important property to internalise.

Producer. The code that writes records to a topic. It picks which partition a record goes to either by a key (records with the same key always land on the same partition, which is how you preserve per-user or per-order ordering) or round-robin if no key is provided.

Consumer group. A set of consumer processes sharing a group id. Kafka assigns partitions to members of the group. If a topic has 12 partitions and the group has 4 consumers, each consumer reads 3 partitions. Add a fifth consumer, the partitions are rebalanced. Lose a consumer, its partitions move to the survivors. This is how Kafka achieves horizontal scale and fault tolerance for reads.

One consequence of that last rule catches everyone once: a partition is read by exactly one consumer in a group, so adding more consumers than partitions does nothing. A topic with 3 partitions and a group of 10 consumers leaves 7 of them sitting idle. Partition count is your ceiling, so pick it with a bit of headroom.

What "Kafka handles billions of requests" actually means

It does not mean Kafka is doing something exotic on each event. The exact opposite. Kafka is fast because the per-event cost is tiny and the architecture forces things into the shapes that hardware is good at.

A few mechanics worth knowing:

Append-only writes. A partition is a log file, and producing a record just appends bytes to the end of it. There are no random updates, no edits in the middle, no index structures to rebalance. Writing to disk in one continuous stream is roughly a hundred times faster than jumping around it, which is exactly what a database has to do. Kafka gets that speed for free by refusing to support editing.

Zero-copy reads. Normally, sending a file over the network means the data is copied from disk into your program's memory, then copied again into the network card's buffer. Kafka skips the detour: it asks the operating system to ship the bytes straight from its disk cache to the network connection. Fewer copies, less CPU, and a single server can push data as fast as its network cable allows. (Worth knowing: this shortcut is lost when you encrypt traffic with TLS, because encryption has to touch the bytes. It is still fast, just not free.)

Batching and compression. Producers do not send one record per network call. They collect records for a few milliseconds, squash them together, compress the bundle, and send it in one go. One round trip ships thousands of events, so the cost per event collapses.

Partitioning is the parallelism knob. Want to handle twice the load? Roughly speaking, add partitions and add consumers. Each partition is read by exactly one consumer per group, so partition count is the upper bound on parallelism inside a group.

Put together, a single broker can do hundreds of thousands of events per second on modest hardware. A real cluster does millions.

Replication: why a broker can die without you noticing

Every partition is stored on more than one machine. The replication factor is how many copies exist, and it is typically 3. One of those copies is the leader — the only one producers and consumers actually talk to. The other two are followers, which do nothing but continuously copy the leader's data so they stay identical.

If the leader's machine dies, Kafka promotes one of the up-to-date followers to leader. Clients notice the change, reconnect, and carry on. Nothing is lost, provided you configured the producer to treat a write as successful only once the followers also have it. That setting is acks=all, and it is the difference between "the leader wrote it to one disk" and "the data survives that disk catching fire".

This is the trade you make: a millisecond or two of extra write latency in exchange for surviving the loss of any single broker without losing committed data. Almost always worth it.

One more thing beginners expect and do not get: Kafka does not delete a record once it has been read. Each topic has a retention setting — keep everything for 7 days, say, or up to 50 GB per partition — and records sit there until that limit passes, read or not. That is what makes replay possible. A consumer that was broken all weekend can rewind to Friday and reprocess, because the data never went anywhere.

A walk through a single order on Flipkart

You tap Buy Now. Here is what flows through Kafka.

1. Order service writes one record to topic orders.placed. The record key is order_id, so all events for the same order land on the same partition and stay in order. Producer call returns in single-digit milliseconds.
2. The user immediately sees "Order placed". The order service is done.
3. Downstream, several consumer groups are subscribed to orders.placed:
   • email-svc group sends the confirmation email.
   • sms-svc group sends the SMS.
   • inventory-svc group decrements stock and may produce a follow-up event to inventory.reserved.
   • recommendations-svc group updates your profile.
   • analytics-pipeline group writes the event to a warehouse.
4. Each consumer group reads at its own pace. The email service can be slow without slowing the SMS service. The recommendations service can be down for an hour, then catch up by replaying from its last committed offset.
5. If a new team wants to react to orders next month, they create a new consumer group, point it at orders.placed, and start reading. The order service does not change at all.

That last point is the entire reason large companies adopt Kafka. New consumers cost the producer nothing.

Where ordering actually lives

Beginners often assume Kafka gives global ordering. It does not, and that is on purpose. Global ordering across a topic would mean a single partition, which would mean a single writer, which would mean no horizontal scale.

What Kafka guarantees is: records with the same key go to the same partition, and a partition is strictly ordered.

So if you key by user_id, every event for that user is in order. Two different users may interleave however the network feels like, and that is fine because they are independent.

Pick your key with care. It is one of the few choices you cannot easily change later.

What about failures

A consumer crashes mid-batch. What happens?

Kafka tracks the committed offset for each consumer group per partition: "this group has processed up to offset 4719 on partition 3". When the consumer restarts, or when the partition is reassigned to a sibling, the new owner picks up at offset 4720.

This gives at-least-once delivery by default: nothing is ever silently dropped, but a record can be processed twice. Picture a consumer that sends the confirmation email and then crashes half a second before it records "done". When it restarts, it sees unfinished work and sends that email again.

The usual fix is not to chase perfect delivery. It is to make processing the same event twice harmless — the property called idempotent. If your handler writes a row keyed by event_id, the second attempt overwrites the first instead of duplicating it, and the problem disappears. Kafka does offer true exactly-once via transactional producers, but it costs throughput and complexity, and most teams find idempotent handlers cheaper.

A broker dies? Leaders fail over to followers, clients reconnect, traffic continues. As a user of Kafka, you mostly notice a small latency spike and move on.

The Dabbawalas analogy, with the right ending

Mumbai's dabbawalas sort thousands of lunchboxes a day using a tiny code stamped on the lid. Every handler along the route only needs to read the code to decide where the box goes next. No central planner. No retries. Just a partitioning scheme that is good enough for parallel humans to follow.

Kafka is that, in software. The key on a record is the code on the lid. The partition is the route. The consumer group is the team of dabbawalas at the destination. The lunchbox is your event. The reason the whole thing scales is that nothing in the middle has to think.

What Kafka is not

It is worth being honest about the boundaries.

• Kafka is not a message queue in the RabbitMQ sense. It does not delete a message when you read it. Records sit in the log until retention expires. Multiple consumer groups can read the same record independently.
• Kafka is not a database. You can store events forever, but querying them by anything other than offset or key range is not its job. Pair it with a database or a stream processor like Flink or ksqlDB for that.
• Kafka is not the right tool for low-throughput request-response. If you have a few thousand messages a day between two services, a REST call or a simple queue will serve you better than running a Kafka cluster.

It earns its place when you need durable, ordered, fan-out streams that many independent consumers can read in parallel. That happens to describe a startling number of problems at scale.

The vocabulary, on one page

Every Kafka tutorial assumes you already know these. Here they are in plain English, so you can skim past the jargon in any other article you read next.

If you learn to watch one number in production, make it lag. Everything unhealthy in a Kafka system shows up there first.

A note on ZooKeeper

If you read older Kafka material, you will find ZooKeeper everywhere: Kafka used it to track which brokers are alive and to elect the one broker in charge of partition assignments. Modern Kafka has replaced that with its own built-in mechanism called KRaft, so new clusters do not run ZooKeeper at all.

It is still worth understanding what it was doing, because the same problem — "how does a cluster agree on who is in charge?" — shows up in every distributed system you will touch. I wrote that up separately in ZooKeeper, Quietly Holding the Cluster Together.

Closing thought

Most of what makes Kafka feel magical is that it picks a small set of strong guarantees and refuses to compromise on them. Append-only logs. Partition-level ordering. Replicated leaders. Consumer groups that rebalance on their own. Once you see those four ideas, every other Kafka concept you read about is just a refinement.

Next time your Flipkart order confirmation arrives a second after you tapped Buy Now, you know the shape of what just happened. A single record landed on a partition, and a small crowd of independent consumers got to work, none of them blocking the others, none of them blocking you.