How Kafka Quietly Runs Flipkart, Netflix, and Half the Internet

You tap Buy Now on Flipkart. In the next second, six things happen. Your order is recorded. The seller is notified. Inventory is decremented. A confirmation email starts moving. An SMS is queued. Your recommendation profile is updated. And somehow, the page just says "Order placed" without making you wait for any of it.

That fan-out is not magic. It is almost always Kafka.

This post is a plain walkthrough of what Kafka actually is, how the pieces fit together, and why it ends up at the center of so many large systems.

The problem Kafka is solving

The natural way to wire services together is direct calls. Order service calls inventory, calls email, calls SMS, calls recommendations. It looks fine on a whiteboard. Then production happens.

Every new consumer of "an order was placed" forces a change in the order service. If the email provider is slow, checkout slows down with it. If recommendations crash, you cannot place an order. Retries and timeouts pile up. One service hiccup turns into a checkout outage.

What you actually want is for the order service to do one thing: announce that an order happened. Whoever cares can listen. Whoever is slow can catch up later. Whoever is down can replay from where they left off when they come back.

That is the job Kafka does.

The shape of Kafka in one paragraph

A Kafka cluster is a set of servers called brokers. They store streams of records organised into topics. Each topic is split into partitions, which are append-only logs persisted on disk. Producers write records to a topic. Consumers read them. A consumer group is a set of consumers that cooperate to read a topic in parallel, with each partition assigned to exactly one consumer in the group.

Everything else is detail.

The four words you need to keep straight

Topic. A named stream of events. For example, orders.placed or payments.failed. You can think of it as the name of a folder of log files.

Partition. A topic is split into N partitions. A partition is a strictly ordered, append-only log. Records inside a partition have a monotonically increasing offset (0, 1, 2, ...). Ordering is guaranteed inside a partition, not across partitions. This is the single most important property to internalise.

Producer. Writes records to a topic. It picks the partition either by a key (records with the same key always land on the same partition, which is how you preserve per-user or per-order ordering) or round-robin if no key is provided.

Consumer group. A set of consumer processes sharing a group id. Kafka assigns partitions to members of the group. If a topic has 12 partitions and the group has 4 consumers, each consumer reads 3 partitions. Add a fifth consumer, the partitions are rebalanced. Lose a consumer, its partitions move to the survivors. This is how Kafka achieves horizontal scale and fault tolerance for reads.

What "Kafka handles billions of requests" actually means

It does not mean Kafka is doing something exotic on each event. The exact opposite. Kafka is fast because the per-event cost is tiny and the architecture forces things into the shapes that hardware is good at.

A few mechanics worth knowing:

Append-only writes. A partition is a log file. Producing a record is write(fd, bytes) at the end of the file. There are no random updates, no in-place mutations, no index trees to rebalance. Sequential writes to disk are roughly two orders of magnitude faster than random ones, and the OS page cache absorbs the rest.

Zero-copy reads. When a consumer fetches a batch, Kafka uses the sendfile() syscall to move bytes from the page cache to the network socket without copying them into userspace. That is how a single broker can saturate a 10 GbE NIC.

Batching and compression. Producers buffer records and send them in batches, compressed (snappy, lz4, zstd). One TCP round trip ships thousands of events. Throughput goes up, per-event overhead goes down.

Partitioning is the parallelism knob. Want to handle twice the load? Roughly speaking, add partitions and add consumers. Each partition is read by exactly one consumer per group, so partition count is the upper bound on parallelism inside a group.

Put together, a single broker can do hundreds of thousands of events per second on modest hardware. A real cluster does millions.

Replication: why a broker can die without you noticing

Each partition has a replication factor, typically 3. One broker holds the leader replica for that partition. The other replicas are followers that pull updates from the leader and stay in sync. Producers and consumers only talk to the leader.

If the leader broker dies, one of the in-sync followers is promoted. Clients re-discover the new leader and resume. No data is lost as long as you acknowledged writes only after they were copied to the in-sync replicas (acks=all).

This is the trade you make: a small write-latency cost in exchange for surviving the loss of any single broker without losing committed data.

A walk through a single order on Flipkart

You tap Buy Now. Here is what flows through Kafka.

1. Order service writes one record to topic orders.placed. The record key is order_id, so all events for the same order land on the same partition and stay in order. Producer call returns in single-digit milliseconds.
2. The user immediately sees "Order placed". The order service is done.
3. Downstream, several consumer groups are subscribed to orders.placed:
   • email-svc group sends the confirmation email.
   • sms-svc group sends the SMS.
   • inventory-svc group decrements stock and may produce a follow-up event to inventory.reserved.
   • recommendations-svc group updates your profile.
   • analytics-pipeline group writes the event to a warehouse.
4. Each consumer group reads at its own pace. The email service can be slow without slowing the SMS service. The recommendations service can be down for an hour, then catch up by replaying from its last committed offset.
5. If a new team wants to react to orders next month, they create a new consumer group, point it at orders.placed, and start reading. The order service does not change at all.

That last point is the entire reason large companies adopt Kafka. New consumers cost the producer nothing.

Where ordering actually lives

Beginners often assume Kafka gives global ordering. It does not, and that is on purpose. Global ordering across a topic would mean a single partition, which would mean a single writer, which would mean no horizontal scale.

What Kafka guarantees is: records with the same key go to the same partition, and a partition is strictly ordered.

So if you key by user_id, every event for that user is in order. Two different users may interleave however the network feels like, and that is fine because they are independent.

Pick your key with care. It is one of the few choices you cannot easily change later.

What about failures

A consumer crashes mid-batch. What happens?

Kafka tracks the committed offset for each consumer group per partition: "this group has processed up to offset 4719 on partition 3". When the consumer restarts, or when the partition is reassigned to a sibling, the new owner picks up at offset 4720.

This gives at-least-once delivery by default. A record may be processed twice if a consumer dies after processing but before committing. If you need exactly-once, Kafka supports it via transactional producers and read_committed consumers, but most teams handle it more cheaply by making the consumer idempotent (for example, keying database writes by event_id).

A broker dies? Leaders fail over to followers, clients reconnect, traffic continues. As a user of Kafka, you mostly notice a small latency spike and move on.

The Dabbawalas analogy, with the right ending

Mumbai's dabbawalas sort thousands of lunchboxes a day using a tiny code stamped on the lid. Every handler along the route only needs to read the code to decide where the box goes next. No central planner. No retries. Just a partitioning scheme that is good enough for parallel humans to follow.

Kafka is that, in software. The key on a record is the code on the lid. The partition is the route. The consumer group is the team of dabbawalas at the destination. The lunchbox is your event. The reason the whole thing scales is that nothing in the middle has to think.

What Kafka is not

It is worth being honest about the boundaries.

• Kafka is not a message queue in the RabbitMQ sense. It does not delete a message when you read it. Records sit in the log until retention expires. Multiple consumer groups can read the same record independently.
• Kafka is not a database. You can store events forever, but querying them by anything other than offset or key range is not its job. Pair it with a database or a stream processor like Flink or ksqlDB for that.
• Kafka is not the right tool for low-throughput request-response. If you have a few thousand messages a day between two services, a REST call or a simple queue will serve you better than running a Kafka cluster.

It earns its place when you need durable, ordered, fan-out streams that many independent consumers can read in parallel. That happens to describe a startling number of problems at scale.

Closing thought

Most of what makes Kafka feel magical is that it picks a small set of strong guarantees and refuses to compromise on them. Append-only logs. Partition-level ordering. Replicated leaders. Consumer groups that rebalance on their own. Once you see those four ideas, every other Kafka concept you read about is just a refinement.

Next time your Flipkart order confirmation arrives a second after you tapped Buy Now, you know the shape of what just happened. A single record landed on a partition, and a small crowd of independent consumers got to work, none of them blocking the others, none of them blocking you.