ZooKeeper, Quietly Holding the Cluster Together

If you run anything in production at scale, you have almost certainly used ZooKeeper without noticing. Kafka used to depend on it. HBase, Solr, Hadoop, Druid, ClickHouse Keeper, and dozens of internal platforms still do. It is one of those pieces of infrastructure that no end user has ever heard of and no operator can do without.

This post walks through what ZooKeeper actually is, what problem it solves, and how the pieces work together. It assumes no distributed systems background — every term gets defined the first time it shows up.

ZooKeeper in one sentence

ZooKeeper is the one machine a group of machines is allowed to believe.

Imagine ten servers doing the same job, and exactly one of them is supposed to be in charge. They cannot just decide amongst themselves — messages get delayed, networks split in half, and two servers cheerfully conclude they are both the boss. So instead they all go ask the same trusted referee: am I the leader? The referee answers one of them yes and everyone else no, and it keeps answering consistently even if some of its own machines die.

ZooKeeper is that referee. It stores a tiny amount of shared information, and its entire value is that every machine reading it sees exactly the same answer at exactly the same moment.

Two terms will keep coming up, so here they are up front:

• Split-brain — when two machines both think they are in charge and both accept writes. Your data quietly forks into two versions and someone spends a weekend merging them by hand. This is the disaster ZooKeeper exists to prevent.
• Quorum — a strict majority. ZooKeeper refuses to accept a change unless more than half of its own servers have saved it. Since two different majorities always overlap, there is no way for two conflicting answers to both be considered official.

The problem: who is in charge

A distributed system is a group of machines that has to agree on something. Which node owns this shard. Which replica is the primary. What the current configuration is. Who is allowed to write right now.

Letting every node decide on its own ends badly, and it ends badly in the specific way described above: two nodes both believe they are the primary, both accept writes, and the data diverges.

The clean answer is to pick one machine to be the source of truth for these decisions. The problem is that the moment you do, you have to answer the next question: what happens when that machine dies. A coordinator that cannot survive its own failure is worse than no coordinator at all.

This is the gap ZooKeeper fills. It is a small, reliable service whose only job is to be the trustworthy place where the rest of your system keeps shared state and elects leaders. It is allowed to be slow and boring. The rest of your stack gets to be fast and interesting.

What ZooKeeper actually is

A ZooKeeper deployment is an odd-numbered cluster of servers, called an ensemble, typically 3 or 5 machines. They run a consensus protocol called Zab (ZooKeeper Atomic Broadcast), which is the set of rules they follow to stay in agreement. Every write goes through a quorum: a majority of the ensemble must save the change to disk before ZooKeeper tells the client "done".

The data it stores looks like a filesystem. A tree of entries called znodes, each with a path like /services/payments/leader, a small payload, and some metadata:

/
├── brokers/
│   └── ids/
│       ├── 1   → {"host": "kafka-1", "port": 9092}   (ephemeral)
│       ├── 2   → {"host": "kafka-2", "port": 9092}   (ephemeral)
│       └── 3   → {"host": "kafka-3", "port": 9092}   (ephemeral)
├── controller  → {"brokerid": 2}                     (ephemeral)
└── config/
    └── feature-flags → {"newCheckout": true}         (persistent)

Clients connect over the network and perform a handful of operations: create, read, update, delete, list children, and set a watch. That is it — there is no query language, no joins, no indexes.

That is the whole shape of it. A replicated tree of tiny entries, accessible over the network, with a strong consistency guarantee. Everything ZooKeeper is famous for — leader election, locks, service discovery — is not built into the server. Clients assemble it from those few operations.

Why the data model is small on purpose

A znode is meant to hold a pointer, not a payload. A few hundred bytes describing "the current primary is host-7 on port 9092" is the right kind of data. A megabyte of application state is the wrong kind.

The reason is that every write goes through quorum and every change is broadcast to every connected client that asked to watch it. If you treat ZooKeeper as a key-value store for your business data, you will saturate it and discover that consensus is expensive in ways your application cache never was. Treat it as a coordination layer and it will stay calm.

Ephemeral nodes: the trick that powers leader election

A znode can be marked ephemeral, which means it is tied to the life of the client that created it.

To understand that, you need one more concept: a session. When a client connects, ZooKeeper opens a session for it and expects a small heartbeat every so often. As long as the heartbeats arrive, the session is alive. Stop hearing from the client for longer than the session timeout — because the process exited, the machine died, or the network cut out — and ZooKeeper declares the session dead and deletes every ephemeral znode that client created.

So an ephemeral znode is a tiny "I am still here" flag that cleans itself up automatically. Nobody has to notice the crash and tidy up. The absence of the node is the notification.

This single feature is the basis for almost every coordination pattern ZooKeeper supports.

Leader election with ephemeral nodes is one short recipe:

1. Each candidate node tries to create an ephemeral znode at a known path, for example /services/payments/leader.
2. Exactly one create succeeds. That client is the leader.
3. Every other client sets a watch on the path so it is notified when the leader znode disappears.
4. If the leader crashes, loses network, or its process simply ends, its session expires and ZooKeeper deletes the znode. Every watcher fires. The candidates race again. A new leader is elected within seconds.

No human in the loop. No external heartbeating to maintain. The cluster heals itself because ZooKeeper has a definition of "this client is still alive" that is honest about TCP and network partitions.

The gotcha nobody warns you about

Leader election is not instant, and that gap matters more than it looks.

Say the old leader is not dead — it is just cut off from the network for twelve seconds. ZooKeeper times out its session, deletes its znode, and a new leader is elected. Meanwhile the old leader has not noticed anything is wrong. For a few seconds, there are genuinely two processes that believe they are the leader, and the one that is wrong is still happily writing.

ZooKeeper is telling the truth; the old leader simply has not been told yet. The fix does not live in ZooKeeper, it lives downstream. Every leader gets a number that increases with each election — a fencing token — and attaches it to every write. The database, or whatever the leader is writing to, remembers the highest number it has seen and rejects anything stamped with an older one. The deposed leader's writes bounce, harmlessly, the moment a newer leader appears.

If you take one practical thing from this post, take this: "I hold the lock" is always a slightly stale belief, so the thing being protected has to do the final check.

Watchers: how clients stay in sync without polling

If a thousand consumers all wanted to know the current leader, polling every second would put thousands of requests per second on ZooKeeper for no good reason. So ZooKeeper inverts the model.

A client reads a znode and asks: "set a watch on this. The next time it changes, tell me." ZooKeeper remembers the registration and pushes a single notification the next time the node is written, deleted, or its children change. The client then re-reads to get the new value and, if it cares, sets a new watch.

Watchers are one-shot. After firing, they need to be re-registered. This is a deliberate design choice. It keeps server state bounded and makes the protocol simple to reason about. The trade-off is that client libraries do the bookkeeping for you, and you should always re-arm the watch right after handling the event.

What strong consistency actually buys you

Zab gives ZooKeeper two guarantees that sound boring and are actually the whole reason it exists.

Linearizable writes. Every write lands in one global order that all servers agree on, as if the whole cluster were a single machine handling one request at a time. If A's write finishes before B's write starts, then every reader afterwards sees A's change before B's. There is no window where one server has the new value and another still serves the old one — which is exactly what "eventually consistent" systems allow, and exactly what you cannot tolerate when the question is "who is the leader?"

FIFO client order. Everything a single client sends is applied in the order it sent it. Nothing overtakes anything else, so you can fire off a sequence of operations without defensively waiting for each one.

The first guarantee is what makes leader election trustworthy. Without linearizability, two clients could each believe their create succeeded and you are back to split-brain. The second guarantee is what makes building higher-level recipes (locks, queues, barriers) tractable without each one re-implementing a happens-before relation.

A real-world example: Kafka's old controller

For years, Kafka used ZooKeeper for exactly this set of jobs:

• Broker membership. Every Kafka broker created an ephemeral znode on startup under /brokers/ids/<broker-id>. The set of children of /brokers/ids was the live broker list. If a broker crashed, its znode vanished and everyone knew.
• Controller election. Exactly one broker is elected the controller, the one responsible for partition leadership decisions. That election was a one-line ephemeral-node recipe.
• Topic and partition metadata. Topic configs and partition assignments were stored as znodes, watched by the controller.

Modern Kafka has moved this to its own built-in metadata quorum (KRaft), because running a second distributed system just to coordinate the first one added real operational overhead at scale. But the design remains the canonical example of what ZooKeeper was built for: small, critical, frequently-read metadata with strong consistency and reliable failure detection. If you want the other half of that story, I wrote about how Kafka itself works.

Why an odd number of servers

A ZooKeeper write needs a quorum, which is a strict majority. With 3 servers, the quorum is 2. With 5 servers, the quorum is 3. With 4 servers, the quorum is also 3. The extra server in the 4-node case buys you nothing: you can tolerate one failure either way, and you have paid for an extra machine. Five gives you tolerance for two simultaneous failures, which is usually the right place to land for production.

This is also why ZooKeeper does not scale linearly with cluster size. Adding more nodes makes writes slower (more replicas to wait for) without giving you more write throughput. It is built to be small, reliable, and trusted, not large and fast.

Common patterns built on the primitives

Once you have ephemeral nodes, sequential nodes (znodes that get an auto-incrementing suffix), and watchers, you can build a surprising amount of coordination logic in a few dozen lines on the client side:

• Distributed locks. Each client creates an ephemeral sequential znode under /locks/resource-x. The client with the lowest sequence number holds the lock. Everyone else watches the znode immediately ahead of them. When that znode disappears (lock released, or holder died), the watcher fires and the next client steps in.
• Service discovery. Each instance of a service registers an ephemeral znode under /services/<name>/. Clients list the children to discover instances and set a watch so the list stays current.
• Configuration. Application config is stored at a known znode. Every service watches it. Pushing a new config is a single write; the entire fleet picks it up within seconds.
• Barriers and gates. Coordinate stages of a job by having workers wait until the children of a barrier znode reach a target count.

The point worth internalising is that ZooKeeper itself does not implement locks or service discovery. It gives you primitives that are strong enough for clients to build those things correctly.

What ZooKeeper is bad at

It is not a queue. The temptation to model a work queue as a directory of znodes shows up in every introduction, and it works for low rates and small payloads. Push it harder and the watcher fan-out, the per-write quorum cost, and the bounded znode size will all hurt you. Use a real queue.

It is not a config server in the GitHub-for-YAML sense. It is good at "here is the current value of this small thing, tell me when it changes". It is not good at storing the last hundred revisions of a 200 KB file.

It is not a general-purpose database. It does not do queries, transactions across znodes (it does support a small multi op but not arbitrary cross-path transactions in the SQL sense), or rich indexing. Keep your data elsewhere.

When in doubt: small payloads, frequent reads, infrequent writes, strong consistency required. That is the ZooKeeper-shaped problem.

The vocabulary, on one page

Closing thought

The reason ZooKeeper has stuck around for well over fifteen years is not that it is fancy. It is the opposite. It does a small number of things, refuses to be talked into doing more, and is honest about what it costs. A handful of replicated nodes running a consensus protocol, holding a tree of tiny entries, with two clever primitives (ephemeral nodes and watchers) on top.

Every time a leader fails over silently, a service rebalances around a dead pod, or a config flag rolls out across a fleet in seconds, there is a decent chance a quiet little ensemble somewhere is doing the boring work that lets the loud parts of your stack feel automatic.