ZooKeeper, Quietly Holding the Cluster Together

If you run anything in production at scale, you have almost certainly used ZooKeeper without noticing. Kafka used to depend on it. HBase, Solr, Hadoop, Druid, ClickHouse Keeper, and dozens of internal platforms still do. It is one of those pieces of infrastructure that no end user has ever heard of and no operator can do without.

This post walks through what ZooKeeper actually is, what problem it solves, and how the pieces work together. No analogies for their own sake, just the parts that helped me make sense of it.

The problem: who is in charge

A distributed system is a group of machines that has to agree on something. Which node owns this shard. Which replica is the primary. What the current configuration is. Who is allowed to write right now.

Letting every node decide on its own ends badly. Two nodes both think they are the primary, both accept writes, and now you have split-brain. The data diverges and you spend a weekend reconciling it.

The clean answer is to pick one machine to be the source of truth for these decisions. The problem is that the moment you do, you have to answer the next question: what happens when that machine dies. A coordinator that cannot survive its own failure is worse than no coordinator at all.

This is the gap ZooKeeper fills. It is a small, reliable service whose only job is to be the trustworthy place where the rest of your system keeps shared state and elects leaders. It is allowed to be slow and boring. The rest of your stack gets to be fast and interesting.

What ZooKeeper actually is

A ZooKeeper deployment is an odd-numbered cluster of servers, called an ensemble, typically 3 or 5 nodes. They run a consensus protocol called Zab (ZooKeeper Atomic Broadcast). Every write goes through a quorum: a majority of the ensemble must persist the write before it is acknowledged.

The data it stores looks like a filesystem. A tree of nodes called znodes, each with a path like /services/payments/leader, a small payload (bytes, kilobytes at most), and some metadata. Clients connect over the network and perform a handful of operations: create, read, update, delete, list children, and set a watch.

That is the whole shape of it. A replicated tree of tiny nodes, accessible over the network, with a strong consistency guarantee. Everything else ZooKeeper is famous for is built on top of those primitives by clients.

Why the data model is small on purpose

A znode is meant to hold a pointer, not a payload. A few hundred bytes describing "the current primary is host-7 on port 9092" is the right kind of data. A megabyte of application state is the wrong kind.

The reason is that every write goes through quorum and every change is broadcast to every connected client that asked to watch it. If you treat ZooKeeper as a key-value store for your business data, you will saturate it and discover that consensus is expensive in ways your application cache never was. Treat it as a coordination layer and it will stay calm.

Ephemeral nodes: the trick that powers leader election

A znode can be marked ephemeral. Ephemeral nodes are bound to the session that created them. When that session ends, whether the client called close or the TCP connection just died, the ephemeral node disappears.

This single feature is the basis for almost every coordination pattern ZooKeeper supports.

Leader election with ephemeral nodes is one short recipe:

1. Each candidate node tries to create an ephemeral znode at a known path, for example /services/payments/leader.
2. Exactly one create succeeds. That client is the leader.
3. Every other client sets a watch on the path so it is notified when the leader znode disappears.
4. If the leader crashes, loses network, or its process simply ends, its session expires and ZooKeeper deletes the znode. Every watcher fires. The candidates race again. A new leader is elected within seconds.

No human in the loop. No external heartbeating to maintain. The cluster heals itself because ZooKeeper has a definition of "this client is still alive" that is honest about TCP and network partitions.

Watchers: how clients stay in sync without polling

If a thousand consumers all wanted to know the current leader, polling every second would put thousands of requests per second on ZooKeeper for no good reason. So ZooKeeper inverts the model.

A client reads a znode and asks: "set a watch on this. The next time it changes, tell me." ZooKeeper remembers the registration and pushes a single notification the next time the node is written, deleted, or its children change. The client then re-reads to get the new value and, if it cares, sets a new watch.

Watchers are one-shot. After firing, they need to be re-registered. This is a deliberate design choice. It keeps server state bounded and makes the protocol simple to reason about. The trade-off is that client libraries do the bookkeeping for you, and you should always re-arm the watch right after handling the event.

What strong consistency actually buys you

Zab gives ZooKeeper two guarantees that sound boring and are actually the whole reason it exists.

Linearizable writes. Every write is ordered with respect to every other write across the entire ensemble. If client A's write is acknowledged before client B starts its write, every subsequent reader sees them in that order. No "eventual" anything.

FIFO client order. All operations from a single client are applied in the order the client sent them. Pipelines work the way you expect.

The first guarantee is what makes leader election trustworthy. Without linearizability, two clients could each believe their create succeeded and you are back to split-brain. The second guarantee is what makes building higher-level recipes (locks, queues, barriers) tractable without each one re-implementing a happens-before relation.

A real-world example: Kafka's old controller

For years, Kafka used ZooKeeper for exactly this set of jobs:

• Broker membership. Every Kafka broker created an ephemeral znode on startup under /brokers/ids/<broker-id>. The set of children of /brokers/ids was the live broker list. If a broker crashed, its znode vanished and everyone knew.
• Controller election. Exactly one broker is elected the controller, the one responsible for partition leadership decisions. That election was a one-line ephemeral-node recipe.
• Topic and partition metadata. Topic configs and partition assignments were stored as znodes, watched by the controller.

Modern Kafka has moved this to its own internal Raft-based metadata quorum (KRaft) because the ZooKeeper dependency added operational overhead at very large scales. But the design pattern remains the canonical example of what ZooKeeper was built for: small, critical, frequently-read metadata with strong consistency and reliable failure detection.

Why an odd number of servers

A ZooKeeper write needs a quorum, which is a strict majority. With 3 servers, the quorum is 2. With 5 servers, the quorum is 3. With 4 servers, the quorum is also 3. The extra server in the 4-node case buys you nothing: you can tolerate one failure either way, and you have paid for an extra machine. Five gives you tolerance for two simultaneous failures, which is usually the right place to land for production.

This is also why ZooKeeper does not scale linearly with cluster size. Adding more nodes makes writes slower (more replicas to wait for) without giving you more write throughput. It is built to be small, reliable, and trusted, not large and fast.

Common patterns built on the primitives

Once you have ephemeral nodes, sequential nodes (znodes that get an auto-incrementing suffix), and watchers, you can build a surprising amount of coordination logic in a few dozen lines on the client side:

• Distributed locks. Each client creates an ephemeral sequential znode under /locks/resource-x. The client with the lowest sequence number holds the lock. Everyone else watches the znode immediately ahead of them. When that znode disappears (lock released, or holder died), the watcher fires and the next client steps in.
• Service discovery. Each instance of a service registers an ephemeral znode under /services/<name>/. Clients list the children to discover instances and set a watch so the list stays current.
• Configuration. Application config is stored at a known znode. Every service watches it. Pushing a new config is a single write; the entire fleet picks it up within seconds.
• Barriers and gates. Coordinate stages of a job by having workers wait until the children of a barrier znode reach a target count.

The point worth internalising is that ZooKeeper itself does not implement locks or service discovery. It gives you primitives that are strong enough for clients to build those things correctly.

What ZooKeeper is bad at

It is not a queue. The temptation to model a work queue as a directory of znodes shows up in every introduction, and it works for low rates and small payloads. Push it harder and the watcher fan-out, the per-write quorum cost, and the bounded znode size will all hurt you. Use a real queue.

It is not a config server in the GitHub-for-YAML sense. It is good at "here is the current value of this small thing, tell me when it changes". It is not good at storing the last hundred revisions of a 200 KB file.

It is not a general-purpose database. It does not do queries, transactions across znodes (it does support a small multi op but not arbitrary cross-path transactions in the SQL sense), or rich indexing. Keep your data elsewhere.

When in doubt: small payloads, frequent reads, infrequent writes, strong consistency required. That is the ZooKeeper-shaped problem.

Closing thought

The reason ZooKeeper has stuck around for fifteen years is not that it is fancy. It is the opposite. It does a small number of things, refuses to be talked into doing more, and is honest about what it costs. A handful of replicated nodes running a consensus protocol, holding a tree of tiny entries, with two clever primitives (ephemeral nodes and watchers) on top.

Every time a leader fails over silently, a service rebalances around a dead pod, or a config flag rolls out across a fleet in seconds, there is a decent chance a quiet little ensemble somewhere is doing the boring work that lets the loud parts of your stack feel automatic.