Testing Raft with a fake clock: deterministic chaos in Go

2026-11-02 · raftlab

Raft is easy to read and hard to trust. The paper is a few pages long, and the bugs live in the interleavings: a message that arrives after a leader change, a timer that fires just before it is stopped, a crash halfway through a write. I wrote raftlab, a Raft implementation in Go with a replicated key-value store and a browser visualizer, and the part I would keep if I had to throw the rest away is how it is tested.

raftlab visualizer during a network partition: five Raft nodes in a pentagon, two of them marked leader, with dashed red links where messages are dropped between the old leader and the rest
The visualizer during a partition. The old leader (N2, term 2) still believes it leads but cannot reach a majority. The other side has elected N5 in term 3. Red dashed links are dropped messages.

Four seams and no clock

The core, raft.Node, implements Figure 2 of the Raft paper and nothing else. It has no goroutines and no knowledge of sockets or wall-clock time. Everything it touches goes through one of four interfaces:

Transport     send RPCs to peers
Clock         election and heartbeat timers
Storage       durable term, vote, log, snapshot
StateMachine  apply committed commands

Each has a real implementation and a test one. Transport is HTTP in a real cluster, an in-process network with fault injection in the visualizer, and a manual in-memory one in unit tests. Clock is the system clock or a fake one. Storage is a checksummed write-ahead log. StateMachine is the key-value store, or a recorder in tests.

Because time and I/O are injected, a whole five-node cluster can run on the fake clock inside a single-threaded test. Nothing sleeps. The test advances virtual time in 10 ms steps and the fake network delivers messages when their virtual delay has passed. A run is then a pure function of its random seed: same seed, same elections, same failures.

The chaos test

The chaos test runs five nodes for 40 virtual seconds. It randomly crashes and restarts nodes, partitions the network, and injects loss, duplication and delay, while four clients write and read. After every 10 ms step it checks two safety properties: election safety (never two leaders in the same term) and committed-prefix agreement (no two nodes disagree about an entry they have both committed).

At the end it heals the cluster and checks three more things. The replicas converge, every acknowledged write survived, and every read returned the last acknowledged write. A failure prints its seed, and replaying it is one command:

CHAOS_SEED=17 go test -run Chaos ./internal/kv
make test-chaos    # 100 seeds

That replay is what makes the test useful. A distributed test that fails once in a hundred runs and never again teaches you nothing. A failing seed can be rerun, stepped through and kept as a regression test.

What a fake clock cannot find

The bug that stayed hidden longest was in the clock itself. A real clock can run a callback that fired just before Stop was called. In raftlab that means a stale election timer can still run, start a spurious election and depose a healthy leader. The fake clock cannot reproduce it, because the fake clock never runs a callback concurrently with a stop.

The fix is to replace timers instead of resetting them and to give each one a generation counter, so a callback from an old generation does nothing. A hand-driven clock covers it in tests. The lesson is the limit of the whole approach: deterministic simulation finds the interleavings you modelled, and the concurrency of the clock was not one of them.

Two bugs about late and repeated messages

Both come from the network delaying, reordering or duplicating messages, which is exactly what the fault injector does.

A delayed AppendEntries erased newer entries. An old request arriving late truncated the follower's log even though nothing in it conflicted. The log is now truncated only on a real conflict.

Reordered and duplicated replies moved nextIndex backwards. A stale reply made the leader forget how much of a follower's log it had already matched. Replies now carry MatchIndex, so a stale one cannot regress the leader's view of a follower.

A bug only Windows could find

I found this one while recording the demo, not in a test. Windows cannot rename over an open file, so the first time a node compacted its log it panicked. Compaction writes a new file, fsyncs it and renames it over the old one, so a crash leaves either the old file or the new one and never a mix. The write-ahead log now closes its handles before the rename and reopens them afterwards, and skips the directory fsync on Windows, where directories cannot be opened for sync. It is a reminder that the simulation checks the algorithm and not the operating system.

What it still does not check

The read check, that every read returns the last acknowledged write, is weaker than a real linearizability checker, which would verify every ordering the history allows. A Porcupine-style checker is on the roadmap, together with PreVote, CheckQuorum and membership changes through joint consensus. The design trade-offs, including why reads go through the log and cost a full replication round, are written down in the repository.

The source is at github.com/nweber23/raftlab.

← Back to the raftlab project