Distributed KV Store

A from-scratch implementation of the Raft consensus algorithm in Rust, layered with a fault-tolerant key–value store. The project serves as both a working distributed database and an exercise in systems programming with Rust's async ecosystem.

All three core Raft subproblems are tackled separately: leader election, log replication, and safety (the election restriction). The implementation is designed for understandability — the state machine is explicit, the event loop is single-threaded, and each RPC handler is a standalone function.

Architecture

flowchart TB
    subgraph Clients
        GRPC["gRPC API<br/>(tonic/prost)"]
    end

    subgraph App["Event Loop (app.rs)"]
        SELECT["tokio::select!"]
    end

    subgraph Core["Raft Core (node.rs)"]
        FSM["State Machine<br/>Follower | Candidate | Leader"]
        PENDING["Pending Ops<br/>poll_events()"]
    end

    subgraph Storage
        WAL["Write-Ahead Log<br/>(log.rs)"]
        KV["KV Store<br/>(store.rs)"]
    end

    subgraph Transport
        GRPC_T["GrpcTransport"]
        SIM_T["SimulatedTransport"]
    end

    GRPC -->|"Propose"| SELECT
    SELECT --> FSM
    FSM --> PENDING
    FSM --> WAL
    PENDING -->|"committed"| KV
    FSM -->|"RPCs"| Transport
    SELECT -->|"heartbeats + poll"| FSM

Raft state machine

stateDiagram-v2
    [*] --> Follower
    Follower --> Candidate : election timeout
    Candidate --> Leader : wins majority
    Candidate --> Follower : discovers leader\nor higher term
    Leader --> Follower : discovers higher term
    Leader --> Leader : heartbeat + replicate
    Follower --> Follower : heartbeat received

Transport layer

Networking is abstracted behind the PeerTransport trait, which gives the Raft core a uniform interface for sending RPCs regardless of whether the cluster is running over real gRPC or in-memory channels.

Transport	Use
`GrpcTransport`	Production — connects to peers over HTTP/2 via `tonic`
`SimulatedTransport`	Testing and the demo — delivers RPCs through `tokio` channels with zero network overhead

The simulated transport is what makes the integration tests deterministic: nodes communicate through in-memory queues, and the test harness drives the cluster step by step without races or real network latency.

Raft core (`src/raft/node.rs`)

The Raft state machine is an async event loop that processes three kinds of event:

ElectionTimeout — the node increments its term, votes for itself, and sends RequestVote RPCs to every peer. Votes are tallied as they arrive; a majority wins the election.
RequestVote / AppendEntries (inbound) — handled synchronously against the current term and log state. Responses are sent immediately via oneshot channels.
Propose — only valid on the leader. The command is appended to the local log and AppendEntries is sent to every follower in parallel. Once a majority acknowledges, the entry is committed and applied to the state machine.

Elections and proposals are non-blocking: spawned tasks run in parallel, and a poll_events() method resolves them as they complete. This keeps the main loop responsive and avoids deadlocks in the simulated test harness.

Write path

sequenceDiagram
    participant C as Client
    participant L as Leader
    participant F1 as Follower A
    participant F2 as Follower B

    C->>L: Put("key", "value")
    L->>L: Append to local log
    L->>F1: AppendEntries(entry)
    L->>F2: AppendEntries(entry)
    F1-->>L: ack
    F2-->>L: ack
    Note over L: Majority reached
    L->>L: Commit entry
    L->>L: Apply to state machine
    L-->>C: OK
    L->>F1: Heartbeat (leader_commit)
    L->>F2: Heartbeat (leader_commit)
    F1->>F1: Commit + apply
    F2->>F2: Commit + apply

Persistence

The write-ahead log (src/raft/log.rs) stores each entry as a JSON line in a .wal file. On startup, the log is replayed to restore the node's state. Snapshots compact the log by serialising the entire state machine to a .snap file and discarding entries that precede the snapshot index.

KV store (`src/kv/store.rs`)

A DashMap-backed concurrent hash map that applies committed log entries in order. Supported operations:

Put — {"op": "put", "key": "...", "value": "..."}
Delete — {"op": "delete", "key": "..."}

All writes flow through Raft; reads are served from the local replica.

Quick start

Prerequisites

Rust toolchain (rustc ≥ 1.96, cargo ≥ 1.96)
Protocol Buffers compiler (protoc) for code generation

Run the demo

The demo starts a 3-node cluster entirely in memory — no networking required — and walks through leader election, log replication, and consistent reads across all replicas:

cargo run --bin demo

Output:

=== Distributed KV Store Demo ===

1. Starting leader election...
   Leader elected: node-b

2. Proposing KV operations...
   {"key":"hello","op":"put","value":"world"}: committed
   {"key":"raft","op":"put","value":"consensus"}: committed
   {"key":"rust","op":"put","value":"async"}: committed

3. Final cluster state:
   node-a: term=3 commit_idx=3
      hello = world
      raft = consensus
      rust = async
   node-b: term=3 commit_idx=3
      hello = world
      raft = consensus
      rust = async
   node-c: term=3 commit_idx=3
      hello = world
      raft = consensus
      rust = async

Run a real gRPC cluster

Start three nodes in separate terminals:

cargo run --bin server -- -i node-a -a 127.0.0.1:50051 \
    -p node-b=127.0.0.1:50052,node-c=127.0.0.1:50053

cargo run --bin server -- -i node-b -a 127.0.0.1:50052 \
    -p node-a=127.0.0.1:50051,node-c=127.0.0.1:50053

cargo run --bin server -- -i node-c -a 127.0.0.1:50053 \
    -p node-a=127.0.0.1:50051,node-b=127.0.0.1:50052

Then interact with the cluster using grpcurl:

# Write a key (value must be base64-encoded)
grpcurl -plaintext -d '{"key":"hello","value":"d29ybGQ="}' 127.0.0.1:50051 raft.KV/Put

# If the node you hit is not the leader, try another
grpcurl -plaintext -d '{"key":"hello","value":"d29ybGQ="}' 127.0.0.1:50052 raft.KV/Put

CLI reference

Usage: server [OPTIONS]

Options:
  -i, --id <ID>            Node identifier (default: node-1)
  -a, --addr <ADDR>        gRPC listen address (default: 127.0.0.1:50051)
  -p, --peers <PEERS>      Comma-separated id=addr pairs
  -w, --wal-dir <WAL_DIR>  WAL directory (default: /tmp/distributed-kv)
  -h, --help               Print help

Run the tests

cargo test

Seven tests cover the critical paths:

Test	What it verifies
`test_leader_election_succeeds_with_majority`	3-node cluster converges on a single leader
`test_only_one_leader_per_term`	Concurrent elections never produce split-brain
`test_term_increments_on_election`	Terms are monotonically increasing
`test_log_replication`	A write proposed on the leader appears on every follower
`test_kv_store_operations`	Put, Get, and Delete on the state machine
`test_log_persistence`	WAL append, index lookup, and term retrieval
`test_leader_election` (unit)	Single node transitions Follower → Candidate on timeout

Project structure

.
├── Cargo.toml
├── build.rs                    # tonic-build proto compilation
├── proto/
│   └── raft.proto              # gRPC service definitions
├── src/
│   ├── lib.rs                  # Module root
│   ├── app.rs                  # Event loop orchestrating Raft + gRPC + KV
│   ├── raft/
│   │   ├── mod.rs              # RaftClient trait, re-exports
│   │   ├── node.rs             # Core state machine (Follower/Candidate/Leader)
│   │   ├── log.rs              # Replicated log with WAL persistence
│   │   ├── rpc.rs              # Internal RPC message types
│   │   ├── transport.rs        # PeerTransport trait
│   │   ├── grpc_transport.rs   # Production gRPC transport
│   │   └── sim_transport.rs    # In-memory transport for testing
│   ├── kv/
│   │   ├── mod.rs
│   │   └── store.rs            # DashMap-backed state machine
│   ├── server/
│   │   ├── mod.rs              # AppEvent enum
│   │   ├── raft_service.rs     # gRPC service for Raft RPCs
│   │   ├── kv_service.rs       # gRPC service for client KV ops
│   │   └── grpc_client.rs      # gRPC client stub wrapper
│   └── bin/
│       ├── server.rs           # Production binary
│       └── demo.rs             # In-memory demonstration
└── tests/
    └── integration_tests.rs    # Multi-node cluster tests

Technical highlights

Async event-driven core — the Raft state machine runs in a single-threaded tokio::select! loop. Elections and replication spawn parallel tasks that resolve asynchronously via poll_events(), keeping the loop responsive without locks on the state machine.
Injectable networking — the PeerTransport trait lets the same Raft code run over real gRPC or in-memory channels. The simulated transport enables fast, deterministic integration tests that exercise the full consensus protocol.
gRPC with tonic/prost — both the internal Raft RPCs (RequestVote, AppendEntries) and the client-facing KV API share the same HTTP/2 transport, defined in a single .proto file.
WAL persistence — each log entry is appended to a JSON-line file before responding to the client. On restart, the full log is replayed. Snapshots compact the log to bound disk usage.
No unsafe code — the entire codebase uses only safe Rust.

What Raft guarantees

Safety — committed entries are never overwritten or lost, even across leadership changes. A candidate can only win an election if its log is at least as up-to-date as a majority of the cluster.
Fault tolerance — with n nodes, the system tolerates up to ⌊(n−1)/2⌋ failures. A 3-node cluster survives 1 crash; a 5-node cluster survives 2.
Linearizable writes — a write is acknowledged only after replication to a majority. If the leader crashes after committing, the next leader will have the entry.

What Raft does not guarantee

Byzantine fault tolerance — the implementation assumes a crash-fault model. Nodes that lie, send conflicting messages, or behave arbitrarily are out of scope.
Throughput at scale — a single leader serialises all writes, which becomes a bottleneck. Production systems layer batching, pipelining, and multi-Raft sharding on top.

Licence

MIT

Name		Name	Last commit message	Last commit date
proto		proto
src		src
tests		tests
.gitignore		.gitignore
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
LICENSE		LICENSE
README.md		README.md
build.rs		build.rs

Navigation Menu

Repository files navigation

Distributed KV Store

Architecture

Raft state machine

Transport layer

Raft core (`src/raft/node.rs`)

Write path

Persistence

KV store (`src/kv/store.rs`)

Quick start

Prerequisites

Run the demo

Run a real gRPC cluster

CLI reference

Run the tests

Project structure

Technical highlights

What Raft guarantees

What Raft does not guarantee

Further reading

Licence

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Distributed KV Store

Architecture

Raft state machine

Transport layer

Raft core (src/raft/node.rs)

Write path

Persistence

KV store (src/kv/store.rs)

Quick start

Prerequisites

Run the demo

Run a real gRPC cluster

CLI reference

Run the tests

Project structure

Technical highlights

What Raft guarantees

What Raft does not guarantee

Further reading

Licence

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Raft core (`src/raft/node.rs`)

KV store (`src/kv/store.rs`)