Skip to content

Peer calls and Raft proposals go through ports - #135

Open
austin-barrington wants to merge 8 commits into
refactor/code-healthfrom
refactor/peer-rpc
Open

austin-barrington wants to merge 8 commits into
refactor/code-healthfrom
refactor/peer-rpc

Conversation

@austin-barrington

Copy link
Copy Markdown
Member

Stacked on #133. Plan: docs/superpowers/plans/2026-11-03-peer-rpc-port.md.

The application layer stops talking HTTP and Raft directly:

  • PeerRpc port (ports/peer_rpc.rs) with a reqwest adapter (adapters/cluster/http_peer_rpc.rs). Every peer call goes through it: shard write, write-batch, query (still streamed chunk by chunk), metadata, delete, bootstrap, transfer push/ack, MV backfill and handoff, region heartbeat, rehome, sync manifest, shard map, health probe, raft client-write.
  • PeerRpcError keeps "answered with status N" apart from "unreachable", so callers still act on 404/405/409/429/503 and 400/422 exactly as before. Headers and timeouts are unchanged.
  • ConsensusPort (ports/consensus.rs) with RaftConsensus. It proposes locally, or forwards to the leader up to 3 times. The query service, scheduler, drain and leader callbacks use it.
  • Moves:
    • the hinted-handoff watcher goes to adapters/cluster/hinted_handoff_watcher.rs;
    • PeerHttp goes to adapters/cluster/peer_http.rs;
    • ShardRoutingContext::peer_client is removed.
  • application/ has no reqwest, PeerClient or Raft handle outside tests; ports/ has no reqwest.

Behaviour changes:

  • A leader's refusal of a forwarded schema mutation is returned rather than retried twice more.
  • Drain and the scheduler forward their proposals when they are followers, instead of failing. Every scheduler op carries an epoch, so stale ops are still refused.

Known and not fixed here:

  • Drain's TransferPrimary proposal ignores ok, as before.
  • coordinator_fill_matches_region_fill and series_cardinality_spans_regions flake on "forward batch to node 2 ... timed out"; this reproduces on the base.

Gate: fmt, clippy -D warnings, lib 885, integration 373, CLI 54, all passing.

…seam

domain::cluster::peer_rpc::PeerRpcError keeps the peer's HTTP status apart
from an unreachable peer (callers act on 404/405/409/429/503).
ports::peer_rpc::PeerRpc is the port; adapters::cluster::http_peer_rpc
implements it on PeerClient's pooled clients. The HttpTransport seam in
adapters/cluster/transport.rs, never wired up, is deleted.
MvHandoffRequest moves to domain::sharding::types.
ShardRoutingContext carries the PeerRpc port and the cluster membership
next to PeerClient, which the following commits stop using.
PeerRpc gains shard_bootstrap, shard_write, shard_write_batch,
shard_query (the Arrow reply as a byte stream, still decoded chunk by
chunk), shard_metadata and shard_delete. The routing, query, metadata
scatter and forward-coalescer call sites use them and keep their status
handling (409 stale epoch, 429/503 capacity, 400/422 refused, 404/405
older peer) by matching PeerRpcError. error_chain moves to crate::error
so the adapter can use it.
@cursor

cursor Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

PR Summary

High Risk
Large refactor across peer RPC, Raft leader forwarding, and sharding/drain paths; incorrect error mapping or consensus forwarding could affect cluster availability and mutation correctness.

Overview
Introduces a PeerRpc port with an HttpPeerRpc adapter so application code no longer builds reqwest calls against PeerClient. Shard routing, forwards, queries (streaming Arrow IPC), transfers, heartbeats, probes, and related internal endpoints route through typed methods; PeerRpcError preserves HTTP status vs unreachable so existing handling of 409, 429/503, 404/405, etc. stays the same.

Adds ConsensusPort and RaftConsensus so schema and shard-map mutations go through propose (local commit or leader forward with up to 3 retries). Drain, ShardScheduler, PeerQueryService, and leader callbacks depend on consensus instead of holding HyperbytedbRaft / ad-hoc client-write HTTP. ShardRoutingContext drops peer_client in favor of peer_rpc plus membership; Peers bundles RPC with membership for transfer paths.

PeerHttp moves to adapters/cluster; the unused transport module is removed. Followers now forward drain/scheduler proposals instead of failing when not leader (stale ops still refused via epochs). A leader’s explicit refusal on a forwarded schema mutation is returned to the caller rather than retried further.

Reviewed by Cursor Bugbot for commit f5260d0. Configure here.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit f5260d0. Configure here.

};
raft.client_write(ClusterRequest::ShardMapMutation(Box::new(tp)))
consensus
.propose(ClusterRequest::ShardMapMutation(Box::new(tp)))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Drain ignores refused primary transfer

High Severity

Drain now calls propose for TransferPrimary, which returns a leader refusal as Ok with ok: false. Only transport errors are checked, so a stale-epoch (or other) refusal is treated as success and drain continues. On a follower this is new: the old local client_write failed instead of returning a response, so a refused handoff can finish drain while this node is still primary.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit f5260d0. Configure here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant