monad-failover runs as root on a Monad full node during a validator migration.
It is a single static binary built from this repository with no third-party
modules; every import is from the Go standard library. Concretely, it:
- reads
KEYSTORE_PASSWORDfrom/home/monad/.env(by parsing the one line, not by executing the file) and reads yoursecp-backup/bls-backupfiles - writes only under
/home/monad/monad-bft/configand its own root-only directory/var/lib/monad-failover(resume state, staging, the identity backup, the re-exported key backups and the run logs) - manages only the
monad-bft,monad-executionandmonad-rpcsystemd units - runs
monad-keystore,monad-sign-name-record,monad-statusandsystemctlwith argument vectors, never through a shell - makes four kinds of outbound requests:
ifconfig.me/ip, thenapi.ipify.org, thenicanhazip.com(HTTPS, the first valid answer wins) to detect the server's public IPv4 (bypassed with--public-ip); Monad Foundation's validator snapshot (bucket.monadinfra.com/validator-data/<network>.json, HTTPS) to read the last published name record sequence for your key, which is used only to suggest a number you can override; whenmonad-statusis not installed, JSON-RPC calls (eth_chainId,eth_blockNumber,eth_syncing) to this node's own RPC on127.0.0.1:8080and to the Foundation's public RPCs of the network (rpc.monad.xyz,rpc-mainnet.monadinfra.com,testnet-rpc.monad.xyz,rpc-testnet.monadinfra.com, HTTPS) to judge sync; and the monval uptime API (validator-api.huginn.tech, operated by Huginn) after cutover for a historical view of the validator identity. The lookup includes its public SECP key in the URL. As with any HTTPS request, the endpoint also sees the caller's IP. Snapshot and uptime failures do not block migration; failed IP detection requires an explicit--public-ipbefore signing can continue - reads
/proc/net/tcpand/proc/net/tcp6in the dry run to report RPC ports listening on non-loopback addresses
It contains no telemetry and never transmits your keys or password anywhere.
The process runs with umask 077, so every file it creates (key backups,
resume state, staging, logs) starts private, and the keystore
password is never written alongside the encrypted keystores.
secp-backup and bls-backup contain unencrypted secret IKM, both when you
provide them and when the tool exports fresh copies after cutover. Anyone with
those files can reconstruct the validator keys without the keystore password.
Their private file permissions are not encryption. Store off-server copies in
an encrypted vault and restrict access to copies left on the node. The separate
failover-<timestamp>/ directory holds this target's original encrypted
keystores and config; it is not a backup of the incoming validator identity.
The Monad key tools take the password and IKM as command-line flags
(monad-keystore --password ... --ikm ...). While those child processes run, their
arguments are visible in /proc/<pid>/cmdline to any local user. This is a property
of the Monad CLI, not something this tool can avoid. On a validator you should
already treat local access as full compromise, but if you want defence in depth,
mount /proc with hidepid=2 so process arguments are not readable across users.
The tool itself never echoes or logs the password or IKM: manual IKM entry is read with terminal echo off, and the output of every key-tool invocation that carries a secret on its command line is discarded or parsed, never written to the run log, so even an error path that echoed its arguments could not land there.
The keystore password and the IKM pass through this process. What the tool does about that inside its own memory:
- Both are held as byte buffers and are turned into strings only for the
instant a child process's argument vector is built. The byte buffers this
code manages are zeroed after use: the
.envcontents, the backup file contents, a typed IKM, the parsed values, and the output ofmonad-keystore recover, which carries the secret alongside the public key. Terminal input is read through one fixed buffer this code owns, and a consumed line is zeroed out of it at once. The password buffer is zeroed when the run ends. - After every command that carried a secret on its argument vector, the garbage collector runs and free memory is returned to the kernel.
- Core dumps are disabled and the process is made non-dumpable; as root, memory locking is attempted so nothing pages out to swap. Each of these is best effort: a failure is printed before the run starts and the run continues.
What it cannot do: the copy Go makes of each argument for execve belongs to
the runtime. It is collected, not zeroed, and can stay in a reused page until
something overwrites it. A memory image of the process taken while a key
command runs, or shortly after, can contain the values. Anyone able to take
such an image is root, and root already holds the password in .env and the
IKM in the backup files; the measures above reduce exposure through core
dumps and swap and are not an unconditional guarantee. A VM snapshot is
outside their reach.
A resume run reads its saved state back and acts on it while running as root: the state names the sequence number to sign, the IP to publish, and the directory a failed step tells the operator to restore from. Whoever can write that file can steer the run, so the file must not be writable by anyone but root.
For that reason the state lives in /var/lib/monad-failover, owned root:root
and mode 0700, with the state file itself 0600. It is not under
/home/monad, which the unprivileged monad service account owns. The install
instructions place the binary in root-owned /usr/local/bin on the same
grounds. On startup the tool:
- refuses to run if the state directory is not owned by root, or if the
directory or the state file is a symlink, or the state file is not a regular
file (an unprivileged user could otherwise pre-stage a symlink to redirect a
root write); an existing directory whose mode is not exactly
0700is refused, never repaired and then trusted; - writes state atomically through a uniquely named temporary file created
inside that directory (
os.CreateTemp,O_EXCL), fsynced and renamed over the old file, never a predictable name; - validates every field it reads back against a narrow shape before use: the
step counter is a closed
1–8range, not just digits, because an out-of-range value would satisfy every completed-step check and skip the whole migration to a fake completion; a key that appears twice in the file is refused rather than read as its first line; - refuses, rather than silently migrating, any state left by an older version
under
/home/monad/.monad-failover: that path is writable by themonadaccount, so it is treated as untrusted and left in place for you to inspect; - refuses the test-only overrides
MF_STATE_DIR,MF_IP_URLandMF_UPTIME_API_BASEin a live (root) run instead of ignoring them, so a stray variable can never be assumed honoured.
Do not remove resume state just to bypass a refusal. If it cannot safely be resumed, follow manual recovery before starting a fresh run.
The three files that make up this node's identity (id-secp, id-bls and
node.toml) cannot be swapped in one atomic step, so the run is built so that
every point it can be interrupted is recoverable:
- Staging lives in
/var/lib/monad-failover/staging, owned by root and mode0700, not in the config directory. That directory belongs to themonadaccount, so anything staged there could be replaced between the checksum check and the rename. - Each staged file is checksummed when it is created; the plan you confirm at cutover covers exactly that content. The recorded value is never refreshed from disk, so a file changed after your confirmation is refused rather than accepted as the new expected content. Symlinks and non-regular files are refused.
- Placement copies the verified content into the destination directory, checks
it again there, and only then renames it, so the final step is a rename
within one filesystem and cannot be interrupted half-written. A move from the
staging filesystem straight to the config filesystem would be a copy, not an
atomic rename. The temporary file is created and written in a single open
with
O_CREAT|O_EXCL|O_WRONLYat mode0600: creating it and then reopening it by name would leave a window in which the name could be replaced with a symlink and the write would follow it, and no checksum afterwards can undo a write to the wrong file. The file is fsynced before the rename. The live file is checked once more after the rename. - One boundary is not fully closed and is worth stating plainly: the
destination directory belongs to the
monadaccount, so the rename target is a path that account can manipulate. The single-open write removes the write-through-a-symlink hazard, and the check after the rename means a substitution is detected and the run stops before the services start, but a rename by file descriptor is not available, so detection rather than prevention is the guarantee for that last step. Operators who want the hazard gone entirely can make the config directory itself root-owned, with themonadaccount holding read and execute only. - The units are masked before the swap and unmasked only once every file is in
place. A plain stop is not enough: the units are normally enabled, so a reboot
between two renames would otherwise bring the node up with a mixed identity.
The mask is persistent, because a
--runtimemask does not survive a reboot, and the run verifies the mask actually took effect before touching anything. What was already masked is recorded before the first mask is applied, so an interruption in between cannot make a resume read this run's own masks as yours. Units you had masked yourself are left masked, and not started. - Placing the files and bringing the services up are two separately resumable stages. An interruption after the swap resumes into the bring-up, so the units cannot be left masked with nothing to unmask them. An unmask that does not take effect is never recorded as done.
- Before the units are unmasked and started, all three live files are checked against the recorded checksums again. The swap may have happened in an earlier run, and anything that changed in between must not be started: the run stops and points at the backup instead.
- The whole live run holds an exclusive
flock. A second run is refused before it reads or writes any state. - Once a cutover has begun that fact is recorded, and a later run without
--resumerefuses to start fresh over an unfinished swap.
An interrupt (Ctrl-C or SIGTERM) restores the terminal, prints the resume
command and exits. Nothing is undone: every step records its progress before
it acts, so the run continues from that record with --resume.
--resume reads the recorded stage and continues from it: before the mask,
between renames, after the swap but before the services start, and during final
verification.
The sequence suggestion comes from Monad Foundation's published validator data. The document is decoded as a whole with the standard JSON decoder; a truncated response, or one with data after the document, is refused before any of it is used. The snapshot's network and chain id must be the ones this node is on and its timestamp must be recent. The entry must match your SECP key exactly and be unique, and its BLS key must match the key you imported. A validator with no published record is reported as unknown, never as sequence zero, and a sequence outside the range a JSON number carries intact is refused. Anything that fails these checks falls back to entering the number yourself, with the reason shown.
After the swap, consensus and execution must report active. RPC must also be active unless it was already masked before this run; that operator choice is preserved. Missing required services fail with recovery steps. Sync is a separate question and gets a bounded window of its own. If the node has not caught up in that window the run reports the cutover as complete with verification pending, keeps the resume state, and does not print success. The monval uptime figure is a 24 hour window keyed on the public key. It does not prove that this new server is currently participating. Missing fields or an inactive result are advisory and do not prevent key backup export.
- Compare
sha256sum /usr/local/bin/monad-failoveragainst the checksum in the README. The build is reproducible: with Go 1.24.7 on linux/amd64, the source install in the README yields the same bytes. CI builds the release the same way and fails any change where the README checksum and the build drift apart, and the release workflow refuses to publish a binary whose checksum differs from the tagged README. - Run
monad-failover --dry-runfirst. It performs every preflight check read-only and changes nothing. - The code is about 3,700 lines under
cmd/andinternal/. The parts that touch secrets, root-owned state and the live node areinternal/state(resume record),internal/place(checksummed placement),internal/monad(the key tools and signer),internal/netinfo,internal/foundationandinternal/uptime(the three network requests) andinternal/promote(the flow).go vet ./...andgo test ./...run without root, network or systemd.
Report vulnerabilities privately via GitHub security advisories rather than public issues. Reports are acknowledged on a best-effort basis, and key-handling issues are treated as top priority.
Before a secret file is created, the backup and log directories are checked:
no symlink at any step of the path, every directory owned by root (or by the
user running the tool), and no ancestor writable by group or others. A
root-owned sticky directory such as /tmp is accepted as an ancestor, never
as the directory itself. A directory that passes is set to 0700. Key
exports are written through uniquely named O_EXCL temporary files and
renamed into place; no predictable temporary path is ever opened. Earlier
exports are kept under unique .bak names, never overwritten.
Preparation changes nothing about the live config directory, .env or the
key files, not even ownership or mode. Placement sets the three destination
files to the monad account and mode 0600 through the open file descriptor
before the rename, and the same ownership and mode are checked again before
the services are unmasked, on resume as well. The surrounding config
directory keeps whatever owner it had.