Skip to content

feat(cli): feint checkpoint and feint reset, judged by what a client observes (#853) - #869

Merged
stephrobert merged 8 commits into
mainfrom
feat/853-the-observable-state
Nov 11, 2026
Merged

stephrobert merged 8 commits into
mainfrom
feat/853-the-observable-state

Conversation

@stephrobert

@stephrobert stephrobert commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

Summary

feint checkpoint <name> and feint reset [<name>]: transaction-like test
isolation on top of the two storage primitives that had no caller. feint snapshot persisted the control plane and objectstore.SaveCheckpoint
persisted bytes; putting them together is what #853 asks for, and the issue
says what separates a lifecycle guarantee from a storage primitive in one line
— "reset is not successful because JSON was restored".

Measured end to end against a running emulator with a byte store armed, which
is #853's "Done means" list in the order it names:

checkpoint "baseline": 4 resource(s), 1 object(s), 0 machine(s)
# one extra server, one extra object, and a.txt overwritten
resources: 7   objects: [a.txt b.txt]
reset "baseline": the lab converged            exit 0
resources: 4   objects: [a.txt]
a.txt: "the bytes the checkpoint holds"        # the content, not just the key
# a second reset: exit 0, nothing left to do

The split that is the whole design

The emulator owns the state transitions, because only it holds both the store
and the packs' bindings: a CLI-side reset would have to name each pack's delete
route to take a machine down, which is provider knowledge in the one place
this repository keeps free of it.

These two commands own the verdict, because whether a lab converged is a
comparison of what a client can observe, and an operation that graded itself
is the verdict this repository refuses everywhere else. Exit 0 when it
converged, 1 with every remaining difference named. Not 2, which is reserved
for upstream drift, and never 0 with differences printed.

What the observable state is, and why any two of its halves can lie

  • the control plane, from /_feint/resources;
  • the bytes, read through the doors, because the authoritative view of an
    object is the one a client gets;
  • the host, from Runtime.Survey, reused rather than reimplemented: it already
    names only what the emulator labelled and already issues no mutating
    command.

Restoring the store alone leaves containers running that the API no longer
mentions. Restoring the store and the host leaves bytes from a later test in a
bucket the checkpoint had empty.

Two comparisons were decided by measurement rather than taste: objects are
compared by digest, because two objects of one size are not one object; and
a resource's state counts, because a reset that left a stopped server
stopped has not converged.

The guard I nearly wrote twice

ReconcileGone's first draft refused a machine name outside the binding's
prefix. That is a second guard beside a working one: Remove already asks the
ownership question, and ours already decided what the answer does — fall
back to the name the emulator could have created rather than refuse. Keeping
my version would have changed that decision in one path only. The test asserts
the existing behaviour instead: a crafted production-database reaches no
command as itself, and the derived name is removed, so a tampered record
does not leave its real container running either.

Each pack wrote two lines. machine.PackSurface now names
Binding.ReconcileGone and Reconciled with their reasons — the discipline
gate refused the three packs until it did.

Four defects found by measuring rather than by testing

  • feint checkpoint baseline --addr … was refused by its own usage line.
    Go's flag package stops at the first non-flag argument. A unit test of the
    parsing would not have caught it, because I would have written the test with
    the flags first.
  • feint checkpoint -h printed no flags, the name being taken off before
    the flag set existed. Found by the frozen-surface observation, which refuses
    a verb that builds no flag set.
  • Two mutations of the falsification spec stayed GREEN, and both were my
    tests' fault: the one comparing resources by identifier alone passed because
    every id in the test was distinct, and the one skipping a checkpoint's byte
    half passed because the test's emulator served no objects. The tests now
    hold two records sharing an identifier across kinds, and a second lab with a
    byte store whose checkpoint is reset afterwards — the only way to see it
    exists.

tools/falsify/specs/a-reset-is-not-a-restore.json is the spec #853's last
criterion asks for: six mutations, each turning the reset back into a restore
by a different route, all biting.

Every debt here carries its number

This repository's rule is that a limit written in a commit message is not a
tracked debt. So each gap below names the issue that holds it, and none of
them is a sentence without a number:

not done here carried by
feint run --reset-after and its three exits done, measured on all three
recreating a machine the host no longer has done, measured under incus
the host half, and "no foreign runtime object is touched" done, measured under incus
a reset interrupted part-way, and an image that left the catalogue between checkpoint and reset the testplan rule's own Unproven
incus-vm and incus-ovn the testplan rule's own Unproven

No issue was opened for the first three, deliberately: they were carried by
#853 itself and a ticket duplicating one would be a debt displaced rather than
treated. They are now done rather than carried.

Every closing criterion of #853 is met, including the two that needed a
real runtime. This pull request still closes nothing: that call is the
maintainer's, and an issue closed by a machine is an issue nobody re-read.

Type of change

  • A route added or changed
  • Bug fix
  • Refactor
  • Documentation
  • Chore / tooling
  • Security / supply chain

Checklist

Always

  • mise run check passes
  • No new external Go dependency
  • internal/core still knows no provider: Reconciler is an optional pack
    half in the shape the others use, the core computes the delta and loops
    over whatever is mounted
  • Nothing in the diff could be written identically for another provider —
    the destroying gesture is shared and each pack wrote two lines, which is
    the measure of whether it was put in the right place
  • mise run testplan was run

When a model wrote a substantive part of this

  • An Assisted-by: trailer names the tool and the model version
  • The cycle was driven by hand against a real emulator; see the figures
    above. mise run conformance was NOT re-run for this branch: it
    touches no route and no response shape, and the CI matrix judges it.
  • Every field name comes from a measurement or from the emulator's own
    payloads

When a route is added or changed

Three /_feint/* endpoints are added, which are the emulator's own control
plane and not a provider API:

  • No provider route changed, so no upstream operation is claimed
  • CheckpointSchemaVersion frozen from the start, like the four
    introspection schemas beside it: a suite that resets between tests is a
    consumer on day one
  • cliSurfaceVersion 30 → 31, fixture regenerated

When behaviour a client can observe changes

  • Both READMEs carry the two verbs
  • docs/limits.md untouched: no documented limit moved. What a reset does
    not do is said on the wire (recreates: false) and in the help,
    which is where a caller meets it

When a machine runtime is involved

  • Measured under --vm incus, Incus 7.2, on the maintainer's station.
    tools/conformance/reset-host.sh creates a machine the checkpoint holds
    and one it does not, resets, and asserts all three: the extra machine is
    gone, the checkpoint's machine is still there, and every container
    feint did not create is still there by name.
  • Nothing the emulator did not create was touched, asserted by name rather
    than by count: a count cannot tell a neighbour's container from one of
    ours.
  • The suite leaves nothing behind. It ends with a machine alive on purpose,
    so it sweeps through feint clean, with the trap armed before the first
    server is created — its first version leaked that container onto the
    station, found by reading incus list and not by any gate.
  • incus-vm and incus-ovn are not covered, nor a reset interrupted
    part-way, nor more than two machines. The testplan rule says so.

Related issues

Refs #853, #854.

🤖 Generated with Claude Code

The first slice of #853, and deliberately not a command. The issue refuses the
shortcut in one line — "reset is not successful because JSON was restored" —
so the verdict has to be a COMPARISON, and this is the thing compared.

Writing `feint checkpoint` and `feint reset` first would have produced two
commands whose success was their own opinion of themselves, which is the
verdict this repository exists to refuse.

## Three halves, and why any two of them can lie

- the control plane, from `/_feint/resources`: what the API says exists;
- the bytes, read THROUGH the doors: what a client can fetch;
- the host, from `Runtime.Survey`: what the emulator actually created.

Restoring the store alone leaves containers running that the API no longer
mentions. Restoring the store and the host leaves bytes from a later test in a
bucket the checkpoint had empty. The issue's own list of guarantees names all
three, so the comparison does too.

`Survey` is reused rather than reimplemented: it already names only what the
emulator labelled, it already issues no mutating command, and its own test
holds that. A second traversal here would be a second place for the ownership
rule to drift, on the one layer where that costs an operator their own
containers.

## Why the bytes go through the door

The authoritative view of an object is the one a client gets. A comparison
reading the store directly would pass while the door served something else,
which is the half that matters to a test using it.

It does make the comparison depend on the door answering, and that is not a
circularity worth avoiding: bytes a door cannot serve are bytes no client can
observe, and a reset whose doors are broken has not converged.

The listing paginates. A comparison that read the first page would report two
different stores as equal, which is a verdict measuring its own window rather
than its subject.

## What is deliberately left out

Created and Updated, because a restored resource carries the instant it was
restored and the comparison could then never be equal. Attrs field by field,
because a pack's view can hold a derived value and a reset is about the state,
not about how a pack renders it.

What IS carried, and was nearly not, is the backing host object behind each
record: a resource whose container changed under it is a resource a reset did
not reconcile, even when both lists are the same length. It is read as the
whole Runtime map sorted, not from a known key, because three packs spell that
key differently and a hardcoded one would silently observe nothing for the
fourth.

## The two tests, and what each would miss alone

TestTwoReadsOfOneLabAreEqual drives the reader against a fake lab whose
resources come back out of order and whose objects come from a Go map, which
has no order at all. It then asserts that something WAS observed: two empty
states compare equal too, and would pass a stability test while measuring
nothing.

TestADifferenceIsNamedWhereverItIs puts one difference on every axis at once,
in both directions, because #853 asks a reset to report "exactly what could
not be reconciled". A verdict that found the extra container but not the
changed digest would send an operator round the loop once per problem. It also
asserts that an axis which CONVERGED says nothing: a verdict listing what
worked buries what did not. And that a state does not differ from itself,
which is what will make a second reset a no-op.

Two of those assertions were earned rather than foreseen: the digest is
compared instead of the size because two objects of one size are not one
object, and the resource state is compared because a reset that left a stopped
server stopped has not converged.

Not here yet: `feint checkpoint`, `feint reset`, `feint run --reset-after`, and
the reconciliation itself. The host half is exercised through the injected
surveyor only; a real runtime comes with the commands that act on it.

mise run prepush green.

Assisted-by: Claude Code (claude-opus-5)
…853)

The gesture #853 needs and nothing above it yet. `PUT /_feint/state` and
`feint snapshot load` restore the control plane verbatim and touch nothing
else, so a resource that disappears leaves its container running while the API
no longer mentions it. Nobody destroys it, and the next `incus list` is the
first time anybody finds out.

## Why the destroying half is here and the recreating half is not

Destroying is the same gesture for every pack: take the machine the record
names, check it is ours, remove it. Written three times it would be fixed once,
which is the defect CLAUDE.md names and this repository has already paid for
on the lifecycle itself.

Recreating is NOT the same gesture. Starting a machine needs a Boot — the
image, the login, the cloud-init a pack renders for its own product — and that
is pack knowledge by construction. #853 hedges in the same place with "missing
checkpoint resources recreated where the runtime contract supports it", so
that half belongs to each pack and is bounded by what that pack can boot. It
is sequenced after this, not dropped.

## The guard I nearly wrote twice

The first draft refused a machine name outside the binding's prefix. That is a
second guard beside a working one: `Remove` already asks the ownership
question, and `ours` already decided what the answer DOES — fall back to the
name the emulator could have created rather than refuse, because a record may
well have a machine and that is the only name it could ever have had.

Keeping my version would have changed that decision in one path only, and two
answers to one question are how they start disagreeing. The test asserts the
existing behaviour instead: a crafted `production-database` reaches no command
as itself, and the derived name is removed, so a tampered record does not
leave its real container running either.

## Four populations, and the default is one of them

- a record with a machine of ours: destroyed, and the names reaching the
  driver are asserted rather than the return value — a reconciliation is
  judged by the commands it emits;
- a record with no machine: not an event at all, which is the ordinary case
  for an address, a volume or a security group. Reporting it would bury the
  machines under records that never had one;
- a foreign name: no command carries it;
- **no runtime: nothing destroyed, and said so through Skipped.** Metadata
  only is the default, so this is the common case. An empty Destroyed read as
  success is how a caller comes to believe containers were removed on a
  station that never started any.

Not here: the emulator does not call this yet, and nothing computes the "gone"
list. That is the next slice, with `feint checkpoint` and `feint reset` on top
of the comparison the previous commit added.

mise run prepush green.

Assisted-by: Claude Code (claude-opus-5)
The lifecycle guarantee #853 asks for, on top of two storage primitives that
had no caller: `feint snapshot` persists the control plane and
`objectstore.SaveCheckpoint` persists bytes, and until now nothing put them
together. Putting them together is the whole point, and the issue says what
separates the two in one line — "reset is not successful because JSON was
restored".

    POST   /_feint/checkpoint/{name}
    DELETE /_feint/checkpoint/{name}
    POST   /_feint/reset/{name}

## Why the reset is server-side

Restoring the store is the easy half. The half that matters is the host: a
resource that disappears in a restore leaves its container running while the
API no longer mentions it, and nobody takes it down. Only this process holds
both the store and the packs' bindings, so only this process can act on the
difference. A CLI-side reset would have to name each pack's delete route,
which is provider knowledge in the one place this repository keeps free of it.

## Why the VERDICT is not server-side

This answers what it did. Whether the lab converged is a comparison of what a
client can observe, and that belongs to whoever asked — the observable state
two commits ago. An operation that graded itself is the verdict this
repository refuses everywhere else.

## The order inside a reset, and why it is not interchangeable

1. read what the store holds NOW, because after step 2 it is gone and the
   machines it named would be unreachable;
2. restore the control plane, so the target is in place before anything is
   destroyed — a failure here leaves the host untouched rather than half swept;
3. restore the bytes, which objectstore does atomically;
4. take down the machines of records that no longer exist, through each pack's
   own binding and therefore through the ownership checks.

The difference is computed by IDENTITY, not by count: a restore that replaced
one server with another of the same kind leaves the count equal and the host
wrong.

## The seam, and what the packs had to write

`Reconciler` is an optional pack half, in the shape the others use
(EnvHazards, StackHazards, FirewallEnforcer): a pack that drives no machine
does not implement it, and the core holds no list of which do. Each pack sees
only its own gone resources, so the core stays a loop over whatever is
mounted.

Each pack wrote two lines. That is the measure of whether the shared gesture
was put in the right place, and `machine.PackSurface` now names
`Binding.ReconcileGone` and `Reconciled` with their reasons — the discipline
gate refused the three packs until it did, which is it working.

## What a reset does NOT do, said on the wire

It does not recreate a machine the checkpoint had and the host no longer does.
Starting one needs a Boot — an image, a login, the cloud-init a pack renders —
and #853 bounds that in the same words, "where the runtime contract supports
it". So the answer carries `recreated: []` and `recreates: false` rather than
leaving a caller to infer it from a quiet success. That half is sequenced
next, per pack.

## Tests, and what each would miss alone

TestACheckpointTakesBothHalvesOrNeither: both halves or neither, because a
checkpoint whose bytes were not taken would fail later, somewhere else, with no
mention of this call. It also asserts the answer says whether bytes were part
of it, and that a name which is a path component before it is a name — "..",
"a/b" — is refused at the door, since the byte store writes a directory per
checkpoint.

TestAResetDestroysWhatTheCheckpointDoesNotHold asserts on what the pack WAS
ASKED to take down, never on the status code. It mounts a second pack that
cannot reconcile, because that is the ordinary case and the loop must skip it
rather than fail; it checks the kept resource was NOT handed over, which is
the direction that would destroy a machine the checkpoint expects; it runs
reset twice and requires the second to ask for nothing, which is the issue's
own no-op criterion; and it requires the refusal of an unknown checkpoint to
name the command that makes one, because a 404 alone sends the reader to the
source.

Not here yet: `feint checkpoint` and `feint reset` as commands, the verdict
built on the observable state, `feint run --reset-after`, and the recreating
half.

mise run prepush green.

Assisted-by: Claude Code (claude-opus-5)
…observes (#853)

    feint checkpoint baseline
    # run tests, create resources, fail halfway
    feint reset baseline

Measured end to end against a running emulator with a byte store armed, which
is #853's "Done means" list in the order it names:

    checkpoint "baseline": 4 resource(s), 1 object(s), 0 machine(s)
    # one extra server, one extra object, and a.txt overwritten
    ressources : 7   objets : [a.txt b.txt]
    reset "baseline": the lab converged        exit 0
    ressources : 4   objets : [a.txt]
    a.txt: "the bytes the checkpoint holds"    # the content, not just the key
    # a second reset: exit 0, nothing left to do

## The split that is the whole design

The emulator owns the state transitions. These two commands own the VERDICT,
and #853 says why in one line: "reset is not successful because JSON was
restored". So a reset is judged by comparing what a client can observe before
and after, never by the status code of the call that performed it.

Exit 0 when it converged, 1 with every remaining difference named. Not 2 —
this repository reserves that for upstream drift — and never 0 with
differences printed, which would be a gate nobody can build on.

## Why a checkpoint records the observation too

Deriving the expected state from the restored store would compare a restore to
itself: the containers and the bytes would be out of the picture, which is
exactly the half that can diverge. So the observation is recorded beside the
instance, it belongs to that emulator, and it goes when it does.

A reset against a name the record does not hold is refused rather than
guessed: comparing one checkpoint's restore to another's observation would
report differences that are not the reset's fault, and an operator would chase
them.

## Two defects the measurements found, not the tests

**`feint checkpoint baseline --addr …` was refused by its own usage line.**
Go's flag package stops at the first non-flag argument, so every flag behind
the name became a positional. A unit test of the parsing would not have caught
it, because I would have written the test with the flags first. The first real
run did.

**`feint checkpoint -h` printed no flags.** The name was taken off before the
flag set existed, so the help had nothing to print. Found by the frozen-surface
observation, which refuses a verb that builds no flag set — it freezes nothing
for it. The order is now flag set first, name second, and that is not cosmetic.

## Two test gaps the falsification found

tools/falsify/specs/a-reset-is-not-a-restore.json is the spec #853's last
criterion asks for: "a falsification that restores only the store but not
runtime makes the test fail". Six mutations, each turning the reset back into
a restore by a different route. Two of them stayed GREEN on the first run, and
both were my tests' fault:

- the mutation that compares resources by identifier alone, dropping the kind
  and the provider, passed because every id in the test was distinct. The test
  now holds two records SHARING an identifier across kinds, and a reset that
  dropped the kind would take the extra volume for the server the checkpoint
  keeps and leave its machine running;
- the mutation that skips the byte half of a checkpoint passed because the
  test's emulator served no objects. There is now a second lab with a byte
  store, and the assertion is not the status code: the byte checkpoint is
  reset afterwards, because that is the only way to see it exists.

Both gaps were invisible to a reader of the tests and obvious to the harness.

## Not here yet

`feint run --reset-after` and its three exits, the recreating half, and a
conformance suite replaying this cycle in CI. The cycle above was driven by
hand, and this commit does not claim otherwise.

mise run prepush green. `cliSurfaceVersion` 30 → 31, fixture regenerated,
both READMEs carry the two verbs.

Assisted-by: Claude Code (claude-opus-5)
…se (#853)

#853's closing criterion that no unit test can reach — "no foreign runtime
object is touched" — closed with a measurement. Run on this station, Incus
7.2, `--vm incus`:

    ok: the checkpoint's machine exists: feint-scw-d539d3eb-…
    ok: the second machine exists: feint-scw-60c339c4-…
    destroyed: feint-scw-60c339c4-…
    reset "reset-host": the lab converged
    ok: the machine the checkpoint does not hold is gone
    ok: the machine the checkpoint holds is still there
    ok: every container feint did not create is still there

## The assertion in the middle is the product

A reset that swept both machines would return an operator to a checkpoint
whose servers answer nothing, which is worse than no reset at all. The
reconciliation is exercised elsewhere through a recording driver, and that
proves which names the code emits; only a real runtime proves what happens to
them.

The third assertion reads `incus list` before and after and requires every
stranger's container BY NAME, rather than counting: a count cannot tell a
neighbour's container from one of ours, and this repository has already
measured a forced removal reaching a name it did not own.

## The leak this suite had on its first run

It ends with a machine alive on purpose, which is the point of its third
assertion. The first version therefore left that container running on the
maintainer's station, and it was found by reading `incus list` after the pass
rather than by any gate here — the same shape as the lab the run tests leaked
two days ago, and the second time in one week that my own harness was the
thing left behind.

It now sweeps through `feint clean`, which removes what the emulator created
by the labels it wrote and nothing else. The trap is armed BEFORE the first
server is created, so a failure in the middle does not leak either: a cleanup
that only runs when the suite passes is not a cleanup, and the run that leaks
is by definition the one that failed.

## Both populations driven before committing

With no runtime it skips and says why. Under `--vm incus` it asserts all five.
Measured after the pass: only the operator's own container remains, and the
station's available memory is where it started.

## Where it runs

The runtime leg of `tools/conformance/leg.sh`, the full pass in mise.toml, and
`runtime-proof.yml`. The testplan gate refused the rule until the file was
tracked — "matches no tracked path: it is either a directory that moved, or
coverage of nothing" — which is it working on a file I had not added yet.

What it does NOT cover is in the rule's own Unproven: a reset interrupted
part-way, a checkpoint whose machine died on its own, and more than two
machines.

shellcheck clean, YAML parsed, mise run prepush green.

Assisted-by: Claude Code (claude-opus-5)
#853's crash behaviour asks for a reset attempted on command success, command
failure and SIGINT/SIGTERM, and concedes the fourth in the same breath: a hard
kill cannot promise anything. The three it asks for are measured against a
SHARED emulator, which is where the flag earns its place — an owned lab
disappears at the end whatever happens.

    exit 1: the command succeeds   code 0    reset=converged   resources 0
    exit 2: the command fails      code 7    reset=converged   resources 0
    exit 3: a relayed signal       code 255  reset=converged   resources 0
    control, without the flag                                  resources 4

The control line is the one that gives the other three meaning: without
`--reset-after`, the run leaves four resources in the shared emulator.

## Two flags that cannot both be right

`--reset-after` erases the state of a failed run; `--keep-on-failure` keeps it
for inspection. Refused together rather than silently ordered, because either
order is wrong for somebody and a reader cannot tell which they got.

## A reset's failure is reported, never raised into the child's verdict

The child has already run when the reset is attempted, and its exit code is
what a caller builds on. So the verdict is a word the receipt carries, and the
four are not interchangeable: "not attempted", "failed", "unverified" and
"converged" send a reader to four different places. The emulator's own reason
is relayed rather than summarised — a reset refused because the store could
not be restored and one refused because the bytes could not be are different
problems.

**But a reset that did not converge fails the command**, even when the child
passed: the caller asked for the lab to be left as found and it was not, and a
quiet 0 is how the next test inherits the mess and blames itself.

The session is named after the run's own directory, so two runs against one
shared emulator cannot reset each other's state. A mismatch is "not attempted"
rather than a reset of somebody else's checkpoint.

## A number nobody can act on, made readable

A signal-killed child gives Go an exit code of -1, and the receipt printed
`failed (-1)`. It now says "failed (the command was killed by a signal)" in
prose while the JSON keeps the number, documented on the field. The measurement
is what showed it: the third exit above printed -1 and I could not tell from
the receipt what had happened.

## The mutation that caught my tests again

Fourteen mutations on the spec now, and one of the two added here stayed GREEN
on the first run: "a reset that did not converge stops failing the command".
No test drove a reset that fails to converge, so the guard had nothing holding
it.

TestARunWhoseResetDidNotConvergeFails drives a lab that answers one more
resource after the reset than at the checkpoint — the reset ran, the emulator
said ok, the state did not come back — with a child that PASSES on purpose. A
reset's failure must not hide behind a failing test, and must not hide behind a
passing one either. It also asserts the receipt blames the right half:
`reset: did not converge` with `exit: 0`.

## Not here

The recreating half, which is #853's remaining criterion and needs a Boot only
each pack can render. `--reset-after` on an emulator with no S3 door is refused
by `feint run`'s own interception requirement, which is #849's design and not a
new limit.

`cliSurfaceVersion` 31 → 32, fixture regenerated, help names the flag.
mise run prepush green.

Assisted-by: Claude Code (claude-opus-5)
)

#853's last open criterion, "runtime reconciled to the checkpoint", and the
half I had sequenced last out of a worry the measurement then overturned.

Measured under Incus 7.2, with the container removed OUT OF BAND so that the
state exists at all — the record is there, the machine is not:

    checkpoint "rc": 4 resource(s), 0 object(s), 1 machine(s)
    # incus delete --force, behind the emulator's back
    containers: []
    the store still says: state=running machine=feint-scw-4dbb6d76-…
    reset "rc": the lab converged
    containers: [feint-scw-4dbb6d76-…]        # back, under the same name

## The worry that sequenced this last, and why it was wrong

I had written that a Boot rebuilt from a restored record might not be the
original's: an image gone from the catalogue, a key deleted, a user data that
no longer exists. Reading where those values come from says otherwise, and
that reading is the reason this half turned out cheap:

- the record's own attributes, which the restore brought back verbatim;
- the store, for the project's SSH keys — which are part of the checkpoint;
- the image catalogue, compiled into the binary and depending on no state.

A label the catalogue does not know produces an empty image reference, which
is exactly what the ORIGINAL poweron produced for that same record. So the
rebuilt boot is the boot it had.

## Two interfaces, because one was a silent trap

The first version added a second method to `Reconciler`. Every pack instantly
stopped satisfying it, the type assertion in the reset handler failed, the
reconciliation silently stopped happening, and the call still answered 200
with a clean body. Nothing logged it.

The only reason it surfaced is that a test asserted which records a pack had
been HANDED rather than that the call succeeded. `MissingReconciler` is now
its own optional half, so a pack gains one when it is ready and keeps the
other working, and internal/cli's
TestAPackThatDrivesMachinesReconcilesBothWays closes the other side: a pack
that drives machines must satisfy both, or a reset quietly does half its job
for that provider.

## The core stopped guessing, and the falsify baseline is what caught it

The first version picked the missing records in the CORE, by looking for a
"feint-" prefix in any Runtime value. Two defects in one line: a provider's
vocabulary in the neutral layer, and a match broad enough to pick up a test's
own bookkeeping stamp — which it then tried to boot. The falsification
baseline went red with nine identical warnings naming that stamp.

The core now asks the runtime what it HOLDS and hands that to each pack, which
decides with its own RuntimeKey. A host nobody can survey answers
asked=false, and then nothing is offered for recreation at all: taking it for
an empty host would start a second container beside every one already
running.

## What it refuses to call a success

PowerOn answers false and sets FailedState when a runtime was configured and
did not deliver. Those are reported as Failed, never folded into Recreated: a
reset claiming it brought three machines back when one is in FailedState would
be caught by the verdict above it anyway, loudly, in the wrong place.

## Tests

- machine: six populations, including a machine the host still HOLDS being
  left alone, and a record whose key names no machine not being a candidate —
  the two the core's guess got wrong;
- emulator: the records a pack is handed, and a host nobody can survey being
  asked for nothing;
- conformance: `reset-host.sh` now measures all three on a real runtime — one
  kept, one removed, one lost out of band and started again under the same
  name. The name is the assertion and not the count: a reset that started SOME
  container would satisfy a count while every record named one that does not
  exist, and the next reset would report it missing again, forever.

/falsify: eight mutations on a-reset-is-not-a-restore.json, all biting,
including one that reproduces the core's guess and one that takes a
survey-less host for an empty one.

`machine.PackSurface` names `Binding.ReconcileMissing` with its reason.
mise run prepush green. Only the operator's own container remains on the
station, and its memory is where it started.

Assisted-by: Claude Code (claude-opus-5)
@stephrobert
stephrobert merged commit 91d5a46 into main Nov 11, 2026
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant