fix(release): the tracker agrees with the tags, and a gate keeps it that way by stephrobert · Pull Request #877 · stephrobert/feint

stephrobert · 2026-11-11T15:53:07Z

Summary

Four documents claimed something the tags disprove. The tracker was corrected by hand; this pull request is the part that stops it happening again, plus the two sentences that were simply false.

What was measured. v0.14.0 and v0.15.0 are published tags. Both carried an open milestone titled after them, holding 19 and 10 open issues, and neither described what its release shipped — 0.14.0 shipped drift triage and block import, 0.15.0 shipped feint run and the S3 data plane. Every minor from 0.9.0 to 0.13.0 has a closed, empty milestone (0.9.0 with 63, 0.11.0 with 79, 0.12.0 with 45), so the convention was real and these two broke it. What let it through: CONTRIBUTING.md still said milestones were "the six waves of the roadmap's sequence", stale since 0.9.0. The sentence that would have caught it was itself out of date.

The gate

tools/release/milestone-gate.sh asks two questions before a tag: does an open milestone claim a version git has already tagged, and does the version being tagged hold open issues under its own name.

It is narrower than the first draft, and the narrowing is the interesting part. "Does any open milestone carry a version number" would have refused every tag from the day a 1.0.0 milestone was created. A milestone for work still ahead is legitimate, and is the one case where the tag should wait for the milestone rather than the milestone being emptied after it. A guard that refuses everything passes its own attack test and breaks the product, so the falsify spec mutates both directions and the accepting half is its own test.

Three exit codes, for the reason preflight.sh already carries at length: a failed measurement and an absent result are different answers, and only one of them is about the repository.

The harness lied twice, and the comments say so

the gh stub returned the raw fixture instead of applying the --jq expression, so the gate read JSON where it expected lines, matched nothing, and reported the repository coherent on the very payload that asked for the gate;
the accepting fixture had the tagged version's milestone closed, which the gate skips before reaching the exception — so the mutation deleting that exception reported STILL GREEN;
and the test passed standalone while failing under the pre-push hook, which exports GIT_DIR and friends: the throwaway repository pointed at the real one. The environment is filtered now.

Each time the fix was the test. The test also builds its own git repository with the tags the fixtures name, because a shallow CI checkout has no tags and reading the real ones would turn this into a measurement of the checkout depth.

Two sentences that were false

README.md: "Object storage is out of scope for now." On the day of the release whose headline is object storage. It sits after the generated coverage:end marker, so feint docs --check could not reach it.

The generated coverage headline named the wrong population. "Of the 548 operations upstream declares", while coverage/scaleway-baseline.json scopes the Scaleway scan to 11 of the 56 products in that SDK. So 548 is the surface of a third of the provider, printed as the provider's surface, with the scoping stated only at docs/limits.md:3316. A reader with scaleway_rdb_instance could not learn their stack was outside the scan, and docs/adoption.md says what then happens: the client reaches the real endpoint with whatever credentials the shell holds.

A third commit: one limit written so an issue could be refused

limits:check flagged two sections the moment #849 and #850 closed, which is the tripwire working. Both were read: the Object Storage bounds are the product's and shipping the doors lifts none of them, so that one is re-dated; the interception section still costed a route the product did not take, so it now says that feint run pays the cost through a scoped forward proxy instead. Only those two lines are re-dated — eleven sections stay flagged, that is #857, and limits-acks.py refresh would have buried every one of them.

A new section then records why the Outscale peering transcript is not committed, so #438 could be refused with its reason where a reader meets it rather than in an issue's history. Both READMEs move with it: the safety banner counts the limits sections it vouches for, and feint docs --check refused the commit until they agreed.

Done outside the diff, on the tracker

Eight milestones lost their version prefix; nothing else about them changed. Nine issues left a milestone that contradicted another document: #191, #192, #193, #194, #480, #482 were in three places at once while docs/roadmap.md:818 says that work carries no milestone at all; #763 planned a Kubernetes API that docs/limits.md:692 refuses under #283; #695 and #679 are a defect and harness work rather than contract text. #755 was retitled off a number that moved without it. A 1.0.0 milestone now holds #847, #848, #851, #852, #854, #405 and the three adoption P0s.

Then all 94 open issues were audited against one question — what does this buy adoption — and twelve were closed because they were already done, already false, or already refused elsewhere: #87, #415, #438, #558, #666, #677, #726, #763, #778, #856, #858, #859. Each closure cites its evidence by file and line, because "done" is not evidence. Five issues were opened to carry what those closures left behind (#872 to #876), including the one this pull request cannot fix: #125 is closed completed while the pull-request runtime gate in its title does not exist, and the PR GitHub lists as closing it says so in its own body.

The backlog went from 94 open to 87, and about nine more are sections of a neighbouring issue rather than issues.

Type of change

Checklist

Always

mise run check passes
No new external Go dependency
internal/core still knows no provider — untouched
Nothing in the diff could be written identically for another provider. The coverage headline change is uniform across the three on purpose, so nothing has to decide which provider deserves a caveat
mise run testplan was run, and what it printed was run

When a model wrote a substantive part of this

An Assisted-by: trailer names the tool and the model version
No conformance suite is touched by this diff; mise run prepush and the testplan's runs were played here, and are named below
No provider field name is added or changed

When a route is added or changed

N/A.

When behaviour a client can observe changes

No client-observable behaviour changes. The README's generated block is regenerated with mise run docs:coverage, and feint docs --check passes
docs/limits.md unchanged: no documented limit moves

When `.github/workflows/` is touched

N/A.

When a machine runtime is involved

N/A — no runtime path is touched.

Related issues

None closed. It is the mechanical half of the milestone realignment, and it exists so the next release cannot recreate the state it was written for.

🤖 Generated with Claude Code

…s the run Four findings from an external audit of merged code, each verified against the source before being corrected, and three of them were real false greens of mine. ## The verdict compared nothing about an object's content `observable.go` took an object's digest from ListObjectsV2's ETag. `internal/s3/list.go` emits no ETag at all, so the field was ALWAYS empty, and the comparison key was the door, the bucket and the key — the size it had already read was dropped too. Two different contents under one key compared equal, and a reset could report convergence over bytes that had changed. The commit message that shipped it claimed objects were compared by digest. What let it through is the shape of the test beside it: TestADifferenceIsNamedWhereverItIs builds observedObject values BY HAND, with digests in them, so it exercised the comparison and never the observation. A fake emitting an ETag would have hidden it just as well. Each object is now FETCHED and hashed by the observer. Not the store's record of it and not the door's checksum header: what is judged is whether a client gets the same bytes back, and a digest supplied by the subject cannot answer that. The delivered size and the listed size travel separately, so an inventory that disagrees with the bytes is a difference rather than a silent one. The audit's decisive test exists — TestTwoObjectsOfOneSizeAndDifferentBytesDoNotConverge — and it mounts a REAL door over a REAL store, because that is the only arrangement in which an absent field is visible. It also asserts the same bytes twice compare EQUAL, so it cannot be satisfied by a comparison that always differs. Writing through the door rather than behind it was not a preference either: a bucket created straight on the store carries no creation metadata and the door refuses to list it with a 501. That cost three runs of blaming the reader. ## `feint run` could return 0 with a broken promise Only "did not converge" failed the run. A reset the emulator refused, one that could not be read back, and one with no checkpoint to return to all left the exit code at the child's, so this was a possible CI run: terraform test success feint reset failed feint run exit 0 The same held for a lab this command owned and could not stop: `cleanup: failed` reached the receipt and nothing else. The rule is now the general one, and it is what a wrapper is for: the child's code travels untouched when the wrapper kept its promises, and the wrapper fails when it did not. `deliveredWhatWasAsked` lists the answers that are NOT a success rather than the ones that are, so a verdict word added later cannot default to "fine" — and the test covers the full cross product including two words nobody has written yet. ## The checkpoint's "both halves or neither" was a sentence The bytes were saved before the control-plane snapshot, so a failure in the second step left an earlier checkpoint of that name holding its OLD store beside the NEW object references. objectstore publishes its half atomically by renaming a temporary file, which is what makes the fix a reordering: snapshot into memory first, save the bytes, publish the store half last. A failure at any step now leaves the previous checkpoint consistent with itself. **The first test for this measured nothing**, and the harness said so: closing the byte store makes BOTH orders refuse at the same step, so the mutation that reverses the order stayed green. The scenario that separates them is the one the audit described — the byte half succeeds and the control-plane half fails — so the store is made unmarshallable and the assertion is on the BYTES a later reset brings back. ## Named checkpoints overwrote each other The emulator holds any number; the CLI wrote one file per instance. So `checkpoint A` then `checkpoint B` made `reset A` impossible, and the refusal that fired was honest while the feature was missing. Records are indexed by name now, with a `last` pointer so `feint reset` with no argument still means something. One of my own tests then passed for the wrong reason — it found a record a neighbouring subtest had written — which the per-name index changed underneath it. It has its own runtime directory now and asserts that the other run's record is left alone. ## What this does NOT fix, and why The audit's third P0: a firewall rule modified in place keeps its rule-set name, so `Survey` returning names cannot see it and a reset could report convergence while packets are still blocked. Its premise is correct — Survey answers names — but the audit says it deduced this from the code rather than reproducing it, and so have I. I am not writing a fix without a test that bites first; it needs a real TCP flow before, after and after the reset, on Incus or OVN. Also untouched: the concurrency question the audit raises at the end. A reset is not a global transaction, and nothing yet proves what a create arriving mid-reset does. /falsify: 11 mutations on a-reset-is-not-a-restore.json and 16 on a-run-owns-its-lab.json, all biting, including the two that reproduce the audit's own P0s and the one that reverses the checkpoint's order. mise run prepush green. Assisted-by: Claude Code (claude-opus-5)

The audit's third P0, reproduced before being believed, then fixed — plus two defects the reproduction itself surfaced, one of them mine. ## The finding, and why it waited The audit deduced from the code that a reset could report convergence while a firewall rule was still missing from the runtime, and said plainly that it had not reproduced it. Neither had anybody here, so the previous commit shipped without a fix and said so. The premise was right. `Survey` answers NAMES; a rule set whose content changed after the checkpoint keeps its name, so nothing in the observation can see the difference. `tools/conformance/scaleway/network.sh` now takes a checkpoint with port 80 open, lets the suite revoke the rule, resets, and asks a TCP connection: - revoking it closes it again ok: port 80 refused again - and a reset puts the rule back on the NIC, not only in the store FAIL: the reset reported convergence and port 80 is still refused: the rule came back in the store and not on the NIC That is the test that bites. The fix came after it. ## The fix: a third optional half `emulator.RuleReconciler`, beside `Reconciler` and `MissingReconciler` and separate from them for the reason MissingReconciler already documents — one interface carrying two methods turned every pack into a non-implementer in silence, and the call still answered 200. The gesture is shared: `machine.GroupSync.Reapply` translates the restored group, writes the set and replays it onto the wearers, which is the same sequence for the three packs. What varies is one word — which restored record a provider calls a group — so each pack writes a loop and a kind. The firewall reconciliation was written once in a pack before, and the two others ran with zero ACLs while their API described a closed port (#475). Ordered last of the three, because a set attaches to the interfaces of the machines that wear it: re-applying before ReconcileMissing has put those machines back would attach nothing. ## Two more, from the same leg **Mine, immediately.** The first version re-applied every restored group, and the runtime leg reported two rule sets appearing on the host that the checkpoint never held — left over from earlier suites whose machines were long swept. A reset puts back what the checkpoint had; it does not furnish the host with objects it did not. Only a group somebody wears is re-applied now, which is also exactly the group the audit's case is about. **Older than the audit.** A record keeps its machine name after the machine is gone — that is how ReconcileGone finds what to destroy — so "names a machine the host does not hold" also describes, word for word, a server the checkpoint says is stopped. A reset was starting containers for terminated VMs and writing `running` over the state it had been asked to return to, and the verdict then named the resource the reset had itself changed: resource present and not in the checkpoint: vm i-a3f536ec state=running resource in the checkpoint and not present: vm i-a3f536ec state=terminated machine present and not in the checkpoint: feint-osc-i-a3f536ec The comparison is in the shared layer, against `Binding.RunningState`: the word varies, the gesture does not, and the Binding already carries that field because PowerOn writes it. Measured as pre-existing before being attributed — the leg was run with these changes stashed, and failed on those three lines alone. ## Proof `FEINT_VM=incus mise run conformance:leg -- runtime`, 94 s, green end to end, including the new reset step and `reset-host.sh`, which the stashed baseline failed. /falsify: 15 mutations on a-reset-is-not-a-restore.json, all biting, including the four added here. Both directions of the firewall fix are mutated, because a guard that refuses everything passes the attack and breaks the product. mise run prepush green. The fixed sleep this adds carries its `waits on silence:` reason, which TestEveryFixedSleepInARuntimeSuiteSaysWhatItWaitsFor demanded before it would pass. ## The debt that remains, with its number The concurrency question the audit raises at the end: a reset is not a transaction, and nothing measures what a create or a delete arriving mid-reset does. It is now #870, with the three paths read from the code, what would close it, and the statement that none of it is reproduced yet. It is not a reason to hold #853 open. Assisted-by: Claude Code (claude-opus-5)

…hat way Three documents and the issue tracker each claimed something the tags disproved. The tracker was corrected by hand; this commit is about the part that stops it happening again, plus the two sentences that were simply false. ## What was measured v0.14.0 and v0.15.0 are published tags. Both carried an OPEN milestone titled after them, holding 19 and 10 open issues, and neither described what its release had shipped: 0.14.0 shipped drift triage and block import, 0.15.0 shipped `feint run` and the S3 data plane. It was not the practice slipping once. Every minor from 0.9.0 to 0.13.0 has a closed, empty milestone — 0.9.0 with 63, 0.11.0 with 79, 0.12.0 with 45 — so the convention was real and these two broke it. What let it through is that `CONTRIBUTING.md` still said milestones were "the six waves of the roadmap's sequence", stale since 0.9.0: the sentence that would have caught it was itself out of date. ## The gate, and why it is narrower than the first draft `tools/release/milestone-gate.sh` asks two questions before a tag: does an open milestone claim a version git has ALREADY tagged, and does the version being tagged hold open issues under its own name. The first draft asked "does any open milestone carry a version number", which is wrong in a way worth recording: it would have refused every tag from the day a `1.0.0` milestone was created. A milestone for work still AHEAD is legitimate, and is the one case where the tag should wait for the milestone rather than the milestone being emptied after the tag. A guard that refuses everything passes its own attack test and breaks the product, so the falsify spec mutates both directions and the accepting half is its own test. Three exit codes, for the reason preflight.sh already carries at length: a failed measurement and an absent result are different answers, and only one of them is about the repository. **The test harness lied to me twice, and the comments say so rather than claiming otherwise.** The `gh` stub first returned the raw fixture instead of applying the `--jq` expression, so the gate read JSON where it expected lines, matched nothing, and reported the repository coherent on the very payload that asked for the gate — green for the only reason that makes a green worthless. And the accepting fixture had the tagged version's milestone CLOSED, which the gate skips before reaching the exception, so the mutation deleting that exception reported STILL GREEN. Both times the fix was the test. The test also builds its own git repository with the tags the fixtures name. A shallow CI checkout has no tags, and reading the real repository's would have turned this into a measurement of the checkout depth. ## Two sentences that were false **`README.md`: "Object storage is out of scope for now."** On the day the release whose headline is object storage. It sits after the generated `coverage:end` marker, so `feint docs --check` could not reach it. It now says what shipped, why it took a year, and what is still not certified (#856, #858). **The generated coverage headline named the wrong population.** It read "Of the 548 operations upstream declares" while `coverage/scaleway-baseline.json` scopes the Scaleway scan to 11 of the 56 products in that SDK — so 548 is the surface of a third of the provider, printed as the provider's surface, with the scoping stated only at `docs/limits.md:3316`. A reader with `scaleway_rdb_instance` could not learn their stack was outside the scan, and `docs/adoption.md` says what then happens: the client reaches the real endpoint with whatever credentials the shell holds. The headline now names the product count, uniformly for the three, so nothing has to decide which provider deserves a caveat. `docs/roadmap.md` also stopped saying "The order 0.11.0 is being worked in" in the present tense, three releases later. ## The tracker, corrected by hand in the same pass Eight milestones lost their version prefix; nothing else about them changed. Nine issues left a milestone that contradicted another document — #191, #192, #193, #194, #480 and #482 were in three places at once while `docs/roadmap.md:818` says that work carries no milestone at all; #763 planned a Kubernetes API that `docs/limits.md:692` refuses under #283; #695 and #679 are a defect and harness work rather than contract text. #755 was retitled off a number that moved without it. A `1.0.0` milestone now holds #847, #848, #851, #852, #854, #405 and the three adoption P0s. Its description is a view: #847 holds the graph, #854 the acceptance, `docs/roadmap.md` the ordering. /falsify: 4 mutations on a-milestone-cannot-claim-a-version.json, all biting. The coverage headline has none, deliberately, and the spec says why: the clause is prose inside a format string, every mutation removing it removes names the harness requires to survive, and contriving one would measure the contrivance. mise run prepush green. The gate answers exit 0 on the live repository, which is the measurement that the hand corrections landed. Assisted-by: Claude Code (claude-opus-5)

…is section costed `limits:check` flagged two sections the moment #850 and #849 closed, which is the tripwire doing its job: a closed issue is not a lifted limit, and somebody has to look. Both were read. The Object Storage section is accurate — its bounds are the product's, and shipping the doors lifts none of them — so it is re-dated and nothing else. The interception section costed a route the product did not take. It still opened on "the DNS redirect is the hard, dangerous one" with no mention that `feint run` now pays the cost through a scoped forward proxy instead: CONNECT accepted, TLS terminated with a per-run authority, every unknown host refused rather than tunnelled, measured on the host network with no sudo and no namespace. The DNS analysis stays, because a reader comparing the two approaches needs it and because the namespace correction in it is a measurement error worth not repeating — but it is no longer the reason anything is refused. Only these two lines are re-dated. Eleven sections stay flagged: that is #857, and `tools/docs/limits-acks.py refresh` would have re-dated every one of them and buried it. Assisted-by: Claude Code (claude-opus-5)

…hat is a decision (#438) #438 asks for a corpus that was recorded and never committed. Closing it needed the reason written where a reader meets it, because a refusal living only in an issue's history is the debt this repository forbids. The surface itself is not in question: the five peering operations each read `driven`, `shape: observed` and `negative: true` in `coverage/evidence.json`, from `shapes/outscale.json`. What is refused is committing a SECOND proof of an already-proven surface, whose replay cascades from one unreproduced `CreateNet` conflict — and `corpus/accepted.json` cannot hold it either, since an entry that excuses nothing fails that gate by its own rule. The recorder findings from that run are not lost: they are the `REDACTED-10` tag-key family, already filed against the instrument as #384 and already recorded in `corpus/accepted.json` as NOT THE EMULATOR. The section names what lifts it, which is what separates a limit from an excuse: the `CreateNet` conflict reproduced, or a client this project drives exercising peering where `shapes/` cannot reach. Both READMEs move with it, and that is the banner doing its job rather than noise: it counts the limits sections it vouches for, so a new section is a regeneration. `feint docs --check` refused the commit until they agreed. Assisted-by: Claude Code (claude-opus-5)

stephrobert added 5 commits November 11, 2026 14:12

stephrobert merged commit cda6f12 into main Nov 11, 2026
31 checks passed

stephrobert mentioned this pull request Nov 11, 2026

chore(release): 0.16.0 #883

Open

12 of 16 tasks

Navigation Menu

Repository navigation

fix(release): the tracker agrees with the tags, and a gate keeps it that way - #877

fix(release): the tracker agrees with the tags, and a gate keeps it that way#877
stephrobert merged 5 commits into
mainfrom
fix/the-tracker-agrees-with-the-tags

stephrobert commented Nov 11, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

stephrobert commented Nov 11, 2026

Summary

The gate

The harness lied twice, and the comments say so

Two sentences that were false

A third commit: one limit written so an issue could be refused

Done outside the diff, on the tracker

Type of change

Checklist

Always

When a model wrote a substantive part of this

When a route is added or changed

When behaviour a client can observe changes

When .github/workflows/ is touched

When a machine runtime is involved

Related issues

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

When `.github/workflows/` is touched