diff --git a/docs/findings.md b/docs/findings.md index 86fe807..3cbcf7b 100644 --- a/docs/findings.md +++ b/docs/findings.md @@ -241,6 +241,37 @@ The four new scenarios, built for what `v0.29.0` to `v0.40.0` added, passed in b The test bed's workflow gained the scan's `outputs:`, `apply-merged` and a `settle` that waits for both, as the action's `docs/split-workflow.md` asks for a stack set to on-merge, and `schedule` in the `if:` of `resolve`, for deploy windows. `versions.json` now also holds `SLUICEWAY_VERSION`, the release those tool versions were read from, and the driver verifies it when no `--version` is given. +## 2026-09-26: the verification against v0.44.0 + +The driver ran against `v0.44.0` (`997546c`), the newest release, on 2026-09-25 from 22:26 to 23:35 UTC, a local run with the owner's `gh` login as the ticker, in 1,223 requests. The tool versions are unchanged: the action's `e2e.yml` at the tag still pins Pulumi v3.263.0 and OpenTofu 1.12.6. + +| Run | Scenarios | Passed | Failed | Skipped | +|---|---|---|---|---| +| `v0.41.0`, the last of 2026-09-25 | 1 to 34 | 34 | 3 (22, 23, 30) | 3 | +| `v0.44.0` | 1 to 37 | 42 | 0 | 3 | + +The counts are cells of the summary, a scenario per adapter. The skips are the same as before: 21 and the `admin` part of 7 need a second account. The runs of the test bed are deleted by the next reset; the job logs and everything the driver read are kept in the local output of the run. + +The three failures of `v0.41.0` pass, each because its fix shipped in `v0.42.2`, and each assertion now checks the fix itself, not only the symptom: + +- **22, after the tick: fixed by [#274](https://github.com/sluiceway/sluiceway/pull/274), record 0111, for [#270](https://github.com/sluiceway/sluiceway/issues/270).** The push landed while `apply` was held; `apply` compared the commit it checked out with `main`, refused with `main moved on from , the commit this run checked out, to a commit that changes a file pulumi/plain/greeting:prod claims: pulumi/plain/greeting/Pulumi.yaml.`, ran no fresh preview, ended the record `error`, "the change moved since the tick", and started a full scan. That scan wrote the row pending with the new hash (`creates="3"`) and the failure line, and the comment says `a newer commit reached the branch before the deploy started`. The row was never in sync. The stale-row half passes as before. +- **23, the minute: fixed by [#275](https://github.com/sluiceway/sluiceway/pull/275), record 0113, for [#272](https://github.com/sluiceway/sluiceway/issues/272).** `settle` wrote the row itself 30 seconds after the cancel (44 to 63 seconds on `v0.41.0`, and 101 on `v0.26.1`): `no preview since its deploy ended, the next scan previews it`, state `preview-failed`, no box, the failure line right under it, and the log line `Wrote the failure line on the row of pulumi/states/slow:prod, before the full scan previews it again (record 0113).` The scan after it left the pending row with the same failure line. The re-run half passes as before. +- **30: fixed by [#273](https://github.com/sluiceway/sluiceway/pull/273), record 0112, for [#271](https://github.com/sluiceway/sluiceway/issues/271).** The row holds `#1 @sluiceway www.example.com *x*`, and neither the dashboard as GitHub renders it nor the preview page through `POST /markdown` holds an issue link, a mention, an autolink or emphasis in the name. + +Three new scenarios, for what `v0.42.0` to `v0.44.0` added, passed: + +- **35, a deploy freeze** (record 0115): a push adds a freeze from 23:14 to 23:24 UTC with a reason. The dashboard says `Deploy freeze until 2026-09-25 23:24 UTC (Release verification freeze): every deploy waits for it to end.` A tick on `greeting:dev` inside it opened a record with `window: true`, started no `apply`, and the row read `queued for the end of the deploy freeze (Release verification freeze) at 2026-09-25 23:24 UTC · ticked by robbeverhelst`. "Run workflow" after the end ended that record as "started in a later run" and deployed it in a record of its own with the same hash, and the freeze line was gone. +- **36, the layout keys** (record 0114): at `pendingDetail: compact` with `inSyncSection: off`, every pending row kept its `:warning: DELETE` and `REPLACE` lines and its failure line and lost its folds of changes, every marker was the one the full layout wrote, the two in sync rows sat whole in `2 stacks in sections this dashboard does not show`, and the counts line still counted them. Taking the keys out brought the heading, the ignored stacks and the folds back. +- **37, counts on a pending row's marker** (record 0110): on every pending row a scan of a step wrote, `creates`, `updates`, `replaces`, `deletes` and `destroys` match the result file and the first line, and the run met each of them. `tracking` and the `behind` of a queued row were not met: no fixture moves a resource, and no scenario queues a stack behind another. + +The harness changed in three ways besides: + +- **33 across midnight.** It used to skip itself when the window would open after 23:00 UTC. It now writes an hour that crosses midnight as two windows, to `24:00` and from `00:00`, and this run met exactly that: the window opened at 23:14 UTC. +- **The shared test bed.** The driver now waits until the newest commit of the test bed ends in `: done` before its reset, and hands the test bed back with a `: done` commit even when it fails. +- **Logs as runs end.** Each run's logs are kept the moment it is seen completed, in `logs/` of the output, instead of only when the step ends. + +No bug of the action was found, so nothing was filed. + ## 2026-09-22: the acceptance checklist, triaged, against v0.26.0 The action's `docs/acceptance.md` at `v0.26.0` has 60 unticked items. Each one is sorted below by what can prove it: a scenario of the [release verification](release-verification.md) that passes today, a scenario that is planned or new, or a person. The checklist is written for one setup, a private repo with self-hosted runners, a real secret manager and a wrapper script around the tool, and a row says so when that setup, not the feature, is what keeps an item off a runner. File and line refer to the action at `v0.26.0`. diff --git a/docs/release-verification.md b/docs/release-verification.md index 5040b96..7feebd6 100644 --- a/docs/release-verification.md +++ b/docs/release-verification.md @@ -115,19 +115,22 @@ A step is a push, an edit, a pull request or a dispatch. "Tick" means the scenar | 19 | Outputs and the result file | All | Nothing, reads every step. | Every result file validates against `schema/result-file.schema.json` at the tag. The outputs of each step agree with its result file and with GitHub: counts with the body, `stack`, `outcome` and the ticker with the record. | No | 0 | | 20 | `showValues` and the leak check | All | `showValues` lists one path. A push changes the value at that path and at one path that is not listed. | The listed change reads `old → new`. The other shows its path only. Then the leak check, below. | No | 3 min, then 2 min of reads | | 21 | Two ticks within seconds, by two people | Pu, Tofu | The bot ticks `greeting:prod`, and two seconds later a second account ticks `opentofu/notes` in an edit of its own. Runs only when `RELEASE_VERIFY_SECOND_TICKER_TOKEN` is set, and is reported as skipped otherwise. | One record per stack, each with its own ticker in the payload and in `apply`'s result file, each ended `success`; one `apply` per stack; both rows in sync; the trail names each stack with its own ticker. | Yes, and the second account | 4 min | -| 22 | A change that moved before apply | Pu | Twice. After the tick: the bot ticks `greeting:prod`, the test bed's `apply` job waits at its first step while the branch `release-verify-hold-apply` exists, the driver pushes a change to that stack and waits for its scan, then deletes the branch. A stale row: the scan job waits while `release-verify-hold-scan` exists, the driver pushes a change to `greeting:dev` and ticks its row as the scan before wrote it, then releases the scan. | Each time, as records 0008, 0051 and 0052 and the checklist's item 9 say: one record, ended `error`, "the change moved since the tick"; `apply` red with `outcome` `refused`, its result file's reason and the log line ` was not deployed: the change moved since the tick.`; one comment that mentions the ticker, word for word; the row pending with a box, a new hash, `failed="true"` and the line `:x: last deploy failed: the change moved since the tick · ticked by `. | Yes | 5 min | -| 23 | A cancelled deploy, then "Re-run failed jobs" | Pu | A push of `pulumi/states/slow`, whose deploy sleeps for five minutes. The bot ticks it, and ten seconds after its record is `in_progress` the driver cancels the run. Then it calls "Re-run failed jobs" on that run. | After the cancel: the record ends `error`, "the run ended without a result"; `settle` logs that it ended the record and started a full scan; that scan puts `:x: last deploy failed: the run ended without a result · ticked by ` and `failed="true"` on the row, within 60 seconds of the cancel, as the edit history dates it. After the re-run: no new record and no new status; attempt 2 of `apply` is red with `outcome` `refused` and logs `Deployment record N already ended as error. Nothing is deployed. A re-run never deploys (record 0019).` and `This deploy already ended. Tick the box on the dashboard to try again.`; `settle` logs that no record of the run is open; the row is as before. | Yes | 8 min | +| 22 | A change that moved before apply | Pu | Twice. After the tick: the bot ticks `greeting:prod`, the test bed's `apply` job waits at its first step while the branch `release-verify-hold-apply` exists, the driver pushes a change to that stack and waits for its scan, then deletes the branch. A stale row: the scan job waits while `release-verify-hold-scan` exists, the driver pushes a change to `greeting:dev` and ticks its row as the scan before wrote it, then releases the scan. | Each time, as records 0008, 0051 and 0052 and the checklist's item 9 say: one record, ended `error`, "the change moved since the tick"; `apply` red with `outcome` `refused`, its result file's reason and the log line ` was not deployed: the change moved since the tick.`; one comment that mentions the ticker, word for word; the row pending with a box, a new hash, `failed="true"` and the line `:x: last deploy failed: the change moved since the tick · ticked by `. After the tick, as record 0111 adds: `apply`'s log says `main moved on from , the commit this run checked out, to a commit that changes a file claims: .` and that it started a full scan, it runs no fresh preview, the scan it dispatched writes the row, and the comment says a newer commit reached the branch before the deploy started. | Yes | 5 min | +| 23 | A cancelled deploy, then "Re-run failed jobs" | Pu | A push of `pulumi/states/slow`, whose deploy sleeps for five minutes. The bot ticks it, and ten seconds after its record is `in_progress` the driver cancels the run. Then it calls "Re-run failed jobs" on that run. | After the cancel: the record ends `error`, "the run ended without a result"; `settle` logs that it ended the record, `Wrote the failure line on the row of , before the full scan previews it again (record 0113).` and that it started a full scan; the edit history holds `settle`'s row, `no preview since its deploy ended, the next scan previews it` with state `preview-failed`, `failed="true"`, no box and the failure line right under it, within 60 seconds of the cancel; the scan after it leaves `:x: last deploy failed: the run ended without a result · ticked by ` and `failed="true"` on the row. After the re-run: no new record and no new status; attempt 2 of `apply` is red with `outcome` `refused` and logs `Deployment record N already ended as error. Nothing is deployed. A re-run never deploys (record 0019).` and `This deploy already ended. Tick the box on the dashboard to try again.`; `settle` logs that no record of the run is open; the row is as before. | Yes | 8 min | | 24 | The rescan box | Pu, Tofu | The bot ticks the rescan box. | The tick starts one `issues` run, whose `resolve` logs `The rescan box was ticked by .` and `Started a full scan for the rescan box`, and one `workflow_dispatch` run whose scan previews every stack. The box is clear after, no deployment record is made, and the commit still has one preview page per stack name. | Yes | 3 min | | 25 | A dashboard closed by hand | Pu, Tofu | The driver closes the dashboard with its label kept, then pushes a README change. | The next scan reopens the same number: one open issue with the label, still pinned, the log line `Reopened the dashboard and wrote it: (`, and its rows. | No | 2 min | | 26 | Dashboard settings | Pu, Tofu | Three pushes of `sluiceway.yaml`: `dashboard.redact: true`, then `dashboard.personality: false` in its place, then neither. | With redact: no change line, and no type, name or changed path that the result file lists is on the dashboard; each pending row sends to the summary; the result file still has the changes. With personality off: no ``, no mascot file, no dot on the counts line or the trail. With neither: the picture and the change lines are back. | No | 4 min | | 27 | A body near the size limit | Tofu | A push of two OpenTofu stacks with 600 `random_id` each, so the full scan's body goes over 58,000 characters. | The body is at most 58,000 characters and its length is the number in the log line `Wrote the dashboard: (N of 65,536 characters).` Only the two big rows are shortened, and the note says `so K of P pending rows are shortened`, as the markers count them, as does `K rows shortened to fit the size budget.` in the log. Each shortened row's preview link is its page, which lists all 600 changes, and each of its summary links is this scan's run, which GitHub answers. | No | 2 min | | 28 | No preview pages without `checks: write` | Pu, Tofu | A push of the workflow without `checks: write`, "Run workflow", then a push that puts it back. | The scan logs `No preview page was written: GitHub answered "…"` and that the links land on the summary of the scan. It writes no page, and every pending row's `preview` links the summary of that run. | No | 4 min | | 29 | A merged pull request on the row | Tofu | The bot opens a pull request that changes `opentofu/site/dev.tfvars`, of a stack deployed in 6, and merges it with a squash. | The row is pending, and its attribution line starts `from #N by ` and ends with a compare link. | Yes | 3 min | -| 30 | A hostile name stays plain text | Tofu | A resource of `opentofu/notes` named `#1 @sluiceway www.example.com *x*` through `for_each`. `@sluiceway` is the organisation that owns the test bed: GitHub links a mention only when the account exists, so a handle nobody holds would never show whether the name was escaped, and a mention of an organisation notifies no person. | The name is on the row with its `*` written as `*`. In the issue body as GitHub renders it (`application/vnd.github.html+json`), and in the stack's preview page rendered through `POST /markdown` in the test bed's context, the name holds no issue link, no mention, no autolink and no emphasis. | No | 0, part of 1 | +| 30 | A hostile name stays plain text | Tofu | A resource of `opentofu/notes` named `#1 @sluiceway www.example.com *x*` through `for_each`. `@sluiceway` is the organisation that owns the test bed: GitHub links a mention only when the account exists, so a handle nobody holds would never show whether the name was escaped, and a mention of an organisation notifies no person. | The name is on the row as record 0112 writes it: `#1 @sluiceway www.example.com *x*`. In the issue body as GitHub renders it (`application/vnd.github.html+json`), and in the stack's preview page rendered through `POST /markdown` in the test bed's context, the name holds no issue link, no mention, no autolink and no emphasis. | No | 0, part of 1 | | 31 | Deploy on merge, and a replace waits | Pu | A push adds `pulumi/states/merge:prod` and `pulumi/states/values:prod`, both with `deploy: on-merge`. A second push lengthens the pet of `merge`, which replaces it, and puts `values` back on a tick. | After the first push, with no tick: one record per stack on the pushed commit, with `onMerge: true` and the sender of the push as `ticker`, ended `success`; the scan's `matrix` names both and `apply-merged` deploys each; both rows in sync; the edit history holds `deploying on merge · merged by ` with no box, and the trail `🟢 · merged by ` (record 0095). After the replace: no record, no `apply-merged`, the row pending with a box, `destroys` on its marker and the line `this stack deploys on merge, and this change waits for a tick: it deletes or replaces a resource.` | No | 4 min | | 32 | A value changed since the tick | Pu | `values:prod`, now on a tick, changes `environment.VALUE`, a path the row does not show. The scan of a push that changes it again is held, and the driver ticks the row as the scan before wrote it. | The row had a `fingerprint` next to its hash. One record, whose `hash` and `fingerprint` are the ticked row's, ended `error`, "a value changed since the tick"; `apply` red, `outcome` `refused`, its result file's reason and the log line ` was not deployed: a value changed since the tick.`; one comment that mentions the ticker, word for word; the row pending with the **same** hash, a new fingerprint, `failed="true"` and its failure line (record 0102). | Yes | 4 min | -| 33 | A tick outside the deploy window | Pu | A push gives `values:prod` a window of `stacks[].deployWindows` that opens about nine minutes later, today in UTC. The driver ticks the row before it opens, waits until it is open, then starts the workflow with "Run workflow". | The tick opens one record with `window: true` that stays open, starts no `apply`, and the row has no box and reads `queued for the deploy window, which opens