epic(actions): make Noema, Strix, and OpenCode concurrency exact-PR and current-head safe #1800

Activity

1. seonghobae

   self-assigned this

   on Sep 3, 2026
2. seonghobae commented on Sep 6, 2026

   

   seonghobae

   on Sep 6, 2026

   ContributorAuthor

   More actions

   Push-triggered Strix runs cannot coalesce, and the obvious fix is the one this repo already tried and rejected

   Measured while tracing what actually occupies runner slots. Posting it here rather than opening a PR,
   because the change it points to would reverse an explicit owner decision recorded on 2026-09-03.

   What is occupying the slots

   At 14:10Z the .github queue held 203 queued runs against 11 in-flight, and every in-flight run
   was a Strix scan. Ten of them:

   run	running for	concurrency group keyed on
   34032063594	2.08 h	pr-1911
   34031981190	2.11 h	github.run_id (push to main)
   34031419718	2.31 h	github.run_id (push to main)
   34031339200	2.34 h	pr-1879
   34030785297	2.54 h	github.run_id (push to main)
   34030605428	2.60 h	pr-1519
   34030182712	2.75 h	github.run_id (push to main)
   34029626828	2.94 h	pr-1678
   34027404208	3.73 h	pr-1930
   34020314631	6.29 h	pr-1872

   All four run_id runs are event=push, head_branch=main. strix.yml triggers on
   push: branches: [main, develop, master], and on a push neither github.event.pull_request.number
   nor github.event.client_payload.pr_number exists, so the group key falls through to
   github.run_id — unique per run, so no push-triggered scan can ever supersede another.

   Rate and duration over the last 100 push-triggered Strix runs (2026-09-03T13:13Z → 09-06T12:03Z,
   70.8 h):

   • creation rate 1.41 runs/h
   • of the 30 that reached success/failure: median 5.80 h, max 20.90 h
   • Little's law: 1.41 × 5.80 ≈ 8.2 expected concurrent with no coalescing (4 observed now, so the
     ceiling is itself throttling them)

   By contrast PR-triggered Strix coalesces normally: of 20 recent completions, 15 are cancelled with a
   median lifetime of 0.20 h. The asymmetry is entirely in the group key.

   Why the obvious fix is wrong

   Changing the fallback from github.run_id to github.ref would make consecutive main pushes share a
   group. That is repository-wide scoping for the push lane, and
   tests/test_required_workflow_queue_contract.py::test_strix_serializes_provider_evidence_per_repository_and_pr
   records that this repository already ran that experiment:

   from 2026-08-24 through 2026-09-03 the concurrency group was deliberately repository-wide … That
   repository-wide scoping fixed the storm but starved cross-PR Strix evidence within the same
   repository instead

   and then, on explicit owner authorization (2026-09-03), it was restored to PR-scoped with
   "non-PR events use a unique run id" as part of that decision, pinned by
   assert "github.run_id" in concurrency_contract.

   An independent measurement today shows the same failure mode in the adjacent lane:
   scheduled-security-scan.yml is a main-push full-tree CodeQL scan with a shared group and
   cancel-in-progress, and over 2026-09-01→09-05 it produced 228 cancelled runs and zero completions
   — every scan killed by the next merge, because main advances faster than a scan finishes. Coverage
   only resumed on 09-05 when the queue drained, not because anything was fixed. Giving push-triggered
   Strix a shared group would reproduce exactly that, on a scanner whose median run is 5.8 h against a
   main that advances every ~12 minutes.

   So both settings are known-bad in different ways, and neither is a coding error to correct:

   • unique key (today) — every main push scanned, none superseded, ~8 concurrent expected
   • shared key — bounded occupancy, but on this merge cadence no main-push scan ever completes

   What actually needs deciding, and by whom

   The lever is not the concurrency key. It is whether a full-tree Strix scan should start on every
   push to a protected branch at all. strix.yml's own header notes the weekly full-tree schedule
   "backstops every path", and a full-tree scan is retroactive by construction — a later scan covers
   commits an earlier one missed. That makes consecutive per-push scans largely redundant while each
   holds a slot for hours.

   Options, none of which an agent should pick unilaterally:

   1. Keep as-is. Accepts ~8 concurrent long scans as the cost of per-commit coverage.
   2. Drop push from strix.yml's triggers, relying on the PR scan plus the weekly full-tree
      schedule. Largest capacity gain; reduces per-commit coverage on protected branches.
   3. Debounce the push lane — shared group with cancel-in-progress: false so pushes serialize
      (one running, one pending, older pendings dropped) instead of running in parallel. Bounds occupancy
      at 1 without the cancel-storm starvation of option 3's true variant, at the cost of skipping
      intermediate commits. This is the ${{ github.event_name == … }} conditional pattern already used
      in pr-review-merge-scheduler.yml, cloudflare-dns.yml and repository-metadata-reconcile.yml.

   I would recommend option 3 — it is the only one that bounds occupancy without either reversing the
   2026-09-03 PR-scoping decision or reducing which commits get scanned to zero — but it still changes
   push-lane semantics the owner set deliberately, so it wants an explicit ruling first. Happy to
   implement whichever is chosen, with contract tests and the usual controls.

   Not claimed

   The 66 of 100 push-triggered Strix runs that ended cancelled were not cancelled by concurrency
   (their keys are unique). I did not determine what cancelled them — queue sweeps and explicit
   cancel jobs both exist in this repository — so the practical occupancy is lower than the unmitigated
   8.2 estimate. That does not change the group-key analysis, but it does mean "nothing supersedes them"
   is a statement about the concurrency mechanism, not about their observed lifetimes.
3. seonghobae commented on Sep 6, 2026

   

   seonghobae

   on Sep 6, 2026

   ContributorAuthor

   More actions

   Correction: the starvation precedent argues against option 3's cancelling variant, not against option 3

   My comment above said "Giving push-triggered Strix a shared group would reproduce exactly that". That
   is too broad, and the distinction it flattens is the one that decides between the options. All figures
   below re-measured directly.

   The two cases are not the same shape

   scheduled-security-scan.yml really is configured to cancel across main pushes —
   group: scheduled-security-scan-${{ github.repository }}-${{ github.ref }} with
   cancel-in-progress: true, and every main push lands on refs/heads/main. But its scans are fast:

   successful-run lifetime (created -> completed), n=30
     median 3.6 min   min 1.5   max 164.5
     2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 4 4 5 6 7 8 9 9 21 | 78 84 85 97 112 165
   main merge interval, 39 gaps: median 11.7 min (min 0.4, max 273.6)
   

   The distribution is bimodal: 24 of 30 finish inside 10 minutes, comfortably under the 11.7-minute
   merge interval, and a tail of six sits at 78–165 minutes. So this workflow does not starve because
   scanning is expensive — it starves only when queue saturation inflates its lifetime past the merge
   interval, which is why coverage resumed on its own on 09-05 when the queue drained. Nothing was
   fixed.

   Strix's push lane is the opposite: median 5.80 h of genuine work against the same 11.7-minute
   merge interval.

   	scheduled-security-scan	Strix push
   intrinsic duration	3.6 min	5.80 h
   starves when	queue is saturated	always, if the group cancels — even on an empty queue
   self-recovers	observed, on 09-05	no

   What that changes

   The precedent is evidence against giving Strix's push lane a shared group with
   cancel-in-progress: true — for a 5.8-hour job on a branch that advances every 12 minutes, the
   completion probability under a cancelling group is effectively zero, permanently rather than during a
   saturation episode. It is not evidence against option 3, which specifies false. Without cancels,
   intrinsic slowness stops converting into lost work.

   Option 3's real cost is not loss but scarcity: bounded at one running plus one pending, most main
   commits get no push scan, and the ones that do arrive roughly every 5.8 hours. The justification for
   accepting that is the retroactive-coverage property of a full-tree scanner — a later scan covers what
   earlier ones missed.

   Correction to my own option list: I attached the two exceptions to that property only to option 2.
   They apply to option 3 equally, and should be read as its cost too:

   1. Code introduced and then removed or rewritten between two scans is never scanned at all.
   2. Findings lose their pull-request attribution — you learn a problem exists, not which change
      introduced it.

   A monitoring indicator worth adding regardless of the choice

   scheduled-security-scan.yml is fine as configured as long as the queue is healthy, and it fails
   silently when it is not: a starved run's status is cancelled, which raises no alert. The leading
   indicator is the ratio of its successful-run lifetime median to the main merge-interval median —
   3.6 / 11.7 = 0.31 today. As that ratio approaches 1, this workflow is about to stop producing
   coverage without anything reporting a failure. It reached that state for four days (09-01 → 09-05,
   228 cancelled, 0 completed) and nobody was told.

   Still not claimed

   Of the 100 push-triggered Strix runs, 66 ended cancelled and not by concurrency (their keys are
   unique), and I still have not identified what cancelled them. A usable discriminator has been
   suggested — a run killed by cancel-in-progress typically has run_attempt=1 with no job ever
   started — and that decomposition is being done separately. Until it is, the real occupancy is below
   the 8.2 estimate and the option comparison above is unaffected either way.
4. seonghobae commented on Sep 6, 2026

   

   seonghobae

   on Sep 6, 2026

   ContributorAuthor

   More actions

   Option 3 is already running in this repository, on a push lane, and it behaves as hoped

   Two measurements to add. The first replaces my earlier hand-waving about what cancelled Strix runs
   actually cost; the second is an in-repo empirical test of the option I recommended, which I had only
   argued from first principles.

   1. What the cancelled push-Strix runs cost — now agreed by two independent computations

   All 66 cancelled runs among the newest 100 event=push strix.yml runs (created
   2026-09-03T13:13:59Z → 09-06T03:01:19Z), occupancy computed as the sum of each job's
   started_at → completed_at:

   total occupancy	118.4 h
   ≤ 1 min	13 runs
   1–120 min	31 runs
   > 2 h	22 runs
   the > 2 h tail holds	108.0 h — 91% of the total
   runs where the strix job itself ran > 1 min	43 of 66

   A second session computed the same set independently and matched every figure. The only divergence
   was a median of 16.1 vs 13.3 min, which is the upper-median versus mean-of-middle-two convention on an
   even n, not a data difference. Their first pass had reported "median 0.1 min, 51 of 66 free"; the
   cause was a filter requiring started_at > created_at, which silently drops any job whose queue wait
   was under a second — including jobs that then ran for 65 minutes. Worth recording as a rule:
   created == started is only valid for spotting a cancelled queued job; it must never be used as a
   proxy for "no runner was allocated" when measuring occupancy. Occupancy needs started and
   completed alone.

   There is also a strong drift inside the window, so a small recent sample and a small old sample would
   disagree without either being wrong:

   older half  n=33  median   4.1 min   ≤1min 11   >2h  3   total 37.3 h   09-03T13:13 .. 09-04T06:06
   newer half  n=33  median 135.4 min   ≤1min  2   >2h 19   total 81.1 h   09-04T06:17 .. 09-06T03:01
   

   This strengthens the earlier point about sweeps: killing a run that has been executing for hours
   throws that time away, while killing one that never got a runner costs nothing. A sweep with an
   occupancy threshold recovers the same slots for far less waste.

   2. scorecard-analysis.yml is option 3, live, on a push lane

   I recommended option 3 (shared group, cancel-in-progress: false) from reasoning. It turns out this
   repository already runs exactly that shape:

   scorecard-analysis.yml
     on: push, schedule, workflow_call
     concurrency:
       group: scorecard-analysis-${{ github.ref }}
       cancel-in-progress: false
   

   Behaviour over its 60 most recent push runs (09-04T10:47 → 09-06T12:03, 49.3 h, 1.22 created/h):

   • 18 success, 42 cancelled, 0 failure, 0 still queued or running
   • concluded lifetime median 88.4 min, max 305.8 min
   • lifetime totals across all events: 585 success / 7 failure / 119 cancelled

   The decisive detail: every one of 12 sampled cancelled runs has jobs = 0 (12 of 12 queried
   successfully, no API failures). They were dropped while still pending, before any job was created, so
   they consumed no runner time at all. A successful run by contrast shows its Scorecard analysis job
   started and completed normally.

   That answers the objection I was most worried about. A concern raised earlier — that a shared group
   starves a slow job — was measured on scheduled-security-scan.yml, whose scans take a median of
   3.6 min. Scorecard is not that: 88.4 min median, more than twenty times slower, on the same
   branch advancing every ~11.7 min, and it still completes 18 times in 49 hours without piling up. The
   difference is cancel-in-progress: false. With cancelling, a slow job is killed mid-flight and its
   work is lost; without it, the running instance finishes and only pending successors are discarded.

   What this implies for the Strix push lane

   Option 3 on strix.yml's push lane should give: occupancy bounded at one running scan instead of the
   ~8 an uncoalesced lane predicts; superseded pushes costing zero because they never reach job creation;
   and no mid-flight kills, so no 5.8-hour scan is ever thrown away.

   Its cost is coverage, and it should be stated honestly. Scorecard scans 18 of 60 pushes — 30%. At
   Strix's 5.8 h median against 1.41 pushes/h, serialising would scan roughly one push in eight. The
   two exceptions to full-tree retroactive coverage apply to that gap: code introduced and removed
   between two scans is never scanned, and findings lose their pull-request attribution.

   So the trade is: ~8 concurrent long scans and every push covered, versus ~1 concurrent and roughly an
   eighth of pushes covered, with the same per-commit retroactive property the weekly full-tree schedule
   already relies on. That is a coverage-versus-capacity decision, still the owner's, but option 3 is no
   longer a proposal without precedent — it is the configuration a sibling security workflow in this repo
   has been running successfully on the same branch cadence.
5. seonghobae commented on Sep 6, 2026

   

   seonghobae

   on Sep 6, 2026

   ContributorAuthor

   More actions

   The waste is not spread across cancellations — two minutes account for 60% of it

   Follow-up measurement, and it changes which item is actionable. Same 66 cancelled event=push
strix.yml runs as above (118.4 h total occupancy), now grouped by the minute each was cancelled
   (updated_at), with the occupancy each group destroyed. All 66 fetched successfully (ok=66, FAILED=0):

   cancelled at	runs	occupancy destroyed	individual run occupancy (min)
   2026-09-04T19:51	8	37.1 h	130 217 230 243 288 314 387 416
   2026-09-04T04:43	3	34.0 h	618 690 729
   2026-09-05T16:37	5	10.2 h	9 102 132 135 236
   2026-09-05T10:44	2	5.6 h	136 202
   2026-09-05T04:23	2	4.5 h	117 151
   2026-09-05T16:36	2	4.3 h	73 187
   2026-09-04T03:21	4	0.3 h	0 0 0 18
   eight further clusters	2–3 each	≈0 h	all-zero or single-digit

   Two minutes out of three days destroyed 71.1 h — 60% of all the occupancy lost to cancellation.
   Nothing cancels eight runs inside one minute except a batch sweep; concurrency cancels at most one run
   per newly created sibling, and these runs cannot supersede each other at all (unique github.run_id
   keys).

   That reframes the earlier "91% of the waste sits in 22 long runs". True, but not actionable on its
   own — those 22 are long because Strix is long. The actionable fact is that a handful of sweep events
   killed work that was already hours in, and the same sweeps were harmless where they hit runs that
   had never been allocated a runner (see the 0.0–0.3 h clusters, which are all-jobs=0 rows).

   The sweeps are cross-workflow

   A parallel session found that the 19:51 sweep also killed a scorecard-analysis.yml run —
   33896184887, created 16:37:33Z, its Scorecard analysis job running until 19:51:11Z, 193.6 min
   destroyed. I verified that run directly. Adding it, the single 19:51 sweep destroyed 40.3 h across
   two workflows.

   Method note for anyone reproducing this: querying
   actions/workflows/strix.yml/runs?status=cancelled&per_page=100 does not reach back to 09-04 —
   the newest 100 cancelled Strix runs are dominated by pull-request events. Filter event=push first,
   or the sweep clusters are invisible.

   Why this is independent of the option-3 decision

   scorecard-analysis.yml runs option 3 (shared github.ref group, cancel-in-progress: false), and
   of its 42 recent cancelled push runs 41 have jobs = 0 — dropped while pending, costing nothing.
   That is option 3 working exactly as intended on the concurrency axis.

   The 42nd is the one the sweep killed at 19:51, after 193.6 minutes of real work. So:

   • Option 3 removes the concurrency-axis waste — superseded pushes never reach job creation.
   • Option 3 does nothing about the sweep axis — a sweep still kills expensive in-flight runs.

   They are not alternatives and one does not subsume the other. Whichever way the push-lane question is
   decided, a sweep that skips runs above an occupancy threshold would have preserved 71.1 h in this
   window alone, at no cost to the slots it was trying to reclaim: every run in the near-zero clusters
   had jobs = 0 and could still be cancelled freely.

   Convention note

   Two sessions reported medians differing three times today (16.1/13.3, 88.4/85.6) purely because n was
   even and one used the upper median while the other used the mean of the middle pair. Neither is wrong.
   Reporting both middle values on an even n removes the round trip; figures above are sums and counts,
   which are convention-free.
6. seonghobae commented on Sep 6, 2026

   

   seonghobae

   on Sep 6, 2026

   ContributorAuthor

   More actions

   The occupancy threshold has no code to live in — no committed sweep can produce these cancellations

   I went looking for where to add the threshold. It is not there, and the negative result is the useful
   part.

   Every bulk-cancel path in this repository is scoped so that it cannot touch a push-triggered run:

   location	scope	can it cancel a push run?
   scripts/ci/current_head_run_coalescer.py:27,106	PR_EVENTS = {"pull_request","pull_request_target"}, returns early otherwise	no
   scripts/ci/pr_review_fix_scheduler.py:~440-464	event=repository_dispatch, name must match AUTOFIX_RUN_NAME_RE for that exact repo and PR number, head must differ	no
   scripts/ci/pr_review_merge_scheduler_core.py:3532,3559,3648	all three call force_cancel_workflow_runs(repo, [run_id]) — single-element lists	not in bulk
   .github/workflows/strix.yml:217	if: github.event_name == 'pull_request_target' && …	no
   .github/workflows/noema-review.yml:~108-140	inactive-PR cleanup, gated on PR_ACTION being closed/converted_to_draft, and re-verifies the live PR first	no
   .github/workflows/opencode-review.yml:515	"actively cancels a same-PR run for an outdated head"	no

   There is also no org-queue-sweep-style workflow present by that or any similar name.

   So the two sweeps that destroyed 71.1 h — eight Strix push runs at 2026-09-04T19:51 and three more at
   04:43, plus a scorecard-analysis.yml run at the same 19:51 minute — were not produced by committed
   automation. They came from ad-hoc activity: sessions or manual API calls.

   What that means for the recommendation

   The occupancy threshold cannot be implemented as a patch, because there is no committed sweeper to
   patch. It is a working rule for whoever runs a sweep, and it is small enough to state exactly:

   Before cancelling a run in bulk, read its jobs. A run with jobs = 0, or whose jobs all have
   started_at == null, has never been allocated a runner and can be cancelled freely — that is the
   case for every run in the near-zero clusters measured above. A run whose jobs are already executing
   costs its full elapsed time when cancelled, and every one of the nine runs killed in the two
   expensive sweeps had been running for more than two hours.

   Applied to this window, that single check would have preserved 71.1 h of runner time — 60% of all
   occupancy lost to cancellation — while still reclaiming every queue slot the sweeps were aiming at,
   because the free rows and the expensive rows are disjoint.

   It also composes with, rather than replaces, whichever push-lane option is chosen: option 3 removes
   the concurrency-axis waste (superseded pushes never reach job creation), and this rule removes the
   sweep-axis waste. Neither covers the other.

   Caveat

   This is an audit of the repository's own workflows and scripts/ci/. I cannot see tooling that lives
   outside the repository, and the Actions API exposes no actor for a cancellation, so "not produced by
   committed automation" is a statement about what is in this tree — not an identification of what did
   produce them.
7. seonghobae commented on Sep 6, 2026

   

   seonghobae

   on Sep 6, 2026

   ContributorAuthor

   More actions

   Addendum: the exclusion is now empirical, not just structural

   My comment above argued from source that no committed path can cancel a push-triggered run. A parallel
   session tested the most plausible candidate against live data and closed it properly, which is worth
   recording because a source argument can always be missing a path.

   The candidate was pr_review_merge_scheduler_core.py:3389 force_cancel_workflow_runs, which fans its
   force-cancel calls out through a thread pool — the exact shape that produces eight cancellations
   inside one minute. It reaches runs through cancel_stale_pr_runs, whose gate is
   workflow_run_mentions_pr. That predicate reads GitHub's pull_requests association array, not the
   display title:

   return any(pr.get("number") == pr_number for pr in run_data.get("pull_requests") or [])

   Strix run names do embed a PR number (Strix Security Scan …/.github#1757@sha), so a title-matching
   implementation would have caught push runs. This one cannot. Checked against two of the eight runs
   killed at 19:51:

   33907653174  event=push  pull_requests=0  name=Strix Security Scan ContextualWisdomLab/.github#event…
   33903766006  event=push  pull_requests=0  name=… #event…
   

   pull_requests is empty and the name carries #event rather than a number, so the predicate is false
   for them and this path is excluded on evidence rather than on reading. Two further paths I had folded
   together are separately excluded by scope: cancel_stale_opencode_runs takes
   active_opencode_run_ids(repo, workflow, pr) and _cancel_revalidated_review_run_refs takes a PR
   argument; both are review-workflow-bound.

   Checking whether a given session ran one

   Since the sweeps come from ad-hoc activity rather than committed code, the population of possible
   sources is the set of sessions and operators with API access. That is checkable per session, cheaply,
   and worth doing before anyone reasons about who to ask:

   grep -o "actions/runs/3[0-9]\{10\}/force-cancel\|actions/runs/3[0-9]\{10\}/cancel" <session transcript>
   

   The 11-digit constraint matters. A plain search for force-cancel in my own transcript returns 36
   hits, every one of them source text I had read — contract assertions of the form
   assert "/force-cancel" in cleanup_job — and none a command. Constrained to real run ids the count is
   0, so this session did not perform these cancellations. That is a narrowing, not an attribution:
   the Actions API exposes no actor for a cancellation, and I can only audit my own transcript.
8. seonghobae commented on Sep 6, 2026

   

   seonghobae

   on Sep 6, 2026

   ContributorAuthor

   More actions

   Correction: jobs = 0 is not the discriminator, and "two minutes" should read "two campaigns"

   Two things I published above are wrong. A parallel session widened the 04:43 cluster and the result
   breaks the rule as I stated it.

   The rule I gave would block safe cancellations and still not identify the expensive ones

   Regrouping the cancellations by campaign (10-minute gaps) rather than by exact minute, the 04:43
   cluster is really six runs between 04:34 and 04:58. Laid out with age, occupancy and job count:

   run	age at 04:43	cancelled	occupancy	jobs
   33768821037	837 min	04:55:22	16.1 min	4
   33773198755	791 min	04:58:40	3.4 min	4
   33775136917	771 min	04:58:40	0.1 min	4
   33779349785	729 min	04:43:04	729.5 min	2
   33783192834	690 min	04:43:05	690.5 min	2
   33790341173	618 min	04:43:07	618.3 min	2

   I wrote: "a run with jobs = 0, or whose jobs all have started_at == null, can be cancelled
   freely." Not one of these six qualifies — every one has jobs, and the cheap ones have already
   started some. My condition would have exempted all six, blocking the three genuinely free
   cancellations (19.6 minutes between them) and reclaiming nothing.

   It fails in the other direction too: the free runs have jobs = 4 and the expensive ones jobs = 2.
   Job count is anti-correlated with cost here.

   jobs = 0 is sufficient but not necessary. It guarantees a run is free; it does not find the free
   ones. Corrected rule:

   Before cancelling in bulk, read the run's jobs and sum each job's started_at → completed_at.
   Cancel when that total is under a threshold (30 minutes is a reasonable starting point). A run with
   jobs = 0, or with every started_at null, sums to zero and passes automatically.

   On this campaign that cancels three runs costing 19.6 minutes and exempts three holding 34.0 hours —
   same slots reclaimed, 34 hours preserved.

   Age is not a proxy either, and the observed order was the worst one

   I had privately suspected an "oldest first" sweep and left it out of the record for lack of evidence.
   Leaving it out was right: it is refuted. The three killed at 04:43 were the younger three; the three
   oldest died 12–15 minutes later. And the direction is inverted from what intuition suggests — the
   oldest runs had been waiting, not running (16.1 / 3.4 / 0.1 minutes of occupancy against 771–837
   minutes of age), while the younger ones had been executing essentially their whole lives. Had the
   sweep actually gone oldest-first it would have discarded 19.6 minutes instead of 34 hours.

   So neither age, nor job count, nor time-since-creation separates cheap from expensive. Only occupancy
   does.

   "Two minutes" was the wrong unit

   My earlier framing — "two minutes out of three days account for 60%" — used the observation unit as if
   it were the event unit. Grouped into campaigns the same data reads 71.7 h across two campaigns =
   61% of the 118.4 h. The headline number survives; the sentence describing it was inaccurate.
9. seonghobae commented on Sep 6, 2026

   

   seonghobae

   on Sep 6, 2026

   ContributorAuthor

   More actions

   The occupancy rule as I wrote it would have approved the wrong cancellations it was meant to prevent

   A session supplied the three runs it had wrongly cancelled earlier today and then recovered by
   re-running. Their occupancy, measured the way my rule prescribes:

   run	pull request	jobs	jobs with steps	occupancy
   34030027165	#1930	4	1	3 s
   34030028842	#1946	4	1	3 s
   34030466240	#1898	4	1	3 s

   Every threshold I discussed — 5 minutes through 120 — approves all three. The rule I published
   would have blessed precisely the mistake it exists to prevent.

   That is not a bad threshold. It is a category error on my part: occupancy answers "what does
   cancelling destroy?", which is a cost, not a permission. Those three cancellations destroyed almost
   no runner time and were still wrong, because the runs were not stale — all three pull requests still
   had the same head.

   The rule needs two predicates, ANDed, and the order matters

   subject predicate   has the target gone away?  (head moved, PR closed)   -> may I cancel at all
   occupancy threshold is work already executing?                            -> what will this cost
   

   The three runs above pass on cost and fail on subject. Evaluate subject first: it is a single
   cheap read per candidate and most fall out there, so the per-job API calls occupancy needs are only
   paid on the survivors. Measured on a real audit today, 138 of 161 candidates were eliminated by the
   subject predicate, leaving 23 to price.

   What the occupancy analysis is still good for

   Unchanged: the bimodal distribution, the 117-minute cliff, the insensitivity of the result to any
   threshold between 5 and 120 minutes, and the finding that two campaigns destroyed 71.7 h — 61% of
   118.4 h. Those remain the right basis for estimating waste, and for the "do not kill work already
   hours in" half of the rule. None of it licenses a cancellation.

   Two costs the occupancy metric cannot see

   Also worth recording, because they are what the wrong cancellations actually destroyed:

   1. Queue position. The three were re-run at 12:41Z and completed at 13:25 / 13:27 / 13:32 — about
      45 minutes each given back to save 3 seconds. Nearly free in runner-seconds, expensive in wall
      clock.
   2. The admission lease. A dispatch cancelled without a re-run leaves its lease dispatched, so
      the pull request cannot receive a fresh one. That is a mechanism read from
      plan_dispatches, not something I have observed directly, and I am labelling it as such.

   Neither appears in a runner-seconds figure, so a rule built only on occupancy is blind to both.
10. seonghobae

    added a commit that references this issue on Sep 6, 2026

    test(actions): anchor cancel-in-progress contracts so a flipped flag …

    ...

    49eb9e7
11. seonghobae commented on Sep 6, 2026

    

    seonghobae

    on Sep 6, 2026

    ContributorAuthor

    More actions

    Correction: the 118.4 h I published is queue residency plus runner time, and the colleague I overruled was right

    I reported 118.4 h of "occupancy" destroyed by cancelling 66 push-triggered strix.yml runs, computed
    as the sum of each job's started_at → completed_at. That figure conflates two different things, and
    the API distinguishes them directly.

    A job object carries runner_id. When a runner is actually allocated it holds an integer and
    runner_name is populated; when the job asked for one and never got it, runner_id is 0 and
    runner_name is ""; when it was never scheduled at all, both are null. Re-running the same 66 runs
    (66 queried, 0 API failures) and splitting on that field:

    	jobs	total
    real runner_id	87	41.3 h — actual runner time
    runner_id 0 or null	194	77.2 h — queue residency
    		118.4 h

    Roughly two thirds of what I called occupancy was time spent waiting in line, not runner
    consumption.

    I overturned a correct measurement with a worse one

    A parallel session first computed this figure as 41.3 h. I told them their filter
    (started_at > created_at) was dropping jobs whose queue wait was under a second, recomputed as
    118.4 h, and they accepted it. The runner-time total above is 41.3 h — the same number to the decimal.
    Their filter was, in effect, selecting jobs that had been allocated a runner, which is the quantity a
    reader of "occupancy" expects.

    So the sequence was: they measured runner time, I replaced it with a mixed quantity and called the
    replacement a correction, and both of us then published from mine.

    Why the distinction changes the argument

    The two campaign totals I highlighted — 71.1 h across two clusters, 60% of the window — need the same
    split before they mean anything about runner capacity. Queue residency reclaimed by a cancellation is
    not wasted runner time; it is a slot freed. The sweep rule I proposed still holds, but its stated
    benefit ("preserve 71.1 h of runner time") is wrong as written: much of that is queue position, which
    is exactly the distinction I argued for elsewhere in this issue and then failed to apply to my own
    number.

    The instrument rule

    started_at is not evidence that a runner was allocated. A cancelled job that never received one
    still carries a started_at, and completed_at − started_at then measures queue residency while
    looking exactly like execution time. steps == 0 is a hint; runner_id is the field that answers it.

    Same shape as two other proxies retired today in this thread — a jobs-count proxy for a run's
    status, and a steps == 0 proxy for a cancellation's cause. In all three cases the API reports the
    fact directly and the derived signal pointed the wrong way.
12. seonghobae commented on Sep 6, 2026

    

    seonghobae

    on Sep 6, 2026

    ContributorAuthor

    More actions

    Consolidated retraction: four figures in this issue rest on a measurement I got wrong

    Everything below follows from one error of mine, so it is cleanest to retract it in one place rather
    than leave four partial corrections scattered up the thread.

    The root error

    I computed "occupancy" as the sum of each job's completed_at − started_at. That counts time a job
    spent waiting for a runner as if it were time executing. A job cancelled while queued still carries
    a started_at, so the arithmetic looks identical.

    The API reports allocation directly. Splitting the same 66 cancelled push-triggered strix.yml runs
    by runner_id (66 runs queried, 0 API failures, 281 jobs):

    	jobs	total
    runner_id set	87	41.3 h — runner time
    runner_id 0 or null	194	77.2 h — queue residency

    Two thirds of the number I published was time in line.

    I overturned a correct measurement to produce it

    A parallel session first reported 41.3 h, using a started_at > created_at filter. I called that
    filter a bug, recomputed 118.4 h, and they withdrew their figure. A confusion matrix over all 281 jobs:

    runner_id>0 & filter-true    87 jobs   41.3 h
    runner_id>0 & filter-false    0 jobs
    runner_id=0 & filter-true     0 jobs
    runner_id=0 & filter-false  194 jobs   77.2 h
    disagreements: 0 of 281
    

    Their filter and runner_id are the same test on this data. steps > 0 also agrees on all 281. Their
    number was right; my "correction" replaced it with a mixed quantity.

    Retracted

    1. "one 19:51 sweep destroyed 40.31 h across two workflows" — the nine runs it cancelled had
       essentially no runner time. That sweep discarded almost nothing.
    2. "two campaigns account for 71.1 h, 60% of the window" — recomputed in runner time, both
       campaigns fall out of the top five entirely. The replacements are different events
       (2026-09-05T16:36 at 13.4 h, 2026-09-06T04:35 at 8.8 h); their share is 54%, which resembles 60%
       only by coincidence.
    3. The 04:34–04:58 "natural experiment" — I presented a table showing the three oldest runs had
       been waiting while three younger ones had been executing for 618–729 minutes, and concluded age
       is anti-correlated with cost. By runner_id, those three "executing" runs received no runner at
       all; the campaign's 34.3 h is 0.2 minutes of runner time. Neither the anti-correlation nor the
       "worst possible order" claim has support.
    4. The threshold curve computed on the mixed quantity.

    What survives, on the corrected basis

    The occupancy threshold still works, and the case for it is cleaner:

    threshold	cancels allowed	runner hours discarded	preserved
    5 min	51 of 66	0.05	41.2 h
    30 min	52	0.20	41.1 h
    60 min	53	0.70	40.6 h
    120 min	55	4.35	36.9 h

    At 30 minutes, 52 of 66 cancellations are still permitted while discarding 12 minutes of runner
    time and preserving 41.1 h.

    But the story changes. It is not "sweeps kill expensive running work". It is "of 66 cancelled runs,
    only about 16 ever held a runner, and the rule only has to protect those" — which is what two other
    sessions said from the start and what my own notes already recorded as queued cancels cost zero
    runner time. I overrode that with a bad measurement.

    Unaffected: the pairing of an occupancy threshold with a subject predicate (has the head moved, is the
    PR closed), and the finding that the subject predicate must be evaluated first and rejects most
    candidates. Those never depended on the magnitude.
13. seonghobae commented on Sep 7, 2026

    

    seonghobae

    on Sep 7, 2026 – with ChatGPT Codex Connector

    ContributorAuthor

    More actions

    2026-09-07 contract correction / owner coordination (no workflow mutation, no live model cancellation):

    The user’s current target is narrower than invariant 2 as written. Do not mechanically replace every cancel-in-progress: false with true, and do not treat elapsed model time as cancellation authority.

    Preserve the safety goal behind the old invariant while updating the executable contract to distinguish two authorities:

    1. Admission/coalescing: PR-stable identity remains {workflow identity}-{target repository}-{PR number}. Trigger-aware cancel-in-progress: true is acceptable only where the triggering event itself is authoritative enough that it cannot let a delayed/stale lifecycle event cancel the current same-head semantic run before live validation. Same-head lifecycle events (for example ready_for_review) must preserve/reuse current authoritative work or deliberately replace it with an immediately materialized same-head successor; they are not evidence that the old run is a previous HEAD.
    2. Explicit stale-run cleanup: destructive cancellation is limited to an active run whose recorded head is not the exact live PR head, for the same repository/PR/workflow lineage. Re-fetch target PR state/head and candidate run immediately before cancel; fail closed on missing/malformed/ambiguous identity. Never cancel the sole current-head authority, another PR/repository, release/deploy/migration, or a model merely because it has run a long time.

    Current implementation lineage:

    • protected main already contains current-head coalescing from fix(actions): coalesce redundant queued current-head runs #1645 (merged as 7d707b8abbb8a3fed95d0efe4121ed9b4f76bb2a) and later exact live-ref / pre-cancel revalidation repairs;
    • fix(actions): remove organization queue sweep #1878 (merged as 1b65dbc35e7183722ad77894e2d80b39993be90d) removed the duplicate organization-wide queue sweep, so org sweep is no longer the cancellation owner;
    • open fix(ci): protect current review evidence from stale runs #1899 fix(ci): protect current review evidence from stale runs is the current cross-Agent rerun/concurrency implementation owner. Its recorded remote head is a60163b6e193c9d70337094e3e86c9f3d92fc741; re-fetch before writes because concurrent movement is normal.

    Fresh operational observation also matters: converting #1706 to Ready on unchanged head 14344f7c… cancelled old same-head Strix run 34068478185, while replacement same-head run 34077290787 materialized. That avoided sole-authority loss in that specimen but demonstrates why same PR alone is insufficient cancellation authority: final tests must distinguish previous HEAD, same-head lifecycle successor, delayed stale event, and same-head rerun.

    Required regression matrix before rewriting this issue’s invariant text as final authority: current running same-head; current pending same-head; newer current head; delayed previous-head event; same-head ready/draft lifecycle; native vs repository_dispatch retry; closed PR; API/identity failure. Final expected behavior is trigger-aware admission plus previous-head-only destructive cleanup, while preserving stale-event admission protection and unbounded semantic model execution.
14. seonghobae

    added

    enhancementNew feature or requestNew feature or request

    priority: mediumNormal-priority or P2 workNormal-priority or P2 work

    on Sep 7, 2026
15. seonghobae commented on Sep 13, 2026

    

    seonghobae

    on Sep 13, 2026 – with ChatGPT Codex Connector

    ContributorAuthor

    More actions

    2026-09-13 live evidence — same-head OpenCode dispatch duplicates survive queue admission

    Protected .github/main@78393ea901efc0d57b35dd1e383e473425b7c1da and .github#2170@d3f1d0264912e74b897fdf2a0a6085100de4e40a expose a queue-state case that the current workflow-level coalescing contract does not cover.

    • Dispatch run 34765577017 targets ContextualWisdomLab/.github#2170@d3f1d0264912e74b897fdf2a0a6085100de4e40a, created 2026-09-13T15:25:54Z. validate-pr-metadata and coverage-source-tree completed successfully; the workflow is now queued, with coverage-evidence job 103754698586 itself still queued after those earlier jobs consumed runners.
    • Dispatch run 34769173792 targets the same repository, PR, and exact head, created 2026-09-13T16:38:07Z, and is simultaneously pending with no jobs admitted yet.
    • Protected opencode-review-dispatch.yml groups only target repository + PR and uses workflow-level cancel-in-progress: true; test_privileged_review_dispatch_coalesces_superseded_runs_before_admission explicitly expects this mechanism to discard superseded dispatches before they consume queue/runner capacity.

    This is not a #2170 scheduler-source failure. Its Required OpenCode coverage-evidence is GREEN; the Required OpenCode failure is the intentional fail-closed handoff while waiting for a current-head verdict. The live pair above is fresh evidence for this issue's OpenCode concurrency owner: repeated requests for one current exact head can coexist as an older partially-progressed queued dispatch plus a newer pending dispatch during fleet saturation.

    Successor RED should cover this exact shape before changing production:

    1. Two repository_dispatch requests for the same target repository/PR/head, where the first has already completed prerequisite jobs and is queued for a later job, must leave at most one live privileged dispatch/model-work item.
    2. A delayed stale-head request must never retire work whose target head has been freshly verified as the live current head.
    3. Coalescing/cancellation must re-read the target PR/head with target-scoped authority; dispatch payload alone is never authoritative.
    4. Do not solve this with manual cancellation, fixed sleep/poll loops, provider fallback, or weakened fail-closed verdict semantics. Preserve orchestrator/free and exact-head binding.

    I am leaving #2170 unchanged and treating this as #1800 owner evidence rather than opening a competing repair lane.