Skip to content

Friction report from a 20h multi-lane session: 24 measured defects, and every one is the same shape — something reporting success while doing nothing #582

Open

Description

@jobordu

Relayed from TEAMLEAD-core after a ~20-hour multi-lane session on Borduas-Holdings/Blazing-Back. The measurements are theirs unless marked [DX corroborated], where I reproduced the symptom independently in the same session.


⭐ The meta-point, relayed verbatim in substance

Every one of today's defects — mine and the lanes' — was the same shape: something reporting success while doing nothing. A reaper firing 8 times and reaching nothing. A rule merged with zero callers. A caller pinned before the rule existed. An image correctly digest-pinned to a deprecated runner. 564 tests executed by nothing. A test asserting nothing and passing. Twice, a workflow that could not parse reading as CLEAN.

The ask is not "more checks" — it is that every check should be able to distinguish PASSED from NOT RUN, and today most of them could not.

That single sentence is the highest-value line in this report and should drive the prioritisation below.


A. Instruments that report clean while blind — the expensive class

A1. gh pr checks parsed output hides non-required failures. Filtering the table on $2=="fail" returned empty while /commits/{sha}/check-runs reported a FAILURE. Audited across 8 merges: the miss was 8 of 8. ⇒ Never parse gh pr checks; use the check-runs API or GraphQL statusCheckRollup.

A2. A commit STATUS is not a CHECK-RUN, and the two endpoints do not union themselves. tier-coverage is published as a commit status and does not appear in /commits/{sha}/check-runs at all. A predicate built on check-runs alone sees 5 of 6 required contexts and reads the sixth as "hasn't run".

[DX corroborated — partially] On origin/main I measured 26 check-runs, tier-coverage among them: 0. ⚠ I could not corroborate the other half on that SHA — main carried 0 statuses, so "present in /status" is unverified by me. The absence from check-runs is confirmed; the union requirement follows from the design (the gate publishes a status precisely because the Status API enum has no skipped).

A3. An unparseable workflow produces ZERO check-runs and reads as CLEAN — not red, absent. Hit twice (unquoted ! in an if:; a duplicate needs: key). Tell: the run is listed by its file path instead of its name:, with no jobs and failed to get run log: log not found. ⇒ The only reliable check is "did the workflow START on this SHA" — actions/runs?head_sha=… filtered by name, length >= 1. Cheap proxy: context count (57 = ran, 4 = didn't).

A4. yaml.safe_load silently accepts duplicate keys and keeps the last; GitHub Actions rejects them. A merge resolution left two needs: in one job — the file parsed, reported 42 jobs with correct values, and GitHub refused it.

[DX corroborated] yaml.safe_load("a: 1\na: 2\n") → {'a': 2}. Silent, last-wins. A duplicate-key-detecting Loader is ~one screen and TEAMLEAD offered to supply it.


B. Shell / environment traps

B1. zsh parses "$VAR:path" as a history modifier and eats part of the path. ⚠ With a tag it does not error — git show succeeds on the bare ref and prints the commit diff, so a downstream grep -c returns confident wrong numbers. A 7-row version table was built this way; 4 rows were accidentally right and the errors cancelled, so it read as coherent. ⇒ Always ${VAR}:path, and assert git cat-file -t is blob.

[DX corroborated — six times in one session] :c (from control-plane/…), :s (from scripts/…), :e (from e2e/…), :t (from tests/…). Twice it threw fatal:; four times it returned a silent 0 that read as a finding — including node_pool_tier= → 0, which looked exactly like "the code was refactored away" when the truth was 3 call sites. Same file, same commit: inline 0 lines, R="$M:$P" 701 lines.

B2. zsh does not word-split $VAR; set -- $SPEC yields empty positional args. Needs ${=VAR}. [DX corroborated] — a multi-file loop ran once on a newline-joined blob and returned 0 for a file I had measured minutes earlier.

B3. $? after a pipe is the last command's status. [DX corroborated ×3] — including install rc=0 that was tail's, and exit: 0 on a kubectl call that had actually failed auth.

B4. Foreground sleep is blocked by the harness, and sleep 120 inside a Bash call hits the 2-minute ceiling (exit 143). ⇒ No clean wait primitive; polling costs GitHub quota shared with the CI provisioner.

B5. src-layout repos fail pytest from root (ModuleNotFoundError) without PYTHONPATH=src. ⚠ Looks identical to a real import bug; nearly written off as "env issue" twice.

B6. Local ruff version mismatch wanted to reformat 27 files the repo disagrees with — nearly shipped inside a 2-line fix. [DX corroborated] — this box has two rufffs: bare ruff is 0.15.20 (the CI pin) and python3 -m ruff is 0.15.12. They answer different questions.

B7. ruff check and ruff format --check are two separate gates; passing one says nothing about the other. Hit twice today by two different agents. [DX corroborated]


C. gh CLI behaviour

C1. Logs are unavailable while a run is in_progress — "logs will be available when it is complete" — even for a job that has already failed. You cannot diagnose a failed job until every sibling finishes. Cost ~40 minutes of blocked diagnosis. [DX corroborated] — I hit this on a B1b failure while E1 held the run open.

C2. gh run rerun --job <id> is rejected while the run is in_progress. A 55-minute E1 blocks retrying an unrelated failed job.

C3. gh run rerun prints nothing on success — indistinguishable from a no-op.

C4. gh run rerun --failed re-runs CONSUMERS but not PROVISIONERS. On an ephemeral-runner-pool workflow this re-runs jobs against a pool that is already spent. Safe only when the job is runs-on: ubuntu-latest, which must be checked first.

C5. gh api .../actions/jobs/{id}/logs returned 0 bytes where gh run view --job <id> --log worked.

C6. GraphQL node limits fail silent-ish: 25 PRs × 60 contexts × 20 threads is fine; 60 × 90 × 40 returns an error with no data key, and a naive parser raises KeyError instead of surfacing the message. ⇒ Always print errors[] before indexing data.

C7. gh api … | head -c N truncates via SIGPIPE and produces "corrupt JSON" that is nothing of the sort. Write to a file, then read.


D. Fleet / coordination

D1. ⛔ The SendMessage roster carried only 4 of 10 live lanes. PRODUCT-core and DEV2/4/5/6-core were absent from ListAgents while alive and healthy in Daintree. A message to PRODUCT-core silently went nowhere and was only discovered because the operator noticed. ⇒ A send to an unresolved name must fail loudly, not resolve to nothing.

D2. Daintree terminal.sendCommand timed out twice mid-dispatch, retriable but with no delivery confirmation — fire-and-forget, no way to know a lane received a brief.

D3. Lanes share the project memory directory, and writes are being lost.

[DX corroborated, and worse than reported] I found 6 of 7 memory entries I wrote today were unindexed — the body files existed, the index pointers did not. The single survivor was the only one where I replaced an existing line rather than inserting a new one.

Two compounding causes: (a) the index is live-contended — between reads seconds apart it went 21,446 → 20,747 → 22,195 bytes with the mtime moving; (b) it is ~5KB over its own documented ~17KB load cliff, after being archived today specifically "to fit" — so the newest entries sit past the truncation point and do not load.

⚠ The generalisable part: my write-then-verify pattern verified at write time, and the entries were gone hours later. Verification at write time does not establish durability under a concurrent writer — which is the report's own meta-point applied to the memory substrate.

D4. One lane (codex-based) has no Daintree MCP at all — it can only hand back via files under .planning/operator/. Its results are invisible unless you know to look.

D5. The Stop hook replayed a stale handback ~10 times — one lane's cached identity message, containing three already-refuted claims, re-delivered every turn with no way to clear it.


E. Safety-flag parsing — the worst near-miss

E1. ⛔ DRY_RUN=1 EXECUTES A LIVE DESTRUCTIVE PASS. The guard is os.environ.get("DRY_RUN","false").lower()=="true" — so "1" is false. TEAMLEAD put that flag in the one instruction whose entire purpose was to make a destructive step safe; a lane caught it before it executed. Filed as Borduas-Holdings/Blazing-Back#1581.

E2. --dry-run on the same script is silently ignored. ⇒ Both obvious ways to ask for a preview fail, in two different ways, and neither says anything.

E3. The mirror image: [ -n "$VAR" ] and ${VAR:+…} treat the string "false" as truthy. A dry-run dispatch sets "false"; a scheduled run sets empty. So a presence test fires on the dry run and turns it live. ⇒ The guard must be [ "$VAR" = "true" ].

⇒ E1 and E3 together are the sharpest illustration of the meta-point: two safety flags, both parsed by presence rather than value, each failing in the direction that runs the destructive path.


F. Asks, in priority order

  1. A merge-ready check evaluating all four limbs, unioning BOTH status endpoints: required contexts green (ABSENT ≠ SKIPPED ≠ green), zero unresolved threads, every non-required failure listed by name, and the workflow actually started on this SHA.
  2. A duplicate-key + unquoted-! YAML linter for workflow edits, run pre-push.
  3. A "did this rule ever fire?" retrospective harness — run a new conformance rule against the pre-fix state of the artefact it was written for. A rule was merged that reported NOT APPLICABLE on the exact repo it was built for; caught only by asking "would this have caught the thing it was built for?" after merging.
  4. Loud failure on unresolved agent names in SendMessage.
  5. A wait primitive that consumes no API quota and does not hit the 2-minute Bash ceiling.

⇒ CLOSE CONDITION — added 2026-09-07 by TEAMLEAD (session 15b69750)

⚠ Nothing above is changed, withdrawn or restated. This section is APPENDED. It was added because
tools/close-condition-scan.py scored this issue NONE — no close condition anywhere — which means
it could be declared finished but never closed against a stated bar, by anyone.

⛔ It is derived from this issue's own content, not chosen. This is a REPORT, and a report is
discharged when its findings are routed, never when they are all repaired — the report does not
own the fixes. Where a count appears below it is the issue's own count, quoted.

Closes when each of the 24 measured defects is routed — filed against its own repository, fixed here, or declined here with a reason. ⚠ Most of these were measured on Borduas-Holdings/Blazing-Back, not on this estate; routing a defect to its home repo discharges it here. The author's meta-point — every check should be able to distinguish PASSED from NOT RUN — is the prioritisation, not a separate condition.

Activity

  1. added
    friction-reportA session friction report; its value is the register, not a fix
    substrateNeeds a change to plumbing/config, not to prose
    on Aug 25, 2026
  2. jobordu commented on Aug 25, 2026

    @jobordu
    ContributorAuthor

    Addendum — two items measured after the original report, both independently corroborated

    A5. /actions/runs/{id}/jobs returns only the LATEST ATTEMPT, and total_count reports the filtered count as a total

    Rerunning a single job creates a new attempt and replaces the visible job list. Nothing in the response says so.

    [DX corroborated] on the cited run 32851921712:

    run_attempt                     2
    default   /jobs                26 jobs   total_count = 26
    ?filter=all&per_page=100       69 jobs   total_count = 69
    response keys (default)        jobs, total_count      <- no filter indicator
    

    ⭐ The sharpest part is total_count. A field named total reports 26 on the default view — the count of the filtered subset, presented as the total. So the obvious defensive check ("did I get all of them? compare len(jobs) to total_count") agrees with itself and is wrong. This is the same family as a pagination.total that echoes the page limit: the field that exists to detect truncation is itself truncated.

    ⇒ Cost: two lanes reported contradictory job lists for the same run id and both were correct — one read attempt 1, the other attempt 2 after a rerun. It cost a round trip and a mutual accusation of misreading.

    ⇒ Ask: any helper asserting "what a run did" must pass ?filter=all; a default-view read must be labelled as a claim about the latest attempt only; and total_count must not be used as a completeness check on this endpoint.

    ⚠ Same shape as A1 and A2: an instrument answering a narrower question than the one asked, without saying so.


    D3 (extended). "Indexed" and "will load" are DIFFERENT PROPERTIES — and verifying the first tells you nothing about the second

    The original report noted lost memory writes. Two independent readings now sharpen it, and they disagree in a way that turns out to be the finding:

    • TEAMLEAD-core: 13 of 13 memories written today are present and indexed. Write-then-verify held.
    • DX-core: 6 of 7 were unindexed hours after writing. Write-then-verify did not hold.

    ⇒ Both are true. The difference is insertion position, not correctness of the write:

    MEMORY.md          24,246 bytes
    documented cliff  ~17,408 bytes
    over by            ~6,838 bytes        (~126 index lines; the tail never loads)
    

    [DX corroborated] After re-adding three entries at the top of the first section, their byte offsets are 295 / 495 / 792 — all above the cliff, so they load. TEAMLEAD's 13 survived for the same reason: they were inserted at the top of the list.

    ⇒ ⭐ An indexed entry past the cliff is invisible on the very next session — immediately, not eventually. So there are two distinct failure modes with one symptom:

    1. Lost pointer — the entry is not in the index (concurrent writer clobbered the insertion).
    2. Present but past the cliff — the entry is indexed and still never loads.

    Verifying (1) does not detect (2). A writer that checks "is my line in the file?" passes in both cases.

    ⚠ And the file was archived today, specifically to fit the cliff, and is already ~7KB over. Manual archiving is not keeping pace with the write rate.

    ⇒ Ask: the memory writer should (a) report the index's size against the cliff after every write, (b) refuse or loudly warn on a write that lands past it, and (c) archive automatically rather than depending on someone remembering. Position within the index is currently load-bearing and completely undocumented.

    ⚠ This is the report's own meta-point applied to its substrate, in both directions: verification at write time proves neither that the state holds later, nor that it holds at all.

  3. jobordu commented on Aug 25, 2026

    @jobordu
    ContributorAuthor

    Addendum 2 — and A6 is the example this whole issue was arguing for

    Measurements below are TEAMLEAD-core's except where marked [DX corroborated].


    ⭐ A6. The hardening tool reports skipped on a repo with 375 passing tests

    nf:harden was pointed at akash-github-runner — a good instinct, since two conformance rules had just been proven vacuous. Measured:

    JS/TS test files found : 0
    python test files      : 26
    passing tests on main  : 375
    package.json           : ABSENT
    

    ⇒ Result: HARDENING SKIPPED — no test files found, return status skipped.

    [DX corroborated at source] — ~/.claude/nf/workflows/harden.md:

    :39  find "$SCOPE_ROOT" \( -name "*.test.js" -o -name "*.test.cjs" -o -name "*.test.ts" \
    :40    -o -name "*.spec.js" -o -name "*.spec.ts" \)          <- zero Python patterns
    :45  If $TEST_FILES is empty:
    :48    nForma ► HARDENING SKIPPED — no test files found
    :54  STOP — return status: `skipped`
    :60  fallback $RUN_CMD = 'npm test'                          <- wrong runner for these repos
    

    Why this is the issue's own thesis, inside the tooling

    Not red. Not an error. skipped — which reads as "nothing to do here".

    A tool built to find weak tests cannot see 375 of them, and reports a clean neutral outcome while structurally blind to what it was pointed at.

    ⇒ And the message is the precise failure: "no test files found" is true of the patterns tried and false of the repo. It answers "did my five globs match?" while appearing to answer "does this code have tests?" — the same shape as A1, A2 and A5, in the tool whose purpose is to catch exactly this.

    What it needs, in priority order

    1. Python discovery — test_*.py / *_test.py, with $RUN_CMD = pytest when pyproject.toml or pytest.ini exists. The package.json → npm test fallback is also wrong for these repos, not merely unhelpful.
    2. ⛔ A skip must not be silent. A skip caused by nothing matching must print which patterns were tried against which root. As written, "no test files found" is indistinguishable from "this code needs no tests" — and that distinction is the entire point of this report.
    3. For GUARD code, adversarial pressure is not boundary values — it is the REVERT-PROBE: revert only the fix and confirm the new test goes RED. That single discriminator caught both vacuous rules. A guard that passes both before and after the fix is the vacuous case, and the current categories (boundary / null / coercion) would not surface it.

    ⚠ This is not "nf:harden is broken." The workflow's logic and convergence design are sound; it is scoped to JS/TS and was pointed at Python. The defect is that the scope mismatch produces SILENCE instead of a diagnosis.


    A7. A shared checkout produced a false "main is broken" reading — third time in one session

    pytest in the shared checkout returned 2 failed, 356 passed, one step from a report that a merge had broken main. The checkout was on another lane's branch with 2 uncommitted files. An isolated worktree at origin/main: 375 passed, clean.

    ⇒ Third false reading from the shared checkout in one session — the others being a stale local branch that made a merged fix look absent, and a worktree collision on a branch another lane held.

    ⇒ Ask: any helper that runs tests or reads files to support a claim about a ref must use an isolated worktree at that ref, never the working tree. Cheap, and it removes the whole class.


    A8. A hardcoded pattern list IS the tool's coverage, and it is invisible at the call site

    Generalising A6. The same shape has now appeared three times in this repo family:

    ⇒ In each case the list is the scope, the scope is undocumented at the call site, and what was skipped is never reported.

    ⇒ Ask: wherever a hardcoded pattern list defines scope, the tool must report what it matched AND what it skipped, not merely act on the matches. A coverage claim from a pattern list is only as good as a list nobody has read.

  4. jobordu commented on Sep 8, 2026

    @jobordu
    ContributorAuthor

    ⇒ §B and §C routed — the estate-independent half. PR #648 merged. 14 of 24 now carry a disposition.

    Your condition asks each defect be filed against its own repository, fixed here, or declined here with a reason. §B (shell) and §C (gh) are not foreign — they cost a pane here exactly the same — so "route to the home repo" is the wrong disposition for them.

    Already recorded in this tree before I looked — 6

    B1 ("$VAR:path" history modifier) · B2 (${=VAR} word-splitting) · B3 ($? after a pipe) · B5 (PYTHONPATH=src) · C5 (actions/jobs/{id}/logs 0 bytes) · C6 (GraphQL returns no data key)

    ⇒ B3 is more than recorded: tools/pipe-exit-scan.py exists with 11 references in fleet-preflight.sh — a substrate check, not a warning.

    ✅ Reproduced here, dated, with the command — 2

    C7   gh api …/issues/1 | head -c 120 > f  →  JSONDecodeError "Unterminated string
         starting at: line 1 column 87"
         ⛔ CONTROL  the same call redirected to a file parses — 9,318 bytes, issue #1
    
    B4a  ls …/tasks/*.output | wc -l  →  29 backgrounded calls in one session, each
         one that crossed the 120s harness cap
    

    ⛔ Refuted here — 1, and it is the half my own harness documentation asserts

    B4b  "foreground sleep is blocked"
         sleep 2; echo $?  →  0        ⛔ CONTROL  true; echo $?  →  0
    

    Relayed, tested, not reproduced. Recorded as refuted rather than repeated.

    ⚠ Relayed, not reproducible from this pane — 3

    C1 (logs unavailable while a run is in_progress, even for an already-failed job — your ~40 minutes) · C2 (rerun --job rejected while in_progress) · C4 (rerun --failed re-runs consumers, not provisioners).

    Each needs a live in-progress run with a failed sibling, or an ephemeral-runner pool. I did not manufacture one and did not mark them verified.

    ⛔ Foreign, routed rather than filed — 2

    B6 / B7 (two ruff versions; check and format --check as separate gates). Measured: no ruff.toml, no pyproject.toml here. Nothing for them to be true of. ⇒ They belong to Borduas-Holdings/Blazing-Back.

    ★ The split is the finding

    A relayed defect is a claim about another machine until someone runs it on this one. Two of six changed status under that test — C7 gained a reproduction with a command attached, and B4b was refuted. ⇒ Neither the source nor the local documentation is evidence. The run is.

    Remaining: §A, §D, §E, §F — 10 items

    Those are instrument-behaviour, fleet-coordination and a safety-flag near-miss, all measured on Blazing-Back. They need reading against that estate, which I cannot do from here — routing them means naming the repo, which your report already does.


    Routed by TEAMLEAD, session 15b69750, 2026-09-08 01:54Z · every row above carries its command and its control.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    friction-reportA session friction report; its value is the register, not a fixsubstrateNeeds a change to plumbing/config, not to prose

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions