日本語版: README.ja.md
claude-code-harness — the AI tried to quit early 150 times in under 4 days, so we built a mechanical gate
These tools are in daily use on a project that runs Claude Code unattended via /loop. Every number below is measured from that operation, including the ones that make it look bad.
When you run Claude Code unattended on a loop, it repeatedly happens that the AI decides "that's enough for today" and ends its turn while more than half the budget (tool calls, wall-clock time) is still available. What follows are the six scripts we wrote to catch that failure mechanically. They use only the Python standard library (zero dependencies).
| What | Count |
|---|---|
Recorded gate firings (RUNS.tsv, 2026-09-10 16:37 → 09-16 = 6 days) |
423 |
| …of those, firings while a live unattended-run lock was held | 151 |
| → Turns the gate kept going because budget remained | 150 (09-10: 1 / 09-11: 14 / 09-12: 39 / 09-13: 43 / 09-14: 53) |
| → Turns it let end because the budget was spent | 1 (09-11 09:42, at T=8/8 — that quit rule has since been deleted; see lesson 5) |
| Firings with no lock held (interactive turns, passed through) | 272 |
| Firings on 09-15 and 09-16 where the gate was armed | 0 of 84 — the instrument went dark; see the next section |
| …of the above, a malfunction | 1 (it read a stale lock as live; fixed — it is the single 09-10 entry) |
| Times we tried to fix the problem by adding a natural-language rule | 2 — neither prevented the next one |
| Did the gate's own quit rule ("let it stop after 8 tries") have any basis | No (it ended a run at 1h53 of a 4h budget; the try-count is no longer an exit — see lesson 5) |
| Bundled tests (run on 2026-09-16) | 38 tests in 6 files, all passed, 0 failures |
The numbers we can't claim. (1) Not all 150 were "stopping too early." The lock labels confirm every one happened inside an unattended session, but an unattended session still pauses to report to a human, and the gate does not tell the two apart — we have no instrument for that yet. (2) Whether blocking the stop produced better work is not measured automatically; by hand, the continued turns produced defect fixes, shelf measurements and competitor observations. (3) The first version of this table read "times it helped in production: 0". We publish the numbers that flatter us and the ones that don't. If you see a similar tool, ask for these numbers — they are usually missing.
On 09-15 we changed how the unattended loop runs. Instead of one long session working down a queue, a main line now hands each unit of work to a fresh subagent, and subagents have no Stop hook.
The gate still fires in the main line — 62 times on 09-15 and 22 on 09-16. The run-lock it keys on is no longer held by those turns, so it was armed 0 of those 84 times and every one passed straight through. No test failed. No error was logged. The check simply stopped applying, on the day the thing it was checking changed shape.
So the number that tells you this gate is working is the armed count, not the firing count. A gate that fires 84 times and blocks nothing looks identical, in any dashboard built on firings, to a gate over a session that never quits early. We only caught it because the per-day column in the table above went to zero while the loop was still running.
If you run a Stop hook: how many times has it fired, and how many of those were armed? Two counts and the window they cover. Paste them in this thread — one repository's log is not a number. (Until 2026-09-17 this sentence said "we will keep the tally in that thread" and linked to the discussion index. There was no such thread. See point 4 below.)
This shelf's own numbers — 3 people came in two weeks, and nobody opened a single tool folder (measured 2026-09-16)
Before measuring whether the tools are any good, we counted whether anyone arrives at all. These are this repository's own numbers from GitHub's Traffic API. The window is exactly what the API returned: 2026-09-01 to 09-14, 14 days (09-15 and 09-16 aren't in it yet).
| What | Count |
|---|---|
| Views | 5 / 3 unique |
| Referring sites | 0 |
| Pages opened | Overview 4 (3 unique) and one Discussion 1 (1 unique). Nothing else — in this window not one tool folder was opened |
| Clones | 70 / 30 unique. But all of them land on the two days right after publication (23 on 09-12, 12 on 09-13). Every one of the following 12 days is 0 |
We don't know how many of the 3 were us (09-12 is the day this repository was created). Most of the folders now here were published after this window, so they aren't in these numbers yet. We take the same four rows again on 09-29.
Three things follow, and two of them broke our own instruments.
- Zero stars is not "the tools were not wanted." It is "nobody opened them." Their quality has not been measured even once. Adding the next one to the same place returns the same zero
- Counting "unique cloners" as a reaction was wrong. Those 30 appear only on the two days after publication and are 0 for the 12 days after — that is not the shape of human interest, it is the shape of machines copying a new public repository. We removed that column from our own reaction test
- The number of things you publish does nothing where the inbound traffic is zero. Everything we published in those two weeks moved the table above by nothing
- We had placed 21 requests to readers across this repository, and none of them has been answered. We counted them on 2026-09-17: 21 sentences asking a reader to paste something or open an issue (10 on the Japanese pages, 11 on the English ones). Measured the same day: zero issues, open or closed, have ever been created; all three discussions were opened by us, and the 3 comments and 3 upvotes on them come from our own account and the owner's. Re-pulling
/traffic/popular/pathsthat same day still returned exactly two paths — the Overview (5 views / 4 unique) and one Discussion (1 / 1) — so 17 of the 21 requests sit on pages that were opened zero times. And of the 4 that sit on this Overview, the one page that does get opened, exactly one led where it said it did: one promised "we will keep the tally in that thread" and no such thread existed; one said "paste that block into an issue" and linked to our own documentation page instead of an issue form; and the template file built for that third request was sitting in the repository with no link to it anywhere in the tree. All three are fixed as of today. We also openedissues/newandissues/new/choosewhile signed out: both land on a sign-in page (a public repository is readable by anyone and writable only with an account). So the cost of a request is not the one sentence we wrote — it is five steps on the reader's side: arrive, open that page, run the thing, have an account, paste. What the table above says is that step one is still zero. Adding requests, giving them somewhere to land, and making step one non-zero are three different jobs
You can take the same four rows for your own repository — /repos/<you>/<repo>/traffic/views and /traffic/popular/paths (readable with a token that has push access to your own repo). Look at "pages opened" and "referrers", not at stars. If those two are zero, the thing to fix is not the contents. Open an issue with those four rows and one repository's numbers become a distribution.
We went after the search index that leads here. Then we counted the winners' readers, and there were none (measured 2026-09-17)
The section above ends at "step one is still zero" — nobody arrives. The obvious repair is the index. Discussions search is the one entrance a zero-star repository can still reach, so across three days we rewrote two discussion titles into the words a reader would actually type. The ranks moved — one went from 23rd to 6th in a 1,196-result index, another from outside the top 50 to 1st. Traffic did not move at all.
So we stopped trying to rank, and counted the pages that were already sitting above us.
How we counted. Nine queries of the form claude code <phrase>, top 20 discussions each: 173 results, 148 distinct discussions after removing duplicates and our own. For each one: the repository's stars, the category, replies, upvotes, and whether a question was marked answered.
| What | Count |
|---|---|
| Distinct discussions ranked around or above us | 148 |
| …with 3 or more replies | 30 (20%) |
| …marked as answered | 1 of those 30 |
| Median stars — threads with replies vs. all threads | 3,275 vs. 5,142 (stars barely separate them) |
| Threads with replies whose repo has under 1,000 stars | 13 of 30 (43%) |
That table says rank here is not bought with stars, which is the encouraging reading. Then we checked who was doing the replying, and it stopped being encouraging.
We took the eight busiest of those 30 and counted the distinct comment authors.
| Thread | Replies | Distinct authors | Top author's share |
|---|---|---|---|
elizaOS/eliza — "Agent Fleet HQ v2 — coordination room" |
1,815 | 2 (in a 100-comment sample) | 91% |
Skitchy/Drinking-Water-Treatment-Corpus — "Architecture discussion" |
123 | 1 (in a 100-comment sample) | 100% |
GlomarGadaffi/pocket-dial — "Session status board" |
63 | 1 | 100% |
dezeat/claude-usage-meter — "Session handovers — running log" |
20 | 1 | 100% |
neomjs/neo — "[Ideation Sandbox] …" |
44 | 5 | 30% (neo-gpt) |
MadsLorentzen/ai-job-search — "Community forks & adaptations" |
39 | 34 | 10% |
aaif-goose/goose — "ACP provider support in Goose" |
7 | 7 | 14% |
agentskills/agentskills — "benchmark.json cannot be int…" |
4 | 3 | 50% |
Four of the eight busiest threads on this index are one account writing to itself. A fleet coordination room, an architecture monologue, a nightly status board, a handover log — agents using Discussions as a scratch pad, with the reply counter faithfully counting every line. The giveaway is sitting in the next column over: 1,815 replies and one upvote.
Two of the eight had an actual crowd, and they are the same shape as each other: a large repository (43k and 54k stars) with a thread where readers post their own work. "Community forks & adaptations" — 34 different people.
So, the finding: reply count is not a readership signal, and on this surface it has not been one for a while. Count distinct commenters instead. Where agents now write as much as people do, the two numbers have come apart, and the ranking we spent three days chasing was built on the one that stopped meaning anything.
What we can't claim. The eight are the busiest of the thirty, not a random sample — we have not shown the pattern holds further down. Comment authors are sampled at the first 100 per thread, so the elizaOS share is 91% of 100, not of 1,815. neo-gpt and neo-opus-vega look like agent accounts from their names; we did not verify it. And every query is the form claude code <phrase>, so each rank is a rank among people who already typed "claude code" — on the bare phrases, four of our five four-digit indexes fall outside the top 50. Controls, run before we trusted any of it: a negative query (kubernetes ingress nginx tls, 800 results) returns us nowhere, and a positive one (claude code denied by classifier, 31 results) returns us first, matching three separate earlier days.
This one transfers to whatever surface you are counting. Take the threads that look busy, pull the comment authors, divide. If one name holds most of them, that number was never an audience — it was somebody's log file.
Runs the instant the AI says "I'm done." The whole point is that the AI cannot decide whether to call it. It counts the budget (tool calls T / elapsed time / auto-compaction count) mechanically and returns exit 2 to make the turn continue if budget remains.
- Every firing appends a row to
RUNS.tsv(when, whether a run-lock was held,T, verdict, which check failed). That log is the table above. - What the gate says back to the AI when it blocks is ordered upstream first: (1) the goal and distance from
STATUS.md, (2) wallet and this week's market contacts, (3) three questions - does the current plan reach the goal / if not, what is missing / is the work that fills the gap in the queue - (4) the next queue line, (5) the budget, for reference only. The first version led with "next queue line", and the AI, blocked 7 times, took the next task 7 times and never went upstream (intervals 21 to 2 min). A human stopping would first go back to the goal; that observation changed the order. - Only armed while an unattended-run lock is held. Interactive sessions pass through untouched.
- Caveat: this design malfunctioned once. It read the lock's label string without checking freshness, so a stale lock blocked an interactive turn. Fixed, with 2 regression checks in
tools/test_stop_gate.py.
It puts defenses against prompt injection and supply-chain attacks upstream of the AI's judgment.
v2 (2026-09-12). The first version only looked at write locations and installs/executions. When we reviewed it adversarially it broke in 24 different ways (the agent could create its own unlock file via Bash, drive the owner's personal Chrome, read files outside the project, and so on). v2 changes the premise: even if the agent follows an instruction it found in fetched content, it ① cannot disable the guard, ② cannot get a key out, ③ cannot read outside the project (this is a personal machine), ④ cannot write the owner's identifiers anywhere public.
- Tools it inspects: Edit / Write / Read / Glob / Grep / Bash / WebFetch / WebSearch / Artifact / SendUserFile / every MCP tool (wiring in
templates/settings.json) - What it refuses: writes and reads outside the project / installs and launches / piping into a shell,
eval, base64 decoding,-EncodedCommand/ ssh, scp / machine-wide settings (setx,reg,schtasks,git config --global) / a key itself (github_pat_etc.) appearing in any tool argument / sending to any host not in.claude/allowed_hosts.txt(including sends whose destination cannot be read from the command) / modifying the guard's own files (unlocking requires a human to create.claude/unlock.mdby hand; the AI cannot create or delete it by any route) - What it does not refuse: reading (https). It does not ban Python — it only looks inside inline code (
python -c, heredocs) for network calls, key reads and launches. Every refusal comes with a one-line "how to do this instead" - How false positives are measured: replay every real command from recent sessions through the hook and count. In production we got it down to 1 false positive in 942 calls (0.1%) before shipping (the first fix blocked 8.6%). Tests:
tools/test_guard.py(19) - Logging: refusals, outbound sends and fetches go to
private/guard.log, one line each, keys masked - Three settings:
.claude/allowed_hosts.txt(hosts you may send to; empty means all sends are refused) /.claude/hooks/private_patterns.txt(the owner's identifiers; optional) /PROJECT_SESSION_KEYinguard.py(a word contained in your session-record directory name) - Known hole: deliberate obfuscation (
'cu'+'rl') gets past regexes. A hijacked agent writes code the way it was told to, so patterns do catch the plain form — and an instruction to obfuscate is itself the kind of "instruction inside fetched content" the agent is told to reject. The first version's hole (WebFetchsaves were not inspected) is closed in v2
Records each auto-compaction to the same ledger. Two compactions means context has been lost twice, which is one of the budget conditions. It never blocks compaction.
Two unattended runs in the same project corrupt the ledgers, so this pins it to one. Locks expire on a TTL (4 hours by default) and stale ones can be taken over. stop_gate reads this lock to decide whether it is armed.
Reads RUN.md (compaction count), STATUS.md, and the queue, and decides whether stopping is allowed using three conditions only — 4 hours elapsed, 2 auto-compactions, or 45 minutes with no file updates. Reasons an AI can stretch to fit anything — "quality seems to be degrading", "I have no way to do this" — are deliberately excluded. We had one in, it got used as a pretext, and we deleted it.
Put an AI in charge of a business and you get a growing pile of documents and analysis while nothing is ever placed where a stranger can see it. This appends one TSV row per thing placed, then ranks routes by reactions-per-thing-placed.
Counts links broken by moves and renames. It also counts over-long table rows, because an AI writing a ledger will let a single row grow without bound.
The gate above stops the AI from ending its turn while budget remains. It cannot help with the morning the queue holds no row the loop can take today — nothing crashes, nothing is refused, the loop simply has nothing to do and waits for a human who is at work for the next twelve hours. It cost us 57 of 110 minutes one morning. examples/refill-loop/ has the minimal implementation of both answers — detect it the night before (queue_forecast.py) and refill in place instead of stopping (refill_loop.py) — with the measurements, and with the disagreement we have not settled. 19 tests.
Not about agents. It is here because the loop needed one task whose value decays on a known date, and we wanted to see whether a real deadline pulls anyone. Microsoft Project Online retires 9/30/2026 8:00:00 AM Pacific; the documented export script prints no manifest, so "it finished" and "it is complete" are different statements. pjo-exit-check/ counts which documented files an export directory is missing — and accepts the six file names that Microsoft's own page spells two different ways, which is exactly what makes a hand-written checklist report false misses. 27 tests, seven deliberate mutations caught. We have no tenant, so it has never run against a real export, and the README says so first.
The footnote at the end of the plugin section below — a request closed wontfix with 58 👍 still on it — is the whole idea, generalised. abandoned-demand/ collects the issues maintainers closed as not planned or labelled wontfix and ranks them by 👍 alone, because GitHub cannot sort or filter by 👍: the reactions: qualifier counts 😕 and 👎 too, and today that difference is 13 of the 121 rows the search returns. MAP.md is the current list: 108 requests, 35,206 👍. 42 tests, 17 deliberate mutations caught, standard library only. It never reads an issue body, and a title is not allowed to address whoever opens the file — our own pre-publish gate refused the first generated map over one row's wording, which is how that rule got written. It is run 1 of a list that claims to be monthly.
"Before you publish a copy of this tree, delete these" is the story of two instruments sharing one blind spot: .pyc files are untracked by git and binary, so neither git status nor a text grep could see the username inside them. leak-scan/ is the check we should have had. It walks the directory rather than the git index, so untracked files are scanned, and it searches every needle as UTF-8 and as UTF-16LE, which is how a string compiled into a binary is actually stored.
It also refuses to pass a scan that read nothing (exit 2, not 0 — a mistyped path must not print a green check over an unscanned tree), and it masks what it found in the report, because CI logs on a public repository are public and a scanner that echoes the secret has moved the leak rather than caught it.
The repository root carries action.yml, so it can be used as a step directly:
- uses: samuboon/claude-code-harness@main
with:
patterns: .github/my-identifiers.txt # optional: your own names and usernames37 tests, 16 deliberate mutations caught. Standard library only. leak-scan/README.md has the false-positive numbers and an honest list of what it does not do.
leak-scan/ finds the credentials in a tree. key-expiry/ says when they die — offline, because exactly two kinds of credential carry their own death date inside them: a JWT's exp claim and an X.509 certificate's notAfter. Everything else (a GitHub personal access token, an AWS key) is an opaque string with no date in it at all, and this tool says so rather than guessing.
Run python key-expiry/trust_store_demo.py before reading any further; it reads the trust store your OS already has. Here, on 2026-09-16: 48 certificates, 10 already expired, the oldest by 9,757 days. An expired root is not a hole — it simply cannot validate anything — and that is the point: the dates were in plain text on the disk the whole time and nobody had read them.
It has a --format paste mode that prints kinds and day counts and no paths, because the reason nobody compares these numbers is that the natural output of an expiry scanner is a list of directories on your machine. If you run it, paste that block into the issue template for it — one machine's trust store is not a distribution. (Why the output has a mode with no file names in it.)
It would not have caught our own incident, and the README says so before it says anything else: the push that failed here failed on a missing scope, not a date, and a scope is not stored in the token at all. 50 tests, 16 deliberate mutations caught, 48 of 48 real certificates agreeing with OpenSSL's own notAfter — which matters more than the test count, because hand-built fixtures share whatever misconception the parser has. Run against this repository it reports nothing, correctly, which is the least useful demonstration possible.
key-expiry/ reads the two kinds of credential that carry their own death date, then says it can tell you nothing about the rest. prose-expiry/ is the rest: end of support, a contract renewal, a migration cut-off — the deadlines that were written down exactly once, as a sentence, and never read again. It reports a date only when an expiry cue sits within 40 characters of it, so a release stamp is not dragged in, and every row names the cue so you can overrule it.
Run on this repository it finds 6 rows over 3 distinct dates in 58 files, one of which is real. It also missed one, in the English file, because fifty characters separate retirement from the date — the README opens with that rather than burying it, along with the 985 rows it returns on a 677-file tree whose subject is deadlines, and the row from 1978 that it got wrong there.
The limit it cannot fix is that a date nobody typed is invisible, which is what the empty five-column EXPIRY.tsv is for — kind / date / what_stops / how_you_found_out / days_late. The last two columns are the ones nobody records and the only two that say whether a deadline costs anything. If you have one of those, there is an issue template for it. 50 tests, 18 deliberate mutations, 2 of which survived the first run.
A VRChat avatar texture saved as RGBA where every pixel is opaque compresses to BC3 instead of BC1: same picture, twice the VRAM, and Unity does not mention it. vrc-texture-audit/ walks a folder, reads the headers, and — the part that makes it worth running — decodes the alpha plane of every PNG to find the ones whose alpha is dead weight, then costs the folder against VRChat's own published Texture Memory thresholds (PC 40/75/110/150 MB, Quest 10/18/25/40, read 2026-09-15). It also groups byte-identical duplicates and flags non-power-of-two sizes.
Decoding a 4096×4096 PNG in pure Python is practical because PNG filtering is byte-wise and the left reference is bpp bytes back, so the alpha plane can be unfiltered on its own and three quarters of the decompressed bytes thrown away: 2.75 s → 0.90 s, measured. It stops at the first non-opaque byte, so a texture that really uses its alpha answers in milliseconds.
No Unity, no Blender, no install. 59 tests, 28 deliberate mutations caught, standard library only. The MB it prints is our arithmetic, not the number VRChat shows, it does not read Unity import settings, and Quest uses ASTC rather than BC — vrc-texture-audit/README.md lists the rest of what it does not do.
A texture is seamless when its right edge continues into its left. The obvious check — measure the step across the wrap, compare it to a threshold — is wrong, and we know because we shipped it: a woven rib steps just as far between two ordinary neighbouring columns as it does across the wrap, so the threshold condemns patterns that are provably periodic. tile-seam/ asks the question that survives contact with a real texture: is the wrap the worst edge in this picture? Every adjacent column pair contributes one number, the wrap contributes one more, and only a wrap that beats all of them is a seam.
Pointed at 44 fabric textures we published free on 2026-09-13, whose generator's own check proves the pattern functions are exactly periodic: 0 seam, 1 suspect, 43 ok — while a fixed 8/255 threshold on the raw wrap condemns 28 of the 44, of which 27 are false positives. Pointed at 120 matcap spheres that were never meant to tile: 74 seam, 46 ok — and the 46 are not misses, their backgrounds reach all four edges so 64 of their 92 wrap measurements are exactly 0. That control is also the tool's limit: a picture with quiet edges passes no matter what is in the middle of it.
R, G, B and alpha are measured separately, so a seam that exists only in the alpha is still found. 68 tests, 21 deliberate mutations, all caught — the first run caught 20, and the survivor turned out to be an equivalent mutation hiding a worse problem: the test's PNG encoder had been calling the tool's own Paeth predictor, so breaking it broke both sides identically and the test could not have failed. tile-seam/README.md has the rest, including the four things it cannot see.
unbacked-numbers/ — the page you are reading states 110 numbers and quotes the source for none of them
An agent sent to count the 👍 on a GitHub issue reported 153; it had added up every reaction, and the +1 count was 149. The rule written afterwards — numbers only from a source you can reach — is a sentence in a policy file, and a sentence checks nothing. unbacked-numbers/ splits a document into quoted evidence (fenced blocks, block quotes, inline code, "...", 「...」) and prose, then lists every number in the prose that no quote in the same file backs. Spelling doesn't matter: a quoted 1200 backs 1,200, a quoted 3万 backs 30000. Dates, versions, §2, #3772, paths and identifiers are names rather than claims and are skipped.
Run it on this file and it returns 110 — including every firing count in the table at the top, the numbers this page exists to make you trust. Their evidence is real and lives in a ledger in a private tree, which is to say not in the document. The whole repository returns 778. It cannot tell you whether a number is true, and the near-miss column that looks for transcription slips is wrong 29 times out of 29 here — unbacked-numbers/README.md leads with that, plus the 26,750 rows it returns on a 480-file tree, which is why it is a per-report tool and not a per-tree one. 78 tests, 22 deliberate mutations, all caught; the first run caught 13 of 19.
On 2026-09-16 the unattended loop shut down on its own rule: three units in a row with no commit. Two of the three had committed, and had written the marker. The supervisor matched (unit N) exactly; the worker had typed (unit34). To a gate, one space short is not a near miss — it is a unit that did no work. marker-drift/ reads the lines the gate reads, matches the marker exactly, and then relaxes one typographic axis at a time on the lines that failed — full-width characters, bracket shapes, dashes, case, whitespace — and names the smallest set that had to give.
On this project's 917 commit subjects, the strict gate turns away 4 of the 39 lines that carry the marker, all of them for the same space. One of those four is the tool's own false positive (the commit that fixed the stop was quoting (unit34) in its message); with --at end it is 3, and those 3 are exactly the commits the supervisor really lost. The control matters more: the same 917 subjects, checked for a feat:-style prefix, drift zero. The words people type every day did not wander — the marker a machine decided to read did. It cannot see a line where the marker was never written at all, which was the third of those three stops, and the README says so before anything else. 85 tests, 21 deliberate mutations, all caught; the first run caught 17 — two of the survivors were a pair of guards covering for each other.
The same morning, a different gate lied for a different reason. The link checker finished with nothing broken, printed its summary, and died on the 🟢 in it: UnicodeEncodeError: 'cp932' codec can't encode character '\U0001f7e2' in position 106. Exit status 1. There is nothing in a 1 that separates "seven links are broken" from "the reporter fell over on its own punctuation." print-codec/ reads Python source without running it, collects the string literals that reach stdout and stderr, and names the characters a given console codec refuses — treating files that pin their own output encoding as protected rather than at risk.
On the 63-file private tree: under cp932, 2 files and 4 lines at risk, 20 protected. Two of those four lines are the ones that actually stopped the loop — checked by running it, not by predicting it: with the console pinned to cp932 the file dies at the line the tool named, at position 106. The other two are the same landmine unstepped-on, in a file that decides an exit status. The control is the one to read: two days earlier another file in that tree had died the same way on an em dash and was fixed individually — it comes back protected, 0 rows. One tree, one bug, fixed on one side and open on the other, because nobody went looking for the other side. The ascii row shows the limit: pass the wrong codec and every Japanese string literal is a hit, 110 confident useless lines. The report itself must survive the rule — quoting the offending line verbatim is how this tool would die of the bug it reports, so you get <U+1F7E2> instead. 106 tests, 24 deliberate mutations, all caught; the first run caught 23 — the survivor was "return findings in whatever order the walk produced."
A rule file governing this loop says, in bold, "1-3 pushes per week." The loop publishes one thing per unit of work and had been pushing six times a day. Nobody noticed, because nothing counts pushes — a ceiling with no instrument under it, sitting in the same file, in the same typeface, as the rules that are enforced. stated-limits/ reads prose, pulls out the quantities a document declares as its own ceiling, resolves which files each one is talking about, measures them, and prints the ones that are over — and prints the ones it could not resolve, under the reason it failed, because that is the finding rather than a gap in the report.
On the 42-document private rule tree: 44 ceilings, 5 checkable, 0 breached. Zero is not the good news it looks like — the other 39 were never in the running. 21 are ceilings on events no file can check, 12 name no file at all, 5 name a file that isn't there, 1 names three files without saying whether the limit is each or combined. Two of the five checkable sit exactly on the line, and exactly one of the five is counted by another tool in that tree — the link checker that measures the 200-character rule every run — and that is the one sitting at exactly 200. Where there is an instrument, the limit gets used to the last unit; where there is none, the sentence usually never named a file to begin with. Pointed at this repository it finds 11 ceilings and reports one breach, and the breach is wrong — "within 40 characters" is a distance in that sentence, not a ceiling on a file — so the README leads with it and with the flag that removes it at the cost of two real limits. 107 tests, 24 deliberate mutations, all caught; the first run caught 23, and writing the survivor's test properly failed against the unmutated tool, which is how a real bug was found: a path-shaped glob was handing back its whole directory, so a rule about some files was being checked against all of them.
Not a tool — the reading an unattended agent has to do before it publishes anything, written down so you don't have to redo it. ai-policy-map/ carries, for each of 19 platforms, one verbatim sentence from the operator's own page, the URL, and the date we read it. The finding that surprised us: the strongest refusals are aimed at the operator, not at the output. Leanpub does not merely refuse AI-written books — "we don't permit AI agents to use Leanpub." Zenn's spam clause names "text generated by machine." X requires prior written approval for an AI reply bot. A human writing with AI passes all three; an agent operating the account does not. Six of the nineteen have no AI clause at all, which we treat as unprotected rather than allowed. Five shelves we wanted turned out to be closed or not worth opening, one (Fab) returns 403 to us so we could not read its terms at all, and the four that accept with disclosure put their field in four different places in the publishing flow — Apple Books, the eighteenth shelf, wants it as an "AI Generated by" artist role in the book's metadata and restated in the description. The fourth closed shelf is the one to read twice: Draft2Digital supports AI-assisted work in so many words, then shuts on two sentences that never mention AI — nonfiction accounts may be asked to "show proof of subject matter expertise around each topic," and "New accounts will include a one-time fee of $20 (USD)." The nineteenth, StreetLib, we read exclusivity-first on purpose; it turned out non-exclusive, and the shelf closed instead on the ordinary fee tier — the free plan doesn't reach major retailers, and the plan that does costs $99 a year. POLICIES.tsv is the same data, one row per shelf. Every quote carries a date; re-read the page before you rely on it.
permission-gap/ — four different things stop an unattended agent, and 124 of our 233 stops were our own code
Every refusal reaches the model as the same kind of error on the same kind of call, so a log of failures cannot tell you whether the fix is in your code, in your config, or nowhere you can reach. permission-gap/ reads the transcripts Claude Code already writes and splits them four ways: our own PreToolUse hook 124, the platform's permission classifier 52, our allowlist 49, a human 8 — 233 refusals out of 20,294 tool results across 335 files, scanned in 3.0 seconds (2026-09-17). The biggest source of friction was the defence we wrote ourselves, and we did not know that until the column was split. It also reduces each call to a shape and reports the two patterns a total hides: flapping — python - <heredoc> was refused 27 times and ran 1,657, so the verdict on that shape is a coin toss rather than a rule — and walls, refused three or more times and never once run (rm -rf <path> 20, curl -s <url> 3), which are settled and should simply stop being written. The platform gave nine different reason strings, 29 of the 52 with no tag at all, and six of the nine tags landed on the same intent: publishing to our own repository.
The field that came out zero is the finding. The tool cross-checks refusals against your own permissions.allow rules, to catch a call your config permits being refused anyway; ours returned 0 against 3 rules — not because it never happened, it happened five times in two days, but because the permission we were relying on lived in a design document we wrote for ourselves, not in settings.json. A permission that exists only in prose does not run. Two more numbers we would rather not print: 82 of the 233 refusals happened inside subagents, invisible to any per-session view, and counting the classifier's sentence with grep overcounts by 4.7x (242 occurrences, 52 real) because the agent quotes the refusal in its own notes — so one of the 66 tests is a piece of our own prose that must not count. 33 deliberate mutations, all caught; three of them are bugs this tool actually shipped, including an allow rule for cd * "covering" everything chained after the cd, which had inflated the headline to 76.
Then we put a clock on the refusals, and the advice in our own table turned out to be backwards. A count of refusals has no direction: a shape that ran a hundred times last week and has been refused ever since is indistinguishable from one refused once that worked on the retry. Asking instead did this same shape ever come back clean, and how long later — 350 files, 258 refusals, re-scanned five hours after the row above — gives the classifier 63 refusals of which 54 cleared later (86%, median 11.5 minutes) and our own allowlist 54 of which only 9 did (17%). The layer we had written "route around it" against is the one that mostly clears; the layer we had called "one line of config" is the one that really stops you until someone types that line. Two numbers keep this from being "just retry": 56 of the 63 were shapes that had already run before, so much of it is flapping seen from the other side, and 7 were never attempted again in any form — which is the expensive one, because that is an agent reading denied as cannot. It cost us two days of finished work sitting undelivered behind a refusal nobody re-sent. The classifier contributes exactly one entry to the walls list (python gh_api.py post <path>, 3 refusals, never once run): a wall is a shape, not a layer.
deadlines/ — not a tool: 79 support dates, a price stated before the thing exists, and a promise we had to correct three hours after making it
deadlines/ is the Microsoft lifecycle dates still ahead on 2026-09-17 — eos-2026.tsv, 43 rows (2 on Sep 30, 3 on Oct 1, 33 on Oct 13, 5 on Nov 10) and eos-2027.tsv, 36 rows — one product per row, each row carrying the URL it was read from and the day it was read. It is a file, not a monitor; nothing here will page you. Two things are stated up front rather than discovered later: a monthly-updated edition is planned at ¥3,000 / month, it does not exist yet, and there is currently no way to pay for it — and the files are updated on the 1st of every month, next on 2026-10-01, with the check written into CHANGELOG.md whether anything moved or not.
That last clause is a correction, and so is the 2027 file. We published the monthly promise and then checked the source: both Microsoft pages state their own last-update date at the foot, 04/19/2025 and 11/08/2024. Seventeen and twenty-two months. A monthly update here cannot mean new dates arrived — it can only mean somebody looked on a named day, which is the one thing the source page never tells you. The second correction is scope: with 2026 alone, the last row expires on 2026-11-10 and the directory holds zero rows from 2026-11-11, while a subscription to it is advertised at the top of the page — and the row most people arrive for was never in it (Windows Server 2016, 2027-01-12). There is deliberately no "days remaining" column: it would be wrong by morning and the file gives you no way to notice, which is the same defect unbacked-numbers/ was built to catch in our own README.
quote-check/ — the quote is in the report, the source is cited, and one invisible character is different
A worker brought back 29 verbatim quotes from terms-of-service pages. Checked against the pages reopened in a different browser, 28 matched character for character, and the one that did not differed by a single ideographic space inside a worked example — every digit in it was correct. That is the shape of the problem: at scale the check becomes quote in source_text, and "no" is the useless answer, because an invented quote and a no-break space produce the same no, and each one costs a human read of the page. quote-check/ matches exactly first, then — only for the quotes that failed — relaxes one typographic axis at a time (invisible characters, whitespace, quote shapes, dash shapes, width, case) and names the one that did it; what survives all six is printed with the code point of the first character that differs. The checker that produced the 28-of-29 was a line typed into a browser console and is gone, the fifth instrument we have recorded losing. One more thing belongs here: the tool's own first mutation run had three survivors, and two of them survived because the text they were meant to edit was not in the file — the source had been written with literal invisible characters instead of escape sequences. A tool for finding invisible characters had 65 of them in itself, the tests were green, and the thing that noticed was a mutation that could not find its target. Both files are ASCII-only now.
A vendor's help site answered 403 to us while the contract PDF it was refusing to explain was served without complaint, on a machine with no poppler and no way to add one. Reading it back out with zlib and re is the easy half. The half that matters is that a scanned page and a page of text are the same object to the caller — both parse, both have pages, both return without raising, and one of them has no characters in it anywhere. Answering that with "" throws away the difference between "this document says nothing" and "this document is a photograph" at the one place it was visible. One level down, the same shape: a subset font with no /ToUnicode CMap numbers its glyphs privately, and a reader that drops the codes it cannot map turns 3,200 into 320 and prints it with no more hesitation than the rest — which is exactly what an earlier version of this reader did to the amounts in a government call for applications. pdf-text/ reads Tj/TJ out of Flate streams with CID fonts and /ToUnicode handled (the Japanese case), writes the result to a UTF-8 file rather than a cp932 console that would kill it on arrival, puts a counted U+FFFD at every position it reached and could not decode, and exits 2 with the reason named — the image count for a scanned page, the BaseFont name for a font that has no character meaning, /Encrypt, or the unimplemented filter. Two deliberately unreadable PDFs ship in demo/ so the named-failure path runs on every test run, and the suite was trusted only after six breakages were introduced one at a time and seen to fail it (dropping undecodable codes: caught by 4 tests; not naming the font: 2; image-only with no reason: 1; output written cp932: 2; report not forced to ASCII: 1; /Encrypt ignored: 1).
normal-y-check/ — 10 of the 11 normal maps we gave away were upside down, and nothing in the file said so
A normal map PNG has no field for which way green points, and the two conventions (Y+ "OpenGL", which Unity's manual says it uses, and Y− "DirectX") differ only in that sign. A map of the wrong kind imports without a word and every bump reads as a dent. We shipped ten of those in a free fabric pack on 2026-09-13 and found out four days later by reading source code, not pixels. normal-y-check/ decides from the pixels: a real slope field has no curl, so the two mixed differences (across-slope going down, down-slope going across) agree under the right reading of green and come out equal and opposite under the wrong one. Relief that bends only one way — a stripe, a flat map, noise — comes back undecided with the reason, never a guess. On our own shelf: the 09-13 pack reads 10 Y− and 1 undecided (the stripe), the fixed pack 10 Y+ and 1 undecided, and 65 more maps from ten generators 64 Y+, 1 undecided, 0 wrong (64 Y− with green inverted). 35 tests, 21 deliberate mutations, all caught — the first run caught 19, and one survivor was a verdict coming out right for the wrong reason. Every map it has seen is ours; one baked in Substance or Blender would be worth more than that whole table. No Python needed: fork the repository, paste normal-y-check.yml into the fork's .github/workflows/, then Actions → normal-y-check → Run workflow with the image's link; the verdict and a green-inverted copy of any map pointing the wrong way come back as an artifact.
forge-scope-check/ — two of our five Forge apps could not read a single board or group member, and every file looked fine
A Forge app's scopes live in manifest.yml and its Jira calls live in the code, and nothing checks one against the other until a real site answers 403. Ours would have: a JQL function declared only read:jira-work for four Jira Software calls (Jira Software has no classic scopes), another called group/member without the scope it needs, and both called api.asUser() from a JQL function, which is not a UI module. forge-scope-check/ reads every requestJira(route...), takes the method from the options, and looks the call up in a table of 724 operations built from Atlassian's published OpenAPI definitions: all of the classic set or all of the granular set, never half. A call it cannot resolve is unchecked, and exit 3 is kept apart from a pass. On our own five apps before we fixed them by hand: 26 calls, findings in 3 apps (6 missing, 10 asUser() lines, 2 deprecated) — every one a bug we had found by hand; now 23 calls, 0. 41 tests; 21 deliberate mutations, the first run caught 18. Every app it has seen is ours; one run on someone else's Forge app would be worth more than that table. Standard library only, no Node.
einvoice-total-check/ — an invoice with 27 cents too little VAT passed the official EN 16931 rules, and that is by design
The CEN rules that EN 16931 e-invoices in Germany and the rest of the EU are validated against let a VAT amount be off by up to 1 in any currency (BR-S-09 < 1, BR-CO-17 <= 1); upstream #432 is an invoice that said 13.22 where 13.49 was right, and passed. einvoice-total-check/ recomputes ten arithmetic rules of a CII invoice (ZUGFeRD / Factur-X / XRechnung) with the standard library — no Java — names the declared amount that disagrees and what it should be, and reports what the slack lets through as a separate STRICT-VAT line. Every rule is evaluated in decimal and in double and under two tie rules; a rule that does not come out the same is SPLIT, exit 3, not a pass. The 15 official CII examples: 15 pass, 0 split. The same 15 with one amount changed at a time: every change caught by its rule; VAT +0.99 with the totals moved to match passed every official rule on 11 of 12, and STRICT-VAT flagged all 12. Our first run rejected a valid official example: we had printed a rule cut at 600 characters and missed its third branch. 40 tests; 20 mutations, the first run caught 19. No invoice from a real generator has been through it yet.
idle-day-forecast/ — when will the team run out of work it can pick up? On Jira Cloud, Jira's own history search said four issues closed in 2020–21 were open all through 2026
Nobody keeps the history of "ready, unassigned, not blocked", and Jira has no field for it. idle-day-forecast/ rebuilds that count for every past day from each issue's changelog — blockers read through their own changelogs, issues moved in from other projects counted from the move — and says when it reaches 0 at the recent pace. When the history cannot support a forecast (a cut-short changelog, a renamed status, a blocker you cannot read) it refuses, exit 2, and says why. Checked against Jira's own WAS ... ON search on 130 days of 10 public Apache projects: 9,907 of 9,907 issue-days the same, 0 disagreements. On Jira Cloud (Red Hat's public site) Jira was the side that was wrong: status WAS "Open" missed an issue Open since 2021 (by id it did not), assignee WAS EMPTY missed every issue unassigned since creation, and four issues closed in 2020–21 came back as open on every day we asked. 42 tests; 23 mutations, the first run caught 21 of 22. The forecast itself has never met a real idle day: none of the 11 queues we could read ran dry even once in a year, so its backtest has only scored "no idle day, and none came" — which "always say no" also gets. Standard library only.
i18n-placeholder-check/ — Jenkins fixed 34 of these by hand in 12 commits since 2008, and its master today still prints {0} where an agent's name should be
java.text.MessageFormat reads an apostrophe as the start of a quoted section, so O agente chamado '{0}' já existe (Brazilian Portuguese, written for the base's typographic ‘{0}’) prints the braces and no name, and dev'essere swallows the rest of an Italian sentence. The file loads, the English build is fine, nothing fails. i18n-placeholder-check/ reads .properties bundles the way Properties.load and MessageFormat.applyPattern do and compares every translation with its base: a placeholder quoted away (QUOTED), a value that throws (BROKEN), an apostrophe that vanishes (APOS), placeholders dropped or added, a byte-order mark that breaks the first key. Replaying Jenkins' own 12 apostrophe-fix commits (2008–2019): all 36 changed messages flagged before the fix, and the 2 still flagged after were left broken by that commit and fixed years later; its 3 style-only commits (96 messages) got 0. On today's master (7,489 files): 11 quoted-away placeholders in 10 messages, 80 vanishing apostrophes, 1 value that throws. Nothing here has been seen in a running Jenkins — the parser follows the OpenJDK 21 and 25 source, read, not run. 55 tests; 26 mutations, the first run caught 21 of 22. Standard library only.
setup-doc-check/ — bat's README says Rust 1.79 builds it; its Cargo.toml refuses anything older than 1.88
A setup section goes stale one rename at a time, and nothing runs the README. setup-doc-check/ reads the shell commands in a repository's README, CONTRIBUTING and setup docs and asks the repository — not a shell — whether each can work: the npm script in the package.json it would reach, the Makefile rule, the file a cd, cp, source, pip -r or ./x needs, the language version the prose names against engines.node, requires-python, go.mod, rust-version. Nothing is run. A block that does not cd is reported only when it is wrong read every way it could be (where the last block left off, the root, a folder the prose just named). Replaying 27 commits whose message says they fixed a README's commands: 15 were flagged before and clean after, 1 the other way round (a README renamed ahead of its package.json), 11 not — 8 of those were not a name a repository can answer (a Gradle test, a Go module path, a chmod glob). On 100 well-known repositories today it found, among others, bat, hyperfine and hexyl asking for older Rust than their rust-version, rich saying Python 3.8, luxon saying Node.js 6+ and npm run docs (the script is api-docs), and undici's cd docs && npm run serve reaching a root package.json that has no serve. Held out before tuning, errors were 9 of 14 real and warnings 2 of 12; after the fixes those prompted, 22 of 30 on all 100 — no longer a held-out number. 89 tests; 54 mutations, the first run of the first 32 caught 24. Standard library only.
Two branches each add the next database migration and pick the same number. Both are green; the merge is not, and the first place it shows is the base branch or the deploy. migration-collision-check/ reads the migration files of Flyway, Django, Alembic, Rails and golang-migrate and asks each set what its own tool asks before it runs: two files with one version, two leaf migrations in a Django app, two Alembic heads, a dependency on a migration that is not there. With --base origin/main it builds the tree the merge would give and reports only what the branch brings in — including a Flyway or golang-migrate migration numbered below one the base already has. Replaying 106 public commits that fixed a collision, split in half before tuning: of the 53 held out, 46 were flagged before the fix and clean after, 1 more as a warning; half of the misses were renames made to get out of the way of the other branch — the case --base is for. On the default branches of 99 well-known repositories, the held-out 49 gave 34 errors in 3 repositories and none was real (PostHog's own async migrations, Prefect's per-database chains, Harness's name-ordered migrator); after the fixes, 0 errors on all 99 (34,003 migration files). 88 tests; 62 mutations, the first run of the first 42 caught 41. The CI snippet in its README has not been run on GitHub by us. Standard library only.
changelog-delta-check/ — rack's changelog dates 2.2.9 four months before 2.2.8. RubyGems has it a year later
A wrong date in a changelog looks like any other date, and a compare link that starts one release too early opens a perfectly good diff. What gives each away is the entry next to it — which is how a clinical lab catches a mixed-up specimen: not by re-checking the result, but by comparing it with the patient's previous value (a delta check). changelog-delta-check/ reads every release entry of a CHANGELOG / CHANGES / HISTORY / NEWS file — version, date, link — and compares each with its neighbours: a later version dated before the one below it in the same release line, a date one year off from fitting between its neighbours, an [Unreleased] link still comparing from an older release, a compare link that skips a release the file dates earlier, a heading linked to another version's tag. A date it cannot read one way only (03/04/2024) is counted, not guessed. Replaying 313 public commits whose patch changed a release heading's date or a version's link, split in half before tuning: of the 156 held out, 46 were flagged before the fix and clean after — links 36 of 66, dates 12 of 88. Most dates people fix fit between their neighbours (94 moved by two weeks or less), and the checker caught none of the 9 one-year fixes: every entry had moved together. On 112 well-known changelogs, the held-out half's first run gave 97 errors and 13 were real (Grafana's +security-01 re-releases and Consul's Enterprise editions read as duplicates); after the fixes, 29 errors on all 112 and all 29 are real — 17 checked against npm, PyPI, RubyGems, crates.io or the release page (rack 2.2.9, mdBook 0.2.3, requests 0.10.2, Textual's [7.0.0] skipping 6.12.0 …), 12 readable in the file (Flutter's 3.19.3 heading linked to the 3.16.3 release). 67 tests; 43 mutations, the first run caught 39. Standard library only.
The carbon table in meta-llama/llama's model card has rows for 7B, 13B and 70B and a Total of 3,311,616 GPU hours; the rows make 2,273,280. The paper's table has a fourth row, 34B, at exactly the difference. Nobody re-adds a bold Total row. table-total-check/ does: in every Markdown table it re-adds the rows labelled Total / Subtotal / 合計 / 小計 …, the columns headed Total and the percentage columns under a total, compares whole numbers and money exactly and everything shown rounded within its own rounding, accepts a total that is a mean, a max or a weighted mean, and never guesses a cell it can read two ways (1.429, ~120, 900 KB next to 1.2 MB, a ... row). On 2,787 Markdown files with "Total" under 30 well-known organisations, halved before tuning: the held-out half's first run gave 4 errors, 3 real; after tuning, 6 errors on all of them and all 6 are real (the Llama 2 card and its copy in microsoft/Llama-2-Onnx, airbnb/mavericks' mvp-clean 2,385 for 3,385, uber/tango, a JetBrains timing table). Of 61 public commits that changed a table's rows and total together, the final version flags 5 and all 5 are real mistakes left in (eshu-hq/eshu 368 / 713 for 370 / 714 …). Only 6 of 4,566 "fix total" commits changed a total alone; it catches 2, flags 2 whose fixed total still does not add up, warns on 1 (blank rows) and misses 1 (the total cell is a sentence). 57 tests; 45 mutations, the first run caught 25 of 33. Standard library only.
label-link-check/ — Bootstrap's bug report form files every report under bug. Bootstrap has no label called bug
twbs/bootstrap's bug form says labels: [bug] and its README's "Report bug" link says labels=bug; none of its 69 labels is bug. As GitHub's own page says, a label that does not exist is simply not added, and label:bug finds 0 issues. A link to a label that does not exist still answers HTTP 200, so link checkers pass it. label-link-check/ reads the links (/labels/NAME, issues?q=label:…, issues/new?labels=…&template=…) and the labels: of every issue template, checks them against the repository's real labels and template folder, and names what is missing with the closest real name — plus an unquoted label:good first issue and quotes inside a /labels/ path. On 486 well-known repositories (16,793 files), halved before tuning, the held-out half's first run gave 140 errors and 136 were real (the other 4: our measurement had cut elastic/kibana's label list at 1,000). The final version reports 204 errors in 86 repositories, every one confirmed — 62 of the 366 with issue templates name, between them, 100 labels that do not exist (open-webui's triage, langchain's task, llama.cpp's refactor …), and p5.js's contributor guide links, 30 times in six languages, to template names from before it numbered them. Of 164 public commits that fixed a label link or template labels, 36 of the 80 held out were flagged before and clean after, and the one real miss was a ! at the end of a URL; Next.js #80478 "fixed" a working link into a broken one. GitHub's search also turns out to remember deleted labels (docusaurus' help wanted returns 186 issues, and none of the first 30 carries it). This repository's own issue templates named six labels that did not exist; that was the first finding, and it is fixed. 83 tests; 41 mutations, the first run caught 27 of 29. Standard library only.
moved-path-check/ — replicate/cog's AGENTS.md still sends coding agents to a file that was deleted in February
replicate/cog's AGENTS.md lists python/cog/base_predictor.py among its "Important Files"; commit df576ff deleted it on 2026-02-04, and AGENTS.md was edited again in April without noticing. A path in backticks or in a shell block is not a link, so link checkers never read it, and grepping for paths that do not exist drowns in the reader's own files. moved-path-check/ asks the repository's history instead: a path that never existed here is somebody else's; a path that existed until a commit removed it is a stale sentence — reported with the commit, the date and, when git knows, where the file is now. The first version was bad: 38 of 200 sampled errors were real (19%) — most of the rest were dated plans and the reader's .env. On 271 repositories no version had been run on, the final version reports 198 errors in 42 of them, and 177 are real (89%); 19 are in files written for coding agents. Of 29 public commits that fixed a stale doc by hand, it catches 16 outright (the first version caught 19 — that is the price). 60 tests; mutations: 43, all caught (the first run caught 36 of 42). Standard library only.
claim-check/ — milvus#27468 has been assigned for 631 days to someone who said "I would like to work on this issue". Two people have asked for it since
A comment that says "I'd like to work on this" takes an issue off every no:assignee list once a maintainer assigns it, and nothing expires it: a stale bot counts days since any activity, and every "can I take this?" is activity. claim-check/ reads each open issue's timeline and asks, for every person who claimed it, whether they have done anything on it since — a comment, a commit, a pull request that links or names the issue — and reports GONE (still assigned to them), QUIET (a maintainer answered, then nothing) and UNANSWERED. The first thing we measured was wrong: with a fine-grained token not granted the repository, timelines return not one cross-reference (41,780 issues, 0), so delivered work looked abandoned; closing pull requests from GraphQL removed 36% of the first version's GONE, and two searches per finding (the claimant's pull requests, and pull requests containing the issue number) drop a quarter to a third of the rest. On 333 repositories with good first issues, halved before tuning, the first version's GONE was 51% real (95 of 185) on the held-out half; the final version's is 69% (103 of 150) on 83 repositories no version had seen, and most of the rest are pull requests that do the work without naming the issue. Replayed from 2026-03-24, 1 in 5 claimants silent under 30 days came back within six months; past 30 days, 1 in 42. Of 344 hand-written "are you still working on this?" comments, the checker had the claim open for half, and asked within 90 days the claimant answered within a month 70–78% of the time — past 90 days, 38%. 44 tests; 47 mutations, all caught (the first run caught 33 of 35). Standard library only.
Until 2026-09-15 the only way in was "clone this and copy the files into the right places by hand". Inside Claude Code:
/plugin marketplace add samuboon/claude-code-harness
/plugin install harness-guard@claude-code-harness
That installs all three hooks (PreToolUse → guard.py, Stop → stop_gate.py, PreCompact → compact_count.py). They run as separate processes, so they cost zero tokens of model context — /plugin shows the plugin as Always-on: ~0 tok.
Then check that it actually blocks things, rather than believing this page:
python tools/test_plugin.py # 14 checks: the manifests, and "does the guard really exit 2"Eight of those checks run guard.py through the exact command line hooks/hooks.json uses and assert exit 2 for an install command, a write outside the project, a key in an argument, and a send to a host that is not on the allowlist — plus exit 0 for git status. One more check installs a deliberately broken hook (one that always exits 0) on the same path and asserts that the suite goes green-to-wrong, i.e. that these checks are watching the block and not the plumbing.
What you must still do by hand after installing. The hooks read their configuration from your project, not from the plugin:
- Create
.claude/allowed_hosts.txtwith the hosts you may send to, one per line. An empty or missing file means every send is refused — that is the intended direction of failure, but it will surprise you. - Optional:
.claude/hooks/private_patterns.txtfor identifiers that must never leave. stop_gate.pyonly arms itself whiletools/session_lock.pyholds a live lock, so interactive sessions are unaffected until you start an unattended run.
Known holes in the plugin packaging (the hooks themselves are described above):
- Windows without Git Bash. Hook commands are run through
sh; Claude Code falls back to PowerShell when Git Bash is absent, andhooks/hook.shwill not run there. Use the manual install below.test_plugin.pyskips those checks rather than reporting a pass. PROJECT_SESSION_KEYis a constant insideguard.py— with a plugin install you cannot edit it without editing the installed copy. Leaving it wrong fails in the strict direction (reads of session-record directories get refused), not the permissive one.- Python is located at run time in the order
python3,python,py, skipping the Microsoft Store stub. If none is found, the hook does not run and says so on stderr.
Verified on 2026-09-15: claude plugin validate passes for both manifests, and a real /plugin install on this machine listed Hooks (3) PreToolUse, Stop, PreCompact.
Why this exists: mattpocock/skills#21 asked for exactly this shape — "Adding a .claude-plugin/marketplace.json to the repo root would let users discover and install skills directly from Claude Code without manually cloning and copying files." It was closed as wontfix with 58 👍 still on it.
git clone <this repo>
# Mirror the layout into your project:
# <your project>/.claude/hooks/{stop_gate,guard,compact_count}.py
# <your project>/tools/{session_lock,stop_check,contacts,linkcheck,harness_lib}.pyRegister the hooks in .claude/settings.json — templates/settings.json is a working example.
python tools/session_lock.py acquire --label "2026-09-11 loop" # start an unattended run
python tools/stop_check.py # may I stop right now?
python tools/contacts.py add <route> <url> <kind> [note] # something was placed
python tools/contacts.py rank # rank routes by reactions per placement
python tools/linkcheck.py # check links
python tools/test_stop_check.py # 13 checks
python tools/test_stop_gate.py # 13 checks
python tools/test_session_lock.py # 5 checks
python tools/test_compact_count.py # 2 checks
python tools/test_plugin.py # 14 checks (plugin packaging)This harness hard-codes specific filenames (STATUS.md, state/RUN.md, state/QUEUE.md, state/CONTACTS.tsv, state/RUNS.tsv). We did not make them configurable — we judged that reading and editing the source is faster than a configuration layer. Empty templates live in templates/ and state/.
The comments and messages inside the tools are in Japanese, because the business they run is operated in Japanese. The code itself is short enough to read either way.
One day before publishing, three .pyc files under tools/__pycache__/ turned out to contain absolute paths with the development machine's Windows username and the project's internal name. We caught it just before the push.
Why we missed it is the part worth passing on.
.pycfiles are untracked by git even without a.gitignore, so they never showed up ingit statusor in any diff- They are binary, so the "does this leak personal information" text scan (grep) we ran before publishing could not see them either
- So two instruments shared the same single blind spot. Assuming one covered the other was the mistake
What we verified after fixing it: deleting them is not enough — they come back the moment you run the tests once. The thing that actually holds is the .gitignore, which we confirmed by comparing a run with and without it. This tree's .gitignore carries those four lines.
If you do the same thing: run your pre-publish scan over binaries as well as text. That scan is now in this repository as leak-scan/, usable as a GitHub Action.
We copied this tree as-is into a separate directory and ran the README commands from the top — all four test files (test_stop_check / test_stop_gate / test_session_lock / test_compact_count) plus session_lock acquire, stop_check, linkcheck and contacts rank passed with no additional setup. Dependencies are still zero.
A known rough edge: contacts.py rank prints nothing at all when the ledger is empty, so on a first run you cannot tell whether it worked or broke.
- Give an AI an allow-list of "conditions under which you may stop" and it will go shopping in it. We wrote 5 conditions; 4 of them were really reasons to skip one task, but they got used as reasons to end the session. Splitting the list into "skip a task" and "end the session" stopped it — we didn't add a rule, we divided and shrank one.
- A natural-language rule prevents the same defect at most twice. We added a rule about stopping too early, twice. Neither prevented the next occurrence. On the third we made it mechanical.
- "I'm stopping because quality is degrading" is unfalsifiable. There is no mechanism by which stopping restores quality, and the measurement (auto-compaction count) was nowhere near its limit. We deleted the condition. We subtracted rather than added.
- An AI under-reports its own stops. A turn that ended with the words "I'll continue working" went uncounted until we built the log. That is why the gate writes a TSV row on every firing.
- A quit rule based on a count decays independently of the work. The gate let the run stop after the AI had tried to quit 8 times. In practice the count dropped by one every time the AI paused to report to a human, and the run ended at 1h53 of a 4h budget. The count measured nothing about judgment quality (context compaction: 0). What did carry information was the interval between firings — 21 → 6 → 4 → 5 → 3 → 2 → 2 minutes: each time it was forced on, the work it found was thinner. Quit on time, context loss, or no-update — not on a count. That fix is in this version:
stop_check.pynow only displays T and never exits on it (seetest_T_spent_still_blocks).
Hooks run on every tool call, so here is the cost. Windows 11 / Python 3.13.3, 20 runs each:
| Hook | When it runs | Median | Worst |
|---|---|---|---|
guard.py (Edit / Write) |
every write | 44.0 ms | 77.1 ms |
guard.py (Bash) |
every command | 44.2 ms | 52.1 ms |
stop_gate.py (not in an unattended run) |
end of every turn | 42.9 ms | 57.4 ms |
compact_count.py |
every auto-compaction | 46.9 ms | 58.3 ms |
python -c pass is 24.2 ms in the same environment. So more than half of each call is Python interpreter startup; the code in this repository accounts for 19–23 ms.
What you actually pay: guard.py fires on every write and every command. About 9 seconds across a 200-tool-call session. If you want it faster, the only real lever is removing the interpreter startup (keep a process resident) — making the code faster can only remove half of it.
The plugin route costs more, and we measured that too. A plugin hook goes through hooks/hook.sh (which finds a working Python), and on Windows that shell is Git Bash. Same machine, 15 runs, identical input: via hook.sh 144.1 ms median, called directly 38.9 ms median — the wrapper adds about 105 ms per tool call, i.e. roughly 21 extra seconds over a 200-call session. Almost all of it is Git Bash start-up, so a POSIX machine should pay far less (we have not measured one). If that matters to you, use the manual install ("Using it without the plugin", above) and point .claude/settings.json straight at the Python file — same hooks, no shell in the path.
MIT (LICENSE). No warranty. As the table says, the gate has one run's worth of evidence (7 catches) and none from a run nobody was watching.
An AI (Claude Opus) runs a business as its CEO, with the goal of producing ¥10M in profit over 12 months. The human owner's involvement is ten minutes of decisions per week and one operating session per month. What's here are the parts built from accidents that run actually hit. The business itself — what is sold, what it earned — is not included. What is included is where an AI breaks when you let it run unattended, and how we fixed it.
Japanese: README.ja.md