Skip to content

About

Reproduce a flaky Go test, shrink it to the smallest polluter→victim witness, and leave a permanent regression test. Used to land flake fixes in tailscale/tailscale, jamf-cli and etcd.

Topics

Resources

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Use this GitHub action with your project
Add this Action to an existing workflow or create a new one
View on Marketplace

Repository files navigation

go-flake-hunter

ci nightly Marketplace Go Reference Go Report Card license

Reproduce the flake in 30 seconds instead of 30 minutes — and leave a permanent regression test in the repo.

247 tests + 2 fuzz targets · CI on linux/macos/windows + a nightly sweep · zero dependencies · eight real upstream flake cases reduced to witnesses (evidence)

Finds flaky Go-test failures caused by order, parallelism, repetition and environment, shrinks them to the smallest witness (polluter → victim) and generates a regression test you can commit.

It does not detect flaky tests — that layer is crowded (BuildPulse, Trunk, gotestsum --rerun-fails). This tool does reproduction and permanence: it turns a failing behavior into a test file that never gets lost again.

go test -json  →  hypothesis order  →  ddmin  →  capsule  →  regression test
                  repeat             smallest
                  order              polluter→victim
                  parallel
                  env

Why not just -count=100?

Approach What it gives you What is left for you
go test -count=100 / -shuffle=on a red log, eventually explain the failure, keep the flake
gotestsum --rerun-fails a rerun of failed tests the flake is still in the repo
stress, -parallel loops a higher reproduction probability still no minimal witness
go-flake-hunter polluter→victim chain, shrunk with ddmin, written as a build-tagged regression test the fix

Detection is not the product; permanence is. A tracker tells you a test is flaky. go-flake-hunter hands you a test file that stays red until the cause is gone.

go-flake-hunter in action

Changing the order requires a temporary witness file, because in Go go test -run '^(TestA|TestB)$' runs in source order — the regex order changes nothing (verified, Go 1.26.2). Zero external dependencies: one binary, stdlib only.

Install

Prebuilt static binaries ship with every release: linux and darwin (amd64/arm64), windows (amd64), each published with a SHA256SUMS file.

# latest release (v0.3.0), linux-amd64 — swap the suffix for your platform
curl -fsSLO https://github.com/emrelab/go-flake-hunter/releases/download/v0.3.0/go-flake-hunter-0.3.0-linux-amd64
curl -fsSLO https://github.com/emrelab/go-flake-hunter/releases/download/v0.3.0/SHA256SUMS
sha256sum -c --ignore-missing SHA256SUMS
chmod +x go-flake-hunter-0.3.0-linux-amd64 && mv go-flake-hunter-0.3.0-linux-amd64 go-flake-hunter

# or build it yourself (Go 1.24+)
go install github.com/emrelab/go-flake-hunter/cmd/go-flake-hunter@v0.3.0

On Windows, save the Windows amd64 executable in a directory on your PATH, then run go-flake-hunter.exe version in PowerShell. You can also run the go install command above and add the output directory from go env GOBIN to PATH. An empty GOBIN defaults to the bin directory under go env GOPATH.

The GitHub Action prefers the prebuilt binary of the tag it is pinned to and falls back to building its own source, so @v0.3.0 needs no toolchain setup beyond actions/setup-go.

First 5 minutes

go install github.com/emrelab/go-flake-hunter/cmd/go-flake-hunter@latest

# 1) scan every package (default: 1000 runs / 5 min budget)
cd ~/projects/service
go-flake-hunter hunt ./...

# 2) turn the found flake into a permanent test file
go-flake-hunter compile
go test -tags goflakehunter -run '^TestFlakeRegression_' .

# 3) explain one capsule (replace the example with the path from your scan)
go-flake-hunter explain .go-flake-hunter/gfh_e3c71933f905.json --format markdown

Real output (testdata/fixtures/order-basic):

flake found: TestVictim
  package      : fixture/orderbasic
  class        : order
  polluter     : TestPolluter
  run          : 3/3 failed
  Wilson 95%   : 43.9%–100.0%
  deterministic: yes
  evidence     : TestPolluter writes, TestVictim reads: seeded

  reproduce:
    go-flake-hunter replay ".go-flake-hunter/gfh_e3c71933f905.json" --pkg '.' -n 1

  capsule      : gfh_e3c71933f905
  status       : confirmed

The generated regression test (compile):

//go:build goflakehunter

// Code generated by go-flake-hunter; DO NOT EDIT.
func TestFlakeRegression_TestVictim(t *testing.T) {
	t.Run("polluter/TestPolluter", TestPolluter)
	t.Run("victim/TestVictim", TestVictim)
}

The price of the build tag: go test ./... does not see it. In exchange the default suite neither slows down nor turns red. The test will be red right now — the artifact's job is to catch the flake, not to fix it. After the fix it turns green and guards the regression permanently.

Scanned real repos (beyond our own fixtures)

Three targeted upstream cases completed the full lifecycle:

Repository / pinned commit Minimal polluter → victim Result
gosimple/slug @ 1eab316 TestSlugMakeUserSubstituteLang → TestSlugMake order; fix merged upstream in #83 (2024), re-validated here
router-for-me/CLIProxyAPI @ 0074897 TestResponsesWebsocketHomeSelectedAuthCallbackPinsAndReusesFirstSelection → TestCodexClientModelsApplyPatchRouting order; proposed fix in #7356 (open), applied locally and validated
Jamf-Concepts/jamf-cli @ 977b704 TestResolveAuth_URLNormalization → TestCheckAPIMatchRefusesAPlatformCommandOnAnInstanceProfile order; witness + fix merged in #474, 1873 preceding tests shrunk to one polluter

The first two (November 10) confirm the lifecycle end to end: confirmed capsule → replay 5/5 → generated regression RED → the fix → the same regression GREEN. Both searches used unmodified upstream test bodies.

The third is the case the tool exists for. A Jamf maintainer had already reported the order dependence and named the mechanism in #370; nobody had shrunk it. repro --from-json on the failing -shuffle log reduced 1873 preceding tests to a single polluter, compile --verify produced a regression that is red on main, and the patch is 11 lines using a helper the repository already had.

None of the three was discovered by this project: every flake report was already public. The contribution is the minimal witness — and, in the jamf-cli case, the fix — never the diagnosis. The table says which fix is merged and which is still a proposal.

Commands, exact revisions, capsules and before/after evidence: real-world validation. These are targeted historical reproductions, not a measurement of detection rate across arbitrary repositories. The seven earlier scans below remain recorded as they ran.

November 15: four Kubernetes packages and istio

Five more upstream defects, each pinned at the base commit of its fix pull request. What separates them from the three above: four are repeat class, which previously had only fixture evidence — pull-kubernetes-unit runs -count=1, so not one of them can appear in upstream CI.

Repository / pinned commit Class Minimal witness After the upstream patch
kubernetes @ 7898e97 (#143133, open) repeat TestAuthPluginWrapTransport package green at -count=2
kubernetes @ 7898e97 (#144048, open) repeat TestObserveCompilation, TestRecordEtcdBookmark green; a third package panics and is reported, not claimed
kubernetes @ 9b3380e (#143132, merged) repeat TestPriorityQueue_UpdateNominatedPodForNode green at -count=5
kubernetes @ 3c690c1 (#142680, merged) order TestInFlightEventAsync + TestRecordGeneratedPlacements → TestQueuedEntitiesRecorder green — and a second repeat witness on the same file stays red
istio @ ac641d0 (#39619, merged 2022) order TestListenerAccessLogs → TestListenerAccessLog green at -shuffle 2/3/7

Capsules and the exact commands: upstream-2026-11.

What a flake hunter should say about a healthy codebase is nothing — so the first real-world metric is false positives, not witnesses:

Repo Scan Result
sirupsen/logrus root package 0 flakes; run budget hit → self-reported partial capsule
rs/zerolog root package 0 flakes; budget-capped, honest partial
spf13/viper root package 0 flakes; budget-capped, honest partial
fatih/color root package 0 flakes; budget-capped, honest partial
mattn/go-colorable root package clean full scan, 0 flakes
urfave/cli v3 root package 0 flakes; budget-capped, honest partial
google/trillian @ pre-#2792 ./log clean full scan, 0 flakes — see note

The trillian note: we checked out the parent of the commit that fixed their order-dependent test and hunted ./log there. The suite passed clean on our machine — their pollution needed a wider run context than the package replay produces. Reported as-is; no witness claimed.

Across all seven scans: zero findings, so zero false findings. We state it that way on purpose — seven scans that emitted nothing do not measure a false positive rate; they only show the hunter stayed quiet on healthy code. Every witness go-flake-hunter does emit must deterministically replay (that is the contract the capsule carries). The targeted cases above add witnesses, but neither they nor these seven scans establish a general false-positive rate. The two targeted reproductions above complete the real-world witness milestone (#1); the seven earlier scans remain evidence only about those specific scans.

Commands

Command What it does Runs tests?
hunt <pkg...> scans packages, classifies, writes capsules ✅
repro <pkg> [Test] shrinks one known suspect (the budget goes to a single test); --from-json FILE|- takes a failed go test -json log instead of typing suspects ✅
compile [id] [--verify] generates a permanent regression test from a capsule; --verify runs the artifact once with --verify
replay [id] runs the witness N times, verifies determinism ✅
explain [id] renders a capsule as a text/JSON/markdown report ❌
init [dir] writes a .github/workflows/go-flake-hunter.yml scaffold ❌

Shared flags: --format text|json|markdown|sarif · --budget N · --timeout 10m · --run-timeout 2m · --seeds N · --repeat N · --capsule-dir DIR · --fail-on-flake · --race · -v · --debug (internal error detail) · --short (scan with -short: the target's own short-skipping tests do not spend the budget). hunt also takes --changed · --base REF · --baseline FILE · --write-baseline · --polluter-pkg PKG (cross-binary suspect, see below) · --sarif FILE (write SARIF 2.1.0 alongside any format — one hunt feeds both the job summary and code scanning). compile also takes --pkg DIR, --print, --dry-run and --verify.

When present, .go-flake-hunter/baseline.json is loaded from the working directory. --baseline FILE selects a different file.

Baselines use commit-independent defect fingerprints (go-flake-hunter/baseline/v2), so an unrelated commit does not turn a known flake into a new finding. The fingerprint covers module, victim package/name, class, polluter identities, environment keys and same-/cross-process shape; run flags and provenance stay in the capsule. Legacy v1 baselines must be regenerated explicitly with go-flake-hunter hunt ./... --write-baseline (add --baseline FILE for a custom path). Scan every previously baselined package: v1 hashes cannot be migrated without their witnesses. Corrupt baseline files are refused, including during --write-baseline.

New regression files include the victim name, for example checkout_TestVictim_goflakehunter_test.go. Compiling a second victim from the same source file preserves the first. Recompiling one victim updates that victim's regression; an existing legacy generated file is reused for its own victim to avoid duplicate declarations. User-owned output files are never overwritten.

compile --verify keeps three verdicts apart — verified (the red reproduced), green-not-reproduced and skipped-missing-env — records the one it got in the capsule's verifyStatus, and exits 3 unless the artifact actually turned red. A skipped run is never a green verification.

replay makes the same distinction on the other side of the contract: a witness that no longer fails is exit 3 ("may already be fixed"), a witness whose package no longer compiles is exit 6. Reading the second as the first is how a mid-refactor tree deletes a real capsule. The witness function name comes from the capsule, so a package that already defines TestFlakeWitness replays under the name the hunt actually used (TestFlakeWitness_2).

--race never narrows the search: the witness is found without race, then verified once with -race. If it does not reproduce, the capsule's "race" field stays false and the report adds a "not reproduced under a race run" note — evidence is never invented.

--format json produces a single document (flakes: [] on a clean run):

{ "schema": "go-flake-hunter/v2", "flakes": [ { "id": "gfh_…", "kind": "order", … } ],
  "runs": 42, "maxRuns": 1000, "packages": [] }

Package-level failures, incomplete runs, and broken tests appear in packages; they do not stop the scan, so the same document also carries every finding from the packages that did run. Partial capsules carry "conclusion": "inconclusive" in reports.

go-flake-hunter init --replay-capsule .go-flake-hunter/gfh_ID.json adds a regression gate for a committed capsule: replayed failure turns CI red, a complete 0/3 replay turns it green, and an incomplete replay reports an unknown result. Cross-binary capsules replay both stages; no generated test is needed.

hunt --jobs N scans up to N package directories concurrently (default min(NumCPU, 4)). All workers share one run/time budget and reports merge in package order. Use --jobs 1 for repositories whose different packages share external state. --polluter-pkg scans are serialized so their two-stage sequences cannot overlap sibling package scans. repro remains serial.

Capsule status Meaning CI behavior
confirmed every verification round reproduced eligible for --fail-on-flake, baseline and compile
suspected some verification rounds reproduced; reports include k/n and Wilson 95% interval advisory SARIF review/note; excluded from gate/baseline; compile refuses
partial scan stopped before a verdict exit 4 for budget, 130 for cancellation

A suspected-only hunt returns 0 with its findings; a suspected-only repro returns 3 because strict reproduction was not achieved. replay can gather new observations from a suspected capsule without promoting its stored status. Intervals summarize the small observed sample; they do not promise a future failure rate. The JSON Action flakes output counts confirmed findings only.

Exit codes

CI scripts bind to these; no path fails silently with 0.

Code Meaning
0 completed hunt (including advisory suspected findings), or confirmed reproduction (gate off)
1 new confirmed flake or test that fails in isolation (--fail-on-flake)
2 usage error
3 repro: the requested test did not reproduce within budget; compile --verify: the artifact stayed green or was skipped for a missing env var
4 budget exceeded (partial capsule written)
5 unsupported witness form or compile of a non-confirmed/cross-clause/cross-binary capsule
6 package does not build
7 a package failed independently of its tests or its run did not complete; the scan still covers the other packages and names this one as skipped
10 internal error (detail with --debug)
130 cancelled (SIGINT)

What it can and cannot find

Class Status Witness
repeat — repeats in the same process ✅ -count=N
order — another test runs first ✅ polluter→victim chain, shrunk with ddmin
parallel — only when concurrent ✅ witness with t.Parallel() on every step + -parallel=N
env — depends on an env var ✅ the key is stored in the capsule, never the value
subtests (t.Run tree) ✅ the parent test is called; verdicts fold onto subtests
TestMain ✅ the witness is installed in the same package, TestMain runs too
internal/external test clauses in one directory ✅ order one test binary, top-level -run selection and recorded shuffle seed; replay supported, compile refuses because one generated file cannot call both clauses
pollution across separate package processes ⚠️ --polluter-pkg only state that survives the process boundary (files, ports): hunt names the two-stage command go test <polluter> && go test <victim>, replay runs both stages. In-memory globals cannot cross two binaries — no witness is claimed. compile refuses these honestly (exit 5) and prints the runnable command
time/network/DB dependent ⚠️ advisory detection static clock/network/database/random calls are named on stderr before probing; external services are not controlled or emulated

Dependency call sites are package-wide syntax hints, not proof of a cause or an excuse to skip a reproducible test. Aliased imports are recognized, values are never captured, and dynamic dispatch may be invisible. A synctest bubble virtualizes its clock but does not make external I/O deterministic. Literal env differences continue to record keys only and give the concrete replay input.

The application-global fixture matrix in testdata/fixtures/app-globals measures a static evidence gap in all three shapes, with no witness recall loss in those three cases. Deeper production-code analysis remains deferred pending measured need.

GitHub Actions

Listed on the Marketplace as go-flake-hunter — the same name as the repository, so the listing and the uses: line below match.

- uses: emrelab/go-flake-hunter@v0.3.0
  with:
    paths: ./...
    format: markdown
    budget: "200"
    # fail-on-flake: "true"   # if you want the gate

Budget defaults differ by surface, on purpose. Local hunt starts at 1000 runs / 5m — a laptop can afford the coverage of ./...; the Action and the go-flake-hunter init scaffold start at 200 runs / 10m, because a shared runner's wall clock is the scarce resource and ten minutes is what a job can spend on one hunt before it looks hung. Pass budget:/--budget and timeout:/--timeout when you want the same shape on both.

The action prefers the prebuilt binary of its own tag (verified against SHA256SUMS) and falls back to building its own source from GITHUB_ACTION_PATH: pinning a release tag is what says which version is running, with no waiting on the proxy. Pin a tag, never a branch — @main means every run of your CI silently executes whatever landed this morning. Go 1.24+ is required (actions/setup-go is on you).

Turn an already-red go test log into a PR comment — repro --from-json shrinks the log's own failures to a witness, no re-scan: docs/CI-REPRO.md.

go-flake-hunter init writes this file into your repo (it never overwrites an existing one). The default is soft mode: a newly adopted flake hunter that turns CI red on day one gets removed on day two.

Design

  • The witness is evidence, not a fix: the generated test does not fix the flake, it reproduces it. It turns green after the fix.
  • Noise-free: the witness file gets a zz_ prefix (Go skips _/. prefixed files) and is deleted after the run; compile output is build-tagged.
  • Budgeted: every run counts, cancellation works; an interrupted run is never read as "passed". --timeout is a wall-clock budget checked between runs: the check never cuts a test in half, so one in-flight run may overshoot by up to its own 2-minute test timeout. A hard ceiling would have to kill a running test, and a killed test reads as "passed" — wrong in the one direction this tool cannot afford.
  • Leak-proof: the env key is stored, the value is not.

Full design: docs/ARCHITECTURE.md.

Development

make test    # go test -race ./...
make build   # ./go-flake-hunter
make fmt     # gofmt

Every module under testdata/fixtures demonstrates a single flake class; adding a new class is the easiest entry point. See CONTRIBUTING.md.

CI dogfoods the tool on its own repo: the self-hunt job runs go-flake-hunter hunt against go-flake-hunter's packages, so a leaked witness file or a global state leak turns our CI red before it could embarrass a user.

License

Apache-2.0 — LICENSE.

About

Reproduce a flaky Go test, shrink it to the smallest polluter→victim witness, and leave a permanent regression test. Used to land flake fixes in tailscale/tailscale, jamf-cli and etcd.

Topics

Resources

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages