Skip to content

About

Where the test time goes in 28 open-source AI projects: wait for green, machine-minutes per commit, share spent getting ready, and whose machines run the jobs. Measured from public GitHub Actions history on 15 Nov 2026. Read-only audit tool included.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

2 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Where the test time goes in open-source AI projects

Measured from public GitHub Actions history on 15 November 2026: 28 projects, 258 pull request commits, 8,531 jobs.

Short answer

  • A pull request waits a median of 9.0 minutes for all its checks to pass. By project, the median runs from 2.3 to 34.5 minutes.
  • About 20% of test machine time is spent getting ready rather than testing: checkout, toolchain setup, dependency install, caches. By project it runs from 2% to 61%.
  • On GitHub's standard Linux machines a job typically starts 2 seconds after it is created, and 0.5% of 5,977 jobs waited over a minute. On GitHub's standard macOS machines 12.7% of 157 jobs waited over a minute.
  • 9 of the 28 projects run 2% or more of their test machine time on machines from another company: Blacksmith (4), Depot (3), Namespace (2), Boat (1). One project uses two of them.

Each number is a reading from one day, not a long-run average. See Limits.

The projects

Each project name links to a page with all of its measures.

project pull requests opened last week wait for green, median min machine-minutes per commit jobs per commit share spent getting ready slowest 10% of jobs wait for a machine, s machines
microsoft/playwright 82 34.5 352.8 23 2% 10 GitHub standard 88%, GitHub larger 12%
pydantic/pydantic-ai 401 22.4 247.2 56 10% 4 GitHub standard 100%
astral-sh/uv 81 19.8 100.8 52 21% 21 Depot 50%, GitHub larger 21%, Namespace 16%
withastro/astro 63 19.2 150.9 23 15% 85 GitHub standard 100%
langchain-ai/langgraph 57 17.3 113.6 62 6% 3 GitHub standard 100%
google-gemini/gemini-cli 56 16.4 139.6 23 12% 65 GitHub larger 100%
n8n-io/n8n 715 14.8 250.9 61 13% 7 Blacksmith 95%, GitHub standard 5%
PostHog/posthog 4,270 14.0 179.2 95 30% 3 GitHub standard 84%, Depot 16%
continuedev/continue 7 13.6 173.3 44 33% 2 GitHub standard 100%
BerriAI/litellm 1,132 12.8 272.7 86 24% 3 GitHub standard 93%, GitHub larger 7%
OpenHands/OpenHands 179 12.5 14.8 3 19% 327 GitHub standard 100%
openclaw/openclaw 3,334 12.3 151.3 50 33% 10 GitHub standard 78%, Boat 22%
zed-industries/zed 176 12.0 56.3 21 16% 23 Namespace 75%, Self-hosted 25%
ollama/ollama 78 9.6 33.7 9 17% 4 GitHub standard 92%, GitHub larger 8%
aaif-goose/goose 102 8.4 70.7 32 34% 2 GitHub standard 100%
e2b-dev/E2B 32 7.3 47.2 22 14% 4 GitHub standard 100%
run-llama/llama_index 114 6.9 20.2 9 24% 2 GitHub standard 77%, GitHub larger 23%
vercel/ai 281 6.6 72.1 48 61% 2 GitHub standard 100%
browser-use/browser-use 64 6.5 39.0 19 25% 4 GitHub standard 100%
cline/cline 74 6.0 14.0 4 21% 4 GitHub standard 100%
langgenius/dify 480 5.9 47.1 29 29% 2 Depot 100%
crewAIInc/crewAI 144 5.8 146.2 49 27% 2 GitHub standard 100%
supabase/supabase 220 5.2 28.7 24 23% 6 Blacksmith 94%, GitHub standard 6%
firecrawl/firecrawl 106 5.2 19.6 7 11% 6 Blacksmith 100%
vitejs/vite 65 4.0 18.4 10 18% 4 GitHub standard 100%
modelcontextprotocol/typescript-sdk 14 3.3 18.5 13 18% 2 GitHub standard 100%
anomalyco/opencode 659 3.1 4.2 4 13% 6 Blacksmith 98%, GitHub standard 2%
langchain-ai/langchain 81 2.3 34.1 66 39% 3 GitHub standard 95%, CodSpeed 5%

Across the 28 projects the median wait for green is 9.0 minutes (range 2.3 to 34.5) and the median getting-ready share is 20% (range 2% to 61%).

Whose machines run the tests

machines projects using them for 2% or more of machine time which
GitHub standard 23 pydantic/pydantic-ai (100%), withastro/astro (100%), langchain-ai/langgraph (100%), continuedev/continue (100%), OpenHands/OpenHands (100%), aaif-goose/goose (100%), e2b-dev/E2B (100%), vercel/ai (100%), browser-use/browser-use (100%), cline/cline (100%), crewAIInc/crewAI (100%), vitejs/vite (100%), modelcontextprotocol/typescript-sdk (100%), langchain-ai/langchain (95%), BerriAI/litellm (93%), ollama/ollama (92%), microsoft/playwright (88%), PostHog/posthog (84%), openclaw/openclaw (78%), run-llama/llama_index (77%), astral-sh/uv (7%), supabase/supabase (6%), n8n-io/n8n (5%)
GitHub larger 6 google-gemini/gemini-cli (100%), run-llama/llama_index (23%), astral-sh/uv (21%), microsoft/playwright (12%), ollama/ollama (8%), BerriAI/litellm (7%)
Blacksmith 4 firecrawl/firecrawl (100%), anomalyco/opencode (98%), n8n-io/n8n (95%), supabase/supabase (94%)
Depot 3 langgenius/dify (100%), astral-sh/uv (50%), PostHog/posthog (16%)
Namespace 2 zed-industries/zed (75%), astral-sh/uv (16%)
CodSpeed 2 astral-sh/uv (7%), langchain-ai/langchain (5%)
Boat 1 openclaw/openclaw (22%)
Self-hosted 1 zed-industries/zed (25%)

How long a job waits for a machine, by whose machine it is

machines system jobs projects typical wait s slowest 10% s slowest 1% s waited over a minute
GitHub standard linux 5,977 24 2 3 21 0.5%
Blacksmith linux 608 4 6 7 8 0.0%
GitHub larger linux 607 6 3 6 71 3.0%
Depot linux 461 3 1 3 3 0.0%
GitHub standard windows 286 15 3 5 239 3.8%
GitHub standard macos 157 10 4 84 500 12.7%
Namespace linux 140 1 9 12 15 0.0%
Boat linux 99 1 10 12 15 0.0%
Namespace macos 66 2 22 39 90 3.0%
Namespace windows 46 1 28 47 105 4.3%
Self-hosted windows 30 1 2 2 79 3.3%
GitHub larger macos 24 1 3 12 15 0.0%

Rough size of each project's pull request test load

Pull requests opened last week times the median machine-minutes of one green commit. A floor, not a bill: most pull requests are pushed more than once, and failed or cancelled runs are not counted.

project pull requests opened last week machine-minutes per commit at least this many machine-minutes a week of which getting ready
PostHog/posthog 4,270 179.2 765,326 226,700
openclaw/openclaw 3,334 151.3 504,601 168,231
BerriAI/litellm 1,132 272.7 308,678 74,959
n8n-io/n8n 715 250.9 179,370 23,661
pydantic/pydantic-ai 401 247.2 99,107 9,928
microsoft/playwright 82 352.8 28,932 714
langgenius/dify 480 47.1 22,604 6,548
crewAIInc/crewAI 144 146.2 21,058 5,737
vercel/ai 281 72.1 20,251 12,283
zed-industries/zed 176 56.3 9,906 1,551
withastro/astro 63 150.9 9,508 1,469
astral-sh/uv 81 100.8 8,161 1,726
google-gemini/gemini-cli 56 139.6 7,818 909
aaif-goose/goose 102 70.7 7,211 2,481
langchain-ai/langgraph 57 113.6 6,475 392
supabase/supabase 220 28.7 6,323 1,441
langchain-ai/langchain 81 34.1 2,765 1,075
anomalyco/opencode 659 4.2 2,751 359
OpenHands/OpenHands 179 14.8 2,646 490
ollama/ollama 78 33.7 2,629 437
browser-use/browser-use 64 39.0 2,493 631
run-llama/llama_index 114 20.2 2,304 543
firecrawl/firecrawl 106 19.6 2,078 225
e2b-dev/E2B 32 47.2 1,510 213
continuedev/continue 7 173.3 1,213 397
vitejs/vite 65 18.4 1,193 213
cline/cline 74 14.0 1,033 221
modelcontextprotocol/typescript-sdk 14 18.5 259 48

Three projects in detail

  • OpenClaw: about 151 machine-minutes per commit, 33% of the machine time getting ready, and the same setup steps taking about the same time on both kinds of machine it uses.
  • vercel/ai: 48 jobs per commit, and getting ready takes 61% of the machine time.
  • PostHog: 95 jobs per commit and the most pull requests of the projects measured.

Method

  • Source: the public GitHub REST API (actions/runs and actions/runs/{id}/jobs), read-only, on 15 November 2026.
  • Sample: for each project, up to 12 of its most recent pull request commits whose pipeline passed cleanly (10 for most projects). A clean pass means: among the runs one push started, the newest run of every workflow finished on its first attempt with success or was skipped.
  • Wait for green: from the moment the first job is queued to the moment the last job finishes.
  • Machine-minutes: the sum of every job's start-to-finish time. Skipped jobs count as zero.
  • Getting ready: time in steps whose names show runner start and cleanup, checkout, toolchain setup, dependency install, cache restore and save, container pull and login, or artifact upload and download. Everything else counts as the work (build, test, lint). Steps are sorted by name, so a step named in an unusual way can land in the wrong group.
  • Wait for a machine: from a job being created to a machine starting it. It includes time a job spends waiting for a concurrency slot the project set itself.
  • Machines: read from each job's runs-on labels and runner name.
  • Pull requests opened last week: GitHub search, 8 to 14 November 2026.
  • Read but left out of the tables for having fewer than 4 clean passes: denoland/deno (1), grafana/grafana (3), open-webui/open-webui (1). Two more, calcom/cal.diy and openai/codex, had none.

Limits

  • Ten or so commits a project is a small sample taken on one day. Treat every number as a reading, not a long-run average.
  • Only clean passes are measured. Failed and cancelled runs, which also use machines, are left out.
  • Pull requests from first-time contributors often wait for a maintainer to approve the run. That wait is not counted.
  • Standard GitHub machines are free for public projects, so machine-minutes here are not a bill.
  • Where steps run in parallel inside one job, their times are summed, so step time can slightly exceed job time.
  • 15 November 2026 was a Sunday. Waits for a machine may be longer on a weekday.

Run it yourself

You need Python 3.9 or newer and the GitHub command line tool, signed in (gh auth login). The tool only reads: it makes GET requests and writes nothing to GitHub. One project takes about 100 to 250 requests.

python3 ci_audit.py fetch owner/repo another/repo   # pull runs, jobs and step timings into data/
python3 ci_audit.py analyze                         # compute the measures into out/
python3 make_summary.py                             # write the cross-project table to out/summary.md
path what
ci_audit.py the audit: fetch and analyze
make_summary.py builds the cross-project table
out/summary.md, out/summary.json, out/summary.csv the cross-project measures
out/detail/ every measure for each project
out/pr_volume.json pull requests opened and merged, 8 to 14 November 2026
reports/ three projects written up

The raw history the tool downloads (data/) is not included here. fetch recreates it.

Corrections

This is an independent measurement. Nobody at the projects or companies named here took part in it or was asked to. If a number is wrong, or a step of your project is grouped wrongly, please open an issue so it can be corrected.

How this was made

A script made the measurements. The write-ups were drafted with AI assistance, and every number in them was checked against the measured files.

About

Where the test time goes in 28 open-source AI projects: wait for green, machine-minutes per commit, share spent getting ready, and whose machines run the jobs. Measured from public GitHub Actions history on 15 Nov 2026. Read-only audit tool included.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages