Skip to content

ci: cut the batch E2E by about an hour without dropping a check - #562

Merged
GeiserX merged 139 commits into
GeiserX:mainfrom
geiresx:pgp/ci-speed-1
Nov 21, 2026
Merged

GeiserX merged 139 commits into
GeiserX:mainfrom
geiresx:pgp/ci-speed-1

Conversation

@geiresx

@geiresx geiresx commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

A batch of pull requests waits about three hours for its E2E run (median 178 minutes over 18 dispatched runs on the fork, range 147 to 226). Most of that is one critical path: og (8 to 12 min), then e2e base (30 to 51 min), then the slowest shard (83 to 122 min). Four things made that path longer than it has to be:

  1. On a run dispatched on a branch, prs-3-panel-measure measures the 1000 Genomes panel for 45 to 58 minutes, but scripts/ci/e2e-case-times.tsv weighs it at 0 seconds. Its shard got a full share of other cases on top and finished last in 12 of 18 runs.
  2. e2e base waited for og, and og built four binaries (Linux x86_64 and aarch64, macOS arm64 and x86_64) when the legs run only og-linux-x86_64.
  3. A change that picks every image starts about 66 Container Test smoke rows at once. They held the account's 20 runners while E2E jobs of the same commit queued for up to 54 minutes.
  4. Per-branch Rust caches filled the fork's 10 GB cache budget (10.69 GB), so the E2E image and data caches could not be saved. Those Rust caches did not make the release builds faster.

Fix

Every case and every check still runs. Only the order, the split and the caching change.

  • scripts/ci/e2e-shards.sh takes E2E_SECONDS (CASE=SECONDS ...) and uses those times in place of the table's for the run. e2e.yml sets prs-3-panel-measure=2900 (its median on a dispatched branch run) for dispatched runs off main, at workflow level, so every job of a run splits alike. A new verb seconds prints the time each case is weighed by; the plan summary uses it and names the override. A name with no row or a time that is not a whole number stops the split.
  • rust/og.sh builds and checks a subset of arches with OG_ARCHES and OG_DARWIN_ARCHES. Unset, it builds and requires all four, as before. The E2E og job and the legs set OG_ARCHES=x86_64 OG_DARWIN_ARCHES=. Guard's rust job still builds all four on every pull request, as does the Rust images job when it runs. The Rust streams job builds only the runner's binary, which is the one it runs. So the nightly Rust run (streams only) now builds only og-linux-x86_64; every pull request still builds all four in Guard rust.
  • e2e base no longer needs the job og. It frees disk, sets up Docker, Java and Nextflow and pulls the images while og builds, then waits for the og-e2e artifact (new step "Wait for og"). The step fails the leg when og ends without the artifact, or after 60 minutes. The shards still start only after the base leg passed.
  • The required check e2e now needs the job og too, and its step "Every case passed once" fails unless og succeeded. Before this change e2e base needed og, so a red og turned e2e red through it; now that e2e base only waits for the artifact, an og job that ends failure or cancelled after its upload would otherwise leave e2e green.
  • Container Test smoke rows run at most 8 at a time (max-parallel: 8). The row count is unchanged.
  • Jobs that only build og --release lose their rust-cache step. Guard's rust job keeps its cache (it saves 1 to 2 minutes on clippy and the tests) but saves it on main only (save-if); branches read main's entry.

Decision, named here: the rust-cache removal also touches docs.yml, nextflow.yml, personal-data.yml and freshness.yml, which are outside this package's file list. The survey's change 8 names them ("the same in the other fast workflows"), and leaving them would keep filling the cache budget the E2E needs. The edit in each is the removal of one rust-cache step and nothing else.

docs/testing.md and CHANGELOG.md describe the new split, the overlap and the cache budget.

How it was checked

  • The og need of e2e, red then green on the step's own logic: the bash of "Every case passed once", cut after its job-result test, run with PLAN, BASE and LEGS success. The old text exits 0 with OG=failure, cancelled and skipped (the hole). The new text exits 1 for each, with "The plan job ended success, the job og failure, the base leg success and the shard legs success; e2e needs all four to succeed.", and exits 0 with OG=success. actionlint 1.7.12 is clean on every workflow at the new head; a copy with og removed from the needs of e2e fails it (property "og" is not defined, rc 1). No new E2E was dispatched for this commit: the next batch-tip E2E runs it.

  • actionlint 1.7.12 (with ShellCheck) on every changed workflow: clean. Positive control: a copy of e2e.yml that reads needs.og.result in the base leg, where og is no longer a need, fails actionlint.

  • ShellCheck 0.11.0 -S style on rust/og.sh and scripts/ci/e2e-shards.sh: clean.

  • tests/test_e2e_shards.sh and tests/test_og_check.sh: pass ("Every one of the 168 cases is listed once" on the real table with 6 shards).

  • rust/og.sh by hand: with only og-linux-x86_64 built, check --required fails by default ("not built for aarch64") and passes with OG_ARCHES=x86_64 OG_DARWIN_ARCHES=; a stale subset binary fails; OG_ARCHES=armv7 and OG_DARWIN_ARCHES=win stop with exit 2.

  • Fork fast checks on the head 4834af1 (dispatched), all green: Guard https://github.com/geiresx/OwnGenome/actions/runs/44901138304, Lint https://github.com/geiresx/OwnGenome/actions/runs/44901151691, Personal data https://github.com/geiresx/OwnGenome/actions/runs/44901164988, Docs https://github.com/geiresx/OwnGenome/actions/runs/44901179104, Nextflow https://github.com/geiresx/OwnGenome/actions/runs/44901194333, Container Test https://github.com/geiresx/OwnGenome/actions/runs/44901209883, Freshness https://github.com/geiresx/OwnGenome/actions/runs/44901218033, Rust https://github.com/geiresx/OwnGenome/actions/runs/44901237349. In Container Test, the row smoke (PARASCOPY_IMAGE) failed once because Zenodo answered 504 Gateway Time-out to its data download. This pull request does not change that row or its data, so only the failed jobs were re-run, and attempt 2 is green. The same checks were green on eed7dc5 too.

  • Dispatched E2E on eed7dc5 on the fork: https://github.com/geiresx/OwnGenome/actions/runs/44842271308, green on attempt 2 ("0 of 168 cases failed or did not run once"). Times next to the survey's medians and ranges:

    this run survey (18 runs)
    og 5.2 min, one binary 10 min (8 to 12)
    e2e base 36.4 min, started during og 43 min (30 to 51), after og
    leg with prs-3-panel-measure 46.9 min (shard 1) 83 to 122 min (shard 6, last in 12 of 18 runs)
    leg with the Nextflow chain 100.7 min (shard 2, EPYC 7763) 68 to 99 min (shard 1)
    slowest shard 100.7 min 94 to 122 min
    dispatch to the last healthy shard 138.6 min wall 178 min (147 to 226)

    Per shard: 1 = 46.9, 2 = 100.7, 3 = 50.2, 4 = 65.7 (attempt 2), 5 = 80.8, 6 = 74.7 min.

    The plan gave every shard 57 to 62 min. Shard 2 ran longest because its four Nextflow cases need each other, so they stay in one leg, and that leg got an EPYC 7763 (the chain took 78 min there).

  • Shard 4 of attempt 1 ran until its 240-minute limit. It was not the split and not a hung case. That runner was slow at everything: the same minimap2 index of 14 sequences took 110 to 116 s there and 12 to 17 s on every other leg. Its image pull took 1428 s, against 274 to 592 s on the others. Every case ran 3.6 times over its table time. On a new runner (rerun of the failed jobs, attempt 2), the same 26 cases took 53 min against a planned 59, and the index took 15 to 18 s. The run already supports this rerun: the shards fetch e2e-base from the earlier attempt, and the report uploads overwrite.

  • Planted failures, throwaway branches off eed7dc5, never merged:

No bead: this package comes from the CI speed survey.

Summary by CodeRabbit

  • CI Improvements
    • Container smoke tests now run up to eight jobs concurrently.
    • E2E test runs support case-time overrides and improved artifact handling; selected builds are limited to the runner’s architecture.
    • Build tooling supports configurable Linux and macOS architectures, including skipping macOS builds when no targets are configured.
  • Documentation
    • Updated testing guidance for E2E timing overrides, artifact handling, image caching, smoke-test concurrency, and runner-specific builds.
  • Chores
    • Updated unreleased notes to reflect CI and resource changes.

A red-first og-browser check on a small demo site with a stub server: results.json,
runs.json and family.json failing, frames.json and results.json unreadable, and
frames.json held. Red on main for every case it adds.

Bead: U05
Bead: S02
Bead: S03
Bead: S04
Bead: S13
A failed results.json read left the report empty, so a finished run read as a
finished report with no results. The report now says so, with Try again.

Bead: U05
A failed or aborted runs.json read left only the server's plain list. The home
page now says so above it, with Try again, and keeps the 30 s retry.

Bead: U05
Bead: S02
family.json answering {} passed as a family, and drawnKey was set before draw()
threw, so the page stayed blank. Such an answer now fails like an error answer.

Bead: S03
A person page whose state could not be read stays closed, but now reads
"Results not loaded" and no longer says child.

Bead: S04
…none

The wire, run health and re-check panels said nothing existed before the first
state arrived. They now read "Waiting for the run's data." until it does.

Bead: S13
data-pgp-status tells whether the run's state arrived, and the live-site tests read
it that way. A failed results.json now shows its line on the page only.

Bead: U05
draw() cleared the page before building the new one, so a later answer that threw
left it blank. The new page now replaces the old one only once it is built.

Bead: S03
…no re-check

Browser::go returns once the navigation starts, so a check could read the page
before. Each visit now goes through about:blank and waits for its address. An
adult whose first state has no re-check must read "No re-check yet".

Bead: S13
A person page whose state parses but carries no plugins kept the gate closed
but named a child. It now marks the page unread, the same condition that
makes it a possible child for want of data. The unread lede now speaks of the
state only, so it stays true when the results are readable and counted.

The three Try again checks now need the click to find the button and the
page to draw within 5 s, so main's own 30 s retry cannot pass them. A new
gate case covers a state with no plugins.

Bead: S04
Bead: U05
og dev evidence y-bench places a published haplogroup (Poznik et al.
2016, written <clade>-<SNP>) and each tool's call on the current YFull
tree, names each call exact, below, ancestor, sibling, wrong, no_call or
unknown_name, and fails when a tool puts fewer than MIN of its men on
the published branch or below it. A published SNP the tree does not
have, or has on two branches the clade letter does not settle, is
refused: it is the truth. Unit tests cover every result, the names as
Yleaf and the paper write them, the gate and its red control (the same
calls given to other men).

Bead: lin-24
The new y-benchmark job slices each man's chrY from his NYGC 30x CRAM,
runs scripts/37-y-haplogroup.sh on it and scores both tools with og dev
evidence y-bench (Y_MIN 0.95). Its red control hands each tool's calls
to the man half the list away and must fail on the gate. The inputs are
pinned: Poznik et al.'s calls and the CRAM index by sha256, chrY of the
reference by its M5. archaic-sites now runs only when an archaic job
reads its file, so a Y-only dispatch skips that download.

Bead: D14
Both tools put all 58 men on the published branch or below it and named
the same branch for 44. The outvote rule fired for one man, whose
placement is on his published line, so it stays as it is. The docs give
how the men were chosen, what each result means, the limits (the
published tree is shallow, so the levels below it are not checked) and
each man's calls.

Bead: lin-24
Bead: D14
…loaded

An answer like {"chapters": {}} passed the shape check, so the page set
results, threw while drawing it, and kept a broken report: the failure line
never showed because results was already set. The person page now takes
chapters only as a list, or a note alone, and says the results could not be
loaded otherwise. A new check covers that answer.

Bead: S02
…fferent haplogroups

The workflow header and the y_samples input said 60 men; the table has 58. The selection rule now says branches with the same count are taken in the sort order of their published name, so the 58 can be re-derived. The y_samples input says the men must come from different haplogroups, because the red control hands each man's calls to another.

Bead: lin-24
…t.md

og dev --help, the og-dev usage and module doc, and docs/rust.md listed only the clashes and hg002-gvcf evidence verbs.

Bead: lin-24
The smn tests wrote each table to a folder keyed by the process id and
the row count. Two one-row tests ran in parallel threads, shared that
folder, and the first to drop removed it while the other still wrote
its table (NotFound or AlreadyExists, about 1 run in 1000).

Each table now gets a folder named by a process-wide counter, and a new
test proves two tables of the same size never share one.
The red control hands each man the calls of the man half the list away. With men on one line (three R1b-DF27 men, say) those calls stay on the published line, the rotated table passes the gate, and a correct run fails as 'red control: the rotated calls passed the gate'. Before any CRAM is read, the job now places each man's published branch and his partner's on the tree with og dev evidence y-bench, and stops when enough partners share a line (exact, below or ancestor) that the handed calls could still reach Y_MIN.

Bead: lin-24
…rror

The viewer page styled only its header, so the browser's white showed
under it, and igv.js's bar widened the page to 767 px on a 390 px phone.
The page now uses the live site's dark palette, keeps igv.js's white
tracks in a framed panel that scrolls sideways, and has one h1 naming
the sample, a main landmark and a line on what the tracks are and what
search takes. When the viewer cannot start, #error (role alert) says
what to check and offers Try again, which loads the page again.

Bead: U08
Bead: U09
Bead: U04
A stub serves the page with og view serve's policy (kept equal to
og-live's), the pinned igv.js and a synthetic reference and VCF. It
checks the dark page, one h1 and one main, the phone width, and the
config error box with its Try again. Each check was red on main's page.

Bead: U08
Bead: U09
Bead: U04
…d source

og dev licences check reads assets/SOURCES.tsv and fails, naming each fault,
when a file under assets/ that git tracks or would add has no row, a file
is in two rows, a row matches no file, a row has no source or licence, a
file lacks the header phrase its row names, or NOTICE does not name a
third-party row. Unit tests plant each fault, and one runs the verb on a
real git tree with an untracked and an ignored file.

Bead: lic-03
assets/SOURCES.tsv lists every file under assets/, and the consumer export
fixture, with its source, version, licence and the header phrase it must
carry. NOTICE names the 15 third-party rows and their terms. Lint's new
data-sources job builds og and runs og dev licences check, then two
controls that must fail: an asset with no row, and sv_rule_regions.tsv
with its source line removed. README and CHANGELOG say so.

Bead: lic-03
pypgx writes Indeterminate for a gene it could not call, and step 32
writes it for CYP2D6 when the depth check fails. Step 27 read it as a
diplotype: the comparison listed it as a pypgx call or a mismatch, and a
gene PharmCAT could not call got a warning that pypgx called
Indeterminate. Step 36 (and the summary, which shares its rule) read a
diplotype with one Indeterminate allele as a call that could disagree
with Cyrius. Indeterminate in any case, and as any allele, is now no call.

Bead: pgx-06
Bead: pgx-30
… list

The medication list was matched by the first word of each line, so
olanzapine/fluoxetine was checked as olanzapine alone, and
naltrexone/bupropion was listed as not in the inhibitor table although
bupropion is a strong CYP2D6 inhibitor. A line now names every word that
is not a dose, a unit, a frequency or "and"; whitespace, commas,
semicolons, "/" and "+" split it. Phenoconversion takes the strongest
inhibitor among all of them, a line is unchecked only when none of its
drugs is in the table, and the contraindication lines see every drug.

port_tests_pgx_cpic.rs pinned the old first-word rule for
lopinavir/ritonavir; its assertion now expects both drugs.

Bead: pgx-16
i411_pypgx_nocall.rs runs og pgx cpic-report: an Indeterminate pypgx
CYP2D6 (any case, or as one allele) gives no comparison row and no
warning, and an Indeterminate CYP2C9 is "PharmCAT only", never a
mismatch; olanzapine/fluoxetine converts CYP2D6 by fluoxetine,
naltrexone/bupropion 8/90 mg by bupropion and is not listed as
unchecked, and the second drug of lamotrigine/valproate is marked on the
medication list. Each case has a control that must not change.

The pgx-parse pheno-boosted case now names both lopinavir and ritonavir
as CYP3A4 and CYP3A5 inhibitors: its two outputs get goldens.

Bead: pgx-06
Bead: pgx-30
Bead: pgx-16
# Conflicts:
#	CHANGELOG.md
#	rust/crates/og-dev/src/lib.rs
GeiserX#560 ports og live serve, so the crate rows of docs/rust.md and
rust/README.md and the stub list now name only results and demo as
not ported.
GeiserX#561 finished og report render and switched the steps to it, but the
og-render row of rust/README.md still said render had no caller and
index was not ported; it now matches docs/rust.md.
GeiserX#547 added rv_live_load_errors.rs and GeiserX#554 added rv_viewer_shell.rs, but
the table of checks in tests/browser/README.md named neither.
GeiserX#560 changed the empty re-check banner in assets/live/plugins/recheck.js to
name bin/og recheck, and GeiserX#547's rv_live_load_errors.rs still expected the
old bin/owngenome_recheck.py text, so its DEMO-002 check failed on the tip.
…rs to it

The entry for the four readers had replaced GeiserX#567's, which still explains
the changed consumer summary counts. Its text is back, with the readers
this branch fixes added at its end, and the test file's header now names
the past-the-end rule it also checks.

Bead: C16
…thon renderer

Since GeiserX#561, step 24, the Nextflow HTML_REPORT module and generate-report.sh
run og report render. The measurement table and its command line still
named render_report.py as what the steps run. Keep the measured numbers,
say they are the Python renderer's, and give the og report render command
line to time today's step the same way.
…oad no longer fails a batch

On a dispatched run, prs-3 waits up to 30 minutes for the panel download
and then gives pgsc_calc 35 minutes. When EBI served the 7.4 GB panel in
31 minutes instead of 13, the case passed the runner's 60-minute limit.
It now raises its own limit to 90 minutes, as the Locityper case does.
…t costs today

A traced run puts every case in one leg with a 355-minute limit, and its
time table keeps prs-3 at 0 s, so the 35 to 80 minute measurement does
not belong there. PGSC_MEASURE=1 still forces it. The case header and
docs/25-prs.md no longer cite the build-fixture job's 150 minutes or say
the download outlasts a pull request's run.
@GeiserX
GeiserX merged commit 9dd5dda into GeiserX:main Nov 21, 2026
116 of 123 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants