Repository navigation
ci: cut the batch E2E by about an hour without dropping a check - #562
Merged
Merged
Conversation
A red-first og-browser check on a small demo site with a stub server: results.json, runs.json and family.json failing, frames.json and results.json unreadable, and frames.json held. Red on main for every case it adds. Bead: U05 Bead: S02 Bead: S03 Bead: S04 Bead: S13
A failed results.json read left the report empty, so a finished run read as a finished report with no results. The report now says so, with Try again. Bead: U05
A failed or aborted runs.json read left only the server's plain list. The home page now says so above it, with Try again, and keeps the 30 s retry. Bead: U05 Bead: S02
family.json answering {} passed as a family, and drawnKey was set before draw()
threw, so the page stayed blank. Such an answer now fails like an error answer.
Bead: S03
A person page whose state could not be read stays closed, but now reads "Results not loaded" and no longer says child. Bead: S04
…none The wire, run health and re-check panels said nothing existed before the first state arrived. They now read "Waiting for the run's data." until it does. Bead: S13
data-pgp-status tells whether the run's state arrived, and the live-site tests read it that way. A failed results.json now shows its line on the page only. Bead: U05
draw() cleared the page before building the new one, so a later answer that threw left it blank. The new page now replaces the old one only once it is built. Bead: S03
…no re-check Browser::go returns once the navigation starts, so a check could read the page before. Each visit now goes through about:blank and waits for its address. An adult whose first state has no re-check must read "No re-check yet". Bead: S13
# Conflicts: # CHANGELOG.md
A person page whose state parses but carries no plugins kept the gate closed but named a child. It now marks the page unread, the same condition that makes it a possible child for want of data. The unread lede now speaks of the state only, so it stays true when the results are readable and counted. The three Try again checks now need the click to find the button and the page to draw within 5 s, so main's own 30 s retry cannot pass them. A new gate case covers a state with no plugins. Bead: S04 Bead: U05
og dev evidence y-bench places a published haplogroup (Poznik et al. 2016, written <clade>-<SNP>) and each tool's call on the current YFull tree, names each call exact, below, ancestor, sibling, wrong, no_call or unknown_name, and fails when a tool puts fewer than MIN of its men on the published branch or below it. A published SNP the tree does not have, or has on two branches the clade letter does not settle, is refused: it is the truth. Unit tests cover every result, the names as Yleaf and the paper write them, the gate and its red control (the same calls given to other men). Bead: lin-24
The new y-benchmark job slices each man's chrY from his NYGC 30x CRAM, runs scripts/37-y-haplogroup.sh on it and scores both tools with og dev evidence y-bench (Y_MIN 0.95). Its red control hands each tool's calls to the man half the list away and must fail on the gate. The inputs are pinned: Poznik et al.'s calls and the CRAM index by sha256, chrY of the reference by its M5. archaic-sites now runs only when an archaic job reads its file, so a Y-only dispatch skips that download. Bead: D14
Both tools put all 58 men on the published branch or below it and named the same branch for 44. The outvote rule fired for one man, whose placement is on his published line, so it stays as it is. The docs give how the men were chosen, what each result means, the limits (the published tree is shallow, so the levels below it are not checked) and each man's calls. Bead: lin-24 Bead: D14
…loaded
An answer like {"chapters": {}} passed the shape check, so the page set
results, threw while drawing it, and kept a broken report: the failure line
never showed because results was already set. The person page now takes
chapters only as a list, or a note alone, and says the results could not be
loaded otherwise. A new check covers that answer.
Bead: S02
# Conflicts: # CHANGELOG.md
# Conflicts: # CHANGELOG.md
…fferent haplogroups The workflow header and the y_samples input said 60 men; the table has 58. The selection rule now says branches with the same count are taken in the sort order of their published name, so the 58 can be re-derived. The y_samples input says the men must come from different haplogroups, because the red control hands each man's calls to another. Bead: lin-24
…t.md og dev --help, the og-dev usage and module doc, and docs/rust.md listed only the clashes and hg002-gvcf evidence verbs. Bead: lin-24
The smn tests wrote each table to a folder keyed by the process id and the row count. Two one-row tests ran in parallel threads, shared that folder, and the first to drop removed it while the other still wrote its table (NotFound or AlreadyExists, about 1 run in 1000). Each table now gets a folder named by a process-wide counter, and a new test proves two tables of the same size never share one.
The red control hands each man the calls of the man half the list away. With men on one line (three R1b-DF27 men, say) those calls stay on the published line, the rotated table passes the gate, and a correct run fails as 'red control: the rotated calls passed the gate'. Before any CRAM is read, the job now places each man's published branch and his partner's on the tree with og dev evidence y-bench, and stops when enough partners share a line (exact, below or ancestor) that the handed calls could still reach Y_MIN. Bead: lin-24
…rror The viewer page styled only its header, so the browser's white showed under it, and igv.js's bar widened the page to 767 px on a 390 px phone. The page now uses the live site's dark palette, keeps igv.js's white tracks in a framed panel that scrolls sideways, and has one h1 naming the sample, a main landmark and a line on what the tracks are and what search takes. When the viewer cannot start, #error (role alert) says what to check and offers Try again, which loads the page again. Bead: U08 Bead: U09 Bead: U04
A stub serves the page with og view serve's policy (kept equal to og-live's), the pinned igv.js and a synthetic reference and VCF. It checks the dark page, one h1 and one main, the phone width, and the config error box with its Try again. Each check was red on main's page. Bead: U08 Bead: U09 Bead: U04
Bead: U08 Bead: U09 Bead: U04
…d source og dev licences check reads assets/SOURCES.tsv and fails, naming each fault, when a file under assets/ that git tracks or would add has no row, a file is in two rows, a row matches no file, a row has no source or licence, a file lacks the header phrase its row names, or NOTICE does not name a third-party row. Unit tests plant each fault, and one runs the verb on a real git tree with an untracked and an ignored file. Bead: lic-03
assets/SOURCES.tsv lists every file under assets/, and the consumer export fixture, with its source, version, licence and the header phrase it must carry. NOTICE names the 15 third-party rows and their terms. Lint's new data-sources job builds og and runs og dev licences check, then two controls that must fail: an asset with no row, and sv_rule_regions.tsv with its source line removed. README and CHANGELOG say so. Bead: lic-03
pypgx writes Indeterminate for a gene it could not call, and step 32 writes it for CYP2D6 when the depth check fails. Step 27 read it as a diplotype: the comparison listed it as a pypgx call or a mismatch, and a gene PharmCAT could not call got a warning that pypgx called Indeterminate. Step 36 (and the summary, which shares its rule) read a diplotype with one Indeterminate allele as a call that could disagree with Cyrius. Indeterminate in any case, and as any allele, is now no call. Bead: pgx-06 Bead: pgx-30
… list The medication list was matched by the first word of each line, so olanzapine/fluoxetine was checked as olanzapine alone, and naltrexone/bupropion was listed as not in the inhibitor table although bupropion is a strong CYP2D6 inhibitor. A line now names every word that is not a dose, a unit, a frequency or "and"; whitespace, commas, semicolons, "/" and "+" split it. Phenoconversion takes the strongest inhibitor among all of them, a line is unchecked only when none of its drugs is in the table, and the contraindication lines see every drug. port_tests_pgx_cpic.rs pinned the old first-word rule for lopinavir/ritonavir; its assertion now expects both drugs. Bead: pgx-16
i411_pypgx_nocall.rs runs og pgx cpic-report: an Indeterminate pypgx CYP2D6 (any case, or as one allele) gives no comparison row and no warning, and an Indeterminate CYP2C9 is "PharmCAT only", never a mismatch; olanzapine/fluoxetine converts CYP2D6 by fluoxetine, naltrexone/bupropion 8/90 mg by bupropion and is not listed as unchecked, and the second drug of lamotrigine/valproate is marked on the medication list. Each case has a control that must not change. The pgx-parse pheno-boosted case now names both lopinavir and ritonavir as CYP3A4 and CYP3A5 inhibitors: its two outputs get goldens. Bead: pgx-06 Bead: pgx-30 Bead: pgx-16
# Conflicts: # CHANGELOG.md # rust/crates/og-dev/src/lib.rs
GeiserX#560 ports og live serve, so the crate rows of docs/rust.md and rust/README.md and the stub list now name only results and demo as not ported.
GeiserX#561 finished og report render and switched the steps to it, but the og-render row of rust/README.md still said render had no caller and index was not ported; it now matches docs/rust.md.
GeiserX#547 added rv_live_load_errors.rs and GeiserX#554 added rv_viewer_shell.rs, but the table of checks in tests/browser/README.md named neither.
GeiserX#560 changed the empty re-check banner in assets/live/plugins/recheck.js to name bin/og recheck, and GeiserX#547's rv_live_load_errors.rs still expected the old bin/owngenome_recheck.py text, so its DEMO-002 check failed on the tip.
…rs to it The entry for the four readers had replaced GeiserX#567's, which still explains the changed consumer summary counts. Its text is back, with the readers this branch fixes added at its end, and the test file's header now names the past-the-end rule it also checks. Bead: C16
# Conflicts: # CHANGELOG.md
# Conflicts: # CHANGELOG.md
# Conflicts: # CHANGELOG.md
# Conflicts: # CHANGELOG.md
# Conflicts: # CHANGELOG.md
# Conflicts: # CHANGELOG.md
# Conflicts: # CHANGELOG.md
# Conflicts: # CHANGELOG.md
…thon renderer Since GeiserX#561, step 24, the Nextflow HTML_REPORT module and generate-report.sh run og report render. The measurement table and its command line still named render_report.py as what the steps run. Keep the measured numbers, say they are the Python renderer's, and give the og report render command line to time today's step the same way.
# Conflicts: # CHANGELOG.md
# Conflicts: # CHANGELOG.md
…oad no longer fails a batch On a dispatched run, prs-3 waits up to 30 minutes for the panel download and then gives pgsc_calc 35 minutes. When EBI served the 7.4 GB panel in 31 minutes instead of 13, the case passed the runner's 60-minute limit. It now raises its own limit to 90 minutes, as the Locityper case does.
…t costs today A traced run puts every case in one leg with a 355-minute limit, and its time table keeps prs-3 at 0 s, so the 35 to 80 minute measurement does not belong there. PGSC_MEASURE=1 still forces it. The case header and docs/25-prs.md no longer cite the build-fixture job's 150 minutes or say the download outlasts a pull request's run.
…) into the T14 batch tip
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A batch of pull requests waits about three hours for its E2E run (median 178 minutes over 18 dispatched runs on the fork, range 147 to 226). Most of that is one critical path:
og(8 to 12 min), thene2e base(30 to 51 min), then the slowest shard (83 to 122 min). Four things made that path longer than it has to be:prs-3-panel-measuremeasures the 1000 Genomes panel for 45 to 58 minutes, butscripts/ci/e2e-case-times.tsvweighs it at 0 seconds. Its shard got a full share of other cases on top and finished last in 12 of 18 runs.e2e basewaited forog, andogbuilt four binaries (Linux x86_64 and aarch64, macOS arm64 and x86_64) when the legs run onlyog-linux-x86_64.Fix
Every case and every check still runs. Only the order, the split and the caching change.
scripts/ci/e2e-shards.shtakesE2E_SECONDS(CASE=SECONDS ...) and uses those times in place of the table's for the run.e2e.ymlsetsprs-3-panel-measure=2900(its median on a dispatched branch run) for dispatched runs off main, at workflow level, so every job of a run splits alike. A new verbsecondsprints the time each case is weighed by; the plan summary uses it and names the override. A name with no row or a time that is not a whole number stops the split.rust/og.shbuilds and checks a subset of arches withOG_ARCHESandOG_DARWIN_ARCHES. Unset, it builds and requires all four, as before. The E2Eogjob and the legs setOG_ARCHES=x86_64 OG_DARWIN_ARCHES=. Guard's rust job still builds all four on every pull request, as does the Rust images job when it runs. The Ruststreamsjob builds only the runner's binary, which is the one it runs. So the nightly Rust run (streams only) now builds onlyog-linux-x86_64; every pull request still builds all four in Guard rust.e2e baseno longer needs the jobog. It frees disk, sets up Docker, Java and Nextflow and pulls the images whileogbuilds, then waits for theog-e2eartifact (new step "Wait for og"). The step fails the leg whenogends without the artifact, or after 60 minutes. The shards still start only after the base leg passed.e2enow needs the jobogtoo, and its step "Every case passed once" fails unlessogsucceeded. Before this changee2e baseneededog, so a redogturnede2ered through it; now thate2e baseonly waits for the artifact, anogjob that ends failure or cancelled after its upload would otherwise leavee2egreen.max-parallel: 8). The row count is unchanged.og --releaselose theirrust-cachestep. Guard's rust job keeps its cache (it saves 1 to 2 minutes on clippy and the tests) but saves it on main only (save-if); branches read main's entry.Decision, named here: the rust-cache removal also touches
docs.yml,nextflow.yml,personal-data.ymlandfreshness.yml, which are outside this package's file list. The survey's change 8 names them ("the same in the other fast workflows"), and leaving them would keep filling the cache budget the E2E needs. The edit in each is the removal of onerust-cachestep and nothing else.docs/testing.mdandCHANGELOG.mddescribe the new split, the overlap and the cache budget.How it was checked
The
ogneed ofe2e, red then green on the step's own logic: the bash of "Every case passed once", cut after its job-result test, run with PLAN, BASE and LEGS success. The old text exits 0 withOG=failure,cancelledandskipped(the hole). The new text exits 1 for each, with "The plan job ended success, the job og failure, the base leg success and the shard legs success; e2e needs all four to succeed.", and exits 0 withOG=success. actionlint 1.7.12 is clean on every workflow at the new head; a copy withogremoved from the needs ofe2efails it (property "og" is not defined, rc 1). No new E2E was dispatched for this commit: the next batch-tip E2E runs it.actionlint 1.7.12 (with ShellCheck) on every changed workflow: clean. Positive control: a copy of
e2e.ymlthat readsneeds.og.resultin the base leg, whereogis no longer a need, fails actionlint.ShellCheck 0.11.0
-S styleonrust/og.shandscripts/ci/e2e-shards.sh: clean.tests/test_e2e_shards.shandtests/test_og_check.sh: pass ("Every one of the 168 cases is listed once" on the real table with 6 shards).rust/og.shby hand: with onlyog-linux-x86_64built,check --requiredfails by default ("not built for aarch64") and passes withOG_ARCHES=x86_64 OG_DARWIN_ARCHES=; a stale subset binary fails;OG_ARCHES=armv7andOG_DARWIN_ARCHES=winstop with exit 2.Fork fast checks on the head 4834af1 (dispatched), all green: Guard https://github.com/geiresx/OwnGenome/actions/runs/44901138304, Lint https://github.com/geiresx/OwnGenome/actions/runs/44901151691, Personal data https://github.com/geiresx/OwnGenome/actions/runs/44901164988, Docs https://github.com/geiresx/OwnGenome/actions/runs/44901179104, Nextflow https://github.com/geiresx/OwnGenome/actions/runs/44901194333, Container Test https://github.com/geiresx/OwnGenome/actions/runs/44901209883, Freshness https://github.com/geiresx/OwnGenome/actions/runs/44901218033, Rust https://github.com/geiresx/OwnGenome/actions/runs/44901237349. In Container Test, the row
smoke (PARASCOPY_IMAGE)failed once because Zenodo answered 504 Gateway Time-out to its data download. This pull request does not change that row or its data, so only the failed jobs were re-run, and attempt 2 is green. The same checks were green on eed7dc5 too.Dispatched E2E on eed7dc5 on the fork: https://github.com/geiresx/OwnGenome/actions/runs/44842271308, green on attempt 2 ("0 of 168 cases failed or did not run once"). Times next to the survey's medians and ranges:
oge2e baseogogprs-3-panel-measurePer shard: 1 = 46.9, 2 = 100.7, 3 = 50.2, 4 = 65.7 (attempt 2), 5 = 80.8, 6 = 74.7 min.
The plan gave every shard 57 to 62 min. Shard 2 ran longest because its four Nextflow cases need each other, so they stay in one leg, and that leg got an EPYC 7763 (the chain took 78 min there).
Shard 4 of attempt 1 ran until its 240-minute limit. It was not the split and not a hung case. That runner was slow at everything: the same minimap2 index of 14 sequences took 110 to 116 s there and 12 to 17 s on every other leg. Its image pull took 1428 s, against 274 to 592 s on the others. Every case ran 3.6 times over its table time. On a new runner (rerun of the failed jobs, attempt 2), the same 26 cases took 53 min against a planned 59, and the index took 15 to 18 s. The run already supports this rerun: the shards fetch
e2e-basefrom the earlier attempt, and the report uploads overwrite.Planted failures, throwaway branches off eed7dc5, never merged:
rust/og/src/main.rs: https://github.com/geiresx/OwnGenome/actions/runs/44843559300 is red.ogfails ("error: planted og build failure").e2e basestops at "Wait for og" ("The job og ended failure: the cases have no og."), ande2eis red.tests/e2e/90-report-assets.sh: https://github.com/geiresx/OwnGenome/actions/runs/44843564682 is red.e2e basereports "1 of 23 cases failed" (the planted check, in90-report-assets), ande2eis red.No bead: this package comes from the CI speed survey.
Summary by CodeRabbit