Repository navigation
test(e2e): stop a slow EBI download from failing a batch on the panel measurement - #572
2 commits merged into
Conversation
…oad no longer fails a batch On a dispatched run, prs-3 waits up to 30 minutes for the panel download and then gives pgsc_calc 35 minutes. When EBI served the 7.4 GB panel in 31 minutes instead of 13, the case passed the runner's 60-minute limit. It now raises its own limit to 90 minutes, as the Locityper case does.
|
No actionable comments were generated in the recent review. Walkthrough
Merge Risk:
|
…t costs today A traced run puts every case in one leg with a 355-minute limit, and its time table keeps prs-3 at 0 s, so the 35 to 80 minute measurement does not belong there. PGSC_MEASURE=1 still forces it. The case header and docs/25-prs.md no longer cite the build-fixture job's 150 minutes or say the download outlasts a pull request's run.
…) into the T14 batch tip
…t traced A traced run skips the panel measure (measure_on in its file, GeiserX#572), so its shard should not be planned around 2900 seconds of work that never runs.
Problem
A batch's E2E run failed on
prs-3-panel-measurewithFAIL (timeout 3600s), and no code in the batch had broken it. On a run dispatched on a branch other than main, the case measures pgsc_calc with the 1000 Genomes ancestry panel. Its time has two parts that each have their own cap:timeout 2100).With the site list and the HAPNEST target on top, the case can take about 80 minutes. The runner stops every case at 60 minutes unless the case raises its own limit. On earlier green dispatched runs, the download took 779 s (case 2013 s, run 44842271308) and 1158 s (case 3021 s, run 44891922054). On the failed run, it took 1852 s, and the runner stopped the case after the HAPNEST target was unpacked.
A traced run is a second problem. It runs every case in one leg with a 355-minute limit, and its time table keeps prs-3 at 0 s. But it is also a dispatched branch run, so it carried the measurement too.
Fix
# case-timeout: 5400, the same lineci-e2e-2-locityper.shuses. Each step inside the case still has its own cap, or none, as before. The 90 minutes is the bound on the whole case. A shard job allows 240 minutes, so it still has room.E2E_TRACE=true) skips the measurement. prs-1 does not start the download, and prs-3 says why it skipped.PGSC_MEASURE=1still forces it.docs/testing.mdno longer says that only the Locityper case keeps 90 minutes. The case header anddocs/25-prs.mdno longer cite the 150-minute limit the single e2e job once had, or say that the download outlasts a pull request's run.How it was checked
bash -nand shellcheck are clean. The shellcheck baseline matches on the code text, so the prs-1 lines it lists still match.scripts/ci/e2e-run.sh:931) reads5400from the file.E2E_TRACEunset or set tofalse;E2E_TRACE=true: skips;PGSC_MEASURE=1withE2E_TRACE=true: measures.E2E_TRACEin the step that runs the cases, before and after ci: cut the batch E2E by about an hour without dropping a check #562.Summary by CodeRabbit
PGSC_MEASURE=1continues to enable them regardless of trace status.