Skip to content

test(e2e): stop a slow EBI download from failing a batch on the panel measurement - #572

Merged
2 commits merged into
GeiserX:mainfrom
geiresx:pgp/prs3-case-timeout
Nov 21, 2026
Merged

2 commits merged into
GeiserX:mainfrom
geiresx:pgp/prs3-case-timeout

Conversation

@geiresx

@geiresx geiresx commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

A batch's E2E run failed on prs-3-panel-measure with FAIL (timeout 3600s), and no code in the batch had broken it. On a run dispatched on a branch other than main, the case measures pgsc_calc with the 1000 Genomes ancestry panel. Its time has two parts that each have their own cap:

  • It waits up to 30 minutes for the 7.4 GB panel download that case prs-1 starts in the background.
  • It then gives pgsc_calc 35 minutes (timeout 2100).

With the site list and the HAPNEST target on top, the case can take about 80 minutes. The runner stops every case at 60 minutes unless the case raises its own limit. On earlier green dispatched runs, the download took 779 s (case 2013 s, run 44842271308) and 1158 s (case 3021 s, run 44891922054). On the failed run, it took 1852 s, and the runner stopped the case after the HAPNEST target was unpacked.

A traced run is a second problem. It runs every case in one leg with a 355-minute limit, and its time table keeps prs-3 at 0 s. But it is also a dispatched branch run, so it carried the measurement too.

Fix

  • The case raises its own limit to 90 minutes with # case-timeout: 5400, the same line ci-e2e-2-locityper.sh uses. Each step inside the case still has its own cap, or none, as before. The 90 minutes is the bound on the whole case. A shard job allows 240 minutes, so it still has room.
  • A traced run (E2E_TRACE=true) skips the measurement. prs-1 does not start the download, and prs-3 says why it skipped. PGSC_MEASURE=1 still forces it.
  • docs/testing.md no longer says that only the Locityper case keeps 90 minutes. The case header and docs/25-prs.md no longer cite the 150-minute limit the single e2e job once had, or say that the download outlasts a pull request's run.

How it was checked

  • bash -n and shellcheck are clean. The shellcheck baseline matches on the code text, so the prs-1 lines it lists still match.
  • The runner's own pattern (scripts/ci/e2e-run.sh:931) reads 5400 from the file.
  • The measure condition, run on six environments:
    • a dispatched branch run: measures, with E2E_TRACE unset or set to false;
    • the same run with E2E_TRACE=true: skips;
    • main: skips;
    • a pull request: skips;
    • PGSC_MEASURE=1 with E2E_TRACE=true: measures.
  • The E2E workflow sets E2E_TRACE in the step that runs the cases, before and after ci: cut the batch E2E by about an hour without dropping a check #562.

Summary by CodeRabbit

  • Documentation
    • Clarified panel download and measurement time estimates, along with their timeout limits.
    • Updated guidance on when measurements run for dispatched end-to-end tests.
  • Bug Fixes
    • Measurements on dispatched runs from non-main branches now run only when tracing is off; PGSC_MEASURE=1 continues to enable them regardless of trace status.

…oad no longer fails a batch

On a dispatched run, prs-3 waits up to 30 minutes for the panel download
and then gives pgsc_calc 35 minutes. When EBI served the 7.4 GB panel in
31 minutes instead of 13, the case passed the runner's 60-minute limit.
It now raises its own limit to 90 minutes, as the Locityper case does.
@coderabbitai

coderabbitai Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The E2E panel download and measurement conditions now exclude traced non-main dispatch runs unless PGSC_MEASURE=1 enables measurement. Documentation and comments describe updated run-time estimates and timeout limits.

Changes: E2E Panel Measurement

Layer / File(s) Summary
Measurement conditions and timeouts
tests/e2e/prs-1-agreement.sh, tests/e2e/prs-3-panel-measure.sh, docs/25-prs.md, docs/testing.md
The scripts skip panel download or measurement on traced non-main dispatch runs; PGSC_MEASURE=1 remains an override. The documentation and comments update duration estimates and describe the 90-minute timeout for Locityper step 53 and prs-3-panel-measure.sh.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other

Merge Risk: Minimal · up to 057dd

This change adjusts when the panel measurement E2E case runs and updates its timeout and documentation. It affects only CI test behavior, not production. No actionable merge risk remains.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: a test(e2e) fix that stops a slow EBI panel download from failing a batch during the panel measurement case. The raw summary and objectives show the case timeout raise…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Full details: Title check

Explanation

The title names the main change: a test(e2e) fix that stops a slow EBI panel download from failing a batch during the panel measurement case. The raw summary and objectives show the case timeout raised from 60 to 90 minutes, and the measurement condition changed for untraced runs. The title covers the core purpose, though it does not mention the trace-based skip condition or the docs updates.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 002bdd5 and 057ddd9.

Files selected for processing: 4

  • docs/25-prs.md
  • docs/testing.md
  • tests/e2e/prs-1-agreement.sh
  • tests/e2e/prs-3-panel-measure.sh

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

geiresx pushed a commit to geiresx/OwnGenome that referenced this pull request Nov 21, 2026
…t costs today

A traced run puts every case in one leg with a 355-minute limit, and its
time table keeps prs-3 at 0 s, so the 35 to 80 minute measurement does
not belong there. PGSC_MEASURE=1 still forces it. The case header and
docs/25-prs.md no longer cite the build-fixture job's 150 minutes or say
the download outlasts a pull request's run.
geiresx pushed a commit to geiresx/OwnGenome that referenced this pull request Nov 21, 2026
@GeiserX GeiserX closed this pull request by merging all changes into GeiserX:main in 9dd5dda Nov 21, 2026
geiresx pushed a commit to geiresx/OwnGenome that referenced this pull request Nov 21, 2026
…t traced

A traced run skips the panel measure (measure_on in its file, GeiserX#572), so its
shard should not be planned around 2900 seconds of work that never runs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants