Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
139 commits
Select commit Hold shift + click to select a range
bbfddf1
test(live): check that pages say when their data did not load
GeiserX Nov 20, 2026
56038ef
fix(live): say when a person's results could not be loaded
GeiserX Nov 20, 2026
1578cc0
fix(live): say when the list of people could not be loaded
GeiserX Nov 20, 2026
b4e4bf0
fix(live): say when a family answer holds no family
GeiserX Nov 20, 2026
5b80e35
fix(live): an unreadable run state closes the page without naming a role
GeiserX Nov 20, 2026
b795ff3
fix(live): panels wait for the run's data instead of saying there is …
GeiserX Nov 20, 2026
948d2ce
docs(changelog): live pages say when their data did not load
GeiserX Nov 20, 2026
e5f07e6
fix(live): leave the run status alone when only the results failed
GeiserX Nov 20, 2026
60a1a05
fix(live): keep the family report when a redraw of it throws
GeiserX Nov 20, 2026
17b7b23
test(live): wait for each page before reading it, cover a state with …
GeiserX Nov 20, 2026
cff5e40
Merge remote-tracking branch 'origin/main' into pgp/rv-live-load-errors
GeiserX Nov 20, 2026
bbe275f
fix(live): a state with no plugins also reads "Results not loaded"
GeiserX Nov 20, 2026
6f86b5f
feat(dev): score Y placements against published haplogroups
GeiserX Nov 20, 2026
c26977e
ci(evidence): rerun step 37 on published 1000 Genomes men with a gate
GeiserX Nov 20, 2026
4bc7356
docs(y): publish the 58-man Y placement benchmark
GeiserX Nov 20, 2026
0716c0f
fix(live): a results.json whose chapters are not a list reads as not …
GeiserX Nov 20, 2026
d4a49fa
Merge remote-tracking branch 'origin/main' into pgp/i411-y-benchmark
GeiserX Nov 20, 2026
7c5bb1d
Merge remote-tracking branch 'origin/main' into pgp/i411-y-benchmark
GeiserX Nov 20, 2026
4db237d
docs(y): say 58 men, how ties are broken, and that y_samples needs di…
GeiserX Nov 20, 2026
2629ea1
docs(dev): name og dev evidence y-bench in og dev --help and docs/rus…
GeiserX Nov 20, 2026
47b7322
test(og-cn): give each SMN test table its own temp dir
GeiserX Nov 20, 2026
3e1c85b
ci(evidence): refuse a y_samples list the red control cannot fail on
GeiserX Nov 20, 2026
2617cdc
fix(viewer): dark page with a heading, a phone width and a retry on e…
GeiserX Nov 20, 2026
c1d4e37
test(viewer): check the viewer page in Chrome at desktop and phone size
GeiserX Nov 20, 2026
04f754f
docs(viewer): describe the page, its error and the new browser check
GeiserX Nov 20, 2026
052ce7e
feat(dev): og dev licences check fails on a data file with no recorde…
GeiserX Nov 20, 2026
69babd8
ci(licences): record the source and licence of every bundled data file
GeiserX Nov 20, 2026
c974f02
fix(pgx): treat a pypgx Indeterminate as no call in steps 27 and 36
GeiserX Nov 20, 2026
cd92a3f
fix(pgx): check every drug of a combination product on the medication…
GeiserX Nov 20, 2026
5ce7535
test(pgx): pin Indeterminate as no call and every drug of a combination
GeiserX Nov 20, 2026
ba53e73
docs(changelog): Indeterminate pypgx calls and combination products
GeiserX Nov 20, 2026
22a2b41
fix(prs): say when the ancestry sits between two reference groups
GeiserX Nov 20, 2026
074492c
test(prs): prove the reported percentile is calibrated
GeiserX Nov 20, 2026
9a8a127
test(parity): pin the summary's ancestry for a sample between two groups
GeiserX Nov 20, 2026
1c4216c
docs(prs): say when a percentile ranks a mixed ancestry among one group
GeiserX Nov 20, 2026
a2e7cd1
test(traits): pin readings that say no more than their studies found
GeiserX Nov 20, 2026
16230a3
fix(traits): word thirteen readings to match their evidence
GeiserX Nov 20, 2026
81c66e4
docs(traits): point scripts at summary.json for the APOL1 result with G2
GeiserX Nov 20, 2026
588b6a8
test(traits-sources): check the lactase base in a sparse FASTA, not a…
GeiserX Nov 20, 2026
0c4b353
docs(changelog): the traits readings and the Ethiopian lactase row
GeiserX Nov 20, 2026
5ebc56d
feat(live): serve the live site from og live serve
GeiserX Nov 20, 2026
01ca58d
chore(recheck): drop the owngenome_recheck.py shim, call og recheck
GeiserX Nov 20, 2026
2bdb3b9
fix(report): write the rest of the report when one section cannot be …
GeiserX Nov 20, 2026
2f69669
test(parity): compare both reports whole against the Python
GeiserX Nov 20, 2026
b5f6c49
feat(report): run og report render for step 24, HTML_REPORT and the t…
GeiserX Nov 20, 2026
0205a33
test(report): move the checks of test_render_report.py to cargo tests
GeiserX Nov 20, 2026
fa8ee87
docs(report): say og report render writes both reports
GeiserX Nov 20, 2026
71b5699
test(report): clear the two new ShellCheck findings
GeiserX Nov 20, 2026
e36aa45
test(parity): pin the mixed ancestry in the whole-file summary cases too
GeiserX Nov 20, 2026
c41baba
fix(demo): give the Python demo sample a genotype for the -14009 row
GeiserX Nov 20, 2026
d55ef4f
fix(report): keep the charts file when a chart cannot be made
GeiserX Nov 20, 2026
7c25bc4
ci(shards): let a run give a case its own seconds (E2E_SECONDS)
GeiserX Nov 21, 2026
58bc720
build(og): build and check a subset of arches (OG_ARCHES, OG_DARWIN_A…
GeiserX Nov 21, 2026
5f5d90c
ci(e2e): split by prs-3's real time, overlap base setup with og, buil…
GeiserX Nov 21, 2026
c634305
ci(container-test): run at most 8 smoke rows at once
GeiserX Nov 21, 2026
4dd0b8d
ci: drop the rust cache where it saves nothing, save Guard rust's on …
GeiserX Nov 21, 2026
02b5165
ci(rust): build only the runner's og for the streams job
GeiserX Nov 21, 2026
eed7dc5
docs(testing): the faster E2E split, the og overlap and the cache budget
GeiserX Nov 21, 2026
2a02763
fix(viewer): keep igv controls readable and tracks in view
GeiserX Nov 21, 2026
32b82a8
docs(licences): ship OwnHealth and HipSTR data notices
GeiserX Nov 21, 2026
39bb54e
test(prs): make the scored fixture meet minimum overlap
GeiserX Nov 21, 2026
ece92a9
ci(evidence): retry each man's chrY read before the y-benchmark job f…
GeiserX Nov 21, 2026
9c001c7
fix(licences): check complete phrases and notice paths
GeiserX Nov 21, 2026
5460589
test: check that the install docs list the requirements first, per pl…
GeiserX Nov 21, 2026
3267622
docs(readme): list what to install before the quick start, and clone …
GeiserX Nov 21, 2026
a7b486a
docs(getting-started): one ordered install path per platform, each en…
GeiserX Nov 21, 2026
8fd8bb1
docs(changelog): the install docs give requirements first, per platform
GeiserX Nov 21, 2026
6e071c5
fix(pgx): match a drug written in brackets or before a stop
GeiserX Nov 21, 2026
05cd1dc
fix(pgx): say "inhibit" when a not-computed note names several drugs
GeiserX Nov 21, 2026
6aebd60
fix(summary): list a selected step that wrote nothing as "selected, n…
GeiserX Nov 21, 2026
261a5f0
fix(report): hand the run's --tools list to og summary collect
GeiserX Nov 21, 2026
d0774be
docs(report): say how a selected step with no result is listed
GeiserX Nov 21, 2026
965b202
fix(report): keep the search index and abort on a panic outside a sec…
GeiserX Nov 21, 2026
2d04d7b
docs(report): name og report render for the charts and the closing list
GeiserX Nov 21, 2026
c0c92e7
fix(nextflow): run docker tasks as the caller, not as root
GeiserX Nov 21, 2026
7b46a15
test(nextflow): check that docker tasks run as the caller
GeiserX Nov 21, 2026
720889d
docs(nextflow): say that docker tasks run as you, CPSR and CNVPYTOR a…
GeiserX Nov 21, 2026
5ae5850
test(vcf): a negative genotype index in the export and the lookup, re…
GeiserX Nov 21, 2026
a4af93b
fix(sv): join consensus support by voting caller and source ID
GeiserX Nov 21, 2026
cb72ab8
test(sv): pin support and depth output changes in parity goldens
GeiserX Nov 21, 2026
21ee5d5
fix(traits): name the limits the review found in eleven readings
GeiserX Nov 21, 2026
538a887
test(traits): reject the old readings and check the documented G2 path
GeiserX Nov 21, 2026
79d1e4b
test(traits-sources): check the FASTA base on a 150-base contig
GeiserX Nov 21, 2026
57bdb0b
fix(vcf): a negative genotype index reads as unreadable, never as the…
GeiserX Nov 21, 2026
e3a5cd8
test(parity): the edge export counts its -1/0 record as unreadable, n…
GeiserX Nov 21, 2026
5976eb2
test(sv): migrate AnnotSV support checks to the Rust collector
GeiserX Nov 21, 2026
4834af1
ci(e2e): keep the job og inside the required check e2e
GeiserX Nov 21, 2026
012eec4
test(install-docs): check the release-tag filter and the step order i…
GeiserX Nov 21, 2026
9132d92
docs(getting-started): name Debian 13's Java package and narrow the S…
GeiserX Nov 21, 2026
d52d3a8
docs(traits): give all three season means beside the Wang 2010 quote
GeiserX Nov 21, 2026
e1362cf
fix(report): handle intentionally skipped Y steps and check missing r…
GeiserX Nov 21, 2026
0763a5c
test(vcf): expose invalid allele indexes in four readers
GeiserX Nov 21, 2026
2d51652
fix(install-docs): make the Apple Silicon notes link work on GitHub a…
GeiserX Nov 21, 2026
efbcfba
fix(vcf): treat out-of-range genotype indexes as missing
GeiserX Nov 21, 2026
75b6a0a
fix(sv): keep Delly's RDCN on added calls and say when consensus supp…
GeiserX Nov 21, 2026
f938b69
fix(prs): drop a user flag with its whole value before pgsc_calc gets it
GeiserX Nov 21, 2026
868e359
fix(nextflow): keep HOME=/root for CPSR and CNVPYTOR, and say who fix…
GeiserX Nov 21, 2026
86c2567
test(summary): pin a half-indeterminate pypgx diplotype as not called
GeiserX Nov 21, 2026
f437b7d
fix(summary): count a pypgx gene as called only when its diplotype is…
GeiserX Nov 21, 2026
aa0a6d2
docs(pgx): say every word of a medication line is matched against Names
GeiserX Nov 21, 2026
7eee1f6
test(pgx): pin a hyphenated combination as checked for each drug
GeiserX Nov 21, 2026
21af407
fix(pgx): check each drug of a hyphenated combination product
GeiserX Nov 21, 2026
9927826
docs(pgx): hyphenated combinations and the pypgx called-gene count
GeiserX Nov 21, 2026
a2cd749
Merge remote-tracking branch 'origin/main' into pgp/rv-install-docs-o…
GeiserX Nov 21, 2026
0200bbe
test(sv): drop the Python single-caller check the Rust collector test…
GeiserX Nov 21, 2026
fde55b4
Merge remote-tracking branch 'origin/main' into pgp/i411-nf-caller-uid
GeiserX Nov 21, 2026
fbd915e
style(pgx): rustfmt the hyphen-part change and its tests
GeiserX Nov 21, 2026
f4f08ab
Merge main (T10) into the T11 batch tip
GeiserX Nov 21, 2026
845341a
Merge #560 port-live-server into the T11 batch tip
GeiserX Nov 21, 2026
bbeccb3
Merge #547 rv-live-load-errors into the T11 batch tip
GeiserX Nov 21, 2026
9b294c2
Merge #554 rv-viewer-shell into the T11 batch tip
GeiserX Nov 21, 2026
cd424de
Merge #551 i411-y-benchmark into the T11 batch tip
GeiserX Nov 21, 2026
72b9efe
docs(rust): say og live serve is ported without login or results
GeiserX Nov 21, 2026
f995d0e
docs(rust): say og report render writes both reports in the crate table
GeiserX Nov 21, 2026
6fba0a4
docs(browser): list the load-error and viewer-shell checks
GeiserX Nov 21, 2026
429978f
fix(ci): expect og recheck in the live page's no-re-check line
GeiserX Nov 21, 2026
bda2e09
docs(vcf): keep #567's changelog entry and add the other readers to it
GeiserX Nov 21, 2026
44e6acc
docs(pgx): name step 32's Status column as the rule the summary shares
GeiserX Nov 21, 2026
9814240
test(prs): pin that a user value is dropped whole or left to the block
GeiserX Nov 21, 2026
ed9b509
Merge the T11 batch tip into the T12 batch tip
GeiserX Nov 21, 2026
e0c8534
Merge #552 ci-smn-tempdir into the T12 batch tip
GeiserX Nov 21, 2026
60d064c
Merge #553 i411-data-register into the T12 batch tip
GeiserX Nov 21, 2026
275df6b
Merge #555 i411-pypgx-nocall into the T12 batch tip
GeiserX Nov 21, 2026
96ab063
Merge #557 i411-prs-guards into the T12 batch tip
GeiserX Nov 21, 2026
d84be85
docs(og-render): say every report part and og report charts are ported
GeiserX Nov 21, 2026
397018a
docs(report): point the card-title comment at rust/og/tests/report_ch…
GeiserX Nov 21, 2026
9b7de3e
Merge the T12 batch tip into the T13 batch tip
GeiserX Nov 21, 2026
befb748
Merge #565 i411-nf-caller-uid into the T13 batch tip
GeiserX Nov 21, 2026
913b6e7
Merge #567 rv-gt-negative-index into the T13 batch tip
GeiserX Nov 21, 2026
f99b3e0
Merge #558 i411-traits-readings-text into the T13 batch tip
GeiserX Nov 21, 2026
c5a1e4d
Merge #563 rv-install-docs-order into the T13 batch tip
GeiserX Nov 21, 2026
171d5c4
docs(hardware): say the report-step numbers were measured with the Py…
GeiserX Nov 21, 2026
b5e38f7
Merge the T13 batch tip into the T14 batch tip
GeiserX Nov 21, 2026
e93d226
Merge #568 rv-gt-negative-index-2 into the T14 batch tip
GeiserX Nov 21, 2026
5448580
Merge #569 i411-pypgx-nocall-2 into the T14 batch tip
GeiserX Nov 21, 2026
3056fab
test(e2e): give the panel measurement 90 minutes, so a slow EBI downl…
GeiserX Nov 21, 2026
46fc7c7
Merge #572 prs3-case-timeout into the T14 batch tip
GeiserX Nov 21, 2026
057ddd9
test(e2e): skip the panel measurement on a traced run, and say what i…
GeiserX Nov 21, 2026
a828fde
Merge #572 prs3-case-timeout (traced runs skip the measurement) into …
GeiserX Nov 21, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/workflows/container-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,12 @@ jobs:
timeout-minutes: 180
strategy:
fail-fast: false
# The account runs 20 jobs at once. A change that picks every image
# starts about 66 rows: they held the runners for up to two hours while
# the E2E run of the same commit waited 54 minutes for runners (runs
# 44721158887 and 44720913761 on the geiresx fork). At 8 at a time
# every row still runs, only later.
max-parallel: 8
matrix:
var: ${{ fromJSON(needs.plan.outputs.images) }}
steps:
Expand Down
4 changes: 0 additions & 4 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -97,10 +97,6 @@ jobs:
rustup toolchain install "$channel" --profile minimal
rustc --version

- uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3
with:
workspaces: rust

- name: Build og
working-directory: rust
run: |
Expand Down
112 changes: 85 additions & 27 deletions .github/workflows/e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,14 @@ env:
# of picked cases runs fewer when they fit (scripts/ci/e2e-shards.sh count). The cases took 230
# minutes in one job on 2026-11-02 (run 41390434697).
E2E_SHARDS: 6
# Times that replace scripts/ci/e2e-case-times.tsv's for this run
# (scripts/ci/e2e-shards.sh). prs-3-panel-measure measures the 1000 Genomes
# panel only on a run dispatched on a branch other than main (measure_on in
# its file), where it took 2684 to 3452 s (median 2900, runs 44488552622,
# 44616213787 and 44790003535 on the geiresx fork); elsewhere it takes 0 s,
# as the table says. Without this its shard got a full share of other cases
# on top and finished last. Set here, so every job of a run splits alike.
E2E_SECONDS: ${{ github.event_name == 'workflow_dispatch' && github.ref != 'refs/heads/main' && 'prs-3-panel-measure=2900' || '' }}

concurrency:
group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.ref }}
Expand Down Expand Up @@ -397,19 +405,24 @@ jobs:
{
echo "### E2E split: base and ${n} shards, $(wc -l < "${RUNNER_TEMP}/plan.tsv") shard cases"
echo
if [ -n "${E2E_SECONDS:-}" ]; then echo "Seconds this run gives in place of the table's: ${E2E_SECONDS}"; echo; fi
echo "| Job | Cases | Measured minutes |"
echo "|---|---|---|"
awk -F'\t' 'FILENAME == ARGV[1] { if ($0 !~ /^#/) t[$1] = $3; next }
FILENAME == ARGV[2] { n["base"]++; s["base"] += ($1 in t) ? t[$1] : 300; next }
{ n[$1]++; s[$1] += ($2 in t) ? t[$2] : 300 }
awk -F'\t' 'FILENAME == ARGV[1] { t[$1] = $2; next }
FILENAME == ARGV[2] { n["base"]++; s["base"] += t[$1]; next }
{ n[$1]++; s[$1] += t[$2] }
END { printf "| base | %d | %.0f |\n", n["base"], s["base"] / 60
for (k = 1; k in n; k++) printf "| shard %d | %d | %.0f |\n", k, n[k], s[k] / 60 }' \
scripts/ci/e2e-case-times.tsv <(scripts/ci/e2e-shards.sh base) "${RUNNER_TEMP}/plan.tsv"
<(scripts/ci/e2e-shards.sh seconds) <(scripts/ci/e2e-shards.sh base) "${RUNNER_TEMP}/plan.tsv"
} | tee -a "$GITHUB_STEP_SUMMARY"

# og for the legs (rust/, docs/rust.md): built once, as scripts/setup.sh
# --og builds it, and handed to each leg as the artifact og-e2e, so no leg
# pulls the build image or compiles.
# pulls the build image or compiles. Only og-linux-x86_64, the one the
# runners run (OG_ARCHES, rust/og.sh): Guard's rust job and the Rust
# workflow's images job still build all four on every pull request that
# changes rust/. The base leg sets up while this builds and waits for the
# artifact only before its first case.
og:
needs: e2e-scope
if: >-
Expand All @@ -420,6 +433,9 @@ jobs:
(github.event_name == 'workflow_dispatch' && inputs.job == 'e2e')) }}
runs-on: ubuntu-latest
timeout-minutes: 20
env:
OG_ARCHES: x86_64
OG_DARWIN_ARCHES: ''
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
Expand All @@ -443,14 +459,17 @@ jobs:
# the artifact e2e-base; the shard legs start once it passed, unpack that
# artifact at the same paths and run their own cases. A shard does not take
# a runner before the base leg is done: in a busy queue, shards that start
# first and wait would hold runners the base legs of every run need. Both
# first and wait would hold runners the base legs of every run need. The
# base leg does not wait for the job og either: it frees its disk, sets up
# Docker, Java and Nextflow and pulls the images while og builds, and waits
# for the artifact og-e2e only then (step "Wait for og"). Both
# jobs run the same steps (the YAML anchor leg-steps); matrix.job tells
# them apart. Only the base leg saves the image cache. The checks are named
# "e2e base" and "e2e shard N"; the required check is the job e2e below,
# which sums them up.
e2e-base:
needs: [e2e-scope, e2e-plan, og]
if: ${{ !cancelled() && needs.e2e-plan.result == 'success' && needs.og.result == 'success' }}
needs: [e2e-scope, e2e-plan]
if: ${{ !cancelled() && needs.e2e-plan.result == 'success' }}
name: ${{ matrix.job == 'base' && 'e2e base' || format('e2e shard {0}', matrix.job) }}
runs-on: ubuntu-latest
# The base leg waits up to 75 minutes for a fixture a pull request bumps
Expand All @@ -464,24 +483,15 @@ jobs:
job: [base]
env:
JOB: ${{ matrix.job }}
# The one og binary of the job og: rust/og.sh check --required and the
# cases' scripts/setup.sh ask for that one only.
OG_ARCHES: x86_64
OG_DARWIN_ARCHES: ''
steps: &leg-steps
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

# og from the job og, checked against this checkout's rust/, and the
# oracles bin/og's shadow mode runs (OG_SHADOW above).
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: og-e2e
path: bin/.og

- name: og and the shadow oracles
run: |
chmod 755 bin/.og/og-linux-*
rust/og.sh check --required
scripts/ci/oracle-diff.sh materialise

# The same cases take about a third longer on some runners than on
# others; the CPU line lets a timing claim name the hardware.
- name: Runner CPU
Expand Down Expand Up @@ -696,11 +706,56 @@ jobs:
grep -qxE 'callability-1-catalog|cn-intervals-1-catalog|followups-ci-3-sv-rule-regions' "$list" && echo "data_catalogs=true" >> "$GITHUB_OUTPUT"
echo "list=${list}" >> "$GITHUB_OUTPUT"

# Before the base outputs: the base leg pulled the same images.
# Before og and the base outputs, so the pulls overlap the wait for
# them. The base leg's cases would pull the same images first thing.
- name: Pull the images the cases use
if: matrix.job != 'base'
run: scripts/ci/e2e-run.sh --pull-images

# The artifact og-e2e of the job og, which the base leg did not wait
# for: a shard leg, which starts after the base leg passed, finds it at
# once. Stops the leg when the job og ended without it.
- name: Wait for og
env:
GH_TOKEN: ${{ github.token }}
RUN_ID: ${{ github.run_id }}
run: |
t0=$(date +%s) done_at=""
while :; do
n=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${RUN_ID}/artifacts?name=og-e2e" --jq '.total_count' 2> /dev/null) || n=""
if [ "${n:-0}" -gt 0 ]; then
echo "og-e2e is there after $(( $(date +%s) - t0 )) s of waiting" | tee -a "$GITHUB_STEP_SUMMARY"
exit 0
fi
state=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${RUN_ID}/jobs?filter=latest&per_page=100" \
--jq '.jobs[] | select(.name == "og") | "\(.status) \(.conclusion)"' 2> /dev/null) || state=""
case $state in
"completed success")
# The upload ends before the job does; give the listing a minute.
done_at=${done_at:-$(date +%s)}
if [ $(( $(date +%s) - done_at )) -gt 60 ]; then
echo "::error::The job og passed, but this run has no og-e2e artifact."; exit 1
fi ;;
completed*) echo "::error::The job og ended ${state#completed }: the cases have no og."; exit 1 ;;
esac
if [ $(( $(date +%s) - t0 )) -gt 3600 ]; then
echo "::error::No og-e2e artifact after 60 minutes (job og: ${state:-not found})."; exit 1
fi
sleep 20
done

# og from the job og, checked against this checkout's rust/, and the
# oracles bin/og's shadow mode runs (OG_SHADOW above).
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: og-e2e
path: bin/.og

- name: og and the shadow oracles
run: |
chmod 755 bin/.og/og-linux-*
rust/og.sh check --required
scripts/ci/oracle-diff.sh materialise

# The base leg uploads e2e-base only once its cases all passed, and this
# leg starts only after it passed. A rerun of failed legs finds the
# artifact of the earlier attempt. A broken transfer is tried again.
Expand Down Expand Up @@ -1130,10 +1185,12 @@ jobs:
job: ${{ fromJSON(needs.e2e-plan.outputs.jobs) }}
env:
JOB: ${{ matrix.job }}
OG_ARCHES: x86_64
OG_DARWIN_ARCHES: ''
steps: *leg-steps

# The required check. Green when e2e-scope finds nothing to run; otherwise
# green only when the plan and every leg succeeded, every picked case
# green only when the plan, og and every leg succeeded, every picked case
# reported exactly one pass (one table of every case goes to the summary)
# and each parity check whose cases ran passed once. Red when e2e-scope
# did not succeed. A leg that was skipped, failed or cut by its time limit
Expand All @@ -1142,7 +1199,7 @@ jobs:
# cancelled with it and reads cancelled, not failed: nothing ran to fail.
# A cancelled check blocks a merge as a red one does.
e2e:
needs: [e2e-scope, e2e-plan, e2e-base, e2e-shards]
needs: [e2e-scope, e2e-plan, og, e2e-base, e2e-shards]
if: >-
${{ !cancelled() && (
github.event_name == 'pull_request' ||
Expand Down Expand Up @@ -1181,6 +1238,7 @@ jobs:
GH_TOKEN: ${{ github.token }}
RUN_ID: ${{ github.run_id }}
PLAN: ${{ needs.e2e-plan.result }}
OG: ${{ needs.og.result }}
BASE: ${{ needs.e2e-base.result }}
LEGS: ${{ needs.e2e-shards.result }}
SHARDS: ${{ needs.e2e-plan.outputs.shards }}
Expand All @@ -1193,10 +1251,10 @@ jobs:
tr ' ' '\n' <<< "$SELECT" > "${RUNNER_TEMP}/select.txt"
export E2E_SELECT="${RUNNER_TEMP}/select.txt"
fi
if [ "$PLAN" != success ] || [ "$BASE" != success ] || [ "$LEGS" != success ]; then
if [ "$PLAN" != success ] || [ "$OG" != success ] || [ "$BASE" != success ] || [ "$LEGS" != success ]; then
# The run was not cancelled (then this job would not run), so a
# job that ended cancelled was cut by its time limit.
echo "::error::The plan job ended ${PLAN:-unknown}, the base leg ${BASE:-unknown} and the shard legs ${LEGS:-unknown}; e2e needs all three to succeed. A job that ended cancelled in a run that was not cancelled hit its time limit."
echo "::error::The plan job ended ${PLAN:-unknown}, the job og ${OG:-unknown}, the base leg ${BASE:-unknown} and the shard legs ${LEGS:-unknown}; e2e needs all four to succeed. A job that ended cancelled in a run that was not cancelled hit its time limit."
rc=1
fi
dir="${RUNNER_TEMP}/reports"
Expand Down
15 changes: 0 additions & 15 deletions .github/workflows/freshness.yml
Original file line number Diff line number Diff line change
Expand Up @@ -144,11 +144,6 @@ jobs:
rustup toolchain install "$channel" --profile minimal
rustc --version

- uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3
with:
workspaces: rust
key: freshness

- name: Build og
working-directory: rust
run: cargo build --release --locked -p og --no-default-features
Expand Down Expand Up @@ -209,11 +204,6 @@ jobs:
rustup toolchain install "$channel" --profile minimal
rustc --version

- uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3
with:
workspaces: rust
key: freshness

- name: Build og
working-directory: rust
run: cargo build --release --locked -p og --no-default-features
Expand Down Expand Up @@ -267,11 +257,6 @@ jobs:
rustup toolchain install "$channel" --profile minimal
rustc --version

- uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3
with:
workspaces: rust
key: freshness

- name: Build og
working-directory: rust
run: cargo build --release --locked -p og --no-default-features
Expand Down
19 changes: 6 additions & 13 deletions .github/workflows/guard.yml
Original file line number Diff line number Diff line change
Expand Up @@ -151,11 +151,6 @@ jobs:
rustup toolchain install "$channel" --profile minimal
rustc --version

- uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3
with:
workspaces: rust
key: fake-docker

- name: Build og into bin/.og
working-directory: rust
run: |
Expand Down Expand Up @@ -225,10 +220,6 @@ jobs:
rustup toolchain install "$channel" --profile minimal
rustc --version

- uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3
with:
workspaces: rust

- name: Build og
working-directory: rust
run: OG_TREE_HASH=$(./og.sh tree-hash) cargo build --release --locked -p og
Expand Down Expand Up @@ -358,9 +349,15 @@ jobs:
rustup toolchain install "$channel" --profile minimal -c rustfmt -c clippy
rustc --version

# The one Guard job a cache speeds up (clippy and the tests, 1 to 2
# minutes). It saves on main only: a branch reads main's entry, and the
# per-branch entries of every job filled the 10 GB the E2E image and
# data caches need. The jobs that only build og --release have no
# cache: it did not make that build faster.
- uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3
with:
workspaces: rust
save-if: ${{ github.ref == 'refs/heads/main' }}

- name: rustfmt
run: cargo fmt --all --check
Expand Down Expand Up @@ -1009,10 +1006,6 @@ jobs:
rustup toolchain install "$channel" --profile minimal
rustc --version

- uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3
with:
workspaces: rust

- name: Build og
working-directory: rust
run: |
Expand Down
62 changes: 62 additions & 0 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -300,3 +300,65 @@ jobs:
exit 1
fi
echo "OK: All internal links valid"

# Every data file the repository ships has a recorded source and licence:
# assets/SOURCES.tsv has a row for each file under assets/, each file
# carries the header phrase its row names, and NOTICE names every
# third-party row (og dev licences check, rust/crates/og-dev/src/licences.rs).
data-sources:
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- name: Install the toolchain rust/rust-toolchain.toml pins
working-directory: rust
run: |
channel=$(sed -n 's/^channel = "\(.*\)"$/\1/p' rust-toolchain.toml)
rustup toolchain install "$channel" --profile minimal
rustc --version

- uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3
with:
workspaces: rust

# A debug build without BAM reading: the check needs no htslib and no
# optimised code.
- name: Build og
working-directory: rust
run: cargo build --locked -p og --no-default-features

- name: Check that every bundled data file has a recorded source and licence
run: rust/target/debug/og dev licences check

# A check that cannot fail is no check: a new asset with no row, and an
# asset with its source line stripped, must both fail it.
- name: Controls, an unregistered asset and a stripped source line must fail
run: |
og=rust/target/debug/og
log=$(mktemp)

planted=assets/lint-control.tsv
printf 'a\tb\n' > "$planted"
rc=0; "$og" dev licences check > "$log" 2>&1 || rc=$?
rm -f "$planted"
cat "$log"
if [ "$rc" -ne 1 ] || ! grep -qF "${planted} has no row in assets/SOURCES.tsv" "$log"; then
echo "ERROR: control 1, an asset with no row passed (exit ${rc})"; exit 1
fi
echo "Control 1: the unregistered ${planted} was caught."

target=assets/sv_rule_regions.tsv
sed -i '/^# Sources and terms:/d' "$target"
if git diff --quiet -- "$target"; then
echo "ERROR: control 2 stripped nothing from ${target}"; exit 1
fi
rc=0; "$og" dev licences check > "$log" 2>&1 || rc=$?
git checkout -- "$target"
cat "$log"
if [ "$rc" -ne 1 ] || ! grep -qF "${target}: its opening # lines lack \"Sources and terms:" "$log"; then
echo "ERROR: control 2, ${target} without its source line passed (exit ${rc})"; exit 1
fi
echo "Control 2: ${target} without its source line was caught."
5 changes: 0 additions & 5 deletions .github/workflows/nextflow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -211,11 +211,6 @@ jobs:
rustup toolchain install "$channel" --profile minimal
rustc --version

- uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3
if: steps.scope.outputs.run == 'true'
with:
workspaces: rust

- name: Build og
if: steps.scope.outputs.run == 'true'
working-directory: rust
Expand Down
Loading
Loading