Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
e2fe74a
fix(chip): give the CHIP filter the contamination estimate
GeiserX Nov 20, 2026
e927d0e
test(chip): check that every CHIP path hands FilterMutectCalls the co…
GeiserX Nov 20, 2026
395bef2
fix(chip): estimate contamination from every common site, not only th…
GeiserX Nov 20, 2026
3ae8ecc
fix(carrier): keep the low-confidence mark and the zygosity on every …
GeiserX Nov 20, 2026
f1d31c8
docs(carrier): say what each carrier row carries and when a row is no…
GeiserX Nov 20, 2026
070d779
fix(chip): warn when the contamination estimate runs without gnomAD, …
GeiserX Nov 20, 2026
0da2276
test(chip): verify the module input order for common sites
GeiserX Nov 21, 2026
799d247
fix(carrier): keep showing rows with no unchanged copy in both report…
GeiserX Nov 21, 2026
e2737d8
test(personal-data): allow the synthetic LPA genotypes of the all rep…
GeiserX Nov 21, 2026
f631b3c
test(parity): fix the time of the report corpus, so its goldens hold …
GeiserX Nov 21, 2026
6785e78
Merge remote-tracking branch 'origin/main' into pgp/i411-carrier-conf…
GeiserX Nov 21, 2026
6b93bf4
Merge the T11 batch tip into #559
GeiserX Nov 21, 2026
6059508
fix(sf): cap a secondary-findings pair that rests on a low-confidence…
GeiserX Nov 21, 2026
fa5f6e6
fix(live): flag the carrier headline of og live serve for hom or hemi…
GeiserX Nov 21, 2026
4df2db0
test(parity): whole-report goldens for the carrier cases
GeiserX Nov 21, 2026
9a27ad7
fix(nextflow): stop the clinical filter crashing on REVEL and AlphaMi…
GeiserX Nov 21, 2026
12341ab
fix(carrier): give the carrier list and the SF card one answer beside…
GeiserX Nov 21, 2026
a857c8c
test(report): check the reason the reports give for an unknown SV sup…
GeiserX Nov 21, 2026
f91075b
fix(report): give the right reason when step 22's consensus cannot te…
GeiserX Nov 21, 2026
44d3398
ci(personal-data): allow the corpus's made-up genotypes in report-ren…
GeiserX Nov 21, 2026
966083d
test(e2e): compress the score-tier tables with the bcftools image, no…
GeiserX Nov 21, 2026
f059dc3
fix(report): count an SV row without a support key as not known
GeiserX Nov 21, 2026
87f194a
fix(carrier): let only confident calls give the phase of a two-copy pair
GeiserX Nov 21, 2026
6ec4394
docs(carrier): say where the carrier list and the SF card agree
GeiserX Nov 21, 2026
327a7c3
fix(live): give the live site's secondary-findings rows the consensus…
GeiserX Nov 21, 2026
ff501c3
fix(carrier): let no weak call make an allele firm or give it a phase
GeiserX Nov 21, 2026
ae476db
docs(carrier): name the weak HIGH-impact call where the two lists differ
GeiserX Nov 21, 2026
4299bb3
docs(carrier): name the real split between the two lists, not allele …
GeiserX Nov 21, 2026
366365f
fix(carrier): let a phased combined allele lend no phase to its weak …
GeiserX Nov 21, 2026
8c2adf8
Merge remote-tracking branch 'origin/main' into pgp/i411-carrier-conf…
GeiserX Nov 21, 2026
a1b27b3
test(parity): regenerate 38 report-render goldens on main's SV suppor…
GeiserX Nov 21, 2026
e5ae524
Merge main into the T15 batch tip
GeiserX Nov 21, 2026
f7cb2a1
Merge #559 i411-carrier-confidence into the T15 batch tip
GeiserX Nov 21, 2026
8a882d8
Merge #570 i411-nf-vcfanno-ops into the T15 batch tip
GeiserX Nov 21, 2026
59df890
Merge #556 i411-chip-contamination into the T15 batch tip
GeiserX Nov 21, 2026
7a6a81d
ci(e2e): weigh prs-3-panel-measure at 2900 s only on a run that is no…
GeiserX Nov 21, 2026
8befa62
docs(prs): say the panel measure adds 35 to 80 minutes, as its case says
GeiserX Nov 21, 2026
f657812
test(parity): regenerate report-render goldens for #559 with #571
GeiserX Nov 21, 2026
cc4b811
docs(carrier): name the deletion-straddle shape where the SF card is …
GeiserX Nov 21, 2026
23eb52a
ci(e2e): say the panel measure time applies only on a run that is not…
GeiserX Nov 21, 2026
9184fd4
test(live): carry the results veil and SMN1 agreement cases
GeiserX Nov 22, 2026
cde3382
feat(live): build the results journal from the Rust report
GeiserX Nov 22, 2026
f1ac07d
test(live): cover search veil tags and child report transitions
GeiserX Nov 22, 2026
0103bfe
refactor(live): switch the remaining results callers to og
GeiserX Nov 22, 2026
9303459
test(live): run the access log check with the workspace og
GeiserX Nov 22, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions .github/workflows/e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -119,12 +119,13 @@ env:
E2E_SHARDS: 6
# Times that replace scripts/ci/e2e-case-times.tsv's for this run
# (scripts/ci/e2e-shards.sh). prs-3-panel-measure measures the 1000 Genomes
# panel only on a run dispatched on a branch other than main (measure_on in
# its file), where it took 2684 to 3452 s (median 2900, runs 44488552622,
# 44616213787 and 44790003535 on the geiresx fork); elsewhere it takes 0 s,
# as the table says. Without this its shard got a full share of other cases
# on top and finished last. Set here, so every job of a run splits alike.
E2E_SECONDS: ${{ github.event_name == 'workflow_dispatch' && github.ref != 'refs/heads/main' && 'prs-3-panel-measure=2900' || '' }}
# panel only on a run dispatched on a branch other than main and not traced
# (measure_on in its file), where it took 2684 to 3452 s (median 2900, runs
# 44488552622, 44616213787 and 44790003535 on the geiresx fork); elsewhere it
# takes 0 s, as the table says. Without this its shard got a full share of
# other cases on top and finished last. Set here, so every job of a run
# splits alike.
E2E_SECONDS: ${{ github.event_name == 'workflow_dispatch' && github.ref != 'refs/heads/main' && !inputs.trace && 'prs-3-panel-measure=2900' || '' }}

concurrency:
group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.ref }}
Expand Down
6 changes: 6 additions & 0 deletions CHANGELOG.md

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions assets/output_contract.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -339,6 +339,7 @@ param viral_ref - - - - experimental - -
param chip_gff - - - - experimental - -
param chip_germline_resource - - - - experimental - -
param chip_pon - - - - experimental - -
param chip_common_sites - - - - experimental - -
param parascopy_population - - - - experimental - -
param parascopy_depth_bed - - - - experimental - -
param paraphase_regions - - - - experimental - -
Expand Down
1,375 changes: 0 additions & 1,375 deletions bin/live_results.py

This file was deleted.

20 changes: 9 additions & 11 deletions bin/owngenome_live.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@
ASSETS = os.path.join(REPO, "assets", "live")
sys.path.insert(0, HERE)
import run_state # noqa: E402
from owngenome_live_results import og_results # noqa: E402

PBKDF2_ITERATIONS = 600000
COOKIE = "owngenome_live"
Expand Down Expand Up @@ -299,7 +300,7 @@ def _h_clinvar(d, summary):

def _h_carrier(d, summary):
lines = _rr().carrier_lines(d)
return lines[0], bool(d.get("carriers"))
return lines[0], bool(d.get("carriers") or d.get("hom_or_hemizygous"))


def _h_couple(d, summary):
Expand Down Expand Up @@ -545,8 +546,8 @@ def __init__(self, app, cfg, password_hash):
self.streams = threading.BoundedSemaphore(cfg.max_streams)
self.headline_cache = {}
self.attention_cache = {}
self.live_overview = {} # run -> (the headline cache key, live_results.overview of that summary)
self.summary_overviews = {} # run -> (the summary summary() gave, live_results.overview of it)
self.live_overview = {} # run -> (the headline cache key, og live results --overview of that summary)
self.summary_overviews = {} # run -> (the summary summary() gave, og live results --overview of it)
self.allowed = set(LOOPBACK_HOSTS) | {h.lower() for h in cfg.allowed_host or []}
if cfg.bind not in ("0.0.0.0", "::"):
self.allowed.add(cfg.bind.strip("[]").lower())
Expand Down Expand Up @@ -810,7 +811,7 @@ def attention(self, d):
return (o["attention"] if o else None), mtime

def overview(self, d):
"""(live_results.overview of <run>/summary.json, mtime), cached by its mtime; (None, None) without one:
"""(og live results --overview of <run>/summary.json, mtime), cached by its mtime; (None, None) without one:
the attention count, the counts by level, the first findings and the lead the home page shows."""
p = os.path.join(d, "summary.json") if d else None
if not p or not self.inside(p, os.path.realpath(d)) or not self.inside(p, self.root):
Expand All @@ -825,9 +826,8 @@ def overview(self, d):
return hit[1], mtime
n = None
try:
import live_results
with open(p) as f:
n = live_results.overview(json.load(f))
n = og_results(json.load(f), "--overview")
except Exception as e: # one bad file must not take the home page down
print(f"owngenome_live: {os.path.basename(os.path.dirname(p))}/summary.json not read: {type(e).__name__}",
file=sys.stderr)
Expand All @@ -840,16 +840,15 @@ def child(self, name, runs=None):
return child_of([r for r in (runs or self.runs()).values() if r.get("kind") == "family"], name)

def summary_overview(self, name, d, st):
"""live_results.overview of the summary results.json reads now (summary()), cached with that summary;
"""og live results --overview of the summary results.json reads now (summary()), cached with that summary;
None while there is none."""
s = self.summary(name, d, st)
if not s:
return None
hit = self.summary_overviews.get(name)
if not hit or hit[0] is not s:
try:
import live_results
hit = (s, live_results.overview(s))
hit = (s, og_results(s, "--overview"))
except Exception as e: # one bad summary must not take a page down
print(f"owngenome_live: overview of {name} not built: {type(e).__name__}", file=sys.stderr)
hit = (s, None)
Expand Down Expand Up @@ -906,8 +905,7 @@ def runs_index(self):
lo = self.live_overview.get(name)
if not lo or lo[0] != hit[0]:
try:
import live_results
lo = (hit[0], live_results.overview(hit[2]))
lo = (hit[0], og_results(hit[2], "--overview"))
except Exception as e: # one bad summary must not take the home page down
print(f"owngenome_live: overview of {name} not built: {type(e).__name__}", file=sys.stderr)
lo = (hit[0], None)
Expand Down
5 changes: 3 additions & 2 deletions bin/owngenome_live_map.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,15 @@

def results_body(live, run):
"""What GET /runs/<run>/results.json sends (bin/owngenome_live_results.py), built the same way."""
import live_results
from owngenome_live_results import og_results
name = run["name"]
d = run["dir"] or os.path.join(live.root, name)
summary = live.summary(name, d, live.base_state(run))
if not summary:
return {"schema_version": 1, "sample": name, "chapters": [],
"note": "No step has finished yet: the results appear here as the steps end."}
return live_results.build(name, summary, live.run_files(run) if run["dir"] else set())
files = live.run_files(run) if run["dir"] else set()
return og_results(summary, "--sample", name, files=files)


def relay(app, req, args, stdin, run, section):
Expand Down
28 changes: 17 additions & 11 deletions bin/owngenome_live_results.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
"""owngenome_live_results.py: a person's results on the live site, a plugin of bin/owngenome_live.py.

GET /runs/<run>/results.json bin/live_results.py's build() of the run's summary: the
GET /runs/<run>/results.json og live results of the run's summary: the
report's seven chapters, one card per section with its
facts, its rows, its caveats and its attention flag, and
the Journal's findings, levels, lead and ten sections
Expand All @@ -17,14 +17,27 @@
what the report prints. Every request writes one access-log row.
"""
import html
import json
import os
import re
import subprocess
import sys
import urllib.parse

HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, HERE)

NAME = r"[A-Za-z0-9][A-Za-z0-9._-]*"
FINDING_ID = re.compile(r"^[a-z0-9][a-z0-9-]{0,79}$")


def og_results(summary, *args, files=(), og=None):
"""Run the results command on a summary already read by Python."""
og = og or os.environ.get("OG") or os.path.join(HERE, "og")
paths = [arg for f in sorted(files) for arg in ("--file", f)]
p = subprocess.run([og, "live", "results", "-", *args, *paths],
input=json.dumps(summary), text=True, capture_output=True, timeout=30, check=True)
return json.loads(p.stdout)


def register(app):
Expand All @@ -38,12 +51,11 @@ def results(req, m):
d = run["dir"] or os.path.join(live.root, name)
out = {"schema_version": 1, "sample": name, "chapters": [], "note": None}
try:
import live_results
st = live.base_state(run)
summary = live.summary(name, d, st)
if summary:
files = live.run_files(run) if run["dir"] else set()
out = live_results.build(name, summary, files)
out = og_results(summary, "--sample", name, files=files)
# For a child whose report the server keeps closed (family.json leaves its count out): the
# flagged-card count of summary.json, as family.json counts it otherwise. The family page shows it
# once every result to act on is opened there (assets/live/core.js released).
Expand Down Expand Up @@ -76,11 +88,11 @@ def finding(req, m):
run = live.run(m.group(1))
fid = m.group(2)
ok = False
if run and run.get("kind") != "family" and live_results_mod().FINDING_ID.match(fid):
if run and run.get("kind") != "family" and FINDING_ID.match(fid):
try:
d = run["dir"] or os.path.join(live.root, run["name"])
summary = live.summary(run["name"], d, live.base_state(run))
found = live_results_mod().build(run["name"], summary)["findings"] if summary else []
found = og_results(summary, "--sample", run["name"])["findings"] if summary else []
ok = any(f["id"] == fid for f in found) or (
any(f["level"] == "act" for f in found) and child_run(live, run["name"]))
except Exception as e: # a reader bug is a 404 here, never a 500
Expand All @@ -103,9 +115,3 @@ def child_run(live, name):
"""Whether the person page treats NAME as a child's run: the one rule of the page and the home page
(bin/owngenome_live.py child_of)."""
return live.child(name)


def live_results_mod():
import live_results
return live_results

25 changes: 16 additions & 9 deletions bin/render_report.py
Original file line number Diff line number Diff line change
Expand Up @@ -836,9 +836,9 @@ def comphet_phase_html(d):
"those calls are listed apart with the reason and are not counted here.")


def annotsv_support_text(x):
def annotsv_support_text(x, consensus=None):
if x["support"] is None:
return "not known (no step 22 consensus)"
return "not known (step 22's consensus cannot tell)" if consensus else "not known (no step 22 consensus)"
if x["support"] == 1:
return f"single caller ({x['caller']})"
return f"{x['support']} callers ({', '.join(x['support_callers'])})"
Expand Down Expand Up @@ -874,7 +874,7 @@ def annotsv_lines(d):
"""Step 05's class 4-5 lines, the same in both reports."""
head = f"Class 4-5 SVs: {d['count']}"
if d["count"]:
head += (f" (single caller: {d['single_caller'] if d['consensus'] else 'not known'}; deletions on exon edges: "
head += (f" (single caller: {str(d['single_caller']) + (f', not known: {m}' if (m := sum(x.get('support') is None for x in d.get('rows') or [])) else '') if d['consensus'] else 'not known'}; deletions on exon edges: "
f"{d['exon_edge'] if d['exon_edge_checked'] else 'not checked'}; common: {d['common']})")
lines = [head]
if d.get("downgraded"):
Expand Down Expand Up @@ -1343,7 +1343,7 @@ def head(key, title=None):
w(f" {line}")
for x in d["rows"]:
w(f" {x['type']:<4} {x['sv']} class {x['acmg_class']} {annotsv_genes_text(x)} "
f"support: {annotsv_support_text(x)}")
f"support: {annotsv_support_text(x, d['consensus'])}")
for f in annotsv_flags(x):
w(f" {f}")
if d.get("downgraded"):
Expand Down Expand Up @@ -1553,7 +1553,7 @@ def head(key, title=None):
f"{len(sf['clinvar_hits'])} ClinVar P/LP, {high_rare(sf)} rare HIGH impact" + high_apart(sf)
+ (f", {len(sf['structural'])} class 4/5 SV(s)" if sf.get("structural_checked") else "")
+ (f", {len(sf['not_reported'])} not reported by the gene's rule" if sf.get("not_reported") else ""))
for g, v, z, ev in sf_rows(sf):
for g, v, z, ev in sf_rows(sf, (S.get("annotsv", MISSING).get("data") or {}).get("consensus")):
w(f" {g:<8} {v} {z} {ev}")
for line in sf_apart_lines(sf):
w(f" {line}")
Expand Down Expand Up @@ -1991,7 +1991,7 @@ def paralogs_unchecked(S):
"change in the same gene (step 39). A hypomorph or pseudodeficiency allele is set aside.")


def sf_rows(sf):
def sf_rows(sf, consensus=None):
"""The secondary findings as (gene, variant, genotype, evidence) rows,
the same in both reports: ClinVar hits, rare HIGH-impact records, then
class 4 and 5 SVs."""
Expand All @@ -2004,7 +2004,7 @@ def extra(x):
for x in sf["high_impact"]]
def support(x):
if x["support"] is None:
return "caller support not known (no step 22 consensus)"
return "caller support not known (step 22's consensus cannot tell)" if consensus else "caller support not known (no step 22 consensus)"
return f"single caller ({x['caller']})" if x["support"] == 1 else f"{x['support']} callers"
rows += [(x["gene"], x["variant"], ".", f"AnnotSV class {x['acmg_class']}, {support(x)}"
+ (f"; through {x['via']}" if x.get("via") else "") + extra(x)) for x in sf.get("structural", [])]
Expand All @@ -2025,13 +2025,20 @@ def carrier_lines(c):
"""The sample's own carrier list (no partner), the same in both reports."""
if c.get("unavailable"):
return [f"Carrier list: not made, {c['unavailable']}"]
both = c.get("hom_or_hemizygous") or [] # no unchanged copy: listed apart, not counted as carriers
lines = [f"Carrier list ({c['genes_on_list']} genes): {len(c['carriers'])} finding(s)"
+ (f", {len(both)} with no unchanged copy" if both else "")
+ (f", {len(c['set_aside'])} set aside" if c["set_aside"] else "")]
def hemi(x):
return f" ({x['hemizygous']})" if x.get("hemizygous") else ""
for x in c["carriers"]:
lines.append(f"{x['gene']} ({x['condition']}; {x['inheritance']}): {x['variant']} {x['genotype']}{hemi(x)}, "
f"ClinVar {x['significance']}")
for x in both:
lines.append(f"{x['gene']} ({x['condition']}; {x['inheritance']}): {x['variant']} {x['genotype']}{hemi(x)}, "
f"ClinVar {x['significance']}: "
+ ("two changed copies" if x["genotype"] == "hom" else "the only copy changed")
+ ", not the pattern of a carrier")
for x in c["set_aside"]:
lines.append(f"Set aside: {x['gene']} {x['variant']} {x['genotype']}{hemi(x)}: {x['name']} is a {x['kind']} "
"allele, not a carrier finding on its own")
Expand Down Expand Up @@ -2843,7 +2850,7 @@ def a(key, lines):
if av.get("rows"):
body.append(f" <table><tr><th>SV</th><th>Class</th><th>Genes</th><th>Callers</th><th>Checks</th>{VIEWER_TH}</tr>"
+ "".join(f"<tr><td>{E(x['type'] + ' ' + x['sv'])}</td><td>{E(x['acmg_class'])}</td>"
f"<td>{E(annotsv_genes_text(x))}</td><td>{E(annotsv_support_text(x))}</td>"
f"<td>{E(annotsv_genes_text(x))}</td><td>{E(annotsv_support_text(x, av['consensus']))}</td>"
f"<td>{E('; '.join(annotsv_flags(x)) or 'none')}</td>{viewer_td_of(x['sv'])}</tr>"
for x in av["rows"])
+ "</table>")
Expand Down Expand Up @@ -3062,7 +3069,7 @@ def a(key, lines):
if sf and (sf["clinvar_hits"] or sf["high_impact"] or sf.get("structural") or sf.get("not_reported")
or sf.get("unavailable")):
rows = "".join(f"<tr><td>{E(g)}</td><td>{E(v)}</td><td>{E(z)}</td><td>{E(ev)}</td>{viewer_td_of(v)}</tr>\n"
for g, v, z, ev in sf_rows(sf))
for g, v, z, ev in sf_rows(sf, (S.get("annotsv", MISSING).get("data") or {}).get("consensus")))
a("clinical", card(S["clinical"], f"Secondary-Findings Genes ({sf['version']})",
[" <table>", f" <tr><th>Gene</th><th>Variant</th><th>Genotype</th><th>Evidence</th>{VIEWER_TH}</tr>",
rows.rstrip("\n"), " </table>"]
Expand Down
12 changes: 4 additions & 8 deletions bin/results_index.py
Original file line number Diff line number Diff line change
Expand Up @@ -216,16 +216,12 @@ def add(term, key, sec, text, locus):
add(n, key, sec, text, locus)
# The veil: an item that names a result to act on (its gene, or a locus inside its position or span)
# carries that result's id, so a child's page leaves it out of the search while the result is closed
# (bin/live_results.py veil_id, the rule its ClinVar rows and chart marks follow).
import live_results # here, not at the top: live_results imports render_report, which imports this module
spans = live_results.closable_of(summary)
for term, items in terms.items():
# og live results applies the same rule to the index as to the ClinVar rows and chart marks.
from owngenome_live_results import og_results
for items in terms.values():
for it in items:
del it["_seen"]
fid = live_results.veil_id(spans, " ".join([term] + [f"{m['text']} {m['locus'] or ''}"
for m in it["mentions"]]))
if fid:
it["veil"] = fid
terms = og_results({"summary": summary, "terms": terms}, "--tag-search")
return {"schema_version": SCHEMA_VERSION, "sample": summary.get("sample"),
"generated_utc": summary.get("generated_utc"),
"terms": {k: terms[k] for k in sorted(terms)}}
Expand Down
2 changes: 1 addition & 1 deletion docs/05-annotsv.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ The reports (`og summary collect`) read each SV's support from step 22's consens
- its ID is a member of two events;
- or the consensus holds a vote of its caller without a readable ID. This covers the CNVpytor calls when step 22 made CNVpytor's VCF from its table, which writes no record IDs. It also covers a consensus without `FORMAT/ID`; SURVIVOR 1.0.7 always writes it, so only a file made another way lacks it.

Without the consensus file the support is "not known" too. `og sv postfilter --consensus` adds the same `Support` and `Support_callers` columns to the table, both empty when the support is not known.
Without the consensus file the support is "not known" too. `og sv postfilter --consensus` adds the same `Support` and `Support_callers` columns to the table, both empty when the support is not known. The reports give the reason: "not known (step 22's consensus cannot tell)" when the consensus ran, "not known (no step 22 consensus)" when it did not. When the consensus ran, the section's count line adds how many SVs it could not tell, as in `single caller: 2, not known: 3`.

Both reports list every class 4 and 5 SV in a section of their own, with its support, the exon-edge flag and the frequency. The bash report (`scripts/24-html-report.sh`, which `run-all.sh` runs) reads the table from the sample folder. The Nextflow `HTML_REPORT` gets the table from `ANNOTSV` and, with `survivor_merge` in `--tools`, the consensus, so a report written by `nextflow run` alone has the same section. A report of a run without AnnotSV lists it under the steps not run.

Expand Down
Loading
Loading