Repository navigation
Conversation
FilterMutectCalls in step 49 ran without a contamination table, so another person's DNA in the sample, whose alleles sit at a low VAF, could pass as a small clone. With step 29's common-sites VCF installed, the step and the Nextflow CHIP_MUTECT2 now run GetPileupSummaries inside the CHIP exons and CalculateContamination, and pass --contamination-table and --tumor-segmentation to FilterMutectCalls. New optional parameter --chip_common_sites; run-all.sh passes it when the VCF and its .tbi are installed. Without the VCF the log says there is no estimate. Bead: trait-62
…ntamination table The fake-docker case runs step 49 without the common-sites VCF, with it, and with it but no .tbi; renders CHIP_MUTECT2's script block with a fake gatk with and without --chip_common_sites; and checks that run-all.sh passes the VCF to the pipeline only with its .tbi. Bead: trait-62
…e CHIP exons The CHIP coding exons hold 8 of the 59,295 sites of the common-sites VCF, too few for CalculateContamination to estimate anything. Contamination is a property of the sample, so GetPileupSummaries now reads every common site of the VCF in both the step and CHIP_MUTECT2. The fake-docker case checks the table first, so main fails it on the --contamination-table check. Bead: trait-62
…carrier row The carrier list parsed each hit with the default call floor (GQ 30, DP 10) and dropped the floor the hits file states, and it wrote no call quality at all, so a weak call read like a sure one. Each row now has zygosity, gq, dp and low_confidence, judged against the file's own ##PGP_CALL_FLOOR, as the ClinVar section judges the same hit. A homozygous or hemizygous P/LP row has no unchanged copy, so it moves from carriers to a new list, hom_or_hemizygous. Two different heterozygous hits in one recessive gene get pair and phase from the haplotype rule of the secondary-findings card (two hits, possible two hits, or one copy in cis). A pair resting on a low-confidence call, and a deletion plus a low-confidence call (two_hits), is at most possible. The schema describes the carrier section; the summary-collect goldens change only in the carrier section. og-render's hemizygous tests read the row from its new list. Bead: clin-15 Bead: clin-19
… carrier Bead: clin-15 Bead: clin-19
…or not at all FilterMutectCalls weighs each allele by its population frequency, which Mutect2 takes from gnomAD's germline resource. On public data (HG002 with HG005's reads mixed in) the estimate removed 31 of 35 PASS calls with gnomAD and none without it. The step now warns in that case, and main.nf warns on the console without --chip_common_sites and with it but without --chip_germline_resource, since a task's stderr is not shown. The docs say the estimate needs gnomAD too and give the measurement. Bead: trait-62
Check the positional binding of the common-sites VCF and index. List the optional VCF in the Nextflow guide. Bead: trait-62
…s and on the live page
Moving homozygous and hemizygous P/LP rows from `carriers` to
`hom_or_hemizygous` left every consumer reading only `carriers`. The live
page lost its act finding for such a row ("two changed copies", "the only
copy changed"), and a gene left with a homozygous row fell to a plain
ClinVar review finding. Both reports dropped the row from the carrier
block.
The live page (finding rows, the carrier panel and its headline) now
reads both lists. Both report renderers (og-render and render_report.py,
still step 24's) print each such row on its own line in the carrier
block, "two changed copies" or "the only copy changed", "not the pattern
of a carrier", and the block's first line counts it apart from the
carriers. The report-render parity cases whose summary holds such rows
get goldens for the new lines.
The two CFTR pair tests now assert their row count, so they cannot pass
with no row.
Bead: clin-15
Bead: clin-19
…ort goldens The report-render goldens of the all and all-history cases print the LPA section of the synthetic corpus summary, whose two risk-allele lines read as an rsID with a genotype. They are invented test data: allow those two lines in those goldens only. Bead: clin-15
…on every run A collected summary carries the minute og summary collect ran (generated_utc), and both reports print it. The report-render goldens of the cases with a row in hom_or_hemizygous therefore failed on the next day's run. report-corpus.sh now writes one fixed time into every collected summary (sed on that one line, so a summary that holds NaN or Infinity on purpose stays as it is), and the goldens are written again from og run inside the checkout, as CI runs it: the all and all-nf summaries now read their repeat tables, as on CI. The pinned oracle matches the old og on every other output (299 of 301; the other 2 are the committed viral golden), and no new golden is stale. The all and all-history text goldens also print the corpus's invented repeat sizes: allow those two lines too. Bead: clin-15
…idence # Conflicts: # CHANGELOG.md
# Conflicts: # CHANGELOG.md
… call The SF card called two_hits(), so a class 4/5 deletion plus a P/LP call below step 06's call floor read 'biallelic' there, while the carrier list of the same report read it as possibly biallelic (low-confidence call). Two phased hets of which one is a low-confidence call also read 'biallelic' with no mark. Each counted allele now carries its low_confidence mark. The deletion rule goes through two_hits_with, and a pair whose proof needs a low-confidence call reads 'possibly biallelic (low-confidence call)'. A confident homozygous call or a confident pair stays 'biallelic'. Bead: clin-19
…zygous rows The Python server flags a sample whose carrier list holds only homozygous or hemizygous rows (a man with only a GLA row). og live serve read only 'carriers', so it gave such a sample no attention flag. Bead: clin-15
The report-render cases now compare whole reports. The 24 cases whose summary holds a homozygous or hemizygous carrier row get whole-report goldens; each differs from the oracle only in the carrier block. The docs and the changelog say what the SF card now does with a pair on a low-confidence call. Bead: clin-15, clin-19
…ssense scores written as text Bead: nf-vcfanno-ops
… a weak call The two cards capped different pairs when a gene held a low-confidence call. The carrier list capped a pair whenever any of its hets was weak, so a confident phased pair beside a weak third het read "possible two hits (low-confidence call)". The secondary-findings card capped only a pair whose proof needed the weak call. Commit 6059508 also said a confident pair stays biallelic beside a low-confidence call. That was false for a confident deletion pair: the card gave it the low-confidence word, while the same row's two-hit line said biallelic. And a combined allele lost its confidence when any one of its calls was weak. Both cards now count a gene's hits twice, with every call and with the confident calls alone: - A pair the confident calls prove is "two hits" on the list and "biallelic" on the card, whatever weak call sits beside it. - On the card, a confident deletion pair that is biallelic or possibly biallelic is used before the low-confidence cap. - A pair that only a weak call proves gets the low-confidence word. - Hits of unknown phase are "possible two hits" and "possibly biallelic" when two of them are confident. Two confident unphased hets plus a weak homozygous call now read "possibly biallelic" on the card, as the carrier list already said. - An allele counts the copies its confident calls give (a count, not a flag), so a combined allele that one confident call proves stays firm. The carrier list counts a combined allele the same way. No reported or not-reported decision moves; only `pair` and `phase` change. A new test runs MUTYH, which is on both lists, through both cards over 25 shapes of hits and requires the same answer from both. Bead: clin-19
…port With step 22's consensus set and an SV whose support is null, both reports must say the consensus cannot tell, never that it did not run, in the AnnotSV rows and in the secondary-findings SV rows, and the AnnotSV count line must count the unknown rows. Without the consensus the old text stays. Red on the code before the fix: the header lacks "not known: 1" and the row says "no step 22 consensus". Bead: sv-01
…ll an SV's support og summary collect writes an SV's support as null when step 22's consensus ran but cannot tell (an SV_VCF run's rows, a row with no ID, an ID in two events, a vote with no readable ID). Both renderers still said "not known (no step 22 consensus)" for every null, and the AnnotSV count line read "single caller: 0". The AnnotSV section's consensus flag now picks the reason in the AnnotSV rows and the secondary-findings SV rows, and the count line adds ", not known: M" when the consensus ran. bin/render_report.py and bin/live_results.py change the same lines for the live site. The pinned Python oracle prints the old reason, so the 50 report-render cases that show it get whole-report goldens. report-corpus.sh pins each collected summary's generated_utc so those goldens stay stable (the same hunk as the carrier-confidence lane). Bead: sv-01
…der goldens The whole-report goldens of the c-full, all and a-/s- report-render cases print the made-up lipoprotein(a) genotypes and repeat sizes of the summary-collect corpus. The scan read them as personal data. Allow those four lines under tests/parity/golden/report-render only. Bead: sv-01
…t the 9 GB GATK one GATK_IMAGE is pulled lazily, so the shard that ran this case pulled about 9 GB to bgzip five small tables. The bcftools image is cached and ships bgzip and tabix.
The new count line read each row's support with d["support"] (Python)
and at("support") (og-render). A live-site summary whose rows have no
support key then raised, and the live site dropped the structural card:
tests/test_owngenome_live_family.py counted 0 flagged cards where it
expects 1. Both now count a missing key as not known, the same as null.
Bead: sv-01
Two confident hets of unknown phase beside a weak het in trans with one of them read "possible two hits (low-confidence call)" on the carrier list but "possibly biallelic" on the secondary-findings card. The card checked two confident copies of unknown phase before the low cap; the list capped first. The list now takes the same step in the same order. A combined allele that is confident only through a folded part kept the GT and phase set of its own weak call, so both cards could call a pair biallelic that the confident calls do not prove. Each allele is now counted with the GT and phase set of the confident call that makes it confident, on both cards. On the card's deletion path a weak combined allele with a confident part is no longer marked low when the confident calls give its copies, and it pairs with the deletion by that confident call's phase. New shapes in the agreement test: R1, R1b, R1c (the list now says possible two hits), R2 and R3d (the card now says possibly biallelic, with its two-hit line), R3 (both now say possibly), and R6 and R7, which keep the low word on both cards. Bead: clin-19
The docs and the changelog said a gene on both lists gets the same answer. That holds for a recessive gene the SF card counts by two copies, and only at the gene level: on the carrier list the answer is the best of `pair`, the `two_hits` verdict and the homozygous rows, and `pair` counts heterozygous hits only. Say so, and name the new rule for two confident copies of unknown phase. Bead: clin-19
… reason too bin/live_results.py called sf_rows without the AnnotSV consensus in the clinical-filter table and in the findings list, so those rows still said "no step 22 consensus" after step 22 ran, next to AnnotSV rows that said it cannot tell. Both now pass the consensus, as the AnnotSV rows do. The parity README names the new golden cases and the generated_utc pin, and the two repeat-size allow lines cover only the text goldens. Bead: sv-01
A HIGH-impact record of the clinical VCF for the same variant made a weak call count as confident on the secondary-findings card, because the card read only step 06's low_confidence mark. With the record's phase set it proved a pair biallelic from a weak phase alone, and beside a deletion it cleared the low-confidence word. The card now also treats a call whose HIGH-impact record has weak_call as low confidence. The carrier list paired a weak combined allele with a deletion by the weak row's own GT and phase set, while the card used its confident part. The list now pairs it the same way: by the first confident row of the same allele with as many copies. The card dropped the phase of a combined allele whose confident call gives fewer copies than the weak one, although that phase came from the confident call. Every small allele now counts with the copies, GT and phase set of its confident call. The agreement test gains eight shapes (H1, H2, H3p, H5, X4, X4b, X9b, X10); each fails on the previous head. Part of GeiserX#411 (clin-19).
A HIGH-impact record is weak on a skewed allele balance too, not only below the clinical call floor. Such a call is now low confidence on the secondary-findings card and confident on the carrier list, even when both files state the same floor. The changelog and the results page said the two lists agree whenever the floors match; they now name this case. Part of GeiserX#411 (clin-19).
…balance The docs said a HIGH-impact call weak on allele balance is low confidence on the card only. It never lowers a hit whose ClinVar row is confident, so that split does not exist. The real one is the HIGH-impact call of a part of a combined allele, which makes the allele confident on the card while the list marks it low confidence. Bead: clin-19
…part The carrier list rescued a weak part by its confident combined allele in every case. The card counts a part through its combined allele only when that allele is unphased and the part is its first part present, so a phased combined allele inside a deletion made its weak part just outside it read biallelic on the list. The SF card's deletion pair on a weak HIGH-impact call also blamed the call floor; it now names both kinds of low-confidence call. Bead: clin-19
…idence # Conflicts: # CHANGELOG.md
…t text
Main now prints the SV caller support as main's code words it ("caller
support not known (no step 22 consensus)" and the like), so the 38 report
goldens this PR adds no longer matched og. Each file is og's own output
from the merged tree; oracle-diff summary-collect report-render agrees on
all 1097 items, and no summary-collect golden moved.
Bead: clin-19
# Conflicts: # CHANGELOG.md # tests/parity/golden/report-render/all-history/all-history.txt # tests/parity/golden/report-render/all/all.txt # tests/parity/golden/report-render/cb-full/cb-full.html # tests/parity/golden/report-render/cb-full/cb-full.txt # tests/parity/golden/report-render/cb-quoted-dosage/cb-quoted-dosage.html # tests/parity/golden/report-render/cb-quoted-dosage/cb-quoted-dosage.txt # tests/parity/golden/report-render/cb-quoted-interval/cb-quoted-interval.html # tests/parity/golden/report-render/cb-quoted-interval/cb-quoted-interval.txt # tests/parity/golden/report-render/cb-quoted-smn/cb-quoted-smn.html # tests/parity/golden/report-render/cb-quoted-smn/cb-quoted-smn.txt # tests/parity/golden/report-render/cb-unicode-contig/cb-unicode-contig.html # tests/parity/golden/report-render/cb-unicode-contig/cb-unicode-contig.txt # tests/parity/golden/report-render/m-clinical-escape/m-clinical-escape.html # tests/parity/golden/report-render/m-clinical-escape/m-clinical-escape.txt # tests/parity/golden/report-render/m-clinical-stale/m-clinical-stale.html # tests/parity/golden/report-render/m-clinical-stale/m-clinical-stale.txt # tests/parity/golden/report-render/m-clinvar-freq-note/m-clinvar-freq-note.html # tests/parity/golden/report-render/m-clinvar-freq-note/m-clinvar-freq-note.txt # tests/parity/golden/report-render/m-clinvar-many/m-clinvar-many.html # tests/parity/golden/report-render/m-clinvar-many/m-clinvar-many.txt # tests/parity/golden/report-render/m-clinvar-none/m-clinvar-none.html # tests/parity/golden/report-render/m-clinvar-none/m-clinvar-none.txt # tests/parity/golden/report-render/m-couple-empty/m-couple-empty.html # tests/parity/golden/report-render/m-couple-empty/m-couple-empty.txt # tests/parity/golden/report-render/m-couple/m-couple.html # tests/parity/golden/report-render/m-couple/m-couple.txt # tests/parity/golden/report-render/m-cpsr-none/m-cpsr-none.html # tests/parity/golden/report-render/m-cpsr-none/m-cpsr-none.txt # tests/parity/golden/report-render/m-cpsr-noown/m-cpsr-noown.html # tests/parity/golden/report-render/m-cpsr-noown/m-cpsr-noown.txt # tests/parity/golden/report-render/m-sf-notes/m-sf-notes.html # tests/parity/golden/report-render/m-sf-notes/m-sf-notes.txt # tests/parity/golden/report-render/m-since-clinical/m-since-clinical.html # tests/parity/golden/report-render/m-since-clinical/m-since-clinical.txt # tests/parity/golden/report-render/m-slivar-phase/m-slivar-phase.html # tests/parity/golden/report-render/m-slivar-phase/m-slivar-phase.txt # tests/parity/golden/report-render/m-slivar-unphased/m-slivar-unphased.html # tests/parity/golden/report-render/m-slivar-unphased/m-slivar-unphased.txt
# Conflicts: # CHANGELOG.md
…t traced A traced run skips the panel measure (measure_on in its file, GeiserX#572), so its shard should not be planned around 2900 seconds of work that never runs.
…eiserX#571 GeiserX#559 (carrier confidence) and GeiserX#571 (SV support reason) each added the same 38 report-render goldens, each written by its own og. On the merged tree og prints both changes, so the 38 files are og's output on this tip: every line of each comes from one of the two sides, and oracle-diff now agrees on all 1097 summary-collect and report-render items (38 differed before).
og live serve now does what the Python server does behind its password: the PBKDF2-SHA256 hash of bin/owngenome_live.py hash-password (an existing site keeps its login), sessions with their hour and idle limits, the limit on failed logins, --allowed-host, --trust-proxy, HTTPS with --tls-cert and --tls-key, --cookie-secure, and the ingest listener that cuts and stores each message. /metrics needs a login or its bearer token once a password is set. This is the parity step; the D12 refusals follow. Bead: D1 Bead: D14
…rd, and take only the server's ingest token Every account on the machine can reach 127.0.0.1, and so can anyone a reverse proxy lets through. Without a password og live serve now exits 2 on --trust-proxy, --allowed-host, --tls-cert/--tls-key and --cookie-secure, and answers 403 to a request with a Forwarded, X-Forwarded-* or X-Real-IP header (forwarded-refused in the access log). The ingest port takes a message only at /<token>/ingest/<run>. The token is made by the server and kept in STATE_DIR/ingest-token, mode 600; a file another account can read, or one that holds anything else, is replaced. A non-loopback --ingest-bind exits 2, and a password site that serves plain HTTP on a non-loopback bind prints a WARNING line. tests/port_live_auth_ingest_d12.rs: six tests, all red on the parent commit. Bead: plat-12 Bead: plat-13 Bead: plat-14 Bead: plat-15 Bead: D12
…s from tests/browser/cdp.py The checks of tests/test_owngenome_live.py are cargo tests of og live serve now (tests/port_live_auth_ingest*.rs). The Chrome driver and the asset check it held move unchanged to tests/browser/cdp.py, which the Python tests that drive Chrome import; test_report_charts.py keeps its own small Python-server start. The fixture plugin only that test loaded is gone too. docs/live-site.md and the changelog describe og live serve's login, the ingest token and the refusals without a password. Bead: D14 Bead: plat-12 Bead: plat-13 Bead: plat-14 Bead: plat-15
|
Important Review skippedToo many files! This PR contains 165 files, which is 65 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. CodeRabbit couldn't verify your organization's usage billing state, so no usage-priced review was started. Retry after the billing service recovers.
|
The personal-data check refuses private-range addresses in tracked files. The test's trusted network and client now sit in 203.0.113.0/24, with the same meaning. Bead: D1
…and the bound address An empty --password-file is refused instead of serving with no login. Expired sessions and old failed-login counts are dropped as new ones come, so clients that never return do not grow memory. The TLS handshake gets ten seconds in all, not ten seconds per read. Two servers started together on one state folder end with one ingest token. A message whose body ends before its Content-Length is refused, and a message after a write that failed part way gets a line of its own. Every X-Forwarded-For field counts, in order. NaN and inf are refused for the session limits. The address actually bound must be loopback for the ingest port, and for the site when no password is set. The start-up URLs bracket IPv6 addresses. The expiry tests leave a second of margin, and the no-password config tests no longer read OWNGENOME_LIVE_PASSWORD_HASH from the shell. Bead: pgp-1-0 port-live-auth-ingest
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
og live serveis the Rust port of the live site, and it had no login. It took no password, so it served loopback only. That alone does not keep a genome private: every account on the machine can reach 127.0.0.1, and so can anyone a reverse proxy lets through to it. It also had no ingest port. On the Python server that port takes a message from any account on the machine, so another user can post fake progress into a run.The fix
og live servenow has the site's login. It takes the PBKDF2 password hash (og live serve hash-passwordwrites one, and a hash the Python server wrote still logs in), the session cookie, the session and failed-login limits, the Host check,--trust-proxy,--cookie-secure, and HTTPS through rustls./metricsthen needs a login or its bearer token. (D1)--trust-proxy,--allowed-host,--tls-certwith--tls-key, and--cookie-secureexit 2. A request with aForwarded,X-Forwarded-*orX-Real-IPheader gets 403. An ssh port forward sends no such header, so it still works. (plat-12, D12)/<token>/ingest/<run>. The server makes the token and keeps it in<state dir>/ingest-tokenwith mode 600. A token file that another account can read, or that holds anything else, is replaced. (plat-13)--ingest-bindexits 2. (plat-14)WARNINGline as it starts. (plat-15)tests/test_owngenome_live.pyis gone. Its checks are cargo tests now (rust/crates/og-live/tests/port_live_auth_ingest*.rs). Its Chrome helpers moved unchanged totests/browser/cdp.py, which the Python browser tests import. (D14)scripts/owngenome-live.shandrun-all.shstill use the Python server.run-all.shcannot post to the token path yet, anddocs/live-site.mdsays so.Stacked on #571 until it lands. This branch starts at the head of #571. The last three commits are this PR.
Files outside the plan's list:
og-live/src/plugins.rs(puts/metricsbehind the login),og-live/tests/port_live_server.rs(one CLI assertion that changed),tests/test_live_viewer.pyandtests/test_report_charts.py(they imported the deleted test),tests/browser/README.md, the fixture plugin only the deleted test loaded, andrust/Cargo.lock(rustls, ring, sha2).Not ported: the Python test's check that a client that stops mid-request is dropped. It needs a 60-second wait, because the request timeout is a constant. Both listeners still set read and write timeouts. The plugin flag and the state hook belonged to the Python plugin interface, which
ogdoes not have;--plugin-diris refused.How it was checked
On an Apple Silicon test Mac, at this head:
cargo test -p og-live: every test passes. That includes 12 inport_live_auth_ingest.rs, 6 inport_live_auth_ingest_d12.rsand the Chrome test inport_live_auth_ingest_browser.rs.port_live_auth_ingest_d12.rsall fail on the parent commit 9d323f7 (0 passed, 6 failed)./metricsopen, a stream that outlives its session, one connection at a time, a cookie withoutHttpOnly, no 413 on a bad length, and a login that ignoresnext.og live serve.cargo fmt --all --checkandcargo clippy --workspace --all-targets --all-features --locked -- -D warningsare clean.tests/browser/cdp.pypass withog:test_live_demo.py(every check, 616 s),test_live_viewer.py,test_owngenome_live_family.pyandtest_report_charts.py.Beads: plat-12, plat-13, plat-14, plat-15, D1, D12, D14