Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,15 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n

## Unreleased

- `og live serve` now has the live site's login and ingest port, in Rust. It takes the password hash (`og live serve hash-password`, or one the Python server wrote), the session cookie, the session and login limits, the Host check, `--trust-proxy`, `--cookie-secure` and HTTPS with `--tls-cert` and `--tls-key`, and logs each login and page read with the first 8 characters of the session's hash. `/metrics` then needs a login or its bearer token. Its ingest port cuts each message to the progress fields as the Python server does, and takes one only at `/<token>/ingest/<run>`: the token is made by the server and kept in `<state dir>/ingest-token` with mode 600, so another account on the machine cannot post fake progress. Without a password it refuses `--trust-proxy`, `--allowed-host`, `--tls-cert`, `--tls-key` and `--cookie-secure` with exit 2, and answers 403 to a request with a `Forwarded`, `X-Forwarded-*` or `X-Real-IP` header, since every account on the machine and anyone behind a proxy can reach 127.0.0.1. It refuses an `--ingest-bind` that is not loopback with exit 2, and warns when a site with a password serves plain HTTP on an address that is not loopback. `scripts/owngenome-live.sh` and `run-all.sh` still use the Python server: `run-all.sh` cannot post to the token path yet. `tests/test_owngenome_live.py` is gone; `cargo test -p og-live` (`tests/port_live_auth_ingest*.rs`) checks the login, the sessions and the ingest, and the Python tests that drive Chrome take their helpers from `tests/browser/cdp.py`. Part of #411 (plat-12, plat-13, plat-14, plat-15).
- The E2E run dispatched on a batch of pull requests finishes sooner, with every case and check it ran before. The shard split now weighs `prs-3-panel-measure` at its measured 2900 seconds on a run dispatched on a branch (`E2E_SECONDS`), where it measures the 1000 Genomes panel, so its shard no longer gets a full share of other cases on top. The base leg sets up while `og` builds, and the job `og` builds only the Linux x86_64 binary the legs run (`OG_ARCHES`, `OG_DARWIN_ARCHES` in `rust/og.sh`); Guard's `rust` job still builds all four on every pull request, as does the Rust `images` job when it runs. The nightly Rust run, which is the `streams` job only, now builds only `og-linux-x86_64`, the binary it runs. Container Test runs at most 8 smoke rows at once, so a change to every image no longer takes the runners the E2E legs wait for. The jobs that only build `og --release` lose their Rust cache, which did not speed them up, and Guard's `rust` job saves its cache on main only, so the E2E image and data caches fit the 10 GB budget.
- The carrier list in `summary.json` keeps each hit's call quality and zygosity. It judged every hit at GQ 30 and DP 10, whatever floor the hits file states (`##PGP_CALL_FLOOR`), and it dropped the low-confidence mark the ClinVar section shows for the same hit. Each row of `sections.carrier.data` now has `zygosity` (`het`, `hom`, `hemizygous` or `unknown`), `gq`, `dp` and `low_confidence`, judged against the file's own floor. A homozygous or hemizygous P/LP row has no unchanged copy, so it is no carrier finding: it moves from `carriers` to a new list, `hom_or_hemizygous`. Both reports still print it in the carrier list, on a line of its own that says it is not the pattern of a carrier, and the list's first line counts it apart (`2 with no unchanged copy`); the live page keeps its finding for it, and its carrier headline (the Python server's and `og live serve`'s) asks for attention. Two different heterozygous hits in one recessive gene are now counted as copies, with the phase rule of the secondary-findings card: each row gets `pair` (`two hits` when read-backed phase puts them on different copies, `possible two hits` when the phase is unknown; none when they sit on one copy) and `phase`, which says why. The carrier list and the secondary-findings card now count a gene's hits the same way, twice: with every call, and with the confident calls alone. A pair that the confident calls prove stays `two hits` on the list and `biallelic` on the card, even when another hit of the gene is a low-confidence call; so does a confident homozygous call or a confident deletion pair. When the confident hits count two copies of unknown phase, the pair is `possible two hits` on the list and `possibly biallelic` on the card, even when a low-confidence call would prove it. Any other pair that needs a low-confidence call is `possible two hits (low-confidence call)` on the list and `possibly biallelic (low-confidence call)` on the card. On the card a low-confidence call is one below step 06's call floor or a weak HIGH-impact call (`weak_call`). A deletion plus a low-confidence call reads `possibly biallelic (low-confidence call)` under `two_hits` and on the card's two-hit line. A combined allele counts as confident when any call of it is, and it then counts with the copies, GT and phase set of that confident call, beside a deletion too. The list's `pair` counts heterozygous hits only. For a recessive gene that the card counts by two copies (MUTYH, ATP7B, CYP27A1, GAA, RPE65, CASQ2, TRDN, and BTD apart from its hypomorph), the two give the gene the same answer, apart from rare deletion shapes where the card is more cautious (a combined allele across a deletion's end) and from a call the card judges by its HIGH-impact record (a lower call floor in the clinical VCF, or a HIGH-impact call of a part of a combined allele): on the list, that answer is the best of `pair`, the gene's `two_hits` verdict and its `hom_or_hemizygous` rows. Output change: rows move from `carriers` to `hom_or_hemizygous`, and fields are added; no field is renamed or removed. `tests/schema/summary.schema.json` now describes the carrier section. Part of #411 (clin-15, clin-19).
- A step you selected that produced nothing now says so. A report-only Nextflow step (TelomereHunter, VNtyper, the CHIP genes and the others in `nextflow.config`) is dropped when it fails, after one retry when the exit was a memory kill. Its section had no file, so the report listed it as not run, the same as a step nobody asked for. `HTML_REPORT` now records the run's `--tools` list, and `og summary collect` lists selected sections with no file under Steps Not Run as "(selected, no result)", with a note naming the tool and the step. Y haplogroup, Y tree placement, and Y-STR markers get this label only for samples declared male. `summary.json` gets a new top-level list, `selected_no_result`, with those section keys. It is present only when the run recorded its tool list, so a bash run's summary is unchanged. The section's state stays `missing`.
- One section that cannot be written no longer stops the whole report. Step 24, the Nextflow `HTML_REPORT` module and `scripts/generate-report.sh` ran `bin/render_report.py`, and a section whose data it could not read stopped it with a Python error: the run got no text report and no HTML report at all. The three now run `og report render` (`rust/crates/og-render`), so no Python runs for the reports. A section that cannot be written says so in its own place, `[could not be read: ...]` in the text report and an Unreadable card in the HTML report, and every other section is still written; `og report render` prints one `WARNING` line on stderr that counts and names those parts, and still exits 0. A panic inside a section writer counts the same way, so the release build now unwinds on a panic (`rust/Cargo.toml`); a panic anywhere else still aborts og at once, through the panic hook og installs first. A search index or charts file that cannot be made is written empty and named in the same `WARNING` line. On the 164 report-render parity summaries (154 before, and 10 new ones that reach each state of the mosaic sex-loss, raw-data export and structural-variant sections) both reports are byte for byte the Python's, whole, with two changes on purpose: a gene the ACMG card names is listed once in the closing gene list, not twice (the `acmg_sf` section repeats the card the clinical section holds), and a run whose BAM has no unplaced unmapped reads says the viral screen was not assessed instead of "no read". The checks of `tests/test_render_report.py` are cargo tests (`rust/og/tests/report_checks.rs`, and `rust/crates/og-render/tests/section_failure.rs` for a section that panics), and the Python file is gone. `bin/render_report.py`, `bin/report_charts.py` and `bin/results_index.py` stay only for the Python live site.
- Nextflow's `docker` profile now runs each task as you, as the step scripts do, not as root. Before, every task ran with `--user root:root`, so a file a task wrote outside its work directory, beside an input under `GENOME_DIR`, stayed root's (Nextflow's `docker.fixOwnership` gives back only the work directory). Tasks now run with `-u` and your UID and GID, with `HOME=/tmp`. `CPSR` and `CNVPYTOR` still run as root, with `HOME=/root` as before: PCGR has not been shown to run unprivileged, which is why step 17 runs it as root too, and the CNVpytor module copies its resources into its image. A file those two write outside their work directory stays root's. The ids are written as bash's `${UID}:${GROUPS}`, which have no space. The PRS module, which hands the run's options on to pgsc_calc, now also drops a user flag with its whole value: a config's common `-u $(id -u):$(id -g)` gave pgsc_calc a stray `-g)` before. pgsc_calc's containers now also get `HOME=/tmp`. A sample folder written by an older run can hold root-owned files that a task running as you cannot overwrite: [troubleshooting](docs/troubleshooting.md#permission-denied-writing-to-mounted-volumes) gives the `chown` that takes it back. `tests/test_prs_runopts.sh` checks the profile's user and the PRS module's user flag drop, and the e2e case `prs-runopts-and-trace-1` checks the docker run line of every task of a stub run.
- The README and Getting started now say what to install before the first command to copy. The README's quick start opens with a "Before you begin" list (Docker, bash 4.4, Java 17, the pinned Nextflow, 16 GB of RAM with swap for step 25, and the disk), and its clone line checks out the newest release tag, so `setup.sh` downloads the `og` binaries instead of building them. Getting started has one ordered install path each for Linux, an Intel Mac, an Apple Silicon Mac and Windows with WSL2, with Homebrew first on a Mac, and each ends in the same four-command check. `tests/test_install_docs_order.sh` keeps that order.
- The reads drawing on a live finding page now says what it is before it is shown. It is the same nine bars on every finding, but its label read "The reads at <place>" above it and only a caption after it said it was a sketch, so it looked like the person's own reads. The label now reads "Where to look at the reads: <place>. An illustration, not your reads.", and the link to the real reads in the genome viewer stays. In the public demo, which has no reads, every offer of reads said so only after the click: the finding page's link, a card's "See the reads at" link, a chart mark's link and a search hit's "reads at <place>" link opened a page saying the demo has none, and the map's card button and reads pill opened a panel saying so. They now read "Reads: not in the demo": the links are plain text, and the map's buttons still open the panel that says why. On any site, the map's reads panel, which opens under the map and often below the screen, now scrolls into view when it opens. The browser check `rust/crates/og-browser/tests/rv_live_reads_sketch_label.rs` builds the demo and checks all of it. No option, output or file name changes.
- `og live serve` serves the live site from the one static `og` binary, in Rust: the home, run and family pages, `state.json` and its headlines, the event stream, the run files, the map, `/metrics` and the search, charts, ops and re-check plugins, with the same policy on every answer. It has no login yet, so it serves loopback only and refuses the password, TLS and proxy options with exit 2; `scripts/owngenome-live.sh` still starts the Python server ([Watch a Run Live](docs/live-site.md)). `bin/owngenome_recheck.py`, the shim that ran `og recheck`, is gone: call `bin/og recheck`. The Python checks of the site that `og live serve` now answers moved to `cargo test -p og-live`.
- `og live serve` serves the live site from the one static `og` binary, in Rust: the home, run and family pages, `state.json` and its headlines, the event stream, the run files, the map, `/metrics` and the search, charts, ops and re-check plugins, with the same policy on every answer. Its login came later in this release (above); `scripts/owngenome-live.sh` still starts the Python server ([Watch a Run Live](docs/live-site.md)). `bin/owngenome_recheck.py`, the shim that ran `og recheck`, is gone: call `bin/og recheck`. The Python checks of the site that `og live serve` now answers moved to `cargo test -p og-live`.
- Step 37's two Y placements are now measured against published haplogroups. On 58 men of the 1000 Genomes Project, with the haplogroups Poznik et al. 2016 published and each man's chrY from his 30x CRAM, Yleaf and `og y tree-place` each put all 58 on the published branch or below it, and named the same branch for 44. `docs/37-y-haplogroup.md` has the table, how the men were chosen and each man's calls. A new verb, `og dev evidence y-bench`, places the published calls and the tools' calls on the current YFull tree and scores them. The public-evidence workflow has a new job, `y-benchmark`: it runs step 37 again on 6 of the men, fails when a tool puts fewer than 95% of them on the published branch or below it, and checks that the same calls given to other men fail. The outvote rule of `og y tree-place` is unchanged: it fired for one man, whose placement is on his published line, so the data give no reason to change it. The `archaic-sites` job of that workflow now runs only when an archaic job needs it.
- The Guard check no longer fails at random on the SMN table tests of `og-cn`. Two of those tests ran at the same time in one folder, and the first to end removed it while the other still wrote its table there. Each test table now gets a folder of its own.
- Every data file the repository ships now has a recorded source and licence, and CI fails on one that has none. Before, nothing tied a table under `assets/` to where it came from, so a new file could ship third-party data with no credit and no terms. `assets/SOURCES.tsv` lists every file under `assets/` with its source, version, licence and the header phrase it must carry; the new top-level `NOTICE` names each third-party file and its terms. `og dev licences check` fails when a file under `assets/` has no row, a row matches no file, a file lacks the header phrase its row names, or `NOTICE` does not name a third-party row. Lint runs it on every push and pull request, with two controls that must fail: an asset with no row, and `assets/sv_rule_regions.tsv` with its source line removed. A change that adds a file under `assets/` now adds its row in the same pull request.
Expand Down
Loading
Loading