Repository navigation
ENH: Add a synthetic signal generator and the tools to score the cutoff selection - #12
Merged
Merged
Conversation
`erb` reproduces the baselines and peak layout of Donald Erb's `test_donnie.py`: a parabola, an exponential, a sinusoid and their combination, each over the one, two and three plateau abscissa cases, with Gaussian peaks and white noise. `erb_native` puts this project's own peak component on those analytic baselines, so the drift is a known closed form while the peaks are exponentially-modified Gaussians of the widths actually observed here. Record lengths are drawn from the quantile function of the real length distribution, so the synthetic set spans the real range without copying any single record. `native_peak_component` is lifted out of `make_signal` unchanged and shared by both, which keeps one definition of what a peak looks like. Output is quantised to the detector step, since a benchmark whose noise is finer than the ADC lattice is not testing what the pipeline receives. Tests cover the shape of each baseline, the reproducibility of a seed, the invariance of the peak positions under the abscissa, that the components sum to the signal before quantisation, that the quantised output lands on the lattice, and that the drawn parameter ranges are those of the source rather than widened versions of them.
The diagnostic called `_fcutoff` with plain `beads` and no regions, while production calls `custom_beads` with the regions and sampling of `_relevant_regions`. It was therefore scoring a different algorithm from the one being tuned, and reported a 15% crash rate and 54% containment that belonged to nothing in the package. Take `peak_regions`, `sampling` and `scut` from `_relevant_regions` and pass the same `method_kwargs` production passes. Tests pin the mirror so the two cannot drift apart again silently. Any cache written by the previous harness is invalid and was discarded.
Aggregates `diag_summary.csv` and the per-signal diagnostics of a synthetic dataset into the test battery: harness validity, accuracy of the current selector, containment of the true optimum, the effect of the stiff-side trim with and without, and the candidate features for stage 3. Includes the sensitivity-minima view, which tests Emmanuel's hypothesis that the optimum sits at a local minimum of the sensitivity curve, against two matched null views so that a coincidence is visible as one. It measures and plots; it fits nothing and proposes no constant.
Collects real chromatographic drifts from three sources of different provenance and writes them to a pool with a manifest and a review plot each, for the `hybrid` family that places known peaks and known noise on an experimental background (Niezen et al. 2022, §4.1, Fig. 1). Provenance decides the trimming rather than a threshold: a trace published as a background is used whole, since its authors already performed the peak removal, while an ordinary run carrying analytes has its longest peak-free stretch extracted. The single criterion cannot serve both, its scale being pinned by the detector quantisation on our records and falling below the drift curvature on the borrowed ones. The `hybrid` family is parked in favour of `erb_native`, so this is experimental work kept for reference and may be deleted later. Nothing in the package imports it.
Two constraints the generator did not enforce, both found by looking at the signals rather than at the summary numbers. Nothing elutes before the dead time. It is the transit time of an unretained species, so it bounds every retention time in the run: a feature earlier than it would have to have travelled the column faster than the mobile phase. The blank ghost peaks drew from a fraction of the record and ignored it entirely, putting a peak ahead of the dead time on 46 of the 72 blanks, by up to 3.32 min. They now draw from the same elution window as the analytes. Fixing that exposed a second route to the same defect. The isocratic case pins an early and a late eluter at fractions of the *record*, not of the elution window; once the dead time pushes the window late, the late pin can land below its lower bound, and the sort that follows then makes that peak the first one. Both pins are clipped into the window. An analyte has eluted when its trailing edge is back in the noise. A record that stops before that is not a short run, it is an impossible one: the trace ends part-way down a peak, which is the endpoint mismatch BEADS' periodicity requirement forbids and which the parabola correction cannot repair. 63 of 432 signals ended with more than 5% of the tallest peak still standing, 12 of them with more than half. Those records carried a 33.3% crash rate against 6.5% for the rest, and 21.4% containment against 78.0%. The analyte set is now redrawn until the component has decayed below the noise at the last sample, the elution window being pulled in when redrawing alone does not get there so the loop terminates. The criterion is the noise the record will carry, which the generator knows exactly, rather than a width rule: a peak has eluted when it is no longer distinguishable from the noise. `native_peak_component` takes it as an optional argument and is byte-identical without it, so the `native` family is unchanged. Measured over 900 draws across every peak case and baseline: no peak before the dead time, and 0.6% of records ending mid-peak, all of them shorter than 700 points. Those short records cannot be fixed by redrawing: a 4.5 min absolute dead time in a 3.9 min record leaves no room for a wide peak to elute, which is a conflict between the measured length distribution and the measured dead time, not a generator bug.
Erb's baseline 3 is a pure sinusoid, ``10 + sin(x/50)``, 2 units peak to peak, oscillating through several full cycles within one record. An HPLC baseline does not do that, so scoring against it measures the method on drift it will never meet. It also fails to provide a target at all. On the wide and mixed peak cases the cutoff minimising the RMSE against it recovers the baseline *worse than a horizontal line*: median ratio 1.19 on ``multi_wide`` and 1.15 on ``multi_mixed``, so the "true optimum" recorded there is the least bad of a grid of bad fits. 35 of 503 scored signals were being measured against such a reference, and 25 of the 72 that select nothing. Removing them changes the reading of the benchmark. Containment goes from 57.3% to 61.6% before and 59.5% to 64.4% after the region-detection work, and the improvement that work produces goes from indistinguishable (gained 37, lost 23, p = 0.092) to detectable (gained 29, lost 14, p = 0.032, McNemar exact). Skipped inside the generation loop rather than dropped from the job list, so the index that seeds every other signal is unchanged and ``erb0`` to ``erb2`` regenerate byte-identical.
The second group with enough members to stand on its own: the benchmark generator, its diagnostic, its report battery, the width sweep, the parked background loader and `synthetic_data.md`. Renames only, but three things break silently on a move and are fixed here: the usage lines and `synthetic_data.md` cross-references inside the moved files; `backgrounds.py`'s repository root, which walked up two directory levels and now needs three; and the two test modules, which load the tools by explicit path since `tools/` is not a package.
Two figures per signal rather than only for the ones that select a cutoff: the production autocorrelation plot with the true optimum drawn on it, and the chromatogram carrying the known baseline, the selected one and the one at the true optimum, over the corrected traces. A signal that selects nothing produces no figure at all today, because `_fcutoff` raises before it builds `plot_data`, so exactly the cases worth looking at are the ones with nothing to look at. Here the selection is rebuilt from the cached curve, and its absence is drawn rather than raised. The true optimum is read from the stored `diag/*__diag.npz` rather than recomputed, so the error curve is not swept again; only the two final BEADS corrections run per signal. It is not added to `plot.r2_plots` itself, and should not be: production has no true optimum to draw.
Decades were the wrong yardstick and Emmanuel called it: on the signals
he judged acceptable by eye, `|d_decades|` has median 0.399 and p90
1.621, against median 0.338 and p90 0.531 on the ones he judged wrong.
The acceptable selections are FURTHER away in decades than the misses,
because a decade measures a distance along the parameter and not in the
corrected chromatogram. `penalty` orders them correctly but is a ratio,
so it diverges as `e_min` approaches zero while both fits sit far below
the noise: that is where the 2479 came from.
A literature check found no convention for dividing by the noise, and
no worst-case metric at all. Every source simply asserts its
normalization. So all of them are reported and the choice is left open:
rms plain RMSE against the known baseline, dividing only by the
point count. Niezen, Schoenmakers & Pirok (2022) Eq. (15).
max maximum absolute departure. Used by no source; here because
the rms averages a local defect over the record, so the same
defect scores lower in a long run than in a short one.
rms, max the two above in units of the record's noise. Used by no
in sigma source either, and misleading when the signal towers over
the noise: a visually perfect fit on a low-noise trace reads
13.8 sigma.
snr_db 10 log10(energy(true) / energy(true - fit)). Ning, Selesnick
& Duval (2014) §5.1, the BEADS paper. Note the energy
includes the baseline's offset, so a trace far from zero
scores high whatever the fit does; their simulated baselines
are near zero and ours are not.
area_pct percent error in recovered peak area against the known one.
Navarro-Huerta et al. (2017) §3.6; Niezen §4.3.3 uses the
same measure. Over the whole record rather than per peak,
since the region boundaries would otherwise be our choice.
All are computed for the fit at `fc_best` and at the selected cutoff.
`d_decades` and `penalty` stay as columns because the report battery
reads them; neither should be quoted as accuracy.
The error curve is now reused across runs. It is 250 BEADS fits per
signal, unchanged by the trimming, the selection or the metrics, and it
was being recomputed every time although `diag/<stem>__diag.npz` already
stored it. It is keyed on the signal, the abscissa, the true baseline,
the grid, the output of `_relevant_regions` and `end_window`, so a
change to the region detection invalidates it -- which is the failure
the key exists to prevent, and it happened four times this week.
`--refresh-err` forces recomputation. Arrays are hashed at float32 as in
`baseline._r2_cache_key`, and the six-signal check on Rorqual confirms
both keys transfer: identical cutoffs to twelve digits under numpy 2.4.2
against 2.5.0.
`synth_figs` reads every figure from `diag_summary.csv` rather than
recomputing, so a plot cannot disagree with its own table, and fills the
optimum's figures itself for the signals that select nothing, whose
optimum does not depend on the selection.
The test that forbade a peak-area column is repointed: the area is now
reported, but `fc_best` must stay the minimum of the baseline error
curve, which is what the test pins.
Follows tools/slurm_script: work in $SLURM_TMPDIR, run the branch checked out in the launch directory, move the results and the log back at the end. The resource form is `--ntasks=1 --cpus-per-task=16` rather than the whole-node one, because the run is a single process with a worker pool over signals: `--ntasks-per-node=16` would give sixteen tasks of one cpu each and the pool would be confined to one. Sixteen of the 192 cores of a Rorqual node is not a whole node either. `--mem-per-cpu=500M` against a measured peak of 287M on the largest signal of the set. The dataset is copied to node-local disk because the run writes one npz per signal, and it carries its r2 cache and stored error curves, so a re-run pays for neither the sweep nor the error curve.
The convention is the whole-node form with the per-cpu one commented above, and it is also the better fit: `--mem` pools the memory across the allocation, while `--mem-per-cpu` budgets each worker separately, so one worker on the largest signal at 287M can be killed while the others sit at a tenth of that. The earlier justification for inverting it was wrong on the mechanism too: the task count does not confine a process pool to one cpu, since Slurm's cgroup grants the job every allocated cpu on the node.
`error_curve` fitted with `parabola_len=end_window(s)` and its docstring claimed that mirrored the final correction of `auto_beads`. It does not. `auto_beads` substitutes `end_window(s)` only when a caller passes `parabola_len=None`, and no caller does: `weaselytics.py` passes nothing, so the default 3 applies to the sweep and to the final correction alike. That branch predates pybaselines exposing `parabola_len` on BEADS and is a fallback, not the production setting. So the answer key was the optimum of a slightly different problem from the one production solves: BEADS anchors its endpoint parabola on 3 points at each edge in production, and on up to 20 in the benchmark. Every stored error curve is invalidated, which the `err_key` handles since the parabola length is part of it. The r2 cache is untouched: the sweep always used 3.
Measured on the 432-signal set, scored by the RMS departure in mV between the baseline at the true optimum and the known one. The assumptions, with pointers: Ning et al. (2014) require the peaks and their first M derivatives to be sparse (§3, §3.2), the baseline to be low-pass with no parametric form and no rule tying the cutoff to peak width (§3.1), the noise to be stationary white Gaussian (Eq. 7-8), and positivity only as a preference encoded in the asymmetry parameter (§3.4). Their validation reports no peak density or resolution at all. Navarro-Huerta et al. (2017) list five limitations (§3.1 i-v) and peak overlap is not among them. **Both papers are silent on overlap, resolution and peak density**, so what follows is a finding outside what either addressed rather than a documented violation. Recorded as such. Failure is not spread evenly. By peak case it runs from 0.0% on blanks to 65.3% on multi_wide; by baseline, erb2 is 48.6% against 18% for the other two; noise barely matters. The cross-tab shows blanks clean on every baseline, so failure needs both a hard baseline and peaks. Emmanuel's reading that the failures are simply not sparse enough holds where it can be tested: over the 288 multi-peak signals, the failing ones have a median adjacent-pair resolution of 0.13 against 0.34 for the passing ones, and cover 19.9% of the record with peaks against 6.9%. Within `multi_narrow x erb0` alone the split is 0.71 against 0.12. But `multi_wide x erb0` inverts and `multi_mixed x erb0/erb1` show nothing, so crowding is not the whole account and the section says so. 252 of 288 multi-peak signals have an adjacent pair below Rs = 1.5, so the generator is producing mostly unresolved chromatograms. Finally the analytic baselines are unlike the real ones: the 339 real baselines have a median range of 0.279 mV against 5.00, 8.11 and 12.13 for erb0, erb1 and erb2. Only 5 of 339 exceed 5 mV. With the warning not to read direction-change counts off a dual-axis plot, where a 0.05 mV wander looks like a 7.8 mV drift.
Brings the synthetic branch up to the current production code. The two copies of the synthetic diagnostic are merged into tools/synthetic/synth_diag.py and the flat tools/synth_diag.py goes. Kept from plateau_segmentation: the run through auto_beads with the sweep spread over the workers, --cache-dir, --diag-dir, --nshards and --shard, the summary written beside the diagnostics, and in_candidates with its candidates mask. Kept from synthetic_signals: the production pick and the error at it, the instability-trim on/off pair with its self-check, the reuse of the error curve, the five error measures and the pilot options. The synthetic tools are brought in line with production: the past-drop exclusion keyword (exclude_collapse was renamed exclude_past_drop) and its gate of 10; a test now ties the three copies of that gate to auto_beads. synth_figs and synth_report take the diagnostics and cache directories as options, since the summary moved. loop_bl.sbatch points at the merged diagnostic, and synth_diag.sbatch is removed: loop_bl is the one job script.
Add a section describing the LPYE dataset and its run conditions, cite Bourret, Stevenson & Cote (2024), renumber the sections after it, and refer to the dataset as LPYE where the text said real, measured or calibrated.
Point the Milani et al. (2024) citation at §3.1.3, where the bounds are stated. State plainly that an exported m of 1000 means the peak is Gaussian, drop the per-peak counts from the bounds comment, and remove the pointer from the pearson7 docstring to a section of synthetic_data.md that no longer exists.
- Peak widths follow the plate model, sigma = tR / sqrt(N), with N drawn per signal. - Peaks start after the injection artifact, end inside the run, and stay at Rs >= 1 from each other. The dead time moves earlier when they do not fit. - Blanks hold only the injection artifact. - Cases are named by peak count (blank, single, few, many, isocratic); Erb's baselines are named parabola, exp and step. - Remove the pyb family and the old erb signal builders, and rename erb_native to erb. - Signal names put a zero-padded replicate before the length, and synth_diag reads it there. - The manifest records the analytes kept and dropped, Rs, coverage, N and the dead time. - Noise levels are 0.02 and 0.05 mV; erb record lengths come from a skew-normal on log10(length).
The injection-artifact dip is now 0 to 2 times the spike as the trace shows it, drawn as twice a Beta(1, 3) variate. `_dip_scale` solves for the negative-lobe height after the two lobes overlap, and the drawn ratio is recorded as `dip_ratio` in the truth entries. The absolute `neg_abs` range is removed. The native baseline wander correlation length moves from 0.3-0.8 min to 1.2-3.2 min. `--family` defaults to `erb`. The docstrings and comments describe a general generator: ranges are picked for this benchmark, and `native` and `lit` are deprecated.
Both review figures gain a third panel, the fitted baseline minus the true one with a +-sigma noise band, which separates fits that differ by less than the noise. Each fit has its own line style, the optimal one wide and solid at the back, and the right column matches the height of the r2 figure beside it. Counts from an older run are dropped from the qo_all docstrings.
`--rate` sets the points per minute of the erb and native signals, default 120. Record lengths are drawn as durations in minutes and the artifact width is set in minutes, so another rate keeps every run as long and samples it more or less densely. `lit` stays at 120. The rate is recorded as `points_per_minute` in the truth metadata and the manifest. At the default rate the output is identical to before.
…d clear their prose
tools/cutoff_overlay/cutoff_overlay.py draws one synthetic signal's baseline at several cutoffs on one merged figure: the production r2 figure with a line per cutoff, and the baselines, corrected traces and baseline errors overlaid. The cutoffs are spaced by how much the fitted baseline moves. The signal's r2 cache entry is copied to a temporary directory first, so the run's cache is never touched. An example figure is kept in tools/cutoff_overlay/figures.
- Correct claims the code contradicts: the dimensionless-parameters closing line (tol depends on the sampling rate, the acetonitrile exception is in mV), the three breakpoints of the peaks_params ladder, and the past-drop fallback attributed to the gate. - Correct the core principle: segment boundaries fall where the slope or the noise level changes, as Figure 4 shows. - Drop statements resting on the old generator: the optimum sitting well inside the region, and the argmin test against the midpoint. - Drop history, the pointers to the local audit, and the passages that repeat the parameter tables or justify a framing. - Reflow the Parameters prose to one line per paragraph and apply the Canadian spellings.
- Carry over the corrections made to segmentation.md: segment boundaries fall where the slope or the noise level changes, and the L-curve literature is conceptually similar, not the source of the method. - Date the right-edge anchoring evidence to the synthetic baselines of the time; the valley shape it cited is reversed on the new sweep. - Cut the custom_bc note, which describes current production and repeats segmentation.md, with the two references only it cited. - Drop the defensive asides, make the examples of baseline drift the same shape, and apply the Canadian spellings.
Only the erb family has built a dataset since the generator was rebuilt. The two deprecated families go with their baselines, Gaussian peaks, constants and the --family option; native_peak_component, which erb shares, becomes peak_component. Job indices, and so the seeds of the erb signals, are unchanged: a 60-signal erb set generated before and after is byte-identical. The tests, WORKFLOW and synthetic_data.md follow.
- Point to where Erb's baselines come from: make_data, posted in issue #4, instead of "his script". - Name the paper behind qo_all's Eq. (11) (Navarro-Huerta et al. 2017) and drop a drifting line number. - Drop emphasis capitals, a passage of history, and passages that repeat another docstring; restate the artifact-versus-baseline split positively.
Follow the pattern of the research_scripts jobs: the whole-node lines are active and the per-task ones commented, CPUs are requested as tasks rather than with --cpus-per-task, and the worker count is read from SLURM_NTASKS, which both forms set. Same resources as before: 8 CPUs, 2G, 3 h.
data_aggregator.py, viz_data.py and stats_analysis.py get module and numpy docstrings, a main(), and lose their commented-out code. Their outputs are unchanged, checked on a synthetic set of peak CSVs. viz_data.py turns its hard-coded selection into --mol and --columns. chauvenets_criterions cites Taylor (1997) §6.2 and states the limits of §6.3: small groups, several outliers, weighted statistics.
One-line docstrings for the nested helpers and the figure entry points that had none, so every function the synthetic_signals branch adds or touches carries one. Code unchanged.
The peak fits export a Pearson VII row since 134f568, and the rejection and averaging already ran on it, but only the Gaussian and Skew-Normal tables were printed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a synthetic signal generator that saves each signal with its true components (baseline, peaks and noise), and the tools to score the cutoff selection of issue #4 against the true baselines. It also cleans up
tools/around them.Choosing the BEADS cutoff frequency can only be judged against a known baseline, which no real chromatogram provides. This branch generates chromatograms whose baseline is known, scores the production selection against it, and draws the figures needed to review the result.
The generator and its scoring, in
tools/synthetic/synth_dataset.pygenerates the signals. Each one is a peak component on one of the analytic baselines Donald Erb wrote for this project in issue ENH: Improve plateaus identification in_fcutoff#4 (a parabola, an exponential decay and a step), plus white noise, rounded to a detector's voltage step. The peaks are exponentially-modified Gaussians whose widths follow the plate model, placed after a bipolar injection artifact at the dead time, with a resolution of at least 1 between neighbours. Record lengths and the sampling rate (--rate) vary. The true baseline, peaks and noise of each signal are saved beside it.synth_diag.pyruns the production selection on every signal and scores it against the true baseline: the RMSE of the fit at the selected cutoff, at the best cutoff of the grid, and the peak-area error.synth_figs.pyandqo_all.pydraw the review figures: one merged figure per signal, and the same figures sorted by how close the best fit gets to the truth and by whether the best cutoff survives stage 2.synthetic_data.mddescribes the benchmark, its sources and its limits.loop_bl.sbatchruns the benchmark withMODE=synthand now draws the review figures too. Its Slurm header uses the whole-node form, and the worker count followsSLURM_NTASKS.Other tools
tools/cutoff_overlay/draws one signal's baseline at several cutoffs on one figure.loop_extract.shis restored for the peak-fitting step;WORKFLOWlists the real and synthetic steps.data_aggregator.py,stats_analysis.pyandviz_data.pyget docstrings and amain(); their outputs are unchanged.viz_data.pygains--moland--columns. The Chauvenet rejection instats_analysis.pynow cites Taylor (1997) §6.2 and states its limits.segmentation.mdcomputen_usedand the past-drop gate from the signal instead of hard-coding them.fcut_dataset.py,fcut_score.py,fcut_gallery.py),fcut_recheck.py,extract_baselines.pyand its job script,qo2_score.py,fig_epsilon.py, the unused peak-width sweep, and the deprecated synthetic families.Documentation
segmentation.mdandfcut_history.md: claims the code contradicts are corrected, statements resting on the old benchmark are dropped, and the prose is cleared of repetition.export.py,peakfitting.py: the Milani et al. (2024) pointer is corrected to §3.1.3, and the meaning of a Pearson VIImof 1000 is restated.Type of Pull Request
Pull Request Checklist
if applicable.
if applicable.
ruff check .passes, if applicable.