Skip to content

ENH: Add a synthetic signal generator and the tools to score the cutoff selection - #12

Merged
physicien merged 49 commits into
developmentfrom
synthetic_signals
Nov 9, 2026
Merged

physicien merged 49 commits into
developmentfrom
synthetic_signals

Conversation

@physicien

Copy link
Copy Markdown
Owner

Description

Adds a synthetic signal generator that saves each signal with its true components (baseline, peaks and noise), and the tools to score the cutoff selection of issue #4 against the true baselines. It also cleans up tools/ around them.

Choosing the BEADS cutoff frequency can only be judged against a known baseline, which no real chromatogram provides. This branch generates chromatograms whose baseline is known, scores the production selection against it, and draws the figures needed to review the result.

The generator and its scoring, in tools/synthetic/

  • synth_dataset.py generates the signals. Each one is a peak component on one of the analytic baselines Donald Erb wrote for this project in issue ENH: Improve plateaus identification in _fcutoff #4 (a parabola, an exponential decay and a step), plus white noise, rounded to a detector's voltage step. The peaks are exponentially-modified Gaussians whose widths follow the plate model, placed after a bipolar injection artifact at the dead time, with a resolution of at least 1 between neighbours. Record lengths and the sampling rate (--rate) vary. The true baseline, peaks and noise of each signal are saved beside it.
  • synth_diag.py runs the production selection on every signal and scores it against the true baseline: the RMSE of the fit at the selected cutoff, at the best cutoff of the grid, and the peak-area error.
  • synth_figs.py and qo_all.py draw the review figures: one merged figure per signal, and the same figures sorted by how close the best fit gets to the truth and by whether the best cutoff survives stage 2.
  • synthetic_data.md describes the benchmark, its sources and its limits.
  • loop_bl.sbatch runs the benchmark with MODE=synth and now draws the review figures too. Its Slurm header uses the whole-node form, and the worker count follows SLURM_NTASKS.

Other tools

  • tools/cutoff_overlay/ draws one signal's baseline at several cutoffs on one figure.
  • loop_extract.sh is restored for the peak-fitting step; WORKFLOW lists the real and synthetic steps.
  • data_aggregator.py, stats_analysis.py and viz_data.py get docstrings and a main(); their outputs are unchanged. viz_data.py gains --mol and --columns. The Chauvenet rejection in stats_analysis.py now cites Taylor (1997) §6.2 and states its limits.
  • The figure scripts of segmentation.md compute n_used and the past-drop gate from the signal instead of hard-coding them.
  • Removed: the scripts built on the deleted hand labels (fcut_dataset.py, fcut_score.py, fcut_gallery.py), fcut_recheck.py, extract_baselines.py and its job script, qo2_score.py, fig_epsilon.py, the unused peak-width sweep, and the deprecated synthetic families.

Documentation

  • segmentation.md and fcut_history.md: claims the code contradicts are corrected, statements resting on the old benchmark are dropped, and the prose is cleared of repetition.
  • export.py, peakfitting.py: the Milani et al. (2024) pointer is corrected to §3.1.3, and the meaning of a Pearson VII m of 1000 is restated.

Type of Pull Request

  • Bug Fix
  • New Feature
  • Miscellaneous Changes (refactor, code improvements, etc.)
  • Documentation or Example Programs

Pull Request Checklist

  • New code and/or documentation is valid for use with the BSD 3-Clause license.
  • New code is fully documented with docstrings that follow Numpy style,
    if applicable.
  • New code follows package conventions (max 80 chars per line, etc.),
    if applicable.
  • Added/updated tests and ensured they pass locally, if applicable.
  • Verified that ruff check . passes, if applicable.

`erb` reproduces the baselines and peak layout of Donald Erb's
`test_donnie.py`: a parabola, an exponential, a sinusoid and their
combination, each over the one, two and three plateau abscissa cases,
with Gaussian peaks and white noise.

`erb_native` puts this project's own peak component on those analytic
baselines, so the drift is a known closed form while the peaks are
exponentially-modified Gaussians of the widths actually observed here.
Record lengths are drawn from the quantile function of the real length
distribution, so the synthetic set spans the real range without
copying any single record. `native_peak_component` is lifted out of
`make_signal` unchanged and shared by both, which keeps one definition
of what a peak looks like.

Output is quantised to the detector step, since a benchmark whose
noise is finer than the ADC lattice is not testing what the pipeline
receives.

Tests cover the shape of each baseline, the reproducibility of a seed,
the invariance of the peak positions under the abscissa, that the
components sum to the signal before quantisation, that the quantised
output lands on the lattice, and that the drawn parameter ranges are
those of the source rather than widened versions of them.
The diagnostic called `_fcutoff` with plain `beads` and no regions,
while production calls `custom_beads` with the regions and sampling of
`_relevant_regions`. It was therefore scoring a different algorithm
from the one being tuned, and reported a 15% crash rate and 54%
containment that belonged to nothing in the package.

Take `peak_regions`, `sampling` and `scut` from `_relevant_regions` and
pass the same `method_kwargs` production passes. Tests pin the mirror
so the two cannot drift apart again silently.

Any cache written by the previous harness is invalid and was discarded.
Aggregates `diag_summary.csv` and the per-signal diagnostics of a
synthetic dataset into the test battery: harness validity, accuracy of
the current selector, containment of the true optimum, the effect of
the stiff-side trim with and without, and the candidate features for
stage 3.

Includes the sensitivity-minima view, which tests Emmanuel's hypothesis
that the optimum sits at a local minimum of the sensitivity curve,
against two matched null views so that a coincidence is visible as one.

It measures and plots; it fits nothing and proposes no constant.
Collects real chromatographic drifts from three sources of different
provenance and writes them to a pool with a manifest and a review plot
each, for the `hybrid` family that places known peaks and known noise
on an experimental background (Niezen et al. 2022, §4.1, Fig. 1).

Provenance decides the trimming rather than a threshold: a trace
published as a background is used whole, since its authors already
performed the peak removal, while an ordinary run carrying analytes
has its longest peak-free stretch extracted. The single criterion
cannot serve both, its scale being pinned by the detector quantisation
on our records and falling below the drift curvature on the borrowed
ones.

The `hybrid` family is parked in favour of `erb_native`, so this is
experimental work kept for reference and may be deleted later. Nothing
in the package imports it.
Two constraints the generator did not enforce, both found by looking at
the signals rather than at the summary numbers.

Nothing elutes before the dead time. It is the transit time of an
unretained species, so it bounds every retention time in the run: a
feature earlier than it would have to have travelled the column faster
than the mobile phase. The blank ghost peaks drew from a fraction of
the record and ignored it entirely, putting a peak ahead of the dead
time on 46 of the 72 blanks, by up to 3.32 min. They now draw from the
same elution window as the analytes.

Fixing that exposed a second route to the same defect. The isocratic
case pins an early and a late eluter at fractions of the *record*, not
of the elution window; once the dead time pushes the window late, the
late pin can land below its lower bound, and the sort that follows then
makes that peak the first one. Both pins are clipped into the window.

An analyte has eluted when its trailing edge is back in the noise. A
record that stops before that is not a short run, it is an impossible
one: the trace ends part-way down a peak, which is the endpoint
mismatch BEADS' periodicity requirement forbids and which the parabola
correction cannot repair. 63 of 432 signals ended with more than 5% of
the tallest peak still standing, 12 of them with more than half. Those
records carried a 33.3% crash rate against 6.5% for the rest, and 21.4%
containment against 78.0%. The analyte set is now redrawn until the
component has decayed below the noise at the last sample, the elution
window being pulled in when redrawing alone does not get there so the
loop terminates.

The criterion is the noise the record will carry, which the generator
knows exactly, rather than a width rule: a peak has eluted when it is
no longer distinguishable from the noise. `native_peak_component` takes
it as an optional argument and is byte-identical without it, so the
`native` family is unchanged.

Measured over 900 draws across every peak case and baseline: no peak
before the dead time, and 0.6% of records ending mid-peak, all of them
shorter than 700 points. Those short records cannot be fixed by
redrawing: a 4.5 min absolute dead time in a 3.9 min record leaves no
room for a wide peak to elute, which is a conflict between the measured
length distribution and the measured dead time, not a generator bug.
Erb's baseline 3 is a pure sinusoid, ``10 + sin(x/50)``, 2 units peak to
peak, oscillating through several full cycles within one record. An HPLC
baseline does not do that, so scoring against it measures the method on
drift it will never meet.

It also fails to provide a target at all. On the wide and mixed peak
cases the cutoff minimising the RMSE against it recovers the baseline
*worse than a horizontal line*: median ratio 1.19 on ``multi_wide`` and
1.15 on ``multi_mixed``, so the "true optimum" recorded there is the
least bad of a grid of bad fits. 35 of 503 scored signals were being
measured against such a reference, and 25 of the 72 that select nothing.

Removing them changes the reading of the benchmark. Containment goes
from 57.3% to 61.6% before and 59.5% to 64.4% after the region-detection
work, and the improvement that work produces goes from indistinguishable
(gained 37, lost 23, p = 0.092) to detectable (gained 29, lost 14,
p = 0.032, McNemar exact).

Skipped inside the generation loop rather than dropped from the job
list, so the index that seeds every other signal is unchanged and
``erb0`` to ``erb2`` regenerate byte-identical.
The second group with enough members to stand on its own: the benchmark
generator, its diagnostic, its report battery, the width sweep, the
parked background loader and `synthetic_data.md`.

Renames only, but three things break silently on a move and are fixed
here: the usage lines and `synthetic_data.md` cross-references inside
the moved files; `backgrounds.py`'s repository root, which walked up two
directory levels and now needs three; and the two test modules, which
load the tools by explicit path since `tools/` is not a package.
Two figures per signal rather than only for the ones that select a
cutoff: the production autocorrelation plot with the true optimum drawn
on it, and the chromatogram carrying the known baseline, the selected
one and the one at the true optimum, over the corrected traces.

A signal that selects nothing produces no figure at all today, because
`_fcutoff` raises before it builds `plot_data`, so exactly the cases
worth looking at are the ones with nothing to look at. Here the
selection is rebuilt from the cached curve, and its absence is drawn
rather than raised.

The true optimum is read from the stored `diag/*__diag.npz` rather than
recomputed, so the error curve is not swept again; only the two final
BEADS corrections run per signal. It is not added to `plot.r2_plots`
itself, and should not be: production has no true optimum to draw.
Decades were the wrong yardstick and Emmanuel called it: on the signals
he judged acceptable by eye, `|d_decades|` has median 0.399 and p90
1.621, against median 0.338 and p90 0.531 on the ones he judged wrong.
The acceptable selections are FURTHER away in decades than the misses,
because a decade measures a distance along the parameter and not in the
corrected chromatogram. `penalty` orders them correctly but is a ratio,
so it diverges as `e_min` approaches zero while both fits sit far below
the noise: that is where the 2479 came from.

A literature check found no convention for dividing by the noise, and
no worst-case metric at all. Every source simply asserts its
normalization. So all of them are reported and the choice is left open:

  rms       plain RMSE against the known baseline, dividing only by the
            point count. Niezen, Schoenmakers & Pirok (2022) Eq. (15).
  max       maximum absolute departure. Used by no source; here because
            the rms averages a local defect over the record, so the same
            defect scores lower in a long run than in a short one.
  rms, max  the two above in units of the record's noise. Used by no
  in sigma  source either, and misleading when the signal towers over
            the noise: a visually perfect fit on a low-noise trace reads
            13.8 sigma.
  snr_db    10 log10(energy(true) / energy(true - fit)). Ning, Selesnick
            & Duval (2014) §5.1, the BEADS paper. Note the energy
            includes the baseline's offset, so a trace far from zero
            scores high whatever the fit does; their simulated baselines
            are near zero and ours are not.
  area_pct  percent error in recovered peak area against the known one.
            Navarro-Huerta et al. (2017) §3.6; Niezen §4.3.3 uses the
            same measure. Over the whole record rather than per peak,
            since the region boundaries would otherwise be our choice.

All are computed for the fit at `fc_best` and at the selected cutoff.
`d_decades` and `penalty` stay as columns because the report battery
reads them; neither should be quoted as accuracy.

The error curve is now reused across runs. It is 250 BEADS fits per
signal, unchanged by the trimming, the selection or the metrics, and it
was being recomputed every time although `diag/<stem>__diag.npz` already
stored it. It is keyed on the signal, the abscissa, the true baseline,
the grid, the output of `_relevant_regions` and `end_window`, so a
change to the region detection invalidates it -- which is the failure
the key exists to prevent, and it happened four times this week.
`--refresh-err` forces recomputation. Arrays are hashed at float32 as in
`baseline._r2_cache_key`, and the six-signal check on Rorqual confirms
both keys transfer: identical cutoffs to twelve digits under numpy 2.4.2
against 2.5.0.

`synth_figs` reads every figure from `diag_summary.csv` rather than
recomputing, so a plot cannot disagree with its own table, and fills the
optimum's figures itself for the signals that select nothing, whose
optimum does not depend on the selection.

The test that forbade a peak-area column is repointed: the area is now
reported, but `fc_best` must stay the minimum of the baseline error
curve, which is what the test pins.
Follows tools/slurm_script: work in $SLURM_TMPDIR, run the branch
checked out in the launch directory, move the results and the log back
at the end.

The resource form is `--ntasks=1 --cpus-per-task=16` rather than the
whole-node one, because the run is a single process with a worker pool
over signals: `--ntasks-per-node=16` would give sixteen tasks of one cpu
each and the pool would be confined to one. Sixteen of the 192 cores of
a Rorqual node is not a whole node either. `--mem-per-cpu=500M` against
a measured peak of 287M on the largest signal of the set.

The dataset is copied to node-local disk because the run writes one npz
per signal, and it carries its r2 cache and stored error curves, so a
re-run pays for neither the sweep nor the error curve.
The convention is the whole-node form with the per-cpu one commented
above, and it is also the better fit: `--mem` pools the memory across
the allocation, while `--mem-per-cpu` budgets each worker separately, so
one worker on the largest signal at 287M can be killed while the others
sit at a tenth of that.

The earlier justification for inverting it was wrong on the mechanism
too: the task count does not confine a process pool to one cpu, since
Slurm's cgroup grants the job every allocated cpu on the node.
`error_curve` fitted with `parabola_len=end_window(s)` and its docstring
claimed that mirrored the final correction of `auto_beads`. It does not.
`auto_beads` substitutes `end_window(s)` only when a caller passes
`parabola_len=None`, and no caller does: `weaselytics.py` passes nothing,
so the default 3 applies to the sweep and to the final correction alike.
That branch predates pybaselines exposing `parabola_len` on BEADS and is
a fallback, not the production setting.

So the answer key was the optimum of a slightly different problem from
the one production solves: BEADS anchors its endpoint parabola on 3
points at each edge in production, and on up to 20 in the benchmark.

Every stored error curve is invalidated, which the `err_key` handles
since the parabola length is part of it. The r2 cache is untouched: the
sweep always used 3.
Measured on the 432-signal set, scored by the RMS departure in mV
between the baseline at the true optimum and the known one.

The assumptions, with pointers: Ning et al. (2014) require the peaks and
their first M derivatives to be sparse (§3, §3.2), the baseline to be
low-pass with no parametric form and no rule tying the cutoff to peak
width (§3.1), the noise to be stationary white Gaussian (Eq. 7-8), and
positivity only as a preference encoded in the asymmetry parameter
(§3.4). Their validation reports no peak density or resolution at all.
Navarro-Huerta et al. (2017) list five limitations (§3.1 i-v) and peak
overlap is not among them.

**Both papers are silent on overlap, resolution and peak density**, so
what follows is a finding outside what either addressed rather than a
documented violation. Recorded as such.

Failure is not spread evenly. By peak case it runs from 0.0% on blanks
to 65.3% on multi_wide; by baseline, erb2 is 48.6% against 18% for the
other two; noise barely matters. The cross-tab shows blanks clean on
every baseline, so failure needs both a hard baseline and peaks.

Emmanuel's reading that the failures are simply not sparse enough holds
where it can be tested: over the 288 multi-peak signals, the failing
ones have a median adjacent-pair resolution of 0.13 against 0.34 for the
passing ones, and cover 19.9% of the record with peaks against 6.9%.
Within `multi_narrow x erb0` alone the split is 0.71 against 0.12. But
`multi_wide x erb0` inverts and `multi_mixed x erb0/erb1` show nothing,
so crowding is not the whole account and the section says so.

252 of 288 multi-peak signals have an adjacent pair below Rs = 1.5, so
the generator is producing mostly unresolved chromatograms.

Finally the analytic baselines are unlike the real ones: the 339 real
baselines have a median range of 0.279 mV against 5.00, 8.11 and 12.13
for erb0, erb1 and erb2. Only 5 of 339 exceed 5 mV. With the warning
not to read direction-change counts off a dual-axis plot, where a
0.05 mV wander looks like a 7.8 mV drift.
Brings the synthetic branch up to the current production code. The two
copies of the synthetic diagnostic are merged into
tools/synthetic/synth_diag.py and the flat tools/synth_diag.py goes.

Kept from plateau_segmentation: the run through auto_beads with the
sweep spread over the workers, --cache-dir, --diag-dir, --nshards and
--shard, the summary written beside the diagnostics, and in_candidates
with its candidates mask. Kept from synthetic_signals: the production
pick and the error at it, the instability-trim on/off pair with its
self-check, the reuse of the error curve, the five error measures and
the pilot options.

The synthetic tools are brought in line with production: the past-drop
exclusion keyword (exclude_collapse was renamed exclude_past_drop) and
its gate of 10; a test now ties the three copies of that gate to
auto_beads. synth_figs and synth_report take the diagnostics and cache
directories as options, since the summary moved. loop_bl.sbatch points
at the merged diagnostic, and synth_diag.sbatch is removed: loop_bl is
the one job script.
Add a section describing the LPYE dataset and its run conditions,
cite Bourret, Stevenson & Cote (2024), renumber the sections after
it, and refer to the dataset as LPYE where the text said real,
measured or calibrated.
Point the Milani et al. (2024) citation at §3.1.3, where the bounds
are stated. State plainly that an exported m of 1000 means the peak
is Gaussian, drop the per-peak counts from the bounds comment, and
remove the pointer from the pearson7 docstring to a section of
synthetic_data.md that no longer exists.
- Peak widths follow the plate model, sigma = tR / sqrt(N), with N
  drawn per signal.
- Peaks start after the injection artifact, end inside the run, and
  stay at Rs >= 1 from each other. The dead time moves earlier when
  they do not fit.
- Blanks hold only the injection artifact.
- Cases are named by peak count (blank, single, few, many,
  isocratic); Erb's baselines are named parabola, exp and step.
- Remove the pyb family and the old erb signal builders, and rename
  erb_native to erb.
- Signal names put a zero-padded replicate before the length, and
  synth_diag reads it there.
- The manifest records the analytes kept and dropped, Rs, coverage,
  N and the dead time.
- Noise levels are 0.02 and 0.05 mV; erb record lengths come from a
  skew-normal on log10(length).
The injection-artifact dip is now 0 to 2 times the spike as the trace
shows it, drawn as twice a Beta(1, 3) variate. `_dip_scale` solves for
the negative-lobe height after the two lobes overlap, and the drawn
ratio is recorded as `dip_ratio` in the truth entries. The absolute
`neg_abs` range is removed.

The native baseline wander correlation length moves from 0.3-0.8 min to
1.2-3.2 min. `--family` defaults to `erb`.

The docstrings and comments describe a general generator: ranges are
picked for this benchmark, and `native` and `lit` are deprecated.
Both review figures gain a third panel, the fitted baseline minus the
true one with a +-sigma noise band, which separates fits that differ by
less than the noise. Each fit has its own line style, the optimal one
wide and solid at the back, and the right column matches the height of
the r2 figure beside it. Counts from an older run are dropped from the
qo_all docstrings.
`--rate` sets the points per minute of the erb and native signals,
default 120. Record lengths are drawn as durations in minutes and the
artifact width is set in minutes, so another rate keeps every run as
long and samples it more or less densely. `lit` stays at 120. The rate
is recorded as `points_per_minute` in the truth metadata and the
manifest. At the default rate the output is identical to before.
tools/cutoff_overlay/cutoff_overlay.py draws one synthetic signal's
baseline at several cutoffs on one merged figure: the production r2
figure with a line per cutoff, and the baselines, corrected traces and
baseline errors overlaid. The cutoffs are spaced by how much the fitted
baseline moves. The signal's r2 cache entry is copied to a temporary
directory first, so the run's cache is never touched. An example figure
is kept in tools/cutoff_overlay/figures.
- Correct claims the code contradicts: the dimensionless-parameters
  closing line (tol depends on the sampling rate, the acetonitrile
  exception is in mV), the three breakpoints of the peaks_params
  ladder, and the past-drop fallback attributed to the gate.
- Correct the core principle: segment boundaries fall where the slope
  or the noise level changes, as Figure 4 shows.
- Drop statements resting on the old generator: the optimum sitting
  well inside the region, and the argmin test against the midpoint.
- Drop history, the pointers to the local audit, and the passages
  that repeat the parameter tables or justify a framing.
- Reflow the Parameters prose to one line per paragraph and apply
  the Canadian spellings.
- Carry over the corrections made to segmentation.md: segment
  boundaries fall where the slope or the noise level changes, and the
  L-curve literature is conceptually similar, not the source of the
  method.
- Date the right-edge anchoring evidence to the synthetic baselines of
  the time; the valley shape it cited is reversed on the new sweep.
- Cut the custom_bc note, which describes current production and
  repeats segmentation.md, with the two references only it cited.
- Drop the defensive asides, make the examples of baseline drift the
  same shape, and apply the Canadian spellings.
Only the erb family has built a dataset since the generator was
rebuilt. The two deprecated families go with their baselines, Gaussian
peaks, constants and the --family option; native_peak_component, which
erb shares, becomes peak_component. Job indices, and so the seeds of
the erb signals, are unchanged: a 60-signal erb set generated before
and after is byte-identical. The tests, WORKFLOW and synthetic_data.md
follow.
- Point to where Erb's baselines come from: make_data, posted in issue
  #4, instead of "his script".
- Name the paper behind qo_all's Eq. (11) (Navarro-Huerta et al. 2017)
  and drop a drifting line number.
- Drop emphasis capitals, a passage of history, and passages that
  repeat another docstring; restate the artifact-versus-baseline split
  positively.
Follow the pattern of the research_scripts jobs: the whole-node lines
are active and the per-task ones commented, CPUs are requested as
tasks rather than with --cpus-per-task, and the worker count is read
from SLURM_NTASKS, which both forms set. Same resources as before:
8 CPUs, 2G, 3 h.
data_aggregator.py, viz_data.py and stats_analysis.py get module and
numpy docstrings, a main(), and lose their commented-out code. Their
outputs are unchanged, checked on a synthetic set of peak CSVs.
viz_data.py turns its hard-coded selection into --mol and --columns.
chauvenets_criterions cites Taylor (1997) §6.2 and states the limits
of §6.3: small groups, several outliers, weighted statistics.
One-line docstrings for the nested helpers and the figure entry points
that had none, so every function the synthetic_signals branch adds or
touches carries one. Code unchanged.
The peak fits export a Pearson VII row since 134f568, and the
rejection and averaging already ran on it, but only the Gaussian and
Skew-Normal tables were printed.
@physicien
physicien merged commit 9e3f216 into development Nov 9, 2026
16 checks passed
@physicien
physicien deleted the synthetic_signals branch November 9, 2026 15:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant