Skip to content

processSEG 0.1.0: the processing library - #1

Merged
xrfez merged 61 commits into
mainfrom
feature/processing-library
Nov 11, 2026
Merged

xrfez merged 61 commits into
mainfrom
feature/processing-library

Conversation

@xrfez

@xrfez xrfez commented Nov 11, 2026

Copy link
Copy Markdown
Owner

The first release of processSEG: seismic processing in Nim on top of parseSEG (>= 0.8.0), built for an interactive viewer. Kernels work in place on parseSEG's sample tensors, chunk by chunk, with no hidden allocation and no hidden threads.

Contents

  • FFT: pure-Nim mixed-radix 2/3/4/5 real FFT, vectorized across traces.
  • Filters: zero-phase FFT band-pass (cos² trapezoid corners, Butterworth magnitude, notches) with edge handling to each trace's live length; Butterworth IIR, zero or minimum phase.
  • Gain: fixed dB, per-trace and per-panel normalization, AGC, time-variant curves, statistics, spectra.
  • Pipelines: ops compiled once to a plan, applied in place or src → dst, with the margin a zoom window needs; a persistent ProcessPool runs a plan across cores.
  • Pre-stack: mutes, datum statics, LMO and NMO, velocity functions and fields (VELF, HANDVEL, CSV), CMP binning, stacking with per-sample fold, a brute stack written as SEG-Y.
  • QC: geometry checks, first-break pickers with confidence, a per-shot refraction fit that flags swapped receivers.
  • Docs: README, guide pages with tested examples, API reference, integration contract.

Testing

35 test files pass locally in debug, release, danger and ASan builds; property and fuzz tests; benchmarks in bench/.

🤖 Generated with Claude Code

xrfez and others added 30 commits November 9, 2026 23:17
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ChunkMeta gains live length, window origin, full geometry and signed
offsets; document the ORC/ptr threading rule, plan generations, edge
handling, SEG-D Rev 1/2 geometry via SPS, and a fuller P2 signature.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
processSEG.nimble requires the parseSEG branch processseg-integration;
nim parseSEGTest runs parseSEG's own suite in deps/parseSEG, and
nim parseSEGBranch re-attaches the checkout after atlas install.
proposedChanges.md is removed: C1-C8 shipped in parseSEG 0.7.0, and the
follow-ups are now processSEG tasks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- src/processseg.nim -> src/processSEG.nim (D9), build/test/docs tasks updated.
- `nim fmt` formats every .nim file under src/, tests/ and bench/.
- CI fmt check walks subdirectories (globstar) and reports every diff.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- tests/tester.nim compiles and runs each t*.nim in its own process, passing
  its arguments through as build flags (debug, -d:release, -d:danger,
  -d:addressSanitizer), and fails listing every failing file.
- tests/config.nims holds the shared paths and the ASan switches
  (-d:useMalloc, -d:noSignalHandler).
- Placeholder Context API removed; tumbrella.nim is the smoke test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Signals added in place into var openArray[T]: sinusoid, spike, Ricker,
  DC, linear drift, seeded Gaussian noise.
- SynthGather[T]: parseSEG TraceKeys with geometry (signed offset, source/
  receiver/midpoint x, elevations, t0Ms, intervalUs, live status), per-row
  live lengths, a zero-padded [rows, cols] tensor, and a TraceRows view.
- Shot (split-spread, end-on) and CMP gathers; hyperbolic and linear events
  with optional per-trace shifts; elevation statics.
- tests/tsynth.nim self-tests every generator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- bench/benchutil.nim: warm-up + timed samples (each >= 20 ms), median and
  minimum per iteration, GB/s; prints CPU, OS, Nim, C compiler and the build
  flags with every run.
- bench/bmemory.nim: copy and in-place scale of a 2000 x 2001 float32 panel.
- `nim bench` runs bench/b*.nim in a danger build with recorded flags;
  results recorded in bench/README.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…data model

- ChunkMeta is a zero-copy wrapper over parseSEG's TraceRows (D12): per-row
  values from the view, liveness rule (tsLive/tsUnknown, not failed), and
  the three view lifetimes.
- Geometry from TraceKeys (inline/crossline from the layout), traceStatics,
  and keysRevision for anything derived from keys.
- Threads: the three ways to run, with the persistent public ProcessPool (D6);
  batch jobs in the prefetcher transform hook (D13); pure-Nim FFT (D2).
- "Proposed parseSEG changes" becomes "parseSEG 0.7.0 and the patch branch"
  (P1-P4 landed; F1-F3 on processseg-integration). Viewer checklist uses
  c.rows, uniform = true and keysRevision. Pool open question closed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
parseSEG 6780373 adds keysAt(rows, row), the non-raising form of traceKeys
for {.raises: [].} kernels and ChunkTransforms (tested there; parseSEG's
whole suite passes through `nim parseSEGTest`).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- integration doc: derived offsets (SEG-D Rev 3, SEG-2, survey joins) are
  unsigned distances, so split-spread side comes from the coordinates;
  kernels read keys with keysAt (F1 marked done); the adapter's interval
  check is a pass over the rows, not O(1).
- synth: unsignOffsets() models derived offsets; the rows() doc forbids
  temporaries.
- tester: runs the compiler with startProcess (no shell quoting).
- tests/config.nims: ASan flags for gcc/clang or MSVC; MinGW is an error.
- bench: the CPU target comes from config.nims (-d:cpuTarget) instead of
  being re-derived; clock read once per doubling batch; a compiler memory
  barrier after every body run (without it gcc merged batched in-place runs
  and reported 2x the real bandwidth).
- atlas.lock: parseSEG with the review's test fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…or tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nting and reject tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NaN/Inf rows left out of averaged spectra; always-on FFT buffer length
checks; short windows tapered and scaled per row; default-plan assertion;
workspaceSize(batched); more lane, NaN and float64 tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed and benched

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…otch, impulse length

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…; warped FD Butterworth

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dget

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… margin, zeroDc)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ip by histogram

TraceStats/traceStatsInto, PanelStats/panelStats (65536-bin float32-bit histogram,
0.4 % relative), exact percentileAbs; Workspace gains a uint32 region. Bench in
bench/bgain.nim. Also applies nim fmt to the Phase 3 filter files (layout only).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scaleDb; NormalizePlan (normMax/normRms/normPercentile, perTrace/perPanel);
AgcPlan (agcRms/agcMeanAbs, centred window, real counts at the ends,
epsilon relative to the trace RMS, live length, kept gains + removeGain).
Window sums from block suffix + prefix sums: no subtraction, exact to a few
ulps at any dynamic range, O(n) whatever the window. tvectorized groups a
kernel's GCC parts and clones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GainCurve (knotGain linear in dB and flat outside, powerGain, exponentialGain),
GainCurvePlan tabulated on the dt grid; applyGainCurve multiplies each live row
by the table at its own times (t0Ms, firstSample), linear between grid points
for a fractional shift, the curve itself outside the table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
xrfez and others added 29 commits November 10, 2026 20:43
…n in CI

Workers start once, sleep on a condition variable and are joined by
=destroy; =copy and =dup are errors. apply validates on the caller's thread,
splits the rows into one range per worker, and runs a per-panel factor on
worker 0 between parts. Identical to the single-thread apply for 1-8
workers; ASan and TSan clean. Bench: 6-12 us per job, 7.4x at 8 workers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…out)

lmoTime/nmoTime, Moveout (linearMoveout, hyperbolicMoveout), and
moveoutTimesInto (ms from a TraceRows view) / moveoutColumnsInto (block
columns from a ChunkMeta). Offsets keep their sign in the keys; the formulas
use |x|. NaN offsets or t0 give NaN.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… real API

tests/utils/viewersim.nim acts out a viewer: uniform prefetch into a raw
cache, ProcessPool into processed buffers tagged by generation, a checking
draw, slider edits, zoom windows with requiredMargin, moveout ticks tagged
with keysRevision and made stale by applySurvey. tviewerloop drives it on
synthetic data and parseSEG's demo files: stale buffers always reprocessed,
0 allocations per steady-state frame, ASan clean.

The doc now names the real API (ops, Pipeline, apply, pool, moveout) and
says what the margin promises: relative to the strongest nearby signal; AGC
after a filter lifts the leak in quiet windows; whole-trace ops use the
window's statistics.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… pipeline checks)

Two fresh reviews: a docs-only API review that wrote its own viewer loop
(kept as tests/tapireview.nim) and a diff review.

- Processed buffers are stale when the plan's generation or the file's
  keysRevision changes (mutes read offsets, gain curves t0, every op the
  status); viewersim tags both and gains refresh(f); new viewer-loop test.
- pipeline: factors checked against the sample type; checkApply(src, dst)
  rejects partial overlaps; runPart/panelFactorOf assert their preconditions
  (kept public for viewers with their own threads); muteTopOp/muteBottomOp;
  requiredMargin(flow, dtMs) without compiling; decodeWindow.
- pool: workers of a moved-from pool is 0, ping asserts, a failed
  createThread joins only started threads.
- moveout: hyperbolic t0Ms >= 0; moveoutTimesInto(meta, ...).
- The umbrella re-exports parseSEG. Integration doc: cached-chunk lifetimes,
  what allocates, zoom windows on the traces' peak, batch marked planned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
import processSEG no longer re-exports parseSEG: the two packages stay
separate, and callers import both (import parseSEG, processSEG). parseSEG
is on every build's path through Atlas's nim.cfg.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
models/pool_handoff.nif: two workers, up to two jobs, destroy at any idle
point, spurious wake-ups. Every job runs exactly once per worker before the
caller returns, and no wake-up is lost. Base passes (235 states), the
witness is reachable, and all five mutants fail (no recheck after a wake,
wait ignoring stop, caller waiting once, no done signal, signal instead of
broadcast).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…L/CSV picks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ifts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rses, zero knot, negative gradient)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ctional shifts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…stretch mute

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r inline/crossline

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…malization, merge

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… NMO, per-worker stacks, progress and cancel

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and intervals skipped, live-sized stack, worker early stop, overflow guard, rcvDepth, header statics column)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…: traces from keys)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… offsets, midpoint, per-trace flags

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ppens; minimum-phase band-pass, LMO-guided window, SNR confidence

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… residuals, neighbour cross-correlation, symmetry; swapped receivers flagged

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… t0 picks, degenerate and long-spread fits, midpoint flag out of badGeometry, writeStack sorting and docs, detection limits)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…range values

tests/tproperty.nim: random pipelines on random gathers (sizes, dt, live
lengths, dead/auxiliary/failed rows, 1e-6..1e6 amplitudes): compile raises
only ValueError, apply is finite, leaves non-live samples alone and equals
src -> dst; a NaN/Inf sample reaches no other row; FFT and IIR filter
linearity; AGC and normalize scale invariance; zero in, zero out; shifts,
NMO and first-break picks with extreme parameters; velocity parsers on
mutated text.

tests/fuzz: libFuzzer + ASan + UBSan targets for the velocity parsers and
for pipelines built from bytes (raw float parameters and samples), with
seeds and a `nim fuzz` task.

UBSan findings fixed: the impulse-length grid of a near-zero ramp (an
18 s run on a garbage FFT size), the NMO taper start for a huge taperMs,
and first-break and shot-fit correlation windows of huge finite lengths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hmark)

- first breaks: a band-passed plan filters fftLanes rows at a time through
  the IIR lanes kernel into workspace copies (was one scalar trace at a
  time): 5-60 Hz picking 100/86/91 -> 44/31/35 ms per 2000-trace panel;
  unfiltered plans pick from the rows in place;
- NMO: the RMS velocity column is reused across rows with the same start
  time (and location), and the input columns are computed by a vectorized
  block loop (moveoutCols, -fno-math-errno in the nmo module): linear
  36 -> 17.5 ms, cubic 55 -> 25 ms;
- shot fit: l1Fit normal equations in local sums (vectorized), golden
  section reuses one point per step: 38 -> 21 ms on 2000 traces;
- tvectorized checks moveoutCols and l1Fit; bench/README records the run.

nim test and nim testRelease pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- README: overview, a compiled quick start, links;
- docs/guide/: getting started, pipelines (zooms, the pool, own threads),
  gain, velocity and stack, QC; every example is a program;
- docs/api.md: conventions, then every exported name by module;
- tests/tdocs.nim: compiles and runs the guide and README examples in
  one build, and checks api.md names all 370 exported names (jsondoc);
- docs/integration.md: status for the release candidate, QC workspace,
  and the zoom margin's measured worst case (2.5 %, not 1 %);
- doc comments: nim docs is warning-free.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Defects:
- mute kernels: opposite infinite bounds made a NaN ramp bound and the
  boundChecks-off loops ran from int.low (heap overflow under ASan); an
  infinite taper is now no taper, the column helpers are NaN-safe, and a
  subnormal dtMs is refused by initChunkMeta;
- intervalVelocity with a first time of 0 changed the first layer (a knot
  at 0 makes Vrms linear from it): the zero-thickness layer is dropped;
- float32 overflow to Inf turned zeros into NaN: gain-curve tables and
  scaleDb refuse such gains, block scale refuses a factor Inf in T, and
  off-table curve samples keep zeros zero;
- FFT/IIR filter and averaged-spectrum block forms now refuse rows whose
  interval is not the plan's (they filtered at the wrong frequencies);
- first breaks: an infinite t0 is no time axis (was a NaN-to-int);
- shot fit: golden-section refinement kept within the candidates' range;
  fitShot no longer infers raises: [Exception] (forward declaration);
- rowBlock(data, rows, cols) refuses an overflowing size;
  decodeWindow takes a negative margin as 0 and cannot overflow;
- velocity parsers accept a UTF-8 byte-order mark.

API: pick(plan, x, ws, offset = NaN, t0Ms = 0) (ws no longer after the
defaults). Docs: zero-mean claim of withZeroDc, asymmetryMs sign, lag
choice, MER weight, pool exceptions, perPanel in bruteStack is per chunk,
the zero-offset tolerance; doc comments from the docs pass. Tests for each
fix, an NMO reference test, allocation-free filtered pickInto.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The patch branch is merged upstream and released as parseSEG 0.8.0
(v0.8.0, 7be5c47); atlas.lock pins it, so CI's atlas rep resolves a
pushed commit again. Integration doc: parseSEG section for 0.8.0, and
the zoom margin's worst case as an open question for the viewer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nimpretty's default 80 columns rewrapped the 100-column sources and, on one
wrap, joined `and p.lagToleranceMs` into `andp.lagToleranceMs`. CI and
`nim fmt` now pass --maxLineLen:100; the remaining diffs are comment
alignment. tfilterapply's spec list is hoisted out of the for header (a
comment inside it crashed nimpretty), and two long test lines are rewrapped
by hand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the CI runner (gcc 15, x86-64-v3) the release build differs from gcc 12
in the last bits: tpipeline's plan-vs-ops check asked for bit equality
(off by 6.7e-16), and tproperty's NaN-containment check compared a
near-zero sample (2e-11) relative to itself. Both tolerances are now
relative to the data's scale (the output's maximum, the row's maximum).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@xrfez
xrfez merged commit 6903084 into main Nov 11, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant