Fix bciciv-2a scorer preprocessing contract - #57
Conversation
bondingelectron
left a comment
There was a problem hiding this comment.
Verified locally on the pinned dataset release — approving.
Local verification (reviewer-side)
Contract unit tests: 5/5 pass.
Real-data check, all 18 GDFs (9 subjects × T/E, extracted from the content-addressed tarballs recorded in data.lock): every file yields (288, 22, 512) float32; T sessions give exactly 72 trials per class; E sessions return labels=None; the runtime 2 s trial-start→cue offset validation passes on every file (500 samples @ 250 Hz); amplitudes land in a plausible 5.9–11.8 µV std range (A01T: 6.867 µV, matching the PR description).
End-to-end ablation (same tiny reference model, same fixed split, same training config, only preprocessing differs):
| Preprocessing | acc | κ |
|---|---|---|
v0.1 (anchor 768, volts) |
0.2500 | 0.0000 (collapsed to a single class) |
this PR (cue-anchored [0,4), µV) |
0.5903 | 0.4537 |
Direction and magnitude reproduce the issue #56 ablation (0.25 → 0.83 with their stronger model). The old pipeline indeed destroyed the learnable signal; the fixed one restores it.
Repo-level gates: tsc build, bp-bench validate bciciv-2a, docs smoke, and the 208-test Node suite all pass on this branch.
Notes for the record
- Label alignment is now strictly stronger than v0.1: labels come directly from the cue annotation identity (769→1 …) rather than a post-hoc window search, and X/y row order is guaranteed by construction from the same events array.
- The runtime validations (cue count = 288, 72/class, 2 s offset, no mixed T/E annotations) fail loudly, so any future data anomaly surfaces as a scorer error instead of silent misalignment.
- Merging this makes bciciv-2a v0.2; v0.1 scores are not comparable. No official v0.1 scores exist, so nothing is invalidated.
Thanks @GTC2333 for the precise report and @HaoxuanLiTHUAI for the clean fix.
Fixes #56
Summary
Validation
Score compatibility
This is a breaking scorer correction. Scores produced with bciciv-2a v0.1 should not be mixed with v0.2 scores because both the temporal window and numerical input scale change.