diff --git a/AGENTS.md b/AGENTS.md index fca448ce9..c7a4f66b9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -63,6 +63,7 @@ This section applies to any agent (Claude, Codex, Cursor, opencode, ...) working ## Architecture references - `ARCHITECTURE.md` - `docs/engineering/acceptance-criteria.md` +- `docs/doctoring/real-audio-accuracy-acceptance.md` - `docs/engineering/harness-engineering.md` - `docs/workflow/one-day-delivery-plan.md` - `docs/workflow/pr-continuity.md` @@ -85,6 +86,7 @@ This section applies to any agent (Claude, Codex, Cursor, opencode, ...) working - Prefer practical, friendly, rehearsal-first wording over academic or authority-heavy language. - Do not reduce the product to a chord analyzer when form, timing, player coordination, simplification, and setup cues are the real rehearsal blockers. - Do not frame usability as a reason to accept weak analysis quality; BandScope should aim for both easy use and high accuracy. +- Do not claim harmony or tempo accuracy from mocked feature matrices. Tier 1 decoded-PCM fixtures and metrics live in `docs/doctoring/real-audio-accuracy-acceptance.md`. ## Safety - Do not add network-dependent runtime paths for local analysis. diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 3302a6fc3..8ee276119 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1,6 +1,6 @@ # ARCHITECTURE.md -Last updated: 2026-03-11 +Last updated: 2026-08-16 ## Brand source @@ -127,3 +127,4 @@ Last updated: 2026-03-11 - Supply-chain docs, workflow pinning, and lockfile verification are part of the default quickcheck path so dependency drift is caught early. - Quickcheck and CI are expected to verify dependency review, audit, supplemental inventory, and SBOM baseline presence as part of bootstrap. - Cross-platform build workflow presence and trigger coverage are part of the default supply-chain verification path. +- Real-audio accuracy acceptance is a separate buyer-facing gate from unit coverage. Tier 1 decoded-PCM fixtures, duration-weighted chord recall, and tempo Acc1 live in `docs/doctoring/real-audio-accuracy-acceptance.md` and `bandscope_analysis.accuracy`. diff --git a/CHANGELOG.md b/CHANGELOG.md index eea696893..6911ebcbe 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,24 @@ ### Added +- Add Tier 1 real-audio accuracy acceptance: a decoded C major WAV must recover `C`, and a 120 BPM click WAV must pass tempo Acc1, with checksum-fail-closed reports. + +### Fixed + +- Reject click-track accuracy fixtures whose beat interval or click pulse cannot be represented by at least one sample, preventing aliased or all-zero tempo evidence from entering acceptance runs. +- Reject fixture timings whose derived sample count overflows or quantizes below one sample, and reject finite BPM inputs whose derived beat interval becomes non-finite, before they can drive real-audio accuracy allocation or loop authority. +- Reject malformed decoded-PCM accuracy evidence before chord recognition: the C-major acceptance buffer must be a non-empty, finite, floating-point, one-dimensional mono array and sample rates must be finite, positive, and non-Boolean. +- Reject accuracy-report metric integers whose conversion overflows floating-point evidence instead of letting an `OverflowError` escape the manifest validation boundary. +- Reject Boolean fixture duration, tempo, and sample-rate inputs so Python `True`/`False` values cannot become numeric real-audio acceptance authority or a 1 Hz WAV contract. +- Reject Boolean chord timing and tempo metric inputs so `True`/`False` cannot masquerade as numeric MIR acceptance evidence through Python's `bool`-as-`int` semantics. +- Reject blank or edge-whitespace-padded accuracy-report case IDs, metric names, and truth labels so portable acceptance evidence preserves exact registered identities. +- Reject empty or reversed chord-estimate intervals before duration-weighted recall so malformed timing evidence cannot be silently ignored by an accuracy acceptance score. +- Fail closed when accuracy-report provenance cannot resolve a non-empty product `VERSION`, instead of publishing an `unknown` engine version as valid evidence. +- Reject non-finite chord annotation and estimate timings before duration-weighted recall so NaN/Inf evidence cannot fabricate covered duration. +- Reject non-finite tempo estimates, ground-truth BPM values, and Acc1 tolerances as invalid accuracy evidence instead of recording them as ordinary misses. +- Union overlapping matching chord-estimate intervals before duration-weighted recall so acceptance scores cannot double-count annotated time or exceed 100%. +- Reject malformed accuracy-report provenance, including non-hex SHA-256 text and non-finite metric values, before acceptance evidence is consumed. +- Score the C major acceptance case from checksummed on-disk WAV bytes instead of the pre-write in-memory triad. - Display the analyzed song tempo (BPM) as a badge in the rehearsal workspace. - 각 합주 역할(Role)별 개인 연습 진행도를 0~100% 범위로 기록 및 시각화할 수 있는 연습 진척도(`practiceProgress`) 트래커 기능 추가. UI 컨트롤(슬라이더 및 +/- 버튼)과 한/영 다국어 지원 포함. diff --git a/CLAUDE.md b/CLAUDE.md index 82c2c704a..62ea64742 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -73,3 +73,4 @@ Supporting packages: - Lockfiles (`package-lock.json`, `uv.lock`, `Cargo.lock`) are committed and must stay in sync; GitHub Actions are SHA-pinned. Adding a direct dependency requires the admission rationale defined in `AGENTS.md` and `docs/security/dependency-policy.md`. - CI beyond quickcheck: `gate / ci / rust-check` (Tauri cargo check on macOS) and `build-baseline` Windows/macOS amd64+arm64 native builds are merge gates, alongside CodeQL, dependency-review, sbom, bandit, trivy, secret-scan, and security-audit workflows. Do not weaken or skip them. - Version metadata lives in `VERSION`, the root `package.json`, and `CHANGELOG.md`; release flow is tag-driven (see `docs/operations/deploy-runbook.md`). +- Real-audio accuracy acceptance (decoded PCM, not mocked chroma) lives in `docs/doctoring/real-audio-accuracy-acceptance.md` and `bandscope_analysis.accuracy`. diff --git a/docs/doctoring/real-audio-accuracy-acceptance.md b/docs/doctoring/real-audio-accuracy-acceptance.md new file mode 100644 index 000000000..37e854adf --- /dev/null +++ b/docs/doctoring/real-audio-accuracy-acceptance.md @@ -0,0 +1,140 @@ +# Real-audio accuracy acceptance + +Next action: before claiming a rehearsal result is accurate, run the Tier 1 +decoded-PCM cases. A green unit suite that never opens a WAV is not buyer +evidence. + +```bash +uv run --project services/analysis-engine pytest \ + services/analysis-engine/tests/test_accuracy_acceptance.py \ + services/analysis-engine/tests/test_accuracy_boolean_fixture_inputs.py \ + services/analysis-engine/tests/test_accuracy_pcm_input_contract.py \ + services/analysis-engine/tests/test_accuracy_manifest_version.py \ + services/analysis-engine/tests/test_accuracy_metric_interval_contract.py \ + services/analysis-engine/tests/test_accuracy_metric_overflow_contract.py +``` + +## Why this lock exists + +BandScope sells rehearsal guidance. A buyer cannot distinguish “the job +succeeded” from “the engine heard the chord and tempo that are in the file” +until decoded PCM is scored against a known label (Raffel et al., 2014). + +This layer is Tier 1 of issue #770: tiny, license-clean, deterministic WAV +fixtures generated in process. It does not claim genre coverage, perceptual +stem quality, or private-corpus readiness. + +## Held cases + +- `c-major-triad`: three seconds of C4+E4+G4 written to WAV, checksummed, + decoded from those bytes, and scored with duration-weighted chord recall. + Pass when recall of `C` is at least `0.70`. That floor is a BandScope Tier 1 + tolerance. Matching estimate intervals are clipped to the annotation window + and unioned before duration is accumulated, so overlapping or duplicate + estimates cannot count the same annotated time twice or produce recall above + `1.0`. Annotation and estimate times must be finite non-Boolean numbers before + clipping, and each estimate interval must have a strictly increasing + start/end pair; Boolean, NaN, infinite, empty, or reversed timing is invalid + acceptance evidence and fails closed instead of being silently ignored or + allowed to fabricate covered duration. File-backed WAV evaluation accepts an + ordinary mono or multichannel WAV and deterministically downmixes decoded + multichannel samples to mono before the production recognizer runs. The direct + `evaluate_c_major_pcm` boundary is narrower: callers must provide a non-empty, + one-dimensional, finite floating-point mono PCM array and a finite positive + non-Boolean sample rate. Unresolved multichannel arrays passed directly, + integer arrays, NaN/infinite samples, or invalid rate evidence are rejected + rather than coerced at that direct evaluator boundary. The metric family is + WCSR/CSR (Odekerken et al., 2021; Raffel et al., 2014). +- `click-120-bpm`: eight seconds of 120 BPM clicks decoded by + `TemporalAnalyzer`. Pass when estimated tempo satisfies Acc1 at 4% + (Schreiber & Müller, 2020). Acc1 does not credit half-time or double-time. + Estimated BPM, true BPM, and the tolerance must all be finite non-Boolean + numbers; Boolean or non-finite metric inputs are invalid acceptance evidence + and fail closed rather than being recorded as an ordinary miss. This matters + in Python because `bool` is an integer subtype and would otherwise satisfy + ordinary numeric comparisons. +- Fixture generation and WAV serialization accept only finite positive + non-Boolean duration, BPM, and sample-rate evidence. In Python, `True` would + otherwise act as numeric `1`, which could create a one-second fixture, a + one-BPM click contract, or a one-Hz WAV while still looking type-compatible at + runtime. Derived fixture timing must also remain representable: the computed + fixture sample count, beat interval, and click-pulse width must each be finite + and must resolve to at least one sample at the requested rate. This prevents a + nominally valid high BPM from placing multiple beats onto the same sample and + prevents a low sample rate from producing an all-zero “click” fixture. Inputs + that overflow or undersample those derived quantities fail closed before + allocation, loop construction, or file serialization. +- Checksum mismatch fails closed on both file evaluators. Do not score a + tampered file as a pass. +- Machine-readable case reports are accepted only when the registered + provenance fields are present and typed, `audio_sha256` is exactly 64 + hexadecimal characters, and `metric_value` is a finite numeric value. + Boolean, NaN, infinity, overflow-to-float, malformed digest, or missing-field + evidence fails closed rather than becoming a portable acceptance record. +- When a caller does not provide an explicit engine version, report creation + resolves the repository product `VERSION`. Missing or empty `VERSION` + provenance fails closed; `unknown` is not accepted as a substitute for the + exact engine version required by the accuracy evidence contract. + +## Claim boundary + +A passing case supports only the registered fixture, metric, engine version, +and tolerance. It does not establish universal musical correctness. + +Keys-left, keys-right, and acoustic-guitar roles still carry arrangement +defaults (`C#`, `Emaj7`, `Eb`). Lead vocal harmony is the role that currently +reflects the `other` stem recognizer. Do not treat those defaults as +measured accuracy. + +## References + +Odekerken, D., Koops, H. V., & Volk, A. (2021). Improving audio chord +estimation by alignment and integration of crowd-sourced symbolic music. +*Transactions of the International Society for Music Information Retrieval, +4*(1), 141–155. https://doi.org/10.5334/tismir.81 + +Raffel, C., McFee, B., Humphrey, E. J., Salamon, J., Nieto, O., Liang, D., & +Ellis, D. P. W. (2014). MIR_EVAL: A transparent implementation of common MIR +metrics. In *Proceedings of the 15th International Society for Music +Information Retrieval Conference* (pp. 367–372). + +Schreiber, H., & Müller, M. (2020). Music tempo estimation: Are we done yet? +*Transactions of the International Society for Music Information Retrieval, +3*(1), 111–125. https://doi.org/10.5334/tismir.43 + +## Security Notes + +- Attack surface: generated WAV bytes, fixture duration/BPM/sample-rate inputs, + decoded PCM shape/dtype/finiteness/sample-rate evidence, SHA-256 digests, + recognizer segment timings, tempo estimates, product-version provenance, and + parsed case-report mappings passed into the accuracy acceptance path. +- Trust boundary: untrusted audio, runtime numeric evidence, decoded buffer + structure, recognizer output, and manifests; trusted repo-controlled fixture + definitions, true labels, metric definitions, registered floors, and the + repository product `VERSION`. +- Mitigations: no network, no shell, checksum fail-closed before C-major + decode and before tempo scoring, deterministic file-decoder downmix for + ordinary multichannel WAV input, non-empty finite floating-point + one-dimensional mono PCM admission at the direct C-major evaluator boundary, + finite positive non-Boolean decoded sample-rate evidence, overlap-safe chord + duration, finite non-Boolean annotation/estimate timing, strictly increasing + estimate intervals, finite non-Boolean tempo metric inputs, finite positive + non-Boolean fixture duration/BPM/sample-rate inputs, finite derived fixture + sample count, beat interval, and click-pulse width with a one-sample minimum, + strict SHA-256 syntax, finite-only report metric values including overflow + rejection, exact non-empty product-version provenance, bounded fixture + durations, and no copyrighted commercial recordings. Fixture paths are pytest + temp files; reports store SHA-256 and labels, not waveform bytes. +- Test points: deterministic digest, C major recall after file decode, ordinary + stereo WAV downmix at the file-decoder boundary, direct decoded-PCM + empty/non-floating/non-finite/non-mono rejection and invalid sample-rate + rejection, overlapping matching intervals do not double-count annotation + duration, non-finite and Boolean chord annotation/estimate timing rejection, + empty/reversed estimate interval rejection, silence-on-disk vs in-memory + triad, 120 BPM Acc1, non-finite and Boolean tempo estimate / truth / tolerance + rejection, non-finite and Boolean fixture generation/WAV sample-rate + rejection, derived sample-count overflow and zero-sample rejection, beat- + interval overflow and sub-one-sample rejection, sub-one-sample click-pulse + rejection, checksum mismatch through both file evaluators, malformed/non-hex + manifest provenance, NaN/infinity/overflow report rejection, missing/empty + product `VERSION` rejection, and silence must not pass as C major. diff --git a/docs/plans/2026-08-16-real-audio-accuracy-acceptance.md b/docs/plans/2026-08-16-real-audio-accuracy-acceptance.md new file mode 100644 index 000000000..94335b2ff --- /dev/null +++ b/docs/plans/2026-08-16-real-audio-accuracy-acceptance.md @@ -0,0 +1,57 @@ +# Real-audio accuracy acceptance (Tier 1) + +**Goal:** Give BandScope a buyer-readable accuracy gate: decoded PCM from a +known WAV must recover the expected chord or tempo, with a versioned report. + +**Architecture:** A new `bandscope_analysis.accuracy` package generates +license-clean fixtures, scores them with documented MIR metrics, and emits a +strict case-report schema. Tests write real WAV files and call production +`ChordRecognizer` and `TemporalAnalyzer` helpers. Stem separation stays out +of this slice. + +**Tech Stack:** NumPy, soundfile, librosa beat tracking, pytest. + +## Security Notes + +### Attack surface + +Generated WAV files, SHA-256 digests, decoded PCM arrays, and JSON-shaped +accuracy reports. + +### Trust boundary + +Untrusted: any on-disk fixture bytes and any parsed report mapping. +Trusted: in-repo generators, metric definitions, and registered floors. + +### Mitigations + +- No network and no shell interpolation. +- Checksum mismatch raises before C-major decode and before tempo scoring. +- Manifest parsing fails closed on missing or mistyped fields. +- Fixtures are short, synthetic, and license-clean. + +### Test points + +- Deterministic C major digest +- Duration-weighted C recall after file decode +- Silence on disk fails even when a C major array exists in memory +- 120 BPM Acc1 after file decode +- Checksum mismatch through both file evaluators +- Malformed report rejection +- Silence must not pass as C major + +### Realistic threats + +A tampered fixture or a missing digest could be scored as a pass and then +cited as release evidence. + +### Logging and privacy + +`TemporalAnalyzer` may log the fixture path. Tests use pytest `tmp_path` +names only. Case reports keep SHA-256, metric, and labels; they do not store +PCM. + +### Remaining risk + +Tier 1 does not cover Demucs stems, private commercial recordings, or +CPU/GPU numeric parity. Those remain later #770 tiers. diff --git a/services/analysis-engine/src/bandscope_analysis/accuracy/__init__.py b/services/analysis-engine/src/bandscope_analysis/accuracy/__init__.py new file mode 100644 index 000000000..03b4f5b57 --- /dev/null +++ b/services/analysis-engine/src/bandscope_analysis/accuracy/__init__.py @@ -0,0 +1,45 @@ +"""Real-audio accuracy acceptance helpers for BandScope rehearsal analysis. + +Next action: run the Tier 1 fixture tests before claiming a harmony or tempo +result is accurate. These helpers score decoded PCM against known labels. They +do not replace Demucs stem separation or private-corpus benchmarks. +""" + +from .evaluate import evaluate_c_major_file, evaluate_c_major_pcm, evaluate_click_tempo_file +from .fixtures import ( + C_MAJOR_LABEL, + DEFAULT_CLICK_BPM, + DEFAULT_SAMPLE_RATE, + assert_fixture_checksum, + read_pcm_wav, + render_c_major_triad, + render_click_track, + write_pcm_wav, +) +from .manifest import ( + AccuracyCaseReport, + build_case_report, + parse_case_report, + read_product_version, +) +from .metrics import duration_weighted_chord_recall, tempo_acc1 + +__all__ = [ + "AccuracyCaseReport", + "C_MAJOR_LABEL", + "DEFAULT_CLICK_BPM", + "DEFAULT_SAMPLE_RATE", + "assert_fixture_checksum", + "build_case_report", + "duration_weighted_chord_recall", + "evaluate_c_major_file", + "evaluate_c_major_pcm", + "evaluate_click_tempo_file", + "parse_case_report", + "read_pcm_wav", + "read_product_version", + "render_c_major_triad", + "render_click_track", + "tempo_acc1", + "write_pcm_wav", +] diff --git a/services/analysis-engine/src/bandscope_analysis/accuracy/evaluate.py b/services/analysis-engine/src/bandscope_analysis/accuracy/evaluate.py new file mode 100644 index 000000000..8c5a41eb0 --- /dev/null +++ b/services/analysis-engine/src/bandscope_analysis/accuracy/evaluate.py @@ -0,0 +1,126 @@ +"""Score decoded PCM fixtures through production analysis helpers.""" + +from __future__ import annotations + +from contextlib import contextmanager +from pathlib import Path +from tempfile import TemporaryDirectory +from typing import Iterator + +import numpy as np +from numpy.typing import NDArray + +from bandscope_analysis.accuracy.fixtures import ( + C_MAJOR_LABEL, + DEFAULT_CLICK_BPM, + DEFAULT_SAMPLE_RATE, + read_pcm_wav, + read_verified_fixture_bytes, +) +from bandscope_analysis.accuracy.manifest import AccuracyCaseReport, build_case_report +from bandscope_analysis.accuracy.metrics import duration_weighted_chord_recall, tempo_acc1 +from bandscope_analysis.chords.chord_recognizer import ChordRecognizer +from bandscope_analysis.temporal.analyzer import TemporalAnalyzer + +C_MAJOR_RECALL_FLOOR = 0.70 + + +@contextmanager +def _verified_fixture_path(audio_path: Path, expected_sha256: str) -> Iterator[Path]: + """Stage the exact checksum-verified bytes at an app-owned temporary path.""" + payload = read_verified_fixture_bytes(audio_path, expected_sha256) + with TemporaryDirectory(prefix="bandscope-accuracy-") as temp_dir: + verified_path = Path(temp_dir) / "verified.wav" + verified_path.write_bytes(payload) + yield verified_path + + +def evaluate_c_major_pcm( + audio: NDArray[np.floating], + sample_rate: int = DEFAULT_SAMPLE_RATE, + audio_sha256: str = "0" * 64, +) -> AccuracyCaseReport: + """Score a C major triad through the production chord recognizer. + + Args: + audio: Decoded one-dimensional non-empty finite floating-point mono PCM. + Do not pass a chroma matrix, integer payload, or unresolved + multichannel buffer. + sample_rate: Finite positive non-Boolean sample rate of ``audio``. + audio_sha256: Digest of the on-disk fixture that produced ``audio``. + + Returns: + A case report whose metric is duration-weighted recall of ``C``. + + Raises: + ValueError: If PCM is empty, non-floating, non-finite, not mono, or the + sample-rate evidence is Boolean, non-finite, or non-positive. + """ + if isinstance(sample_rate, bool) or not np.isfinite(sample_rate) or sample_rate <= 0: + raise ValueError("sample_rate must be a finite positive non-Boolean number") + if ( + not isinstance(audio, np.ndarray) + or audio.ndim != 1 + or audio.size == 0 + or not np.issubdtype(audio.dtype, np.floating) + or not np.isfinite(audio).all() + ): + raise ValueError("audio must be non-empty finite floating-point mono PCM") + samples = np.asarray(audio, dtype=np.float32) + + recognizer = ChordRecognizer() + tracked = recognizer.recognize(samples, sr=sample_rate) + segments = [(item["start_time"], item["end_time"], item["chord"]) for item in tracked] + duration = float(len(samples) / sample_rate) + recall = duration_weighted_chord_recall(segments, C_MAJOR_LABEL, 0.0, duration) + return build_case_report( + case_id="c-major-triad", + audio_sha256=audio_sha256, + metric_name="duration_weighted_chord_recall", + metric_value=recall, + passed=recall >= C_MAJOR_RECALL_FLOOR, + true_label=C_MAJOR_LABEL, + ) + + +def evaluate_c_major_file(audio_path: Path, expected_sha256: str) -> AccuracyCaseReport: + """Checksum, decode, and score one immutable C major WAV snapshot. + + Args: + audio_path: On-disk WAV written by ``write_pcm_wav``. + expected_sha256: Registered digest. Mismatch fails closed before decode. + + Returns: + A case report whose metric is duration-weighted recall of ``C``. + """ + with _verified_fixture_path(audio_path, expected_sha256) as verified_path: + audio, sample_rate = read_pcm_wav(verified_path) + return evaluate_c_major_pcm(audio, sample_rate, expected_sha256) + + +def evaluate_click_tempo_file( + audio_path: Path, + expected_sha256: str, + true_bpm: float = DEFAULT_CLICK_BPM, +) -> AccuracyCaseReport: + """Score one immutable checksum-verified click-track WAV snapshot. + + Args: + audio_path: On-disk WAV written by ``write_pcm_wav``. + expected_sha256: Registered digest. Mismatch fails closed. + true_bpm: Known click tempo. + + Returns: + A case report whose metric is 1.0 on Acc1 pass and 0.0 on Acc1 fail. + """ + with _verified_fixture_path(audio_path, expected_sha256) as verified_path: + features = TemporalAnalyzer().analyze(verified_path) + passed = tempo_acc1(features["bpm"], true_bpm) + return build_case_report( + case_id=f"click-{true_bpm:g}-bpm", + audio_sha256=expected_sha256, + metric_name="tempo_acc1", + metric_value=1.0 if passed else 0.0, + passed=passed, + true_label=f"{true_bpm:g} bpm", + ) diff --git a/services/analysis-engine/src/bandscope_analysis/accuracy/fixtures.py b/services/analysis-engine/src/bandscope_analysis/accuracy/fixtures.py new file mode 100644 index 000000000..ab6750825 --- /dev/null +++ b/services/analysis-engine/src/bandscope_analysis/accuracy/fixtures.py @@ -0,0 +1,206 @@ +"""Deterministic, license-clean PCM fixtures for accuracy acceptance.""" + +from __future__ import annotations + +import hashlib +from pathlib import Path + +import numpy as np +import soundfile as sf # type: ignore[import-untyped] +from numpy.typing import NDArray + +C_MAJOR_LABEL = "C" +DEFAULT_SAMPLE_RATE = 22_050 +DEFAULT_CLICK_BPM = 120.0 +C4_HZ = 261.63 +E4_HZ = 329.63 +G4_HZ = 392.00 +_CLICK_FREQUENCY_HZ = 1_000.0 +_CLICK_DURATION_SECONDS = 0.01 +_CLICK_DECAY = 80.0 + + +def _fixture_sample_count(duration_seconds: float, sample_rate: int) -> int: + """Convert fixture timing evidence to at least one finite integer sample.""" + scaled_sample_count = duration_seconds * sample_rate + if not np.isfinite(scaled_sample_count) or scaled_sample_count < 1: + raise ValueError("fixture sample count must be finite and at least one sample") + return int(scaled_sample_count) + + +def render_c_major_triad( + duration_seconds: float = 3.0, + sample_rate: int = DEFAULT_SAMPLE_RATE, +) -> NDArray[np.float32]: + """Render a unit-peak C major triad as float32 PCM. + + Args: + duration_seconds: Finite positive non-Boolean fixture length in seconds. + sample_rate: Finite positive non-Boolean samples-per-second rate. + + Returns: + Mono float32 samples in ``[-1, 1]``. + + Raises: + ValueError: If duration or sample rate is Boolean, non-finite, or not positive, + or if their derived sample count is non-finite or below one sample. + """ + if ( + isinstance(duration_seconds, bool) + or not np.isfinite(duration_seconds) + or duration_seconds <= 0 + ): + raise ValueError("duration_seconds must be a finite positive non-Boolean number") + if isinstance(sample_rate, bool) or not np.isfinite(sample_rate) or sample_rate <= 0: + raise ValueError("sample_rate must be a finite positive non-Boolean number") + + sample_count = _fixture_sample_count(duration_seconds, sample_rate) + times = np.arange(sample_count, dtype=np.float32) / np.float32(sample_rate) + mix = ( + np.sin(2 * np.pi * np.float32(C4_HZ) * times) + + np.sin(2 * np.pi * np.float32(E4_HZ) * times) + + np.sin(2 * np.pi * np.float32(G4_HZ) * times) + ) / np.float32(3.0) + return np.asarray(mix, dtype=np.float32) + + +def render_click_track( + bpm: float = DEFAULT_CLICK_BPM, + duration_seconds: float = 8.0, + sample_rate: int = DEFAULT_SAMPLE_RATE, +) -> NDArray[np.float32]: + """Render a click track at a known tempo. + + Args: + bpm: Finite positive non-Boolean true tempo in beats per minute. + duration_seconds: Finite positive non-Boolean fixture length in seconds. + sample_rate: Finite positive non-Boolean samples-per-second rate. + + Returns: + Mono float32 samples with a decaying click on each beat. + + Raises: + ValueError: If tempo, duration, or sample rate is Boolean, non-finite, + or not positive, if the derived sample count is non-finite or below + one sample, if the derived beat interval is non-finite or shorter + than one sample, or if the click pulse itself is shorter than one + sample at the requested rate. + """ + if isinstance(bpm, bool) or not np.isfinite(bpm) or bpm <= 0: + raise ValueError("bpm must be positive, finite, and non-Boolean") + if ( + isinstance(duration_seconds, bool) + or not np.isfinite(duration_seconds) + or duration_seconds <= 0 + ): + raise ValueError("duration_seconds must be a finite positive non-Boolean number") + if isinstance(sample_rate, bool) or not np.isfinite(sample_rate) or sample_rate <= 0: + raise ValueError("sample_rate must be a finite positive non-Boolean number") + + sample_count = _fixture_sample_count(duration_seconds, sample_rate) + audio = np.zeros(sample_count, dtype=np.float32) + with np.errstate(over="ignore", divide="ignore", invalid="ignore"): + interval_seconds = float(np.divide(60.0, bpm)) + interval_samples = float(np.multiply(interval_seconds, sample_rate)) + if not np.isfinite(interval_seconds): + raise ValueError("bpm must produce a finite beat interval") + if not np.isfinite(interval_samples) or interval_samples < 1: + raise ValueError("beat interval must be finite and at least one sample") + + click_sample_count = _CLICK_DURATION_SECONDS * sample_rate + if not np.isfinite(click_sample_count) or click_sample_count < 1: + raise ValueError("click length must be finite and at least one sample") + click_length = int(click_sample_count) + click_times = np.arange(click_length, dtype=np.float32) / np.float32(sample_rate) + click = ( + np.sin(2 * np.pi * np.float32(_CLICK_FREQUENCY_HZ) * click_times) + * np.exp(-click_times * np.float32(_CLICK_DECAY)) + ).astype(np.float32) + beat_time = 0.0 + while True: + start = int(beat_time * sample_rate) + if start >= sample_count: + break + end = min(sample_count, start + click_length) + audio[start:end] += click[: end - start] + beat_time += interval_seconds + + peak = float(np.max(np.abs(audio))) + if peak > 0: + audio /= np.float32(peak) + return audio + + +def write_pcm_wav(path: Path, audio: NDArray[np.floating], sample_rate: int) -> str: + """Write a WAV file and return the SHA-256 digest of the bytes on disk. + + Args: + path: Destination path. Parent directories are created. + audio: Mono PCM samples. + sample_rate: Finite positive non-Boolean samples-per-second rate used to write the file. + + Returns: + Lowercase hex SHA-256 of the written file. + + Raises: + ValueError: If the sample rate is Boolean, non-finite, or not positive. + """ + if isinstance(sample_rate, bool) or not np.isfinite(sample_rate) or sample_rate <= 0: + raise ValueError("sample_rate must be a finite positive non-Boolean number") + path.parent.mkdir(parents=True, exist_ok=True) + sf.write(path, np.asarray(audio, dtype=np.float32), sample_rate) + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def read_pcm_wav(path: Path) -> tuple[NDArray[np.float32], int]: + """Decode a WAV file to mono float32 PCM. + + Args: + path: Existing WAV path written by ``write_pcm_wav``. + + Returns: + A tuple of mono samples and the file sample rate. + + Raises: + ValueError: If the file has no samples after decode. + """ + audio, sample_rate = sf.read(path, dtype="float32", always_2d=False) + samples = np.asarray(audio, dtype=np.float32) + if samples.ndim > 1: + samples = np.mean(samples, axis=1).astype(np.float32) + if samples.size == 0: + raise ValueError("decoded WAV has no samples") + return samples, int(sample_rate) + + +def read_verified_fixture_bytes(path: Path, expected_sha256: str) -> bytes: + """Read one immutable fixture snapshot and verify its registered digest. + + Args: + path: Existing WAV path. + expected_sha256: Lowercase hex digest recorded in the case manifest. + + Returns: + The exact bytes whose SHA-256 matched ``expected_sha256``. + + Raises: + ValueError: If the snapshot digest does not match. + """ + payload = path.read_bytes() + actual = hashlib.sha256(payload).hexdigest() + if actual != expected_sha256: + raise ValueError("Accuracy fixture checksum mismatch") + return payload + + +def assert_fixture_checksum(path: Path, expected_sha256: str) -> None: + """Fail closed when a fixture file does not match its registered digest. + + Args: + path: Existing WAV path. + expected_sha256: Lowercase hex digest recorded in the case manifest. + + Raises: + ValueError: If the file digest does not match. + """ + read_verified_fixture_bytes(path, expected_sha256) diff --git a/services/analysis-engine/src/bandscope_analysis/accuracy/manifest.py b/services/analysis-engine/src/bandscope_analysis/accuracy/manifest.py new file mode 100644 index 000000000..b1d07634c --- /dev/null +++ b/services/analysis-engine/src/bandscope_analysis/accuracy/manifest.py @@ -0,0 +1,169 @@ +"""Accuracy-case report schema for exact-value acceptance evidence.""" + +from __future__ import annotations + +import math +from pathlib import Path +from typing import TypedDict + +REQUIRED_REPORT_KEYS = frozenset( + { + "case_id", + "audio_sha256", + "metric_name", + "metric_value", + "passed", + "engine_version", + "true_label", + } +) +_HEX_DIGITS = frozenset("0123456789abcdefABCDEF") + + +class AccuracyCaseReport(TypedDict): + """One scored fixture case that a buyer can read without opening logs.""" + + case_id: str + audio_sha256: str + metric_name: str + metric_value: float + passed: bool + engine_version: str + true_label: str + + +def read_product_version(start: Path | None = None) -> str: + """Return the nearest non-empty ``VERSION`` file contents. + + Accuracy reports are provenance evidence, so an unavailable product version + is not converted into an ``unknown`` value that could still pass report + validation. + + Args: + start: File or directory to walk upward from. Defaults to this module. + + Returns: + Stripped product version text. + + Raises: + ValueError: If no non-empty ``VERSION`` file exists in the ancestor path. + """ + current = start if start is not None else Path(__file__).resolve() + cursor = current.parent if current.is_file() else current + for parent in (cursor, *cursor.parents): + candidate = parent / "VERSION" + if not candidate.is_file(): + continue + text = candidate.read_text(encoding="utf-8").strip() + if text: + return text + raise ValueError("Product VERSION file is missing or empty") + + +def build_case_report( + *, + case_id: str, + audio_sha256: str, + metric_name: str, + metric_value: float, + passed: bool, + true_label: str, + engine_version: str | None = None, +) -> AccuracyCaseReport: + """Build a validated accuracy case report. + + Args: + case_id: Stable fixture identifier such as ``c-major-triad``. + audio_sha256: Digest of the decoded fixture bytes. + metric_name: Registered metric id such as ``duration_weighted_chord_recall``. + metric_value: Numeric score for this run. + passed: Whether the score met the registered tolerance. + true_label: Ground-truth label the buyer should hear. + engine_version: Optional exact override. Defaults to the product ``VERSION``. + + Returns: + A report that ``parse_case_report`` will accept. + + Raises: + ValueError: If required report evidence is invalid or the default product + ``VERSION`` cannot be resolved. + """ + report: dict[str, object] = { + "case_id": case_id, + "audio_sha256": audio_sha256, + "metric_name": metric_name, + "metric_value": metric_value, + "passed": passed, + "engine_version": engine_version if engine_version is not None else read_product_version(), + "true_label": true_label, + } + return parse_case_report(report) + + +def parse_case_report(value: object) -> AccuracyCaseReport: + """Validate an accuracy case report and return a typed copy. + + Args: + value: Untrusted mapping, typically loaded from JSON. + + Returns: + The same fields after type and presence checks. + + Raises: + ValueError: If the payload is missing keys or uses invalid values. + """ + if not isinstance(value, dict): + raise ValueError("Accuracy case report must be an object") + + missing = REQUIRED_REPORT_KEYS.difference(value) + if missing: + missing_names = ", ".join(sorted(missing)) + raise ValueError(f"Accuracy case report is missing: {missing_names}") + + case_id = value["case_id"] + audio_sha256 = value["audio_sha256"] + metric_name = value["metric_name"] + metric_value = value["metric_value"] + passed = value["passed"] + engine_version = value["engine_version"] + true_label = value["true_label"] + + if not isinstance(case_id, str) or not case_id or case_id.strip() != case_id: + raise ValueError("case_id must be an exact non-empty string") + if ( + not isinstance(audio_sha256, str) + or len(audio_sha256) != 64 + or any(character not in _HEX_DIGITS for character in audio_sha256) + ): + raise ValueError("audio_sha256 must be a 64-character hex digest") + if not isinstance(metric_name, str) or not metric_name or metric_name.strip() != metric_name: + raise ValueError("metric_name must be an exact non-empty string") + if isinstance(metric_value, bool) or not isinstance(metric_value, (int, float)): + raise ValueError("metric_value must be a finite number") + try: + normalized_metric_value = float(metric_value) + except OverflowError as error: + raise ValueError("metric_value must be a finite number") from error + if not math.isfinite(normalized_metric_value): + raise ValueError("metric_value must be a finite number") + if not isinstance(passed, bool): + raise ValueError("passed must be a boolean") + if ( + not isinstance(engine_version, str) + or not engine_version + or engine_version.strip() != engine_version + or engine_version.lower() == "unknown" + ): + raise ValueError("engine_version must be an exact non-empty version, not unknown") + if not isinstance(true_label, str) or not true_label or true_label.strip() != true_label: + raise ValueError("true_label must be an exact non-empty string") + + return { + "case_id": case_id, + "audio_sha256": audio_sha256, + "metric_name": metric_name, + "metric_value": normalized_metric_value, + "passed": passed, + "engine_version": engine_version, + "true_label": true_label, + } diff --git a/services/analysis-engine/src/bandscope_analysis/accuracy/metrics.py b/services/analysis-engine/src/bandscope_analysis/accuracy/metrics.py new file mode 100644 index 000000000..c565a8306 --- /dev/null +++ b/services/analysis-engine/src/bandscope_analysis/accuracy/metrics.py @@ -0,0 +1,121 @@ +"""Recognized MIR metrics for BandScope accuracy acceptance.""" + +from __future__ import annotations + +import math +from collections.abc import Sequence + + +def duration_weighted_chord_recall( + segments: Sequence[tuple[float, float, str]], + expected_chord: str, + start_seconds: float, + end_seconds: float, +) -> float: + """Return the annotated-interval fraction labeled with ``expected_chord``. + + This is a single-label duration-weighted recall on one interval, the + smallest WCSR-style score that still answers “did the engine hear the + known chord for most of the fixture?” (Odekerken et al., 2021; Raffel + et al., 2014). Matching estimate intervals are unioned after clipping to + the annotation window, so overlapping estimates cannot count the same + annotated time more than once. Non-finite, Boolean, empty, or reversed + annotation and estimate timing is invalid acceptance evidence and fails + closed before clipping. + + Args: + segments: ``(start, end, chord)`` estimates in seconds. Segment timing + values must be finite non-Boolean numbers and each end must be + greater than its start. + expected_chord: Ground-truth chord symbol for the interval. + start_seconds: Inclusive finite non-Boolean annotation start. + end_seconds: Exclusive finite non-Boolean annotation end. Must be + greater than start. + + Returns: + A value in ``[0, 1]``. + + Raises: + ValueError: If annotation or estimate timing is Boolean, non-finite, + empty, or reversed. + """ + if ( + isinstance(start_seconds, bool) + or isinstance(end_seconds, bool) + or not math.isfinite(start_seconds) + or not math.isfinite(end_seconds) + ): + raise ValueError("annotation times must be finite numbers") + if end_seconds <= start_seconds: + raise ValueError("annotation end_seconds must be greater than start_seconds") + + matching_intervals: list[tuple[float, float]] = [] + for segment_start, segment_end, chord in segments: + if ( + isinstance(segment_start, bool) + or isinstance(segment_end, bool) + or not math.isfinite(segment_start) + or not math.isfinite(segment_end) + ): + raise ValueError("segment times must be finite numbers") + if segment_end <= segment_start: + raise ValueError("segment end_seconds must be greater than start_seconds") + overlap_start = max(start_seconds, segment_start) + overlap_end = min(end_seconds, segment_end) + if overlap_end > overlap_start and chord == expected_chord: + matching_intervals.append((overlap_start, overlap_end)) + + if not matching_intervals: + return 0.0 + + matching_intervals.sort(key=lambda interval: (interval[0], interval[1])) + current_start, current_end = matching_intervals[0] + covered = 0.0 + for interval_start, interval_end in matching_intervals[1:]: + if interval_start <= current_end: + current_end = max(current_end, interval_end) + continue + covered += current_end - current_start + current_start, current_end = interval_start, interval_end + covered += current_end - current_start + return covered / (end_seconds - start_seconds) + + +def tempo_acc1( + estimated_bpm: float, + true_bpm: float, + relative_tolerance: float = 0.04, +) -> bool: + """Return whether estimated tempo is within Acc1 tolerance of the true tempo. + + Acc1 accepts an estimate within ``relative_tolerance`` of the true BPM and + does not credit octave errors (Schreiber & Müller, 2020). Boolean or + non-finite estimates, ground truth, or tolerances are invalid evidence and + fail closed instead of being converted into an ordinary metric miss. + + Args: + estimated_bpm: Engine tempo in beats per minute. Must be a finite + non-Boolean number. + true_bpm: Known fixture tempo. Must be a finite positive non-Boolean + number. + relative_tolerance: Finite non-negative non-Boolean Acc1 window. The + default is 4%. + + Returns: + ``True`` when the estimate is inside the Acc1 window. + + Raises: + ValueError: If any metric input is Boolean/non-finite, ``true_bpm`` is + not positive, or the tolerance is negative. + """ + if isinstance(estimated_bpm, bool) or not math.isfinite(estimated_bpm): + raise ValueError("estimated_bpm must be finite") + if isinstance(true_bpm, bool) or not math.isfinite(true_bpm) or true_bpm <= 0: + raise ValueError("true_bpm must be finite and positive") + if ( + isinstance(relative_tolerance, bool) + or not math.isfinite(relative_tolerance) + or relative_tolerance < 0 + ): + raise ValueError("relative_tolerance must be finite and non-negative") + return abs(estimated_bpm - true_bpm) / true_bpm <= relative_tolerance diff --git a/services/analysis-engine/tests/test_accuracy_acceptance.py b/services/analysis-engine/tests/test_accuracy_acceptance.py new file mode 100644 index 000000000..069653d83 --- /dev/null +++ b/services/analysis-engine/tests/test_accuracy_acceptance.py @@ -0,0 +1,351 @@ +"""Real-audio accuracy acceptance for decoded PCM fixtures. + +These cases prove a buyer-visible claim: a known waveform written to disk, +decoded, and analyzed yields the expected chord or tempo. Mocked chroma +matrices are not acceptance evidence. +""" + +from __future__ import annotations + +from pathlib import Path + +import numpy as np +import pytest +import soundfile as sf + +from bandscope_analysis.accuracy import ( + C_MAJOR_LABEL, + DEFAULT_CLICK_BPM, + DEFAULT_SAMPLE_RATE, + build_case_report, + duration_weighted_chord_recall, + evaluate_c_major_file, + evaluate_c_major_pcm, + evaluate_click_tempo_file, + parse_case_report, + read_pcm_wav, + read_product_version, + render_c_major_triad, + render_click_track, + tempo_acc1, + write_pcm_wav, +) +from bandscope_analysis.accuracy.evaluate import C_MAJOR_RECALL_FLOOR +from bandscope_analysis.api import build_demo_rehearsal_song + + +def test_c_major_fixture_is_deterministic(tmp_path: Path) -> None: + """Two writes of the same triad must share one SHA-256 digest.""" + audio = render_c_major_triad() + first = write_pcm_wav(tmp_path / "a.wav", audio, DEFAULT_SAMPLE_RATE) + second = write_pcm_wav(tmp_path / "b.wav", audio, DEFAULT_SAMPLE_RATE) + assert first == second + assert len(first) == 64 + + +def test_c_major_wav_recovers_c_after_file_decode(tmp_path: Path) -> None: + """A decoded C major WAV must recover C for most of the fixture duration.""" + audio = render_c_major_triad(duration_seconds=3.0) + path = tmp_path / "c-major.wav" + digest = write_pcm_wav(path, audio, DEFAULT_SAMPLE_RATE) + report = evaluate_c_major_file(path, digest) + assert report["true_label"] == C_MAJOR_LABEL + assert report["metric_name"] == "duration_weighted_chord_recall" + assert report["metric_value"] >= C_MAJOR_RECALL_FLOOR + assert report["passed"] is True + assert report["audio_sha256"] == digest + + +def test_click_wav_recovers_120_bpm_acc1(tmp_path: Path) -> None: + """A decoded 120 BPM click WAV must pass tempo Acc1.""" + audio = render_click_track(bpm=DEFAULT_CLICK_BPM, duration_seconds=8.0) + path = tmp_path / "click-120.wav" + digest = write_pcm_wav(path, audio, DEFAULT_SAMPLE_RATE) + report = evaluate_click_tempo_file(path, digest, DEFAULT_CLICK_BPM) + assert report["passed"] is True + assert report["metric_name"] == "tempo_acc1" + assert report["true_label"] == "120 bpm" + + +def test_silence_does_not_pass_c_major_recall() -> None: + """Silence must not be reported as a passing C major acceptance case.""" + silence = np.zeros(DEFAULT_SAMPLE_RATE, dtype=np.float32) + report = evaluate_c_major_pcm(silence, DEFAULT_SAMPLE_RATE, "b" * 64) + assert report["passed"] is False + assert report["metric_value"] < C_MAJOR_RECALL_FLOOR + + +def test_click_tempo_acc1_fails_when_true_tempo_is_wrong(tmp_path: Path) -> None: + """Acc1 must fail when the registered true tempo is not the click tempo.""" + audio = render_click_track(bpm=DEFAULT_CLICK_BPM, duration_seconds=8.0) + path = tmp_path / "click-wrong-label.wav" + digest = write_pcm_wav(path, audio, DEFAULT_SAMPLE_RATE) + report = evaluate_click_tempo_file(path, digest, true_bpm=40.0) + assert report["passed"] is False + assert report["metric_value"] == 0.0 + + +def test_checksum_mismatch_fails_closed(tmp_path: Path) -> None: + """A tampered fixture must not be scored as a passing case.""" + click = render_click_track() + click_path = tmp_path / "click.wav" + write_pcm_wav(click_path, click, DEFAULT_SAMPLE_RATE) + with pytest.raises(ValueError, match="checksum mismatch"): + evaluate_click_tempo_file(click_path, "0" * 64) + + triad = render_c_major_triad() + triad_path = tmp_path / "c-major.wav" + write_pcm_wav(triad_path, triad, DEFAULT_SAMPLE_RATE) + with pytest.raises(ValueError, match="checksum mismatch"): + evaluate_c_major_file(triad_path, "0" * 64) + + +def test_c_major_file_decode_scores_disk_not_memory(tmp_path: Path) -> None: + """Silence on disk must fail even when a C major array exists in memory.""" + triad = render_c_major_triad(duration_seconds=3.0) + silence = np.zeros_like(triad) + path = tmp_path / "silence.wav" + digest = write_pcm_wav(path, silence, DEFAULT_SAMPLE_RATE) + report = evaluate_c_major_file(path, digest) + assert report["passed"] is False + assert report["metric_value"] < C_MAJOR_RECALL_FLOOR + memory_report = evaluate_c_major_pcm(triad, DEFAULT_SAMPLE_RATE, digest) + assert memory_report["passed"] is True + + +def test_read_pcm_wav_mixes_stereo_to_mono(tmp_path: Path) -> None: + """A stereo fixture must collapse to mono before scoring.""" + path = tmp_path / "stereo.wav" + stereo = np.column_stack([np.ones(8, dtype=np.float32), np.zeros(8, dtype=np.float32)]) + sf.write(path, stereo, DEFAULT_SAMPLE_RATE) + audio, sample_rate = read_pcm_wav(path) + assert sample_rate == DEFAULT_SAMPLE_RATE + assert audio.shape == (8,) + assert np.allclose(audio, 0.5, atol=1e-3) + + +def test_read_pcm_wav_rejects_empty_file(tmp_path: Path) -> None: + """An empty WAV must fail closed instead of scoring as a pass.""" + path = tmp_path / "empty.wav" + sf.write(path, np.zeros(0, dtype=np.float32), DEFAULT_SAMPLE_RATE) + with pytest.raises(ValueError, match="no samples"): + read_pcm_wav(path) + + +def test_pipeline_surfaces_c_on_active_lead_vocal() -> None: + """Unmocked assembly must put measured C on lead vocal when that stem is active.""" + audio = render_c_major_triad(duration_seconds=3.0) + silence = np.zeros_like(audio) + song = build_demo_rehearsal_song( + { + "stems": { + "vocals": audio * np.float32(0.35), + "bass": silence, + "drums": silence, + "other": audio, + }, + "sr": DEFAULT_SAMPLE_RATE, + "separation": {"duration_seconds": 3.0, "chunk_count": 1, "notes": "accuracy"}, + } + ) + assert song["id"] == "analyzed-song" + lead_chords = [ + role["harmony"]["chord"] + for section in song["sections"] + for role in section["roles"] + if role["id"] == "lead-vocal" + ] + assert C_MAJOR_LABEL in lead_chords + + +def test_render_helpers_reject_non_positive_inputs() -> None: + """Fixture helpers must refuse empty or reversed generation parameters.""" + with pytest.raises(ValueError, match="duration_seconds"): + render_c_major_triad(duration_seconds=0) + with pytest.raises(ValueError, match="sample_rate"): + render_c_major_triad(sample_rate=0) + with pytest.raises(ValueError, match="must be positive"): + render_click_track(bpm=0) + with pytest.raises(ValueError, match="sample_rate"): + write_pcm_wav(Path("unused.wav"), np.zeros(4, dtype=np.float32), 0) + + +def test_render_helpers_reject_non_finite_inputs() -> None: + """Fixture generation must reject non-finite timing before allocation/loops.""" + with pytest.raises(ValueError, match="duration_seconds.*finite"): + render_c_major_triad(duration_seconds=np.nan) + with pytest.raises(ValueError, match="bpm.*finite"): + render_click_track(bpm=np.nan) + with pytest.raises(ValueError, match="duration_seconds.*finite"): + render_click_track(duration_seconds=np.inf) + + +def test_click_track_rejects_zero_length_click_evidence() -> None: + """A sample rate that cannot represent one click sample must fail closed.""" + with pytest.raises(ValueError, match="click length"): + render_click_track(bpm=60.0, duration_seconds=1.0, sample_rate=1) + + +def test_duration_weighted_recall_covers_overlap_and_misses() -> None: + """Recall must count only overlapping time that matches the expected chord.""" + assert duration_weighted_chord_recall([(0.0, 2.0, "C"), (2.0, 4.0, "G")], "C", 0.0, 4.0) == 0.5 + assert duration_weighted_chord_recall([(5.0, 6.0, "C")], "C", 0.0, 2.0) == 0.0 + with pytest.raises(ValueError, match="end_seconds"): + duration_weighted_chord_recall([], "C", 1.0, 1.0) + + +def test_duration_weighted_recall_unions_overlapping_matching_estimates() -> None: + """Overlapping matching estimates must not count annotation time twice.""" + recall = duration_weighted_chord_recall( + [(0.0, 2.0, "C"), (1.0, 3.0, "C"), (4.0, 5.0, "C")], + "C", + 0.0, + 5.0, + ) + assert recall == pytest.approx(0.8) + + +@pytest.mark.parametrize( + ("segments", "start_seconds", "end_seconds"), + [ + ([(np.nan, 2.0, "C")], 0.0, 4.0), + ([(0.0, np.inf, "C")], 0.0, 4.0), + ([(0.0, 2.0, "C")], np.nan, 4.0), + ([(0.0, 2.0, "C")], 0.0, np.inf), + ], +) +def test_duration_weighted_recall_rejects_non_finite_timing( + segments: list[tuple[float, float, str]], + start_seconds: float, + end_seconds: float, +) -> None: + """Non-finite estimate or annotation times must not fabricate recall.""" + with pytest.raises(ValueError, match="finite"): + duration_weighted_chord_recall(segments, "C", start_seconds, end_seconds) + + +@pytest.mark.parametrize( + ("segments", "start_seconds", "end_seconds"), + [ + ([(True, 2.0, "C")], 0.0, 4.0), + ([(0.0, True, "C")], 0.0, 4.0), + ([(0.0, 2.0, "C")], False, 4.0), + ([(0.0, 2.0, "C")], 0.0, True), + ], +) +def test_duration_weighted_recall_rejects_boolean_timing_evidence( + segments: list[tuple[float | bool, float | bool, str]], + start_seconds: float | bool, + end_seconds: float | bool, +) -> None: + """Boolean timestamps must not be accepted as numeric MIR timing evidence.""" + with pytest.raises(ValueError, match="times must be finite numbers"): + duration_weighted_chord_recall(segments, "C", start_seconds, end_seconds) # type: ignore[arg-type] + + +def test_tempo_acc1_window_and_guards() -> None: + """Acc1 must accept a 4% window and reject octave errors and bad inputs.""" + assert tempo_acc1(120.0, 120.0) is True + assert tempo_acc1(124.8, 120.0) is True + assert tempo_acc1(240.0, 120.0) is False + with pytest.raises(ValueError, match="true_bpm"): + tempo_acc1(120.0, 0.0) + with pytest.raises(ValueError, match="relative_tolerance"): + tempo_acc1(120.0, 120.0, relative_tolerance=-0.01) + + +@pytest.mark.parametrize( + ("estimated_bpm", "true_bpm", "relative_tolerance", "message"), + [ + (np.nan, 120.0, 0.04, "estimated_bpm"), + (np.inf, 120.0, 0.04, "estimated_bpm"), + (120.0, np.nan, 0.04, "true_bpm"), + (120.0, np.inf, 0.04, "true_bpm"), + (120.0, 120.0, np.nan, "relative_tolerance"), + (120.0, 120.0, np.inf, "relative_tolerance"), + ], +) +def test_tempo_acc1_rejects_non_finite_evidence( + estimated_bpm: float, + true_bpm: float, + relative_tolerance: float, + message: str, +) -> None: + """Non-finite estimate, truth, or tolerance must fail closed.""" + with pytest.raises(ValueError, match=message): + tempo_acc1(estimated_bpm, true_bpm, relative_tolerance) + + +@pytest.mark.parametrize( + ("estimated_bpm", "true_bpm", "relative_tolerance", "message"), + [ + (True, 120.0, 0.04, "estimated_bpm"), + (120.0, True, 0.04, "true_bpm"), + (120.0, 120.0, True, "relative_tolerance"), + ], +) +def test_tempo_acc1_rejects_boolean_numeric_evidence( + estimated_bpm: float | bool, + true_bpm: float | bool, + relative_tolerance: float | bool, + message: str, +) -> None: + """Boolean values must not satisfy numeric Acc1 evidence contracts.""" + with pytest.raises(ValueError, match=message): + tempo_acc1(estimated_bpm, true_bpm, relative_tolerance) # type: ignore[arg-type] + + +def test_parse_case_report_rejects_malformed_payloads() -> None: + """Manifest parsing must fail closed on missing or mistyped fields.""" + valid = build_case_report( + case_id="c-major-triad", + audio_sha256="a" * 64, + metric_name="duration_weighted_chord_recall", + metric_value=0.9, + passed=True, + true_label="C", + engine_version="0.1.3", + ) + assert parse_case_report(valid)["passed"] is True + + with pytest.raises(ValueError, match="must be an object"): + parse_case_report(["not", "an", "object"]) + with pytest.raises(ValueError, match="missing"): + parse_case_report({"case_id": "only"}) + with pytest.raises(ValueError, match="case_id"): + parse_case_report({**valid, "case_id": ""}) + with pytest.raises(ValueError, match="audio_sha256"): + parse_case_report({**valid, "audio_sha256": "short"}) + with pytest.raises(ValueError, match="audio_sha256"): + parse_case_report({**valid, "audio_sha256": "g" * 64}) + with pytest.raises(ValueError, match="metric_name"): + parse_case_report({**valid, "metric_name": ""}) + with pytest.raises(ValueError, match="metric_value"): + parse_case_report({**valid, "metric_value": True}) + with pytest.raises(ValueError, match="metric_value"): + parse_case_report({**valid, "metric_value": "0.9"}) + with pytest.raises(ValueError, match="metric_value"): + parse_case_report({**valid, "metric_value": np.nan}) + with pytest.raises(ValueError, match="metric_value"): + parse_case_report({**valid, "metric_value": np.inf}) + with pytest.raises(ValueError, match="passed"): + parse_case_report({**valid, "passed": 1}) + with pytest.raises(ValueError, match="engine_version"): + parse_case_report({**valid, "engine_version": ""}) + with pytest.raises(ValueError, match="true_label"): + parse_case_report({**valid, "true_label": ""}) + + +def test_read_product_version_uses_version_file_and_fails_closed(tmp_path: Path) -> None: + """Version lookup must read VERSION and reject missing provenance.""" + versioned = tmp_path / "versioned" + versioned.mkdir() + (versioned / "VERSION").write_text("9.9.9\n", encoding="utf-8") + assert read_product_version(versioned) == "9.9.9" + assert read_product_version(versioned / "VERSION") == "9.9.9" + empty = tmp_path / "empty-tree" + empty.mkdir() + (empty / "VERSION").write_text(" \n", encoding="utf-8") + with pytest.raises(ValueError, match="VERSION"): + read_product_version(empty) + assert read_product_version() != "unknown" diff --git a/services/analysis-engine/tests/test_accuracy_boolean_fixture_inputs.py b/services/analysis-engine/tests/test_accuracy_boolean_fixture_inputs.py new file mode 100644 index 000000000..5735ded11 --- /dev/null +++ b/services/analysis-engine/tests/test_accuracy_boolean_fixture_inputs.py @@ -0,0 +1,94 @@ +"""Boolean and derived-arithmetic guards for deterministic real-audio fixtures.""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any, cast + +import numpy as np +import pytest + +from bandscope_analysis.accuracy import render_c_major_triad, render_click_track, write_pcm_wav + + +@pytest.mark.parametrize( + ("keyword", "value", "message"), + [ + ("duration_seconds", True, "duration_seconds"), + ("sample_rate", True, "sample_rate"), + ], +) +def test_c_major_fixture_rejects_boolean_numeric_evidence( + keyword: str, + value: bool, + message: str, +) -> None: + """Boolean duration/rate values must not become numeric fixture authority.""" + kwargs = {keyword: cast(Any, value)} + with pytest.raises(ValueError, match=message): + render_c_major_triad(**kwargs) + + +@pytest.mark.parametrize( + ("keyword", "value", "message"), + [ + ("bpm", True, "bpm"), + ("duration_seconds", True, "duration_seconds"), + ("sample_rate", True, "sample_rate"), + ], +) +def test_click_fixture_rejects_boolean_numeric_evidence( + keyword: str, + value: bool, + message: str, +) -> None: + """Boolean tempo/duration/rate values must fail before allocation or loops.""" + kwargs = {keyword: cast(Any, value)} + with pytest.raises(ValueError, match=message): + render_click_track(**kwargs) + + +@pytest.mark.parametrize("factory", [render_c_major_triad, render_click_track]) +def test_fixture_rejects_nonfinite_scaled_sample_count(factory: Any) -> None: + """Finite inputs whose product overflows must fail before allocation authority.""" + with pytest.raises(ValueError, match="sample count"): + factory(duration_seconds=1e308) + + +@pytest.mark.parametrize("factory", [render_c_major_triad, render_click_track]) +def test_fixture_rejects_duration_shorter_than_one_sample(factory: Any) -> None: + """Positive durations that quantize to zero samples must fail as evidence.""" + sub_sample_duration = np.nextafter(0.0, 1.0) + + with pytest.raises(ValueError, match="sample count"): + factory(duration_seconds=sub_sample_duration) + + +def test_click_fixture_rejects_tempo_whose_beat_interval_overflows() -> None: + """A finite positive tempo must not become infinite loop timing authority.""" + smallest_positive = np.nextafter(0.0, 1.0) + + with pytest.raises(ValueError, match="bpm"): + render_click_track(bpm=smallest_positive, duration_seconds=1.0) + + +def test_click_fixture_rejects_tempo_shorter_than_one_sample() -> None: + """A click cadence faster than one sample must not alias acceptance evidence.""" + with pytest.raises(ValueError, match="beat interval"): + render_click_track(bpm=6_001.0, duration_seconds=0.02, sample_rate=100) + + +def test_click_fixture_rejects_sample_rate_too_low_for_one_click_sample() -> None: + """A click fixture must contain at least one sample of click evidence.""" + with pytest.raises(ValueError, match="click length"): + render_click_track(bpm=60.0, duration_seconds=0.1, sample_rate=99) + + +def test_wav_writer_rejects_boolean_sample_rate(tmp_path: Path) -> None: + """A Boolean sample rate must not be serialized as a 1 Hz WAV contract.""" + with pytest.raises(ValueError, match="sample_rate"): + write_pcm_wav( + tmp_path / "boolean-rate.wav", + np.zeros(4, dtype=np.float32), + cast(Any, True), + ) diff --git a/services/analysis-engine/tests/test_accuracy_manifest_string_contract.py b/services/analysis-engine/tests/test_accuracy_manifest_string_contract.py new file mode 100644 index 000000000..fb0fbff14 --- /dev/null +++ b/services/analysis-engine/tests/test_accuracy_manifest_string_contract.py @@ -0,0 +1,37 @@ +"""Regression tests for accuracy-report string evidence authority.""" + +from __future__ import annotations + +import pytest + +from bandscope_analysis.accuracy import build_case_report, parse_case_report + + +@pytest.mark.parametrize( + ("field_name", "invalid_value"), + [ + ("case_id", " "), + ("case_id", " c-major-triad "), + ("metric_name", " "), + ("metric_name", " tempo_acc1 "), + ("true_label", " "), + ("true_label", " C "), + ], +) +def test_accuracy_report_rejects_blank_or_padded_string_evidence( + field_name: str, + invalid_value: str, +) -> None: + """Identifiers and truth labels must be exact non-blank evidence strings.""" + valid = build_case_report( + case_id="c-major-triad", + audio_sha256="a" * 64, + metric_name="duration_weighted_chord_recall", + metric_value=0.9, + passed=True, + true_label="C", + engine_version="0.1.3", + ) + + with pytest.raises(ValueError, match=field_name): + parse_case_report({**valid, field_name: invalid_value}) diff --git a/services/analysis-engine/tests/test_accuracy_manifest_version.py b/services/analysis-engine/tests/test_accuracy_manifest_version.py new file mode 100644 index 000000000..62c8f0ea3 --- /dev/null +++ b/services/analysis-engine/tests/test_accuracy_manifest_version.py @@ -0,0 +1,49 @@ +"""Regression tests for accuracy-report engine-version provenance.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from bandscope_analysis.accuracy import build_case_report, parse_case_report, read_product_version + + +def test_missing_or_empty_product_version_fails_closed(tmp_path: Path) -> None: + """Accuracy evidence must not publish an ``unknown`` engine version.""" + missing_tree = tmp_path / "missing-tree" + missing_tree.mkdir() + with pytest.raises(ValueError, match="VERSION"): + read_product_version(missing_tree) + + empty_tree = tmp_path / "empty-tree" + empty_tree.mkdir() + (empty_tree / "VERSION").write_text(" \n", encoding="utf-8") + with pytest.raises(ValueError, match="VERSION"): + read_product_version(empty_tree) + + +@pytest.mark.parametrize("engine_version", ["unknown", "UNKNOWN", " 0.1.3 "]) +def test_inexact_engine_version_is_rejected_at_report_boundary(engine_version: str) -> None: + """Unknown or whitespace-obscured versions must not become valid evidence.""" + report = { + "case_id": "c-major-triad", + "audio_sha256": "a" * 64, + "metric_name": "duration_weighted_chord_recall", + "metric_value": 0.9, + "passed": True, + "engine_version": engine_version, + "true_label": "C", + } + with pytest.raises(ValueError, match="engine_version"): + parse_case_report(report) + with pytest.raises(ValueError, match="engine_version"): + build_case_report( + case_id="c-major-triad", + audio_sha256="a" * 64, + metric_name="duration_weighted_chord_recall", + metric_value=0.9, + passed=True, + true_label="C", + engine_version=engine_version, + ) diff --git a/services/analysis-engine/tests/test_accuracy_metric_interval_contract.py b/services/analysis-engine/tests/test_accuracy_metric_interval_contract.py new file mode 100644 index 000000000..814b12790 --- /dev/null +++ b/services/analysis-engine/tests/test_accuracy_metric_interval_contract.py @@ -0,0 +1,28 @@ +"""Regression tests for MIR acceptance interval authority.""" + +from __future__ import annotations + +import pytest + +from bandscope_analysis.accuracy import duration_weighted_chord_recall + + +@pytest.mark.parametrize( + ("segment_start", "segment_end"), + [ + (2.0, 2.0), + (3.0, 2.0), + ], +) +def test_duration_weighted_recall_rejects_empty_or_reversed_estimate_intervals( + segment_start: float, + segment_end: float, +) -> None: + """Malformed estimate intervals must fail closed before acceptance scoring.""" + with pytest.raises(ValueError, match="segment end_seconds must be greater than start_seconds"): + duration_weighted_chord_recall( + [(segment_start, segment_end, "C")], + "C", + 0.0, + 4.0, + ) diff --git a/services/analysis-engine/tests/test_accuracy_metric_overflow_contract.py b/services/analysis-engine/tests/test_accuracy_metric_overflow_contract.py new file mode 100644 index 000000000..6177df77e --- /dev/null +++ b/services/analysis-engine/tests/test_accuracy_metric_overflow_contract.py @@ -0,0 +1,37 @@ +"""Regression tests for accuracy-report numeric overflow authority.""" + +from __future__ import annotations + +import pytest + +from bandscope_analysis.accuracy import build_case_report, parse_case_report + + +def test_accuracy_report_rejects_integer_metric_that_overflows_float() -> None: + """Huge integer metrics must fail closed instead of escaping validation.""" + valid = build_case_report( + case_id="c-major-triad", + audio_sha256="a" * 64, + metric_name="duration_weighted_chord_recall", + metric_value=0.9, + passed=True, + true_label="C", + engine_version="0.1.3", + ) + + with pytest.raises(ValueError, match="metric_value"): + parse_case_report({**valid, "metric_value": 10**400}) + + +def test_build_case_report_rejects_boolean_metric_evidence() -> None: + """Builder coercion must not turn Boolean evidence into a numeric score.""" + with pytest.raises(ValueError, match="metric_value"): + build_case_report( + case_id="c-major-triad", + audio_sha256="a" * 64, + metric_name="duration_weighted_chord_recall", + metric_value=True, + passed=True, + true_label="C", + engine_version="0.1.3", + ) diff --git a/services/analysis-engine/tests/test_accuracy_pcm_input_contract.py b/services/analysis-engine/tests/test_accuracy_pcm_input_contract.py new file mode 100644 index 000000000..c234502fa --- /dev/null +++ b/services/analysis-engine/tests/test_accuracy_pcm_input_contract.py @@ -0,0 +1,42 @@ +"""Fail-closed decoded-PCM evidence contract for accuracy acceptance.""" + +from __future__ import annotations + +from typing import Any, cast + +import numpy as np +import pytest + +from bandscope_analysis.accuracy import evaluate_c_major_pcm, render_c_major_triad + + +@pytest.mark.parametrize("sample_rate", [True, 0, -1, float("nan"), float("inf")]) +def test_c_major_pcm_rejects_invalid_sample_rate(sample_rate: object) -> None: + """Invalid rate evidence must fail before production recognition or division.""" + audio = render_c_major_triad(duration_seconds=0.25) + + with pytest.raises(ValueError, match="sample_rate"): + evaluate_c_major_pcm(audio, cast(Any, sample_rate), "a" * 64) + + +def test_c_major_pcm_rejects_non_mono_audio() -> None: + """Accuracy acceptance must not reinterpret multichannel arrays as mono PCM.""" + stereo = np.zeros((32, 2), dtype=np.float32) + + with pytest.raises(ValueError, match="audio"): + evaluate_c_major_pcm(stereo, 22_050, "a" * 64) + + +@pytest.mark.parametrize( + "audio", + [ + np.zeros(0, dtype=np.float32), + np.array([0.0, np.nan], dtype=np.float32), + np.array([0.0, np.inf], dtype=np.float32), + np.zeros(32, dtype=np.int16), + ], +) +def test_c_major_pcm_rejects_malformed_decoded_audio(audio: np.ndarray) -> None: + """Empty, non-finite, or non-floating decoded evidence must fail closed.""" + with pytest.raises(ValueError, match="audio"): + evaluate_c_major_pcm(audio, 22_050, "a" * 64) diff --git a/services/analysis-engine/tests/test_accuracy_same_byte_provenance.py b/services/analysis-engine/tests/test_accuracy_same_byte_provenance.py new file mode 100644 index 000000000..771dbad07 --- /dev/null +++ b/services/analysis-engine/tests/test_accuracy_same_byte_provenance.py @@ -0,0 +1,95 @@ +"""Same-byte provenance regressions for real-audio accuracy evidence.""" + +from __future__ import annotations + +from pathlib import Path + +import numpy as np +import pytest + +from bandscope_analysis.accuracy import ( + DEFAULT_CLICK_BPM, + DEFAULT_SAMPLE_RATE, + assert_fixture_checksum, + evaluate_c_major_file, + evaluate_click_tempo_file, + render_c_major_triad, + render_click_track, + write_pcm_wav, +) + + +def _replace_after_first_read( + monkeypatch: pytest.MonkeyPatch, + target: Path, + replacement_bytes: bytes, +) -> None: + """Replace ``target`` immediately after its first ``Path.read_bytes`` snapshot.""" + original_read_bytes = Path.read_bytes + replaced = False + + def read_bytes(path: Path) -> bytes: + nonlocal replaced + payload = original_read_bytes(path) + if path == target and not replaced: + replaced = True + path.write_bytes(replacement_bytes) + return payload + + monkeypatch.setattr(Path, "read_bytes", read_bytes) + + +def test_assert_fixture_checksum_accepts_matching_snapshot(tmp_path: Path) -> None: + """The compatibility checksum boundary must remain directly executable.""" + target = tmp_path / "matching.wav" + digest = write_pcm_wav(target, render_c_major_triad(), DEFAULT_SAMPLE_RATE) + + assert_fixture_checksum(target, digest) + + +def test_c_major_file_scores_the_bytes_that_satisfied_checksum( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Checksum evidence and chord scoring must bind to one immutable byte snapshot.""" + target = tmp_path / "c-major.wav" + digest = write_pcm_wav(target, render_c_major_triad(), DEFAULT_SAMPLE_RATE) + + replacement = tmp_path / "silence.wav" + write_pcm_wav( + replacement, + np.zeros(DEFAULT_SAMPLE_RATE * 3, dtype=np.float32), + DEFAULT_SAMPLE_RATE, + ) + _replace_after_first_read(monkeypatch, target, replacement.read_bytes()) + + report = evaluate_c_major_file(target, digest) + + assert report["audio_sha256"] == digest + assert report["passed"] is True + + +def test_click_file_scores_the_bytes_that_satisfied_checksum( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Checksum evidence and tempo scoring must bind to one immutable byte snapshot.""" + target = tmp_path / "click.wav" + digest = write_pcm_wav( + target, + render_click_track(bpm=DEFAULT_CLICK_BPM), + DEFAULT_SAMPLE_RATE, + ) + + replacement = tmp_path / "silence.wav" + write_pcm_wav( + replacement, + np.zeros(DEFAULT_SAMPLE_RATE * 8, dtype=np.float32), + DEFAULT_SAMPLE_RATE, + ) + _replace_after_first_read(monkeypatch, target, replacement.read_bytes()) + + report = evaluate_click_tempo_file(target, digest, DEFAULT_CLICK_BPM) + + assert report["audio_sha256"] == digest + assert report["passed"] is True diff --git a/services/analysis-engine/tests/test_accuracy_tempo_case_identity.py b/services/analysis-engine/tests/test_accuracy_tempo_case_identity.py new file mode 100644 index 000000000..c39264638 --- /dev/null +++ b/services/analysis-engine/tests/test_accuracy_tempo_case_identity.py @@ -0,0 +1,26 @@ +"""Regression contract for tempo accuracy report identity.""" + +from __future__ import annotations + +from pathlib import Path + +from bandscope_analysis.accuracy import ( + DEFAULT_CLICK_BPM, + DEFAULT_SAMPLE_RATE, + evaluate_click_tempo_file, + render_click_track, + write_pcm_wav, +) + + +def test_click_report_case_id_matches_registered_true_tempo(tmp_path: Path) -> None: + """A non-default truth label must not be published under the 120 BPM case ID.""" + audio = render_click_track(bpm=DEFAULT_CLICK_BPM, duration_seconds=8.0) + path = tmp_path / "click-wrong-label.wav" + digest = write_pcm_wav(path, audio, DEFAULT_SAMPLE_RATE) + + report = evaluate_click_tempo_file(path, digest, true_bpm=40.0) + + assert report["case_id"] == "click-40-bpm" + assert report["true_label"] == "40 bpm" + assert report["passed"] is False