Skip to content

feat(metrics): add F1Score metric and factory functions (#1229) - #1462

Merged
michalharakal merged 1 commit into
SKaiNET-developers:developfrom
EdinEd:feature/1229-f1-metric
Oct 30, 2026
Merged

michalharakal merged 1 commit into
SKaiNET-developers:developfrom
EdinEd:feature/1229-f1-metric

Conversation

@EdinEd

@EdinEd EdinEd commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Implements Lane 2 (Kotlin core) sub-issue #1229 for the Precision, Recall, and F1Score metric feature.

This PR introduces the public F1Score metric class and its companion factory functions (f1Score() and binaryF1Score()) in skainet-lang-core. The implementation computes $F_1$ directly from raw confusion-matrix counts via the shared ConfusionMatrixAccumulator and reduce() engine from #1226, walking batches once without allocating or delegating to separate Precision or Recall instances.

Part of DARC tracking issue #1222.
Closes #1229.
Depends on #1226.
Unblocks Lane 3 platform verification (#1230–#1233) and review (#1235).

Note

Unit tests only; ground-truth wiring tracked in SKEEP-005 ([SKEEP-005]: Ground-truth validation for stateful metrics #1223).


Changes

  1. F1Score class (sk.ainet.lang.nn.metrics.F1Score):

    • Public implementation of Metric with override val name: String = "f1".
    • Single-pass batch traversal: maintains one private ConfusionMatrixAccumulator(dim, threshold). Does not allocate or delegate to Precision / Recall, preventing duplicate tensor traversal.
    • Computes F1 directly via:
      $$\text{per-class } F_1 = \frac{2 \cdot \text{TP}}{2 \cdot \text{TP} + \text{FP} + \text{FN}}$$
      using reduce(acc.counts(), averaging) { tp, fp, fn -> if (2 * tp + fp + fn == 0L) 0.0 else (2.0 * tp) / (2 * tp + fp + fn) }.
    • Comprehensive KDocs explicitly documenting the mean-of-F1s vs. F1-of-means distinction under Averaging.MACRO.
  2. Factory functions (sk.ainet.lang.nn.metrics.F1ScoreKt):

  3. API Dump (skainet-lang-core.api):

    • JVM binary dump synchronized via ./gradlew :skainet-lang:skainet-lang-core:apiDump.
    • Registers only F1Score and F1ScoreKt in alphabetical order with zero leakage of internal accumulator types.

Lane 1 Contract Alignment (#1224)

  • Zero-Division: Evaluates to 0.0 (never NaN) on a fresh accumulator, empty batch, or whenever $2 \cdot \text{TP} + \text{FP} + \text{FN} = 0$, requiring only a single guard without separate checks on $P$ and $R$.
  • Macro Class Set: Operates over observed classes (targets ∪ predictions), ensuring unobserved classes in mini-batches do not artificially dilute macro F1.
  • Binary Mode: Validates that all observed classes are binary (0 or 1); evaluates class 1, returning 0.0 if class 1 was never observed. Throws IllegalArgumentException on multiclass inputs.

Testing Strategy

Tests are placed in skainet-backends/skainet-backend-cpu/src/commonTest/kotlin/sk/ainet/exec/metrics/F1ScoreTest.kt using DirectCpuExecutionContext(phase = Phase.EVAL), mirroring AccuracyTest.kt, PrecisionTest.kt, and RecallTest.kt.

Test Coverage Highlights:

  1. Mean-of-F1s vs. F1-of-Means (Reference Fixture [Lane 1 · numerics] Precision/Recall/F1 — averaging-mode and zero-division conventions #1224):
    • On preds = [2, 0, 1, 2], targets = [2, 1, 1, 0], per-class F1 scores are $[0, 2/3, 2/3]$.
    • Macro F1 evaluates to $4/9 \approx 0.4444...$.
    • Explicitly asserts that Macro F1 does not equal $2 \cdot P_{macro} \cdot R_{macro} / (P_{macro} + R_{macro}) = 0.5$, verifying immunity to the common F1-of-means bug.
  2. Micro Multi-Class:
    • Evaluates to $0.5$ on the reference fixture and confirms exact equality with accuracy() and precision(Averaging.MICRO).
    • Verified on asymmetric imbalanced fixture ($0.25 == \text{Accuracy}$).
  3. Soft Targets:
    • Exact parity between hard Int32 index targets and FP32 one-hot targets.
  4. Binary F1 & Thresholding:
    • 2-class logits with default threshold 0.5f ($F_1 = 0.8$) and custom threshold 0.7f ($F_1 = 0.5$).
    • Single-logit sigmoid tensors with threshold 0.5f, verifying exact tie 0.5f maps strictly to negative class 0.
    • Threshold-less binary F1 falling back to argmax.
  5. Multi-Batch Accumulation:
    • Two update() calls match a single concatenated batch; reset() restores compute to 0.0.
  6. Zero-Division & Edge Cases:
    • Fresh accumulator returns 0.0.
    • Class with zero ground-truth samples ($TP=0, FP=1, FN=0$) yields finite $0.0$ (never NaN).
    • Unseen classes in tensor dimension width do not dilute the macro average.
    • All predictions wrong evaluates to 0.0.
    • Empty batch (Shape(0, 3)) evaluates to 0.0 without errors.
  7. Shape Boundaries & Helpers:
    • 1D prediction with 0D scalar target.
    • 3D predictions with custom class axis dim = 1.
    • computeForBatch helper reset round-trip.
    • assertFailsWith<IllegalArgumentException> when Averaging.BINARY receives multiclass labels.

Acceptance Checklist

  • F1Score, f1Score(), and binaryF1Score() are public and registered in skainet-lang-core.api.
  • All test cases pass on JVM (:skainet-backends:skainet-backend-cpu:jvmTest).
  • Multiplatform tests pass on Linux Native (:skainet-backends:skainet-backend-cpu:linuxX64Test).
  • Batches are walked once per update() via a single ConfusionMatrixAccumulator.
  • :skainet-lang:skainet-lang-core:apiCheck and :skainet-backends:skainet-backend-cpu:apiCheck passed.
  • ./gradlew checkUnmarkedStubs passed.

@michalharakal
michalharakal merged commit 42939b1 into SKaiNET-developers:develop Oct 30, 2026
21 checks passed
michalharakal added a commit to EdinEd/SKaiNET that referenced this pull request Oct 30, 2026
The branch was cut before the Precision metric (SKaiNET-developers#1460) and F1Score
(SKaiNET-developers#1462) merged into develop, so the checked-in JVM API dump was
missing the Precision/PrecisionKt entries, failing jvmApiCheck on
the merge commit. Merge develop and regenerate the dump via
:skainet-lang:skainet-lang-core:apiDump.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Lane 2 · kotlin-core] F1Score metric

2 participants