Skip to content

feat(metrics): Averaging enum and internal ConfusionMatrixAccumulator (#1226) - #1458

Merged
michalharakal merged 1 commit into
SKaiNET-developers:developfrom
EdinEd:feature/1226-confusion-matrix-accumulator
Oct 29, 2026
Merged

michalharakal merged 1 commit into
SKaiNET-developers:developfrom
EdinEd:feature/1226-confusion-matrix-accumulator

Conversation

@EdinEd

@EdinEd EdinEd commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Implements Lane 2 (Kotlin core) sub-issue #1226 for the Precision, Recall, and F1Score metric feature.

This PR introduces the public Averaging enum strategy and the shared internal ConfusionMatrixAccumulator / reduce() engine in skainet-lang-core. These components provide the foundational state and reduction logic for classification metrics, leaving Precision (#1227), Recall (#1228), and F1Score (#1229) to contribute only their respective formulas.

Part of DARC tracking issue #1222.
Closes #1226.
Unblocks #1227, #1228, and #1229.


Changes

  1. Averaging enum (sk.ainet.lang.nn.metrics.Averaging):

    • Public API with BINARY, MACRO, and MICRO strategies.
    • Comprehensive KDocs specifying reduction semantics and the 0.0 zero-division convention.
    • Validated against binary-compatibility check via :skainet-lang:skainet-lang-core:apiDump.
  2. ClassCounts internal data class:

    • Immutable snapshot of (tp: Long, fp: Long, fn: Long) with non-negative validation.
  3. ConfusionMatrixAccumulator internal accumulator:

    • Walks sample batches via ClassificationSupport.forEachPrediction(predictions, targets, dim, threshold).
    • Tracks sparse counts across targets ∪ predictions using unboxed mutable Long counters to avoid heap allocation overhead during training loops.
    • Provides counts(): Map<Int, ClassCounts> snapshot and reset() for accumulator reuse across epochs.
  4. reduce() internal aggregation function:

    • Centralizes BINARY, MACRO, and MICRO calculations in one location.
    • Zero-division handling: Any non-finite score (NaN or ±Infinity caused by zero denominators) is normalized to 0.0 via evalSafe().
    • BINARY: Evaluates positive class index 1 only. Enforces that only binary class labels (0 or 1) are present; returns 0.0 if class 1 was never observed.
    • MACRO: Computes the unweighted arithmetic mean over observed classes (targets ∪ predictions). Never-seen classes from predictions.shape[dim] do not dilute the average.
    • MICRO: Pools ∑TP, ∑FP, and ∑FN across all observed classes before evaluating the metric formula.

Lane 1 Contract Decisions (#1224)

  • Zero-Division: Evaluates to 0.0 (never NaN), consistent with Accuracy.compute() on empty inputs and scikit-learn defaults.
  • Macro Class Set: Operates over targets ∪ predictions (the set of observed classes), preventing unobserved classes in mini-batches from artificially suppressing the score.
  • Binary Mode: Strict validation for class indices 0 and 1. Evaluates positive class index 1, returning 0.0 when no positive samples exist.

Testing Strategy

As outlined in #1226, Option 2 was chosen: the accumulator and reduction suite is placed directly in skainet-lang-core's commonTest (ConfusionMatrixAccumulatorTest.kt). Tensors are constructed using DenseTensorDataFactory and VoidOpsTensor.

This keeps skainet-lang-core self-contained and allows tests to run across all Kotlin Multiplatform targets (JVM, Linux Native, etc.) without introducing cross-module test dependencies on backend modules.

Test Coverage Highlights:

  • Reference Fixture from [Lane 1 · numerics] Precision/Recall/F1 — averaging-mode and zero-division conventions #1224 (preds = [2, 0, 1, 2], targets = [2, 1, 1, 0]): Confirms counts {0: (0,1,1), 1: (1,0,1), 2: (1,1,0)} and verifies expected Macro F1 (0.4444...) and Micro F1 (0.5).
  • Reduction Modes: Exhaustive verification of BINARY, MACRO, and MICRO including empty accumulator edge cases.
  • Zero-Division & Non-Finite Safeguards: Division by zero yielding NaN or ±Infinity safely normalizes to 0.0.
  • Soft Targets & Multi-Dimensional Slices: Validated along default (dim = -1) and custom dimensions.
  • Binary Thresholding & Ties: Validated with explicit thresholds and tie-breaking behavior.

Acceptance Checklist

  • Averaging is public, documented, and registered in skainet-lang-core.api.
  • ConfusionMatrixAccumulator, ClassCounts, and reduce() are internal with zero public surface leak.
  • Reference fixture yields exact counts {0: (0,1,1), 1: (1,0,1), 2: (1,1,0)}.
  • reduce() tested for MACRO, MICRO, BINARY, and zero-division cases.
  • Multiplatform tests pass (:skainet-lang:skainet-lang-core:jvmTest, :skainet-lang:skainet-lang-core:linuxX64Test).
  • :skainet-lang:skainet-lang-core:apiCheck passed.
  • ./gradlew checkUnmarkedStubs passed.

@michalharakal
michalharakal merged commit 9082c69 into SKaiNET-developers:develop Oct 29, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Lane 2 · kotlin-core] Precision/Recall/F1 — Averaging enum + internal ConfusionMatrixAccumulator

2 participants