Repository navigation
[Lane 1 · numerics] Precision/Recall/F1 — averaging-mode and zero-division conventions #1224
Description
Sub-issue of #1222 (Precision, Recall and F1Score metrics).
Lane: 1 · Numerics / Research
Skill needed: comfortable reading scikit-learn / torchmetrics source and docs. No Kotlin required, no SKaiNET knowledge required.
Size: s (~2–4 h)
Blocked by: nothing — this is the first task in the feature and unblocks #1226.
What to do
- Read
sklearn.metrics.precision_recall_fscore_support(docs + source) and
torchmetrics.classification.{Precision, Recall, F1Score}(docs + source). - Produce a short table — one row per edge case, one column per library — for:
- empty batch (no samples at all);
- a class that appears in neither predictions nor targets (TP+FP+FN = 0);
- a class that appears only in targets (never predicted) and vice versa;
- all predictions wrong;
- ties in the argmax (two logits exactly equal).
For each: what does the library return for precision / recall / F1 undermacroand
micro, and do the two libraries agree?
- Answer the three decisions the parent issue leaves open, each as a one-paragraph
recommendation with a link a reviewer can click:- Zero-division:
0.0(sklearn default) vs.NaNvs. a parameter. The parent
recommends0.0to matchAccuracy.compute()on an empty accumulator. - Class set for macro averaging: union of classes seen in targets ∪ predictions
(sklearn) vs. the full class dimensionpredictions.shape[dim](which counts
never-seen classes as F1 = 0). These differ on small batches. - Binary mode: does binary mean "report the positive class (index 1) only", and
shouldF1Scoretake athreshold: Float?likeAccuracydoes, or is binary purely
averaging = BINARY?
- Zero-division:
- Post the table and the three recommendations as a comment on [Feature]: Precision, Recall and F1Score metrics #1222 under
its Research section. That comment is the deliverable — not code.
Acceptance
- Edge-case table posted on [Feature]: Precision, Recall and F1Score metrics #1222 with links to the exact sklearn / torchmetrics
doc sections or source lines - Three explicit recommendations (zero-division, macro class set, binary/threshold)
- A maintainer has replied "adopted" (or amended) so [Lane 2 · kotlin-core] Precision/Recall/F1 — Averaging enum + internal ConfusionMatrixAccumulator #1226 can start from a fixed contract
Notes
The hand-computed reference fixture the Kotlin sub-issues will test against is:
predicted classes [2, 0, 1, 2], targets [2, 1, 1, 0] (3 classes). Expected under
sklearn: per-class precision [0, 1, 0.5], recall [0, 0.5, 1], F1 [0, 0.667, 0.667];
macro P = R = 0.5, macro F1 = 0.444; micro P = R = F1 = 0.5 (= accuracy). Please confirm
those numbers by actually running precision_recall_fscore_support once and paste the
output — that turns a hand calculation into a citable reference.
Activity
michalharakal commentedon Oct 30, 2026
The three convention decisions are settled and implemented in the merged metrics (PRs #1460, #1461, #1462): zero-division returns 0.0 (never NaN), macro averages over observed classes (targets ∪ predictions, matching sklearn), and binary mode reports positive class 1 with an optional threshold. Closing with parent issue #1222.