Repository navigation
Description
This is a second issue in the released isolated-label evaluation path, separate from the cluster/celltype label-source mismatch in #4.
At commit 0c68f87d554a52e363dbfda2a9d1fa9538eeae9f, evaluation_pipelines/scib_metrics/scib_metrics.py constructs batches numbered 1..B and then sets:
batch = read_batch(args.cty_path)
num = np.max(batch)+1
...
iasw = scib.me.isolated_labels_asw(
adata_integrated,
batch_key="batch",
label_key="cluster",
embed="X_emb",
iso_threshold=num,
)
if1 = scib.me.isolated_labels_f1(
adata_integrated,
batch_key="batch",
label_key="celltype",
cluster_key="cluster",
embed="X_emb",
iso_threshold=num,
)For B batches, num is B+1. In scib v1.1.5, iso_threshold is the maximum number of batches in which a label may occur and still be considered isolated. No observed label can occur in more than B batches, so threshold B+1 admits every label. That changes these metrics from isolated-label metrics into averages over all labels.
scMultiBench's environment file installs scib without a version pin. I therefore use v1.1.5 as a source-verified reference implementation, not as a claim about the exact publication environment. The same threshold semantics are stated in the public scib API: iso_threshold is the maximum number of batches per label.
Native scib counterexample
I ran the released threshold rule with actual scib 1.1.5 and AnnData 0.10.9. The complete scMultiBench evaluator and complete scib isolated_labels.py and silhouette.py sources were all Git-blob verified; the imported scib function ASTs match the pinned v1.1.5 source.
The fixture has three batches. Four common labels (C1..C4) occur in all three batches. One label (R) occurs only in batch 1.
- scib default isolation policy (
iso_threshold=None) selects onlyR. - scMultiBench's threshold is
num = 4, and selectsC1,C2,C3,C4, andR: every label.
Two synthetic embeddings make the effect on iASW visible:
| Candidate | scib default isolated-label iASW | released threshold 4 iASW |
|---|---|---|
| A: isolated label R is well preserved; common labels overlap | 0.9354775053 | 0.5656422721 |
| B: common labels are well separated; R overlaps a common label | 0.5251509457 | 0.7723953788 |
The metric preference reverses. Under the isolated-label policy A is better; with the released threshold B is better.
This is a selected synthetic counterexample, not a reconstruction of the paper's embeddings or method ranking. It demonstrates that the threshold can change what the metric rewards.
What was executed
Successful public run:
https://github.com/dnncha/Dnncha/actions/runs/34139256190
Source-verified test branch/PR:
dnncha/Dnncha#4
Environment: Python 3.10.21, scib 1.1.5, AnnData 0.10.9, NumPy 1.26.4, pandas 2.2.3, scikit-learn 1.5.2.
Pinned source blobs:
- scMultiBench evaluator:
b8549d293d2a6eaef9af91bd7842dc68e15e9fcf - scib v1.1.5
isolated_labels.py:2e8258ab79f972cd983f902f20e3cb247db820cc - scib v1.1.5
silhouette.py:8d44b3aa8c2e90e3e1792db35efc7ca4fdc25a7a
The same native run also reproduces issue #4's ordinary ASW label-source reversal using the real scib silhouette function: the compact technical/mixed-celltype embedding scores 0.9963883078 using cluster, but 0.4918026995 using celltype; the biologically separated embedding scores 0.9192562199 under either label source in the constructed case.
Suggested correction / clarification
If the intended metric is scib's default isolated-label metric, omit iso_threshold or pass None. If the benchmark intended a different isolation definition, choose and document an explicit threshold that does not mechanically include every label.
Simply replacing B+1 with B is not necessarily a meaningful fix: a threshold equal to all B batches would still admit labels present in every batch in implementations following this definition.
The iASW line also has the independent label-source problem tracked in #4; fixing this threshold alone does not fix that call's use of cluster rather than celltype.
The graph evaluator passes the same num = max(batch)+1 threshold to isolated-label F1, so the policy issue is not restricted to embedding iASW.
To measure publication impact, the next step is to rerun the released/original embeddings with their batch and biological-label metadata under the intended isolation rule. I am doing that where the repository provides sufficient released inputs; this issue itself does not claim a changed original-paper recommendation.
Activity
dnncha commentedon Sep 7, 2026
The released embedding demo confirms that this threshold is active on real repository data, not only in the synthetic fixture.
I reanalysed the complete demo inputs referenced by the evaluator's example command after Git-blob verification: 28,574 observations, 3 batches and 28 biological cell types. The released rule computes
num = 4and therefore selects all 28 cell types. Under scib 1.1.5's default isolation policy, the minimum batch presence is 1, and onlyErythroidis selected. Two other types occur in 2 batches and the remaining 25 occur in all 3.Using the same per-observation cell-type silhouette samples:
The actual released iASW call is 0.6303640834438349, but it also uses inferred
clusterlabels, the independent defect tracked in #4. Every inferred cluster in this demo is present in all three batches.Public run: https://github.com/dnncha/Dnncha/actions/runs/34139782478
Report and durable summary: https://github.com/dnncha/Dnncha/tree/2c068640f9b32386b9573790765e56c837d8a537/research/scmultibench-scib-native
Again, v1.1.5 is a source-verified reference implementation because scMultiBench's environment file leaves
scibunpinned. These released files are not yet identified as a named original-paper method run, so I am not treating 0.865638 as a corrected publication result.