Repository navigation
Convert to landmark rectangular matrix system, also big speedup - #21
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #21 +/- ##
==========================================
+ Coverage 96.84% 97.74% +0.90%
==========================================
Files 4 4
Lines 317 621 +304
==========================================
+ Hits 307 607 +300
- Misses 10 14 +4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical and moderate correctness, reproducibility, and memory-safety issues remain.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Refactors PTED to use rectangular landmark matrices for faster permutation testing, adds containment testing, and updates diagnostics, tests, and documentation.
Changes:
- Adds landmark allocation, batching, subgroup permutation, and enumeration.
- Introduces directional containment testing and PIT diagnostics.
- Updates public APIs, exports, tests, and documentation.
File summaries
| File | Summary and final review comments |
|---|---|
tests/test_utils.py |
Adds landmark, backend, PIT, and enumeration coverage. |
tests/test_pted.py |
Adds API, calibration, containment, backend, and RNG coverage. |
src/pted/utils.py |
Implements landmarking, batching, containment, and PIT logic. Critical (3 votes): default batch sizing can permit multi-GB allocations and OOM. Moderate (3 votes): tie-unsafe cumulative ranks occur at lines 786 and 858. Moderate (1 vote): subsampled ECDF points do not validate the stated whole-curve simultaneous band. Moderate (1 vote): singleton containing samples cause containment allocation failures. |
src/pted/tests.py |
Updates package smoke tests. |
src/pted/pted.py |
Integrates public APIs. Moderate (2 votes): plots recompute landmark allocation, so plots and p-values can use different landmarks. |
src/pted/__init__.py |
Exports containment testing. |
README.md |
Updates API and statistical documentation. Nit (2 votes): duplicated word and subject-verb agreement. Nit (1 vote): overstates the meaning of being below the upper null envelope. Nit (1 vote): reverses the interpretation of pit_confidence=0.95. |
Review details
Suppressed comments (5)
README.md:237
- Being below a one-sided upper null envelope is compatible with uniform p-values and does not establish containment; only an excursion above the bound is evidence of excess peripheral x. This wording overstates what the diagnostic can conclude and conflicts with the caution immediately below.
line indicates the `x` values are likely contained at the threshold level (95%
by default).
README.md:97
pit_confidence=0.95corresponds to a 5% significance level, not 95%. Calling 95% the significance level reverses the interpretation in this user-facing explanation.
Ultimately, this means that the fact we only use a finite number of simulations to compute the coverage test does not affect the validity of the bands in the PIT plot; any point that sticks out of the grey shaded area indicates possible miscalibration at the given significance level (default 95%).
src/pted/utils.py:861
- The cumulative rank here is not tie-safe:
argsortcan place an x point before a y point with the same depth, soranked_y.cumsumomits that y even though the documented count is{j in y : d_j >= d_i}. Duplicate or symmetric samples therefore get order-dependent per-point p-values and can miscalibrate the containment test. Compute the count at the end of each equal-depth group (or use an explicit>=comparison) before combining the p-values.
order = (-depth).argsort(0)
ranked_x = _take_along0(Xs.T, order, backend)
ranked_y = _take_along0((1.0 - Xs).T, order, backend)
p = (1.0 + ranked_y.cumsum(0)) / (prep["n_y"] + 1.0)
src/pted/utils.py:1159
- When
latticeexceedsmax_points(and for continuous p-values whenever20*n > max_points), this retains only a subset of attainable ECDF evaluation points._band_accept_probabilitythen calibrates the event of staying inside only at those points, while the API and plot describe a simultaneous band for the whole curve; the ECDF can jump outside between retained points, so the reported confidence is not valid for the full curve. Either account for all skipped points in the recursion or explicitly label this as an approximate/subsampled diagnostic.
k = (
np.arange(1, lattice + 1)
if lattice <= max_points
else np.unique(np.round(np.linspace(1, lattice, max_points)).astype(int))
)
t = k / lattice
src/pted/utils.py:824
- When
xis the larger sample and the containing sampleyhas one point,_allocate_landmarksalways enters the singleton regime withn_small_landmarks == 0, so every landmark run ofpted_containment_test(x, y)raises here. The message suggests using more landmarks, but nom < ncan fix this allocation; allocate the containing singleton as a landmark for containment or reject this shape with a truthful error.
n_y_landmarks = alloc["n_large_landmarks"] if x_is_small else alloc["n_small_landmarks"]
if n_y_landmarks < 1:
raise ValueError(
"the containing sample has no landmarks, so depth is unestimable; "
"use more landmarks or the full matrix"
- Files reviewed: 7/7 changed files
- Comments generated: 4
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
The whole underlying system is refactored for more speed and better "chunk" system.
Now called "landmarks" it creates a rectangular distance matrix rather than the full square, for big speedups.
This has some small API breaking changes, but mostly the improvements are internal and should produce near identical results just with better speed. The statistic also picks up a scale factor, but this doesn't matter for the sake of rank ordering.