Skip to content

Convert to landmark rectangular matrix system, also big speedup - #21

Merged
ConnorStoneAstro merged 7 commits into
mainfrom
betterchunk
Sep 15, 2026
Merged

ConnorStoneAstro merged 7 commits into
mainfrom
betterchunk

Conversation

@ConnorStoneAstro

Copy link
Copy Markdown
Owner

The whole underlying system is refactored for more speed and better "chunk" system.
Now called "landmarks" it creates a rectangular distance matrix rather than the full square, for big speedups.
This has some small API breaking changes, but mostly the improvements are internal and should produce near identical results just with better speed. The statistic also picks up a scale factor, but this doesn't matter for the sake of rank ordering.

Copilot AI lite review requested due to automatic review settings September 15, 2026 13:15
@codecov-commenter

codecov-commenter commented Sep 15, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.51485% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.74%. Comparing base (73f46f8) to head (3dfc1a1).

Files with missing lines Patch % Lines
src/pted/utils.py 98.39% 6 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #21      +/-   ##
==========================================
+ Coverage   96.84%   97.74%   +0.90%     
==========================================
  Files           4        4              
  Lines         317      621     +304     
==========================================
+ Hits          307      607     +300     
- Misses         10       14       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical and moderate correctness, reproducibility, and memory-safety issues remain.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Refactors PTED to use rectangular landmark matrices for faster permutation testing, adds containment testing, and updates diagnostics, tests, and documentation.

Changes:

  • Adds landmark allocation, batching, subgroup permutation, and enumeration.
  • Introduces directional containment testing and PIT diagnostics.
  • Updates public APIs, exports, tests, and documentation.
File summaries
File Summary and final review comments
tests/test_utils.py Adds landmark, backend, PIT, and enumeration coverage.
tests/test_pted.py Adds API, calibration, containment, backend, and RNG coverage.
src/pted/utils.py Implements landmarking, batching, containment, and PIT logic. Critical (3 votes): default batch sizing can permit multi-GB allocations and OOM. Moderate (3 votes): tie-unsafe cumulative ranks occur at lines 786 and 858. Moderate (1 vote): subsampled ECDF points do not validate the stated whole-curve simultaneous band. Moderate (1 vote): singleton containing samples cause containment allocation failures.
src/pted/tests.py Updates package smoke tests.
src/pted/pted.py Integrates public APIs. Moderate (2 votes): plots recompute landmark allocation, so plots and p-values can use different landmarks.
src/pted/__init__.py Exports containment testing.
README.md Updates API and statistical documentation. Nit (2 votes): duplicated word and subject-verb agreement. Nit (1 vote): overstates the meaning of being below the upper null envelope. Nit (1 vote): reverses the interpretation of pit_confidence=0.95.
Review details

Suppressed comments (5)

README.md:237

  • Being below a one-sided upper null envelope is compatible with uniform p-values and does not establish containment; only an excursion above the bound is evidence of excess peripheral x. This wording overstates what the diagnostic can conclude and conflicts with the caution immediately below.
line indicates the `x` values are likely contained at the threshold level (95%
by default).

README.md:97

  • pit_confidence=0.95 corresponds to a 5% significance level, not 95%. Calling 95% the significance level reverses the interpretation in this user-facing explanation.
Ultimately, this means that the fact we only use a finite number of simulations to compute the coverage test does not affect the validity of the bands in the PIT plot; any point that sticks out of the grey shaded area indicates possible miscalibration at the given significance level (default 95%).

src/pted/utils.py:861

  • The cumulative rank here is not tie-safe: argsort can place an x point before a y point with the same depth, so ranked_y.cumsum omits that y even though the documented count is {j in y : d_j >= d_i}. Duplicate or symmetric samples therefore get order-dependent per-point p-values and can miscalibrate the containment test. Compute the count at the end of each equal-depth group (or use an explicit >= comparison) before combining the p-values.
    order = (-depth).argsort(0)
    ranked_x = _take_along0(Xs.T, order, backend)
    ranked_y = _take_along0((1.0 - Xs).T, order, backend)
    p = (1.0 + ranked_y.cumsum(0)) / (prep["n_y"] + 1.0)

src/pted/utils.py:1159

  • When lattice exceeds max_points (and for continuous p-values whenever 20*n > max_points), this retains only a subset of attainable ECDF evaluation points. _band_accept_probability then calibrates the event of staying inside only at those points, while the API and plot describe a simultaneous band for the whole curve; the ECDF can jump outside between retained points, so the reported confidence is not valid for the full curve. Either account for all skipped points in the recursion or explicitly label this as an approximate/subsampled diagnostic.
        k = (
            np.arange(1, lattice + 1)
            if lattice <= max_points
            else np.unique(np.round(np.linspace(1, lattice, max_points)).astype(int))
        )
        t = k / lattice

src/pted/utils.py:824

  • When x is the larger sample and the containing sample y has one point, _allocate_landmarks always enters the singleton regime with n_small_landmarks == 0, so every landmark run of pted_containment_test(x, y) raises here. The message suggests using more landmarks, but no m < n can fix this allocation; allocate the containing singleton as a landmark for containment or reject this shape with a truthful error.
        n_y_landmarks = alloc["n_large_landmarks"] if x_is_small else alloc["n_small_landmarks"]
    if n_y_landmarks < 1:
        raise ValueError(
            "the containing sample has no landmarks, so depth is unestimable; "
            "use more landmarks or the full matrix"
  • Files reviewed: 7/7 changed files
  • Comments generated: 4
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/pted/utils.py Resolved
Comment thread src/pted/pted.py Resolved
Comment thread src/pted/utils.py Resolved
Comment thread README.md Resolved
@ConnorStoneAstro
ConnorStoneAstro merged commit 42aadab into main Sep 15, 2026
20 of 30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants