Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# keep text files with LF endings in the repository on every platform
* text=auto eol=lf
*.png binary
2 changes: 1 addition & 1 deletion .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,4 +25,4 @@ jobs:
- name: Fast tests (unit + regression)
run: pytest -q
- name: Validation tests
run: pytest -q --runslow tests/test_validation.py tests/test_joint_validation.py
run: pytest -q --runslow tests/test_validation.py tests/test_joint_validation.py tests/test_notebook.py
44 changes: 42 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,25 @@ pytest -q # fast tests; add --runslow for the validation tes

The scripts can also be run directly with `python maxcal_poisson.py ...`.

**Demo notebook.** `notebooks/demo.ipynb` walks through the method interactively,
using the test datasets:
- attempt counting as Q‡ moves;
- the reweighting weights, their saturation, and the resulting CV and N_eff;
- stitching and the emergence of Poisson statistics;
- validation against a simulated higher barrier, including what goes wrong with Q‡
on the barrier top;
- the joint-mode checks (kinetic ΔG, transition-path symmetry across temperatures).

```bash
pip install -e ".[demo]"
jupyter lab notebooks/demo.ipynb
```

The committed copy is rendered with static figures, so it can be read on GitHub.
The sliders appear when you run it live with `ipywidgets` installed. To refresh the
committed outputs:
`MAXCAL_DEMO_STATIC=1 jupyter nbconvert --to notebook --execute --inplace notebooks/demo.ipynb`.

```bash
python maxcal_poisson.py "runs/q_*.xvg" --qu 0.3 --qf 0.8 \
--qts 0.40 --qtse 0.55 --out results
Expand All @@ -277,6 +296,16 @@ Lines starting with `#` or `@` are ignored, so GROMACS `.xvg` files work directl
KS p-values, λ* with bootstrap CI, N_eff, stitching threshold, and
`cv_stitch0`/`ksp_stitch0` (stitching at λ = 0).
- `tse_frames_*.csv`: TSE frame per completed trajectory, with its weight.

**Status columns.** Every NaN in `summary.csv` comes with a reason code, so it can be
told apart from a failure of the code.

| column | values |
|---|---|
| `geom_status` | `ok`; `too_few_trajectories` (fewer than 10 completed); `too_few_bins` (too few distinct k for the χ² test) |
| `lag1_status` | `ok`; `too_few_pairs` (fewer than 10 consecutive failure cycles); `constant_cycles` (correlation undefined) |
| `reweight_status` | `ok`; `no_root` (weighted CV never reaches 1, §2.4); `low_neff` (N_eff below `--neff-min`); `cv_ge_1_at_lambda0` (already over-dispersed: a missing barrier does not explain the deviation) |
| `stitch_status` | `ok`; `ok_pool_fallback` (initial segments had to be used); `no_pass` (Poisson criterion never met in the scan); `empty_pool` (no failed attempts past this Q‡, typically at or beyond the barrier) |
- `survival.png`, `lambda_scan.png`, `robustness_qts.png`.

**Reading the results**
Expand All @@ -294,8 +323,8 @@ Lines starting with `#` or `@` are ignored, so GROMACS `.xvg` files work directl
## 5. Tests

```bash
pytest -q tests/ # 41 fast tests (~8 s)
pytest -q tests/ --runslow # + 13 validation tests against known answers (~15 s)
pytest -q tests/ # 46 fast tests (~8 s)
pytest -q tests/ --runslow # + 13 validation tests and the demo notebook (~25 s)
MAXCAL_UPDATE_REF=1 pytest tests/test_regtest.py # regenerate the reference
```

Expand All @@ -306,6 +335,17 @@ MAXCAL_UPDATE_REF=1 pytest tests/test_regtest.py # regenerate the reference
trajectories, compared with `tests/regtest/reference_summary.csv`. Deterministic
columns use rtol 1e-5; RNG-dependent stitching columns use loose absolute
tolerances.
- **`test_reweight_path.py`**: end-to-end test of the path where reweighting
succeeds, which the Langevin data never reach. It uses hand-built trajectories
(`tests/synthetic.py`) in which the weighted CV crosses 1 at λ* ≈ 1.1 with
N_eff ≈ 115. It checks that:
- λ* is finite and bracketed by its bootstrap confidence interval;
- N_eff and the KS p-value at λ* are finite;
- the weighted CV equals 1 at λ*;
- the TSE weight ratio between trajectories with 3 and 0 failures is exactly c³;
- every NaN has the right reason code.
- The regression test also checks that every NaN in the summary has a non-`ok`
reason code, and it compares the reason codes with the reference.
- **`test_validation.py`**: low-barrier and high-barrier Langevin runs; the high
barrier is the low one plus a Gaussian bump of 2 or 3 kT on the barrier top. It
checks four things.
Expand Down
5 changes: 2 additions & 3 deletions maxcal_joint.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,9 +51,7 @@
import numpy as np
from scipy import stats

import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import matplotlib.pyplot as plt # backend set to Agg in main(), not at import

import maxcal_poisson as mp

Expand Down Expand Up @@ -507,6 +505,7 @@ def report(res):


def main(argv=None):
plt.switch_backend("Agg") # command-line use: files only, no display
args = parse_args(argv)
res, F, B, feat_table, pplot, rng = analyse(args)
write_outputs(args, res, F, B, feat_table, pplot, rng)
Expand Down
58 changes: 50 additions & 8 deletions maxcal_poisson.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,7 @@
from scipy import stats
from scipy.special import logsumexp

import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import matplotlib.pyplot as plt # backend set to Agg in main(), not at import


# ----------------------------------------------------------------------------- I/O
Expand Down Expand Up @@ -290,6 +288,41 @@ def stitch_scan(lams, p, A, n_sub_size, args, rng):
return lam_pass, res


# ------------------------------------------------------------------ status flags
# Every NaN in summary.csv comes with a reason in one of these columns.
def geom_status(n, pval):
if n < 10:
return "too_few_trajectories"
return "too_few_bins" if not np.isfinite(pval) else "ok"


def lag1_status(pairs):
"""pairs: (n, 2) array of consecutive failure-cycle durations within trajectories."""
if len(pairs) < 10:
return "too_few_pairs"
if np.ptp(pairs[:, 0]) == 0 or np.ptp(pairs[:, 1]) == 0:
return "constant_cycles" # correlation undefined
return "ok"


def reweight_status(cv0, lam_star, neff, neff_min):
if cv0 >= 1:
return "cv_ge_1_at_lambda0" # already over-dispersed: a missing barrier can't explain it
if not np.isfinite(lam_star):
return "no_root" # CV never reaches 1: observed attempt counts too few
if not neff >= neff_min:
return "low_neff"
return "ok"


def stitch_status(lam_stitch, n_fail_pool, n_succ_pool, pool_note):
if n_fail_pool == 0 or n_succ_pool == 0:
return "empty_pool" # no failed attempts past this Q‡ (at/after the barrier?)
if not np.isfinite(lam_stitch):
return "no_pass"
return "ok_pool_fallback" if pool_note else "ok" # fallback: initial segments were used


# --------------------------------------------------------------------------- main
def analyse(trajs, qts, args, rng, lams, full=False):
A = attempts(trajs, qts, args.t0, args.include_initial, args.qtse)
Expand All @@ -299,9 +332,11 @@ def analyse(trajs, qts, args, rng, lams, full=False):
frac_k0=float(np.mean(k[done] == 0)) if done.any() else np.nan, p=p,
note=A["note"])
out["geom_p"], out["geom_bins"] = geometric_test(k[done])
out["geom_status"] = geom_status(int(done.sum()), out["geom_p"])
pr = A["pairs"]
out["lag1_status"] = lag1_status(pr)
out["lag1_rho"], out["lag1_p"] = (stats.spearmanr(pr[:, 0], pr[:, 1])
if len(pr) >= 10 else (np.nan, np.nan))
if out["lag1_status"] == "ok" else (np.nan, np.nan))

cv, neff = cv_curve(lams, k, T, done, p)
lam_star = first_crossing(lams, cv)
Expand Down Expand Up @@ -332,9 +367,12 @@ def analyse(trajs, qts, args, rng, lams, full=False):
out["lam_lo"], out["lam_hi"] = (np.percentile(fin, [2.5, 97.5]) if fin.size
else (np.nan, np.nan))
out["boot_noroot"] = float(np.mean(~np.isfinite(boots))) if boots.size else np.nan
out["reweight_status"] = reweight_status(out["cv0"], lam_star, out["neff_star"], args.neff_min)

lam_st, st_res = stitch_scan(lams, p, A, k.size, args, rng)
out["lam_stitch"] = lam_st
out["stitch_status"] = stitch_status(lam_st, A["fail_pool"].size, A["succ_pool"].size,
A["note"])
out["cv_stitch0"], out["ksp_stitch0"] = (st_res[0][1], st_res[0][2]) if st_res else (np.nan, np.nan)
out["cv_lammax"] = cv[-1]
out["kmax"] = int(k.max()) if k.size else 0
Expand All @@ -345,6 +383,7 @@ def analyse(trajs, qts, args, rng, lams, full=False):


def main():
plt.switch_backend("Agg") # command-line use: files only, no display
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("files", nargs="+", help="trajectory files or glob patterns")
Expand Down Expand Up @@ -399,9 +438,10 @@ def main():
cols = ["qts", "N", "n_done", "sum_k", "frac_k0", "p", "geom_p", "lag1_rho", "lag1_p",
"cv0", "ks_p0", "lam_star", "lam_lo", "lam_hi", "boot_noroot", "neff_star",
"ks_p_star", "reweight_ok", "cv_lammax", "kmax", "cv_stitch0", "ksp_stitch0",
"lam_stitch", "p_stitch", "note"]
"lam_stitch", "p_stitch", "geom_status", "lag1_status", "reweight_status",
"stitch_status", "note"]
with open(os.path.join(args.out, "summary.csv"), "w", newline="") as fh:
wr = csv.writer(fh)
wr = csv.writer(fh, lineterminator="\n")
wr.writerow(cols)
for r in results:
wr.writerow([r[c] if isinstance(r[c], str) else f"{r[c]:.6g}" for c in cols])
Expand All @@ -415,7 +455,9 @@ def main():
f"{r['lam_stitch']:8.2f}" + (" *ref" if r is ref else ""))

# ---- warnings for the reference surface
print(f"\nReference Q‡ = {qref:.3f}")
print(f"\nReference Q‡ = {qref:.3f} [geometric test: {ref['geom_status']}, "
f"lag-1: {ref['lag1_status']}, reweighting: {ref['reweight_status']}, "
f"stitching: {ref['stitch_status']}]")
if ref["note"]:
print(" " + ref["note"])
if np.isfinite(ref["geom_p"]) and ref["geom_p"] < args.alpha:
Expand Down Expand Up @@ -445,7 +487,7 @@ def main():
A, w = ref["A"], ref["w"]
pos = {i: j for j, i in enumerate(A["idx"])}
with open(os.path.join(args.out, f"tse_frames_qts{qref:.3f}_qtse{(args.qtse or qref):.3f}.csv"), "w", newline="") as fh:
wr = csv.writer(fh)
wr = csv.writer(fh, lineterminator="\n")
wr.writerow(["trajectory", "frame", "time", "k_failed", "weight_at_lambda_star"])
for i, fr in A["tse"]:
j = pos[i]
Expand Down
650 changes: 650 additions & 0 deletions notebooks/demo.ipynb

Large diffs are not rendered by default.

3 changes: 2 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@ requires-python = ">=3.9"
dependencies = ["numpy>=1.22", "scipy>=1.8", "matplotlib>=3.5"]

[project.optional-dependencies]
test = ["pytest>=7"]
test = ["pytest>=7", "nbformat", "nbclient", "ipykernel"]
demo = ["jupyterlab", "ipywidgets>=8"]

[project.scripts]
maxcal-poisson = "maxcal_poisson:main"
Expand Down
12 changes: 6 additions & 6 deletions tests/regtest/reference_summary.csv
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
qts,N,n_done,sum_k,frac_k0,p,geom_p,lag1_rho,lag1_p,cv0,ks_p0,lam_star,lam_lo,lam_hi,boot_noroot,neff_star,ks_p_star,reweight_ok,cv_lammax,kmax,cv_stitch0,ksp_stitch0,lam_stitch,p_stitch,note
0.383333,100,100,286,0.21,0.259067,0.241822,-0.0816005,0.329201,0.805936,0.00124969,nan,nan,nan,1,nan,nan,0,0.540971,12,0.938403,0.0847288,0,0.259067,
0.466667,100,100,123,0.44,0.44843,0.914989,-0.0628215,0.70789,0.805936,0.00124969,nan,nan,nan,1,nan,nan,0,0.464694,8,0.884336,0.0366158,0.4,0.352741,
0.55,100,100,57,0.62,0.636943,0.475826,nan,nan,0.805936,0.00124969,nan,nan,nan,1,nan,nan,0,0.495261,4,0.876319,0.0228693,0.6,0.490531,
0.633333,100,100,25,0.77,0.8,nan,nan,nan,0.805936,0.00124969,nan,nan,nan,1,nan,nan,0,0.618446,2,0.765308,0.000249938,2.3,0.286242,failure pool <10: initial segments included
0.716667,100,100,11,0.89,0.900901,nan,nan,nan,0.805936,0.00124969,nan,nan,nan,1,nan,nan,0,0.649382,1,0.551963,0.000249938,3.35,0.24182,failure pool <10: initial segments included
qts,N,n_done,sum_k,frac_k0,p,geom_p,lag1_rho,lag1_p,cv0,ks_p0,lam_star,lam_lo,lam_hi,boot_noroot,neff_star,ks_p_star,reweight_ok,cv_lammax,kmax,cv_stitch0,ksp_stitch0,lam_stitch,p_stitch,geom_status,lag1_status,reweight_status,stitch_status,note
0.383333,100,100,286,0.21,0.259067,0.241822,-0.0816005,0.329201,0.805936,0.00124969,nan,nan,nan,1,nan,nan,0,0.540971,12,0.938403,0.0847288,0,0.259067,ok,ok,no_root,ok,
0.466667,100,100,123,0.44,0.44843,0.914989,-0.0628215,0.70789,0.805936,0.00124969,nan,nan,nan,1,nan,nan,0,0.464694,8,0.884336,0.0366158,0.4,0.352741,ok,ok,no_root,ok,
0.55,100,100,57,0.62,0.636943,0.475826,nan,nan,0.805936,0.00124969,nan,nan,nan,1,nan,nan,0,0.495261,4,0.876319,0.0228693,0.6,0.490531,ok,too_few_pairs,no_root,ok,
0.633333,100,100,25,0.77,0.8,nan,nan,nan,0.805936,0.00124969,nan,nan,nan,1,nan,nan,0,0.618446,2,0.765308,0.000249938,2.3,0.286242,too_few_bins,too_few_pairs,no_root,ok_pool_fallback,failure pool <10: initial segments included
0.716667,100,100,11,0.89,0.900901,nan,nan,nan,0.805936,0.00124969,nan,nan,nan,1,nan,nan,0,0.649382,1,0.551963,0.000249938,3.35,0.24182,too_few_bins,too_few_pairs,no_root,ok_pool_fallback,failure pool <10: initial segments included
42 changes: 42 additions & 0 deletions tests/synthetic.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
"""Hand-built first-passage trajectories with a controlled attempt structure.

Used to exercise the reweighting path end to end: the population is chosen so that the
weighted CV of folding times starts below 1 and crosses 1 at a finite lambda with a
healthy effective sample size (lambda* ≈ 1.1 kT, N_eff ≈ 115 for the defaults), which
does not happen for the Langevin regression data.

n0 trajectories with no failed attempt, folding time ≈ T0
n3 trajectories with 3 failed attempts, folding time ≈ TS (short) or TL (long, fraction f)

Up-weighting trajectories with more failures (the MaxCal tilt) shifts weight to the
over-dispersed k = 3 group, so the CV rises through 1. The construction is a test of
the code path, not a physical model.
"""
import os
import numpy as np

Q_U, Q_EXC, Q_F = 0.1, 0.5, 0.9 # U basin, excursion level (fails), folded


def trajectory(k, T, b=2):
"""Q(t) folding exactly at frame T with k failed excursions to Q_EXC.
Each failure is 2 frames at Q_EXC followed by b frames back in U."""
a = T - 1 - k * (2 + b)
if a < 1:
raise ValueError("folding time too short for k failures")
q = [Q_U] * a + ([Q_EXC, Q_EXC] + [Q_U] * b) * k + [Q_EXC, Q_F]
return np.array(q)


def write_reweight_set(outdir, n0=200, n3=40, f=0.25, T0=50, TS=20, TL=200, seed=0):
rng = np.random.default_rng(seed)
os.makedirs(outdir, exist_ok=True)
nl = int(round(f * n3))
specs = ([(0, T0 + int(rng.integers(-5, 6))) for _ in range(n0)]
+ [(3, TS + int(rng.integers(-3, 4))) for _ in range(n3 - nl)]
+ [(3, TL + int(rng.integers(-20, 21))) for _ in range(nl)])
for i, (k, T) in enumerate(specs):
q = trajectory(k, T)
np.savetxt(os.path.join(outdir, f"s_{i:03d}.xvg"),
np.column_stack([np.arange(q.size, dtype=float), q]), fmt="%.6f")
return specs
24 changes: 24 additions & 0 deletions tests/test_notebook.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
"""The demo notebook must run top to bottom (slow: pytest --runslow).
Executed in static mode (MAXCAL_DEMO_STATIC=1), exactly as the committed copy is rendered."""
import os

import pytest

pytestmark = pytest.mark.slow

nbformat = pytest.importorskip("nbformat")
nbclient = pytest.importorskip("nbclient")
pytest.importorskip("ipykernel")

HERE = os.path.dirname(os.path.abspath(__file__))
NB = os.path.join(HERE, os.pardir, "notebooks", "demo.ipynb")


def test_demo_notebook_executes(monkeypatch):
monkeypatch.setenv("MAXCAL_DEMO_STATIC", "1")
nb = nbformat.read(NB, as_version=4)
nbclient.NotebookClient(nb, timeout=600, kernel_name="python3",
resources={"metadata": {"path": os.path.dirname(NB)}}).execute()
n_fig = sum(1 for c in nb.cells if c.cell_type == "code" for o in c.outputs
if "data" in o and "image/png" in o["data"])
assert n_fig >= 10
21 changes: 21 additions & 0 deletions tests/test_regtest.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,22 @@
"lam_star", "neff_star", "cv_lammax", "kmax"]
# stochastic but seeded columns: loose tolerance, robust to RNG-algorithm changes
LOOSE = {"cv_stitch0": 0.08, "lam_stitch": 0.5}
# reason codes for NaNs: must match exactly
STATUS = ["geom_status", "lag1_status", "reweight_status"]
# stitch_status depends on the stochastic pass/fail near the threshold only through
# "ok" vs "no_pass"; pool availability is deterministic
POOL_STATUS = "stitch_status"


def check_nan_reasons(row):
"""Every NaN in a summary row must come with a non-ok reason code."""
nan = lambda c: row[c] in ("", "nan")
assert nan("geom_p") == (row["geom_status"] != "ok"), row
assert nan("lag1_rho") == (row["lag1_status"] != "ok"), row
if nan("lam_star"):
assert row["reweight_status"] == "no_root", row
if nan("lam_stitch"):
assert row["stitch_status"] in ("empty_pool", "no_pass"), row


def read(path):
Expand Down Expand Up @@ -65,6 +81,8 @@ def test_physics_sanity(run):
assert np.all(np.diff(p) > 0), "success probability must grow as Q‡ approaches Qf"
ref = rows[len(rows) // 2]
assert np.isfinite(as_float(ref["lam_stitch"])), "stitching must reach Poisson statistics"
for r in rows:
check_nan_reasons(r)
tse = read(next(out.glob("tse_frames_qts*.csv")))
assert len(tse) == 100
assert sum(float(t["weight_at_lambda_star"]) for t in tse) == pytest.approx(1.0, abs=1e-4)
Expand All @@ -84,6 +102,9 @@ def test_against_reference(run):
for c in EXACT:
np.testing.assert_allclose(as_float(g[c]), as_float(e[c]), rtol=1e-5,
atol=1e-9, equal_nan=True, err_msg=f"{c} @ Q‡={e['qts']}")
for c in STATUS:
assert g[c] == e[c], f"{c} @ Q‡={e['qts']}: {g[c]} != {e[c]}"
assert (g[POOL_STATUS] == "empty_pool") == (e[POOL_STATUS] == "empty_pool")
for c, tol in LOOSE.items():
np.testing.assert_allclose(as_float(g[c]), as_float(e[c]), atol=tol,
equal_nan=True, err_msg=f"{c} @ Q‡={e['qts']}")
Loading
Loading