Add selftest(), clearer dataset docs and errors, ATE row

CHANGELOG.md

@@ -6,6 +6,10 @@ All notable changes to graphdml are listed here. Versions follow
 ## [Unreleased]
 
 ### Added
+- `graphdml.selftest()` and `make_signal_check()`: a built-in check, on synthetic data with
+  planted effects and a null twin that has none, that the estimator finds real signals and
+  does not invent false ones. Reports bias, coverage, power and false-alarm rates against
+  fixed criteria, next to a network-blind baseline.
 - **ATE (total effect)**: the average effect of treating every node vs none, reported first
   in `summary()` and as a `"total"` row in `summary_frame()`, `cluster_summary_frame()` and
   `conf_int()`, with `model.ate_` and `model.total_weights_`. Pass `total=False` for the
@@ -15,6 +19,13 @@ All notable changes to graphdml are listed here. Versions follow
 - `GraphData.with_values(..., feature_names=...)`; covariates may change their number of
   columns.
 - Installation guide, badges and project links in the README and on PyPI.
+- README datasets table now says which datasets need outside data (LastFM downloads on
+  first use, the insurance experiment must be downloaded from openICPSR), and lists
+  `make_toy_graph`.
+
+### Improved
+- `load_insurance_experiment` explains where to get the data when the files are missing,
+  and `fetch_lastfm_asia` explains what to do when the download fails.
 
 ### Changed
 - `summary_frame()` includes a `"total"` row by default when a peer exposure is fitted.

README.md

@@ -135,16 +135,76 @@ per-node predictions and residuals, and never modifies your data. Try it locally
 
 ## Example datasets
 
-| Dataset | Story | Shows |
-|---|---|---|
-| `make_flu_town` | flu shots and sick days among neighbors | direct vs peer effects, network confounding |
-| `make_referral_app` | promo coupons in an app with influencers | choosing the exposure map |
-| `make_classroom_tutoring` | tutoring hours and test scores | continuous treatment, who the estimate is about |
-| `make_homophily_trap` | a course whose true peer effect is zero | a failure mode and how to catch it |
-| `make_lastfm_promo` | a promotion on the real LastFM Asia network | real topology and covariates, known truth |
-| `load_insurance_experiment` | a randomized field experiment with 4,902 farmers | real data (download from openICPSR) |
-
-Every dataset comes with a story card: `print(ds.DESCR)`.
+Six of these ship inside the package. Two need data from outside: one downloads
+automatically, one you download yourself.
+
+| Dataset | Story | Shows | Data needed |
+|---|---|---|---|
+| `make_flu_town` | flu shots and sick days among neighbors | direct vs peer effects, network confounding | none (generated offline) |
+| `make_referral_app` | promo coupons in an app with influencers | choosing the exposure map | none (generated offline) |
+| `make_classroom_tutoring` | tutoring hours and test scores | continuous treatment, who the estimate is about | none (generated offline) |
+| `make_homophily_trap` | a course whose true peer effect is zero | a failure mode and how to catch it | none (generated offline) |
+| `make_toy_graph` | 30 hand-drawn nodes | pictures of focal sets and folds (too small to estimate anything) | none (generated offline) |
+| `make_lastfm_promo` | a promotion on the real LastFM Asia network | real topology and covariates, known truth | **internet on first use** (6.5 MB, automatic) |
+| `load_insurance_experiment` | a randomized field experiment with 4,902 farmers | real data, no known truth | **you download it** (free account) |
+
+The first five are simulated with known true effects, so you can check an estimate against
+the answer. Every dataset comes with a story card: `print(ds.DESCR)`.
+
+### Datasets that need outside data
+
+**LastFM Asia** (`make_lastfm_promo`): the social network and covariates are real, the
+treatment and outcome are simulated. On first use it downloads the network from
+[SNAP](https://snap.stanford.edu/data/feather-lastfm-social.html), checks its checksum and
+caches it in `~/.cache/graphdml` (set `GRAPHDML_DATA` to change the location). After that it
+works offline. If you are offline or behind a firewall, download `lastfm_asia.zip` yourself
+and put it in the cache folder. Please cite Rozemberczki and Sarkar (CIKM 2020), also shown
+in `ds.extras["citation"]`.
+
+**Insurance experiment** (`load_insurance_experiment`): a real randomized experiment with a
+real friendship network, from Cai, de Janvry and Sadoulet (2015). graphdml does not
+redistribute it. Download the replication package from
+[openICPSR project 113593](https://www.openicpsr.org/openicpsr/project/113593/version/V1/view)
+(free account, CC BY 4.0 license), unzip it, and point the loader at the folder that holds
+`0422survey.dta` and `0422allinforawnet.dta`:
+
+```python
+from graphdml.datasets import load_insurance_experiment
+
+ds = load_insurance_experiment("~/Downloads/113593-V1/data/data")
+```
+
+If the files are not found, the error tells you where to get them. See the dataset's story
+card for the published estimates to compare against.
+
+## Does it find real signals? Run the self-test
+
+graphdml ships a check you can run on your own machine. It simulates two worlds with the
+same strong network confounding: one with planted effects (direct +1.0, peer +0.6) and a
+**null twin** with none. It fits the model many times and compares with the known truth.
+
+```python
+import graphdml
+print(graphdml.selftest(n_reps=60))     # about 30 seconds on a 10-core laptop
+```
+
+```
+graphdml self-test: PASSED (60 repetitions per world)
+
+                truth  mean_estimate  coverage  detected  naive_detected
+world   effect
+planted direct  1.000          1.049     0.917     1.000           1.000
+        peer    0.600          0.635     0.983     1.000           1.000
+null    direct  0.000          0.050     0.917     0.083           1.000
+        peer    0.000          0.036     0.967     0.033           1.000
+```
+
+In the planted world graphdml finds both effects every time. In the null world, where
+nothing is happening, it raises a false alarm 3 to 8% of the time (5% is the target), while
+a method that ignores the network reports an effect **every time**, because the
+confounding looks like a treatment effect. The full output also lists the pass criteria.
+You can test your own settings with `graphdml.selftest(estimator=GraphDML(...))`, and use
+the data directly with `make_signal_check()`.
 
 ## Checking an analysis
 

src/graphdml/__init__.py

@@ -18,6 +18,7 @@
     WeightedExposure,
 )
 from graphdml.features import NeighborhoodFeatures, OwnFeatures, PrecomputedFeatures
+from graphdml.selftest import selftest
 
 __version__ = "0.1.0"
 
@@ -37,5 +38,6 @@
     "__version__",
     "negative_control_test",
     "placebo_test",
+    "selftest",
     "two_hop_test",
 ]

src/graphdml/datasets/__init__.py

@@ -6,6 +6,7 @@
 ``make_classroom_tutoring``        continuous treatment; communities; focal-set estimand
 ``make_homophily_trap``            a failure mode (latent homophily) and how to detect it
 ``make_toy_graph``                 a 30-node graph for pictures of focal sets and folds
+``make_signal_check``              planted effects plus a null twin, for graphdml.selftest()
 ``make_lastfm_promo``              semi-synthetic: real LastFM Asia network and covariates
 ``fetch_lastfm_asia``              the raw LastFM Asia network (downloaded and cached)
 ``load_insurance_experiment``      real randomized network experiment (Cai et al. 2015)
@@ -18,6 +19,7 @@
 from graphdml.datasets._fetch import RealNetwork, fetch_lastfm_asia
 from graphdml.datasets._insurance import load_insurance_experiment
 from graphdml.datasets._semisynthetic import make_lastfm_promo
+from graphdml.datasets._signal import make_signal_check
 from graphdml.datasets._stories import (
     make_classroom_tutoring,
     make_flu_town,
@@ -36,5 +38,6 @@
     "make_homophily_trap",
     "make_lastfm_promo",
     "make_referral_app",
+    "make_signal_check",
     "make_toy_graph",
 ]

src/graphdml/datasets/_fetch.py

@@ -66,8 +66,15 @@ def fetch_lastfm_asia(
     if not path.exists():
         if not download:
             raise FileNotFoundError(f"{path} not found and download=False.")
-        with urllib.request.urlopen(_LASTFM_URL, timeout=60) as r:
-            payload = r.read()
+        try:
+            with urllib.request.urlopen(_LASTFM_URL, timeout=60) as r:
+                payload = r.read()
+        except OSError as e:  # URLError and timeouts are OSErrors
+            raise OSError(
+                f"Could not download LastFM Asia from {_LASTFM_URL} ({e}). Check your "
+                f"internet connection, or download lastfm_asia.zip yourself and place it at "
+                f"{path}."
+            ) from e
         if hashlib.sha256(payload).hexdigest() != _LASTFM_SHA256:
             raise OSError("Checksum mismatch for lastfm_asia.zip; refusing to use it.")
         path.write_bytes(payload)

src/graphdml/datasets/_insurance.py

@@ -43,6 +43,9 @@
 _COVARIATES = ["male", "age", "agpop", "ricearea_2010", "literacy", "risk_averse",
                "disaster_prob"]
 
+_REQUIRED_FILES = ("0422survey.dta", "0422allinforawnet.dta")
+_DATA_URL = "https://www.openicpsr.org/openicpsr/project/113593/version/V1/view"
+
 
 def load_insurance_experiment(data_dir: str | os.PathLike) -> GraphDataset:
     """Load the insurance experiment as a :class:`GraphDataset` (no ground truth).
@@ -62,6 +65,14 @@ def load_insurance_experiment(data_dir: str | os.PathLike) -> GraphDataset:
       estimates.
     """
     data_dir = Path(data_dir).expanduser()
+    missing = [f for f in _REQUIRED_FILES if not (data_dir / f).exists()]
+    if missing:
+        raise FileNotFoundError(
+            f"Could not find {', '.join(missing)} in {data_dir}. This dataset is not bundled "
+            "with graphdml: download the replication package (free openICPSR account, CC BY "
+            f"4.0) from {_DATA_URL}, unzip it, and pass the folder that contains the .dta "
+            "files (the 'data/data' folder)."
+        )
     survey = pd.read_stata(data_dir / "0422survey.dta")
     noms = pd.read_stata(data_dir / "0422allinforawnet.dta")
     noms = noms[~((noms["network_missname"] == 1) & noms["network_id"].isna())]

src/graphdml/datasets/_signal.py

@@ -0,0 +1,74 @@
+"""A synthetic dataset with planted effects, and a null twin with none.
+
+Used by :func:`graphdml.selftest` to check, on your machine, that the estimator finds
+real signals and does not invent false ones.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+import numpy as np
+from scipy.special import expit
+from sklearn.utils import check_random_state
+
+from graphdml.data import GraphData
+from graphdml.datasets._base import GraphDataset, load_descr
+from graphdml.exposure import row_normalize
+from graphdml.simulate.graphs import erdos_renyi
+
+__all__ = ["make_signal_check"]
+
+
+def make_signal_check(
+    n: int = 6000,
+    *,
+    direct_effect: float = 1.0,
+    peer_effect: float = 0.6,
+    confounding: float = 1.0,
+    avg_degree: float = 4.0,
+    noise_sd: float = 2.0,
+    random_state: Any = 0,
+) -> GraphDataset:
+    """Planted direct and peer effects, hidden behind strong network confounding.
+
+    Set ``direct_effect=0, peer_effect=0`` for the **null twin**: the same data-generating
+    process with no true effect. Methods that ignore the network still "find" effects in
+    the null twin, because the confounding is strong; a correct method should not.
+
+    ``T`` is binary. Both treatment and outcome depend nonlinearly on a node's own
+    covariates and on its neighbors' (mean and maximum), with strength ``confounding``.
+    The exposure is ``"mean"``: the share of treated neighbors.
+    """
+    rng = check_random_state(random_state)
+    A = erdos_renyi(n, avg_degree, rng)
+    M = row_normalize(A)
+    X = rng.standard_normal((n, 3))
+    nbr_mean = M @ X
+    nbr_max = np.zeros_like(X)
+    has = np.diff(A.indptr) > 0
+    nbr_max[has] = np.maximum.reduceat(X[A.indices], A.indptr[:-1][has], axis=0)
+
+    logit = (
+        -0.2
+        + 0.5 * X[:, 0]
+        + 0.4 * X[:, 1] * X[:, 2]
+        + confounding * (0.9 * nbr_mean[:, 0] + 0.5 * nbr_mean[:, 1])
+    )
+    T = rng.binomial(1, expit(logit)).astype(float)
+    g = (
+        1.0 * X[:, 0]
+        + 0.8 * np.sin(2 * X[:, 1])
+        + 0.5 * X[:, 2] ** 2
+        + confounding * (1.6 * nbr_mean[:, 0] + 1.0 * np.tanh(nbr_max[:, 1]))
+    )
+    Y = g + direct_effect * T + peer_effect * (M @ T) + noise_sd * rng.standard_normal(n)
+    return GraphDataset(
+        name="signal_check" if (direct_effect or peer_effect) else "signal_check_null",
+        data=GraphData(X, T, Y, A, feature_names=["x1", "x2", "x3"]),
+        truth={"ade": direct_effect, "ape": peer_effect},
+        exposure="mean",
+        treatment_name="treated",
+        outcome_name="outcome",
+        DESCR=load_descr("signal_check"),
+    )

src/graphdml/datasets/descr/classroom_tutoring.md

@@ -40,3 +40,8 @@ print("estimate:         ", m.ade_)
 With constant effects (the default) both truths coincide. With varying effects, report the
 estimand honestly: "the average effect among students like those in the focal set", and check
 how the focal set differs from everyone (`m.diagnostics_["mean_degree_focal"]`).
+
+**Known accuracy on this dataset.** Over 200 draws, estimates run slightly toward zero
+(direct about 1.5%, peer about 10%, total about 3%) and 95% intervals cover the truth about
+91-92% of the time, a little under nominal. The signs and sizes are right, and ignoring the
+network is far worse: it gets the sign of the peer effect wrong.

src/graphdml/datasets/descr/signal_check.md

@@ -0,0 +1,26 @@
+# Signal Check (synthetic, planted effects)
+
+**Question.** Does the estimator find a real effect when one exists, and stay quiet when
+none does?
+
+**Teaches.** What a trustworthy estimate looks like. The data come in two versions with the
+same graph structure and the same strong network confounding:
+
+| | Direct effect | Peer effect |
+|---|---|---|
+| **planted** (`make_signal_check()`) | +1.0 | +0.6 (share of treated neighbors, 0 to 100%) |
+| **null twin** (`make_signal_check(direct_effect=0, peer_effect=0)`) | 0 | 0 |
+
+Treatment and outcome both depend nonlinearly on a node's own covariates and on its
+neighbors' (mean and maximum), so adjusting only for own covariates, or ignoring the
+network, produces false signals. In the null twin, a method that ignores the network still
+reports effects, because confounding looks like a treatment effect. A correct method should
+report none.
+
+Run the full check with `graphdml.selftest()`: it repeats both worlds many times and
+reports bias, interval coverage, power and false-alarm rates against fixed criteria.
+
+```python
+import graphdml
+print(graphdml.selftest())
+```