Card-not-present fraud detection that is graded the way production grades it: trained on the past, scored on a future it has never seen.
A stolen card number leaves no physical trace, only a swipe that never happened. phantom-swipe reconstructs the card behind each transaction, compares every purchase with that card's own history, and ranks transactions with a blended, calibrated ensemble of XGBoost, LightGBM and CatBoost.
Data: the 590,540 labelled transactions of the IEEE-CIS Fraud Detection dataset (3.5% fraud, 182 days). The model trains on the first 80% of the timeline (472,432 transactions) and is scored once on the last 20% (118,108 transactions, about six weeks).
| Model | ROC AUC (95% CI) | PR AUC | Fraud caught at 1% false alarms | Fraud caught reviewing top 2% |
|---|---|---|---|---|
| Raw columns, one XGBoost | 0.9046 (0.898–0.910) | 0.543 | 46.1% | 41.7% |
| phantom-swipe, frozen | 0.9344 (0.931–0.938) | 0.618 | 54.5% | 46.8% |
| phantom-swipe, batch | 0.9457 (0.941–0.949) | 0.664 | 58.8% | 48.4% |
The two phantom-swipe rows are the same model evaluated under two deployment assumptions:
- frozen: every group statistic is fixed at the training cutoff. This is a model deployed once and left alone for six weeks.
- batch: group statistics may also use the unlabelled transactions of the period being scored (how often a card appears, its usual amount). This matches scoring a whole period at once, which is how the Kaggle competition worked. Labels of the scored period are never used.
In the batch setting, reviewing the 1% highest-scored transactions yields 92% real fraud; reviewing the top 10% catches 82% of fraud cases and 81% of the fraud value.
- These are not Kaggle leaderboard scores. The competition's test labels were never released, so everything here is measured on a held-out slice of the labelled data. The winning private leaderboard AUC was about 0.946, on a later and different test period; the numbers are not directly comparable.
- The same raw-column model scores 0.972 under a shuffled split, against 0.905 under the chronological one. A random split lets the model see each card's future, and that 0.07 gap is pure optimism. Every number in the table above uses the chronological split.
Ranking quality declines as the holdout moves away from the training cutoff, most visibly for the frozen variant (0.954 in week one, 0.923 in week six). Refreshing the group statistics recovers about half of that loss without any retraining.
Scores are calibrated with isotonic regression fitted on out-of-fold predictions. The batch variant is well calibrated on the holdout (expected calibration error 0.005). The frozen variant is over-confident in the 1–10% range (ECE 0.013, worse than its uncalibrated 0.007): its features drift after the cutoff, and a calibrator fitted on the training period does not follow.
Half of the top twenty features are engineered card-level statistics (uid_*,
card1_addr1_*). Importance is spread thin: the top feature carries under 4%
of total gain.
Full metrics, including per-learner scores and the review curve, are in
docs/results/metrics_full.json.
- Rebuild the card. The dataset has no customer id.
D1is "days since the card was first seen", sotransaction day − D1is constant for a card. Combined withcard1andaddr1it gives a pseudo-identity (217,850 of them). It is used only to compute features, never as a model input. - Compare each purchase with its card's past. Prior transaction count, seconds since the previous purchase, amount relative to the card's running mean, purchases so far today, device change. These look only backwards, so they cannot leak.
- Describe each card with label-free statistics. Counts, amount mean and spread, distinct devices and email domains per card. Estimated on training rows only (frozen) or on all rows (batch).
- Prune redundancy. The 339 anonymised
Vcolumns are grouped by shared missing-value pattern and de-correlated, keeping 131–141. - Validate forward in time. Five folds, one per 30-day block of the training period, each validated on its own block. Blend weights (in log-odds space) and the calibrator are fitted on out-of-fold predictions; the holdout is scored once at the end.
Leakage guarantees are enforced by tests (tests/test_features.py): features
are identical whether labels are present, flipped or absent, and training-row
features in frozen mode are identical whether or not the future rows exist.
git clone https://github.com/manchelov9/phantom-swipe.git
cd phantom-swipe
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
phantom-swipe run # trains on the bundled simulated sample
pytest -qThe repository ships data/sample/, 12,000 simulated transactions with the
real schema, so every command runs out of the box. Numbers measured on it are
meaningless; it exists to exercise the code.
The real dataset is owned by Vesta Corporation and may not be redistributed, so
it is not in this repository. To reproduce the results, accept the rules on the
competition page, place
train_transaction.csv and train_identity.csv in data/raw/, and run:
phantom-swipe run --data data/raw --out runs/full --preset fullThis took 28 minutes on an RTX 5070 with 16 CPU threads. Without a GPU the code falls back to CPU and is considerably slower.
- One dataset, one six-week holdout. The confidence intervals cover sampling noise, not how the model would behave on another merchant or another year.
- Labels in this dataset mark a card's later transactions as fraud after a chargeback, which rewards recognising a card more than recognising an attack.
- The pseudo-identity is a heuristic and merges or splits some real cards.
- The frozen variant's probabilities need recalibration as it ages.
- Hyperparameters were set by hand from common practice and not tuned.
- No fairness or adverse-impact analysis has been done; the fields are anonymised, so it cannot be done from this data.
src/phantom_swipe/
data.py loading, caching, chronological split
features.py card reconstruction and feature engineering
models.py the three learners, blending, calibration
metrics.py recall at fixed false-alarm rate, review curve, ECE, bootstrap CI
pipeline.py time-blocked CV, holdout report, baselines
plots.py report figures
synth.py simulated sample generator
cli.py `phantom-swipe run` / `make-sample`
tests/ leakage guarantees and an end-to-end run
A ground-up rewrite of a hackathon entry, utec-hackaton-fraud-detection.
MIT for the code. The IEEE-CIS data keeps its own terms.




