Version and CI note (7 November 2026): The later 6 November manuscript PDF is not yet established as line-by-line identical to
paper/main_methodological_2026.tex. Workflow run 40310733288 completed the calculations and numerical gates but failed on the final non-fast-forward Git push. A proposed workflow fix and the verification boundaries are documented in the evidence reconciliation note. Do not characterize the entire run as a successful CI job.
Current working paper: A Source-Correction and Matched-Information Audit on ASSISTments 2009–2015, Devis Wawan Saputra (2026). This is a working manuscript for final author review, not a peer-reviewed publication. The repository also preserves the earlier ASSISTments 2009 benchmark documented below.
The paper asks how model rankings change when the source version is corrected and models receive the same maximum number of prior learner events. In a 2009 source-provenance comparison, 817 of 4,148 shared learners had different normalized ordered skill-response sequences between the corrected official release and a pinned third-party mirror. This demonstrates non-equivalence; it does not identify every altered original event.
A separate, nonoverlapping ASSISTments 2015 stage-3 cohort contained 4,000 sampled learners (2,800 train, 600 validation, 600 test). Of the test learners, 582 had evaluable post-first-response targets: 20,592 identical test responses across all six model/window combinations. For 20 available prior events, Brier scores were 0.176674 (count-based logistic regression), 0.173885 (order-aware logistic regression), and 0.175140 (GRU). The GRU-minus-order-aware contrast was +0.001255, with learner-cluster bootstrap 95% interval [0.000075, 0.002426]. Lower Brier is better. Model rankings depend on comparator and history representation; this does not establish a pure architectural or causal effect.
- Full methodological manuscript (LaTeX)
- Frozen 2015 matched-history protocol
- 2015 matched-history experiment script
- Executed results (JSON)
- 2009 corrected-source audit script and results
- Executed research summary
- Figure-generation script and document/numeric QA record
- Successful matched-history GitHub Actions run
For the pinned computational environment, install requirements-repro.txt and gdown>=5.2,<6, then run from the repository root:
PYTHONPATH=src python scripts/audit_corrected_2009.py
PYTHONPATH=src python scripts/matched_history_budget_2026.pyThese scripts retrieve their declared public sources and write aggregate evidence; they are not claimed to have been independently rerun in this README update. Raw student-level datasets are not committed. Interpretation is limited to the evaluated historical cohorts, protocol, models, and source-processing choices.
This study asks how much predictive value different forms of learner history add when the evaluation is kept genuinely learner-disjoint. It uses 4,148 ASSISTments 2009 learners and 274,331 interactions, splitting learners rather than individual attempts into 70% training, 15% validation, and 15% final test partitions. Under one frozen protocol, it compares a global prior, a skill prior, fixed and validation-tuned Bayesian Knowledge Tracing, a PFA-style logistic model based on prior learner-skill successes and failures, and a compact GRU sequence model. Every stateful prediction is generated before the response being predicted is observed, preventing the current answer from leaking into the model state.
On 623 held-out learners, the seed-42 GRU achieves ROC-AUC 0.7470 and Brier score 0.1800, compared with 0.6982 and 0.1976 for PFA. The three GRU seeds average ROC-AUC 0.7471 with sample SD 0.0008. These results support better response prediction under this historical benchmark; they do not establish instructional benefit or justify high-stakes interpretations of a learner’s knowledge.
- Calculations, evidence and verification scope
- Figure sources and exact numerical paths
- Working paper
- Data and provenance
Scope of the earlier benchmark review: That review inspected stored empirical results without independently rerunning the complete original experiment. The subsequent 2026 source-correction and matched-history evidence is documented separately above.
Research Bundle · AI in Education · learner modeling · empirical knowledge tracing
A reproducible, learner-disjoint comparison of transparent and recurrent next-response models on real ASSISTments 2009 sequences. The benchmark compares simple priors, fixed and validation-tuned Bayesian Knowledge Tracing, a PFA-style logistic model, and a compact GRU knowledge tracer under one controlled protocol.
This repository is designed to make the evidence chain inspectable: source provenance, leakage safeguards, model-selection boundaries, test metrics, uncertainty, error analysis, ethics, and generated research artifacts are all kept in the repository.
- Do stateful learner models improve held-out next-response probability quality over global and skill priors?
- Does validation-tuned BKT improve over a fixed parameterization?
- Does a PFA-style model using only prior learner-skill successes and failures improve probability quality?
- Does a compact GRU knowledge tracer add useful sequence information beyond transparent baselines?
- Do conclusions differ between a learner's first encounter with a skill key and repeated encounters?
- Are Brier-score differences robust when uncertainty is resampled at the learner level?
- How stable is the recurrent result across prespecified random seeds?
The executable adapter retrieves the public Atomi/ASSISTments2009 sequence representation pinned to revision:
c72a664a9693547fb206652ed2ce18e62d320c7d
The runner records the exact parquet SHA-256, learner count, unique learner count, and interaction count. The mirror is a retrieval representation; the original ASSISTments collection remains the scientific data source. See DATA.md and docs/dataset_card.md.
Raw learner data are not committed to the repository.
Before partitioning, the runner requires exactly one serialized row per user_id. Duplicate learner rows raise an error.
After that validation:
- 70% of learners are assigned to training;
- 15% to validation;
- 15% to the final test set;
- the split seed is 42;
- no learner crosses partitions;
- every stateful probability is emitted before the response being predicted is observed.
Training learners fit priors, PFA, GRU parameters, and the GRU skill vocabulary. Validation learners are used for BKT parameter selection and GRU early stopping. Final test learners are reserved for evaluation.
- Global prior — training correctness rate.
- Skill prior — training skill rate with global fallback.
- Fixed BKT — explicit fixed Bayesian Knowledge Tracing parameters.
- Validation-tuned BKT — parameters selected from the frozen validation grid.
- PFA-style logistic model — skill identity plus prior learner-skill successes and failures.
- Compact GRU knowledge tracer — recurrent sequence model with documented prior fallback.
The GRU is a compact recurrent benchmark, not a claim of reproducing every canonical Deep Knowledge Tracing implementation. The full empirical run uses seeds 13, 42, and 73 and records both individual runs and repeated-seed summary statistics.
Modern attention-based KT models are discussed in docs/related_work.md as extensions; they are not claimed as implemented models.
The final learner-disjoint test set reports:
- ROC-AUC;
- average precision;
- Brier score;
- log loss;
- ECE-10;
- first-seen skill for learner versus repeated-skill slices;
- learner-block bootstrap Brier differences against the skill-prior baseline;
- repeated-seed GRU stability;
- high-error skill diagnostics with minimum support.
Terminology: first_seen_skill_for_learner means the first occurrence of the observed skill key in that learner's sequence. It does not mean the skill is globally unseen in training.
The machine-readable source of truth is results/metrics.json. The human-readable table in results/summary.md, manuscript results in paper/results.md, and measured calibration figure are generated by the same runner.
For the closest reproduction of the successful empirical GitHub Actions environment:
python -m venv .venv
source .venv/bin/activate
pip install -r requirements-repro.txt
pip install pytest==9.1.1
PYTHONPATH=src python -m pytest -q
PYTHONPATH=src python src/run_experiment.pyFor normal development against bounded compatible package versions, use requirements.txt.
See REPRODUCIBILITY.md for the environment boundary and reproducibility notes.
| Evidence | Location |
|---|---|
| Research questions and protocol | docs/research_protocol.md |
| Dataset provenance and split unit | DATA.md |
| Dataset card | docs/dataset_card.md |
| Model / benchmark card | reports/model_card.md |
| Related work and benchmark positioning | docs/related_work.md |
| Executable experiment | src/run_experiment.py |
| Core BKT implementation | src/knowledge_tracing_benchmark/core.py |
| Automated tests | tests/ |
| Machine-readable results | results/metrics.json |
| Generated result summary | results/summary.md |
| Manuscript draft | paper/paper.md |
| Ethics and non-claims | ETHICS.md |
| Evidence contract | RESEARCH_BUNDLE.md |
Knowledge-tracing probabilities are model-dependent predictive summaries. They are not direct measurements of intelligence, motivation, effort, disability, potential, or stable mastery.
Better held-out prediction does not by itself demonstrate better teaching, causal learning improvement, fairness, or suitability for automated educational decisions.
This repository therefore presents a controlled empirical benchmark, not a deployment claim and not a state-of-the-art leaderboard claim.
For a fast technical review:
README → DATA → research protocol → experiment runner → tests → generated results → model card → ethics → manuscript
The central question for review is not whether one model has the highest single score, but whether the comparison is reproducible, leakage-resistant, appropriately scoped, and supported by inspectable evidence.