Skip to content

About

AI in Education research bundle benchmarking priors, BKT, PFA, and GRU knowledge tracing on learner-disjoint ASSISTments 2009 data.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

131 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Version and CI note (7 November 2026): The later 6 November manuscript PDF is not yet established as line-by-line identical to paper/main_methodological_2026.tex. Workflow run 40310733288 completed the calculations and numerical gates but failed on the final non-fast-forward Git push. A proposed workflow fix and the verification boundaries are documented in the evidence reconciliation note. Do not characterize the entire run as a successful CI job.

When Knowledge-Tracing Rankings Depend on Available History (2026)

Current working paper: A Source-Correction and Matched-Information Audit on ASSISTments 2009–2015, Devis Wawan Saputra (2026). This is a working manuscript for final author review, not a peer-reviewed publication. The repository also preserves the earlier ASSISTments 2009 benchmark documented below.

Current paper: question, evidence and replication

The paper asks how model rankings change when the source version is corrected and models receive the same maximum number of prior learner events. In a 2009 source-provenance comparison, 817 of 4,148 shared learners had different normalized ordered skill-response sequences between the corrected official release and a pinned third-party mirror. This demonstrates non-equivalence; it does not identify every altered original event.

A separate, nonoverlapping ASSISTments 2015 stage-3 cohort contained 4,000 sampled learners (2,800 train, 600 validation, 600 test). Of the test learners, 582 had evaluable post-first-response targets: 20,592 identical test responses across all six model/window combinations. For 20 available prior events, Brier scores were 0.176674 (count-based logistic regression), 0.173885 (order-aware logistic regression), and 0.175140 (GRU). The GRU-minus-order-aware contrast was +0.001255, with learner-cluster bootstrap 95% interval [0.000075, 0.002426]. Lower Brier is better. Model rankings depend on comparator and history representation; this does not establish a pure architectural or causal effect.

Start here for this 2026 study

For the pinned computational environment, install requirements-repro.txt and gdown>=5.2,<6, then run from the repository root:

PYTHONPATH=src python scripts/audit_corrected_2009.py
PYTHONPATH=src python scripts/matched_history_budget_2026.py

These scripts retrieve their declared public sources and write aggregate evidence; they are not claimed to have been independently rerun in this README update. Raw student-level datasets are not committed. Interpretation is limited to the evaluated historical cohorts, protocol, models, and source-processing choices.


Earlier ASSISTments 2009 benchmark (preserved)

This study asks how much predictive value different forms of learner history add when the evaluation is kept genuinely learner-disjoint. It uses 4,148 ASSISTments 2009 learners and 274,331 interactions, splitting learners rather than individual attempts into 70% training, 15% validation, and 15% final test partitions. Under one frozen protocol, it compares a global prior, a skill prior, fixed and validation-tuned Bayesian Knowledge Tracing, a PFA-style logistic model based on prior learner-skill successes and failures, and a compact GRU sequence model. Every stateful prediction is generated before the response being predicted is observed, preventing the current answer from leaking into the model state.

On 623 held-out learners, the seed-42 GRU achieves ROC-AUC 0.7470 and Brier score 0.1800, compared with 0.6982 and 0.1976 for PFA. The three GRU seeds average ROC-AUC 0.7471 with sample SD 0.0008. These results support better response prediction under this historical benchmark; they do not establish instructional benefit or justify high-stakes interpretations of a learner’s knowledge.

Start here

Study question, data, design and interpretation

Defined calculation and source-linked evidence

Scope of the earlier benchmark review: That review inspected stored empirical results without independently rerunning the complete original experiment. The subsequent 2026 source-correction and matched-history evidence is documented separately above.

Detailed project documentation

CI Empirical Study

Research Bundle · AI in Education · learner modeling · empirical knowledge tracing

A reproducible, learner-disjoint comparison of transparent and recurrent next-response models on real ASSISTments 2009 sequences. The benchmark compares simple priors, fixed and validation-tuned Bayesian Knowledge Tracing, a PFA-style logistic model, and a compact GRU knowledge tracer under one controlled protocol.

This repository is designed to make the evidence chain inspectable: source provenance, leakage safeguards, model-selection boundaries, test metrics, uncertainty, error analysis, ethics, and generated research artifacts are all kept in the repository.

Research questions

  1. Do stateful learner models improve held-out next-response probability quality over global and skill priors?
  2. Does validation-tuned BKT improve over a fixed parameterization?
  3. Does a PFA-style model using only prior learner-skill successes and failures improve probability quality?
  4. Does a compact GRU knowledge tracer add useful sequence information beyond transparent baselines?
  5. Do conclusions differ between a learner's first encounter with a skill key and repeated encounters?
  6. Are Brier-score differences robust when uncertainty is resampled at the learner level?
  7. How stable is the recurrent result across prespecified random seeds?

Real dataset and provenance

The executable adapter retrieves the public Atomi/ASSISTments2009 sequence representation pinned to revision:

c72a664a9693547fb206652ed2ce18e62d320c7d

The runner records the exact parquet SHA-256, learner count, unique learner count, and interaction count. The mirror is a retrieval representation; the original ASSISTments collection remains the scientific data source. See DATA.md and docs/dataset_card.md.

Raw learner data are not committed to the repository.

Leakage safeguards and split protocol

Before partitioning, the runner requires exactly one serialized row per user_id. Duplicate learner rows raise an error.

After that validation:

  • 70% of learners are assigned to training;
  • 15% to validation;
  • 15% to the final test set;
  • the split seed is 42;
  • no learner crosses partitions;
  • every stateful probability is emitted before the response being predicted is observed.

Training learners fit priors, PFA, GRU parameters, and the GRU skill vocabulary. Validation learners are used for BKT parameter selection and GRU early stopping. Final test learners are reserved for evaluation.

Compared models

  1. Global prior — training correctness rate.
  2. Skill prior — training skill rate with global fallback.
  3. Fixed BKT — explicit fixed Bayesian Knowledge Tracing parameters.
  4. Validation-tuned BKT — parameters selected from the frozen validation grid.
  5. PFA-style logistic model — skill identity plus prior learner-skill successes and failures.
  6. Compact GRU knowledge tracer — recurrent sequence model with documented prior fallback.

The GRU is a compact recurrent benchmark, not a claim of reproducing every canonical Deep Knowledge Tracing implementation. The full empirical run uses seeds 13, 42, and 73 and records both individual runs and repeated-seed summary statistics.

Modern attention-based KT models are discussed in docs/related_work.md as extensions; they are not claimed as implemented models.

Evaluation

The final learner-disjoint test set reports:

  • ROC-AUC;
  • average precision;
  • Brier score;
  • log loss;
  • ECE-10;
  • first-seen skill for learner versus repeated-skill slices;
  • learner-block bootstrap Brier differences against the skill-prior baseline;
  • repeated-seed GRU stability;
  • high-error skill diagnostics with minimum support.

Terminology: first_seen_skill_for_learner means the first occurrence of the observed skill key in that learner's sequence. It does not mean the skill is globally unseen in training.

The machine-readable source of truth is results/metrics.json. The human-readable table in results/summary.md, manuscript results in paper/results.md, and measured calibration figure are generated by the same runner.

Reproduce

For the closest reproduction of the successful empirical GitHub Actions environment:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements-repro.txt
pip install pytest==9.1.1
PYTHONPATH=src python -m pytest -q
PYTHONPATH=src python src/run_experiment.py

For normal development against bounded compatible package versions, use requirements.txt.

See REPRODUCIBILITY.md for the environment boundary and reproducibility notes.

Research-bundle evidence map

Evidence Location
Research questions and protocol docs/research_protocol.md
Dataset provenance and split unit DATA.md
Dataset card docs/dataset_card.md
Model / benchmark card reports/model_card.md
Related work and benchmark positioning docs/related_work.md
Executable experiment src/run_experiment.py
Core BKT implementation src/knowledge_tracing_benchmark/core.py
Automated tests tests/
Machine-readable results results/metrics.json
Generated result summary results/summary.md
Manuscript draft paper/paper.md
Ethics and non-claims ETHICS.md
Evidence contract RESEARCH_BUNDLE.md

Interpretation boundary

Knowledge-tracing probabilities are model-dependent predictive summaries. They are not direct measurements of intelligence, motivation, effort, disability, potential, or stable mastery.

Better held-out prediction does not by itself demonstrate better teaching, causal learning improvement, fairness, or suitability for automated educational decisions.

This repository therefore presents a controlled empirical benchmark, not a deployment claim and not a state-of-the-art leaderboard claim.

Professor review path

For a fast technical review:

README → DATA → research protocol → experiment runner → tests → generated results → model card → ethics → manuscript

The central question for review is not whether one model has the highest single score, but whether the comparison is reproducible, leakage-resistant, appropriately scoped, and supported by inspectable evidence.

About

AI in Education research bundle benchmarking priors, BKT, PFA, and GRU knowledge tracing on learner-disjoint ASSISTments 2009 data.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages