Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
68cc90e
research: add real-data robustness, richer PFA and paired learner boo…
devissaputra Oct 21, 2026
716f0fb
ci: run full empirical reproducibility and robustness on GitHub Actions
devissaputra Oct 21, 2026
13cab47
test: verify pooled learner bootstrap and leakage-free enriched PFA f…
devissaputra Oct 21, 2026
cab0ca8
ci: execute full real-data reproduction and robustness in PR with dow…
devissaputra Oct 21, 2026
1005f60
research: freeze independent 2015 test — CONFIRMATORY_PROTOCOL_2015.md
devissaputra Oct 22, 2026
6c94c91
research: freeze independent 2015 test — scripts/confirmatory_2015.py
devissaputra Oct 22, 2026
4c932c6
research: freeze independent 2015 test — .github/workflows/confirmato…
devissaputra Oct 22, 2026
35fb537
research: freeze ASSISTments 2015 external-cohort test outcome
github-actions[bot] Oct 22, 2026
43a119b
research: freeze disjoint learner-cohort validation — CONFIRMATORY_ST…
devissaputra Oct 22, 2026
d7dc193
research: freeze disjoint learner-cohort validation — scripts/confirm…
devissaputra Oct 22, 2026
a924f1f
research: freeze disjoint learner-cohort validation — .github/workflo…
devissaputra Oct 22, 2026
2df2f46
research: record untouched cohort confirmatory findings
github-actions[bot] Oct 22, 2026
f811ad8
research: document source-audited independent-cohort replication and …
devissaputra Oct 22, 2026
6766472
paper: integrate independently verified 2015 cohort replication into …
devissaputra Oct 22, 2026
f983265
paper: add documented PROVENANCE_NOVELTY_AUDIT_20261022.md
devissaputra Oct 22, 2026
b1dab5e
paper: add documented FINAL_QA_RESULTS_20261022.json
devissaputra Oct 22, 2026
a5ad580
paper: correct outdated research overview and link integrated 2015 pr…
devissaputra Oct 22, 2026
455e328
ci: quote workflow name to repair YAML parse error
devissaputra Oct 22, 2026
b761cb5
paper: integrate pilot diagnostics and confirmed 2015 results in manu…
devissaputra Oct 22, 2026
fe43081
paper: establish updated manuscript as canonical LaTeX main.tex
devissaputra Oct 22, 2026
57d7b66
paper: add reproducible figure generator from executed metrics
devissaputra Oct 22, 2026
7e4683b
paper: document canonical manuscript build and author review gates
devissaputra Oct 22, 2026
b1f4939
paper: link canonical main.tex and build instructions
devissaputra Oct 22, 2026
29c6019
research: investigate corrected 2009 source parity (scripts/audit_cor…
devissaputra Oct 22, 2026
f25d1be
research: investigate corrected 2009 source parity (.github/workflows…
devissaputra Oct 22, 2026
b276fca
research: archive corrected 2009 versus mirror provenance audit
github-actions[bot] Oct 22, 2026
5270606
research: frozen corrected-2009 sensitivity CORRECTED_2009_REANALYSIS…
devissaputra Oct 22, 2026
9592c30
research: frozen corrected-2009 sensitivity scripts/corrected_2009_re…
devissaputra Oct 22, 2026
7651f5d
research: frozen corrected-2009 sensitivity .github/workflows/correct…
devissaputra Oct 22, 2026
6017da7
research: record exploratory corrected-2009 source sensitivity
github-actions[bot] Oct 22, 2026
996a973
paper: finalize official corrected-2009 parity audit and exploratory …
devissaputra Oct 22, 2026
303c046
paper: finalize FINAL_RESEARCH_AUDIT_20261022.md
devissaputra Oct 22, 2026
6c3fd64
paper: finalize FINAL_RELEASE_QA_20261022.json
devissaputra Oct 22, 2026
3cd2e02
paper: finalize ARXIV_METADATA_FOR_AUTHOR_REVIEW.md
devissaputra Oct 22, 2026
3d682c2
paper: update project overview with proven corrected-source discrepan…
devissaputra Oct 22, 2026
4dd3dd4
research: prospectively freeze 2026 stage-3 history-budget matched-in…
devissaputra Oct 26, 2026
9bed80e
research: implement same-target count/order/GRU history-budget study …
devissaputra Oct 26, 2026
9f54dd1
ci: execute fresh 2015 cohort matched-history budget study and archiv…
devissaputra Oct 26, 2026
776f3d1
paper: draft 2026 methodological research upgrade with bounded novelt…
devissaputra Oct 26, 2026
bc89bbb
ci: distinguish split-level 600 learners from 582 evaluable with a pa…
devissaputra Oct 26, 2026
26ae8e0
protocol: disclose 18 single-event test learners and correction of re…
devissaputra Oct 26, 2026
9c20400
research: archive matched-history budget empirical results
github-actions[bot] Oct 26, 2026
c87230e
paper: integrate verified matched-history experiment into 2026 method…
devissaputra Oct 26, 2026
06f0f37
paper: archive numerical and clean-room manuscript QA 41 checks
devissaputra Oct 26, 2026
136696a
figures: regenerate matched-history budget Brier figure from verified…
devissaputra Oct 26, 2026
901ba2d
paper: document actual fresh-cohort matched-window findings, correcte…
devissaputra Oct 26, 2026
28817c0
paper: mark pending-results draft as archived, point to executed manu…
devissaputra Oct 26, 2026
a8a53cc
paper: final 2026 author-review QA, verified DOI corrections and natu…
devissaputra Oct 26, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions .github/workflows/arxiv-verification.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
name: "arXiv verification: real-data robustness"
on:
push:
branches:
- research/kt-arxiv-verification-20261022
workflow_dispatch:
permissions:
contents: write
jobs:
verify:
if: github.actor != 'github-actions[bot]'
runs-on: ubuntu-latest
timeout-minutes: 120
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Install pinned scientific packages
run: |
python -m pip install --upgrade pip
pip install -r requirements-repro.txt
pip install pytest
- name: Run repository unit tests
run: PYTHONPATH=src python -m pytest -q
- name: Reproduce original real-data experiment
run: PYTHONPATH=src python src/run_experiment.py
- name: Run enhanced baselines and extended GRU on actual data
run: PYTHONPATH=src python scripts/robustness_extension.py --epochs 25 --patience 4 --boots 2000 --seeds 13 42 73
- name: Validate empirical outputs
run: |
python - <<'PY'
import json
from pathlib import Path
original=json.loads(Path("results/metrics.json").read_text())
extension=json.loads(Path("results/robustness_extension.json").read_text())
assert original["dataset"]["n_learners"]==extension["dataset"]["n_learners"]
assert original["dataset"]["n_interactions"]==extension["dataset"]["n_interactions"]
assert extension["status"]=="complete_exploratory_real_data"
for run in extension["extended_gru"].values():
assert len(run["training_history"])>=1
assert run["paired_vs_pfa"]["n_learners"]==623
assert run["paired_vs_stronger_pfa"]["n_interactions"]==41228
print("REAL-DATA VERIFICATION PASSED: numeric artifacts match sample identifiers")
PY
- name: Upload verification results
if: always()
uses: actions/upload-artifact@v4
with:
name: kt-arxiv-verification
path: |
results/metrics.json
results/summary.md
results/robustness_extension.json
results/figures/
if-no-files-found: warn
- name: Commit empirical records to verification branch
run: |
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add results/metrics.json results/summary.md results/figures/ results/robustness_extension.json
if git diff --cached --quiet; then
echo "No changes to empirical artifacts"
else
git commit -m "research: record rerun original and extended real-data results"
git push origin HEAD:research/kt-arxiv-verification-20261022
fi
47 changes: 47 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,50 @@ jobs:
pip install -r requirements.txt
pip install pytest
- run: PYTHONPATH=src python -m pytest -q

empirical-arxiv-check:
if: github.event_name == 'pull_request' && github.head_ref == 'research/kt-arxiv-verification-20261022'
runs-on: ubuntu-latest
timeout-minutes: 180
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with:
python-version: '3.11'
- name: Install reproducibility environment
run: |
python -m pip install --upgrade pip
pip install -r requirements-repro.txt
pip install pytest
- name: Rerun original experiment with real ASSISTments mirror
run: PYTHONPATH=src python src/run_experiment.py
- name: Execute extended GRU and richer PFA robustness
run: PYTHONPATH=src python scripts/robustness_extension.py --epochs 25 --patience 4 --boots 2000 --seeds 13 42 73
- name: Validate outputs against real-data identifiers
run: |
python - <<'PY'
import json
from pathlib import Path
original=json.loads(Path("results/metrics.json").read_text())
extension=json.loads(Path("results/robustness_extension.json").read_text())
assert original["dataset"]["n_learners"]==extension["dataset"]["n_learners"]==4148
assert original["dataset"]["n_interactions"]==extension["dataset"]["n_interactions"]==274331
assert original["dataset"]["parquet_sha256"]==extension["dataset"]["parquet_sha256"]
assert extension["status"]=="complete_exploratory_real_data"
assert len(extension["extended_gru"])==3
for run in extension["extended_gru"].values():
assert run["paired_vs_pfa"]["n_learners"]==623
assert run["paired_vs_stronger_pfa"]["n_interactions"]==41228
print("Original + robustness real-data results: numerical validation passed")
PY
- name: Upload numerical evidence for manuscript update
if: always()
uses: actions/upload-artifact@v4
with:
name: assistments-arxiv-reproduction
path: |
results/metrics.json
results/summary.md
results/robustness_extension.json
results/figures/
if-no-files-found: warn
60 changes: 60 additions & 0 deletions .github/workflows/confirmatory-2015-stage2.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
name: External cohort confirmation — second untouched ASSISTments 2015 cohort
on:
push:
branches:
- research/kt-fresh-cohort-20261022
paths:
- CONFIRMATORY_STAGE2_PROTOCOL_2015.md
- scripts/confirmatory_2015_stage2.py
- .github/workflows/confirmatory-2015-stage2.yml
workflow_dispatch:
permissions:
contents: write
jobs:
second-cohort:
runs-on: ubuntu-latest
timeout-minutes: 120
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Install frozen computational packages
run: |
python -m pip install -r requirements-repro.txt
python -m pip install 'gdown>=5.2,<6'
- name: Compile and inspect frozen code
run: python -m py_compile scripts/confirmatory_2015.py scripts/confirmatory_2015_stage2.py
- name: Run untouched population study
run: PYTHONPATH=src python scripts/confirmatory_2015_stage2.py
- name: Check full independent source and test outputs
run: |
python - <<'PY'
import json
from pathlib import Path
a=json.loads(Path('results/confirmatory_assist2015_stage2.json').read_text())
assert a['data']['sha256']=='75e1f46131d5897a388c82411bc3032279899cb1f1e66a9043354d41234407a0'
assert a['data']['stage2_intersection_stage1']==0
assert a['data']['stage1_excluded_learners']==5000
assert a['data']['stage2_sampled_learners']==5000
assert sum(a['protocol'][x] for x in ('train_learners','validation_learners','test_learners'))==5000
assert a['paired_brier']['n_learners']==750
assert a['paired_brier']['n_bootstrap']==2000
print('Recorded status:',a['status'],'directional replication:',a['directional_replication_confirmed'])
PY
- uses: actions/upload-artifact@v4
if: always()
with:
name: assistments2015-stage2-validation
path: results/confirmatory_assist2015_stage2.json
if-no-files-found: warn
- name: Commit frozen results (not raw student records)
if: success()
run: |
git config user.name 'github-actions[bot]'
git config user.email '41898282+github-actions[bot]@users.noreply.github.com'
git add results/confirmatory_assist2015_stage2.json
if ! git diff --cached --quiet; then
git commit -m 'research: record untouched cohort confirmatory findings'
git push origin HEAD:research/kt-fresh-cohort-20261022
fi
62 changes: 62 additions & 0 deletions .github/workflows/confirmatory-2015.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
name: External cohort confirmation — ASSISTments 2015
on:
push:
branches:
- research/kt-fresh-cohort-20261022
paths:
- scripts/confirmatory_2015.py
- .github/workflows/confirmatory-2015.yml
- CONFIRMATORY_PROTOCOL_2015.md
- scripts/robustness_extension.py
- src/run_experiment.py
workflow_dispatch:
permissions:
contents: write
jobs:
external-cohort:
runs-on: ubuntu-latest
timeout-minutes: 120
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install verified environment and Google Drive downloader
run: |
python -m pip install -r requirements-repro.txt
python -m pip install 'gdown>=5.2,<6' pytest
- name: Compile frozen confirmatory analysis
run: |
python -m py_compile scripts/confirmatory_2015.py
- name: Run independent-cohort analysis
run: PYTHONPATH=src python scripts/confirmatory_2015.py
- name: Validate output schema and data isolation
run: |
python - <<'PY'
import json
from pathlib import Path
a = json.loads(Path('results/confirmatory_assist2015.json').read_text())
assert a['protocol']['sample_learners'] == 5000
assert sum(a['protocol'][k] for k in ('train_learners','validation_learners','test_learners')) == 5000
assert a['brier_comparison']['n_learners'] == a['protocol']['test_learners']
assert a['brier_comparison']['n_bootstrap'] == 2000
assert len(a['data']['source_sha256']) == 64
print('External-cohort result recorded. Confirmation:', a['directional_replication_confirmed'])
PY
- name: Upload results and logs even if run fails
if: always()
uses: actions/upload-artifact@v4
with:
name: knowledge-tracing-2015-confirmation
path: results/confirmatory_assist2015.json
if-no-files-found: warn
- name: Commit metrics to branch without changing previous paper results
if: success()
run: |
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add results/confirmatory_assist2015.json
if ! git diff --cached --quiet; then
git commit -m "research: freeze ASSISTments 2015 external-cohort test outcome"
git push origin HEAD:research/kt-fresh-cohort-20261022
fi
55 changes: 55 additions & 0 deletions .github/workflows/corrected-2009-reanalysis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
name: "Exploratory corrected-source 2009 GRU versus enriched PFA"
on:
push:
branches:
- research/kt-fresh-cohort-20261022
paths:
- scripts/corrected_2009_reanalysis.py
- .github/workflows/corrected-2009-reanalysis.yml
- CORRECTED_2009_REANALYSIS_PROTOCOL.md
workflow_dispatch:
permissions:
contents: write
jobs:
reanalysis:
runs-on: ubuntu-latest
timeout-minutes: 100
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Install frozen numeric and raw-source packages
run: |
python -m pip install -r requirements-repro.txt
python -m pip install 'gdown>=5.2,<6'
- name: Inspect Python code
run: python -m py_compile scripts/audit_corrected_2009.py scripts/corrected_2009_reanalysis.py
- name: Analyze corrected official-source data without redistributing student records
run: PYTHONPATH=src python scripts/corrected_2009_reanalysis.py
- name: Check required source-audit provenance and output schema
run: |
python - <<'PY'
import json
r=json.load(open('results/corrected_2009_sensitivity.json'))
assert r['status'] in ('completed_exploratory_corrected_source','inconclusive_diagnostics','blocked_or_failed')
if r['status']=='completed_exploratory_corrected_source':
assert r['source']['corrected_sha256']=='162ef8d2d28bcbfea6591a282994062bd8d5eaa00636544292a0d268dca6e5da'
assert r['paired_brier']['n_bootstrap']==2000
print('Corrected-source research status:',r['status'])
PY
- uses: actions/upload-artifact@v4
if: always()
with:
name: corrected-2009-aggregate-sensitivity
path: results/corrected_2009_sensitivity.json
- name: Archive aggregate outcome in research branch
if: success()
run: |
git config user.name 'github-actions[bot]'
git config user.email '41898282+github-actions[bot]@users.noreply.github.com'
git add results/corrected_2009_sensitivity.json
if ! git diff --cached --quiet; then
git commit -m 'research: record exploratory corrected-2009 source sensitivity'
git push origin HEAD:research/kt-fresh-cohort-20261022
fi
71 changes: 71 additions & 0 deletions .github/workflows/matched-history-budget-2026.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
name: "Paper 1 - 2026 matched-history-budget experiment"
on:
push:
branches:
- research/kt-methodological-2026-history-budget
paths:
- METHODOLOGICAL_2026_PROTOCOL.md
- scripts/matched_history_budget_2026.py
- .github/workflows/matched-history-budget-2026.yml
workflow_dispatch:
permissions:
contents: write
jobs:
matched-budget:
if: github.actor != 'github-actions[bot]'
runs-on: ubuntu-latest
timeout-minutes: 120
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Install pinned scientific stack
run: |
python -m pip install -r requirements-repro.txt
python -m pip install 'gdown>=5.2,<6'
- name: Code compilation and causal prefix smoke checks
run: |
python -m py_compile scripts/matched_history_budget_2026.py
PYTHONPATH=src python scripts/matched_history_budget_2026.py --smoke
- name: Execute same-target independent-cohort budget comparison
run: PYTHONPATH=src python -u scripts/matched_history_budget_2026.py
- name: Validate learner and calculation invariants
run: |
python - <<'PY'
import json
a=json.load(open("results/matched_history_budget_2026.json"))
assert a["status"]=="passed_all_computational_gates",a.get("diagnostic_failures")
assert a["source"]["source_sha256"]=="75e1f46131d5897a388c82411bc3032279899cb1f1e66a9043354d41234407a0"
assert a["source"]["cohort_overlap_previous"]==0
assert a["source"]["stage3_learners"]==4000
assert a["design"]["test_learners"]==600
assert set(a["comparisons"])=={"1","20"}
for b in a["comparisons"].values():
assert set(b["models"])=={"window_count","ordered_logistic","window_gru"}
for x in b["models"].values():
assert 0<x["brier"]<1 and x["n"]>10000
for x in b["paired_GRU_minus"].values():
assert 400 <= x["n_learners"] <= 600 and x["n_bootstrap"]==2000
assert x["n_interactions"]==b["models"]["window_gru"]["n"]
print("Verified fresh learner cohort and history-parity output gates.")
PY
- name: Upload aggregate evidence even if a research gate fails
uses: actions/upload-artifact@v4
if: always()
with:
name: matched-history-budget-2026-aggregate
path: results/matched_history_budget_2026.json
if-no-files-found: warn
- name: Archive de-identified aggregate results
if: success()
run: |
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add results/matched_history_budget_2026.json
if ! git diff --cached --quiet; then
git commit -m "research: archive matched-history budget empirical results"
git push origin HEAD:research/kt-methodological-2026-history-budget
fi
Loading
Loading