diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index b632f4d..4a2bb43 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "idstack", - "version": "3.4.0.0", + "version": "3.4.0.1", "description": "Evidence-based instructional design pipeline. 11 skills backed by peer-reviewed research across 11 domains. Analyze, design, build, review, and export courses with every recommendation citing its evidence tier.", "author": { "name": "Philippos Savvides", diff --git a/CHANGELOG.md b/CHANGELOG.md index 51265fc..de07e80 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,33 @@ # Changelog +## v3.4.0.1 (2026-08-07) + +Accuracy pass over the public surfaces before sharing the project more widely. No skill behavior changes. + +To update: `cd` into your idstack clone, then `git pull && ./setup`. + +### Fixed — three evidence cards claimed better evidence than idstack holds + +- **Two cards on idstack.org advertised T2 for domains whose strongest reference is T3**, and a third advertised T5 for a domain that bottoms out at T4. Needs Analysis showed `T2–T5` when all 7 of its references are T3; Evaluation Models showed `T2–T5` against an actual `T3–T5`; Online Course Quality Frameworks showed `T1–T5` against an actual `T1–T4`. The other eight cards were correct, and every study count already matched. + + Overstating a tier is the one inaccuracy this project cannot ship — labelling evidence honestly is the entire claim. The cards are no longer hand-maintained: `test/check-evidence-cards.py` derives every count and tier span from `evidence/references.md` and smoke-test fails on any disagreement, naming the domain and flagging when a card overstates. Two mutations pin it, one per direction (tier drift, count drift). + +### Fixed — PRIVACY.md did not disclose two outbound calls + +- **`/idstack:course-export` uploads to Canvas and the privacy policy never said so.** PRIVACY.md stated "No data is sent to external servers by idstack" and carved out only `/idstack:course-import` *fetching* from Canvas. But `course-export` POSTs modules, pages, assignments, and discussions to the Canvas instance you point it at. That is your institution's server rather than ours, and it only runs when you invoke the skill and confirm the target course, but an undisclosed upload in a privacy policy is a defect regardless of where the bytes land. +- **The hourly update check was also undisclosed.** `bin/idstack-update-check` runs `git fetch` against this repository. It carries nothing but the fetch and never touches course data, and it only runs for git installs, but it is a network call and now says so. +- Both are listed under Third-party services. Audited by enumerating every outbound host across `bin/`, `templates/`, and `skills/`: the Canvas API and that `git fetch` are the only two. + +### Fixed — the Windows install path could not work + +- **`README.md` told Windows users to open PowerShell and run `./setup`.** `setup` is an extensionless bash script; PowerShell cannot execute it, and no `.ps1` ships. The ZIP instructions now name the shell that works (WSL or Git Bash) and say why PowerShell does not. This lived in the collapsed "Download ZIP" block, the path written for people who cannot use `git clone`. + +### For contributors + +- `ROADMAP.md` claimed "18 mutations" against an actual 24. The count is gone rather than corrected, so it cannot drift a third time; run the suite to see the number. +- `TODOS.md`'s landing-page demo item described a "See it work" transcript section that a redesign had already removed, so the task read as an upgrade to something that no longer existed. Rewritten against the page as it stands. +- v3.4.0.0 was merged but never tagged, so `.github/workflows/release.yml` never fired and GitHub still listed v3.3.0.4 as the latest release. Tagged retroactively at its merge commit. + ## v3.4.0.0 (2026-08-06) idstack is a Claude Code plugin now, and only that. The OpenAI Codex CLI target that shipped in v2.5.0.0 is removed, and the Gemini CLI target that was planned is off the roadmap. Nothing about the 11 skills, the evidence base, the manifest schema, or the report contract changes. diff --git a/PRIVACY.md b/PRIVACY.md index ac482b6..8095e72 100644 --- a/PRIVACY.md +++ b/PRIVACY.md @@ -20,13 +20,20 @@ Designer profile and cross-project learnings are stored locally: - `~/.idstack/profile.yaml` — your experience level preference - `~/.idstack/global/learnings.jsonl` — learnings promoted across projects -No data is sent to external servers by idstack. No analytics, no tracking, no telemetry. +idstack adds no analytics, no tracking, and no telemetry. Two things do reach the network, both only when you ask for them, and both are listed under Third-party services below: the Canvas API calls made by `/idstack:course-import` and `/idstack:course-export`, and an update check against this repository on GitHub. ## Third-party services idstack runs inside Claude Code, which is operated by Anthropic. Your conversation with Claude Code is subject to [Anthropic's privacy policy](https://www.anthropic.com/privacy). idstack itself does not add any data collection beyond what Claude Code already does. -If you use `/idstack:course-import` with the Canvas API, your Canvas access token is used locally to fetch course data. idstack does not store or transmit your token beyond the API call. +**Canvas API.** Two skills talk to the Canvas instance you point them at, using the access token you supply: + +- `/idstack:course-import` **downloads** your course data from Canvas. +- `/idstack:course-export` **uploads** generated course content to Canvas — modules, pages, assignments, and discussions are POSTed to your Canvas instance so they appear in the course. + +Both run only when you invoke that skill and confirm the target course. Your token is used for those API calls and is neither stored by idstack nor sent anywhere else. The receiving Canvas instance is your institution's, not ours. + +**Update check.** On skill startup idstack runs `git fetch` against this repository to see whether a newer version exists, at most once an hour. That is a request to GitHub carrying nothing but the fetch itself; it never uploads your course data. It only runs for git installs, and removing the repo's `.git` directory disables it. ## Questions diff --git a/README.md b/README.md index 86fdcf7..3f70e8a 100644 --- a/README.md +++ b/README.md @@ -123,7 +123,8 @@ Claude clones the repo, runs setup, and confirms the skills are registered. 1. Click the green "Code" button on GitHub, then "Download ZIP" 2. Unzip the file -3. Open Terminal (Mac) or PowerShell (Windows) +3. Open a bash shell: Terminal on macOS or Linux; WSL or Git Bash on Windows. + `./setup` is a bash script, so PowerShell and `cmd` cannot run it. 4. Navigate to the folder: `cd path/to/idstack` 5. Run: `./setup` diff --git a/ROADMAP.md b/ROADMAP.md index 9019ba3..5cf98ba 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -21,7 +21,7 @@ What's coming next for idstack. Priorities are shaped by user feedback. [Tell us - The suite had never run automatically. GitHub Actions runs every suite on each push and pull request, across ubuntu (Python 3.9 + 3.12) and macOS — eight suites at v3.3.0.0, ten today. - `./setup` — the primary deliverable — went from zero coverage to 17 behavioral tests. - `bin/idstack-doctor` and `bin/idstack-status --readiness` gained their first execution coverage in v3.3.0.4. Both are what a user reaches for when something has already gone wrong, and neither had any. -- `test/mutation-test.sh` reintroduces each fixed defect and asserts its guarding test fails, which is how a test that only appeared to test something gets caught. 18 mutations, all guarded. +- `test/mutation-test.sh` reintroduces each fixed defect and asserts its guarding test fails, which is how a test that only appeared to test something gets caught. Every mutation in it is guarded; the suite prints the current count when you run it. ### Install through the Claude Code plugin marketplace (v3.2.0.0) - `./setup` registers idstack as a Claude Code plugin marketplace and installs from there. Recent Claude Code versions stopped discovering plugins from the bare symlink older setups created, so `/idstack:` commands silently never appeared in the slash picker. If that happened to you, pull the latest and re-run `./setup`. diff --git a/TODOS.md b/TODOS.md index 540961f..a6e7886 100644 --- a/TODOS.md +++ b/TODOS.md @@ -90,12 +90,13 @@ vs project-local learnings interact. **Priority:** P2 ## v2: Interactive Landing Page Demo -Upgrade the "See it work" section on idstack.org from static text to an interactive -pre-install demo. The current section shows a text transcript of /course-import and -/course-quality-review. A richer version could include: animated terminal replay -(CSS-only, building on the existing typing animation), clickable pipeline explorer -showing what each of the 11 skills does, or expanded sample output with evidence -citations. The goal is to let prospective users experience the value before installing. +Give idstack.org a pre-install demo. The redesign removed the old "See it work" +transcript section, so today a visitor sees the evidence cards, the pipeline diagram, +and a sample report excerpt, but never watches a skill run. Options: animated terminal +replay (CSS-only, building on the existing typing animation), a clickable pipeline +explorer showing what each of the 11 skills does, or expanded sample output with +evidence citations. The goal is to let prospective users experience the value before +installing. **Why:** CEO review killed a post-install /demo command because IDs always have a course. The real TTHW problem is pre-install: visitors on idstack.org need to understand the value before committing to install. The landing page demo is where that conversion happens. diff --git a/VERSION b/VERSION index 163144c..0ae4a78 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -3.4.0.0 +3.4.0.1 diff --git a/docs/index.html b/docs/index.html index fcdaac8..a716edb 100644 --- a/docs/index.html +++ b/docs/index.html @@ -902,7 +902,7 @@ MIT - v3.4.0.0 — Claude Code only + v3.4.0.1 — Claude Code only

Evidence-based instructional design, transparently cited.

108 peer-reviewed studies across 11 research domains. Every recommendation tagged with its evidence tier — T1 meta-analyses and RCTs through T5 expert opinion. Runs in Claude Code.

@@ -950,7 +950,7 @@

Constructive Alignment & Learning Objectives

Needs Analysis

Three layers — organizational, task, learner — analyzed before any content gets written. When this step is skipped, courses tend to teach what the designer found interesting rather than what the audience came for.

-

7 studiesT2–T5

+

7 studiesT3

@@ -980,7 +980,7 @@

Learner Analysis & Differentiation

Evaluation Models

Evaluation only measures what it was designed to measure, so the design has to happen alongside the course. Kirkpatrick's four levels (reaction, learning, behavior, results) map what's possible; most courses report only the first.

-

8 studiesT2–T5

+

8 studiesT3–T5

@@ -992,7 +992,7 @@

Rapid Prototyping & Design-Based Research

Online Course Quality Frameworks

Quality Matters' eight standards and the Community of Inquiry framework (teaching, social, cognitive presence) are the two rubrics with the most published validation studies behind them. idstack's quality audit runs against both.

-

10 studiesT1–T5

+

10 studiesT1–T4

@@ -1146,10 +1146,11 @@

v3.4.0.0 — idstack is a Claude Code plugin.

-

August 6, 2026 · v3.4.0.0

+

August 6, 2026 · v3.4.0.0 · patched through v3.4.0.1

One host, supported properly.

idstack shipped an OpenAI Codex CLI build from v2.5.0.0 through v3.3.0.4. It is removed. Every skill, the evidence base, the project manifest, and the HTML reports are unchanged — idstack now targets Claude Code and nothing else, which is the only place it was ever tested end to end.

To update: cd into your idstack clone, then git pull && ./setup. If you had the Codex build installed, ./setup can no longer clean it up — the release notes carry the two paths to remove.

+

Patched in v3.4.0.1 (August 7): three evidence cards on this page overstated their domain's evidence tier. They are derived from evidence/references.md now, and the test suite fails if they ever disagree again.

diff --git a/test/check-evidence-cards.py b/test/check-evidence-cards.py new file mode 100755 index 0000000..97b8ef9 --- /dev/null +++ b/test/check-evidence-cards.py @@ -0,0 +1,130 @@ +#!/usr/bin/env python3 +"""Assert the landing page's evidence cards agree with evidence/references.md. + +docs/index.html carries one card per research domain, each restating two facts +from evidence/references.md: how many studies the domain holds, and the span of +evidence tiers across them. Both were hand-typed, and three had drifted by +v3.4.0.0 — two cards advertised T2 for domains whose strongest reference is T3. + +Overstating tier strength is the one inaccuracy idstack cannot afford: the +product's entire claim is that it labels evidence honestly. So the cards are +derived from the reference file here rather than trusted. + +Usage: check-evidence-cards.py +Prints one line per mismatch and exits 1; prints nothing and exits 0 when clean. +Runs on Python 3.9 (the macOS system interpreter) — see test/test-preamble-python.sh +for why that floor matters. +""" + +import html +import io +import os +import re +import sys + +# A domain heading: "## Domain 7: Learner Analysis & Differentiation" +DOMAIN_RE = re.compile(r"^## Domain \d+: (.+?)\s*$") +# A reference key: "[Alignment-14]" — one per study. +CITATION_RE = re.compile(r"\[[A-Za-z][A-Za-z-]*-\d+\]") +TIER_RE = re.compile(r"\bT([1-5])\b") +# One card: title, study count, tier range. Order-independent; matched by title. +CARD_RE = re.compile( + r'evidence-card-title">(.*?)

' + r".*?" + r'evidence-card-meta">(\d+)\s+studies' + r".*?" + r'meta-tier">(.*?)', + re.S, +) + + +def tier_span(tiers): + """Render a sorted tier set the way the cards do: 'T3' or 'T1-T4' (en dash).""" + lo, hi = tiers[0], tiers[-1] + if lo == hi: + return "T%d" % lo + return "T%d–T%d" % (lo, hi) + + +def parse_domains(path): + text = io.open(path, encoding="utf-8").read().split("\n") + starts = [] + for i, line in enumerate(text): + m = DOMAIN_RE.match(line) + if m: + starts.append((i, m.group(1))) + starts.append((len(text), None)) + + domains = [] + for k in range(len(starts) - 1): + begin, name = starts[k] + end = starts[k + 1][0] + body = "\n".join(text[begin:end]) + tiers = sorted(set(int(t) for t in TIER_RE.findall(body))) + domains.append((name, len(CITATION_RE.findall(body)), tiers)) + return domains + + +def main(): + if len(sys.argv) != 2: + print("usage: check-evidence-cards.py ") + return 2 + root = sys.argv[1] + refs = os.path.join(root, "evidence", "references.md") + landing = os.path.join(root, "docs", "index.html") + + for p in (refs, landing): + if not os.path.isfile(p): + print("missing file: %s" % p) + return 1 + + domains = parse_domains(refs) + if not domains: + print("no '## Domain N:' sections found in evidence/references.md") + return 1 + + cards = CARD_RE.findall(io.open(landing, encoding="utf-8").read()) + by_title = {} + for raw_title, count, tier in cards: + by_title[html.unescape(raw_title).strip()] = (int(count), tier.strip()) + + problems = [] + if len(cards) != len(domains): + problems.append( + "card count %d != domain count %d — a domain was added or removed " + "without updating docs/index.html" % (len(cards), len(domains)) + ) + + for name, count, tiers in domains: + if name not in by_title: + problems.append("no evidence card for domain %r" % name) + continue + card_count, card_tier = by_title.pop(name) + if card_count != count: + problems.append( + "%s: card says %d studies, references.md has %d" + % (name, card_count, count) + ) + if not tiers: + problems.append("%s: references.md lists no evidence tiers" % name) + continue + want = tier_span(tiers) + if card_tier != want: + note = "" + if int(card_tier[1]) < tiers[0]: + note = " <- OVERSTATES: claims stronger evidence than the domain holds" + problems.append( + "%s: card says %s, references.md spans %s%s" + % (name, card_tier, want, note) + ) + + for leftover in sorted(by_title): + problems.append("evidence card %r matches no domain in references.md" % leftover) + + for line in problems: + print(line) + return 1 if problems else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/test/mutation-test.sh b/test/mutation-test.sh index cb9325c..4c0da9b 100755 --- a/test/mutation-test.sh +++ b/test/mutation-test.sh @@ -327,6 +327,42 @@ open(p,'w').write(s) PY expect_fail "claim carrying the bot's name is not exempt" "$WORK/r/test/smoke-test.sh" "$WORK/r" +# 18. an evidence card overstates its domain's tier strength -> smoke-test must +# fail. Three cards had drifted this way before v3.4.0.1, two of them claiming +# T2 for domains whose strongest reference is T3. idstack's whole claim is that +# it labels evidence honestly, so a card advertising better evidence than the +# repo holds is the most expensive inaccuracy it can ship. Mutating toward +# OVERSTATES rather than a harmless typo pins the direction that matters. +fresh +python3 - "$WORK/r/docs/index.html" <<'PY' +import io, sys +p = sys.argv[1]; s = io.open(p, encoding='utf-8').read() +old = 'T3' +assert s.count(old) == 1, 'anchor not unique: %d' % s.count(old) +io.open(p, 'w', encoding='utf-8').write(s.replace(old, 'T1–T5')) +PY +expect_fail "evidence card overstates its tier range" "$WORK/r/test/smoke-test.sh" "$WORK/r" + +# 19. a domain's study count drifts from the reference file -> smoke-test must +# fail. The count and the tier range are separate assertions; a checker that +# only compared tiers would let "108 peer-reviewed studies" rot silently. +fresh +python3 - "$WORK/r/evidence/references.md" <<'PY' +import io, sys +p = sys.argv[1]; s = io.open(p, encoding='utf-8').read() +lines = s.split('\n') +for i, line in enumerate(lines): + if line.startswith('## Domain 3: '): + # Drop the first citation line under this heading. + for j in range(i + 1, len(lines)): + if '[Needs-' in lines[j]: + del lines[j] + break + break +io.open(p, 'w', encoding='utf-8').write('\n'.join(lines)) +PY +expect_fail "domain study count drifts from the cards" "$WORK/r/test/smoke-test.sh" "$WORK/r" + echo "" echo "guarded: $pass NOT guarded: $fail skipped: $skip" [ "$fail" -eq 0 ] diff --git a/test/smoke-test.sh b/test/smoke-test.sh index d8fe565..9732d61 100755 --- a/test/smoke-test.sh +++ b/test/smoke-test.sh @@ -113,6 +113,23 @@ check "landing: Output section present" "grep -q 'id=.output.' '$LANDING'" check "landing: gradient text keeps a color fallback (no bare 'color: transparent')" \ "! grep -Eq '[^-]color: *transparent' '$LANDING'" +# The landing page's evidence cards restate evidence/references.md: a study count +# and a tier range per domain. They were hand-maintained and three had drifted — +# two advertised T2 for domains whose strongest evidence is T3, which is the one +# error idstack cannot afford, since tier honesty is the whole product. Derived +# from the reference file now, so the numbers cannot disagree. +# +# Skipped without python3 rather than silently passing; a crash in the checker +# becomes drift text so it fails loudly instead of vacuously. +EVIDENCE_DRIFT="" +if command -v python3 &>/dev/null; then + EVIDENCE_DRIFT="$(python3 "$IDSTACK_DIR/test/check-evidence-cards.py" "$IDSTACK_DIR" 2>&1)" \ + || EVIDENCE_DRIFT="evidence-card checker failed: +$EVIDENCE_DRIFT" + check "landing evidence cards match evidence/references.md" \ + "if [ -n \"\$EVIDENCE_DRIFT\" ]; then printf '%s\n' \"\$EVIDENCE_DRIFT\"; false; fi" +fi + # Open Graph card template (docs/og-template.html) - same gradient-text fallback rule. OG_TEMPLATE="$IDSTACK_DIR/docs/og-template.html" check "og-template.html exists" "[ -f '$OG_TEMPLATE' ]"