Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ jobs:
- name: Self-test for the inventory generator
run: node --test .harness/scripts/ci/07-generate-inventories.test.mjs

# 21 cases over `09-reconcile-maturity.mjs`, including the GT-576 rule that a
# 24 cases over `09-reconcile-maturity.mjs`, including the GT-576 rule that a
# capability may not be marked Validated on an ADR citation alone and the GT-596
# ISO/IEC 33020 threshold rule. Both were closed with "ships with a negative
# self-test" as the evidence; the self-test ran nowhere.
Expand Down
124 changes: 124 additions & 0 deletions .github/workflows/maturity-evidence-freshness.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
name: Maturity Evidence Freshness (GT-711)

# The four runtime checks in `maturity-evidence.json` are valid for 30 days; after that
# `09-reconcile-maturity.mjs --check` rejects them and `Validate documentation` — a REQUIRED
# check — goes red on every open PR, whatever the PR touched. That is by design: evidence
# that ages out is re-taken, not extended. What was not by design is HOW the expiry got
# noticed: twice (2026-08-18, 2026-09-19) by a PR author whose change had nothing to do with
# it, on the day it started blocking merges. The expiry date is known thirty days ahead, so
# this workflow reads it daily and opens ONE issue a week before, with the date and the
# procedure. Nothing here moves the window or touches the evidence.

concurrency:
group: ${{ github.workflow }}
cancel-in-progress: false

on:
schedule:
# 06:45 UTC daily — after opa-parity (06:00) and the published canary (06:30), so a
# red morning still has one cause per thread.
- cron: '45 6 * * *'
workflow_dispatch:
inputs:
now:
description: 'Ask what the report will say on this day (YYYY-MM-DD); empty = today'
required: false
default: ''

permissions:
contents: read
issues: write

jobs:
freshness:
name: Report when the runtime evidence turns stale
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
# The evidence is re-observed on `develop` and reaches `main` by promotion, so the
# branch to read is the one where the fix is authored. A shallow checkout is enough:
# `--freshness` reads one JSON file and never calls git.
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: develop

- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: '20'

- name: Age the four checks against the 30-day window
id: freshness
run: |
node .harness/scripts/ci/09-reconcile-maturity.mjs --freshness \
${{ inputs.now && format('--now={0}', inputs.now) || '' }} \
2>&1 | tee freshness.log
exit "${PIPESTATUS[0]}"

# A warning nobody is subscribed to is the failure mode this workflow exists to end
# (the Actions tab was already red for the canary and nobody looked — GT-635). The
# issue is UPDATED rather than duplicated, so seven days of warning is one thread.
- name: Open or update the freshness issue
if: failure() && github.event_name == 'schedule'
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
const fs = require('fs');
const log = fs.existsSync('freshness.log') ? fs.readFileSync('freshness.log', 'utf8') : '(no log captured)';
const today = new Date().toISOString().slice(0, 10);
const title = 'Runtime maturity evidence turns stale soon — re-observe it before Validate documentation goes red';
const body = [
`Checked on ${today} against \`develop\`.`,
'',
'From the day named below, `09-reconcile-maturity.mjs --check` rejects the evidence and',
'**`Validate documentation` (required) is red on every PR**, whatever the PR touched.',
'',
'```',
log.slice(-4000),
'```',
'',
'**Procedure** (the one in `3e5aac80` and `2ee3f9a0`): for each of the four checks take a',
'fresh green run of its workflow, rewrite `observedAt` / `commit` / `source` / `summary` in',
'`reference/core/control-center/maturity-reports/maturity-evidence.json` as a NEW observation',
'(what the run shows, never a date bump), run `node .harness/scripts/ci/09-reconcile-maturity.mjs`',
'to regenerate the reconciliation, and open a PR to `develop`; promote to `main` afterwards.',
'',
`Run: ${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`,
].join('\n');

const existing = await github.rest.issues.listForRepo({
owner: context.repo.owner, repo: context.repo.repo,
state: 'open', labels: 'maturity-evidence',
});
if (existing.data.length > 0) {
await github.rest.issues.createComment({
owner: context.repo.owner, repo: context.repo.repo,
issue_number: existing.data[0].number, body,
});
} else {
await github.rest.issues.create({
owner: context.repo.owner, repo: context.repo.repo,
title, body, labels: ['maturity-evidence'],
});
}

# Once re-observed, the thread says so and closes itself; an issue nobody closes is an
# issue nobody believes.
- name: Close the freshness issue once the evidence is fresh again
if: success() && github.event_name == 'schedule'
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
const existing = await github.rest.issues.listForRepo({
owner: context.repo.owner, repo: context.repo.repo,
state: 'open', labels: 'maturity-evidence',
});
for (const issue of existing.data) {
await github.rest.issues.createComment({
owner: context.repo.owner, repo: context.repo.repo, issue_number: issue.number,
body: `Fresh again on ${new Date().toISOString().slice(0, 10)} — all four checks are inside the window with more than seven days to spare.`,
});
await github.rest.issues.update({
owner: context.repo.owner, repo: context.repo.repo,
issue_number: issue.number, state: 'closed',
});
}
106 changes: 105 additions & 1 deletion .harness/scripts/ci/09-reconcile-maturity.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -626,6 +626,82 @@ const OUTPUT = expected('maturityReports', 'maturity-reconciliation.json');
const EVIDENCE_STATUSES = new Set(['PASS', 'BLOCKED', 'RESOLVED']);
const REQUIRED_CHECKS = new Set(['cli-baseline', 'coverage', 'documentation', 'release']);

// GT-711: the 0..30 day window on runtime evidence is the point — evidence that ages out
// is re-taken, not extended — but the only thing that ever noticed the window closing was
// the REQUIRED `Validate documentation` check going red on whichever PR happened to be
// open that morning. Twice: a promotion on 2026-08-18 and a README-only PR (#724) on
// 2026-09-19, neither of which had touched the evidence. Expiry is a date known thirty
// days in advance, so it is announced as one: every check inside EVIDENCE_WARN_DAYS of
// turning stale is reported with the day it turns, on every invocation, and
// `--freshness` turns that report into an exit code a scheduled workflow can act on
// BEFORE the day it starts blocking merges. The window itself does not move.
export const EVIDENCE_MAX_AGE_DAYS = 30;
export const EVIDENCE_WARN_DAYS = 7;

function isoDate(date) {
return date.toISOString().slice(0, 10);
}

/**
* Age every check against the window and say, per check, when it turns stale. The day it
* turns stale is `observedAt + EVIDENCE_MAX_AGE_DAYS + 1`, the first day on which
* `validateRuntimeEvidence` rejects it — the same arithmetic, not a second opinion.
* `state` is `fresh`, `expiring` (stale within EVIDENCE_WARN_DAYS, today included),
* `stale` (already rejected) or `future` (observedAt after today, also rejected).
*/
export function assessEvidenceFreshness(evidence, now = new Date()) {
const checks = Array.isArray(evidence?.checks) ? evidence.checks : [];
return checks.map((check) => {
const observedAt = /^\d{4}-\d{2}-\d{2}$/.test(check?.observedAt || '') ? check.observedAt : null;
if (!observedAt) return { id: check?.id, observedAt: check?.observedAt, ageDays: null, staleFrom: null, daysLeft: null, state: 'invalid' };
const observed = new Date(`${observedAt}T00:00:00Z`);
const ageDays = Math.floor((now - observed) / 86400000);
const staleFrom = isoDate(new Date(observed.getTime() + (EVIDENCE_MAX_AGE_DAYS + 1) * 86400000));
const daysLeft = EVIDENCE_MAX_AGE_DAYS - ageDays;
let state = 'fresh';
if (ageDays < 0) state = 'future';
else if (ageDays > EVIDENCE_MAX_AGE_DAYS) state = 'stale';
else if (daysLeft <= EVIDENCE_WARN_DAYS) state = 'expiring';
return { id: check?.id, observedAt, ageDays, staleFrom, daysLeft, state };
});
}

/** One line per check, in the shape a human reads in a log or an issue body. */
export function formatEvidenceFreshness(rows) {
const width = Math.max(...rows.map((row) => String(row.id).length), 2);
return rows.map((row) => {
const id = String(row.id).padEnd(width);
if (row.state === 'invalid') return `❌ ${id} observedAt ${row.observedAt}: not a date`;
if (row.state === 'future') return `❌ ${id} observed ${row.observedAt}: in the future — a date, not an observation`;
if (row.state === 'stale') return `❌ ${id} observed ${row.observedAt}: STALE since ${row.staleFrom} (${row.ageDays} days old) — Validate documentation is red on every PR until it is re-observed`;
if (row.state === 'expiring') return `⚠️ ${id} observed ${row.observedAt}: turns stale on ${row.staleFrom} (${row.daysLeft === 0 ? 'today is the last valid day' : `${row.daysLeft} day(s) left`}) — re-observe it before then`;
return `✅ ${id} observed ${row.observedAt}: ${row.daysLeft} day(s) left, turns stale on ${row.staleFrom}`;
});
}

/**
* `--freshness`: read the evidence alone, print its ages, and exit 1 when any check is
* within the warning band or past it. Deliberately cheaper than the reconciliation — no
* board, no git, no ISO audit — so the scheduled workflow that runs it needs a shallow
* checkout and nothing else.
*/
function reportEvidenceFreshness(now = new Date()) {
const evidence = JSON.parse(fs.readFileSync(RUNTIME_EVIDENCE, 'utf8'));
const rows = assessEvidenceFreshness(evidence, now);
console.log(`Runtime maturity evidence on ${isoDate(now)} — window ${EVIDENCE_MAX_AGE_DAYS} days, warning ${EVIDENCE_WARN_DAYS} days before it closes:`);
for (const line of formatEvidenceFreshness(rows)) console.log(line);
const failing = rows.filter((row) => row.state !== 'fresh');
if (failing.length) {
console.log(
`\n${failing.length} of ${rows.length} check(s) need re-observing. The procedure is the one in commit 3e5aac80 / 2ee3f9a0:\n`
+ ' take a fresh green run of each workflow, rewrite observedAt/commit/source/summary as a NEW observation\n'
+ ' (never a date bump), then `node .harness/scripts/ci/09-reconcile-maturity.mjs` and commit both files.',
);
process.exit(1);
}
console.log(`\nAll ${rows.length} checks are inside the window with more than ${EVIDENCE_WARN_DAYS} days to spare.`);
}

function countFiles(directory, pattern, excludePattern) {
if (!fs.existsSync(directory)) return 0;
return fs.readdirSync(directory, { withFileTypes: true }).reduce((total, entry) => {
Expand Down Expand Up @@ -686,7 +762,7 @@ export function validateRuntimeEvidence(evidence, board, root = ROOT, now = new
errors.push(`${check?.id} has invalid observedAt`);
} else {
const ageDays = Math.floor((now - new Date(`${check.observedAt}T00:00:00Z`)) / 86400000);
if (ageDays < 0 || ageDays > 30) errors.push(`${check.id} evidence is stale or future-dated`);
if (ageDays < 0 || ageDays > EVIDENCE_MAX_AGE_DAYS) errors.push(`${check.id} evidence is stale or future-dated`);
}
if (!/^[0-9a-f]{7,40}$/i.test(check?.commit || '') || !commitExists(root, check.commit)) {
errors.push(`${check?.id} references an unavailable commit`);
Expand Down Expand Up @@ -774,6 +850,20 @@ function serialize(snapshot) {
}

function run() {
// GT-711: the freshness report is its own mode so the scheduled workflow can run it on a
// shallow checkout; it reads one file and never touches the reconciliation.
if (process.argv.includes('--freshness')) {
// `--now=YYYY-MM-DD` asks what the report will say on a given day, so the red path can
// be observed on demand (workflow_dispatch) instead of waited for.
const asOf = process.argv.find((arg) => arg.startsWith('--now='))?.slice('--now='.length);
if (asOf && !/^\d{4}-\d{2}-\d{2}$/.test(asOf)) {
console.error(`❌ --now expects YYYY-MM-DD, got "${asOf}"`);
process.exit(2);
}
reportEvidenceFreshness(asOf ? new Date(`${asOf}T12:00:00Z`) : new Date());
return;
}

// GT-576/GT-596: prove both rules still bite BEFORE trusting their verdict on the real
// document. A guard that has never been observed failing is the defect, not the control.
const { assertions } = selfTestValidatedEvidenceRule();
Expand Down Expand Up @@ -811,6 +901,20 @@ function run() {
);

const expected = serialize(buildSnapshot());

// GT-711: buildSnapshot has just accepted the evidence, so nothing here can be stale —
// but it can be about to be. Say so on every run, in the log a PR author actually reads,
// with the date; the scheduled `--freshness` run is what turns this into an issue.
const expiring = assessEvidenceFreshness(JSON.parse(fs.readFileSync(RUNTIME_EVIDENCE, 'utf8')))
.filter((row) => row.state === 'expiring');
if (expiring.length) {
console.warn(
`⚠️ ${expiring.length} runtime maturity check(s) turn stale within ${EVIDENCE_WARN_DAYS} days — `
+ 'from that day `Validate documentation` is red on every PR until they are re-observed:\n'
+ formatEvidenceFreshness(expiring).map((line) => ` ${line}`).join('\n'),
);
}

if (process.argv.includes('--check')) {
if (!fs.existsSync(OUTPUT) || fs.readFileSync(OUTPUT, 'utf8') !== expected) {
console.error('❌ Maturity reconciliation is stale. Run: node .harness/scripts/ci/09-reconcile-maturity.mjs');
Expand Down
49 changes: 49 additions & 0 deletions .harness/scripts/reconcile-maturity.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@ import assert from 'node:assert/strict';
import {
parseBoard,
validateRuntimeEvidence,
assessEvidenceFreshness,
formatEvidenceFreshness,
EVIDENCE_MAX_AGE_DAYS,
EVIDENCE_WARN_DAYS,
auditIsoRatings,
auditAssessmentRatings,
bandFor,
Expand Down Expand Up @@ -125,6 +129,51 @@ test('runtime evidence rejects stale and unowned blockers', () => {
assert.ok(errors.some((error) => error.includes('Missing required maturity check')));
});

// ---------------------------------------------------------------------------
// GT-711 — the window is announced before it closes
// ---------------------------------------------------------------------------

// The real incident, replayed: four checks observed 2026-08-18 were accepted by the develop
// run of 2026-09-15 and rejected by a README-only PR on 2026-09-19. The report has to say
// "turns stale on 2026-09-18" from the day it is within the warning band.
const observedAugust18 = (id) => ({ id, status: 'PASS', observedAt: '2026-08-18', commit: 'abc1234', source: 'https://github.com/o/r/actions/runs/1', summary: 'x' });

test('freshness names the first day a check turns stale, and it is the day validateRuntimeEvidence starts rejecting it', () => {
const evidence = { schemaVersion: '1.0.0', asOf: '2026-09-05', checks: [observedAugust18('cli-baseline')] };
const [row] = assessEvidenceFreshness(evidence, new Date('2026-09-15T17:00:00Z'));
assert.equal(row.staleFrom, '2026-09-18');
assert.equal(row.ageDays, 28);
assert.equal(row.daysLeft, 2);
assert.equal(row.state, 'expiring');

const board = { ...parseBoard('**Last Updated:** 2026-09-05\n| [`GT-1`](./c.md#gt-1) | x | Cross | P0 | M | `DONE` |\n'), content: '' };
const lastValidDay = validateRuntimeEvidence(evidence, board, process.cwd(), new Date('2026-09-17T23:59:00Z'));
const firstStaleDay = validateRuntimeEvidence(evidence, board, process.cwd(), new Date('2026-09-18T00:00:00Z'));
assert.ok(!lastValidDay.some((error) => error.includes('stale')), 'day 30 is still inside the window');
assert.ok(firstStaleDay.some((error) => error.includes('stale')), 'day 31 is outside it — the same day the report announced');
});

test('freshness bands: fresh beyond the warning band, expiring inside it (today included), stale past the window, future ahead of it', () => {
const evidence = { schemaVersion: '1.0.0', asOf: '2026-09-05', checks: [observedAugust18('coverage')] };
const stateOn = (day) => assessEvidenceFreshness(evidence, new Date(`${day}T12:00:00Z`))[0].state;
assert.equal(EVIDENCE_MAX_AGE_DAYS, 30);
assert.equal(EVIDENCE_WARN_DAYS, 7);
assert.equal(stateOn('2026-09-09'), 'fresh', '22 days old: 8 left, outside the band');
assert.equal(stateOn('2026-09-10'), 'expiring', '23 days old: 7 left, first day of the band');
assert.equal(stateOn('2026-09-17'), 'expiring', '30 days old: last valid day, 0 left');
assert.equal(stateOn('2026-09-18'), 'stale', '31 days old: rejected');
assert.equal(stateOn('2026-08-17'), 'future', 'observed tomorrow: a date, not an observation');
assert.equal(assessEvidenceFreshness({ checks: [{ id: 'release', observedAt: 'soon' }] })[0].state, 'invalid');
});

test('the freshness report is one line per check and carries the date on the lines that matter', () => {
const evidence = { schemaVersion: '1.0.0', asOf: '2026-09-05', checks: [observedAugust18('cli-baseline'), { ...observedAugust18('release'), observedAt: '2026-09-15' }] };
const lines = formatEvidenceFreshness(assessEvidenceFreshness(evidence, new Date('2026-09-19T09:00:00Z')));
assert.equal(lines.length, 2);
assert.match(lines[0], /^❌ cli-baseline .*STALE since 2026-09-18/);
assert.match(lines[1], /^✅ release .*turns stale on 2026-10-16/);
});

// ---------------------------------------------------------------------------
// GT-596 — ISO/IEC 33020:2019 rating scale
// ---------------------------------------------------------------------------
Expand Down
Loading
Loading