feat: verify immutable Design policy exports for WP00.01

.github/workflows/pr-gate.yml

@@ -35,7 +35,7 @@ jobs:
           if ($LASTEXITCODE) { throw 'Invalid package version.' }
           "value=$version" >> $env:GITHUB_OUTPUT
   managed-packages:
-    needs: [allocate-version, native, secret-scan, repository-hooks]
+    needs: [allocate-version, native, secret-scan, repository-hooks, design-policy]
     uses: ./.github/workflows/package-validation.yml
     with:
       version: ${{ needs.allocate-version.outputs.version }}
@@ -71,9 +71,32 @@ jobs:
       - name: Run repository hooks
         run: pre-commit run --all-files --show-diff-on-failure --color=always
 
+  design-policy:
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+        with:
+          persist-credentials: false
+      - name: Reject source, export, citation and dependency drift
+        run: python -m unittest discover -s eng -p test_design_policy.py -v
+      - name: Verify exports against an isolated immutable Design checkout
+        run: python eng/design_policy.py
+      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
+        if: always()
+        with:
+          name: design-policy-${{ matrix.os }}-${{ github.run_id }}-${{ github.run_attempt }}
+          path: artifacts/evidence/design-policy.json
+          if-no-files-found: error
+          retention-days: 30
+
   ci:
     if: always()
-    needs: [allocate-version, managed-packages, native, secret-scan, repository-hooks]
+    needs: [allocate-version, managed-packages, native, secret-scan, repository-hooks, design-policy]
     runs-on: ubuntu-latest
     env:
       CI_NEEDS: ${{ toJSON(needs) }}

README.md

@@ -17,6 +17,23 @@ Product applications and services build independently from published packages.
 See [extraction scope and evidence](docs/platform-bootstrap.md). The accepted product family design is
 maintained in [ArcForges-Design](https://github.com/ArcForges/ArcForges-Design).
 
+## Design policy data
+
+`eng/policy` contains generated glossary and invariant data from the exact Design
+commit in `design-source.json`. The portable checker compares fresh exports and
+validates document links, scoped citations, occurrence classifications and the
+work-package graph. CI requires these checks on Windows and Linux before packaging.
+
+```text
+python -m unittest discover -s eng -p test_design_policy.py -v
+python eng/design_policy.py
+```
+
+The second command fetches only the pinned public documentation into a temporary
+checkout and retains a report under `artifacts/evidence/design-policy.json`. It
+does not execute Design code examples. For reviewed source changes and read-only
+pre-merge checks, see [policy maintenance and evidence](docs/design-policy.md).
+
 ## Build and verify
 
 Install the .NET SDK selected by `global.json` and Python 3.11 or newer. No Mobile/Web workloads are needed.

docs/design-policy.md

@@ -0,0 +1,76 @@
+# WP00.01: Design policy exports and continuing integrity
+
+Design remains the authored authority. [P2-016](https://github.com/ArcForges/ArcForges-Design/blob/1607374e81955f0a47f319cd6cc8ba1c6e254157/docs/decisions/phase-2-specification-decisions.md#rule-p2-016)
+assigns this AGPL exporter, derived data and CI check to DesktopPlatform. Consumers
+can read the checked-in JSON without a Design checkout. These files are policy data;
+distribution through build packages belongs to WP02/WP05, and this change adds no
+packable capability or product source dependency.
+
+The reviewed pin is `1607374e81955f0a47f319cd6cc8ba1c6e254157`. Collection and
+negative fixtures found missing scoped anchors, ambiguous and compound citations,
+sentence-final citations, an omitted reverse dependency and stale mechanism totals.
+Design PRs [15](https://github.com/ArcForges/ArcForges-Design/pull/15),
+[16](https://github.com/ArcForges/ArcForges-Design/pull/16) and
+[17](https://github.com/ArcForges/ArcForges-Design/pull/17) repair those at their owner.
+
+The exported 135 term rows preserve 148 marked names, five explicit spaces and all
+16 contextual forbidden-alias rows. The 429 invariant records retain both catalogue
+and coverage locations, statements, architecture homes, mechanisms, planned
+verification, owning packages and completion gates. Three retired rows keep their
+absence tests. `verificationState: planned-only` does not close PG-11 or prove runtime
+behavior. Contracts recognizes only the digest-bound forbidden-alias array at its
+registered path; all remaining values and files stay subject to naming scans.
+
+The checker reads 172 current Markdown documents while excluding deprecated input
+bodies before reading. The current receipt indexes 7,601 document-scoped rules and
+9,436 explicit rule citations, checks 11,411 local links and validates 165 exact
+occurrence classifications covering 171 occurrences. It compares both forward
+graphs, every active package header/dependency section and the exact reverse graph:
+51 active nodes and 158 edges. It checks topological order, unique numbered sections,
+mandatory package sections and the owned `.90` evidence row. Future/retired packages
+have no active edges. The commerce substep ordering remains explicit.
+
+## Source updates and review
+
+For a proposed Design worktree, run the read-only preview before its documentation
+PR merges. Its report includes commit/dirty state and cannot be used as an immutable
+export result:
+
+```text
+python eng/design_policy.py --design-root /absolute/path/to/design-worktree --preview --report artifacts/evidence/design-preview.json
+```
+
+After reviewing and merging the Design change, update the exact commit and source
+hashes in `eng/policy/design-source.json` on a DesktopPlatform worktree. Source hashes
+use UTF-8 text with normalized LF newlines, matching Git's text content on Windows
+and Linux. The corpus digest covers the sorted array of document path and text digest
+pairs using compact JSON with sorted keys and unescaped UTF-8. Obtain these values
+from the reviewed preview, then generate and independently verify the exports:
+
+```text
+python eng/design_policy.py --refresh
+python eng/design_policy.py
+```
+
+A supplied `--design-root` must be the canonical repository, clean and at the exact
+pin for immutable mode. The default obtains a fresh isolated public checkout at that
+commit, with checkout hooks disabled. It runs no source programs and never advances
+to a branch tip. A source change also requires Contracts to update its exact derived
+declaration registration before the family scan accepts the new glossary identity.
+
+## Verification and limits
+
+The 15 test groups use independent small documents and real temporary Git repositories.
+They remove, duplicate, reorder and edit export records; alter source statements,
+spaces, owners and architecture homes; corrupt links, anchors and classification
+counts/hashes; and break every graph representation, node ordering and evidence rows.
+They exercise dirty/wrong-pin refusal, preview immutability, deprecated-body exclusion,
+sentence punctuation, same-spelled rules in different documents and the CLI's real
+failure exit/report. A current-corpus pass alone is insufficient.
+
+Both CI platforms run the fixtures and fetch/verify the pinned real corpus before
+package creation. Their full reports are retained as `design-policy-*` artifacts.
+Existing native, managed, independent package-consumer and publication gates remain
+required. Local immutable export verification passed; PR CI, merge and publication
+remain pending until the corresponding run artifacts are reviewed. Policy checks
+establish no provider, device, product behavior or commercial activation evidence.

eng/design_corpus.py

@@ -0,0 +1,159 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+"""Document-scoped Markdown integrity checks; no historical programs are executed."""
+
+import collections
+import html
+import posixpath
+import re
+import subprocess
+from urllib.parse import unquote, urlsplit
+
+ID = re.compile(r'(?<![\w-])(?:WP-\d{2}(?:\.\d{2})?|P2-\d{3}|F-[A-Z]{2}-[1-9]\d*|[A-Z][A-Z0-9]{0,5}-(?:[A-Z]\d{1,3}|\d{2,3}[a-z]?))(?![\w-]|\.\d)')
+LINK = re.compile(r'\[([^\]\n]+)\]\(([^)\n]+)\)')
+ANCHOR = re.compile(r'<a\s+id="([^"]+)"\s*>\s*</a>')
+STANDARD = {'SHA-256', 'SHA-512', 'UTF-16', 'UTF-32', 'IEEE-754', 'P-256'}
+
+def visible(s):
+    return re.sub(r'[`*~]', '', html.unescape(LINK.sub(lambda m: m[1], re.sub(r'<[^>]*>', '', s))))
+
+def cells(line):
+    return [c.strip() for c in re.split(r'(?<!\\)\|', line.strip())[1:-1]]
+
+def lines(text):
+    fence = None
+    for n, line in enumerate(text.splitlines(), 1):
+        match = re.match(r'^\s{0,3}(`{3,}|~{3,})(.*)$', line)
+        if match:
+            if fence is None:
+                fence = match[1]
+            elif match[1][0] == fence[0] and len(match[1]) >= len(fence) and not match[2].strip():
+                fence = None
+            continue
+        if fence is None:
+            yield n, line
+    if fence:
+        raise ValueError('unclosed code fence')
+
+def heading_slug(text):
+    return re.sub(r'[^\w\- ]', '', visible(text).lower()).replace(' ', '-')
+
+def anchor_identifier(anchor):
+    if not anchor.startswith('rule-'):
+        return None
+    candidate = anchor[5:].upper()
+    if ID.fullmatch(candidate):
+        return candidate
+    candidate = candidate[:-1] + candidate[-1:].lower()
+    return candidate if ID.fullmatch(candidate) else None
+
+def definition(line):
+    if line.startswith('|'):
+        cs = cells(line)
+        if not cs or LINK.search(cs[0]):
+            return None
+        value = visible(cs[0]).strip()
+    else:
+        m = re.match(r'^#{1,6} (.+)$|^- (.+)$|^(\*\*[A-Z].+)$', line)
+        if not m:
+            return None
+        value = visible(m[1] or m[2] or m[3]).strip()
+        if LINK.match((m[1] or m[2] or m[3]).lstrip('*')): return None
+    m = ID.match(value)
+    if m and (len(value) == len(m[0]) or re.match(r'\s*[—:·]', value[len(m[0]):])):
+        return m[0]
+    return None
+
+def load(root):
+    entries = subprocess.check_output(['git', 'ls-files', '--stage', '-z'], cwd=root).decode().split('\0')
+    paths = []
+    for entry in filter(None, entries):
+        metadata, path = entry.split('\t', 1)
+        mode, _, stage = metadata.split()
+        if mode not in {'100644', '100755'} or stage != '0':
+            raise ValueError(f'unsupported Git source entry: {path}')
+        paths.append(path)
+    paths += subprocess.check_output(['git','ls-files','--others','--exclude-standard','-z'],cwd=root).decode().split('\0')
+    paths = sorted(set(p for p in paths if p.endswith('.md') and not (p.startswith('docs/deprecated-inputs/') and p != 'docs/deprecated-inputs/README.md')))
+    return {p: read_document(root, p) for p in paths}
+
+def read_document(root, path):
+    candidate = root / path
+    if not candidate.resolve().is_relative_to(root.resolve()):
+        raise ValueError(f'source escapes Design root: {path}')
+    for current in [candidate, *candidate.parents]:
+        if current == root:
+            break
+        if current.is_symlink() or (hasattr(current, 'is_junction') and current.is_junction()):
+            raise ValueError(f'linked source is unsupported: {path}')
+    return candidate.read_text(encoding='utf-8')
+
+def collect(root, docs=None):
+    docs = load(root) if docs is None else docs
+    anchors = {}; defs = {}; errors = []
+    for p, text in docs.items():
+        parsed = list(lines(text)); existing = {}; counts = collections.Counter(); found = {}
+        for n, line in parsed:
+            for a in ANCHOR.findall(line):
+                if a in existing: errors.append([p,n,'duplicate anchor',a])
+                existing[a] = n
+            h = re.match(r'^#{1,6} (.*)$', line)
+            if h:
+                base = heading_slug(h[1]); count = counts[base]; counts[base] += 1
+                slug = base + (f'-{count}' if count else '')
+                if slug in existing: errors.append([p,n,'duplicate anchor',slug])
+                existing[slug] = n
+            key = definition(line)
+            if key:
+                if key in found: errors.append([p,n,'duplicate definition',key])
+                found[key] = n
+        for a, n in existing.items():
+            key = anchor_identifier(a)
+            if key: found.setdefault(key,n)
+        authored = {definition(line) for _, line in parsed}
+        defs[p] = {k: {'line':n, 'anchor':'rule-'+k.lower(), 'stable':'rule-'+k.lower() in existing,
+                       'kind':'definition' if k in authored else 'preserved-anchor'} for k,n in found.items()}
+        anchors[p] = existing
+    return docs, anchors, defs, errors
+
+def audit(root, docs=None):
+    docs, anchors, defs, errors = collect(root, docs)
+    citations=[]; local_links=[]; raw=[]; missing=[]
+    homes=collections.defaultdict(list)
+    for p, dd in defs.items():
+        for key, d in dd.items():
+            homes[key].append(p)
+            if not d['stable']: missing.append([p,d['line'],key])
+    for p, text in docs.items():
+        for n, line in lines(text):
+            prose = re.sub(r'`+[^`]*`+', '', line)
+            if re.search(r'\[[^\]]+\]\[[^\]]*\]|^\s*\[[^\]]+\]:', prose):
+                errors.append([p,n,'unsupported reference-link syntax'])
+            for match in LINK.finditer(line):
+                uri=match[2].strip('<>')
+                label_ids = [key for key in ID.findall(visible(match[1])) if key not in STANDARD]
+                if urlsplit(uri).scheme or uri.startswith('//'):
+                    if label_ids: errors.append([p,n,'external rule home',uri,label_ids])
+                    continue
+                name, _, fragment=uri.partition('#')
+                target=posixpath.normpath(posixpath.join(posixpath.dirname(p),unquote(name))) if name else p
+                local_links.append(dict(document=p, line=n, label=match[1], target=target, anchor=unquote(fragment)))
+                if target.startswith('/') or target == '..' or target.startswith('../'):
+                    errors.append([p,n,'link escapes Design root',uri])
+                    continue
+                if not (root/target).exists(): errors.append([p,n,'missing target',uri])
+                elif fragment and unquote(fragment) not in anchors.get(target,{}): errors.append([p,n,'missing fragment',uri])
+                for label in label_ids:
+                    if fragment!='rule-'+label.lower():errors.append([p,n,'wrong rule anchor',uri,label])
+                    if label not in defs.get(target,{}):errors.append([p,n,'missing rule definition',uri,label])
+                    citations.append([p,n,label,target,fragment])
+            spans=[m.span() for m in LINK.finditer(line)]+[m.span() for m in ANCHOR.finditer(line)]
+            defid=definition(line)
+            firstdef=True
+            for m in ID.finditer(line):
+                key=m[0]
+                if any(a<=m.start()<b for a,b in spans) or key in STANDARD:continue
+                if key==defid and firstdef:firstdef=False;continue
+                raw.append([p,n,key,homes[key],line])
+    return dict(documents=len(docs), links=len(local_links), localLinks=local_links, definitions=sum(map(len,defs.values())),
+                index=[dict(document=p, identifier=k, **v) for p,dd in defs.items() for k,v in dd.items()],
+                missingAnchors=missing, errors=errors, citations=citations, raw=raw)

eng/design_graph.py

@@ -0,0 +1,81 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+"""Compare every current work-package graph representation and its schedule."""
+
+from design_corpus import cells, lines, load
+import re
+
+INACTIVE = {'20', '27', '29'}
+
+def graph(root, docs=None):
+    docs=load(root) if docs is None else docs
+    errors=[]
+    index=docs['docs/planning/work-packages/README.md']
+    sequence=docs['docs/planning/implementation-sequence.md']
+    forward={};phase={};reverse={};packages={}
+    ids=lambda value:set(re.findall(r'`(\d{2})`',value))
+    for line in sequence.split('## 9. ',1)[1].splitlines():
+        m=re.match(r'^\| (\d{2}) \| (.*) \|$',line)
+        if m:
+            if m[1] in forward:errors.append(['duplicate forward',m[1]])
+            forward[m[1]]=ids(m[2])
+    for line in index.split('## Downstream dependency index',1)[0].splitlines():
+        cs=cells(line)
+        if len(cs)==3 and re.fullmatch(r'\d{2}',cs[0]):
+            if cs[0] in INACTIVE:
+                if ids(cs[2]):errors.append(['inactive phase edges',cs[0]])
+                continue
+            if cs[0] in phase:errors.append(['duplicate phase',cs[0]])
+            phase[cs[0]]=ids(cs[2])
+    for line in index.split('## Downstream dependency index',1)[1].split('## Deferred-gate scheduling',1)[0].splitlines():
+        cs=cells(line)
+        if len(cs)==2 and re.fullmatch(r'\d{2}',cs[0]):
+            if cs[0] in reverse:errors.append(['duplicate reverse',cs[0]])
+            reverse[cs[0]]=ids(cs[1])
+    for p,text in docs.items():
+        numbers=[m[1] for n,l in lines(text) if (m:=re.match(r'^## (\d+(?:\.\d+)*)(?:\.)?\s',l))]
+        if len(numbers)!=len(set(numbers)):errors.append([p,'duplicate numbered section',numbers])
+        m=re.match(r'docs/planning/work-packages/(\d{2})-',p)
+        if not m:continue
+        if m[1] in INACTIVE:
+            for line in text.splitlines():
+                if re.match(r'^(?:> Upstream:|\*\*(?:Upstream|Downstream):\*\*)',line) and ids(line):
+                    errors.append([p,'inactive package edges'])
+            continue
+        key=m[1]
+        if key in packages:errors.append([p,'duplicate package identifier',key])
+        packages[key]=p
+        if set(numbers)!=set(str(i) for i in range(1,10)):errors.append([p,'mandatory sections',numbers])
+        header=next((l for l in text.splitlines() if l.startswith('> Upstream:')),None)
+        if not header or 'Downstream:' not in header:errors.append([p,'missing graph header']);continue
+        hu,hd=header.split('Downstream:',1)
+        try:
+            dep=text.split('## 9. ',1)[1].split('\n## ',1)[0]
+            su,sd=dep.split('**Downstream:**',1)
+            evidence=text.split('## 7. ',1)[1].split('\n## ',1)[0]
+        except (IndexError,ValueError):errors.append([p,'missing dependency/evidence section']);continue
+        up,down=ids(su),ids(sd)
+        if ids(hu)!=up or ids(hd)!=down:errors.append([p,'header/dependency mismatch'])
+        if up!=forward.get(key):errors.append([p,'forward mismatch'])
+        if down!=reverse.get(key):errors.append([p,'downstream mismatch'])
+        rows=[l for l in evidence.splitlines() if l.startswith('|') and f'#rule-wp-{key}.90)' in l]
+        if len(rows)!=1:errors.append([p,'owned .90 evidence rows',len(rows)])
+    if forward!=phase:errors.append(['forward/phase mismatch'])
+    if set(forward)!=set(packages) or set(reverse)!=set(packages):errors.append(['node sets differ'])
+    for key, deps in forward.items():
+        if not deps<=forward.keys():errors.append([key,'inactive producer'])
+        if reverse.get(key)!={k for k,v in forward.items() if key in v}:errors.append([key,'reverse is not transpose'])
+    s=re.search(r'^Serial execution: ([\d, ]+)\.',sequence,re.M)
+    order=s[1].split(', ') if s else []
+    if len(order)!=len(set(order)) or set(order)!=set(forward):errors.append(['serial node set/order'])
+    pos={v:n for n,v in enumerate(order)}
+    for k, values in forward.items():
+        for v in values:
+            if pos.get(v,999)>=pos.get(k,-1):errors.append([k,'producer ordered later',v])
+    if f'All {len(forward)} active packages' not in sequence:errors.append(['declared node count'])
+    edges=sum(map(len,forward.values()))
+    if f'Total active dependency edges: {edges}.' not in sequence:errors.append(['declared edge count'])
+    if 'WP42.11 precedes 42.10.' not in sequence:errors.append(['commerce substep order missing'])
+    return {'nodes':len(forward),'edges':edges,'order':order,
+            'forward':{k:sorted(v) for k,v in forward.items()},
+            'reverse':{k:sorted(v) for k,v in reverse.items()},
+            'packages':packages,'errors':errors}