Skip to content

fix(schema-gate): check well-formedness as emitted and reject repeated mc:Ignorable tokens - #1501

Merged
stevenobiajulu merged 5 commits into
mainfrom
1484-schema-gate-duplicate-ns-20261115
Nov 15, 2026
Merged

stevenobiajulu merged 5 commits into
mainfrom
1484-schema-gate-duplicate-ns-20261115

Conversation

@stevenobiajulu

Copy link
Copy Markdown
Member

Problem

Issue #1484 says the emitted-document schema gate (scripts/check_emitted_document_schema.mjs) accepts a w:document root that declares xmlns:w14 and xmlns:mc twice. Reproducing at base 1b1149e showed a narrower gap:

  • Repeated declarations were already rejected. The gate parses each part with xmldom's default handler, which throws Attribute xmlns:w14 redefined. The issue's literal shape fails on main (exit 1).
  • What passed was the comparison output built from such a fixture. docx-core's parseXml drops an exact repeat of a namespace declaration on input (docx-core: a part that declares xmlns:w twice on one element throws a raw ParseError instead of loading or failing closed #1032) but keeps the attribute value, so the redline's root carries mc:Ignorable="w14 w14 wps". The gate accepted that (exit 0). Reproduced with the pre-d98d03db fixture from fix(docx-compare): track header/footer table rows beside a text box and never wrap a cell in w:ins #1479: the original fails the gate, the compared output passes.
  • The gate checked only the preprocessed projection. When mc:Ignorable drops a namespace, the gate re-serializes the DOM and gives xmllint the projection, not the bytes as emitted. Two prefixes bound to one URI can give one element the same expanded attribute name twice, which is not namespace-well-formed (Namespaces in XML 1.0 §6.3). That passed whenever preprocessing removed the namespace. Without preprocessing, xmllint caught it.
  • mc:Ignorable rules were only partly checked. An unbound prefix failed. A prefix bound to the MC namespace, a repeated token, and mc:MustUnderstand/mc:ProcessContent tokens were not checked.

LibreOffice 26.2.5.2 (headless --convert-to odt, through the heavy-run gate) on the three shapes:

Input Gate on main Gate after LibreOffice
Fixture document.xml with xmlns:w14/xmlns:mc declared twice fail fail "source file could not be loaded"
Its compared output: declarations once, mc:Ignorable="w14 w14 wps" pass fail loads
The same output with mc:Ignorable="w14 wps" pass pass loads

So the LibreOffice refusal in #1479 came from the input fixture, not the redline. The repeated declarations are a well-formedness error that any XML parser rejects. The repeated Ignorable token is not a LibreOffice problem. It is what such a fixture leaves behind in compared output.

Approach

Gate (scripts/check_emitted_document_schema.mjs)

  1. Every instance is checked for well-formedness as emitted, before preprocessing (checkWellFormedBatch). This uses libxml2 (xmllint --noout, no schema), the parser LibreOffice's XML fast parser is built on. libxml2 reports namespace errors with exit status 0, so the gate fails on any error : line naming the file, not on the exit code. Instances that fail this check skip schema validation, so one problem is not reported twice.
  2. A pre-pass over the whole tree, checkMarkupCompatibilityAttributes, checks the attributes against ECMA-376 Part 3, 5th edition (2015). I checked these clauses against the vendored spec-compliance/ecma-376/source-artifacts/ECMA-376-3_5th_edition_december_2015.zip:
    • §7.2 mc:Ignorable and §7.4 mc:MustUnderstand: each prefix is bound to an in-scope namespace that is not the MC namespace.
    • §7.3 mc:ProcessContent: each token is prefix:localName or prefix:*, its prefix is bound to a namespace other than MC, and that namespace is declared ignorable at the same element or an ancestor.
    • No list repeats a token. This rule is the gate's, not Part 3's. §7.2 does not forbid a repeat, and LibreOffice loads one. The code comment says so. Annex A.1.2 and A.1.4 allow different prefixes that resolve to one namespace, and a test keeps them passing.
      The pre-pass visits content that preprocessing later removes (ignorable elements, mc:AlternateContent).
  3. The --self-test now also proves that the well-formedness pass rejects a namespace error hidden by mc:Ignorable.
  4. Inputs that share a basename used to collide on one scratch path, so a smoke over several runs' compared.docx validated only the last one. This was a latent bug, and it is fixed: each package and each projection gets its own scratch directory, and failures name the source package.

Fixture helper buildDocxFromBodyXml (packages/docx-core/src/testing/ooxml-fixtures.ts and its copy in packages/docx-compare/src/testing/ooxml-fixtures.ts)

  • It merges caller namespaces into the built-in w/w14/mc bindings. Restating a binding with the same URI is a no-op. A different URI for a built-in prefix throws.
  • ignorablePrefixes are listed once. Each one must be bound on the root and must not be the MC namespace. Every existing caller (git grep -n "ignorablePrefixes\|namespaces:" packages/*/src) passes bound, non-MC prefixes and keeps working. The fix(docx-compare): track header/footer table rows beside a text box and never wrap a cell in w:ins #1479 workaround that strips w14/mc is now unnecessary but harmless, so I left it alone.

Callers that built malformed input on purpose. The #1032 tests (packages/docx-core/src/primitives/xml_repeated_namespace_declaration.test.ts and packages/docx-compare/src/tagged/xmlPartRepeatedNamespaceDeclaration.test.ts) passed namespaces: { w: … } to get a repeated xmlns:w. They now build the clean package and write the repeat into word/document.xml themselves.

There is no @conformance tag. Part 3 has no entry in spec-compliance/registry/, and the citation lint scans only .ts. The Part 3 clauses are cited in prose.

Evidence

  • RED on base 1b1149e. I ran the new check_emitted_document_schema.test.mjs against the base gate (git show 1b1149e1:scripts/check_emitted_document_schema.mjs): 4 of 11 failed. The failures were the doubled-w14 compared root, the §6.3 duplicate hidden by preprocessing, an Ignorable prefix bound to MC, and the MustUnderstand/ProcessContent checks. The test for the literal repeated-declaration root passes on base and stays as a guard. After the fix, 11 of 11 pass.
  • RED for the fixture helper. The new tests, run against the base helper, fail with ParseError: Attribute xmlns:w14 redefined and "promise resolved instead of rejecting". After the fix, 5 of 5 pass in both packages.
  • Schema corpus: test:run ran with SDX_SCHEMA_CORPUS_DIR set, and the gate (--self-test --known-failures coverage/emitted-schema-known-failures.json) then ran over that corpus: 2339 of 2339 instances validate. Nothing new failed.
  • Pre-submit: build, lint:workspaces, check:spec-coverage, check:conformance-citations, check:conformance-doc, check:emitted-schema and check:real-corpus (4 files, 37 tests) all pass. test:run ran while the host's load average was over 100. Every failure was a 5 s timeout except the two docx-core: a part that declares xmlns:w twice on one element throws a raw ParseError instead of loading or failing closed #1032 tests above. I fixed those tests, then reran each failing file once the load had dropped (docx-compare 23 files, docx-markdoc 10, docx-core 4): all pass.
  • Dogfood compared outputs: 9 of 9 pass. Word-staged files (~/.cache/safe-docx-compare-fanout/word-staged/**/*.docx): 141 of 142 pass. The one failure, 1471/word-staged/sdt-del-word-children.docx, is a schema error (customXmlDelRangeStart as a direct w:sdt child) that the base gate reports the same way. It predates this change and is not a well-formedness or Markup Compatibility finding.

Left out

  • The gate still drops an ignorable element wholesale. It does not keep the content that mc:ProcessContent asks to be processed. Only the attribute's syntax and scope are checked now. No emitted corpus uses ProcessContent.
  • CI does not run node --test scripts/check_emitted_document_schema.test.mjs. It runs inside npm run check:emitted-schema, which is not a CI step. CI does run the gate's --self-test, which now covers the well-formedness pass. I did not edit the workflows.

Fixes #1484

…d mc:Ignorable tokens

The emitted-document gate rejected a root that repeats xmlns:w14 or
xmlns:mc, because xmldom throws on a repeated attribute. It did not reject
what such a fixture leaves in a comparison's output: docx-core's parser
drops the repeated declaration (#1032) but keeps mc:Ignorable="w14 w14 ...",
and the gate passed that. It also validated only the Markup Compatibility
projection, so two prefixes bound to one URI that gave an element the same
expanded attribute name (Namespaces in XML 1.0 section 6.3) passed whenever
mc:Ignorable dropped that namespace.

The gate now runs libxml2 over every instance as emitted, before
preprocessing, and fails on any error line (libxml2 reports namespace errors
with exit 0). A pre-pass checks mc:Ignorable, mc:MustUnderstand and
mc:ProcessContent against ECMA-376 Part 3 sections 7.2 to 7.4 (bound
prefix, not the MC namespace, ProcessContent token shape and ignorable
scope) and rejects a repeated token. The self-test now proves the
well-formedness pass rejects a namespace error.

buildDocxFromBodyXml (both copies) merges caller namespaces into its
built-in w/w14/mc bindings: restating a binding is a no-op, a different URI
for a built-in prefix throws, Ignorable prefixes are listed once and must be
bound and not the MC namespace.

Fixes: #1484
Every comparison run writes compared.docx, so a smoke over several runs
passes inputs that share a basename. The gate extracted each package to one
scratch path per basename, so only the last package was validated, once per
input. Each package and each projection now gets its own scratch directory,
and failures name the source package.

Ref: #1484
The #1032 tests built their malformed input by passing w in namespaces to
buildDocxFromBodyXml, which now declares each prefix once and rejects a
conflicting URI. They now build the clean package and write the repeated
declaration into word/document.xml themselves. The fixture comment also stops
naming the spec without a registry-backed citation.

Ref: #1484
@vercel

vercel Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
site Ready Ready Preview Nov 15, 2026 2:17pm UTC

Request Review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-15T14:01:31.438252Z 470acce PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@usejunior-llm-gate

usejunior-llm-gate Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

LLM gate (advisory)

All evaluated rules passed - 8 pass, 0 warn, 0 error, 8 skipped, 16 total

Findings

None.

All 16 rules (8 evaluated, 8 skipped)
Rule Verdict Detail
read_file response metadata parity SKIPPED paths not touched by this PR
Live DOM namespace-safe OOXML writes SKIPPED paths not touched by this PR
Complex-field revisions preserve complete accept/reject state machines PASS The PR touches XML namespace declaration handling and emitted schema validation in ooxml-fixtures.ts and check_emitted_document_schema.mjs, and does not touch field atomization, validateFieldStructure, w:fldChar, w:instrText, w:delInstrText, or collapsed-field comparison logic.
Field validation per story, not global SKIPPED paths not touched by this PR
Revision IDs seeded from all revision-bearing side parts SKIPPED paths not touched by this PR
Accept/reject sweep side parts and caches PASS The PR modifies OOXML fixture namespace handling and the emitted-document schema check in scripts/check_emitted_document_schema.mjs, and does not touch accept/reject logic or side-part revision handling.
DocumentViewNode.heading stays canonical SKIPPED paths not touched by this PR
AI-author parity across entry points SKIPPED paths not touched by this PR
Property-change wrapper discipline SKIPPED paths not touched by this PR
SUPPORT.md Table A drift vs. implementation PASS The PR only refactors test fixture generation in packages/docx-core/src/primitives/xml_repeated_namespace_declaration.test.ts and does not modify OOXML revision emission behavior or touch SUPPORT.md.
Table A / Table B boundary on side-part revisions SKIPPED paths not touched by this PR
Canonical-emission surface completeness PASS The PR only modifies test logic in packages/docx-core/src/primitives/xml_repeated_namespace_declaration.test.ts and does not touch tracked-edit surfaces in docx-core or docx-mcp.
Unit-test quality (avoid tautological / change-detector tests) PASS The tests added in packages/docx-core/src/testing/ooxml-fixtures.test.ts:85-126 and packages/docx-compare/src/testing/ooxml-fixtures.test.ts:83-124 evaluate XML output against independent DOMParser instances and verify explicit error messages, avoiding tautological checks or SUT mocking.
Re-derived facts vs canonical sources PASS The PR does not add logic that re-computes facts derived elsewhere (such as field walks, footnote references, or DOM visibility passes); it only updates root namespace deduplication in test fixture builders and adds well-formedness MCE checks to scripts/check_emitted_document_schema.mjs.
.openspec tag ↔ test-assertion drift PASS This PR modifies OOXML fixture namespace declaration helper functions and test cases, and does not add, move, or change any .openspec(...) tags.
Library stays general (no downstream-domain leakage) PASS The PR updates XML namespace handling in test fixture helpers (packages/docx-core/src/testing/ooxml-fixtures.ts). All added symbols use general XML/OOXML concepts and avoid any downstream domain concepts.

…ocessContent parts and stage instances under controlled names

Review of #1501: mc:MustUnderstand="xml" was rejected as unbound because
xmldom's lookup does not report the predefined xml binding (Namespaces in
XML 1.0 section 3). ProcessContent local parts such as ext:1bad or ext:**
passed; ECMA-376 Part 3 section 7.3 requires a local name (an NCName) or
exactly *. xmllint diagnostics were attributed by matching the input path,
which a path containing a newline, or one path prefixing another, defeats;
every instance is now staged as instance-<n>.xml in the scratch directory
and reported under its source name.

Ref: #1484
@stevenobiajulu

Copy link
Copy Markdown
Member Author

Codex peer review, round 1 (gpt-6.1-sol, medium) at 470acce: CHANGES_REQUIRED

Codex ran the gate tests, the self-test and both fixture suites, and probed the gate further. I accepted all three findings and fixed them in 6f4b3c9.

  1. Blocking. The implicit xml prefix was rejected. Confirmed: xmldom's lookupNamespaceURI('xml') returns null, so mc:MustUnderstand="xml" failed as unbound. The prefix is bound by definition (Namespaces in XML 1.0 §3). A resolvePrefix helper now maps xml to its namespace in both the check and the projection, and a test covers it.
  2. Blocking. Malformed ProcessContent local parts passed. Confirmed: ext:1bad and ext:** passed. Part 3 §7.3 requires a local name or exactly *. Prefixes and local parts are now validated against the NCName grammar (XML 1.0 §2.3 [4]/[4a] without the colon), including the Unicode ranges. A test rejects ext:1bad, ext:** and ext:a* and accepts ext:*, ext:wrap, ext:élément-2 and ext:_x.y.
  3. Non-blocking. Attributing stderr lines by input path could fail. Confirmed by reading the code: matching on ${file}: breaks when one path is a prefix of another or a path contains a newline. Every instance is now staged as instance-<n>.xml in the scratch directory, and xmllint only ever sees those names. Reports use the source name. A test covers prefix.xml beside prefix.xml:tail.xml: only the second one is reported.

Re-verified after the fixes:

  • Gate unit tests: 14/14 pass. --self-test passes.
  • Schema corpus captured from test:run: 2339/2339 pass.
  • Dogfood compared outputs: 9/9 pass.
  • Word-staged files: 141/142 pass. The one failure is the pre-existing customXmlDelRangeStart schema error, which the base gate reports the same way.

A confirmation round follows.

Review round 2 of #1501: like xml, the xmlns prefix is bound by definition
(Namespaces in XML 1.0 section 3), and xmldom's lookup does not report it,
so mc:Ignorable="xmlns" or mc:MustUnderstand="xmlns" was rejected as unbound.

Ref: #1484
@stevenobiajulu

Copy link
Copy Markdown
Member Author

Codex peer review, rounds 2 and 3

  • Round 2 at 6f4b3c9: CHANGES_REQUIRED. One blocking finding: resolvePrefix bound xml but not xmlns, which Namespaces in XML 1.0 §3 also binds by definition. Codex reproduced it: mc:Ignorable="xmlns" and mc:MustUnderstand="xmlns" were rejected as unbound. I accepted the finding and fixed it in bd41331, with a test covering xml/xmlns in Ignorable, MustUnderstand and ProcessContent="xmlns:*".
  • Round 3 at bd41331: APPROVE. Codex reproduced no findings. The prefix binding, the NCName validation and the instance staging all passed its confirmation probes.

After both rounds, the review left nothing behind: the working tree is clean, HEAD is unchanged and the branch is unchanged.

@stevenobiajulu

Copy link
Copy Markdown
Member Author

Pre-submit rerun at bd41331 passes:

  • build, lint:workspaces and test:run, which passed in every package with no failures this time
  • check:spec-coverage, check:conformance-citations, check:conformance-doc and check:emitted-schema
  • the gate over the SDX_SCHEMA_CORPUS_DIR corpus: 2352 of 2352 instances validate (--self-test --known-failures)
  • check:real-corpus: 4 files pass

@stevenobiajulu
stevenobiajulu enabled auto-merge (squash) November 15, 2026 14:33
@codecov

codecov Bot commented Nov 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@stevenobiajulu
stevenobiajulu merged commit e8d02e9 into main Nov 15, 2026
28 checks passed
@stevenobiajulu
stevenobiajulu deleted the 1484-schema-gate-duplicate-ns-20261115 branch November 15, 2026 14:46
@stevenobiajulu

Copy link
Copy Markdown
Member Author

Post-merge smoke passed

Squash commit e8d02e98 is on origin/main. Its tree (78ff1d57) is identical to the PR head bd41331, so I ran the smoke from the PR worktree, built at that head. The scripts are in the session scratchpad (scratchpad/1484/smoke/smoke.mjs and smoke-plain.mjs), and the gate is node scripts/check_emitted_document_schema.mjs <inputs>.

Gate over emitted corpora

Corpus Result
Dogfood compared outputs (scratchpad/dogfood/out/*/*/compared.docx) 9 of 9 pass
Word-staged compared files (~/.cache/safe-docx-compare-fanout/word-staged/**/*compared*.docx) 50 of 50 pass
Word-staged, all files, including originals and probes (run before the merge) 141 of 142 pass

The one failure, 1471/…/sdt-del-word-children.docx, has the same customXmlDelRangeStart schema error under the base gate.

#1484 reproduction

Instance Gate
Root with xmlns:w14 and xmlns:mc declared twice and mc:Ignorable="w14 w14" fail (Attribute xmlns:w14 redefined)
Pre-d98d03db fixture, original .docx fail
Its compared output, mc:Ignorable="w14 w14 wps" fail (mc:Ignorable repeats token 'w14'). It passed on main.
The same output with the token deduplicated pass
Two prefixes bound to one URI, a:k/b:k, hidden by mc:Ignorable fail (namespace error). It passed on main.

Real documents. NVCA Indemnification Agreement and NVCA Voting Agreement, from the real comparison corpus, each compared through compareDocuments (dist):

Follow-up (not filed). The #1032 repair in docx-core parseXml drops a repeated declaration but keeps a doubled Ignorable token from a malformed input. The comparison then carries that token into its output root. LibreOffice loads such a root, and Part 3 §7.2 does not forbid a repeat, so this is hygiene rather than a reader failure. The fix belongs in the repair or root-merge path (packages/docx-core/src/primitives/xml.ts).

Cleanup: remote branch 1484-schema-gate-duplicate-ns-20261115 deleted. The worktree is kept for the supervisor.

This branch was successfully deployed

1 active deployment
Preview — bd413316 Deployed Nov 15, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix PR type: bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(schema-gate): reject duplicate namespace declarations and Ignorable tokens that LibreOffice refuses

1 participant