Skip to content

test: the audit measures what the tool supports; figures re-measured on 0.3.0 - #58

Merged
elvogel merged 1 commit into
mainfrom
fix/the-audit-measures-what-the-tool-supports
Oct 16, 2026
Merged

elvogel merged 1 commit into
mainfrom
fix/the-audit-measures-what-the-tool-supports

Conversation

@elvogel

@elvogel elvogel commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

The audit that produces docmd's published coverage figures measured less than the product
supports, and the figures it produced were a year-old release behind. Both fixed here, in that
order, because the second depends on the first.

The instrument

Corpus_Audit globbed *.docx. The converter accepts .docx, .docm, .dotx and .dotm, and
on Linux that glob matches case-sensitively as well -- so a .dotx or a .DOCX in an
audited folder was left out of the figure rather than counted. One of the 50 documents sampled
for this measurement is a .dotx, and the tree it came from holds seven files with an upper-case
extension.

private static readonly string[] WordExtensions = [".docx", ".docm", ".dotx", ".dotm"];

private static bool IsWordDocument(string path) =>
    !Path.GetFileName(path).StartsWith("~$", StringComparison.Ordinal)
    && WordExtensions.Contains(Path.GetExtension(path), StringComparer.OrdinalIgnoreCase);

~$ is Word's own lock file -- ownership metadata, never a readable package -- so it is skipped
rather than counted as a failure.

A file that is not an OPC package at all is now its own reported category instead of failing
the audit's one hard assertion. A real folder of client documents usually holds one: a Word
97-2003 binary that someone renamed to .docx. The CLI already calls that input error and exits
2 rather than 1, and the audit now draws the same line. Every other exception type still fails
the run, because an exception really does end a batch.

The figures

Re-measured on 0.3.0, corpus 394b2cb31e7459a1, 2026-10-15, run time 13 m 44 s:

Word documents found 50
Not a Word package 1 — reported, and excluded from the denominator
Converted without error 49 of 49
Lost not one word 45
Words lost 21 of 245,034 — 0.009%
Worst document, by words 11 words (99.8% kept)
Worst document, by share 2 of 177 words (98.9% kept)

Every one of the 21 words falls into a cause docs/limitations.md already records -- unmapped
w:sym characters, <pict>/<shape> wrappers, words fused at a figure-caption boundary, one
bare URL -- and the two causes 0.3.0 fixed produced no losses at all.

The corpus the old figures used is unreachable

README.md and docs/limitations.md quoted 27 of 50 against dca5084324a9022c. That corpus
cannot be re-measured. A corpus id is a hash of its documents' content, so it recognises a
corpus and cannot reconstruct one, and the _source.tsv that mapped those documents was not
kept anywhere on the machine.

This is the second time a corpus behind a published figure has become unreachable, and the first
time is already written into scripts/verify-corpus.sh: "by the time anyone tried to re-measure,
20 of its 21 names no longer resolved to a file." Content hashing was the fix for names; keeping
the manifest is the fix for hashes. Both documents now say so.

The two figures are therefore not comparable -- different documents and a different
converter. Both documents say that too, rather than inviting the reader to read an improvement
into the difference.

Also

Two limitations still said Status: open for defects that shipped in 0.3.0 -- the tab-indented
code block (#38) and the non-numeric w:ilvl (#37). Both are marked fixed, with the commit and
the verification that closed them, in the style the file already uses for fixed entries.

Suite: 394 total, 393 pass, 1 skip -- the skip being this audit with no DOCMD_CORPUS set,
which is how it is meant to behave.

🤖 Generated with Claude Code

https://claude.ai/code/session_018wZEgtuzaZPswykiGEBxbz

…on 0.3.0

The corpus audit globbed *.docx. The converter accepts .docx, .docm, .dotx and
.dotm, and on Linux that glob is case-sensitive as well, so a .dotx or a .DOCX
in an audited folder was left out of the figure rather than counted. One of the
50 sampled documents used here is exactly that case, and the tree it came from
holds seven files with an upper-case extension.

A file that is not an OPC package at all -- a Word 97-2003 binary renamed to
.docx -- is now its own reported category rather than a failed assertion. The
CLI already calls that input error and exits 2 rather than 1, so the audit
draws the same line; a real folder usually holds one, and failing for it buries
everything else the audit found. Every other exception type still fails.

Re-measured on 0.3.0, corpus 394b2cb31e7459a1, 2026-10-15
  Word documents found     50
  not a Word package       1, reported and excluded from the denominator
  documents converted      49 of 49, none threw
  lost not one word        45
  words lost               21 of 245,034, 0.009%
  worst document           11 words, 99.8% kept
  run time                 13 m 44 s

README.md and docs/limitations.md carried the 0.2.0 figure against corpus
dca5084324a9022c, which cannot be re-measured: a corpus id is a hash of its
documents' content, so it recognises a corpus and cannot reconstruct one, and
the _source.tsv that mapped that one was not kept. Both documents now record
that, and that the two figures are not comparable -- the converter changed too.

Two limitations that still said "Status: open" are marked fixed, with the
commit and the verification that closed them: #37 and #38 shipped in 0.3.0.

Suite: 394 total, 393 pass, 1 skip, the skip being this audit with no
DOCMD_CORPUS set.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018wZEgtuzaZPswykiGEBxbz
@elvogel
elvogel merged commit 3697b9b into main Oct 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant