Hand edit a generated file on purpose and prove the build objects

[tamnd]

The third of the four M1 blocking gates, and half of the exit criterion for this milestone.

What was untested

Two claims hold this repository up.

No number in a lesson is typed by a human.

The blueprints are generated, not transcribed.

Both are enforced the same way. The tool regenerates the file, compares it against what is committed, and fails on any difference. Both are therefore exactly as true as that comparison is, and nothing had ever checked that the comparison works.

That is the same criticism I made of the citation gate in #31 and the assertion checker in #33, and it applies here more sharply, because a comparison that always passed would look precisely like a repository where nobody has ever hand edited anything. Nobody would find out which of the two this is until a reader noticed a page saying something the program does not.

The test

xray check --selftest copies a real lesson and a real blueprint out of the tree, breaks each copy in one specific way, and runs the ordinary check over it.

ok    passes an untouched lesson
ok    refuses a number changed by hand in a captured output
ok    refuses a captured output deleted
ok    refuses a sentence added by hand to a generated page
ok    passes an untouched blueprint
ok    refuses a number changed by hand in a generated blueprint section
ok    refuses a sentence added by hand to a generated blueprint page
xray check --selftest: 0 failure(s)

The tamper for a number is to find the first digit in the file and bump it, which is the smallest edit that is still a lie, and it is the exact thing the first claim above says cannot survive. Each refusal is asserted for the message as well as the verdict, so a check that failed for some unrelated reason does not count as a pass.

The two cases that change nothing are the reason the other five mean anything. A harness that failed on everything, including work that is correct, would report a clean sweep without them.

Which is how the bug turned up

The control failed on its first run, and it was right to.

FAIL  passes an untouched lesson: it came back with 1 and said:
  line 114 produced: dotnet run --project tools/xray -- boss ../../../../var/folders/.../xray-check-selftest-98473
  line 114 on disk:  dotnet run --project tools/xray -- boss lessons/smoke-pipeline

The boss fight section of a generated page names the lesson so the reader can copy the grader command, and it was computing that path relative to the current working directory. So the page depended on where the tool was run from. Here it is before this change, on a clean checkout with nothing edited:

$ cd docs && dotnet run --project ../tools/xray -- check ../lessons
../lessons/smoke-pipeline/lesson.md: does not match what the code produces
  line 114 produced: dotnet run --project tools/xray -- boss ../lessons/smoke-pipeline
  line 114 on disk:  dotnet run --project tools/xray -- boss lessons/smoke-pipeline
xray check: 2 lesson(s), 2 problem(s)

A red build, on correct work, with a diff that makes no sense to whoever is looking at it. It never fired in CI because every job happens to run from the repository root, which is the sort of luck that runs out on the day somebody adds a job that does not.

The path is now taken relative to the top of the repository, found by walking up for ClrXray.slnx. Anything that goes into a generated file has to be relative to that rather than to the working directory, and Files.Root says so in a comment for the next person to put a path in a page.

Worth being plain about the order here: the five tamper cases were all passing while this bug was live, and they were passing for the wrong reason. Every one of them was detecting the path difference rather than the tamper. The control is the only thing in this pull request that could have told me that.

Also in here

The self test builds its copy under a marker file at a synthetic repository root, so the copy keeps its position in the tree and renders the same page the original does. That is not a workaround for the bug above, it is what makes the cases test the tamper rather than the move.

A new tamper job runs it on ubuntu-latest.

Checked locally

xray lint       18 files, 0 problems
xray check lessons     2 lessons, 0 problems
xray check docs        13 diagrams, 0 problems
xray check blueprints  1 blueprint, 0 problems
xray numbers lessons   2 lessons, 0 problems
xray numbers --selftest  0 failures
xray cite --selftest     0 failures
xray assert --selftest   0 failures
xray check --selftest    0 failures, 7 cases, 8 seconds
dotnet format --verify-no-changes  clean

Also checked that check lessons now passes from docs/ as well as from the root, which is the thing that was broken.

One thing this does not fix

main has no branch protection, so none of these gates is blocking in the sense the milestone means. A pull request with every check red can be merged today. That is a repository setting rather than code, and it is the next thing I am doing.