Check coverage, overlap and structure

CHANGELOG.md

@@ -10,4 +10,5 @@
 - A draft is filled in by `in2lambda draft question add`, `in2lambda draft part add QUESTION` and `in2lambda draft question solution QUESTION`. Each takes `--text` to copy the wording out of the frozen source, as a block id such as `b3` or as lines such as `s10:14`, or `--literal TEXT` where the source does not say it in a form the field can take, which records the field as edited and written by layer 4 rather than 3. Question and part numbers are worked out from the fields already written rather than given, so a replay arrives at the same ids. `in2lambda draft split block BLOCK AT` cuts a block the parser made one of two things into `b3a` and `b3b`, so that each half can be quoted on its own. A command writing a field that is already written, or quoting lines another field was taken from, is refused naming both fields.
 - `in2lambda draft field replace FIELD OLD NEW` changes the wording inside a field that is already written, for the faults only an edit can fix - a brace the OCR dropped out of some maths, which no range of the source says correctly. OLD has to occur in the field exactly once, or the command is refused saying how many times it occurs; `--regex` reads it as a regular expression and NEW as what to replace it with. The field is left quoting the lines it was taken from, at the layer that wrote it, but recorded as edited and by whoever replaced the wording, so the change can be shown against the source.
 - `in2lambda spec run SPEC` runs a YAML file of selectors over the frozen source: it says which blocks are questions, parts and solutions, which to ignore, what to strip off the front of each one, and which of the four filters lays the solutions out. It fills in the draft's fields with the markdown of the lines each was taken from, records the spec's name and hash in the log so a replay runs the same file, and reports every block it made nothing of. Running an edited spec over a draft it has already filled in is refused, as freezing a document that has changed is: `in2lambda source add --start-over` begins the draft again. Reading a spec needs pyyaml, which the `convert` extra now installs alongside panflute. See [the spec page](https://lambda-feedback.github.io/in2lambda/spec.html) for the selectors and layouts.
+- `in2lambda validate` checks a draft over as a whole and writes what it finds into it as a `report`: source blocks in no field and not marked ignore, two fields taken from the same lines, gaps in the numbering of the questions or their parts, parts nothing answers, and fields holding nothing. Each finding names the field and the lines it is about, so it can be acted on without reading the draft. Finding something is not a failure and the command still exits 0; the report is replaced by the next run of the checks and dropped by the next command that changes the draft, since it describes the draft as it stood.
 - The Python API is unchanged: `in2lambda.main.runner` and everything under `in2lambda.api` take the same arguments and return the same objects.

docs/source/spec.md

@@ -7,11 +7,11 @@ every question comes out of the source rather than being retyped:
 ```bash
 $ in2lambda source add questions.docx
 $ in2lambda spec run spec.yaml
-b6 is in no field.
+b6 (lines 12-13) is in no field and not marked ignore.
 ```
 
-The last line is the point of it: a spec run reports every block it made nothing of, so what is
-left to account for is in front of you rather than quietly missing.
+The last line is the point of it: a spec run reports every block it made nothing of, naming the
+lines it is, so what is left to account for is in front of you rather than quietly missing.
 
 The fields a draft holds belong to the spec that wrote them, so a spec is run over a draft once.
 Running an edited one again is refused; freeze the document afresh and run it, which is two
@@ -91,9 +91,10 @@ the solutions are, and what each of them answers.
 
 ## What it writes
 
-Each question is `q1`, `q2` and so on in the order they appear, and each of its parts `q1.a`,
-`q1.b`. So a spec fills in `q1.text`, `q1.a.text`, `q1.a.solution` and, for a question answered
-as a whole, `q1.solution`. Every one of them records the lines it was copied from, and that a
+Each question is `q1`, `q2` and so on in the order they appear, and each of its parts `q1.p1`,
+`q1.p2`. So a spec fills in `q1.text`, `q1.p1.text`, `q1.p1.solution` and, for a question
+answered as a whole, `q1.solution`. They are the names the `in2lambda draft` commands give out
+as well, so a draft filled in either way is the same draft. Every one of them records the lines it was copied from, and that a
 spec wrote it.
 
 The spec is recorded in the draft's log with its hash, so `in2lambda draft replay` rebuilds the

in2lambda/draft/__init__.py

@@ -16,6 +16,7 @@
 from typing import Any
 
 import in2lambda.spec
+from in2lambda.draft.report import checks, overlapping, uncovered
 from in2lambda.source import (
     DRAFT,
     SourceError,
@@ -148,8 +149,11 @@ def record(
             "in2lambda source add --start-over to begin the draft again."
         )
     for filled, field in draft["fields"].items():
+        # Each of the field's ranges on its own, so that the refusal names the one in
+        # the way: a field edited by hand can be quoted from several, and the rest of
+        # them may be lines nobody wants.
         for taken in field["ranges"]:
-            if any(taken[0] <= end and start <= taken[1] for start, end in ranges):
+            if overlapping(ranges, [taken]):
                 raise AlreadyFilled(
                     f"Lines {taken[0]}-{taken[1]} are where {filled} came from, so "
                     f"they cannot also be {key}. Run in2lambda source show to see "
@@ -251,6 +255,9 @@ def apply(
     written = handler(draft, markdown, entry["args"], entry["by"], directory)
     # After the handler, so a command that was refused is not recorded as having run.
     draft["log"].append(entry)
+    # A report is about the draft as it was, so the command that changes it takes the
+    # report with it rather than leaving one that describes something else.
+    draft.pop("report", None)
     return written
 
 
@@ -306,6 +313,10 @@ def replay(directory: str = ".") -> None:
     }
     for entry in draft["log"]:
         apply(rebuilt, markdown, entry, directory)
+    # The one thing in a draft that no command wrote: the checks did, over the draft the
+    # commands left, so rebuilding it is running them again rather than copying it.
+    if "report" in draft:
+        rebuilt["report"] = checks(rebuilt)
 
     path = Path(directory) / DRAFT
     if serialise(rebuilt) != path.read_bytes():
@@ -316,32 +327,6 @@ def replay(directory: str = ".") -> None:
         )
 
 
-def coverage(draft: dict[str, Any]) -> list[str]:
-    """The blocks of a draft that nothing has made anything of yet.
-
-    Args:
-        draft: The draft to look over.
-
-    Returns:
-        The ids of the blocks that are in no field and have not been ignored, in
-        document order. A spec run prints these: they are what is left to account for,
-        and an empty list is the whole document spoken for.
-    """
-    ranges = [
-        line_range
-        for field in draft["fields"].values()
-        for line_range in field["ranges"]
-    ]
-    return [
-        block["id"]
-        for block in draft["blocks"]
-        if f"{block['id']}.ignore" not in draft["fields"]
-        and not any(
-            start <= block["end"] and block["start"] <= end for start, end in ranges
-        )
-    ]
-
-
 def _block(draft: dict[str, Any], block: str) -> dict[str, Any]:
     """One block of the frozen source, given the draft has one of that id.
 
@@ -675,20 +660,21 @@ def _spec_run(
     # The blocks the selectors run over are the ones the parser makes of the source, and
     # a `split block` since has left the draft holding halves the parser never made. So
     # an ignored block is named and ranged from here rather than from the draft: the
-    # field then spans the whole of what was ignored, and coverage, which goes by lines
-    # as well as by name, counts each half of a split block as covered by it.
+    # field then spans the whole of what was ignored, and `uncovered`, which goes by the
+    # lines a field was taken from, counts each half of a split block as covered by it.
     elements = _elements(markdown)
     fields, ignored = in2lambda.spec.fields(spec, elements, markdown)
     for found in fields:
         record(draft, found.key, found.value, layer=1, ranges=found.ranges, by=by)
     lines = {block.id: [block.start, block.end] for block, _ in elements}
-    # The field `mark ignore` writes, so that coverage need not care which said so.
+    # The field `mark ignore` writes, so that `uncovered` need not care which said so.
     for block_id in ignored:
         record(
             draft, f"{block_id}.ignore", True, layer=1, ranges=[lines[block_id]], by=by
         )
     # A spec writes a draft's worth of fields, so what it hands back is the other way
-    # round: what it made nothing of, which is what is left for anyone to act on.
-    if uncovered := coverage(draft):
-        return "\n".join(f"{block} is in no field." for block in uncovered)
+    # round: what it made nothing of, which is what is left for anyone to act on. Said
+    # in the words `in2lambda validate` says it in, since it is the same check.
+    if left_out := uncovered(draft):
+        return "\n".join(finding["message"] for finding in left_out)
     return "Every block is in a field or ignored."