Compare two sets as a library function and a command

CHANGELOG.md

@@ -25,4 +25,5 @@
 - `in2lambda convert FILE PartsOneSol` now exports the worked solution a document writes in a `solution` environment. Pandoc writes that environment as a Div whose classes hold `solution`, and the filter recognised only a Div whose first block reads `Solution`, so a document using the environment exported every question with an empty worked solution. A Div whose first block reads `Solution` is still recognised.
 - Importing `in2lambda.katex_convert` no longer writes a file called `log` into the working directory. That module reports what it changed in an expression to the `in2lambda.katex_convert` logger, which is silent unless the application configures logging.
 - `in2lambda convert` now reads a .docx that holds an image. in2lambda looks in the document for the directories a `\graphicspath` names, and read the document as UTF-8 text to find them. A .docx is a zip file, so converting a Word document holding a figure raised `UnicodeDecodeError`. in2lambda now reads a document that is not UTF-8 text as naming no directory, which is what a .docx names.
+- `in2lambda compare BUILT_ZIP EXPORT_DIR` compares two Lambda Feedback sets, each given as a folder or a zip, so that a set in2lambda wrote can be checked against the export it should reproduce. Each question's main text is compared, and each part's text and worked solution, and every difference is printed naming the question, the part and the field. Three differences in wording are taken off both sides first: a run of whitespace is compared as one space, an image is compared by the file's name, and a part holding neither text nor a worked solution is dropped where it is the question's only part. `--known FILE` names the differences the two sets are known to have, one line per difference with the ticket that would close it written after `  # `, and `in2lambda compare` exits 1 where the differences found are not the differences that file names. `in2lambda.compare.differences` and `in2lambda.compare.known` are the two functions behind the command.
 - The rest of the Python API is unchanged: `in2lambda.main.runner` and everything under `in2lambda.api` take the same arguments and return the same objects.

in2lambda/compare.py

@@ -0,0 +1,156 @@
+"""Compares two sets question by question, naming every place they say something else.
+
+`in2lambda convert` writes a set, `in2lambda build` writes a set from a draft, and Lambda
+Feedback exports a set. :func:`differences` compares any two of them in question and part
+order - each question's main text, and each part's text and worked solution - and returns
+one line per difference, naming the question, the part and the field as
+`in2lambda.validation` names them.
+
+Three differences in wording are not differences in what a question says, and are taken
+off both sides before comparing:
+
+- **Whitespace.** Every run of whitespace is compared as one space, because a draft
+  quotes the lines pandoc wrapped where `in2lambda convert` writes a paragraph on one
+  line.
+- **Image references.** An image is compared by the file's name, because
+  `in2lambda convert` writes every image as ``![pictureTag](path)`` where a draft keeps
+  the alt text the document wrote, and an export names each file as ``media/`` holds it
+  where the set `in2lambda convert` returns holds the path the document wrote.
+- **A lone empty part.** A single part holding neither text nor a worked solution is
+  dropped from both sides, because a question written without parts or solution exports
+  as one part holding nothing, where `in2lambda convert` writes no part at all.
+
+:func:`known` reads the differences two sets are known to have from a file: one line per
+difference as :func:`differences` words it, with the ticket that would close it written
+after ``  # ``.
+"""
+
+from itertools import zip_longest
+from pathlib import Path
+from typing import Any, Optional
+
+from in2lambda.api.question import Question
+from in2lambda.api.set import Set
+from in2lambda.json_convert.json_convert import _IMAGE
+from in2lambda.validation import _location
+
+_TICKET = "  # "
+"""What a line of a differs.txt names the ticket closing it after."""
+
+
+def _text(markdown: str) -> str:
+    """A field with the differences in wording that are not differences taken off.
+
+    Every run of whitespace becomes one space, and every image reference is written as
+    the file's name alone. The module docstring says why.
+    """
+    named = _IMAGE.sub(lambda reference: f"![]({Path(reference[1]).name})", markdown)
+    return " ".join(named.split())
+
+
+def _parts(question: Question) -> list[tuple[str, str]]:
+    """Each part's text and worked solution, dropping a lone part holding neither."""
+    parts = [(_text(part.text), _text(part.worked_solution)) for part in question.parts]
+    return [] if parts == [("", "")] else parts
+
+
+def _only(left: Optional[Any], thing: str, left_name: str, right_name: str) -> str:
+    """Which of the two sets holds a question or a part the other one does not."""
+    if left is None:
+        return f"{right_name} wrote this {thing} and {left_name} did not"
+    return f"{left_name} wrote this {thing} and {right_name} did not"
+
+
+def _differing(
+    where: str, left: str, right: str, left_name: str, right_name: str
+) -> list[str]:
+    """The line naming a field the two sets write differently, or no line at all."""
+    if left == right:
+        return []
+    return [f"{where}: {left_name} says {left!r} and {right_name} says {right!r}"]
+
+
+def differences(
+    built: Set,
+    expected: Set,
+    left_name: str = "the draft",
+    right_name: str = "convert",
+) -> list[str]:
+    r"""Every place the two sets say something different, in question and part order.
+
+    Args:
+        built: The set being checked, such as the one `in2lambda build` wrote.
+        expected: The set it should reproduce, such as a Lambda Feedback export.
+        left_name: What to call `built` in each line.
+        right_name: What to call `expected` in each line.
+
+    Returns:
+        One line per difference, naming the question, the part and the field as
+        `in2lambda.validation` names them and quoting what each set says there.
+
+    Examples:
+        >>> from in2lambda.api.set import Set
+        >>> from in2lambda.compare import differences
+        >>> built, expected = Set(), Set()
+        >>> built.add_question(main_text="The rocket  is at\n45 degrees.")
+        >>> expected.add_question(main_text="The rocket is at 45 degrees.")
+        >>> differences(built, expected)
+        []
+        >>> expected.add_question(main_text="Find the impulse.")
+        >>> differences(built, expected)
+        ['Question 2 "": convert wrote this question and the draft did not']
+    """
+    found = []
+    questions = zip_longest(built.questions, expected.questions)
+    for number, (built_question, expected_question) in enumerate(questions, start=1):
+        if built_question is None or expected_question is None:
+            found.append(
+                f"{_location(number, '')}: "
+                f"{_only(built_question, 'question', left_name, right_name)}"
+            )
+            continue
+        found += _differing(
+            _location(number, "", field="main text"),
+            _text(built_question.main_text),
+            _text(expected_question.main_text),
+            left_name,
+            right_name,
+        )
+        parts = zip_longest(_parts(built_question), _parts(expected_question))
+        for index, (built_part, expected_part) in enumerate(parts):
+            if built_part is None or expected_part is None:
+                found.append(
+                    f"{_location(number, '', index)}: "
+                    f"{_only(built_part, 'part', left_name, right_name)}"
+                )
+                continue
+            for field, built_value, expected_value in zip(
+                ("text", "worked solution"), built_part, expected_part
+            ):
+                found += _differing(
+                    _location(number, "", index, field),
+                    built_value,
+                    expected_value,
+                    left_name,
+                    right_name,
+                )
+    return found
+
+
+def known(path: str | Path) -> list[str]:
+    """The differences two sets are known to have, as a differs.txt file holds them.
+
+    Args:
+        path: The file to read. A file that does not exist names no difference, so that
+            a folder of fixtures holds one only where the two sets differ.
+
+    Returns:
+        Each line as :func:`differences` words it, with the ticket written after ``  # ``
+        taken off and blank lines dropped.
+    """
+    path = Path(path)
+    if not path.is_file():
+        return []
+    return [
+        line.split(_TICKET)[0] for line in path.read_text().splitlines() if line.strip()
+    ]

in2lambda/main.py

@@ -5,12 +5,14 @@
 import os
 import shlex
 import warnings
+import zipfile
 from collections.abc import Callable  # Rather than typing's, which beartype warns on.
 from contextlib import contextmanager
 from typing import Any, Optional
 
 import rich_click as click
 
+import in2lambda.compare
 import in2lambda.draft
 import in2lambda.draft.export
 import in2lambda.draft.report
@@ -606,5 +608,74 @@ def render(output_dir: str, draft: Optional[str]) -> None:
         click.echo(f"Wrote {pdf}")
 
 
+def _set_at(path: str) -> Set:
+    """The set at `path`, or a message naming `path` where it holds no set.
+
+    `Set.from_json` raises `ValueError` where a folder or a zip holds no ``set_*.json``,
+    and `zipfile.BadZipFile` where a path named ``.zip`` is not a zip at all. `compare`
+    reads two paths, so the message names which of the two is at fault.
+    """
+    try:
+        return Set.from_json(path)
+    except (ValueError, zipfile.BadZipFile):
+        raise click.ClickException(
+            f"{path} is not a Lambda Feedback set. A set is a folder or a zip holding "
+            "one set_*.json file beside a question_*.json file per question."
+        ) from None
+
+
+@cli.command("compare")
+@click.argument("built_zip", type=click.Path(exists=True))
+@click.argument("export_dir", type=click.Path(exists=True))
+@click.option(
+    "--known",
+    "known_path",
+    type=click.Path(exists=True, dir_okay=False),
+    help="File naming the differences the two sets are known to have, one per line.",
+)
+def compare(built_zip: str, export_dir: str, known_path: Optional[str]) -> None:
+    """Compares the set in BUILT_ZIP with the set in EXPORT_DIR, and prints each difference.
+
+    Each argument is a Lambda Feedback set, as a folder or as a zip. Each question's main
+    text is compared, and each part's text and worked solution, and every difference is
+    printed naming the question, the part and the field. Three differences in wording are
+    taken off both sides first: a run of whitespace is compared as one space, an image is
+    compared by the file's name, and a lone empty part is dropped. in2lambda.compare says
+    why. --known names a file of the differences the two sets are known to have, one per
+    line as this command prints it, with a ticket written after "  # ". in2lambda compare
+    exits 1 where the differences found are not the differences --known names.
+    """
+    with _message_not_traceback():
+        found = in2lambda.compare.differences(
+            _set_at(built_zip),
+            _set_at(export_dir),
+            left_name=built_zip,
+            right_name=export_dir,
+        )
+    for line in found:
+        click.echo(line)
+
+    expected = in2lambda.compare.known(known_path) if known_path else []
+    if found == expected:
+        if not found:
+            click.echo("Identical.")
+        return
+    # Echoed rather than put in the message, because a difference is a long line and
+    # the message is printed in a box that wraps it.
+    for line in expected:
+        if line not in found:
+            click.echo(f"Not found: {line}")
+    if not known_path:
+        raise click.ClickException(
+            "The two sets differ in the places printed above. Pass --known FILE to "
+            "name the differences the two sets are known to have."
+        )
+    raise click.ClickException(
+        f"The differences printed above are not the differences {known_path} names. "
+        f"Write one line of {known_path} per difference found, with the ticket that "
+        'would close it after "  # ".'
+    )
+
+
 if __name__ == "__main__":
     cli()

tests/fixtures/against_convert/README.md

@@ -23,6 +23,9 @@ document itself, as the ranges in `fixtures/sources` are. They were produced wit
 
 ## What the comparison ignores
 
+The comparison is `in2lambda.compare.differences`, whose docstring states these rules, so
+that this README and the code do not drift apart.
+
 Each question's main text is compared, and each part's text and worked solution. Three
 differences between the routes are not differences in what a question says, and are taken
 off both sides before comparing: