Skip to content

fix: make ingest the base layer, stop serving from tools/ - #582

Open
chris-dare-dev wants to merge 6 commits into
mainfrom
fix/batch-m-layering
Open

chris-dare-dev wants to merge 6 commits into
mainfrom
fix/batch-m-layering

Conversation

@chris-dare-dev

Copy link
Copy Markdown
Owner

Fix the package layering: ingest becomes the base layer, the server stops importing tools/, and the order is now tools → server → ingest.

#317: ingest imported server at module level

  • Before, four ingest modules imported server at module level, and ingest/store.py had to import server.corpus function-locally to dodge the cycle that made.
  • The modules ingest needs move into ingest as pure renames, and every importer points at the new paths:
    • server/application_paths.py → ingest/application_paths.py
    • server/corpus.py → ingest/corpus.py
    • server/theorem_names_store.py → ingest/theorem_names_store.py
    • server/retrieval/equations.py → ingest/mathml_tree.py
  • ApplicationPaths recognises a source checkout by finding its own directory under the repository root, so it now compares against ingest/. Otherwise every checkout run would fall through to the platform data directory.
  • server.ssl_pin stays in server: it resolves the CA bundle beside itself and builds server.config.Config. Ingest imports it at its fetch call sites only.

#293: the served process imported tools/

  • Before, the config validator, routes, handlers, resources, health scrape and corpus manifest imported the tools/ developer-utility package at 14 sites, so reorganising tools could break serving.
  • Every module the server needed is also used by ingest, or builds on it, so they move into ingest:
    • tools/_notebook_common.py → ingest/notebooks.py
    • tools/arxiv_fetch.py → ingest/arxiv_fetch.py
    • tools/_arxiv_api.py → ingest/arxiv_api.py
    • tools/oai_license.py → ingest/oai_license.py
    • tools/ingest_sentinel.py → ingest/sentinel.py
    • tools/security/ → ingest/security/
  • discover_for_notebook reads the server's notebooks store, so it moves to server/discovery.py instead.
  • The ingest-paused sentinel CLI now resolves its default path from ApplicationPaths. It used to write beside its own source, which on an installed wheel meant site-packages, and it ignored ARXMCP_OPS_DIR.
  • The ingest data-path probe skips the path resolver itself and the paths that name checkout code (the repository root and the LaTeXML sandbox profile), which the moved modules hold.

Changed commands

  • python -m tools.ingest_sentinel is now python -m ingest.sentinel.
  • python -m tools.discover_for_notebook is now python -m server.discovery.

The last commit merges main, which already had the equations and retrieval-routing PRs. It keeps search_papers on main's corpus_route code and points the new imports from those PRs at the moved modules.

Closes #317
Closes #293

Verification

  • ruff check . is clean, the Markdown allow-list check passes, and make wheel-check passes.
  • Full suite on this branch after the merge: 5618 passed, 182 skipped, 1 xfailed (Python 3.12, Ubuntu).
  • The merged tree is the same as the three-batch test merge (equations, retrieval routing and this branch on 7cc5620), plus only fix(server): reject non-string protocolVersion #579's changes.
  • server/ has no tools import, and ingest/ has no module-level server import.

About 1,340 changed lines over 177 files, above CREDO §5's 400-line guide. Almost all of it is renames and one-line import changes. Reviewing with git diff -M shows the moved files as renames.

Two READMEs outside the Markdown allow-list still name the old paths: ingest/security/README.md and apps/desktop/README.md. Any wiki page that gives the old module paths or the two old commands is also out of date.

ingest sits below the serving layer, but four ingest modules imported
server at module level, and ingest/store.py had to import server.corpus
function-locally to dodge the cycle that made.

Move the modules ingest needs into ingest, as pure renames:

  server/application_paths.py    -> ingest/application_paths.py
  server/corpus.py               -> ingest/corpus.py
  server/theorem_names_store.py  -> ingest/theorem_names_store.py
  server/retrieval/equations.py  -> ingest/mathml_tree.py

and point every importer at the new paths. server.ssl_pin stays in
server: it resolves the CA bundle beside itself and builds
server.config.Config, so bulk_ingest now imports it at its one call
site, as ingest already does for operator settings.

Refs #317
ApplicationPaths recognises a source checkout by finding its own
directory under the repository root. Now that the module lives in
ingest/, compare against that directory, not server/, or every
checkout run falls through to the platform data directory.

Refs #317
The served process imported the tools/ developer-utility package from
its config validator, routes, handlers, resources, health scrape and
corpus manifest, so reorganising tools could break serving. Every
module it needed is also used by ingest, or builds on ingest, so they
move into ingest, below the server:

  tools/_notebook_common.py   -> ingest/notebooks.py
  tools/arxiv_fetch.py        -> ingest/arxiv_fetch.py
  tools/_arxiv_api.py         -> ingest/arxiv_api.py
  tools/oai_license.py        -> ingest/oai_license.py
  tools/ingest_sentinel.py    -> ingest/sentinel.py
  tools/security/             -> ingest/security/

discover_for_notebook reads the server's notebooks store, so it moves
to server/discovery.py instead, CLI included (python -m
server.discovery). The sentinel CLI is now python -m ingest.sentinel.

ingest.notebooks imports server.ssl_pin at its one fetch call site,
since ssl_pin builds server.config.Config. The ingest data-path probe
now skips the path resolver itself and paths naming checkout code
(the repository root, the LaTeXML sandbox profile), which the moved
modules hold.

Refs #293
The ingest-paused sentinel CLI fell back to var/ beside the package's
own source, so an installed wheel wrote into site-packages and an
ARXMCP_OPS_DIR override was ignored. Resolve the default from
ApplicationPaths like every other ingest data path.

Refs #293
Docstrings and comments still described corpus as a server-layer
reader that ingest never imports, and named _notebook_common and
_arxiv_api by their old tools/ paths. Point them at ingest.corpus,
ingest.notebooks and ingest.arxiv_api.

Refs #293
Refs #317
Bring in the equations, retrieval-routing and protocolVersion changes.
search_papers keeps main's corpus_route seam, with the notebook helpers
imported from ingest.notebooks; the new equation, corpus and notebook
imports in ingest, server and tests point at the moved modules.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant