Repository navigation
fix: make ingest the base layer, stop serving from tools/ - #582
Open
chris-dare-dev wants to merge 6 commits into
Open
chris-dare-dev wants to merge 6 commits into
chris-dare-dev wants to merge 6 commits into
Conversation
ingest sits below the serving layer, but four ingest modules imported server at module level, and ingest/store.py had to import server.corpus function-locally to dodge the cycle that made. Move the modules ingest needs into ingest, as pure renames: server/application_paths.py -> ingest/application_paths.py server/corpus.py -> ingest/corpus.py server/theorem_names_store.py -> ingest/theorem_names_store.py server/retrieval/equations.py -> ingest/mathml_tree.py and point every importer at the new paths. server.ssl_pin stays in server: it resolves the CA bundle beside itself and builds server.config.Config, so bulk_ingest now imports it at its one call site, as ingest already does for operator settings. Refs #317
ApplicationPaths recognises a source checkout by finding its own directory under the repository root. Now that the module lives in ingest/, compare against that directory, not server/, or every checkout run falls through to the platform data directory. Refs #317
The served process imported the tools/ developer-utility package from its config validator, routes, handlers, resources, health scrape and corpus manifest, so reorganising tools could break serving. Every module it needed is also used by ingest, or builds on ingest, so they move into ingest, below the server: tools/_notebook_common.py -> ingest/notebooks.py tools/arxiv_fetch.py -> ingest/arxiv_fetch.py tools/_arxiv_api.py -> ingest/arxiv_api.py tools/oai_license.py -> ingest/oai_license.py tools/ingest_sentinel.py -> ingest/sentinel.py tools/security/ -> ingest/security/ discover_for_notebook reads the server's notebooks store, so it moves to server/discovery.py instead, CLI included (python -m server.discovery). The sentinel CLI is now python -m ingest.sentinel. ingest.notebooks imports server.ssl_pin at its one fetch call site, since ssl_pin builds server.config.Config. The ingest data-path probe now skips the path resolver itself and paths naming checkout code (the repository root, the LaTeXML sandbox profile), which the moved modules hold. Refs #293
The ingest-paused sentinel CLI fell back to var/ beside the package's own source, so an installed wheel wrote into site-packages and an ARXMCP_OPS_DIR override was ignored. Resolve the default from ApplicationPaths like every other ingest data path. Refs #293
Bring in the equations, retrieval-routing and protocolVersion changes. search_papers keeps main's corpus_route seam, with the notebook helpers imported from ingest.notebooks; the new equation, corpus and notebook imports in ingest, server and tests point at the moved modules.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix the package layering:
ingestbecomes the base layer, the server stops importingtools/, and the order is now tools → server → ingest.#317: ingest imported server at module level
serverat module level, andingest/store.pyhad to importserver.corpusfunction-locally to dodge the cycle that made.server/application_paths.py→ingest/application_paths.pyserver/corpus.py→ingest/corpus.pyserver/theorem_names_store.py→ingest/theorem_names_store.pyserver/retrieval/equations.py→ingest/mathml_tree.pyApplicationPathsrecognises a source checkout by finding its own directory under the repository root, so it now compares againstingest/. Otherwise every checkout run would fall through to the platform data directory.server.ssl_pinstays inserver: it resolves the CA bundle beside itself and buildsserver.config.Config. Ingest imports it at its fetch call sites only.#293: the served process imported
tools/tools/developer-utility package at 14 sites, so reorganisingtoolscould break serving.tools/_notebook_common.py→ingest/notebooks.pytools/arxiv_fetch.py→ingest/arxiv_fetch.pytools/_arxiv_api.py→ingest/arxiv_api.pytools/oai_license.py→ingest/oai_license.pytools/ingest_sentinel.py→ingest/sentinel.pytools/security/→ingest/security/discover_for_notebookreads the server's notebooks store, so it moves toserver/discovery.pyinstead.ApplicationPaths. It used to write beside its own source, which on an installed wheel meantsite-packages, and it ignoredARXMCP_OPS_DIR.Changed commands
python -m tools.ingest_sentinelis nowpython -m ingest.sentinel.python -m tools.discover_for_notebookis nowpython -m server.discovery.The last commit merges
main, which already had the equations and retrieval-routing PRs. It keepssearch_papersonmain'scorpus_routecode and points the new imports from those PRs at the moved modules.Closes #317
Closes #293
Verification
ruff check .is clean, the Markdown allow-list check passes, andmake wheel-checkpasses.7cc5620), plus only fix(server): reject non-string protocolVersion #579's changes.server/has notoolsimport, andingest/has no module-levelserverimport.About 1,340 changed lines over 177 files, above CREDO §5's 400-line guide. Almost all of it is renames and one-line import changes. Reviewing with
git diff -Mshows the moved files as renames.Two READMEs outside the Markdown allow-list still name the old paths:
ingest/security/README.mdandapps/desktop/README.md. Any wiki page that gives the old module paths or the two old commands is also out of date.