Repository navigation
Conversation
…nal name The journal argument is only the storage folder, so rules keyed on it never fired for ns-pond's IngestionWorkflow runs. Rules now key on the final URL or page content: - PLoS: any journals.plos.org article URL gets its NLM XML in requests mode; the slug and DOI come from the URL. Browser mode never opens the manuscript download. PLOS_JOURNAL_SITES is gone. - Frontiers: keyed on frontiersin.org, replacement r'/xml/nlm'. - Wiley and J Cognitive Neuroscience URL rewrites and the Wiley relatedArticles wait are removed (they do nothing on current sites). - Table-expand links are clicked on every page that has them, and page_source is read after the clicks and waits. - The retry loop reloads the substituted URL, and the first Driver is quit before the second is created. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ked is skipped, not fatal The first load loop quit its driver on a timeout and then retried with the same dead driver, so no retry could succeed. Expanding tables now runs on every page that has them, so one stale or covered link must not fail the whole article: it is logged and that table stays collapsed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s mode The browser path now runs in one try/finally, so an exception from a page call no longer leaves Chrome running. Every driver gets the page-load timeout, and a slow substitute or retry load is read as far as it got. html starts as None, so ten failed page_source reads return None instead of raising. The PLoS manuscript and the Frontiers NLM XML come from one function, _xml_url, used only in requests mode: a browser downloads the first and wraps the second in its XML viewer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jdkent
marked this pull request as ready for review
October 14, 2026 19:11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #70.
journalnow does one job: it names the folder a page is saved under (html/<journal>/<pmid>.html). The publisher rules key on the page's own URL or content. ns-pond passes"IngestionWorkflow", so none of the journal-name rules ever fired there; see #70 for the measurements.Changes (
ace/scrape.py)journaldropped fromcheck_for_substitute_url,get_htmlandget_html_by_pmid;process_articlekeeps it for the storage path.journals.plos.org/<site>/article?id=<doi>URL; the site slug and DOI come from the path.PLOS_JOURNAL_SITESis gone, so newer titles (Mental Health, Digital Health, Global Public Health, ...) are covered. The manuscript XML is used in requests mode only: in browser mode the manuscript is served as an attachment, which Chrome downloads instead of showing (Never open PubMed web pages; open no-LinkOut PLoS articles at doi.org from the keyed esummary DOI (ace-no-pubmed-page) #69).frontiersin.orgURL, with replacementr'/xml/nlm'. The old'xml\/nlm'produced a literal backslash, and that URL returns 404.doi/abs→doi/fullrewrite and the WileyrelatedArticleswait. These are no-ops on current URLs: LinkOut lands ononlinelibrary.wiley.com/doi/<doi>(full text) anddirect.mit.edu..table-expand-inlinelink (any journal: J Neurosci, AJNR, J Nucl Med, BMJ Case Rep, ...), now happen beforepage_sourceis read. Before this,htmlwas read first and that earlier copy was returned, so expanded tables never reached the result.Table pages (requested follow-up)
These were checked from a laptop with a few requests, without getting around bot protection:
T<n>.expansion.html(sources.py, J Neurosci10.1523/JNEUROSCI.1359-19.2019, AJNR10.3174/ajnr.A6000): a Cloudflare challenge, so it can't be verified and is unchanged. Expanding the inline tables in the article page (above) is the route that doesn't depend on those pages./article/<doi>/tables/<n>(10.1007/s10548-018-0650-y): 200, but it redirects to?error=cookies_not_supportedwithout a session. Unchanged.Tests
xml/nlm. Existing PLoS/Frontiers expectations were updated for the URL-keyed rule.--block-network --allowed-hosts=127.0.0.1), pinned venv (urllib3 1.26.20, requests 2.31.0, vcrpy 6.0.1, pytest 8.0.1, seleniumbase 4.24.0):Browser-mode check (beast, by hand)
get_html(url, mode="browser")with this branch (934db97) and with master (a21b603), each in its own fresh venv and an empty working directory. The table count is<tablein the returned HTML./content/36/5/1758)/content/20/20/7752)/content/36/2/375)/content/61/1/96)10.3174/ajnr.A600010.1002/hbm.2501010.1016/j.neuroimage.2019.116189downloaded_files/..table-expand-inlinelinks and 0 saved tables.After merge (ns-pond)
J Neurosci pages and the other collapsed-table pages need their ACE downloads refreshed to pick up the tables: 2,818 J Neurosci articles have ACE as their only text and no table from any source.
Review fix pass (703bb6d)
try/finally, so an exception fromdriver.title,find_elementsor a page load no longer leaves Chrome running. This leak is real on master: after ns-pond's ACE passes, beast had 359 orphaned Chrome processes holding about 13 GB. Killing them dropped memory in use from 20 GB to 7 GB.htmlstarts asNone, so ten failedpage_sourcereads returnNoneinstead of raisingUnboundLocalError._xml_url. A browser downloads the first and wraps the second in its XML viewer, so browser mode reads the HTML page, andcheck_for_substitute_urlkeeps only the HTML rewrites (ScienceDirect, Springer)./fullpage loads as HTML. No Chrome left running after the run, and no downloaded files.Callers this breaks
journalis gone fromget_html,get_html_by_pmidandcheck_for_substitute_url;process_articlestill takes it, for the storage folder. ns-pond only callsprocess_article, so it is unaffected. These callers in other repos break:CurateNSPond/src/curate_ns_pond/fulltext.py:131passesjournal=toget_html_by_pmid(TypeError).autonima-results/scripts/ace_get_unavailable.py:444callscheck_for_substitute_url(url, html, journal)(TypeError). Its subclassget_html(url, journal, ...)at :504-515 passesjournalpositionally intomode.They need a one-line update each when they move to this ACE.
🤖 Generated with Claude Code