Skip to content

Key publisher rewrites and page actions on the page URL, not the journal folder name - #71

Merged
jdkent merged 3 commits into
masterfrom
ace-70
Oct 14, 2026
Merged

jdkent merged 3 commits into
masterfrom
ace-70

Conversation

@jdkent

@jdkent jdkent commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #70.

journal now does one job: it names the folder a page is saved under (html/<journal>/<pmid>.html). The publisher rules key on the page's own URL or content. ns-pond passes "IngestionWorkflow", so none of the journal-name rules ever fired there; see #70 for the measurements.

Changes (ace/scrape.py)

  • journal dropped from check_for_substitute_url, get_html and get_html_by_pmid; process_article keeps it for the storage path.
  • PLoS: keyed on the journals.plos.org/<site>/article?id=<doi> URL; the site slug and DOI come from the path. PLOS_JOURNAL_SITES is gone, so newer titles (Mental Health, Digital Health, Global Public Health, ...) are covered. The manuscript XML is used in requests mode only: in browser mode the manuscript is served as an attachment, which Chrome downloads instead of showing (Never open PubMed web pages; open no-LinkOut PLoS articles at doi.org from the keyed esummary DOI (ace-no-pubmed-page) #69).
  • Frontiers: keyed on the frontiersin.org URL, with replacement r'/xml/nlm'. The old 'xml\/nlm' produced a literal backslash, and that URL returns 404.
  • Removed: the Wiley abstract→full rewrite, the JoCN doi/abs→doi/full rewrite and the Wiley relatedArticles wait. These are no-ops on current URLs: LinkOut lands on onlinelibrary.wiley.com/doi/<doi> (full text) and direct.mit.edu.
  • Read after acting: the ScienceDirect and Wiley waits, and clicking every .table-expand-inline link (any journal: J Neurosci, AJNR, J Nucl Med, BMJ Case Rep, ...), now happen before page_source is read. Before this, html was read first and that earlier copy was returned, so expanded tables never reached the result.
  • Browser handling:
    • the old driver is quit before a substitute URL opens a new one;
    • the page-source retry reloads the substituted URL;
    • the first load loop now starts a new browser for each retry (it used to retry with the driver it had just quit);
    • a table link that is stale or covered is logged and skipped instead of failing the article.

Table pages (requested follow-up)

These were checked from a laptop with a few requests, without getting around bot protection:

  • HighWire T<n>.expansion.html (sources.py, J Neurosci 10.1523/JNEUROSCI.1359-19.2019, AJNR 10.3174/ajnr.A6000): a Cloudflare challenge, so it can't be verified and is unchanged. Expanding the inline tables in the article page (above) is the route that doesn't depend on those pages.
  • Springer /article/<doi>/tables/<n> (10.1007/s10548-018-0650-y): 200, but it redirects to ?error=cookies_not_supported without a session. Unchanged.
  • No table-URL pattern was found to 404 or to move to a new form, so no table URL builder changed.

Tests

  • New: recorded cassettes for the PLoS-by-URL rule (PLoS One and PLOS Mental Health) and for Frontiers xml/nlm. Existing PLoS/Frontiers expectations were updated for the URL-keyed rule.
  • Suite, offline (--block-network --allowed-hosts=127.0.0.1), pinned venv (urllib3 1.26.20, requests 2.31.0, vcrpy 6.0.1, pytest 8.0.1, seleniumbase 4.24.0):
    • master a21b603: 60 passed, 10 failed;
    • this branch: 64 passed, 10 failed.
    • The 10 failures are the same tests on both: they make live calls and have no cassette.
  • No cassette contains the API key (grepped against the real key).
  • Not covered by automated tests: the browser-mode changes (clicks, waits, driver lifecycle), because they can't be recorded. They were checked by hand instead, below.

Browser-mode check (beast, by hand)

get_html(url, mode="browser") with this branch (934db97) and with master (a21b603), each in its own fresh venv and an empty working directory. The table count is <table in the returned HTML.

Page master branch
J Neurosci, PMID 26843655 (/content/36/5/1758) 0 1
J Neurosci, PMID 11027238 (/content/20/20/7752) 0 1
J Neurosci, PMID 26758830 (/content/36/2/375) 0 0
J Nucl Med, PMID 31324712 (/content/61/1/96) 0 2
AJNR, 10.3174/ajnr.A6000 0 2
Wiley HBM, 10.1002/hbm.25010 3 3
ScienceDirect, 10.1016/j.neuroimage.2019.116189 0 0
  • 4 of the 5 collapsed-table pages now return their tables.
  • For 26758830 the expand links didn't produce a table. The fail-safe handling logged it and returned the page, as before.
  • Neither version left a file in downloaded_files/.
  • Of 25 recent J Neurosci pages in ns-pond's ACE cache, 18 have .table-expand-inline links and 0 saved tables.

After merge (ns-pond)

J Neurosci pages and the other collapsed-table pages need their ACE downloads refreshed to pick up the tables: 2,818 J Neurosci articles have ACE as their only text and no table from any source.

Review fix pass (703bb6d)

  • Chrome is closed on every way out. The browser path runs in one try/finally, so an exception from driver.title, find_elements or a page load no longer leaves Chrome running. This leak is real on master: after ns-pond's ACE passes, beast had 359 orphaned Chrome processes holding about 13 GB. Killing them dropped memory in use from 20 GB to 7 GB.
  • Every driver gets the 10 s page-load timeout. A slow substitute or retry load is read as far as it got instead of failing the article.
  • html starts as None, so ten failed page_source reads return None instead of raising UnboundLocalError.
  • XML rewrites are requests-mode only. The PLoS manuscript and the Frontiers NLM XML both come from one function, _xml_url. A browser downloads the first and wraps the second in its XML viewer, so browser mode reads the HTML page, and check_for_substitute_url keeps only the HTML rewrites (ScienceDirect, Springer).
  • Suite (offline, same pins): 65 passed, 10 failed, the same 10 live-call failures as master.
  • Browser re-check at 703bb6d (beast): J Neurosci 26843655 1 table, J Nucl Med 31324712 2, AJNR 2. The Frontiers /full page loads as HTML. No Chrome left running after the run, and no downloaded files.

Callers this breaks

journal is gone from get_html, get_html_by_pmid and check_for_substitute_url; process_article still takes it, for the storage folder. ns-pond only calls process_article, so it is unaffected. These callers in other repos break:

  • CurateNSPond/src/curate_ns_pond/fulltext.py:131 passes journal= to get_html_by_pmid (TypeError).
  • autonima-results/scripts/ace_get_unavailable.py:444 calls check_for_substitute_url(url, html, journal) (TypeError). Its subclass get_html(url, journal, ...) at :504-515 passes journal positionally into mode.

They need a one-line update each when they move to this ACE.

🤖 Generated with Claude Code

jdkent and others added 3 commits October 14, 2026 12:37
…nal name

The journal argument is only the storage folder, so rules keyed on it
never fired for ns-pond's IngestionWorkflow runs. Rules now key on the
final URL or page content:

- PLoS: any journals.plos.org article URL gets its NLM XML in requests
  mode; the slug and DOI come from the URL. Browser mode never opens the
  manuscript download. PLOS_JOURNAL_SITES is gone.
- Frontiers: keyed on frontiersin.org, replacement r'/xml/nlm'.
- Wiley and J Cognitive Neuroscience URL rewrites and the Wiley
  relatedArticles wait are removed (they do nothing on current sites).
- Table-expand links are clicked on every page that has them, and
  page_source is read after the clicks and waits.
- The retry loop reloads the substituted URL, and the first Driver is
  quit before the second is created.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ked is skipped, not fatal

The first load loop quit its driver on a timeout and then retried with the
same dead driver, so no retry could succeed. Expanding tables now runs on
every page that has them, so one stale or covered link must not fail the
whole article: it is logged and that table stays collapsed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s mode

The browser path now runs in one try/finally, so an exception from a page
call no longer leaves Chrome running. Every driver gets the page-load timeout,
and a slow substitute or retry load is read as far as it got. html starts as
None, so ten failed page_source reads return None instead of raising. The
PLoS manuscript and the Frontiers NLM XML come from one function, _xml_url,
used only in requests mode: a browser downloads the first and wraps the
second in its XML viewer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jdkent
jdkent marked this pull request as ready for review October 14, 2026 19:11
@jdkent jdkent self-assigned this Oct 14, 2026

@jdkent jdkent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jdkent
jdkent merged commit e50b384 into master Oct 14, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Key publisher rewrites and page actions on the page URL/content, not the caller's journal folder name

1 participant