Skip to content

Never open PubMed web pages; open no-LinkOut PLoS articles at doi.org from the keyed esummary DOI (ace-no-pubmed-page) - #69

Merged
jdkent merged 2 commits into
masterfrom
ace-no-pubmed-page
Oct 14, 2026
Merged

jdkent merged 2 commits into
masterfrom
ace-no-pubmed-page

Conversation

@jdkent

@jdkent jdkent commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Follow-up to #68. ACE no longer opens PubMed web pages (pubmed.ncbi.nlm.nih.gov) in its browser.

Why

After #68, get_html_by_pmid opens the page _prlinks_target picks from the keyed elink JSON. When an article has no LinkOut record, that page was the PubMed abstract page, https://pubmed.ncbi.nlm.nih.gov/<pmid>/.

Since about 00:00 CDT on 2026-10-14, PubMed answers the ingestion browser on beast with an Incapsula/Imperva block page ("Request unsuccessful. Incapsula incident ID ..."). Before that it answered "Too Many Requests". Between 05:00 and 12:30 UTC, 1,639 of these pages appeared, and a sample of 30 were all this no-LinkOut fallback. NCBI wants scripts to use E-utilities, not the website.

Decision. On 2026-10-14 the recommendation was: "don't bypass this block; it's PubMed telling us to stop ... stop ACE opening PubMed web pages entirely ... get the PLOS DOI from the keyed E-utilities lookup". The user asked for this PR on that basis ("create this PR"). So this PR doesn't try to get around the block in any way. It stops asking.

The abstract page was only useful for PLoS: check_for_substitute_url read its citation_doi meta tag to build the PLoS manuscript URL. For any other journal it isn't full text.

Change

With no LinkOut record, get_html_by_pmid makes one keyed esummary call (PubMedAPI.get('esummary', retmode=json), the same client and key as the elink).

  • PLoS article: if articleids has a DOI starting 10.1371/journal., it opens https://doi.org/<doi>. That redirects (via dx.plos.org) to the journals.plos.org/<site>/article?id=<doi> HTML page, which is the same page the LinkOut path reaches through dx.plos.org.
  • Anything else: it opens nothing, logs No LinkOut record for PMID <pmid> and not a PLoS article. Skipping... once, and returns None. process_article then returns (filename, None) and writes no file. retrieve_articles counts the article as invalid and writes it to invalid_article_log_file, as for any other failed scrape.
  • Failed lookup: this covers a request error, NCBI's HTTP 200 per-UID error reply ({"uid": "99999999", "error": "cannot get document summary"}), and a record without articleids or idtype. Each is logged once at ERROR (E-utilities summary failed for PMID <pmid>: ..., through _without_api_key) and returns None, which is reported invalid the same way.

Why doi.org and not the manuscript URL. The first version of this PR opened journals.plos.org/<site>/article/file?id=<doi>&type=manuscript. That URL 302s to an XML file served with content-disposition: attachment. requests follows it, but Chrome downloads it. In browser mode, which ns-pond uses, ACE therefore got Chrome's "New Tab" page, _validate_scrape accepted it, and the XML piled up in ./downloaded_files/ (found in review). The doi.org page loads in both modes. In requests mode with a PLoS journal name, check_for_substitute_url still rewrites the article page to the manuscript, which is the path test_journal_scraping takes.

Why the DOI prefix decides "is it PLoS". The caller's journal can't be used: ns-pond calls process_article(pmid, "IngestionWorkflow", ...). Matching fulljournalname against PLOS_JOURNAL_SITES misses newer titles, among them PLOS Mental Health (616 records), Global Public Health (5,004), Digital Health (1,293), Climate and Water. Every PLOS journal article DOI starts 10.1371/journal., and doi.org needs no site slug.

DOI. It is taken from esummary articleids (idtype == "doi"), read strictly. elocationid is a display string ("doi: 10.1371/...").

Also:

  • _is_pubmed_landing_page and _extract_citation_doi are removed, along with the citation_doi branch of check_for_substitute_url. Nothing else used them.
  • _prlinks_target(providers) no longer takes a PMID or returns a PubMed URL. It is only called when there is at least one provider.
  • Before, a non-PLoS no-LinkOut article got the abstract page. That page passes _validate_scrape (see test_validate_scrape_allows_embedded_recaptcha_widget), so ACE could save it as the article. Now nothing is saved and the article is reported invalid.
  • Each no-LinkOut article now costs one extra keyed E-utilities request, which replaces one request to the PubMed website.

Not changed here: two rules for "is it PLoS". check_for_substitute_url still decides PLoS from the caller's journal. Because ns-pond passes "IngestionWorkflow", that branch, and the other journal-name branches there and in browser-mode get_html, never fire in production. Settling that is a separate item. This PR only makes both routes land on the same journals.plos.org article page.

Browser mode, verified by hand on beast

Browser mode can't be recorded, so the cassette tests run requests mode. In a fresh venv at this head, from an empty working directory with PUBMED_API_KEY set, I ran Scraper(store).get_html_by_pmid(pmid, "IngestionWorkflow", mode="browser"):

PMID journal page returned _validate_scrape citation_doi on the page
42868208 PLoS One the article page (2.28 MB), title "Practice-based initial teacher education in alternative teaching certification programs: ..." True 10.1371/journal.pone.0360047
42868163 PLOS Mental Health the article page (2.36 MB), title "Associations between hurricane stressors, social capital, and climate anxiety ..." True 10.1371/journal.pmen.0000749

Afterwards downloaded_files/ held only seleniumbase's driver_fixing.lock: no XML.

Tests

The cassettes are real keyed responses, recorded on beast through conftest.vcr_config (api_key filtered from the query, and the key scrubbed from bodies and headers).

conftest.py now also records, through vcr's before_record_request, the URI of every request a cassette is asked for, whether it was served or refused (fixture attempted_urls). A refused request raises inside the code under test, where a broad except can swallow it, so play_counts / all_played alone can't show that a PubMed request was never made. The no-LinkOut tests assert set(attempted_urls) == set(played).

test PMID checks
..._without_linkout_opens_the_plos_article_by_doi[plos-one] 42868208 (PLoS One, no LinkOut) journal passed is 'Other journal'; played: prlinks elink, esummary, https://doi.org/10.1371/journal.pone.0360047, then dx.plos.org → journals.plos.org/plosone/article?id=...; nothing else attempted; the page has the article's citation_doi and passes _validate_scrape
...[plos-mental-health] 42868163 (PLOS Mental Health, not in PLOS_JOURNAL_SITES) same, ending on journals.plos.org/mentalhealth/article?id=...
..._opens_nothing_for_other_journals[book-chapter] 28520367 (Bookshelf chapter, no DOI) journal passed is 'PLoS ONE'; returns None; elink + esummary only, played and attempted; one "not a PLoS article" line; no ERROR
...[article-with-a-doi] 42869849 (ACR Open Rheumatol, DOI 10.1002/...) same. Only this one stops "any DOI → open it"
..._logs_an_esummary_error_reply_as_a_failed_lookup 99999999 (no such PMID: elink has no LinkOut, esummary gives HTTP 200 + "error") returns None; logs E-utilities summary failed for PMID 99999999: cannot get document summary; no "Skipping" line; nothing else attempted
test_process_article_without_linkout_reports_an_invalid_article 28520367 (replays the chapter cassette) (filename, None), no file written, nothing else attempted
test_plos_summary_failure_is_logged_without_the_api_key 42868208, key not-a-real-key NCBI's real 400 "API key invalid" (request + 5 retries); returns None; only the esummary attempted; the log has <api_key> and not the key, and no "Skipping" line
test_journal_scraping (existing) 38814972, 38814901 (PLoS One) Its cassette was recorded when these had no LinkOut, so it runs end to end: retrieve_articles → esummary → doi.org → article page → manuscript XML (the journal passed is 'PLoS ONE'). The cassette keeps only interactions the code plays: the originals byte for byte, plus the recorded esummary replies and doi.org chains. The old retmode=ref hops and PubMed pages are removed. I added one assertion: no attempted request to pubmed.ncbi.nlm.nih.gov.

A PLoS record without a DOI: I found none. Under the prefix rule such a record has no separate path. It takes the no-DOI branch, which the book chapter covers (B7 below).

Existing tests that encoded the removed behaviour:

  • test_get_html_by_pmid_without_linkout_opens_the_pubmed_page (Open the prlinks target from the keyed elink lookup instead of the unkeyed eutils hop #68) asserted that the PubMed page was opened. It is replaced by the no-LinkOut tests above, and its cassette is deleted.
  • test_plos_substitute_url_from_pubmed_landing_page tested the citation_doi path, so it is removed.
  • test_plos_substitute_url_covers_sibling_journals reached the PLoS Biology mapping through a PubMed URL plus citation_doi. It now uses the journals.plos.org/plosbiology/article?id=... publisher URL, with the same assertion.

All other #68 tests are unchanged and pass.

Suite. I used a fresh Python 3.12 uv venv on beast: requirements.txt plus urllib3<2, vcrpy==6.0.1, pytest-recording==0.13.1, requests==2.31.0, pytest==8.0.1. seleniumbase resolves to 4.24.0, and setuptools<81 is needed for its pytest-html plugin. The run was pytest --block-network --allowed-hosts=127.0.0.1, without PUBMED_API_KEY:

  • master 787ea9f: 55 passed, 10 failed
  • this branch: 60 passed, 10 failed

The 10 failures are the same on both. They are the tests that make live calls with no cassette (test_pmc_embedded_table, test_wiley_label, test_elsivier_table_parse, test_multi_column_float_conversion, test_frontier_table_identification, test_schizophrenia_research_source, test_find_tables_in_old_sciencedirect, test_old_springer_source, test_old_table_parser, test_empty_columns).

I grepped every cassette for the real key, not-a-real-key and api_key=: no matches.

Mutations

These are edits to ace/scrape.py, each run against the same offline suite. 20 of 22 are killed: B6 survives and B10 is equivalent. The M series is mine, and the B series is the test reviewer's, adapted to the new code.

id mutant result killed by
M1 no LinkOut → open pubmed.ncbi.nlm.nih.gov/<pmid>/ again killed both PLoS, both other-journal, error-reply, process_article, journal_scraping
M2 drop the PLoS DOI lookup (no LinkOut → nothing) killed same 7
M3 DOI from elocationid as given killed both PLoS, journal_scraping
M4 DOI = first articleids value, any type killed both PLoS, journal_scraping
M5 PLoS decided by the caller's journal killed both PLoS, [article-with-a-doi], summary-failure
M6 PLoS decided by fulljournalname in PLOS_JOURNAL_SITES (the first version's rule) killed [plos-mental-health]
M7 open the manuscript download URL instead of doi.org killed both PLoS
M8 esummary error logged without _without_api_key killed summary-failure
M9 open the lookup result even when it is None killed both other-journal, error-reply, process_article
M10 PLoS publisher URL not rewritten to the manuscript killed substitute_url publisher + sibling, journal_scraping
M11 any DOI opened as a PLoS article killed [article-with-a-doi]
M12 a failed lookup also logged as a skip killed error-reply
B1 non-PLoS no-LinkOut falls back to the PubMed page killed both other-journal, process_article
B2 PubMed page fetched inside the esummary-failure except, errors swallowed killed error-reply, summary-failure (attempted-request hook)
B3 PubMed page fetched for every no-LinkOut PMID, errors swallowed killed all 8 no-LinkOut tests incl. journal_scraping (attempted-request hook)
B4 esummary failure swallowed as "not PLoS" (no error log) killed error-reply, summary-failure
B5 lenient esummary parse (no error check, .get defaults) killed error-reply
B6 any 10.1371/ DOI accepted, not only 10.1371/journal. survived none. The only non-journal 10.1371/ DOIs are PLoS Currents (10.1371/currents.*), and all 10 I sampled have LinkOut, so there is no no-LinkOut record to pin it with.
B7 drop the doi is None guard killed [book-chapter], process_article
B8 no-LinkOut skip flagged as a configured skip (article silently dropped) killed process_article
B9 esummary sent without the API key killed summary-failure
B10 summary read from uids[-1] instead of str(pmid) survived, equivalent (one UID per request) n/a

🤖 Generated with Claude Code

jdkent and others added 2 commits October 14, 2026 08:11
… by DOI

When an article has no LinkOut record, get_html_by_pmid opened
https://pubmed.ncbi.nlm.nih.gov/<pmid>/ in the browser. Since 2026-10-14
PubMed answers beast's browser with an Incapsula block page, and the
abstract page was only ever useful for a PLoS citation_doi.

Now, with no LinkOut, the keyed esummary decides: a PLoS journal
(fulljournalname) with a DOI in articleids goes straight to the PLoS
manuscript URL; anything else opens nothing and is logged, which
process_article and retrieve_articles report as an invalid article.
_is_pubmed_landing_page and _extract_citation_doi are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of #69: in browser mode the PLoS manuscript URL is a download
(content-disposition: attachment), so Chrome saved the XML to
downloaded_files/ and ACE kept Chrome's "New Tab" page as the article.
https://doi.org/<doi> lands on the journals.plos.org article page, the
same page LinkOut gives, and loads in both modes.

PLoS is now decided by the 10.1371/journal. DOI prefix, so titles missing
from PLOS_JOURNAL_SITES (PLOS Mental Health, Global Public Health, ...)
are found. An esummary error reply (HTTP 200, per-UID "error") and a
malformed record are logged once as a failed lookup, not as "not PLoS".

Tests record every request a cassette is asked for, so a PubMed request
swallowed by a broad except still fails them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jdkent jdkent changed the title Never open PubMed web pages; address no-LinkOut PLoS articles by their esummary DOI (ace-no-pubmed-page) Never open PubMed web pages; open no-LinkOut PLoS articles at doi.org from the keyed esummary DOI (ace-no-pubmed-page) Oct 14, 2026
@jdkent
jdkent marked this pull request as ready for review October 14, 2026 14:05
@jdkent jdkent self-assigned this Oct 14, 2026

@jdkent jdkent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant