Repository navigation
Never open PubMed web pages; open no-LinkOut PLoS articles at doi.org from the keyed esummary DOI (ace-no-pubmed-page) - #69
Merged
Conversation
… by DOI When an article has no LinkOut record, get_html_by_pmid opened https://pubmed.ncbi.nlm.nih.gov/<pmid>/ in the browser. Since 2026-10-14 PubMed answers beast's browser with an Incapsula block page, and the abstract page was only ever useful for a PLoS citation_doi. Now, with no LinkOut, the keyed esummary decides: a PLoS journal (fulljournalname) with a DOI in articleids goes straight to the PLoS manuscript URL; anything else opens nothing and is logged, which process_article and retrieve_articles report as an invalid article. _is_pubmed_landing_page and _extract_citation_doi are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of #69: in browser mode the PLoS manuscript URL is a download (content-disposition: attachment), so Chrome saved the XML to downloaded_files/ and ACE kept Chrome's "New Tab" page as the article. https://doi.org/<doi> lands on the journals.plos.org article page, the same page LinkOut gives, and loads in both modes. PLoS is now decided by the 10.1371/journal. DOI prefix, so titles missing from PLOS_JOURNAL_SITES (PLOS Mental Health, Global Public Health, ...) are found. An esummary error reply (HTTP 200, per-UID "error") and a malformed record are logged once as a failed lookup, not as "not PLoS". Tests record every request a cassette is asked for, so a PubMed request swallowed by a broad except still fails them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jdkent
marked this pull request as ready for review
October 14, 2026 14:05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #68. ACE no longer opens PubMed web pages (
pubmed.ncbi.nlm.nih.gov) in its browser.Why
After #68,
get_html_by_pmidopens the page_prlinks_targetpicks from the keyed elink JSON. When an article has no LinkOut record, that page was the PubMed abstract page,https://pubmed.ncbi.nlm.nih.gov/<pmid>/.Since about 00:00 CDT on 2026-10-14, PubMed answers the ingestion browser on beast with an Incapsula/Imperva block page ("Request unsuccessful. Incapsula incident ID ..."). Before that it answered "Too Many Requests". Between 05:00 and 12:30 UTC, 1,639 of these pages appeared, and a sample of 30 were all this no-LinkOut fallback. NCBI wants scripts to use E-utilities, not the website.
Decision. On 2026-10-14 the recommendation was: "don't bypass this block; it's PubMed telling us to stop ... stop ACE opening PubMed web pages entirely ... get the PLOS DOI from the keyed E-utilities lookup". The user asked for this PR on that basis ("create this PR"). So this PR doesn't try to get around the block in any way. It stops asking.
The abstract page was only useful for PLoS:
check_for_substitute_urlread itscitation_doimeta tag to build the PLoS manuscript URL. For any other journal it isn't full text.Change
With no LinkOut record,
get_html_by_pmidmakes one keyedesummarycall (PubMedAPI.get('esummary', retmode=json), the same client and key as the elink).articleidshas a DOI starting10.1371/journal., it openshttps://doi.org/<doi>. That redirects (viadx.plos.org) to thejournals.plos.org/<site>/article?id=<doi>HTML page, which is the same page the LinkOut path reaches throughdx.plos.org.No LinkOut record for PMID <pmid> and not a PLoS article. Skipping...once, and returnsNone.process_articlethen returns(filename, None)and writes no file.retrieve_articlescounts the article as invalid and writes it toinvalid_article_log_file, as for any other failed scrape.{"uid": "99999999", "error": "cannot get document summary"}), and a record withoutarticleidsoridtype. Each is logged once at ERROR (E-utilities summary failed for PMID <pmid>: ..., through_without_api_key) and returnsNone, which is reported invalid the same way.Why doi.org and not the manuscript URL. The first version of this PR opened
journals.plos.org/<site>/article/file?id=<doi>&type=manuscript. That URL 302s to an XML file served withcontent-disposition: attachment.requestsfollows it, but Chrome downloads it. In browser mode, which ns-pond uses, ACE therefore got Chrome's "New Tab" page,_validate_scrapeaccepted it, and the XML piled up in./downloaded_files/(found in review). The doi.org page loads in both modes. In requests mode with a PLoS journal name,check_for_substitute_urlstill rewrites the article page to the manuscript, which is the pathtest_journal_scrapingtakes.Why the DOI prefix decides "is it PLoS". The caller's
journalcan't be used: ns-pond callsprocess_article(pmid, "IngestionWorkflow", ...). MatchingfulljournalnameagainstPLOS_JOURNAL_SITESmisses newer titles, among them PLOS Mental Health (616 records), Global Public Health (5,004), Digital Health (1,293), Climate and Water. Every PLOS journal article DOI starts10.1371/journal., and doi.org needs no site slug.DOI. It is taken from esummary
articleids(idtype == "doi"), read strictly.elocationidis a display string ("doi: 10.1371/...").Also:
_is_pubmed_landing_pageand_extract_citation_doiare removed, along with thecitation_doibranch ofcheck_for_substitute_url. Nothing else used them._prlinks_target(providers)no longer takes a PMID or returns a PubMed URL. It is only called when there is at least one provider._validate_scrape(seetest_validate_scrape_allows_embedded_recaptcha_widget), so ACE could save it as the article. Now nothing is saved and the article is reported invalid.Not changed here: two rules for "is it PLoS".
check_for_substitute_urlstill decides PLoS from the caller'sjournal. Because ns-pond passes"IngestionWorkflow", that branch, and the other journal-name branches there and in browser-modeget_html, never fire in production. Settling that is a separate item. This PR only makes both routes land on the samejournals.plos.orgarticle page.Browser mode, verified by hand on beast
Browser mode can't be recorded, so the cassette tests run
requestsmode. In a fresh venv at this head, from an empty working directory withPUBMED_API_KEYset, I ranScraper(store).get_html_by_pmid(pmid, "IngestionWorkflow", mode="browser"):_validate_scrapecitation_doion the pageAfterwards
downloaded_files/held only seleniumbase'sdriver_fixing.lock: no XML.Tests
The cassettes are real keyed responses, recorded on beast through
conftest.vcr_config(api_keyfiltered from the query, and the key scrubbed from bodies and headers).conftest.pynow also records, through vcr'sbefore_record_request, the URI of every request a cassette is asked for, whether it was served or refused (fixtureattempted_urls). A refused request raises inside the code under test, where a broadexceptcan swallow it, soplay_counts/all_playedalone can't show that a PubMed request was never made. The no-LinkOut tests assertset(attempted_urls) == set(played)...._without_linkout_opens_the_plos_article_by_doi[plos-one]'Other journal'; played: prlinks elink, esummary,https://doi.org/10.1371/journal.pone.0360047, thendx.plos.org→journals.plos.org/plosone/article?id=...; nothing else attempted; the page has the article'scitation_doiand passes_validate_scrape...[plos-mental-health]PLOS_JOURNAL_SITES)journals.plos.org/mentalhealth/article?id=......_opens_nothing_for_other_journals[book-chapter]'PLoS ONE'; returnsNone; elink + esummary only, played and attempted; one "not a PLoS article" line; no ERROR...[article-with-a-doi]10.1002/...)..._logs_an_esummary_error_reply_as_a_failed_lookup"error")None; logsE-utilities summary failed for PMID 99999999: cannot get document summary; no "Skipping" line; nothing else attemptedtest_process_article_without_linkout_reports_an_invalid_article(filename, None), no file written, nothing else attemptedtest_plos_summary_failure_is_logged_without_the_api_keynot-a-real-keyNone; only the esummary attempted; the log has<api_key>and not the key, and no "Skipping" linetest_journal_scraping(existing)retrieve_articles→ esummary → doi.org → article page → manuscript XML (the journal passed is'PLoS ONE'). The cassette keeps only interactions the code plays: the originals byte for byte, plus the recorded esummary replies and doi.org chains. The oldretmode=refhops and PubMed pages are removed. I added one assertion: no attempted request topubmed.ncbi.nlm.nih.gov.A PLoS record without a DOI: I found none. Under the prefix rule such a record has no separate path. It takes the no-DOI branch, which the book chapter covers (B7 below).
Existing tests that encoded the removed behaviour:
test_get_html_by_pmid_without_linkout_opens_the_pubmed_page(Open the prlinks target from the keyed elink lookup instead of the unkeyed eutils hop #68) asserted that the PubMed page was opened. It is replaced by the no-LinkOut tests above, and its cassette is deleted.test_plos_substitute_url_from_pubmed_landing_pagetested thecitation_doipath, so it is removed.test_plos_substitute_url_covers_sibling_journalsreached the PLoS Biology mapping through a PubMed URL pluscitation_doi. It now uses thejournals.plos.org/plosbiology/article?id=...publisher URL, with the same assertion.All other #68 tests are unchanged and pass.
Suite. I used a fresh Python 3.12 uv venv on beast:
requirements.txtplusurllib3<2,vcrpy==6.0.1,pytest-recording==0.13.1,requests==2.31.0,pytest==8.0.1. seleniumbase resolves to 4.24.0, andsetuptools<81is needed for its pytest-html plugin. The run waspytest --block-network --allowed-hosts=127.0.0.1, withoutPUBMED_API_KEY:The 10 failures are the same on both. They are the tests that make live calls with no cassette (
test_pmc_embedded_table,test_wiley_label,test_elsivier_table_parse,test_multi_column_float_conversion,test_frontier_table_identification,test_schizophrenia_research_source,test_find_tables_in_old_sciencedirect,test_old_springer_source,test_old_table_parser,test_empty_columns).I grepped every cassette for the real key,
not-a-real-keyandapi_key=: no matches.Mutations
These are edits to
ace/scrape.py, each run against the same offline suite. 20 of 22 are killed: B6 survives and B10 is equivalent. The M series is mine, and the B series is the test reviewer's, adapted to the new code.pubmed.ncbi.nlm.nih.gov/<pmid>/againelocationidas givenarticleidsvalue, any typejournal[article-with-a-doi], summary-failurefulljournalnameinPLOS_JOURNAL_SITES(the first version's rule)[plos-mental-health]_without_api_keyNone[article-with-a-doi]except, errors swallowed.getdefaults)10.1371/DOI accepted, not only10.1371/journal.10.1371/DOIs are PLoS Currents (10.1371/currents.*), and all 10 I sampled have LinkOut, so there is no no-LinkOut record to pin it with.doi is Noneguard[book-chapter], process_articleuids[-1]instead ofstr(pmid)🤖 Generated with Claude Code