Repository navigation
Open the prlinks target from the keyed elink lookup instead of the unkeyed eutils hop - #68
Merged
Merged
Conversation
get_html_by_pmid sent the browser to elink.fcgi?cmd=prlinks&retmode=ref,
a request to E-utilities without the API key, so NCBI rate limited it per
IP and the saved page was often {"error":"API rate limit exceeded"}. The
keyed JSON elink call already asks for cmd=prlinks; open the URL the
redirect would have gone to instead: the first LinkOut provider that isn't
PMC, PMC when it is the only one, and the PubMed abstract page when there
are none. Errors from the keyed calls are logged with the key redacted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The URL tests read cassette.requests, which is the stored cassette, so they passed without the code making those requests. Assert on the interactions the cassette actually served instead, and add recorded tests for PMC preference, PMC as the only provider, both "only" exits, the skip_url_substrings pre-check, and an invalid-key 400 from the keyed elink (returns None, no unkeyed fallback, key redacted at the real log sites). The helper-only redaction test is replaced by those. Parse the elink JSON into (provider, URL) pairs once. Drop the raise_for_status call: PubMedAPI.get already raises on any non-200. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- "only" on 25709573 replays the prefers-PMC recording and must open
PMC4325919; before, an "only" run that never looked up a PMCID passed.
- The "only" + failed-PMCID test now replays the real recorded 400s for
the PMCID lookup and checks its log lines, instead of relying on vcr
refusing an unrecorded request.
- Record the real {"linksets": []} reply (empty id) and check it counts
as a failed lookup.
- NCBI echoes the key in error bodies, which filter_query_parameters
doesn't reach: store bodies decoded and replace the key (env and the
tests' fake key) in bodies and headers. A local server echoing the key
checks the scrub on every run. The two 400 cassettes are re-recorded
through it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jdkent
marked this pull request as ready for review
October 14, 2026 00:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
get_html_by_pmidmade a keyed JSONelinkcall (thePubMedAPIthat was givenapi_key). For articles that don't come from PMC, it then sent the browser tohttp://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi?dbfrom=pubmed&id=<pmid>&cmd=prlinks&retmode=ref. That second request goes to E-utilities without the key, so NCBI applies its per-IP limit of 3 requests a second. With 3 ACE workers on one host, nearly half of those hops came back asand that was saved as the page (e.g. PMID 20979788). In the ingestion run on 2026-10-13, 39 of 63 pages set aside in 4 minutes were this error. The previous night's run had 2,664.
Fix
The keyed JSON call already sends
cmd=prlinks(PubMedAPI.elinkwith the defaultaccess_db), so it asks the same question as the browser hop.get_html_by_pmidnow opens the URL that theretmode=refredirect would have landed on, and never sends the browser through E-utilities:https://pubmed.ncbi.nlm.nih.gov/<pmid>/when there is no LinkOut record. That is where the redirect lands today (it redirects to/<pmid>, which then 301s to/<pmid>/), so the PubMed-landing-page handling incheck_for_substitute_url(PLoScitation_doi) still applies unchanged.The PMC path (
prefer_pmc_source, including"only") is unchanged. The elink JSON is parsed once into (provider, URL) pairs. A response withoutlinksets[0].idurllist[0].objurlsnow counts as a failed lookup, instead of being read as "no LinkOut" and sent to the PubMed page. Theskip_url_substringspre-check over all provider URLs now runs on every path. Before, it was skipped whenprefer_pmc_source=Falsewithout skip substrings (nothing to check), and when PMC was listed but the PMCID lookup failed. In that caseget_htmlstill checked the URL after the redirect.Behaviour changes worth knowing:
elinkcall fails,get_html_by_pmidlogs the error and returnsNone. Before, it fell back to the unkeyed hop, which is the request that was rate limited.retrieve_articlesrecords the article as invalid, as it does for any other failed scrape.prefer_pmc_source=Falseand there are no skip substrings. This replaces the unkeyed request, so there is one NCBI request per article either way, and it is now keyed.retmodeargument ofget_html_by_pmidis removed. It only set theretmodeof the browser hop, and nothing in ACE passes it.logger.info(query)logged the eutils URL. It now logs the provider URL. Errors from the keyedelinkand PMCID calls are logged with the API key replaced by<api_key>, becauserequestsexceptions quote the full query string.Provider URL equivalence
I checked the JSON
objurlsagainst theLocationofcmd=prlinks&retmode=ref(keyed, redirects not followed) for 96 distinct PMIDs from NeuroImage, HBM, J Neurosci, Cereb Cortex, Brain Res, Front Hum Neurosci, PLoS ONE, eLife, PNAS, Sci Rep, Brain, Cortex, J Physiol 1960, Yale J Biol Med 1980, Cochrane and others. The rule above matched the redirect target in all 96:linkinghub.elsevier.com/...; 20979788 →www.yiigle.com/LinkIn.do?...doi.org/10.3389/fnhum.2014.01073, not PMCwww.jneurosci.org/cgi/pmidlookup?...pmc.ncbi.nlm.nih.gov/articles/pmid/20311739/pubmed.ncbi.nlm.nih.gov/28520367The JSON list puts PMC wherever its name sorts, so taking the first URL would be wrong for the 23 PMC-first cases; the redirect skips PMC whenever another provider exists. No PMID in the sample had two non-PMC providers. For that case the code takes the first one in the JSON order.
Tests
The cassette tests run the real
get_html_by_pmidandget_htmlinrequestsmode. The URL choice is shared withbrowsermode, which can't be recorded. The assertions check the interactions the cassette actually served (play_counts/all_played), notcassette.requests: that is the stored file, so asserting on it passes even when the code makes no request. As a result, a test fails on the run that records its cassette and passes once it plays back...._opens_the_provider_url_from_the_keyed_lookupprefer_pmc_source=Falsedoi.org/10.3389/fnhum.2014.01073(PMC is listed first), and no other eutils request..._without_linkout_opens_the_pubmed_pagepubmed.ncbi.nlm.nih.gov/28520367/. In requests mode PubMed returns its proof-of-work challenge page (HTTP 203), so only the URL is asserted...._prefers_pmcTruepubmed_pmcelink, thenpmc.ncbi.nlm.nih.gov/articles/PMC4325919; no publisher request..._opens_pmc_when_it_is_the_only_providerFalsepmc.ncbi.nlm.nih.gov/articles/pmid/20311739/..._pmc_only_skips_articles_without_pmc"only"None; only the elink is served..._pmc_only_opens_pmc"only"PMC4325919..._pmc_only_skips_when_the_pmcid_lookup_fails"only"None; the log has both "Failed to resolve PMCID" and "PMC source detected, but PMCID lookup failed"..._treats_an_elink_reply_without_linksets_as_a_failed_lookup''{"linksets": []}reply (HTTP 200) returnsNoneand opens nothing..._skips_a_blocked_provider_it_would_not_openpmc.ncbi.nlm.nih.govNone; HighWire is the target and PMC is listed second; the skip flag is set; only the elink is served..._returns_none_when_the_keyed_lookup_failsnot-a-real-keyPubMedAPI's 5 retries); returnsNone; nothing but those elinks is served;caploghas<api_key>and not the keytest_pmcid_lookup_failure_is_logged_without_the_api_key_get_pmcid_from_pmidlog sitetest_recorded_cassettes_do_not_store_the_api_keyvcr_confighas<api_key>in both places and no keybackoff_factorto 0. The retries stay, but the 30 s of sleeps go.pubmed_pmclink (0 of 62), so the "only"/failed-PMCID test pairs two real recordings: the prlinks reply for 25709573 and the invalid-key 400s for its PMCID lookup.ace/tests/conftest.pyfiltersapi_keyfrom the query. That filter also applies to matching, so cassettes play back with or withoutPUBMED_API_KEY. NCBI also echoes the key in its error bodies ("api-key":"<key>"), so bodies are stored decoded (decode_compressed_response), and abefore_record_responsehook replaces the key fromPUBMED_API_KEYand the tests' fake key with<api_key>in bodies and headers. The two 400 cassettes were re-recorded through the hook. I grepped every cassette for the real key,not-a-real-keyandapi_key=: no matches.Mutations
I ran the 53 offline tests with
--block-network --allowed-hosts=127.0.0.1against one-line mutants oface/scrape.pyandace/tests/conftest.py. This deselects the 10 tests that make live calls without a cassette, plustest_stroke_tableandtest_taylor_and_francis_source, which have no cassette committed. 25 of 26 mutants are killed.'pmc'retmode=refhop[]_without_api_keydoesn't replacelinksets→ no providers)"only"with no PMC falls through"only"with a failed PMCID lookup falls through"only"never looks up a PMCIDTruenever takes PMCraise_for_statuswas dropped fromget_html_by_pmid:PubMedAPI.getalready raises on any non-200, so the call could never fire, and a mutant removing it would be equivalent.Suite
Fresh uv venv, Python 3.12, serial
pytest -p no:cacheprovider. Pins:urllib3<2,vcrpy==6.0.1,pytest-recording==0.13.1,requests==2.31.0,pytest==8.0.1,seleniumbase>=4.20,setuptools<70. Results: master 19b6340 53 passed; this branch 65 passed with the key unset and with it set. With currenturllib32.x andvcrpy8.x, 14 tests fail on master too: gzip cassette bodies aren't decoded.🤖 Generated with Claude Code