Skip to content

Open the prlinks target from the keyed elink lookup instead of the unkeyed eutils hop - #68

Merged
jdkent merged 3 commits into
masterfrom
ace-prlinks-key
Oct 14, 2026
Merged

jdkent merged 3 commits into
masterfrom
ace-prlinks-key

Conversation

@jdkent

@jdkent jdkent commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

get_html_by_pmid made a keyed JSON elink call (the PubMedAPI that was given api_key). For articles that don't come from PMC, it then sent the browser to http://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi?dbfrom=pubmed&id=<pmid>&cmd=prlinks&retmode=ref. That second request goes to E-utilities without the key, so NCBI applies its per-IP limit of 3 requests a second. With 3 ACE workers on one host, nearly half of those hops came back as

{"error":"API rate limit exceeded","api-key":"128.62.223.147","count":"4","limit":"3"}

and that was saved as the page (e.g. PMID 20979788). In the ingestion run on 2026-10-13, 39 of 63 pages set aside in 4 minutes were this error. The previous night's run had 2,664.

Fix

The keyed JSON call already sends cmd=prlinks (PubMedAPI.elink with the default access_db), so it asks the same question as the browser hop. get_html_by_pmid now opens the URL that the retmode=ref redirect would have landed on, and never sends the browser through E-utilities:

  • the first LinkOut provider that isn't PMC;
  • the PMC URL, when PMC is the only provider;
  • https://pubmed.ncbi.nlm.nih.gov/<pmid>/ when there is no LinkOut record. That is where the redirect lands today (it redirects to /<pmid>, which then 301s to /<pmid>/), so the PubMed-landing-page handling in check_for_substitute_url (PLoS citation_doi) still applies unchanged.

The PMC path (prefer_pmc_source, including "only") is unchanged. The elink JSON is parsed once into (provider, URL) pairs. A response without linksets[0].idurllist[0].objurls now counts as a failed lookup, instead of being read as "no LinkOut" and sent to the PubMed page. The skip_url_substrings pre-check over all provider URLs now runs on every path. Before, it was skipped when prefer_pmc_source=False without skip substrings (nothing to check), and when PMC was listed but the PMCID lookup failed. In that case get_html still checked the URL after the redirect.

Behaviour changes worth knowing:

  • If the keyed elink call fails, get_html_by_pmid logs the error and returns None. Before, it fell back to the unkeyed hop, which is the request that was rate limited. retrieve_articles records the article as invalid, as it does for any other failed scrape.
  • The keyed lookup now also runs when prefer_pmc_source=False and there are no skip substrings. This replaces the unkeyed request, so there is one NCBI request per article either way, and it is now keyed.
  • The retmode argument of get_html_by_pmid is removed. It only set the retmode of the browser hop, and nothing in ACE passes it.
  • logger.info(query) logged the eutils URL. It now logs the provider URL. Errors from the keyed elink and PMCID calls are logged with the API key replaced by <api_key>, because requests exceptions quote the full query string.

Provider URL equivalence

I checked the JSON objurls against the Location of cmd=prlinks&retmode=ref (keyed, redirects not followed) for 96 distinct PMIDs from NeuroImage, HBM, J Neurosci, Cereb Cortex, Brain Res, Front Hum Neurosci, PLoS ONE, eLife, PNAS, Sci Rep, Brain, Cortex, J Physiol 1960, Yale J Biol Med 1980, Cochrane and others. The rule above matched the redirect target in all 96:

JSON objurls PMIDs example
one non-PMC provider 31 29768909 → linkinghub.elsevier.com/...; 20979788 → www.yiigle.com/LinkIn.do?...
PMC listed first, then publisher 23 25709573 → doi.org/10.3389/fnhum.2014.01073, not PMC
publisher listed first, then PMC 30 21159981 → www.jneurosci.org/cgi/pmidlookup?...
PMC only 9 20311739 → pmc.ncbi.nlm.nih.gov/articles/pmid/20311739/
none 3 28520367 → pubmed.ncbi.nlm.nih.gov/28520367

The JSON list puts PMC wherever its name sorts, so taking the first URL would be wrong for the 23 PMC-first cases; the redirect skips PMC whenever another provider exists. No PMID in the sample had two non-PMC providers. For that case the code takes the first one in the JSON order.

Tests

The cassette tests run the real get_html_by_pmid and get_html in requests mode. The URL choice is shared with browser mode, which can't be recorded. The assertions check the interactions the cassette actually served (play_counts / all_played), not cassette.requests: that is the stored file, so asserting on it passes even when the code makes no request. As a result, a test fails on the run that records its cassette and passes once it plays back.

test PMID checks
..._opens_the_provider_url_from_the_keyed_lookup 25709573, prefer_pmc_source=False keyed JSON elink first, then doi.org/10.3389/fnhum.2014.01073 (PMC is listed first), and no other eutils request
..._without_linkout_opens_the_pubmed_page 28520367 served: the elink, then pubmed.ncbi.nlm.nih.gov/28520367/. In requests mode PubMed returns its proof-of-work challenge page (HTTP 203), so only the URL is asserted.
..._prefers_pmc 25709573, True prlinks elink, pubmed_pmc elink, then pmc.ncbi.nlm.nih.gov/articles/PMC4325919; no publisher request
..._opens_pmc_when_it_is_the_only_provider 20311739, False pmc.ncbi.nlm.nih.gov/articles/pmid/20311739/
..._pmc_only_skips_articles_without_pmc 28520367, "only" returns None; only the elink is served
..._pmc_only_opens_pmc 25709573, "only" replays the prefers-PMC recording and opens PMC4325919
..._pmc_only_skips_when_the_pmcid_lookup_fails 25709573, "only" replays the prlinks reply with the recorded PMCID-lookup 400s; returns None; the log has both "Failed to resolve PMCID" and "PMC source detected, but PMCID lookup failed"
..._treats_an_elink_reply_without_linksets_as_a_failed_lookup '' NCBI's real {"linksets": []} reply (HTTP 200) returns None and opens nothing
..._skips_a_blocked_provider_it_would_not_open 21159981, skip pmc.ncbi.nlm.nih.gov returns None; HighWire is the target and PMC is listed second; the skip flag is set; only the elink is served
..._returns_none_when_the_keyed_lookup_fails 25709573, key not-a-real-key records NCBI's 400 "API key invalid" (6 responses: the request plus PubMedAPI's 5 retries); returns None; nothing but those elinks is served; caplog has <api_key> and not the key
test_pmcid_lookup_failure_is_logged_without_the_api_key 25709573, same key the same 400 at the _get_pmcid_from_pmid log site
test_recorded_cassettes_do_not_store_the_api_key - a local server stands in for NCBI and echoes the key in a gzip body and a header; the cassette recorded through vcr_config has <api_key> in both places and no key
  • The two invalid-key tests set the client's retry backoff_factor to 0. The retries stay, but the 30 s of sleeps go.
  • No PMID with a PMC LinkOut lacks a pubmed_pmc link (0 of 62), so the "only"/failed-PMCID test pairs two real recordings: the prlinks reply for 25709573 and the invalid-key 400s for its PMCID lookup.
  • ace/tests/conftest.py filters api_key from the query. That filter also applies to matching, so cassettes play back with or without PUBMED_API_KEY. NCBI also echoes the key in its error bodies ("api-key":"<key>"), so bodies are stored decoded (decode_compressed_response), and a before_record_response hook replaces the key from PUBMED_API_KEY and the tests' fake key with <api_key> in bodies and headers. The two 400 cassettes were re-recorded through the hook. I grepped every cassette for the real key, not-a-real-key and api_key=: no matches.

Mutations

I ran the 53 offline tests with --block-network --allowed-hosts=127.0.0.1 against one-line mutants of ace/scrape.py and ace/tests/conftest.py. This deselects the 10 tests that make live calls without a cassette, plus test_stroke_table and test_taylor_and_francis_source, which have no cassette committed. 25 of 26 mutants are killed.

id mutant result killed by
M1 drop the non-PMC loop killed provider_url
M20 pick PMC when it is listed killed provider_url
M17 compare 'pmc' killed provider_url
M2 no LinkOut → unkeyed retmode=ref hop killed without_linkout
M3 always open the unkeyed hop killed 3 tests
M18 provider parser returns [] killed 6 tests
M19 skip the keyed elink killed 10 tests
M10 _without_api_key doesn't replace killed both invalid-key tests
M4 elink failure → unkeyed hop killed returns_none, without_linksets
M4b elink failure → carry on with no providers (PubMed page) killed returns_none, without_linksets
M5 lenient parse (missing linksets → no providers) killed without_linksets
M6 drop the skip pre-check killed skips_a_blocked_provider
M7 pre-check only the first provider killed skips_a_blocked_provider
M23 pre-check only the chosen target killed skips_a_blocked_provider
M8 elink-failure log unredacted killed returns_none
M9 PMCID-failure log unredacted killed pmcid_lookup_failure
M11 "only" with no PMC falls through killed pmc_only_skips_articles_without_pmc
M12 "only" with a failed PMCID lookup falls through killed pmc_only_skips_when_the_pmcid_lookup_fails
M21 "only" never looks up a PMCID killed pmc_only_opens_pmc, pmc_only_skips_when_the_pmcid_lookup_fails
M22 open the LinkOut PMC URL, not the resolved PMCID killed prefers_pmc, pmc_only_opens_pmc
M13 True never takes PMC killed 3 tests
M14 flip the PMC condition killed 3 tests
M16 PMC only → PubMed page killed opens_pmc_when_it_is_the_only_provider
M24 conftest hook doesn't scrub bodies killed do_not_store_the_api_key
M25 conftest hook doesn't scrub headers killed do_not_store_the_api_key
M26 bodies stored compressed killed do_not_store_the_api_key
M15 two non-PMC providers → take the last survived none of the ~100 PMIDs sampled here, or the 700 the reviewer sampled, had two non-PMC providers, so a test would encode a guess

raise_for_status was dropped from get_html_by_pmid: PubMedAPI.get already raises on any non-200, so the call could never fire, and a mutant removing it would be equivalent.

Suite

Fresh uv venv, Python 3.12, serial pytest -p no:cacheprovider. Pins: urllib3<2, vcrpy==6.0.1, pytest-recording==0.13.1, requests==2.31.0, pytest==8.0.1, seleniumbase>=4.20, setuptools<70. Results: master 19b6340 53 passed; this branch 65 passed with the key unset and with it set. With current urllib3 2.x and vcrpy 8.x, 14 tests fail on master too: gzip cassette bodies aren't decoded.

🤖 Generated with Claude Code

jdkent and others added 3 commits October 13, 2026 17:52
get_html_by_pmid sent the browser to elink.fcgi?cmd=prlinks&retmode=ref,
a request to E-utilities without the API key, so NCBI rate limited it per
IP and the saved page was often {"error":"API rate limit exceeded"}. The
keyed JSON elink call already asks for cmd=prlinks; open the URL the
redirect would have gone to instead: the first LinkOut provider that isn't
PMC, PMC when it is the only one, and the PubMed abstract page when there
are none. Errors from the keyed calls are logged with the key redacted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The URL tests read cassette.requests, which is the stored cassette, so
they passed without the code making those requests. Assert on the
interactions the cassette actually served instead, and add recorded tests
for PMC preference, PMC as the only provider, both "only" exits, the
skip_url_substrings pre-check, and an invalid-key 400 from the keyed
elink (returns None, no unkeyed fallback, key redacted at the real log
sites). The helper-only redaction test is replaced by those.

Parse the elink JSON into (provider, URL) pairs once. Drop the
raise_for_status call: PubMedAPI.get already raises on any non-200.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- "only" on 25709573 replays the prefers-PMC recording and must open
  PMC4325919; before, an "only" run that never looked up a PMCID passed.
- The "only" + failed-PMCID test now replays the real recorded 400s for
  the PMCID lookup and checks its log lines, instead of relying on vcr
  refusing an unrecorded request.
- Record the real {"linksets": []} reply (empty id) and check it counts
  as a failed lookup.
- NCBI echoes the key in error bodies, which filter_query_parameters
  doesn't reach: store bodies decoded and replace the key (env and the
  tests' fake key) in bodies and headers. A local server echoing the key
  checks the scrub on every run. The two 400 cassettes are re-recorded
  through it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jdkent
jdkent marked this pull request as ready for review October 14, 2026 00:28
@jdkent jdkent self-assigned this Oct 14, 2026

@jdkent jdkent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant