Repository navigation
SpringerSource: resolve table links against the page URL the page states - #73
Merged
Merged
Conversation
A Nature page without citation_fulltext_html_url left its root-relative table hrefs unjoined, so requests failed with No scheme supplied and the article lost its tables. The page URL is now <base href>, the canonical link, og:url, then citation_fulltext_html_url. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jdkent
marked this pull request as ready for review
October 17, 2026 09:14
Collaborator
Author
|
LGTM |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SpringerSource joined a table page's root-relative href against the
citation_fulltext_html_urlmeta, and used the href as it was when that meta was missing. A Nature page that has no such meta (article s41380-026-03719-9, PMID 42401715) therefore requested/articles/s41380-026-03719-9/tables/1, gotInvalid URL ... No scheme suppliedfive times, and the article lost its tables.The page URL now comes from
Source._page_url:<base href>, then the canonical link, thenog:url, thencitation_fulltext_html_url. SpringerSource joins table hrefs against it withurljoin.Measured on the chain19 extract log: 5 warnings, all for that one URL (the retries of one request, one article). Across the 301,549 saved ACE html files on beast, 9,079 have root-relative
/article(s)/.../tables/Nlinks; 9,078 carrycitation_fulltext_html_url, and this one page is the only one without it.Test:
test_springer_table_links_resolve_against_page_urlruns the real SpringerSource on that saved page (scripts and styles removed) and checks that the first request is the absolute table URL. The table page is a VCR cassette recorded from nature.com (set-cookie headers removed). Withcontent_url = Nonerestored, the test fails. Full suite: 76 passed.Not changed:
_normalize_table_link(DefaultSource) still returns the href unchanged when_get_base_urlfinds nothing. No log line shows that path failing.🤖 Generated with Claude Code