Repository navigation
HighWireSource: parse the page's tables when it has no citation_public_url - #72
Merged
Merged
Conversation
Current science.org pages are identified as HighWire but carry no citation_public_url meta tag. HighWireSource.parse_article indexed that tag unconditionally, so parsing raised TypeError and the article got no extraction. When the tag is missing, the tables printed in the page are parsed with the same helper DefaultSource uses for inline tables. The T<n>.expansion download path is unchanged when the tag is present. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jdkent
marked this pull request as ready for review
October 15, 2026 12:50
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Current science.org pages are identified as HighWire, but they carry no
citation_public_urlmeta tag.HighWireSource.parse_articlesubscripted the missing tag (sources.py:1417) and raisedTypeError: 'NoneType' object is not subscriptable, so the article got no extraction. Seen in ns-pond's production extract on 2026-10-15 for PMIDs 19407193 and 17170308 (canonicalhttps://www.science.org/doi/10.1126/science.1167768).Change: when the tag is missing, parse the tables printed in the page with the existing
_append_leaf_candidate_tableshelper, and return the article. With no tables,missing_sourceis set the same wayDefaultSourcesets it. When the tag is present, the oldT<n>.expansion.htmlpath is unchanged. Those pages are behind Cloudflare now, which is also what #70 noted.Tests:
test_highwire_page_without_public_url_parses_inline_tables, with a minimal saved fixture (a HighWire page, no public URL, one inline table). It fails on master with the originalTypeError.--block-network --allowed-hosts=127.0.0.1, same pinned venv as Open the prlinks target from the keyed elink lookup instead of the unkeyed eutils hop #68/Key publisher rewrites and page actions on the page URL, not the journal folder name #71): master e50b384 65 passed / 10 failed, this branch 66 passed / 10 failed. The same 10 tests fail on both; they make live calls and have no cassette.🤖 Generated with Claude Code