Skip to content

HighWireSource: parse the page's tables when it has no citation_public_url - #72

Merged
jdkent merged 1 commit into
masterfrom
ace-highwire
Oct 26, 2026
Merged

jdkent merged 1 commit into
masterfrom
ace-highwire

Conversation

@jdkent

@jdkent jdkent commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator

Current science.org pages are identified as HighWire, but they carry no citation_public_url meta tag. HighWireSource.parse_article subscripted the missing tag (sources.py:1417) and raised TypeError: 'NoneType' object is not subscriptable, so the article got no extraction. Seen in ns-pond's production extract on 2026-10-15 for PMIDs 19407193 and 17170308 (canonical https://www.science.org/doi/10.1126/science.1167768).

Change: when the tag is missing, parse the tables printed in the page with the existing _append_leaf_candidate_tables helper, and return the article. With no tables, missing_source is set the same way DefaultSource sets it. When the tag is present, the old T<n>.expansion.html path is unchanged. Those pages are behind Cloudflare now, which is also what #70 noted.

Tests:

🤖 Generated with Claude Code

Current science.org pages are identified as HighWire but carry no
citation_public_url meta tag. HighWireSource.parse_article indexed that
tag unconditionally, so parsing raised TypeError and the article got no
extraction.

When the tag is missing, the tables printed in the page are parsed with
the same helper DefaultSource uses for inline tables. The T<n>.expansion
download path is unchanged when the tag is present.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jdkent
jdkent marked this pull request as ready for review October 15, 2026 12:50
@jdkent jdkent self-assigned this Oct 15, 2026

@jdkent jdkent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jdkent
jdkent merged commit aec9898 into master Oct 26, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant