fix(preview): serve Open Graph to iMessage, Reddit and Bluesky scrapers (SIT-359) - #42
Merged
Merged
Conversation
…rs (SIT-359) Core's scraper allowlist had drifted behind Cloud's. It held the same 13 user-agents it has since the beginning, so self-hosted deployments still redirect Applebot, redditbot and Bluesky's Cardyb to the destination instead of serving them the preview page — the share card then shows the destination site's metadata rather than the link's. Cloud verified all three against production and fixed its own copy in cloud#217. Cloud registers its hook ahead of these routes, so this list governs self-hosted deployments only. Bluesky is the one worth prioritising: CardyB bakes the fetched card into the post record permanently, so a post that captured a wrong card can never be re-scraped the way a Facebook or LinkedIn share can. Adds Applebot, redditbot, Cardyb, Mastodon, vkShare, the Embedly and Iframely aggregators, and the common preview validators. Every entry is a token only a server-side fetcher sends. Deliberately NOT added: Instagram, Threads, Snapchat, Tumblr. Each name appears in that app's in-app browser user-agent, so matching it would serve a real person the meta-refresh interstitial instead of their redirect. Instagram and Threads previews come from facebookexternalhit, already covered. A test pins all four as non-matches. Lifts the list to a module-level SCRAPER_PATTERNS and exports it with isSocialScraper so both are testable. First test coverage for this gate: 232 tests pass, tsc clean.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gap
isSocialScraper(src/routes/preview.ts) has held the same 13 user-agents since the beginning. Anything unlisted falls through to the redirect handler, so the share card renders the destination site's metadata instead of the link's.Three real scrapers were affected:
Applebot/0.1(iMessage, Notes, Spotlight)redditbot/1.0Bluesky Cardyb/1.1Why Bluesky is the urgent one
CardyB bakes the fetched card into the post record permanently. Unlike Facebook or LinkedIn, there is no force-re-scrape — a post that captured a wrong card keeps it forever. Its user-agent masquerades as Chrome, so
Cardybis the only usable token:What is added
Applebot,redditbot,Cardyb,Mastodon,vkShare,Embedly,Iframely, plus the common validators (opengraph,MetaInspector,PreviewBot,LinkPreview). Every entry is a token only a server-side fetcher sends.What is deliberately NOT added
Instagram,Threads,Snapchat,Tumblr. Each of those strings appears in that app's in-app browser user-agent, so matching the bare name would serve a real person the meta-refresh interstitial instead of their redirect — worse than a missing preview card.Instagram and Threads previews are generated by Meta's
facebookexternalhit, already in the list, so adding them buys nothing and only creates the regression. A test asserts all four stay absent, with the reason attached.WhatsApphas the same defect and predates the rule. Left alone deliberately — narrowing it risks breaking previews that currently work. Documented in a comment.Structural change
scraperPatternsmoves from a function-local array to a module-levelSCRAPER_PATTERNS, exported alongsideisSocialScraperso both are testable. Behaviour and signature are unchanged.Verification
npm run buildclean,tsc --noEmitclean.npx vitest run— 232 tests across 15 files (was 198 across 14). The 34 new cases cover every scraper pattern, every human UA including the four in-app browsers, and the absent-pattern guard.