fix(docs): disallow crawling of RSC payload files in robots.txt - #243
Merged
Conversation
RSC payload files under /funstack__/ were being crawled by Google and reported as duplicate content in Search Console. They are internal assets fetched by the client runtime, not pages, so exclude them from crawling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FtR8TfdEtzckdyAuTsAbnP
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Google Search Console reports RSC payload URLs such as
https://router.funstack.work/funstack__/fun%3Arsc-payload/36397a7cea311570.txtas "crawled — marked as duplicate". These.txtfiles are internal assets emitted by@funstack/static(path pattern/funstack__/{rscPayloadDir}/{hash}.txt), not pages, so there is no value in Google crawling them, and they surface as duplicate-content noise in Search Console.Change
Add a
Disallow: /funstack__/rule topackages/docs/public/robots.txt. The directory prefix covers both the olderfun:rsc-payloadpayload paths currently in Google's index reports and the currentfun__rsc-payloaddefault, since all payloads live under/funstack__/. Google applies the longest-matching rule, so thisDisallowtakes precedence over the existingAllow: /for anything under that directory.SEO safety
Blocking a subresource is only risky when Googlebot needs it to render page content. The docs site builds with
ssr: true, so full page content is pre-rendered into the HTML at build time and the RSC payloads are only used for client-side hydration (verified against the live site — the content is present in the raw HTML). Blocking/funstack__/therefore does not affect how Google sees the actual pages.Verification
robots.txtis copied verbatim intodist/public/.@funstack/staticdocs that all RSC payloads are emitted under/funstack__/.🤖 Generated with Claude Code
https://claude.ai/code/session_01FtR8TfdEtzckdyAuTsAbnP
Generated by Claude Code