Skip to content

[codex] Improve Bing metadata image crawlability - #3

Draft
RayLi-Muye wants to merge 1 commit into
mainfrom
codex/issue-2-bing-seo-metadata
Draft

RayLi-Muye wants to merge 1 commit into
mainfrom
codex/issue-2-bing-seo-metadata

Conversation

@RayLi-Muye

Copy link
Copy Markdown

Refs #2

Scope

  • Route Open Graph, Twitter, Organization JSON-LD, and project JSON-LD image/logo URLs through the site-owned /api/media endpoint so crawlers can fetch metadata assets without an OSS page Referer.
  • Enable /api/media in production with bucket-relative path validation, a stable https://mist-arch.com/ OSS Referer, and cache headers.
  • Update robots.txt to allow /api/media while keeping the broader /api/ path disallowed.
  • Add explicit index, follow robots metadata for localized pages.
  • Add Playwright coverage for metadata image URLs, the metadata image fetch path, JSON-LD logo URL, and robots rules.
  • Allow local Playwright runs to use PLAYWRIGHT_PORT so tests can run without stopping unrelated local services on port 3000.
  • Update SEO, image pipeline, and development-log docs.

Verification

  • npm run typecheck
  • npm run lint
  • npm run build
  • PLAYWRIGHT_PORT=3005 npm run test:e2e — 9 Chromium tests passed
  • git diff --check

Live/diagnostic proof from the audit:

  • Current production HTML already has valid title, description, canonical, Open Graph, Twitter, JSON-LD, and non-empty image alt output.
  • Direct no-Referer requests to current OSS metadata image URLs return 403 Forbidden, while the same OSS object succeeds when fetched with the production site Referer.
  • This PR moves metadata image URLs to a site-owned fetch path; it does not change visible page imagery.

Docs Impact

  • docs/SEO.md documents crawler-safe metadata image URLs and the intentional /api/media robots allowance.
  • docs/IMAGE_PIPELINE.md documents mediaMetadataUrl() as a metadata-only exception to direct OSS/CDN browser imagery.
  • docs/DEVELOPMENT_LOG.md records the change and verification.

Risk / Rollback

  • Main risk: /api/media becomes publicly reachable for safe bucket-relative OSS paths. The route rejects absolute/data/empty/traversal paths and sends cache headers to limit repeated fetch load.
  • Main benefit: Bing and other crawlers can fetch logo/social preview assets without depending on OSS Referer behavior.
  • Rollback: revert this PR to return metadata images to direct OSS URLs and restore /api/media as development-only.

Owner Decision Needs

  • Review/merge decision for this PR.
  • Separate explicit approval is still required for merge, preview/production deployment, Bing Webmaster manual actions, favicon/logo visual changes, or any customer-visible copy/visual direction changes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant