perplexitybot

See which AI crawlers can read your site — GPTBot, ClaudeBot, PerplexityBot & 20 more. Curated, operator-sourced bot list + a zero-dep CLI and GitHub Action to audit robots.txt, test reachability, read access logs, and gate it all in CI.

Validates whether GPTBot, ClaudeBot, PerplexityBot and other LLM crawlers are allowed by a robots.txt file.

Complete database of AI search engine crawler user-agents (GPTBot, PerplexityBot, ClaudeBot) with robots.txt configuration examples

Agent-native multi-bot web crawler simulator. See your site through the eyes of Googlebot, GPTBot, ClaudeBot, and PerplexityBot.

The best AEO/GEO skill for Claude Code. Audit, fix, and monitor your website's visibility across ChatGPT, Claude, Perplexity, Gemini, and Google AI Overviews. 100+ optimization techniques, 33 evidence collectors, 4-vector GEO Score. Built on Princeton KDD 2024 paper. Free, MIT.

Visual robots.txt manager for the AI era. Block or allow GPTBot, ClaudeBot, PerplexityBot, and 20+ AI crawlers with one click.

User-agent simulator for major LLM crawlers: measures response code, content length, and JS-dependency

Free no-tracking browser utility for drafting and auditing AI crawler robots.txt rules.

Record which AI crawlers read your pages. Forwards GPTBot, ClaudeBot, PerplexityBot and friends to rankveo, and nothing else.

SEO and GEO agent skills for WordPress. Server log forensics for AI crawlers, passage level citability scoring, a 90 day GEO plan, and portfolio reporting for agencies. Every analysis ends in a self-contained HTML report.

Check whether AI crawlers can access a URL (robots.txt) plus page legibility signals. Zero dependency CLI and library.

21 AI crawler user-agents we detect, and which 14 of them actually reached one real site over 74 days. First-party detection list plus first-party access logs. Not observed is written "not observed", never 0.

AI Observatory — verified AI and search crawler retrieval monitoring for business websites

Audit robots.txt for sitemap hygiene, crawl mistakes, and explicit AI crawler policy.

Four free MCP tools to check whether AI engines can find, read and recognise a website. No account, no API key, no LLM calls.

Machine-readable dataset of AI crawler user-agents — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, and more. Includes robots.txt recipes and weekly verification.

Open core of the CrawlCheck AI-crawler visibility scanner: robots.txt policy per crawler agent, user-agent group shadowing detection, and log-line forgery verification against operators' published IP feeds. Dependency-free, parity-tested against production.