← jwatte.com

Robots LLM Drift Simulator

Different LLM crawlers parse robots.txt differently. GPTBot honors the most-specific block. ClaudeBot honors the LAST matching block. PerplexityBot ignores wildcards in some patterns. The same robots.txt produces different access maps per bot — and most operators never check.

Context and background

Read the story behind this tool: When the same robots.txt blocks one LLM and allows another →

Input