Different LLM crawlers parse robots.txt differently. GPTBot honors the most-specific block. ClaudeBot honors the LAST matching block. PerplexityBot ignores wildcards in some patterns. The same robots.txt produces different access maps per bot — and most operators never check.
Read the story behind this tool: When the same robots.txt blocks one LLM and allows another →