Pick a stance per known AI crawler — separately for crawling (robots.txt) and training (ai.txt). The tool emits three consistent files — robots.txt, /ai.txt, and /.well-known/ai.txt — expressing the same intent per bot. Covers 22 bots including OpenAI, Anthropic, Google, Perplexity, Meta, Apple, Bytespider, and more.
Read the story behind this tool: Your AI Bot Policy Lives In Three Files, And They Probably Disagree →
Starting point. You can customize any bot's individual directive below after applying a preset.
Each row lets you set the bot's crawl directive (robots.txt) and training directive (ai.txt) independently. "allow crawl + disallow training" is the canonical AEO-friendly, training-hostile pattern.
| Bot | Crawl (robots.txt) | Training (ai.txt) | Operator / notes |
|---|
Catalog covers every known AI/LLM crawler as of 2026-04. Classified by operator (OpenAI, Anthropic, Google, Perplexity, Meta, Apple, ByteDance, Amazon, Common Crawl, Cohere, You, DuckDuckGo, Mistral, Diffbot, Timpi). Does not include classic search bots (Googlebot, Bingbot) by default — they're assumed-allowed and regulated by their own non-AI policies.
Source documentation: each bot card links to the operator's official docs. Full policy reasoning in the companion blog post.