Pick a stance per known AI crawler — separately for crawling (robots.txt) and training (ai.txt). The tool emits three consistent files — robots.txt, /ai.txt, and /.well-known/ai.txt — expressing the same intent per bot. Covers 22 bots including OpenAI, Anthropic, Google, Perplexity, Meta, Apple, Bytespider, and more.
Starting point. You can customize any bot's individual directive below after applying a preset.
Each row lets you set the bot's crawl directive (robots.txt) and training directive (ai.txt) independently. "allow crawl + disallow training" is the canonical AEO-friendly, training-hostile pattern.
| Bot | Crawl (robots.txt) | Training (ai.txt) | Operator / notes |
|---|
Catalog covers every known AI/LLM crawler as of 2026-04. Classified by operator (OpenAI, Anthropic, Google, Perplexity, Meta, Apple, ByteDance, Amazon, Common Crawl, Cohere, You, DuckDuckGo, Mistral, Diffbot, Timpi). Does not include classic search bots (Googlebot, Bingbot) by default — they're assumed-allowed and regulated by their own non-AI policies.
Source documentation: each bot card links to the operator's official docs. Full policy reasoning in the companion blog post.