AI crawlers aren't one thing
Most AI companies run several bots with different jobs. OpenAI, for example, runs GPTBot for training, OAI-SearchBot for ChatGPT search and ChatGPT-User for pages a user asks ChatGPT to open. Anthropic and Perplexity follow similar patterns.
Because each bot has its own user-agent name, robots.txt can treat them differently. That's what makes a middle-ground policy possible.
Three questions to decide your policy
- Do you want to appear in AI search answers? If yes, allow the search crawlers and user fetchers.
- Is your content your product? Publishers, course creators and data businesses often block training crawlers; most other brands benefit from being well represented in models.
- Are some sections different? You can block paywalled, premium or archive paths while leaving the rest open.
The most common policy
Many brands allow AI search crawlers and user fetchers, and decide training separately. That keeps them citable in ChatGPT search, Perplexity, Claude and Copilot while setting their own terms for model training.
Note that Google-Extended only controls Gemini training and grounding. Google's AI Overviews and AI Mode are part of Google Search and follow Googlebot rules, so blocking Google-Extended doesn't remove you from them.
Check that the policy actually works
robots.txt is only one layer. CDN bot-protection settings and firewalls can block AI crawlers regardless of what robots.txt says. After any change, test the live response for each bot and confirm in your access logs.