01Know the three kinds of AI crawler
Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) collect content for future models. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the index AI search answers from. User fetchers (ChatGPT-User, Claude-User, Perplexity-User) load a page live because someone asked about it.
Blocking a training crawler keeps content out of future models but doesn't remove you from AI answers. Blocking search crawlers or user fetchers does.
02Read your robots.txt
Open yoursite.com/robots.txt. Look for groups that name AI bots, and for a catch-all group (User-agent: *) that disallows everything, which also applies to every AI bot not named elsewhere.
A bot follows the most specific group that matches its name. If GPTBot has its own group, the * group no longer applies to it.
User-agent: GPTBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml03Check page-level directives
A page can opt out even when robots.txt allows it: a meta robots tag with noindex, or an X-Robots-Tag response header. Some sites also add noai or noimageai, which some AI tools honor.
Check templates as well as single pages. A noindex left over from a staging site is one of the most common causes of missing visibility.
04Test the live server response
robots.txt is a request, not a lock. Firewalls, CDNs and bot-protection services can block AI crawlers at the network level, often from a single setting such as a CDN's “block AI bots” switch. The crawler then gets a 403 error page however your robots.txt reads.
The only reliable test is to request the page with each crawler's real user agent and look at the status code. The checker below does exactly that.
AI Crawler Access Checker
Enter any URL to see whether each AI crawler is allowed by robots.txt and page directives, and what your server actually returns to it.
05Make sure the content is in the HTML
Most AI crawlers don't run JavaScript. If your page is rendered entirely in the browser, they see an empty shell. Use server-side rendering or static generation for any page you want cited.
06Write your policy and confirm it in your logs
Decide by purpose: most brands allow search crawlers and user fetchers, and make a separate call on training. Write the rules, deploy them, then check your access logs a week later to confirm each bot behaves as expected.
Checklist
- robots.txt doesn't block AI search crawlers or user fetchers you want
- No stray noindex or X-Robots-Tag on important templates
- CDN and firewall let verified AI crawlers through
- Key content renders without JavaScript
- Training-crawler policy decided and documented
- Logs checked a week after changes
Questions
Will blocking GPTBot remove me from ChatGPT?+
No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, and live page fetches use ChatGPT-User. Each is controlled separately.
Do I need to name every AI bot in robots.txt?+
No. Bots that aren't named follow your User-agent: * group. Name a bot only when you want different rules for it.