AEOEngine
20 crawlers · 12 operators

The AI Bot Directory

Every major AI crawler: who runs it, what it does with your content, and the exact robots.txt lines to allow or block it.

Check which bots can reach your siteAnalyze your server logs
AI search index

Build the index AI search engines cite from.

Blocking these removes you from AI search results and citations. Most brands should allow them.

Live user request

Fetch a page live because a person asked the assistant to.

Blocking stops assistants from reading your page even when a user shares the link.

Model training

Collect content to train future AI models.

Blocking keeps your content out of future models but doesn't remove you from live AI answers.

GPTBot

OpenAI

Model training

If blocked: Content can't be used to train future OpenAI models.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot

User-agent: GPTBot
Disallow: /
User-agent: GPTBot
Allow: /

OAI-SearchBot

OpenAI

AI search index

If blocked: Pages can't appear as sources in ChatGPT search results.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot

User-agent: OAI-SearchBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /

ChatGPT-User

OpenAI

Live user request

If blocked: ChatGPT can't open your pages when a user asks it to.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

User-agent: ChatGPT-User
Disallow: /
User-agent: ChatGPT-User
Allow: /

ClaudeBot

Anthropic

Model training

If blocked: Content can't be used to train future Claude models.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +claudebot@anthropic.com

User-agent: ClaudeBot
Disallow: /
User-agent: ClaudeBot
Allow: /

Claude-SearchBot

Anthropic

AI search index

If blocked: Pages may be left out of Claude's search results.

User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /

Claude-User

Anthropic

Live user request

If blocked: Claude can't fetch your pages when a user asks it to.

User-agent: Claude-User
Disallow: /
User-agent: Claude-User
Allow: /

PerplexityBot

Perplexity

AI search index

If blocked: Pages can't be indexed or cited in Perplexity answers.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

User-agent: PerplexityBot
Disallow: /
User-agent: PerplexityBot
Allow: /

Perplexity-User

Perplexity

Live user request

If blocked: Perplexity can't open your pages for a user's question.

User-agent: Perplexity-User
Disallow: /
User-agent: Perplexity-User
Allow: /

Google-Extended

Google

Model training

If blocked: Content can't be used for Gemini training and grounding. Doesn't affect Google Search or AI Overviews.

robots.txt token only. Googlebot does the crawling, so this name never appears in your server logs.

User-agent: Google-Extended
Disallow: /
User-agent: Google-Extended
Allow: /

Googlebot

Google

AI search index

If blocked: Pages drop out of Google Search, including AI Overviews and AI Mode.

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

User-agent: Googlebot
Disallow: /
User-agent: Googlebot
Allow: /

Bingbot

Microsoft

AI search index

If blocked: Pages drop out of Bing, which also feeds Copilot and other AI products.

Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

User-agent: Bingbot
Disallow: /
User-agent: Bingbot
Allow: /

Applebot

Apple

AI search index

If blocked: Pages may be left out of Siri and Spotlight suggestions.

User-agent: Applebot
Disallow: /
User-agent: Applebot
Allow: /

Applebot-Extended

Apple

Model training

If blocked: Content can't be used to train Apple's AI models.

robots.txt token only. Applebot does the crawling, so this name never appears in your server logs.

User-agent: Applebot-Extended
Disallow: /
User-agent: Applebot-Extended
Allow: /

Meta-ExternalAgent

Meta

Model training

If blocked: Content can't be used to train Meta's AI models.

User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Meta-ExternalAgent
Allow: /

Meta-ExternalFetcher

Meta

Live user request

If blocked: Meta AI can't fetch your pages for a user's request.

User-agent: Meta-ExternalFetcher
Disallow: /
User-agent: Meta-ExternalFetcher
Allow: /

Amazonbot

Amazon

AI search index

If blocked: Content may be left out of Alexa and Amazon's AI answers.

User-agent: Amazonbot
Disallow: /
User-agent: Amazonbot
Allow: /

DuckAssistBot

DuckDuckGo

Live user request

If blocked: Pages can't be used for DuckDuckGo's AI-assisted answers.

User-agent: DuckAssistBot
Disallow: /
User-agent: DuckAssistBot
Allow: /

MistralAI-User

Mistral

Live user request

If blocked: Le Chat can't fetch your pages for a user's request.

User-agent: MistralAI-User
Disallow: /
User-agent: MistralAI-User
Allow: /

CCBot

Common Crawl

Model training

If blocked: Content is left out of Common Crawl, an open dataset many AI models train on.

User-agent: CCBot
Disallow: /
User-agent: CCBot
Allow: /

Bytespider

ByteDance

Model training

If blocked: Content can't be used to train ByteDance's AI models.

User-agent: Bytespider
Disallow: /
User-agent: Bytespider
Allow: /
FAQ

AI crawler questions

Want to see these bots on your own site? The AI Bot Log Analyzer reads your access log in the browser.

Should I block AI crawlers?+

Decide by purpose. Blocking training bots keeps content out of future models. Blocking search and user bots removes you from AI answers, which usually costs visibility. Many brands allow search and user bots and make a separate call on training bots.

Does blocking Google-Extended remove me from AI Overviews?+

No. Google-Extended only controls whether content is used for Gemini training and grounding. AI Overviews and AI Mode are part of Google Search and follow Googlebot rules.

Can a bot fake its user agent?+

Yes. Anyone can send a user-agent string like GPTBot. Major operators publish IP ranges or support reverse DNS checks so you can verify real crawler traffic.

Do AI crawlers obey robots.txt?+

The major operators listed here say their crawlers respect robots.txt. Some user-triggered fetchers may behave differently, so check each operator's documentation and your server logs.

Be the brand AI recommends, and the one customers choose.

Run a free AI-readiness audit of your site in under a minute. No credit card, no sign-up.

Run a free site auditSee pricing