Publishers & Media
Track how AI uses and cites your journalism.
Find which articles AI engines quote and link.
Decide which AI crawlers to allow, with data on what each fetches.
Measure visits that arrive from AI answers.
AI assistants read, summarize and cite journalism every day. For publishers, the questions are which articles get cited, which crawlers to let in, and whether AI answers send readers back to the original reporting.
Crawler policy is a business decision. Training crawlers, AI search crawlers and live user fetchers are separate bots that can be allowed or blocked independently, so publishers can stay visible in AI search while setting their own terms for training.
Choose which AI crawlers may read your site, grouped by what they do with your content, and get robots.txt rules to paste. Block only paywalled or archive sections if that suits your model.
13 allowed · 7 blocked
# AI crawler policy, generated with AEO Engine
# AI search index
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
User-agent: Amazonbot
Allow: /
# Live user request
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Meta-ExternalFetcher
User-agent: DuckAssistBot
User-agent: MistralAI-User
Allow: /
# Model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
User-agent: Bytespider
Disallow: /Merge these lines into your existing robots.txt rather than replacing it. Bot names change; check the AI Bot Directory for the current list.
See which AI bots fetch which sections, from your logs.
Choose allow or block per bot purpose, and per section if needed.
Find which articles AI answers cite, and for which prompts.
Count readers who arrive from AI assistants.
Your articles cited in AI answers, by engine, topic and prompt.
Allow or block by purpose and by section, with the robots.txt to match.
What each AI bot fetches, from paywalled sections to archives.
Visits that arrive from ChatGPT, Perplexity, Copilot and Gemini.
Answers that use your reporting without citing you.
How quickly breaking stories appear as sources.
Related optimizations
Know which bots crawl you and serve them well.
Tool: AI Bot Log AnalyzerLIVE
Use AI to run your optimization work faster.
Tool: AI Crawler Access CheckerLIVE
Optimize for language models of every size.
Tool: Fact Density AnalyzerLIVE
Make your content the chunk that gets retrieved.
Tool: Chunk Retrievability TesterLIVE
Yes. Blocking training bots like GPTBot or Google-Extended is separate from allowing search bots like OAI-SearchBot and PerplexityBot.
Citation tracking lists your articles that AI answers cite for tracked prompts, with frequency and engine.
AI referrals are smaller than search traffic today but growing. Tracking them shows which content AI audiences value.
Yes. robots.txt rules can target paths, so you can block a crawler from paywalled or archive sections while leaving the rest open.
Blocking training-only bots like GPTBot, CCBot or Google-Extended doesn't affect Google or Bing search. Blocking search bots like OAI-SearchBot or PerplexityBot does remove you from those AI search engines.
More solutions
Run a free AI-readiness audit of your site in under a minute. No credit card, no sign-up.