What is BSO?
Bot search optimization manages how search and AI crawlers access your site: which bots visit, how often, what they fetch, and what you allow or block.
Server logs are the ground truth. They show real crawler behavior, including bots wasting visits on broken URLs, important pages that never get crawled, and AI bots you didn't know were visiting.
Why BSO matters
Crawling is the first step toward every search result and AI citation. If the right bots can't fetch your important pages, or waste their visits on errors, nothing downstream works.
How BSO works
- 01
Request
A crawler fetches a URL and identifies itself with a user-agent string.
- 02
Policy
robots.txt, meta robots tags and server rules decide what it may access.
- 03
Response
Your server returns the content, a redirect or an error.
- 04
Log
Each request is recorded with its time, path, status and user agent.
BSO best practices
- Review server or CDN logs monthly for search and AI bot activity.
- Fix URLs that return 4xx or 5xx errors to bots.
- Make sure important pages are linked and in the sitemap, so they get crawled.
- Set your AI crawler policy by bot purpose: training, search or user requests.
- Verify important bots by IP address or reverse DNS, since user agents can be faked.
Common BSO mistakes
- Blocking bots by accident through firewall or CDN rules.
- Wasting crawl budget on endless parameter URLs.
- Assuming every request with a bot's user agent is genuine.
How to measure BSO
Metrics to track
- Hits per bot over time
- Share of bot requests that return errors
- Crawl frequency of key pages
- New AI crawlers appearing in logs
What it improves
- Crawler visit frequency
- Pages fetched per bot
- Blocked vs. allowed bots
- Crawl errors
BSO vs. related optimizations
These overlap with BSO. Work on one usually helps the others.
BSO FAQ
Where do I find server logs?+
On your web server (Apache or Nginx access logs), from your hosting provider, or from your CDN. Cloudflare, Fastly, Vercel and others offer log exports.
How can I tell if a bot is real?+
Check the requesting IP against the operator's published IP ranges, or do a reverse DNS lookup. Major operators such as Google, Microsoft and OpenAI document how to verify their crawlers.
What is crawl budget?+
The number of URLs a search engine will crawl on your site in a given period. It matters most for large sites, where wasted crawls on duplicates or errors delay important pages.
Should I block AI training bots?+
That's a content policy decision. Blocking training bots doesn't remove you from AI search, as long as search and user-request bots are allowed.