AI Crawler Database
22 user agents from 11 operators, each taken from that operator's own documentation rather than from another crawler list. Sorted by what actually matters: whether it trains a model, or builds the index your citations come from.
- Search
- 8
- Training
- 6
- User-triggered
- 6
- Usage control
- 2
- Operators
- 11
Every source link checked August 2026
The distinction that decides everything
Most crawler lists give you a table of names. The useful question is what each one does, because blocking a training crawler and blocking a search crawler have opposite consequences.
- Search
- Builds the retrieval index that AI answers are assembled from. Blocking one of these removes you from answers now.
- Training
- Collects material for future model versions. Blocking costs you nothing this quarter and is a licensing decision, not an SEO one.
- User-triggered
- Fetches a page because a person asked. Most operators state robots.txt may not apply, so a line here records intent rather than enforcing anything.
- Usage control
- Not a crawler at all. A token that governs how data collected by another agent may be used. It fetches nothing.
Surfaces websites in ChatGPT's search features.
If you block it: You stop appearing in ChatGPT search results. This is the crawler most robots.txt files forget while carefully allowing GPTBot.
Crawls content that may be used to train OpenAI's foundation models.
If you block it: Your content is not used for future model training. No effect on whether ChatGPT can find or cite you today.
Handles user-initiated actions in ChatGPT and custom GPTs.
If you block it: Little, in practice. OpenAI states robots.txt rules may not apply to user-initiated visits. Enforce at the server if you genuinely need to.
Operator states robots.txt may not applyOfficial docsValidates the safety of pages submitted as ads in ChatGPT.
If you block it: Pages cannot be validated for advertising in ChatGPT.
Operator documents that it honours robots.txtOfficial docsNavigates the web to improve search result quality for Claude users.
If you block it: You are less likely to be surfaced in Claude's search results.
Operator documents that it honours robots.txtOfficial docsFetches pages when a person asks Claude a question.
If you block it: Claude cannot retrieve your page to answer a user's question about it.
Operator documents that it honours robots.txtOfficial docsCollects web content for AI model training and development.
If you block it: Your content is not used to train future Claude models. No effect on citations today.
Operator documents that it honours robots.txtOfficial docsBuilds the Search index, which is also what AI Overviews and AI Mode draw on.
If you block it: You leave Google Search entirely, AI Overviews included. This is the crawler behind AI Overviews, not Google-Extended.
Operator documents that it honours robots.txtOfficial docsControls whether content Google has already crawled may train future Gemini models. It does not crawl.
If you block it: Gemini is not trained on your content. Google states plainly that it does not affect inclusion in Search and is not a ranking signal, so this does not remove you from AI Overviews.
Operator documents that it honours robots.txtOfficial docsGeneric crawler used by various Google product teams for research and development.
If you block it: No effect on Search. Affects internal Google fetching only.
Operator documents that it honours robots.txtOfficial docsFetches sites on a Vertex AI customer's request when they build their own agents.
If you block it: Vertex AI customers cannot ingest your pages into their own agents.
Operator documents that it honours robots.txtOfficial docsIndexes pages so they can be surfaced and linked in Perplexity results. Not a training crawler.
If you block it: You are removed from Perplexity's index and stop being cited there.
Visits a page because a person asked Perplexity about it.
If you block it: Generally nothing. Perplexity states this agent ignores robots.txt because a human initiated the request.
Crawls the web to train Meta's AI models and improve its products.
If you block it: Meta's models are not trained on your content.
Operator documents that it honours robots.txtOfficial docsFetches individual links at a user's request, including agentic AI navigating a site on someone's behalf.
If you block it: Little. Meta states this crawler may bypass robots.txt because the fetch was user-requested.
Operator states robots.txt may not applyOfficial docsPowers Spotlight, Siri and Safari search. The agent that actually fetches your pages.
If you block it: You disappear from Siri, Spotlight and Safari suggestions.
Operator documents that it honours robots.txtOfficial docsDoes not crawl. It determines whether data already collected by Applebot may train Apple's foundation models.
If you block it: Apple Intelligence is not trained on your content, and you stay fully discoverable in Spotlight, Siri and Safari. Blocking it does not reduce your visibility.
Operator documents that it honours robots.txtOfficial docsCrawls to improve Alexa's answers and support Amazon's AI services. Honours allow and disallow, but not crawl-delay.
If you block it: Alexa and Amazon's assistant features stop drawing on your content.
Operator documents that it honours robots.txtOfficial docsRetrieves pages in real time when someone asks Le Chat a question. Not used for training.
If you block it: Le Chat cannot cite your pages when answering.
Operator documents that it honours robots.txtOfficial docsCrawls in real time for DuckDuckGo's AI-assisted answers, which cite their sources. Not used for training.
If you block it: You are not cited in DuckDuckGo's AI answers.
Operator documents that it honours robots.txtOfficial docsBuilds the open Common Crawl corpus, which many organisations use as training data.
If you block it: You leave a widely reused public dataset. Note that copies already published cannot be recalled.
Operator documents that it honours robots.txtOfficial docsCollects data associated with ByteDance's models.
If you block it: Intended to stop collection, though compliance has been reported as inconsistent. Enforce at the server if it matters to you.
Operator does not document robots.txt behaviourOfficial docs
Your robots.txt
Toggle any crawler above and this updates. It starts with everything that drives citations allowed, and training left for you to decide deliberately rather than by default.
# AI crawler policy
# Generated from sharaki.me/tools/ai-crawlers
# Every token below is taken from the operator's own documentation.
User-agent: *
Allow: /
# Search
User-agent: OAI-SearchBot
Allow: /
User-agent: OAI-AdsBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Googlebot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Applebot
Allow: /
User-agent: Amazonbot
Allow: /
User-agent: DuckAssistBot
Allow: /
# Training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: GoogleOther
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
# User-triggered
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Google-CloudVertexBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: meta-externalfetcher
Allow: /
User-agent: MistralAI-User
Allow: /
# Usage control
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
Sitemap: https://example.com/sitemap.xmlReplace the sitemap line with your own. Remember that for any agent marked user-triggered, this file records your intent rather than enforcing it: those operators state robots.txt may not apply, so a real block belongs at the server.