The 16 AI crawlers that matter in 2026
The single most expensive mistake in AI visibility is treating "AI bots" as one category. Every major vendor runs separate agents for model training and for answering a user's question right now. GPTBot is not OAI-SearchBot. ClaudeBot is not Claude-SearchBot. Sites that blanket-blocked AI in 2023 to opt out of training routinely switched themselves off inside ChatGPT Search and Claude without ever intending to.
Search & answer agents — blocking these costs you citations today
| User agent | Operator | Purpose | What blocking it costs |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Search / answers | Builds the index behind ChatGPT Search. Blocking it removes you from ChatGPT's search answers entirely. |
| ChatGPT-User | OpenAI | Search / answers | Fetches your page when a ChatGPT user's question needs it. OpenAI notes robots.txt may not apply to these user-initiated fetches. |
| Claude-SearchBot | Anthropic | Search / answers | Builds Claude's search index. Blocking it removes you from Claude's search results. |
| Claude-User | Anthropic | Search / answers | Fetches your page when a Claude user asks a question that needs it. Anthropic states this agent does honour robots.txt. |
| PerplexityBot | Perplexity | Search / answers | Indexes content for Perplexity answers and citations. Blocking it removes you from one of the highest-referral AI search engines. |
| Perplexity-User | Perplexity | Search / answers | Live fetch triggered by a user's question. Perplexity states robots.txt generally does not apply to user-initiated requests. |
| GoogleOther | Search + training | General-purpose Google fetcher used for research and development crawls outside core Search indexing. | |
| Applebot | Apple | Search + training | Powers Siri, Spotlight and Apple Intelligence answers. Blocking it removes you from Apple's assistant surfaces. |
| meta-externalagent | Meta | Search + training | Crawls for Meta AI training and product features across Facebook, Instagram and WhatsApp assistants. |
| bingbot | Microsoft | Search + training | Bing's index is the retrieval layer behind Copilot and, historically, parts of ChatGPT Search. Blocking it has outsized AI cost. |
Training-only agents — blocking these is a licensing decision
| User agent | Operator | Purpose | What blocking it costs |
|---|---|---|---|
| GPTBot | OpenAI | Model training | Collects content for training foundation models. Blocking it is a licensing choice and does not remove you from ChatGPT Search. |
| ClaudeBot | Anthropic | Model training | Collects content for model training. Blocking it does not block Claude-SearchBot or Claude-User. |
| Google-Extended | Model training | Controls use of your content for Gemini model training and grounding. It does NOT affect Google Search ranking or AI Overviews inclusion — those follow Googlebot. | |
| Applebot-Extended | Apple | Model training | Opts your content out of Apple foundation-model training without affecting Siri or Spotlight results. |
| Bytespider | ByteDance | Model training | Collects training data for ByteDance models. Widely reported to ignore robots.txt, so blocking may not be enforced. |
| CCBot | Common Crawl | Model training | Feeds the Common Crawl corpus, an upstream training source for many open models. Blocking removes you from future snapshots only. |
How the score works
100 points across five pillars. The weighting follows causality: if crawlers can't reach the page, nothing else matters, so access carries the most weight. Every check is deterministic — the same URL produces the same score, with no model calls involved.
| Pillar | Points | What it measures |
|---|---|---|
| Crawler Access | 30 | Whether each of the 16 AI crawlers is allowed to fetch the page, weighted so that blocking a live search agent costs far more than blocking a training-only agent. |
| llms.txt | 15 | Presence and structure of /llms.txt and /llms-full.txt, scored against the llmstxt.org format: an H1 title, a blockquote summary, and markdown links to key pages. |
| Answer Extractability | 25 | Single H1, logical heading nesting, question-shaped headings, self-contained answer paragraphs, lists and tables, and the density of concrete numeric facts. |
| Entity & Schema | 20 | Valid JSON-LD, an Organization or Person entity with sameAs links, content-type schema such as Article or FAQPage, author and date properties, and a self-referencing canonical. |
| Render Parity | 10 | How much content exists in the raw HTML before JavaScript executes, the script-to-text ratio, and whether a semantic main or article landmark is present. |
How to fix the most common failures
1. Allow AI search agents while still opting out of training
This is the configuration most sites actually want: stay out of training corpora, stay visible in AI answers.
# Opt out of model training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
# Stay visible in AI search and answers
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Applebot
Allow: /2. Publish an llms.txt
Save this at /llms.txt. The format is from llmstxt.org: an H1, a blockquote summary, then sections of markdown links.
# Acme Analytics
> Acme Analytics is a product analytics platform for
> engineering teams. Self-hosted, open source, SQL-native.
## Docs
- [Quickstart](https://acme.com/docs/quickstart): Install and send your first event.
- [SQL reference](https://acme.com/docs/sql): Full query syntax.
## Optional
- [Changelog](https://acme.com/changelog): Release history.3. Make answers extractable
Follow each heading with a self-contained answer of two to three sentences that makes sense with no surrounding context — that passage is the unit a model quotes. Phrase headings the way people ask questions. Convert comparisons and specifications into real <table> and <ul> markup rather than prose. Above all, make sure the content is in the server response: most AI crawlers do not execute JavaScript, so client-rendered text is invisible to them no matter how good it is.
How this differs from AI mention trackers
Tools from Ahrefs, Semrush and others prompt live models and count how often your brand appears. That is a genuinely useful outcome metric, and this tool does not replace it. What it can't tell you is why the number is what it is. This checker measures the inputs: access, structure, markup and rendering. It runs no model calls, which is exactly why it can be instant, unlimited and free — and why the result is identical every time you run it. Use this to fix the plumbing, then use a mention tracker to watch the curve move.
Frequently asked questions
What is an AI visibility checker?
An AI visibility checker audits whether AI systems like ChatGPT, Claude, Perplexity and Google's AI surfaces can access, read and quote your pages. This one checks the technical prerequisites: whether AI crawlers are allowed in your robots.txt, whether you publish an llms.txt, whether your content is structured so an answer can be extracted from it, whether you declare structured data, and whether your content exists in the raw HTML before JavaScript runs.
How is this different from tools that track AI brand mentions?
Mention trackers query ChatGPT or Perplexity with prompts and count how often your brand appears. That tells you the outcome. This tool tells you the cause: whether AI systems can technically reach and parse your content in the first place. If a crawler is blocked or your content only renders client-side, no amount of content work will get you cited. Run this first, then use a mention tracker to measure the result.
Does blocking GPTBot hurt my visibility in ChatGPT?
Not directly. GPTBot collects data for training OpenAI's models. ChatGPT's search feature uses a separate crawler called OAI-SearchBot. You can block GPTBot to opt out of training while remaining fully visible in ChatGPT Search — as long as OAI-SearchBot is allowed. Many sites blocked every OpenAI agent at once and unintentionally removed themselves from ChatGPT's search results.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot crawls content that may be used to train OpenAI's foundation models. OAI-SearchBot crawls content to surface websites in ChatGPT's search results. They are separate user agents with separate robots.txt rules. Anthropic makes the same split: ClaudeBot is for training, Claude-SearchBot builds Claude's search index, and Claude-User fetches pages when a user's question needs them.
Is llms.txt actually used by any AI system?
No major AI provider has publicly confirmed that it consumes llms.txt in production. It is a proposed standard from Answer.AI, not a ratified one. This tool scores it because it costs nothing to publish, the format is stable, and the exercise of choosing your most important pages is independently useful. Treat it as cheap optionality rather than a guaranteed ranking factor — and be sceptical of any tool that promises otherwise.
Why doesn't my site appear in ChatGPT or Perplexity?
The three most common technical causes, in order: AI search crawlers are blocked in robots.txt; the page is client-rendered so the raw HTML a crawler receives is nearly empty; or the content has no clear heading structure and no self-contained answers, so there is no clean passage to quote. This audit checks all three. Beyond the technical layer, being cited also depends on topical authority and whether your page is genuinely the best answer.
Do AI crawlers respect robots.txt?
Mostly, with important exceptions. OpenAI, Anthropic and Google document that their crawlers honour robots.txt. However, OpenAI notes that robots.txt may not apply to ChatGPT-User, and Perplexity states that user-initiated fetches generally fall outside robots.txt. Some crawlers, notably Bytespider, have been widely reported to ignore it. If you need enforcement rather than a request, block at the edge or firewall level.
What is answer extractability?
Answer extractability measures how easily a model can lift a clean, self-contained passage out of your page. In practice that means one clear H1, logical H2/H3 sections without skipped levels, headings phrased the way people actually ask questions, and a direct 2–3 sentence answer immediately after each heading. Pages structured this way get quoted; walls of undifferentiated prose don't, even when the information is there.
Does this tool store the URLs I audit?
No. The audit runs on request and the result is returned directly to your browser. Nothing is written to a database, there is no account, no email capture, and no log of audited URLs. The ?url= parameter in the address bar exists purely so you can share a link that re-runs the same audit.
Is this tool really free?
Yes — free, unlimited, and no signup. It runs entirely deterministic checks against public files and your page's HTML, with no AI API calls behind it, so there is no per-check cost to pass on and no reason to gate it.