> ## Documentation Index
> Fetch the complete documentation index at: https://docs.surfais.com/llms.txt
> Use this file to discover all available pages before exploring further.

# AI crawler access

> Which AI bots to allow in robots.txt — and what blocking them actually costs you.

`robots.txt` is the file at the root of your site that tells crawlers which parts of your site they may visit. Several AI companies now publish their own crawlers, and how you treat each one in that file has a real — but very uneven — effect on your AI visibility. This page explains which crawlers matter, why, and what a sensible `robots.txt` looks like.

## Not all AI crawlers do the same job

The biggest misconception is treating every AI bot the same way. Surfais groups the AI crawlers it checks into three classes, because blocking each one has a completely different consequence:

<CardGroup cols={2}>
  <Card title="Search-index crawlers — blocking costs you citations">
    These fetch your pages specifically to include you in AI search results and answers. Block one of these and you lose citations on that platform, full stop. This is the class worth checking first.
  </Card>

  <Card title="Live fetchers — your firewall matters more than robots.txt">
    These visit a page at the exact moment a real user asks an AI platform about you. Most ignore `robots.txt` rules by the vendor's own admission — the real control is whether your firewall or CDN challenges or blocks the request.
  </Card>

  <Card title="Training crawlers — blocking is a legitimate choice">
    These harvest content to train a future model. Blocking one is a deliberate, defensible decision with no measured visibility cost today — it is not a mistake, and Surfais never flags it as one.
  </Card>
</CardGroup>

## The crawlers Surfais checks

**Search-index crawlers — should be allowed.** Block any of these and you lose citations on that platform directly.

| Crawler          | Platform                                              |
| ---------------- | ----------------------------------------------------- |
| OAI-SearchBot    | ChatGPT search                                        |
| Claude-SearchBot | Claude web search                                     |
| PerplexityBot    | Perplexity                                            |
| Bingbot          | Bing, Copilot, and part of ChatGPT's search retrieval |
| Googlebot        | Google Search and AI Overviews                        |
| Applebot         | Siri, Spotlight, and Apple Intelligence               |
| Amzn-SearchBot   | Alexa answers                                         |
| DuckAssistBot    | DuckDuckGo's AI answers                               |
| MistralAI-Index  | Mistral's Le Chat search                              |
| meta-webindexer  | Meta AI search                                        |

**Live fetchers — your firewall matters more than `robots.txt`.** These fire when someone is actively asking about you. Most ignore `robots.txt` by the vendor's own admission, so the meaningful check is whether your firewall or CDN challenges them.

| Crawler              | Platform                 |
| -------------------- | ------------------------ |
| ChatGPT-User         | ChatGPT live browsing    |
| Claude-User          | Claude live browsing     |
| Perplexity-User      | Perplexity live browsing |
| MistralAI-User       | Le Chat live browsing    |
| meta-externalfetcher | Meta AI live browsing    |

**Training crawlers — blocking is a legitimate choice.** These build a future model's general knowledge. Allowing or blocking them is your call, and Surfais never marks a block here as a problem.

| Crawler                      | Platform                           |
| ---------------------------- | ---------------------------------- |
| GPTBot                       | OpenAI model training              |
| ClaudeBot                    | Anthropic model training           |
| Google-Extended              | Gemini training and grounding      |
| Applebot-Extended            | Apple model training               |
| Amazonbot                    | Amazon model training              |
| CCBot                        | Common Crawl (feeds many trainers) |
| Bytespider                   | ByteDance model training           |
| cohere-training-data-crawler | Cohere model training              |
| meta-externalagent           | Meta model training                |

<Note>
  Google-Extended is a training-and-grounding control for Gemini, not a switch for AI Overviews. AI Overviews eligibility follows regular Google Search indexing via Googlebot — blocking Google-Extended does not remove you from AI Overviews, but it does cost you Gemini's grounded citations.
</Note>

## Recommended posture

For most brands, the sensible default is straightforward:

<Steps>
  <Step title="Allow every search-index crawler">
    This is the highest-leverage technical change available to you. There's essentially never a good reason to block one of these — doing so removes you directly from that platform's answers.
  </Step>

  <Step title="Check your firewall for live fetchers">
    Make sure ChatGPT-User, Claude-User, and Perplexity-User aren't being challenged or blocked by your WAF or CDN. A `robots.txt` rule alone won't stop these anyway — the block that matters happens at the network layer.
  </Step>

  <Step title="Decide training crawlers on their own merits">
    Blocking GPTBot, ClaudeBot, and the rest is entirely reasonable if you don't want your content used for model training. It has no measured effect on whether you're cited in that platform's answers today.
  </Step>
</Steps>

Check your current rules any time on your brand's [AI Readiness page](/readiness/readiness-page) — it reads your live `robots.txt`, classifies every AI crawler it finds a rule for, and shows only the blocks that are actually costing you visibility as critical.

## About llms.txt

`llms.txt` is a proposed convention — a plain-text file at your site root summarising your content for AI tools to read. You may see it recommended elsewhere as a must-have. Surfais treats it as **optional**: no major AI platform is confirmed to read it today, and independent studies of large numbers of domains have found no measurable citation effect from having one.

That doesn't mean you shouldn't add one — it's a low-effort, zero-risk hedge, and some documentation and agent tooling does consume it. It simply means its absence is never flagged as a problem on your Readiness page, and it should never be where you spend limited effort that could go toward allowing the right crawlers or earning genuine editorial coverage.

## Related pages

<CardGroup cols={2}>
  <Card title="The AI Readiness page" href="/readiness/readiness-page">
    Where your live crawler-access results are shown, alongside the rest of your readiness check.
  </Card>

  <Card title="Schema & structured data" href="/readiness/schema-structured-data">
    The other half of on-site technical readiness.
  </Card>

  <Card title="What audits check" href="/audits/what-audits-check">
    How Technical AI-Readiness feeds into your audit's Readiness Score.
  </Card>

  <Card title="AIS Score vs Readiness Score" href="/metrics/ais-vs-readiness">
    How this technical groundwork relates to your measured visibility.
  </Card>
</CardGroup>