robots.txt is the file at the root of your site that tells crawlers which parts of your site they may visit. Several AI companies now publish their own crawlers, and how you treat each one in that file has a real — but very uneven — effect on your AI visibility. This page explains which crawlers matter, why, and what a sensible robots.txt looks like.
Not all AI crawlers do the same job
The biggest misconception is treating every AI bot the same way. Surfais groups the AI crawlers it checks into three classes, because blocking each one has a completely different consequence:Search-index crawlers — blocking costs you citations
These fetch your pages specifically to include you in AI search results and answers. Block one of these and you lose citations on that platform, full stop. This is the class worth checking first.
Live fetchers — your firewall matters more than robots.txt
These visit a page at the exact moment a real user asks an AI platform about you. Most ignore
robots.txt rules by the vendor’s own admission — the real control is whether your firewall or CDN challenges or blocks the request.Training crawlers — blocking is a legitimate choice
These harvest content to train a future model. Blocking one is a deliberate, defensible decision with no measured visibility cost today — it is not a mistake, and Surfais never flags it as one.
The crawlers Surfais checks
Search-index crawlers — should be allowed. Block any of these and you lose citations on that platform directly.
Live fetchers — your firewall matters more than
robots.txt. These fire when someone is actively asking about you. Most ignore robots.txt by the vendor’s own admission, so the meaningful check is whether your firewall or CDN challenges them.
Training crawlers — blocking is a legitimate choice. These build a future model’s general knowledge. Allowing or blocking them is your call, and Surfais never marks a block here as a problem.
Google-Extended is a training-and-grounding control for Gemini, not a switch for AI Overviews. AI Overviews eligibility follows regular Google Search indexing via Googlebot — blocking Google-Extended does not remove you from AI Overviews, but it does cost you Gemini’s grounded citations.
Recommended posture
For most brands, the sensible default is straightforward:1
Allow every search-index crawler
This is the highest-leverage technical change available to you. There’s essentially never a good reason to block one of these — doing so removes you directly from that platform’s answers.
2
Check your firewall for live fetchers
Make sure ChatGPT-User, Claude-User, and Perplexity-User aren’t being challenged or blocked by your WAF or CDN. A
robots.txt rule alone won’t stop these anyway — the block that matters happens at the network layer.3
Decide training crawlers on their own merits
Blocking GPTBot, ClaudeBot, and the rest is entirely reasonable if you don’t want your content used for model training. It has no measured effect on whether you’re cited in that platform’s answers today.
robots.txt, classifies every AI crawler it finds a rule for, and shows only the blocks that are actually costing you visibility as critical.
About llms.txt
llms.txt is a proposed convention — a plain-text file at your site root summarising your content for AI tools to read. You may see it recommended elsewhere as a must-have. Surfais treats it as optional: no major AI platform is confirmed to read it today, and independent studies of large numbers of domains have found no measurable citation effect from having one.
That doesn’t mean you shouldn’t add one — it’s a low-effort, zero-risk hedge, and some documentation and agent tooling does consume it. It simply means its absence is never flagged as a problem on your Readiness page, and it should never be where you spend limited effort that could go toward allowing the right crawlers or earning genuine editorial coverage.
Related pages
The AI Readiness page
Where your live crawler-access results are shown, alongside the rest of your readiness check.
Schema & structured data
The other half of on-site technical readiness.
What audits check
How Technical AI-Readiness feeds into your audit’s Readiness Score.
AIS Score vs Readiness Score
How this technical groundwork relates to your measured visibility.