GPT Visibility

ChatGPT SEO: let AI crawlers read your site

The most basic part of “ChatGPT SEO” is technical: if ChatGPT’s search crawler can’t fetch your pages, it won’t show them. Here’s which bots matter, what each one does, and how to check.

Last updated 15 October 2026 · Written by GPT Visibility

The bots that matter

User agentWhoWhat it’s for
OAI-SearchBotOpenAISurfaces sites in ChatGPT search. Opted-out sites won’t be shown in ChatGPT search answers.
GPTBotOpenAICrawls content that may be used to train OpenAI’s models. Independent of OAI-SearchBot.
ChatGPT-UserOpenAIFetches pages when a user’s action asks for them. OpenAI says robots.txt rules may not apply and it isn’t used to decide search inclusion.
PerplexityBotPerplexitySurfaces and links sites in Perplexity search; Perplexity says it isn’t used to train foundation models.
Perplexity-UserPerplexityUser-requested fetches; Perplexity says it generally ignores robots.txt.
GooglebotGoogleCrawls for Google Search, including AI Overviews and AI Mode.
Google-ExtendedGoogleA robots.txt token controlling use of content for training and grounding in some other Google AI systems. Google says it does not affect Search inclusion.

Recommended robots.txt for most small businesses

If you want to be visible in AI search, allow the search crawlers. Whether you allow training bots such as GPTBot is your choice – it doesn’t affect ChatGPT search inclusion according to OpenAI.

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

# Optional – training crawlers (your choice)
User-agent: GPTBot
Allow: /

User-agent: *
Allow: /

Sitemap: https://www.yourdomain.co.uk/sitemap.xml

OpenAI and Perplexity both say changes can take up to about 24 hours to be picked up.

The hidden blocker: firewalls and CDNs

robots.txt is only a request. Many sites are blocked at a different layer – a CDN, hosting “bot fight” mode or a security plugin returns 403 Forbidden or a challenge page to crawlers. Both OpenAI and Perplexity publish IP ranges and recommend allowing them:

Perplexity’s documentation includes step-by-step allow rules for Cloudflare and AWS WAF that combine the user agent with the published IP ranges – the safest pattern, because anyone can fake a user-agent string.

How to check your site

  1. Open yourdomain.co.uk/robots.txt and look for any Disallow: / under these user agents or under *.
  2. Check your CDN or hosting security settings for AI-bot blocking toggles.
  3. Look in your server or CDN logs for requests from OAI-SearchBot and PerplexityBot, and the status codes they received.
  4. A quick (imperfect) test is fetching your homepage with the bot’s user-agent string. A 403 suggests user-agent blocking; a 200 doesn’t prove the real bot gets through, because real bots come from specific IP ranges.

Beyond access

Access is necessary, not sufficient. Once crawlers can read you, what they find matters – see how to get recommended by ChatGPT.

Frequently asked questions

What is OAI-SearchBot?

OAI-SearchBot is OpenAI’s crawler for ChatGPT search. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.

Is GPTBot the same as OAI-SearchBot?

No. OpenAI documents them as independent: GPTBot relates to training data for its models, OAI-SearchBot to search. You can allow one and block the other.

Does blocking Google-Extended remove me from AI Overviews?

No. Google says Googlebot controls crawling for Search, including AI features, and Google-Extended relates to some other Google AI systems. To limit what appears from your pages in Search, Google points to nosnippet, max-snippet and noindex controls.

Do I need an llms.txt file?

It isn’t required. Google says you don’t need new machine-readable or AI text files to appear in its AI features. An llms.txt file is harmless and can help some tools, but it doesn’t replace crawler access or clear content.

Sources

Related guides