AI Visibility: Can GPTBot and Other AI Crawlers Read Your Site?

Probably, but check three things. Your robots.txt file must let the right AI crawlers in. Your firewall must not turn them away. And your text must be in the page your server sends, not added later by JavaScript. Also know which bot does what. Blocking GPTBot, for example, stops OpenAI training on your site, but OpenAI says it doesn't keep you out of ChatGPT search.

We checked the websites of 26 IT and office-tech firms from our October AI tests. None of the robots.txt files we could read blocked an AI crawler. The problems we did find were elsewhere.

The short version

• Each AI company runs more than one bot. One collects pages for training. Another fetches pages for search answers. A third visits a page when a user asks about it. They do different jobs.

• GPTBot is the training bot. OpenAI says blocking it keeps your content out of training. Its search bot, OAI-SearchBot, is what decides whether you can appear in ChatGPT search answers.

• Google's AI Overviews and AI Mode use the normal Googlebot. Google-Extended, Google's AI opt-out, does not affect Google Search.

• Robots.txt was not the problem on the sites we checked. 22 of the 26 had a readable robots.txt, and none of the 22 blocked an AI crawler.

• Firewalls were. Two sites turned away our ordinary browser visit with a block page, and two more didn't load at all. Robots.txt shows none of that.

• Most AI crawlers don't run JavaScript. All 22 homepages we could fetch had their text in the HTML, which is what you want.

What does GPTBot do?

GPTBot is OpenAI's crawler for training its AI models. In OpenAI's own words, "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models."

It is not the bot that feeds ChatGPT's search answers. That job belongs to OAI-SearchBot, which "is used to surface websites in search results in ChatGPT's search features." OpenAI is direct about what blocking it means: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers."

The third bot, ChatGPT-User, visits a page when someone asks ChatGPT about it. Because a person started that request, OpenAI says "robots.txt rules may not apply."

The other AI companies split the work the same way.

Sources: OpenAI, Anthropic, Perplexity and Google, all checked 27 October 2026. Perplexity says PerplexityBot "is not used to crawl content for AI foundation models," and that its user fetcher "generally ignores robots.txt rules."

Does blocking Google-Extended remove you from AI Overviews?

No. Google is clear on this. Its crawler documentation says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." It controls whether Google can use your content to train and ground its Gemini models.

AI Overviews and AI Mode are part of Google Search. Google's guide to AI features says the control for them is your robots.txt rules for Googlebot, plus the usual snippet settings. It also says there are "no additional requirements to appear in AI Overviews or AI Mode," and that you don't need "new machine readable files, AI text files, or markup." That includes llms.txt, which we'll look at in a later post.

Should you block GPTBot?

It depends on what you want, and the two choices are separate.

If you want to be recommended in AI answers, allow the search bots. That means OAI-SearchBot, Claude-SearchBot, PerplexityBot and, of course, Googlebot. For a local business that wants to be named when a buyer asks for help, there is little reason to block any of them.

Training is a separate decision. Blocking GPTBot, ClaudeBot or Google-Extended keeps your pages out of future model training. OpenAI and Google both say that doesn't remove you from their search answers. Some owners block training on principle; others allow it in the hope that models learn their brand. We haven't seen solid evidence either way on what training access does for a small business's visibility, so treat it as a values choice, not a ranking tactic.

What you shouldn't do is block everything with one line because a plugin offered it. That shuts out the search bots along with the trainers.

What did we find when we checked 26 small-business websites?

On 27 October 2026 we checked the websites of 26 IT and office-tech firms that came up in our October AI tests. Some were firms AI kept naming; others were firms it left out. For each one we read the robots.txt file, loaded the homepage, and checked whether its text was in the HTML the server sends.

Robots.txt was fine everywhere we could read it. 22 of the 26 sites returned a robots.txt file. None blocked any named AI crawler, and none blocked all crawlers. Two went further and welcomed AI crawlers by name, listing GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot with "Allow." That's tidy, but it changes nothing: a crawler that isn't blocked is already allowed.

Firewalls and broken sites were the real gaps. Two sites refused our ordinary browser visit with a block page. One said "Sorry, you have been blocked," from a security service. The other said our "access to this site has been limited by the site owner." Two more didn't load at all. A fifth made us wait on a security check page before showing anything.

We were visiting from outside the US, so some of those blocks may be country rules. We can't see from the outside how they treat AI crawlers. But a rule strict enough to turn away a normal visitor can also turn away a crawler, and you won't find it in robots.txt.

The text was in the HTML. All 22 homepages we could fetch had their words in the page the server sends, from under 100 words on the thinnest site to over 3,000 on the longest. That matters for the reason below.

Can AI crawlers read JavaScript?

Mostly not. In October 2026, technical SEO consultant Glenn Gabe re-tested whether AI assistants can read pages that build their text with JavaScript, 434 days after his first test. ChatGPT, Claude and Perplexity failed every case. They saw the menu, the footer or an error, but not the main content. In his words, "over a year later, AI Search platforms still cannot render JavaScript-based content." Gemini, which uses Google's systems, read all three test pages. Microsoft Copilot, which uses Bing's, read two of the three.

That matches what Vercel saw in its server logs in December 2024: ChatGPT's and Claude's crawlers download script files but don't run them.

So if your page shows its text only after scripts run, the crawlers behind ChatGPT, Claude and Perplexity may see a blank page. All 22 homepages we checked sent their text with the page. But custom-built sites, booking widgets, reviews loaded from another service, and pricing tables built in JavaScript are common gaps. We also cover why this sinks schema markup in does schema help AI citations?

Is Cloudflare blocking AI crawlers on your site?

It might be, and you may not know it.

On 1 July 2025, Cloudflare announced that it would block AI crawlers by default unless site owners chose otherwise. It said "every new domain will now be asked if they want to allow AI crawlers," and that "customers can easily check their settings and enable crawling at any time." Cloudflare says it handles traffic for 20% of the web. It added that more than one million customers had already turned on its one-click AI-crawler block since September 2024.

If your site uses Cloudflare, or your web developer set it up, log in and look at the bot and AI-crawler settings. Many of these settings group AI bots together, so blocking one can block search bots too. Make sure the search bots you want are allowed.

How to check your own site in 10 minutes

1. Open yoursite.com/robots.txt. Look for lines that name GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot or Googlebot followed by "Disallow: /". Also look for "User-agent: *" followed by "Disallow: /", which blocks everyone.

2. Check your firewall or host. If you use Cloudflare or a security plugin, find its bot settings. Ask your web developer whether any country or bot blocking is switched on.

3. Check the text is in the page. In your browser, open the page, choose "View page source," and search for a sentence from your main heading. If you can't find it, the text is being added by JavaScript.

4. Ask an AI to read a page. Paste one of your service-page links into ChatGPT and ask it to summarise the page. If it says it can't open the page, something is in the way.

5. Look at your server logs, if your host offers them. Search for "GPTBot" and "OAI-SearchBot." Regular visits mean they can reach you.

Being readable is the floor, not the win. Once the crawlers can get in, what they find has to answer the buyer's question. That is where pages for each type of client you serve come in, which we covered in industry landing pages. Getting the technical side right is the first part of our technical SEO work.

Not sure whether AI crawlers can reach your site?

Our written audit checks your robots.txt, your firewall settings and what each crawler actually sees on your key pages. Then it runs your buyer questions on ChatGPT, Google's AI Mode, Gemini and Claude to show whether you're being named. The first audit is free, and you get a document, not a sales call.

Get a free audit →