Article: 44.9% of the top 107 websites block at least one AI crawler, according to a 2026 analysis. The risk isn’t just a loss of traffic; it’s a potential disappearance from the new channels where users are getting information today.

The scope of the problem

Robots.txt has been the go-to file for telling Googlebot and other traditional search crawlers which pages to index. A similar file now controls AI crawlers—software agents that read web content to generate answers for tools like ChatGPT, Claude and Perplexity. The study found that 44.9 % of the examined sites have deliberately disallowed at least one of these bots. GPTBot, the crawler used by OpenAI’s models, tops the list of blocked agents.

A surprising share of the blocks comes from one-click settings in services such as Cloudflare, where site owners may unintentionally deny AI access while trying to tighten security.

What “AI crawlability” really means

Crawlability is the ability of a bot to fetch a page. For AI, crawlability determines whether a model can pull the latest facts about a brand, product or service when a user asks a question. If a site is not crawlable, the AI has no source to cite, and the brand disappears from the answer feed.

The old SEO playbook focused on Googlebot crawling pages so they could be ranked in a list of links. AI crawlability shifts the goal: instead of a ranking, the outcome is a recommendation inside a conversational answer.

Training bots vs. answer bots

Not all AI crawlers serve the same purpose.

  1. Training bots – examples include GPTBot, ClaudeBot and Google-Extended. They scrape large swaths of the web to feed the massive datasets that power the underlying language models. Site owners can block these if they want to keep proprietary content out of future model training.
  2. Answer bots – examples include OAI-SearchBot, Claude-SearchBot and PerplexityBot. These operate in real time, pulling specific snippets to answer a user’s query. Blocking an answer bot means the site’s content will never be surfaced in a conversational reply, even if the same page is still indexed by traditional search engines.

The analysis shows many sites are conflating the two, ending up with a blanket “block all AI” rule that harms visibility without protecting any real IP.

Why the wrong bots get blocked

A handful of factors explain the misconfiguration:

  • Default security presets – platforms that offer a single “block AI” toggle often apply it to every known crawler, including answer bots that the site would actually want to reach.
  • Lack of awareness – most webmasters know robots.txt for Googlebot, but the newer AI-specific directives are less familiar.
  • Fear of data misuse – owners worry that training bots will incorporate their content into future models, a legitimate concern that does not apply to answer bots that only fetch data on demand.

How to get the right balance

  1. Edit robots.txt – explicitly allow the answer bots you want to reach. A line such as User-agent: OAI-SearchBot\nAllow: / lets the OpenAI answer bot crawl the whole site while still keeping other agents blocked.
  2. Decide on training data – if protecting proprietary material is a priority, keep a Disallow rule for GPTBot, ClaudeBot and similar training agents.
  3. Adopt llms.txt – this emerging standard lets publishers highlight the most important pages for AI consumption, speeding up the discovery process for answer bots.
  4. Audit third-party settings – review any security or CDN configuration that may have auto-blocked AI crawlers, and adjust the rules manually.

The trade-off you must weigh

Allowing answer bots improves brand exposure in AI-driven conversations, but it also means the content can be reproduced verbatim in user-facing answers. Blocking training bots protects the data from being baked into future model weights, yet it does not stop answer bots from pulling the same public pages.

If a company’s core value is strict control over its IP, a tighter block may be justified, but the cost is reduced presence in the channels where many users now start their research. Conversely, an e-commerce site that thrives on product discovery will likely benefit more from being AI-crawlable than from the marginal risk of a few quoted snippets.

What to watch next

  • Adoption of llms.txt – as the standard gains traction, tools that parse it could become the primary way AI answer bots locate high-value content.
  • Policy shifts from AI providers – OpenAI, Anthropic and others may refine their crawler policies, potentially offering more granular opt-in mechanisms.
  • Legal guidance on AI training data – ongoing debates about whether scraped public content can be used for model training could influence how many sites choose to block training bots outright.

The bottom line: if a brand wants to stay visible where users are increasingly asking questions instead of typing keywords, it must make its site AI-crawlable. The first step is a simple robots.txt edit; the second is a clear decision about what data it is comfortable sharing with the next generation of search. Ignoring the distinction between training and answer bots could leave a company invisible in the very space that’s reshaping how information is found.