Article: 44.9% of the top 107 websites block at least one AI crawler, according to a 2026 analysis. The risk isn’t just a loss of traffic; it’s a potential disappearance from the new channels where users are getting information today.

The scope of the problem

Robots.txt has been the go-to file for telling Googlebot and other traditional search crawlers which pages to index. A similar file now controls AI crawlers—software agents that read web content to generate answers for tools like ChatGPT, Claude and Perplexity. The study found that 44.9 % of the examined sites have deliberately disallowed at least one of these bots. GPTBot, the crawler used by OpenAI’s models, tops the list of blocked agents.

A surprising share of the blocks comes from one-click settings in services such as Cloudflare, where site owners may unintentionally deny AI access while trying to tighten security.

What “AI crawlability” really means

Crawlability is the ability of a bot to fetch a page. For AI, crawlability determines whether a model can pull the latest facts about a brand, product or service when a user asks a question. If a site is not crawlable, the AI has no source to cite, and the brand disappears from the answer feed.

The old SEO playbook focused on Googlebot crawling pages so they could be ranked in a list of links. AI crawlability shifts the goal: instead of a ranking, the outcome is a recommendation inside a conversational answer.

Training bots vs. answer bots

Not all AI crawlers serve the same purpose.

  1. Training bots – examples include GPTBot, ClaudeBot and Google-Extended. They scrape large swaths of the web to feed the massive datasets that power the underlying language models. Site owners can block these if they want to keep proprietary content out of future model training.
  2. Answer bots – examples include OAI-SearchBot, Claude-SearchBot and PerplexityBot. These operate in real time, pulling specific snippets to answer a user’s query. Blocking an answer bot means the site’s content will never be surfaced in a conversational reply, even if the same page is still indexed by traditional search engines.

The analysis shows many sites are conflating the two, ending up with a blanket “block all AI” rule that harms visibility without protecting any real IP.

Why the wrong bots get blocked

A handful of factors explain the misconfiguration:

  • Default security presets – platforms that offer a single “block AI” toggle often apply it to every known crawler, including answer bots that the site would actually want to reach.
  • Lack of awareness – most webmasters know robots.txt for Googlebot, but the newer AI-specific directives are less familiar.
  • Fear of data misuse – owners worry that training bots will incorporate their content into future models, a legitimate concern that does not apply to answer bots that only fetch data on demand.

How to get the right balance

  1. Edit robots.txt – explicitly allow the answer bots you want to reach. A line such as User-agent: OAI-SearchBot\nAllow: / lets the OpenAI answer bot crawl the whole site while still keeping other agents blocked.
  2. Decide on training data – if protecting proprietary material is a priority, keep a Disallow rule for GPTBot, ClaudeBot and similar training agents.
  3. Adopt llms.txt – this emerging standard lets publishers highlight the most important pages for AI consumption, speeding up the discovery process for answer bots.
  4. Audit third-party settings – review any security or CDN configuration that may have auto-blocked AI crawlers, and adjust the rules manually.

The trade-off you must weigh

Allowing answer bots improves brand exposure in AI-driven conversations, but it also means the content can be reproduced verbatim in user-facing answers. Blocking training bots protects the data from being baked into future model weights, yet it does not stop answer bots from pulling the same public pages.

If a company’s core value is strict control over its IP, a tighter block may be justified, but the cost is reduced presence in the channels where many users now start their research. Conversely, an e-commerce site that thrives on product discovery will likely benefit more from being AI-crawlable than from the marginal risk of a few quoted snippets.

What to watch next

  • Adoção do llms.txt – à medida que o padrão ganha tração, ferramentas que o interpretam podem se tornar a principal forma de os bots de resposta de IA localizarem conteúdo de alto valor.
  • Mudanças nas políticas dos provedores de IA – OpenAI, Anthropic e outros podem refinar suas políticas de rastreamento, potencialmente oferecendo mecanismos de opt-in mais granulares.
  • Orientações jurídicas sobre dados de treinamento de IA – os debates em curso sobre se o conteúdo público coletado pode ser usado para o treinamento de modelos podem influenciar quantos sites optam por bloquear bots de treinamento de forma definitiva.

Em resumo: se uma marca deseja permanecer visível onde os usuários estão cada vez mais fazendo perguntas em vez de digitar palavras-chave, ela deve tornar seu site rastreável por IA. O primeiro passo é uma simples edição no robots.txt; o segundo é uma decisão clara sobre quais dados ela se sente confortável em compartilhar com a próxima geração de busca. Ignorar a distinção entre bots de treinamento e bots de resposta pode deixar uma empresa invisível no exato espaço que está remodelando a forma como a informação é encontrada.