Article: 44.9% of the top 107 websites block at least one AI crawler, according to a 2026 analysis. The risk isn’t just a loss of traffic; it’s a potential disappearance from the new channels where users are getting information today.

The scope of the problem

Robots.txt has been the go-to file for telling Googlebot and other traditional search crawlers which pages to index. A similar file now controls AI crawlers—software agents that read web content to generate answers for tools like ChatGPT, Claude and Perplexity. The study found that 44.9 % of the examined sites have deliberately disallowed at least one of these bots. GPTBot, the crawler used by OpenAI’s models, tops the list of blocked agents.

A surprising share of the blocks comes from one-click settings in services such as Cloudflare, where site owners may unintentionally deny AI access while trying to tighten security.

What “AI crawlability” really means

Crawlability is the ability of a bot to fetch a page. For AI, crawlability determines whether a model can pull the latest facts about a brand, product or service when a user asks a question. If a site is not crawlable, the AI has no source to cite, and the brand disappears from the answer feed.

The old SEO playbook focused on Googlebot crawling pages so they could be ranked in a list of links. AI crawlability shifts the goal: instead of a ranking, the outcome is a recommendation inside a conversational answer.

Training bots vs. answer bots

Not all AI crawlers serve the same purpose.

  1. Training bots – examples include GPTBot, ClaudeBot and Google-Extended. They scrape large swaths of the web to feed the massive datasets that power the underlying language models. Site owners can block these if they want to keep proprietary content out of future model training.
  2. Answer bots – examples include OAI-SearchBot, Claude-SearchBot and PerplexityBot. These operate in real time, pulling specific snippets to answer a user’s query. Blocking an answer bot means the site’s content will never be surfaced in a conversational reply, even if the same page is still indexed by traditional search engines.

The analysis shows many sites are conflating the two, ending up with a blanket “block all AI” rule that harms visibility without protecting any real IP.

Why the wrong bots get blocked

A handful of factors explain the misconfiguration:

  • Default security presets – platforms that offer a single “block AI” toggle often apply it to every known crawler, including answer bots that the site would actually want to reach.
  • Lack of awareness – most webmasters know robots.txt for Googlebot, but the newer AI-specific directives are less familiar.
  • Fear of data misuse – owners worry that training bots will incorporate their content into future models, a legitimate concern that does not apply to answer bots that only fetch data on demand.

How to get the right balance

  1. Edit robots.txt – explicitly allow the answer bots you want to reach. A line such as User-agent: OAI-SearchBot\nAllow: / lets the OpenAI answer bot crawl the whole site while still keeping other agents blocked.
  2. Decide on training data – if protecting proprietary material is a priority, keep a Disallow rule for GPTBot, ClaudeBot and similar training agents.
  3. Adopt llms.txt – this emerging standard lets publishers highlight the most important pages for AI consumption, speeding up the discovery process for answer bots.
  4. Audit third-party settings – review any security or CDN configuration that may have auto-blocked AI crawlers, and adjust the rules manually.

The trade-off you must weigh

Allowing answer bots improves brand exposure in AI-driven conversations, but it also means the content can be reproduced verbatim in user-facing answers. Blocking training bots protects the data from being baked into future model weights, yet it does not stop answer bots from pulling the same public pages.

If a company’s core value is strict control over its IP, a tighter block may be justified, but the cost is reduced presence in the channels where many users now start their research. Conversely, an e-commerce site that thrives on product discovery will likely benefit more from being AI-crawlable than from the marginal risk of a few quoted snippets.

What to watch next

  • Adoption de llms.txt – à mesure que la norme gagne du terrain, les outils qui l'analysent pourraient devenir le principal moyen pour les bots de réponse IA de localiser du contenu à haute valeur ajoutée.
  • Évolutions des politiques des fournisseurs d'IA – OpenAI, Anthropic et d'autres pourraient affiner leurs politiques de crawling, en proposant potentiellement des mécanismes d'opt-in plus granulaires.
  • Orientations juridiques sur les données d'entraînement de l'IA – les débats en cours sur la possibilité d'utiliser du contenu public extrait pour l'entraînement des modèles pourraient influencer le nombre de sites choisissant de bloquer purement et simplement les bots d'entraînement.

L'essentiel : si une marque souhaite rester visible là où les utilisateurs posent de plus en plus de questions au lieu de taper des mots-clés, elle doit rendre son site accessible au crawling de l'IA. La première étape est une simple modification du fichier robots.txt ; la seconde est une décision claire sur les données qu'elle accepte de partager avec la prochaine génération de recherche. Ignorer la distinction entre les bots d'entraînement et les bots de réponse pourrait rendre une entreprise invisible dans l'espace même qui est en train de remodeler la manière dont l'information est trouvée.