Article: 44.9% of the top 107 websites block at least one AI crawler, according to a 2026 analysis. The risk isn’t just a loss of traffic; it’s a potential disappearance from the new channels where users are getting information today.

The scope of the problem

Robots.txt has been the go-to file for telling Googlebot and other traditional search crawlers which pages to index. A similar file now controls AI crawlers—software agents that read web content to generate answers for tools like ChatGPT, Claude and Perplexity. The study found that 44.9 % of the examined sites have deliberately disallowed at least one of these bots. GPTBot, the crawler used by OpenAI’s models, tops the list of blocked agents.

A surprising share of the blocks comes from one-click settings in services such as Cloudflare, where site owners may unintentionally deny AI access while trying to tighten security.

What “AI crawlability” really means

Crawlability is the ability of a bot to fetch a page. For AI, crawlability determines whether a model can pull the latest facts about a brand, product or service when a user asks a question. If a site is not crawlable, the AI has no source to cite, and the brand disappears from the answer feed.

The old SEO playbook focused on Googlebot crawling pages so they could be ranked in a list of links. AI crawlability shifts the goal: instead of a ranking, the outcome is a recommendation inside a conversational answer.

Training bots vs. answer bots

Not all AI crawlers serve the same purpose.

  1. Training bots – examples include GPTBot, ClaudeBot and Google-Extended. They scrape large swaths of the web to feed the massive datasets that power the underlying language models. Site owners can block these if they want to keep proprietary content out of future model training.
  2. Answer bots – examples include OAI-SearchBot, Claude-SearchBot and PerplexityBot. These operate in real time, pulling specific snippets to answer a user’s query. Blocking an answer bot means the site’s content will never be surfaced in a conversational reply, even if the same page is still indexed by traditional search engines.

The analysis shows many sites are conflating the two, ending up with a blanket “block all AI” rule that harms visibility without protecting any real IP.

Why the wrong bots get blocked

A handful of factors explain the misconfiguration:

  • Default security presets – platforms that offer a single “block AI” toggle often apply it to every known crawler, including answer bots that the site would actually want to reach.
  • Lack of awareness – most webmasters know robots.txt for Googlebot, but the newer AI-specific directives are less familiar.
  • Fear of data misuse – owners worry that training bots will incorporate their content into future models, a legitimate concern that does not apply to answer bots that only fetch data on demand.

How to get the right balance

  1. Edit robots.txt – explicitly allow the answer bots you want to reach. A line such as User-agent: OAI-SearchBot\nAllow: / lets the OpenAI answer bot crawl the whole site while still keeping other agents blocked.
  2. Decide on training data – if protecting proprietary material is a priority, keep a Disallow rule for GPTBot, ClaudeBot and similar training agents.
  3. Adopt llms.txt – this emerging standard lets publishers highlight the most important pages for AI consumption, speeding up the discovery process for answer bots.
  4. Audit third-party settings – review any security or CDN configuration that may have auto-blocked AI crawlers, and adjust the rules manually.

The trade-off you must weigh

Allowing answer bots improves brand exposure in AI-driven conversations, but it also means the content can be reproduced verbatim in user-facing answers. Blocking training bots protects the data from being baked into future model weights, yet it does not stop answer bots from pulling the same public pages.

If a company’s core value is strict control over its IP, a tighter block may be justified, but the cost is reduced presence in the channels where many users now start their research. Conversely, an e-commerce site that thrives on product discovery will likely benefit more from being AI-crawlable than from the marginal risk of a few quoted snippets.

What to watch next

  • Việc áp dụng llms.txt – khi tiêu chuẩn này dần trở nên phổ biến, các công cụ phân tích nó có thể trở thành phương thức chính để các bot trả lời AI định vị được các nội dung có giá trị cao.
  • Sự thay đổi chính sách từ các nhà cung cấp AI – OpenAI, Anthropic và các bên khác có thể tinh chỉnh các chính sách thu thập dữ liệu (crawler) của họ, có khả năng cung cấp các cơ chế cho phép tham gia (opt-in) chi tiết hơn.
  • Hướng dẫn pháp lý về dữ liệu huấn luyện AI – những tranh luận đang diễn ra về việc liệu nội dung công khai được thu thập (scraped) có thể được sử dụng để huấn luyện mô hình hay không có thể ảnh hưởng đến việc có bao nhiêu trang web chọn cách chặn hoàn toàn các bot huấn luyện.

Điểm mấu chốt là: nếu một thương hiệu muốn duy trì sự hiện diện ở nơi mà người dùng đang ngày càng chuyển sang đặt câu hỏi thay vì gõ từ khóa, họ phải làm cho trang web của mình có khả năng được AI thu thập dữ liệu (AI-crawlable). Bước đầu tiên là một chỉnh sửa robots.txt đơn giản; bước thứ hai là một quyết định rõ ràng về những dữ liệu mà họ sẵn lòng chia sẻ với thế hệ tìm kiếm tiếp theo. Việc phớt lờ sự khác biệt giữa bot huấn luyện và bot trả lời có thể khiến một công ty trở nên vô hình trong chính không gian đang tái định hình cách tìm kiếm thông tin.