AI crawlers are inflating your traffic numbers, and the usual shortcut—checking the User-Agent header—doesn’t catch it.

A recent eight-day log sweep uncovered 468 genuine article fetches but 991 requests that pretended to be AI bots while actually probing for files like .env or .git. The culprit? Anyone can insert a self-declared User-Agent string such as GPTBot or ChatGPT-User into an HTTP request.

Why the metric matters

Websites treat a surge in “AI traffic” as a sign of relevance. They use the numbers to justify ad rates, allocate server resources, and brag to investors. When half the reported AI hits are merely noise, budgets go astray and dashboards become misleading.

The two filters that separate claim from reality

  1. Cloudflare’s verifiedBotCategory – Cloudflare fills this field only after it resolves the request’s IP back to a known bot domain via reverse DNS. If the header says “GPTBot” but the IP fails the lookup, the request lands in the “unverified” bucket.
  2. Sitemap cross-check – A bot truly reads your content only when the requested URL appears in your XML sitemap. Requests for paths absent from the sitemap likely scout for vulnerabilities rather than index articles.

Applying both filters to the same eight-day sample produced stark numbers:

  • ChatGPT-User – 39 % of requests passed verification.
  • GPTBot – 13 % verified.
  • PerplexityBot – 0 % verified.
  • Google-Extended – 0 % verified; the string does not correspond to any official Google crawler.

Even among the verified calls, many bots fetched only robots.txt or the sitemap itself, not the article pages you care about.

What developers can do today

  • Enable Cloudflare bot verification in your analytics pipeline. The verifiedBotCategory field appears in the request headers and can be stored alongside your own logs.
  • Maintain an up-to-date sitemap and automate a check that every incoming request’s path exists there before you count it as a content view.
  • Filter out non-GET methods and requests that target typical development files (.env, .git, .bak). Those are almost always malicious scans.

The counter-point

Some argue that reverse DNS checks can be spoofed, and that a bot’s legitimate purpose may be to discover new URLs not yet listed in the sitemap. Those concerns are valid: a determined attacker could compromise a DNS record, and new content will naturally be absent from the current sitemap until the next generation cycle.

The pragmatic response is to treat verification as a confidence score rather than an absolute gate. Combine DNS verification, sitemap presence, and request-method checks to raise the bar for what you count as “real AI traffic.” If a request passes two of three checks, flag it for manual review instead of discarding it outright.

What to watch next

  • Changes to Cloudflare’s verification API – any alteration to the verifiedBotCategory logic could shift verification rates.
  • Emergence of new self-identified bots – watch the User-Agent strings that appear in your logs; a sudden spike may indicate a new scanner masquerading as an AI crawler.
  • Sitemap generation frequency – longer intervals increase the chance that legitimate crawlers will be misclassified.

The takeaway is simple: stop treating a User-Agent string as proof. Layer DNS verification and sitemap validation, and you’ll see a clearer picture of who is actually reading your content.

Source: https://dev.to/aulvem/ai-crawler-user-agents-are-self-reported-468-real-fetches-991-fake-ones-bgo