For the first time in over a decade, bots now account for the majority of internet traffic. The driving force is AI—crawlers from OpenAI, Anthropic, Google, and others are aggressively indexing the web to power their models and answer engines. Your site is being read by machines far more than by humans.

User-driven AI bots—crawlers that visit sites in real time when a user asks an AI a question—grew 15x in 2025. Unlike training crawlers that scrape content in bulk, user-driven bots indicate real purchase intent behind each visit. A surge in user-driven crawl traffic is a direct signal that AI users are actively asking questions about the brands and topics those sites cover.

GPTBot traffic growth in 2025

GPTBot—OpenAI's crawler for both training and search—grew 305% in 2025, making it one of the fastest-growing crawlers on the web. Sites that block GPTBot risk being deprioritized in ChatGPT's knowledge base and citation pool. Sites that are well-structured and openly accessible are positioned to benefit disproportionately as crawl volume surges.

Pages ClaudeBot crawls per visitor it refers

ClaudeBot crawls 38,000 pages for every single visitor it ends up referring. This extreme ratio reveals that ClaudeBot is primarily a training crawler, not a search-indexing tool. It reads at massive scale to build Anthropic's understanding of the web, but only a fraction of that reading translates directly into citations. Being accessible to ClaudeBot today shapes how Claude answers questions tomorrow.

Nearly 4 in 5 major news publishers now block AI training crawlers via robots.txt. As more high-authority sources opt out, the remaining open web becomes more valuable to AI models—and brands that remain accessible and well-structured gain a larger share of citations by default.

The vast majority of AI bot traffic—80%—is for training future models, not powering current search results. Only 18% is for search indexing and 2% is user-driven queries. This means most AI crawling today doesn't translate directly to citations yet, but it shapes how the next generation of models understand your brand.