How does Scrunch detect AI bot traffic visiting my site?

Scrunch detects AI bot traffic through direct integrations with your CDN or hosting provider. These connections capture real-time bot requests that never appear in traditional analytics tools like Google Analytics 4 (GA4), then classify them by bot type and platform — so you can see which AI agents are accessing your content, what they're requesting, and how often. Because most AI crawlers don't execute JavaScript, standard pageview-based analytics miss the majority of this traffic entirely.

Bot Category Purpose Known Examples
Training bots Collect data to improve model performance GPTBot (OpenAI), ClaudeBot (Anthropic)
Indexer bots Crawl content to build searchable indexes used by AI search platforms PerplexityBot (Perplexity), Google-Extended (Google AI Overviews)
Retrieval bots Fetch content in real time to answer specific user prompts, at the moment a query is submitted ChatGPT browsing, Perplexity real-time fetch

Image from Agent Traffic tab in Scrunch

What you can see in Agent Traffic

Once your CDN or host is connected, the Agent Traffic tab provides five categories of insight:

  1. Bot traffic trends over time, broken down by specific AI models (e.g., ChatGPT, Perplexity)
  2. Request type classification — training, indexing, or retrieval
  3. Top bot agents and the webpages they access most frequently
  4. Real-time monitoring of the most recent bot requests

This visibility helps you understand which AI platforms actively engage with your content, which pages attract the most crawler attention, and how your AI search visibility compares across platforms like ChatGPT, Perplexity, and Google AI Overviews.

How this differs from GA4 and server logs

GA4 and standard pageview-based analytics rarely capture AI bot activity because bots typically don't execute page scripts — meaning crawls from agents like GPTBot, PerplexityBot, or ClaudeBot go entirely unrecorded in conventional dashboards. Raw server logs can surface some of this activity, but they are often noisy, incomplete, and provide no classification of requests by AI model, platform, or intent. The result is a systematic blind spot: brands have no reliable way to know whether their content is being read, indexed, or cited by AI systems.

Scrunch goes beyond both by offering three capabilities standard tools lack:

  1. Direct CDN/host integration to monitor AI crawlers in real time, capturing "invisible" bot traffic that powers AI answers but never triggers pageviews
  2. Request classification by bot type (retrieval, indexer, training) and by platform or model name
  3. AI Referrals (via GA4 integration) to surface the human visits, conversions, and revenue that arrive from AI platforms — connecting crawler activity to downstream business impact

Crawler traffic vs. AI-driven referral traffic

Scrunch uses two complementary data sources to separate bot activity from human visits influenced by AI:

Together, these two views give a complete picture: supply — how often and how deeply AI systems pull from your content — and demand — how often users arrive at your site having been influenced by an AI answer. Neither metric alone is sufficient; a site can receive heavy crawler traffic without generating AI-driven human visits, or attract AI referrals from content that is rarely re-crawled.

Which AI platforms and models are tracked

Scrunch currently tracks bot and referral traffic across nine major AI platforms:

Support for Grok is coming soon.

Technical AI-readiness and diagnostics

Scrunch provides three tools to verify that AI agents can read and use your content — and to improve how they do it:

  1. Site Optimization — runs technical and editorial checks to identify issues that may limit AI visibility, covering access controls and content quality problems
  2. Robots.txt and bot accessibility guidance — flags settings that unintentionally block legitimate AI crawlers such as GPTBot or ClaudeBot
  3. Agent Pages — automatically converts your site into an AI-optimized format and delivers it directly to AI agents for clearer understanding and higher citation rates

Tip: Unless your content is paywalled or otherwise monetized behind access controls, allow legitimate AI crawlers to access your site. Blocking bots like GPTBot or ClaudeBot via robots.txt prevents LLMs from accurately representing your brand in AI-generated answers. The more accurately AI models can read and index your content, the more likely they are to cite it in responses to relevant user queries.

Supported CDN and hosting integrations

Scrunch offers native integrations with six major CDN and hosting providers:

If your CMS doesn’t expose raw CDN logs (e.g., HubSpot CMS, BigCommerce, Shopify, Squarespace, Uberflip, Webflow, Wix), you can still integrate via a Cloudflare proxy.

Keep exploring