How Scrunch detects AI bot traffic visiting your site

Scrunch detects AI bot traffic through direct CDN and hosting integrations that capture real-time bot visits. These integrations surface which AI platforms are accessing your content, what they request, how often they visit, and how those requests are classified. Scrunch also distinguishes between training bots (collecting data to train models), indexer bots (building search indexes), and retrieval bots (fetching content for live user queries).

What you’ll see in Agent Traffic

When you connect your CDN or hosting provider, the Agent Traffic tab shows:

• Bot traffic trends over time by specific models such as ChatGPT and Perplexity
• Request type classification across training, indexing, and retrieval
• Top bot agents and the webpages they visit most
• Human vs AI bot traffic to understand relative volume
• A real-time log of recent bot requests

This visibility helps you see which AI platforms engage with your content and which pages drive the most visibility in AI search.

Why this works better than GA4 or server logs for bot detection

Most AI bots don’t trigger traditional pageviews, so they rarely appear in standard analytics. Scrunch connects directly to your CDN or hosting layer (e.g., Cloudflare, Akamai, Vercel, WordPress) to capture bot requests as they happen. This “invisible” traffic often won’t show up in GA4 or typical server analytics.

• Scrunch Agent Traffic monitors crawler activity in real time via CDN/hosting integrations.
• Scrunch AI Referrals connects to Google Analytics 4 to measure actual human visitors who arrive from AI platforms.
• The key difference: GA4 and standard server logs miss much of the non-pageview bot activity. Scrunch’s CDN-level visibility identifies the platform/model and classifies the request type so you can act on it.

Crawler traffic vs AI-driven referrals

Scrunch separates crawler activity from human referral visits so you can measure both sides of the AI journey:

• Agent Traffic: Real-time monitoring of AI bots crawling your site, including traffic over time, traffic by model, request type, human vs bot comparison, top agents, top pages, and recent requests.
• AI Referrals: GA4-connected reporting on human traffic from AI platforms, including AI-referred sessions, views, purchase revenue (if configured in GA4), AI-referred conversion rate, non-AI conversion rate, AI sources, and AI-referred landing pages.

Learn more in the FAQ on how Scrunch tracks AI search visits. You can also connect GA4 by following the steps in the guide to connecting AI Referrals.

Supported integrations

Scrunch offers native integrations with major CDN and hosting providers, including Akamai, AWS CloudFront, Cloudflare, Fastly, Vercel, and WordPress.

If your CMS doesn’t expose raw CDN logs, you can still integrate with Scrunch via a Cloudflare proxy for HubSpot CMS, BigCommerce, Shopify, Squarespace, Uberflip, Webflow, and Wix. See how to integrate via proxy through Cloudflare.

AI-readiness checks Scrunch supports

Scrunch helps you assess and improve how accessible and useful your content is to AI platforms:

• Connection status and freshness: See whether your tracking connection is active and when it was last checked.
• Site Audit: Run a site audit to find access, delivery, or content issues that hurt AI visibility.
• Robots.txt guidance: Ensure you’re not blocking legitimate AI bots so your content can be crawled, indexed, and retrieved.
• Prioritized optimization: Use Top Bot Agents and Top Content Pages to focus updates on the pages AI retrieval bots hit most often.
• Agent Experience Platform (AXP): Deliver an AI-optimized version of your site directly to AI agents to improve how platforms read, understand, and cite your content. Learn more about AXP.

Multi-LLM coverage

Scrunch supports eight major AI platforms, including ChatGPT, Claude, Gemini, Perplexity, Google AI Mode, Google AI Overviews, Meta AI, and Microsoft Copilot, with additional platforms coming soon. See the full list of supported platforms.

Getting started

• Connect your CDN or hosting provider to start monitoring crawler activity. Follow the integration instructions for Agent Traffic.
• If you can’t access CDN logs directly, integrate via a Cloudflare proxy.
• Connect your GA4 property to track human traffic from AI platforms with AI Referrals. Follow the GA4 connection guide.

Related resources

• Learn how Scrunch tracks AI search visits to your website
• Step-by-step: How to track if AI bots are visiting your website
• Guide to AI search

Links: - Learn how Scrunch tracks AI search visits to your website: https://scrunchai.com/faqs/how-does-scrunch-track-ai-search-visits-to-my-website - Connect Agent Traffic integrations: https://intercom.help/scrunchai/en/collections/15678483-agent-traffic - Integrate via Cloudflare proxy: https://intercom.help/scrunchai/en/articles/12845473-supported-cdn-integrations-for-agent-traffic - Connect GA4 to AI Referrals: https://intercom.help/scrunchai/en/articles/11826932-connecting-the-ai-traffic-tool-to-your-google-analytics-account - Run a site audit: https://intercom.help/scrunchai/en/articles/11833527-understanding-the-site-audit-tool-in-scrunch - Learn about AXP: https://scrunchai.com/platform/agent-experience/ - Supported AI platforms: https://scrunchai.com/faqs/which-ai-platforms-and-llms-can-scrunch-track-and-monitor - How-to guide for tracking AI bot visits: https://scrunchai.com/how-tos/how-to-track-if-ai-bots-are-visiting-your-website