Measure AI crawler traffic in your logs and find crawl errors

Use this guide to detect AI bot traffic (e.g., GPTBot, ChatGPT-User, PerplexityBot), see which agents are most active, and spot which webpages are most popular.

Quick answer

Scrunch tracks AI platform activity in two complementary ways: - Agent Traffic shows AI bots crawling your website in real time and classifies them as retrieval, indexer, or training bots. - AI Referrals shows human visitors who arrive from AI platforms by connecting to your GA4 data.

Together, you can see which AI tools access your content and whether AI search sends you traffic and conversions.

Get set up

• Connect your CDN or hosting provider to Agent Traffic. Scrunch offers native integrations with Akamai, AWS CloudFront, Cloudflare, Fastly, Vercel, and WordPress. If your CMS doesn’t expose raw CDN logs (HubSpot, BigCommerce, Shopify, Squarespace, Uberflip, Webflow, Wix), you can still integrate via a Cloudflare proxy.
• To monitor AI-driven visits, revenue, and conversions, connect your GA4 property to AI Referrals. Follow the GA4 connection guide.

Monitor AI bot visits in Agent Traffic

1) Open the Agent Traffic tab in Scrunch and select your domain.
2) Choose a date range to analyze trends and spikes.
3) Read the panels to understand activity: - Agent Traffic Over Time and Traffic Distribution show total bot visits and the share by bot type (retrieval, indexer, training). - Bot Requests counts show how your content is being used now (retrieval) versus prepared for future use (indexing, training). - Human vs. Bot Traffic compares real-time human traffic with AI bot traffic. - Top Bot Agents highlights which AI platforms (e.g., ChatGPT, Perplexity, Claude, Gemini, Meta AI) are most active on your site. - Top Content Pages reveals your most accessed pages by AI agents. - Recent Bot Requests lists bot hits with timestamp, agent, type, path, and response status so you can spot errors fast.

Scrunch automatically classifies bot activity: - Retrieval bots fetch content for live user queries. - Indexer bots crawl to build searchable indexes. - Training bots collect content to improve model performance.

Tip: Prioritize retrieval traffic. It’s the closest proxy to real buyer intent because it’s tied to live prompts.

See which pages bots hit most

Use the Top Content Pages panel in Agent Traffic. Sort by visits to identify pages AI agents rely on most to answer user questions, high-intent pages with frequent retrieval requests, and opportunities to improve access, speed, and content clarity on pages that LLMs already value.

Segment AI traffic the right way

• By model/platform: Compare activity from ChatGPT, Claude, Perplexity, Gemini, Meta AI, and others to see where you’re most visible.
• By bot type: Separate retrieval from indexing and training to isolate intent-driven activity.
• By page and path: Identify product pages, docs, pricing, and comparison content that attract agents.
• By time: Track spikes after content launches, PR hits, or site changes to understand cause and effect.

Figure out which AI tools matter most right now

• For bot activity: Check Top Bot Agents in Agent Traffic to see which AI platforms crawl your site most often and which pages get the most attention.
• For human visits: Use AI Referrals to identify which platforms drive sessions, conversions, and revenue from your GA4 data.

Using both views together shows which models engage with your content and which ones actually send people to your site.

What Scrunch tracks today (models and agents)

Scrunch currently supports eight major AI platforms for monitoring and insights: ChatGPT, Claude, Gemini, Perplexity, Google AI Mode, Google AI Overviews, Meta AI, and Microsoft Copilot. Support for Grok is coming soon. In Agent Traffic, Scrunch identifies and classifies visits from AI agents associated with these ecosystems, including well-known retrieval bots like GPTBot, ClaudeBot, and PerplexityBot.

Scrunch vs. GA4 vs. server logs

• Scrunch vs. GA4: AI bots typically don’t execute JavaScript or trigger pageviews, so GA4 misses most bot activity. Scrunch connects directly to your CDN/hosting to capture these “invisible” bot visits in real time.
• Scrunch vs. raw logs: Logs contain bot hits, but Scrunch enriches them with AI-specific identification, bot-type classification (retrieval, indexer, training), model-level insights, and purpose-built dashboards—removing the need for manual parsing and inference.

Result: You get complete visibility into both sides of AI search—non-human bot crawling (Agent Traffic) and human referrals and conversions (AI Referrals).

Can Scrunch distinguish crawlers from AI-driven referral traffic?

Yes. Agent Traffic captures non-human bot visits via CDN/hosting integrations. AI Referrals reports human sessions that originate from AI platforms by reading your GA4 property, including conversion metrics if you’ve configured them in GA4.

Practical tips and next steps

• Ensure access. Double-check robots.txt and security settings so legitimate AI agents can read your content.
• Run a site audit. Use Scrunch’s Site Audits to fix access, delivery, or content issues that limit AI visibility.
• Optimize high-interest pages first. Start with pages that retrieval bots hit most often.
• Broaden measurement. In addition to traffic, track your brand presence in AI search and citations in AI answers to grow visibility and share of voice.

New to Scrunch? Start tracking in minutes

• Connect your CDN or host to Agent Traffic for real-time AI bot monitoring. Setup guides are available for Cloudflare Workers, Akamai DataStream 2, Vercel, and WordPress. If your CMS doesn’t expose logs, use the Cloudflare proxy approach.
• Connect GA4 to AI Referrals to attribute sessions, revenue, and conversions from AI platforms. Follow the GA4 connection guide.

Want to see it first? Start a 7-day free trial or request a demo to see these reports in action.