Does Scrunch show me which webpages are most visited by AI bots?
Yes. Scrunch’s Agent Traffic shows exactly which webpages AI bots visit most, how often they visit, which AI platforms/models are responsible, and what type of request each bot is making.
Agent Traffic classifies every request as:
- Training: collecting data to build or fine-tune models
- Indexer: crawling to build searchable indexes
- Retrieval: fetching content in real time to answer live user queries
What enables this visibility: Scrunch connects directly to your CDN or hosting provider to capture real-time bot visits and identify which AI platforms are accessing content, what they’re requesting, and how frequently they come back.
What you’ll see in Agent Traffic
Top Content Pages
The specific URLs on your site that AI bots access most often (with counts), so you can spot high-value pages at a glance.
Top Bot Agents
Which AI platforms/models (for example, ChatGPT, Perplexity, Gemini, Claude) are most active on your site.
Bot traffic trends over time
Volume of visits across a selected date range, with filters for retrieval, indexer, and training activity.
Request type classification
Clear breakdown of whether visits are for training, indexing, or retrieval, so you can separate demand signals from background crawling.
Human vs. AI comparison
Side-by-side view of human visitors versus AI bot traffic to understand relative volume.
Real-time monitoring
A chronological log of recent bot requests (including bot ID, type, path, status, and timestamp) to track current activity.
Tip: Use the date range selector and bot-type filters to analyze frequency patterns, isolate retrieval traffic, and investigate spikes tied to launches or campaigns.
Example
A SaaS team connects their CDN to Agent Traffic and sees that:
- The pricing page gets heavy bot activity overall, but less retrieval traffic than expected.
- Key product pages show the highest retrieval traffic from several models.
They prioritize fixing any technical issues that could limit AI consumption on the pricing page and optimize the product pages to ensure AI answers include critical positioning and differentiators. They also reference the Human vs. AI panel to gauge how large their AI audience is relative to human visitors.
How to get set up
Connect your CDN or hosting provider
Scrunch offers native integrations with Akamai, AWS CloudFront, Cloudflare, Fastly, Vercel, and WordPress.
If your CMS doesn’t expose raw CDN logs (e.g., HubSpot CMS, BigCommerce, Shopify, Squarespace, Uberflip, Webflow, Wix), you can still integrate via a Cloudflare proxy.
(Optional) Track human traffic from AI platforms with AI Referrals
Connect your GA4 property to Scrunch to measure AI-referred sessions, views, conversions, revenue (if configured), sources, and landing pages. Follow the steps to connect GA4 to Scrunch.
How to use these insights
Focus on retrieval signals
Prioritize pages that receive the most retrieval traffic—these are directly tied to live user prompts.
Map pages to monitored prompts
Compare high-traffic URLs to your tracked prompts to confirm you’re monitoring the right questions and to spot gaps.
Optimize the pages AI uses most
Refine structure, clarity, and on-page signals on your top retrieval pages so AI agents reliably extract your key value props.
Keep AI access open (when possible)
Unless your content is paywalled or monetized, ensure AI bots aren’t unintentionally blocked by robots.txt or other controls so they can read and cite your content.