How to track if AI bots are visiting your website
Quick answer
Track AI bot traffic from Scrunch's Agent Traffic tab.
AI bot traffic is how often different AI crawlers visit your website. Here's how to track it:
View overview metrics in the Agent Traffic tab:
- Agent Traffic Over Time: Line chart showing how retrieval, indexer, and training bots have accessed your site over a selected time period
- Traffic Distribution: Percentage breakdown of traffic by bot type
- Bot Requests: Separate counts for retrieval, training, and indexer bots
Analyze bot activity:
- Human vs. Bot Traffic: Real-time comparison of human traffic versus AI traffic
- Top Bot Agents: Which AI platforms (ChatGPT, Gemini, Perplexity, etc.) visit your site most often
- Top Content Pages: Your most frequently accessed webpages by AI retrieval bots
- Recent Bot Requests: Chronological listing of all bot visits with timestamps and webpage path
Track changes over time: Use the date selector (default: last 1 day) to monitor bot activity patterns and identify traffic surges.
Filter by bot type: Focus analysis on retrieval, indexer, or training bots by selecting or deselecting checkboxes.
Bottom line: Higher AI bot traffic indicates more content retrieved and indexed, which results in better visibility in AI search results.
Start here
Why you should track if AI bots are visiting your website
AI bot traffic—aka traffic to your website from AI web crawlers—directly impacts brand presence in AI search. The less AI bots visit your site, the less likely it is that you’ll show up in AI responses (or show up accurately). To optimize AI search performance, you need to understand AI bot activity on your site and make sure your content is accessible.
Here’s how to do it using Scrunch.
Before you begin:
- Log in to Scrunch or sign up for a free 7-day trial to follow along.
- Connect your website to Scrunch’s Agent Traffic feature.
Step 1: Track AI bot traffic in the Agent Traffic tab
Log in to Scrunch and navigate to the Agent Traffic tab.
Select Agent Traffic from left-hand sidebar
This page shows you all the website domains you have connected to Scrunch’s Agent Traffic feature.
Find the domain you’re looking for and click the “View Details” button.
This page gives you details on how AI bots are accessing your website, AI traffic versus human traffic, which agents are visiting your site most often, which webpages they’re visiting most frequently, and a chronological listing of all AI bot visits.
You can see AI bot activity across your website over time by using the date selector at the top of the page (it’s set to the last 1 day by default).
Click the “Last 1 day” button to view your options or set a custom date range.
View date options
This data tells you whether your website is accessible to AI models and provides a high-level overview of whether your site content is being used to answer questions in AI search (via retrieval bots) or may be used in the future (via indexer and training bots).
💡 Pro tip: Unless the content on your site is monetized, Scrunch recommends giving AI bots access to your site. The more that LLMs know about your brand, products, and services, the likelier it is that they’ll represent them accurately in AI responses.
At the top of the page you’ll see:
- Status: Whether the CDN you’re using to track AI bot traffic is active or inactive
- Agent Traffic: How many AI bot visits your website has received over a selected time period
- Last Checked: When AI bot traffic counts were last updated
View Status, Agent Traffic, and Last Checked
Below this you’ll see panels for Agent Traffic Over Time and Traffic Distribution.
View Agent Traffic Over Time and Traffic Distribution
The former is a line chart showing how indexer, retrieval, and training bots have accessed your website over a selected time period. The latter is a percentage breakdown of traffic over a selected time period by bot type.
Below this you’ll see panels showing counts of retrieval, training, and indexer bot requests.
View bot requests
📖 Scrunch definition: Bot types
There are three AI bot types to be aware of: retrieval, indexer, and training. Retrieval bots execute real-time web searches and fetch results to provide up-to-date information in AI responses. Indexer bots crawl and analyze web content to help build a searchable database for faster information retrieval. Training bots scrape websites and use the content to improve model performance.
Use this data to gauge demand for content on your website in AI search, as well to see if surges in activity correlate with specific campaigns or content releases.
The higher and more consistent your traffic growth, the more your content is actively being referenced in AI responses and reaching the right audiences.
Next you’ll see the Human vs. Bot Traffic panel. This is a real-time comparison of human traffic to your website versus AI traffic.
View Human vs. Bot Traffic
You can configure your view of bot traffic in this panel by selecting or unselecting the “Retrieval,” “Training,” and “Indexer” checkboxes in the upper right-hand corner.
Below this is the Top Bot Agents panel. This shows you which AI bots are most active on your website over time, as well as a breakdown of top bots (and bot type) by percentage of traffic.
View Top Bot Agents
Use this data to understand which AI platforms users are asking questions on related to your brand. Lower traffic from certain bots may indicate lower demand on that platform or an opportunity to optimize content for that specific model.
💡 Pro tip: Focus on retrieval bot traffic. While allowing bots to crawl your site for indexing and training purposes is useful, retrieval traffic is connected directly to human prompts.
Below this you’ll see a panel for Top Content Pages. This shows you exactly which webpages on your site are most frequently accessed by AI retrieval bots.
View Top Content Pages
Use this data to understand which content on your website most closely matches what human users are searching for, as well as what content on your site AI platforms find most valuable and relevant for user queries.
Finally you’ll see a table for Recent Bot Requests. This is a chronological listing of all bot requests on your website that includes timestamp, bot ID, bot type, webpage path, and status (i.e., whether the request was successful or not).
View Recent Bot Requests
With this information, you can clearly see how AI bots are accessing (or not accessing) your website, which bot types are on your site most often, and which webpages bots are pulling content from most frequently—all of which impacts your performance in AI search results.
What’s next
Start increasing AI bot traffic volume:
- Prioritize technical updates: Make sure AI bots can access and read website content before focusing on content updates or creation.
- Fix robots.txt files: Make sure robots.txt files aren’t blocking legitimate AI bot traffic.
- Run a site audit: Fix access, delivery, or content issues hurting your visibility.
- Optimize frequently crawled webpages: Focus on making technical and content updates to the pages most commonly crawled by AI bots.
Bot traffic analysis helps you understand how often your website is being considered in responses to AI prompts.
However, bot traffic on its own doesn’t guarantee that your brand will show up consistently or accurately in AI answers. Scrunch also recommends tracking brand presence and citations to understand and improve brand performance in AI search.
Not a Scrunch customer yet? Start your 7-day free trial or request a demo to see Scrunch in action.