How does Scrunch serve bot-friendly HTML and pre-rendered pages to AI crawlers?
- Also asked as:
- How does AXP handle AI crawler requests?
- What technology does Scrunch use to optimize pages for AI?
Scrunch serves bot-friendly HTML and pre-rendered pages to AI crawlers via its Agent Experience Platform (AXP). The user’s CDN routes AI user agents to AXP, which delivers AI-optimized content directly to AI crawlers.
Additional context: Content served via AXP is clean, server-rendered HTML with no JavaScript dependency. This includes structured summaries and context blocks, clarified entity definitions and claims, removed or simplified dynamic components, and any approved page-level or template-level enhancements.
Example
For example, a Scrunch user using Cloudflare as their CDN would follow this automated workflow:
Initial request: AI user agent (e.g., ChatGPT-User, PerplexityBot, etc.) retrieves webpage using the existing URL already indexed in search.
Routing logic: Cloudflare routes AI agent to the Agent Experience Platform origin.
Optimized delivery: AXP serves the agent an optimized version of the page designed for AI comprehension.
Result: AI agents receive structured, easily parseable content while the human web visitor experience remains unchanged.
Follow-up question: Does Agent Experience Platform change search indexing?
No, Agent Experience Platform does not change search indexing. AXP only affects real-time AI retrieval bots, not search indexing bots. Search engine crawlers like Googlebot and Bingbot are not routed to AXP.
Related FAQs
What methods does Scrunch use to collect data from AI platforms?
Scrunch uses multiple methodologies to collect prompt data from AI platforms like ChatGPT, Perplexity, Google AI Overviews, and others, including browser automation and official platform APIs.
How do I create and use customer personas in Scrunch?
Scrunch allows users to create customer personas based on unique characteristics and geographies and either auto-generate prompts based on those personas or assign personas to existing prompts for targeted tracking and filtering.
Which AI platforms and LLMs can Scrunch track and monitor?
Scrunch currently supports seven major AI platforms: ChatGPT, Claude, Gemini, Perplexity, Google AI Mode, Google AI Overviews, and Meta AI. Support for Microsoft Copilot and Grok is coming soon.