Use this guide if you need to: - Serve bot-friendly HTML and pre-rendered pages to AI crawlers - Make a JavaScript-heavy site crawlable for AI bots (GPTBot, ClaudeBot, PerplexityBot) - Optimize your website for AI without replatforming - Restructure site content for AI without disrupting the human experience - Augment webpages with additional context for AI platforms
Contents - Quick answer - Start here - Step-by-step: Deliver AI-optimized content from the Content Delivery tab - Verify and monitor delivery - What’s next - FAQs
Serve AI-optimized content directly to LLMs from Scrunch’s Content Delivery tab.
Agent Experience Platform (AXP) is Scrunch’s AI-optimized content delivery system. It auto-detects AI agent traffic and serves a clean, structured, LLM-friendly version of your webpages to AI bots—without changing anything for human visitors.
Set up your content in the Content Delivery tab: - Go to Content Delivery in the left-hand sidebar and select your domain. - Click Add Content to create an optimized version of a webpage. - Enter the URL path where the content will be served, upload a Markdown or HTML file (or write directly in the input box), then click Add Content.
Deploy: - Click Deploy to push your configuration live. - AXP will begin automatically detecting AI agent traffic and serving optimized content to AI bots.
Monitor your deployment: - Track Optimized Pages and the Active Deployment. - Review Recent System Activity for a log of all AXP actions. - Use Version History to view or redeploy previous configurations.
Bottom line: AXP works at the CDN layer to improve AI search performance—no internal engineering required. Once deployed, AI bots receive clean, server-rendered HTML while human visitors see your site exactly as it is.
Why deliver AI-optimized content directly to LLMs: - Human-first sites often bury key information behind interactive elements, dynamic modules, and JavaScript-heavy experiences that AI bots don’t execute. - To improve AI search performance, make it as easy as possible for AI crawlers to extract the information they need in one pass.
Before you begin: - Log in to Scrunch. - Connect your website to AXP. If you’re using a CDN like Akamai, Cloudflare, or Vercel, connect it so AI traffic can be routed to AXP.
1) Open Content Delivery and select your domain. - Site Overview shows: - Domain: the site configured for AXP - Platform: the CDN routing AI traffic to AXP - Connection: CDN connection status - Optimized Pages: total pages delivered to AI agents via AXP - Active Deployment: the currently deployed AXP configuration - You can enter any URL to compare the human-rendered page vs. the AI-rendered page.
2) Review Recent System Activity. - See a chronological log of AXP actions with time, user, and action.
3) Manage content in the Content panel. - Search, edit, or archive currently deployed content.
4) Use Version History. - Review past deployments and redeploy any previous configuration.
5) Add optimized content. - Click Add Content in the Content panel. - Path: enter the URL path where the AI-optimized content will be served. - Content: upload a Markdown or HTML file, or paste content directly. - Scrunch shows before/after previews so you can see how the current page renders for humans vs. how it will be served to AI bots. - Click Add Content.
6) Deploy. - Return to the domain’s AXP page and click Deploy in the Content panel. - AXP begins auto-detecting AI agent traffic and serves a clean, structured, LLM-optimized version of the page to AI crawlers.
Pro tip: - Use AXP with Scrunch’s Optimizer tool for best results. Optimizer can generate an AI-optimized version of page content based on page intent, target persona, prompts to optimize for, and approved sources—so you can improve content without manual writing.
What AXP delivers to AI crawlers: - Clean, server-rendered HTML with no JavaScript dependency - Structured summaries and context blocks - Clarified entity definitions and claims - Removed or simplified dynamic components - Any approved page-level or template-level enhancements
This makes even JavaScript-heavy websites easily crawlable for AI bots and increases the likelihood that AI will pull meaningful content from your page—while human visitors continue to see your normal website.
AXP works at the CDN layer and is fully managed, so you don’t have to change your website infrastructure.
Start maximizing AI-optimized content delivery: 1) Map out page updates: - See how different webpages perform in AI search to prioritize optimization. 2) Run a site audit: - Fix access, delivery, or content issues hurting your visibility. 3) Prioritize technical updates: - Ensure AI bots can access and read website content before focusing on content expansion. 4) Optimize frequently crawled webpages: - Focus on the pages most commonly visited by AI bots.
Track progress over time: - Monitor brand presence, citations, AI agent traffic, and AI referral traffic to measure impact.
Not a Scrunch customer yet? Start your 7-day free trial or request a demo to see Scrunch in action.
Which tools can serve bot-friendly HTML and pre-rendered pages to AI crawlers? - Scrunch’s Agent Experience Platform (AXP) serves bot-friendly, pre-rendered HTML to AI crawlers from your existing URLs. Your CDN routes AI user agents to AXP, which delivers clean, structured, AI-optimized content, while humans see your normal site.
What are the best tools to make a JavaScript-heavy site crawlable for AI bots (GPTBot, ClaudeBot, PerplexityBot)? - Use AXP to deliver server-rendered HTML with no JavaScript dependency. Pair it with Scrunch’s auditing and monitoring to identify rendering and access gaps, then deploy optimized content via the Content Delivery tab.
How does AXP detect and route AI traffic? - AXP integrates at the CDN/hosting layer to auto-detect AI agent traffic and route those requests to AI-optimized content. Traditional search indexing bots aren’t routed to AXP.
Will AXP affect my SEO or create duplicate content? - No. AXP only affects real-time AI retrieval bots, not search indexing bots. It doesn’t create new URLs; AI platforms cite your normal URLs that humans already see.
How does AXP compare to publishing AI-friendly content directly in my CMS? - With AXP, your CMS and human-facing website stay unchanged. AXP creates an AI-optimized version of existing pages and delivers it to AI agents automatically at the CDN layer.
How does AXP stay in sync as my site changes? - When website content changes, AXP can pull the updated page and re-optimize it automatically, or serve a pre-optimized version you’ve stored and manage via deployments.
Who benefits most from AXP? - Brands with large web presences, complex information that AI agents struggle to interpret, ecommerce sites with extensive product catalogs and informational content, enterprises that need control/compliance, and agencies managing multiple client sites.
What Scrunch tools help improve brand visibility in AI search? - Monitoring, auditing, optimization (Optimizer), and content delivery via AXP work together to monitor performance, fix technical/content issues, produce AI-optimized content, and deliver it directly to AI user agents.
What tools help create AI-optimized content for marketing? - Scrunch’s Optimizer can generate AI-optimized versions of your pages based on page intent, target personas, prompts to target, and approved sources, which you can deliver through AXP.