How to serve AI-optimized content directly to LLMs
Quick answer
Serve AI-optimized content directly to LLMs from Scrunch's Content Delivery tab.
Agent Experience Platform (AXP) is Scrunch's AI-optimized content delivery system. It auto-detects AI agent traffic and serves a clean, structured, LLM-friendly version of your webpages to AI bots—without changing anything for human visitors. Here's how to use it:
Set up your content in the Content Delivery tab:
- Navigate to Content Delivery in the left-hand sidebar and select your domain.
- Click the “Content” button, then the “Add content” button, to upload or create an optimized version of a webpage.
- Enter the URL path where the content will be served, upload a markdown or HTML file (or write directly in the input box), and click the “Add Content” button.
Configure AXP behavior rules:
- Use the Actions panel to set rules that block, redirect, augment, or redact specific pages or content for AI agents.
- Target rules by AI platform or bot type for more granular control.
Deploy:
- Click the “Deploy” button to push your configuration live.
- AXP will begin automatically detecting AI agent traffic and serving optimized content to AI bots.
Monitor your deployment:
- Track Active Rules and Optimized Content counts at the top of the domain page.
- Review Recent Activity for a log of all AXP actions.
- Use Version History to view or redeploy previous configurations.
Bottom line: AXP works at the CDN layer to improve AI search performance—no internal engineering required. Once deployed, AI bots receive clean, structured HTML while human visitors see your site exactly as it is.
Start here
Why you should deliver AI-optimized content directly to LLMs
Your website is built for humans, not AI bots. If information is buried beneath interactive elements and dynamic content, JavaScript-heavy experiences, visual navigation AI bots can’t interpret, and content structures optimized for browsing instead of answering questions, your AI search performance will suffer. To optimize AI search performance, you need to make it as easy as possible for AI bots to extract the information they need.
Here’s how to do it using Scrunch.
Before you begin:
- Log in to Scrunch.
- Connect your website to Scrunch’s Agent Experience Platform (AXP) feature.
Step 1: Serve AI-optimized content directly to LLMs in the Content Delivery tab
Log in to Scrunch and navigate to the Content Delivery tab.
Select Content Delivery from left-hand sidebar
This page shows you all the website domains you have connected to Scrunch’s Agent Experience Platform (AXP) feature.
Find the domain you’re looking for and click the “View Details” button.
This page acts as your control center for AXP, allowing you to auto-detect AI agent traffic and serve bot-friendly HTML to AI crawlers without changing the site experience for human visitors.
At the top of the page you’ll see:
- Site: The website you currently have configured for AXP
- Status: Whether the CDN you’re using to route AI traffic to AXP is active or inactive
- Active Rules: How many custom AXP behavior rules are currently active
- Optimized Content: How many webpages are currently delivered to AI agents via AXP
View Site, Status, Active Rules, and Optimized Content
Below this you’ll see the Actions panel. This is where you can configure AXP behavior rules, create and manage optimized content, and compare AXP content to original content.
View Actions
Rules include the ability to:
- Block specific pages from AI access.
- Redirect AI traffic to alternative content.
- Augment pages with additional context.
- Redact sensitive or irrelevant information.
- Target specific AI platforms or bot types.
💡 Pro tip: Unless the content on your site is monetized, Scrunch recommends giving AI bots access to your site. The more that LLMs know about your brand, products, and services, the likelier it is that they’ll represent them accurately in AI responses.
Below this you’ll see panels for Recent Activity and Version History.
View Recent Activity and Version History
The former is a chronological listing of all AXP activity that includes time, user, action, and details. The latter is a chronological listing of AXP deployment history that allows you to view details of AXP deployments and redeploy previous versions of your AXP configuration.
Click the “Content” button in the Actions panel to create AXP content.
View Content
This page shows you all the content currently configured for AXP by URL.
Click the “Add content” button in the top right corner of the page.
View Add New Content modal
Enter the URL where content will be served in the “Path” box.
Next, add a markdown or HTML file to the “Content” upload box.
You can also write or copy-paste text into the input box below the upload box.
The goal is to provide AXP with an optimized version of site content—clear, information-dense, and comprehensive.
Scrunch will display before and after versions of the page to show you how the current page is rendered for human visitors versus how it will look to AI bots.
Click the “Add Content” button, click back to the AXP page for your website domain, and click the “Deploy” button in the Version History panel.
💡 Pro tip: Use AXP in conjunction with Scrunch’s Optimizer tool for the best results. Optimizer will help you generate an AI-optimized version of page content based on the intent of the page, your target persona, prompts you want the page to be optimized for, and sources it should pull from to improve content without any manual writing or uploading.
Once the new content is deployed, AXP will automatically detect AI agent traffic and serve a clean, structured, LLM-optimized version of the page to AI bots minus any superfluous code or other issues that may hinder AI consumption.
AXP works at the CDN layer and is fully managed, meaning no internal engineering is required.
Scrunch uses optimization rules and AI to restructure, summarize, and translate content into greatly simplified HTML for LLMs in your existing URL structure so key facts and messages are surfaced clearly.
This includes:
- Clean, server-rendered HTML with no JavaScript dependency
- Structured summaries and context blocks
- Clarified entity definitions and claims
- Removed or simplified dynamic components
- Any approved page-level or template-level enhancements
This makes even JavaScript-heavy websites easily crawlable for AI bots and increases the likelihood that AI will pull meaningful content from your page. Meanwhile, human visitors continue to see your website as is.
With AXP deployed, you can serve AI-optimized content directly to LLMs at scale and improve your AI search performance without touching your existing website infrastructure.
What’s next
Start maximizing AI-optimized content delivery:
- Map out page updates: See how different webpages perform in AI search to prioritize optimization efforts.
- Run a site audit: Fix access, delivery, or content issues hurting your visibility.
- Prioritize technical updates: Make sure AI bots can access and read website content before focusing on content updates or creation.
- Optimize frequently crawled webpages: Focus on making technical and content updates to the pages most commonly crawled by AI bots.
AI-optimized content delivery helps you improve how AI agents retrieve content from your website without disrupting the site experience for human visitors.
Scrunch recommends monitoring how your AI search performance improves over time by tracking brand presence, citations, AI agent traffic, and AI referral traffic.
Not a Scrunch customer yet? Start your 7-day free trial or request a demo to see Scrunch in action.