Scrunch works with teams across sizes, industries, and technology stacks to help them understand and improve how their brand appears in AI platforms — including ChatGPT, Perplexity, Google AI Overviews, Gemini, Claude, and Microsoft Copilot. Because these AI systems now retrieve content from the web in real time, visibility depends on how accessible and parseable your site is to AI user agents. Bots such as OAI-SearchBot (OpenAI), PerplexityBot, and Googlebot-Extended fetch pages synchronously and render only server-side HTML — meaning sites that block these agents or rely on client-side JavaScript are effectively invisible to AI-generated answers.
AI optimization is fast-evolving, but five technical issues surface most consistently across Scrunch audits. Fixing them improves AI citation rates and typically benefits traditional SEO and human UX at the same time.
To appear in web search, a site must allow indexing bots such as Googlebot and Bingbot — and most hosting platforms permit these by default. AI platforms, however, use separate real-time retrieval bots that fetch pages synchronously when a user asks a relevant question. Firewalls and anti-bot services (e.g., AWS WAF Bot Control, Cloudflare, Datadome, Imperva) often block these agents unintentionally, because they were configured before AI retrieval bots existed or were categorized as generic scrapers.
Many teams block all bots from OpenAI to prevent model training. OpenAI's published documentation explicitly states that OAI-SearchBot — their real-time retrieval agent — does not train on retrieved content. This means a site can allow OAI-SearchBot for live retrieval while still blocking GPTBot, OpenAI's training crawler, via separate robots.txt rules.
How to check: Review firewall rules and robots.txt. Ask brand-specific questions in ChatGPT and see whether your content is cited. Or use Scrunch’s Site Audit to scan and flag bot-blocking patterns across pages.
How to fix: Update rules in your web host, WAF, or anti-bot provider (e.g., AWS WAF Bot Control, Cloudflare, Datadome, Imperva) to permit reputable AI retrieval user agents.
More detail: Read our overview of AI user agents.
Current-generation AI retrieval bots — including OAI-SearchBot, PerplexityBot, and Googlebot-Extended — do not execute JavaScript. They consume the raw server response: the HTML delivered before any scripts run. If primary content is injected by JavaScript (the default behavior for single-page applications and app-shell frameworks such as React, Vue, and Angular), AI agents receive a near-empty document and cannot cite that content.
Googlebot does execute JavaScript for indexing — a capability Google officially launched in 2014 — but AI retrieval operates differently: it is synchronous and time-budgeted. ChatGPT and Perplexity fetch multiple sources in parallel for each query and cannot wait for client-side rendering to complete, making server-rendered HTML a hard requirement for AI citation.
How to check: In Chrome DevTools, enable “Disable JavaScript” and load your pages. Scrunch’s Site Audit compares the “human” (JS-enabled) and “AI” (server-rendered) views to spot gaps.
How to fix: Prefer server-rendered or pre-rendered HTML for meaningful content (articles, product details, FAQs). If a refactor isn’t feasible—or you want fine-grained control—use Scrunch’s Agent Experience Platform (AXP). AXP detects AI agents and serves clean, server-rendered HTML without JS dependencies.
AI search relies on plain, well-structured text. Pages dominated by embedded media — videos, interactive diagrams, audio players — with little explanatory copy give AI agents no extractable facts to cite. At the other extreme, pages exceeding roughly 5,000–10,000 words can dilute key information and exceed the practical token budgets AI systems apply during retrieval. The target is a focused, fact-dense page: comprehensive enough to answer a specific question, concise enough for AI to summarize accurately.
A good rule of thumb: think “feature article,” not “novel.” If your page reads like James Joyce, it’s probably too long for AI retrieval.
How to check: Review pages manually or use Site Audit’s content-length checks.
How to fix:
- For thin pages, add relevant prose: summaries, captions, transcripts, and clear FAQs.
- For very long pages, segment logically and ensure prominent summaries with scannable sections. Avoid simple pagination that splits a single article into multiple URLs.
Even when content is server-rendered, certain markup patterns fragment meaning when converted to plain text — the format most AI systems use internally during summarization. Complex CSS layouts, visual page-builder output, and table-heavy designs often lose their structure when formatting is stripped, leaving the AI with text that is out of order or contextually disconnected.
Example: neat-looking pricing tables can degrade into jumbled text the model struggles to interpret.
How to check: Preview with Reader Mode or examine the DOM with JS disabled. In Scrunch’s Site Audit, the AI Preview shows how agents likely “see” your page, and Content Clarity flags risky patterns.
How to fix:
- Adjust HTML structure and content order so important meaning survives when formatting is stripped.
- Provide prose summaries for structured data that AI might misread (e.g., a short “Pricing at a glance” section).
Schema markup (JSON-LD) drives Google's rich results, but current AI retrieval systems — including ChatGPT, Perplexity, and Google AI Overviews — are still primarily driven by unstructured body text, not structured data. Relying on schema in place of readable prose is a common gap: a page with well-formed JSON-LD but thin copy will still be skipped by AI agents that cannot extract citable facts from the text itself.
How to fix: Keep your schema, but ensure the page itself includes the same facts in clear prose. Review the four items above to make the text AI-friendly.
Traditional crawlers focus on search indexing and UX signals for engines like Google: internal links, meta tags, canonicalization, sitemaps, and JS rendering as Googlebot might do.
Scrunch’s Site Audit is built for AI retrieval and citation:
- Simulates AI user agents and fetches the server-rendered “no-JS” view that ChatGPT, Perplexity, and others consume.
- Identifies robots.txt, WAF, and host-level blocks for AI bots specifically.
- Compares “human” vs. “AI” views to detect content that disappears without JS.
- Checks content length, clarity, and markup that can confuse AI summarization.
- Surfaces page-level issues correlated with citation likelihood across AI platforms.
Pair this with Scrunch Monitoring & Insights to see brand presence, citations, and competitor benchmarks across major AI platforms—data traditional SEO tools don’t provide.
Scrunch audits fetch pages the way AI agents do, then compare:
- Detects when primary content is missing from the server-rendered output.
- Flags reliance on client-side rendering, dynamic components, and heavy JS that prevents AI access.
- Validates agent access by testing known AI user agents and reporting blocked responses.
- Confirms AXP delivery by verifying that AI agents receive clean, server-rendered HTML.
This approach reliably highlights pages where AI crawlers cannot access or interpret content—even when those pages look fine in a normal browser or pass a traditional SEO crawl.
If you’re evaluating solutions for SEO or content strategy, prioritize:
- Cross-platform monitoring of AI answers and citations (ChatGPT, Perplexity, Google AI Overviews, Gemini, Claude, Copilot, Meta AI).
- Prompt-level tracking with personas and geographies for realistic, decision-stage queries.
- Citation and link analysis, including unlinked brand mentions.
- Auditing for AI-specific blockers: robots/WAF rules, JS dependency, content clarity, and content length.
- An “AI view” or preview of how agents parse your pages.
- Actionable recommendations and page-level fixes.
- Optional delivery of AI-optimized content to agents without changing the human site (e.g., Agent Experience Platform (AXP)).
- Change tracking, reporting, and integrations for content ops.
To go deeper on measurement and action, explore Scrunch’s Monitoring & Insights and Agent Experience Platform.