Agency_
Check What GPTBot, ClaudeBot & PerplexityBot Actually See Without JavaScript
Most AI retrieval bots—including GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot—do not execute heavy client-side JavaScript bundles. If your pages rely on client hydration or hidden React streaming fallbacks, answer engines read an empty shell. Agency verifies server-rendered HTML parity, bot access rules, JSON-LD entities, and security headers across 7 crawlers.
1. Why Answer Engines & AI Search Bots Silently Miss Client-Rendered Websites
Modern React, Vue, and Next.js applications often score cleanly in desktop browsers while returning nearly empty HTML in the initial HTTP 200 response. Unlike evergreen headless Chromium renderers, AI grounding and search bots frequently fetch initial HTML without running deferred client JavaScript—and misconfigured streaming boundaries can leave key copy inside hidden templates.
Agency’s AI Crawler & No-JS Visibility Auditor (/evaluation/ai-crawler) simulates an unrendered bot fetch against your live URLs, measures exact visible server words inside <main>, flags empty client shells and accidental Disallow directives across all 7 major search and AI bots, and scores JSON-LD structured data and security headers on a clear 0–100 health scale.
7-Bot Robots.txt & Access Validation
Tests live robots.txt and meta crawler directives individually for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, and Googlebot.
No-JS Visible Word & <main> SSR Parity
Measures raw server-rendered words and primary <main> landmark copy before JavaScript executes so you catch CSR blindspots and React Suspense streaming traps early.
JSON-LD Structured Data Health Check
Verifies valid server-delivered Organization, WebSite, WebPage, Article, and BreadcrumbList JSON-LD blocks rather than JS-injected schema that bots never parse.
HTTPS, Canonicals & Security Header Check
Checks HTTP 200 latency, self-referencing rel="canonical", single H1 structure, HSTS, CSP, and X-Frame-Options headers alongside initial script tag bloat.
2. Step-by-Step: How to Verify AI Crawler & SSR Readiness
Confirm both search engines and AI citation crawlers can parse your pages in three steps:
Step 1: Test Live Robots.txt Rules Across All 7 Crawlers
Bot AccessInspect /evaluation/ai-crawler (or test any live URL free on /seo-checker) to see whether CDN WAF defaults or robots.txt wildcards block OpenAI, Anthropic, or Perplexity retrieval bots.
Step 2: Verify Initial HTML Word Count & <main> Copy Without JavaScript
SSR ParityCompare server-delivered visible copy against full rendered text to ensure pricing tables, core product explanations, and editorial articles render 100% server-side.
Flags hidden streaming containers, empty React root divs, and missing canonicals.
Step 3: Fix Blockers in /seo/actions or /website/auto-fixer
Code RepairUnlock the full 95-point deterministic technical action queue and deploy SSR/JSON-LD improvements directly across your Next.js, WordPress, or static stack.
3. Four AI Crawler & Rendering Workflows in Agency
Catch Client-Side Hydration Traps Before Launching Next.js or React Builds
Confirm key commercial landing pages deliver complete semantic headings, body copy, and JSON-LD in raw HTML rather than loading spinners.
Separate Training Crawlers from Live AI Search Citation Bots
Audit robots.txt nuance so you intentionally allow OAI-SearchBot, ChatGPT-User, and PerplexityBot for live citations while controlling training crawlers.
Detect Stealth 403s Blocking AI Retrieval Agents
Identify security rules that return 200 OK to desktop browsers while serving challenge pages or blocks to automated AI and SEO crawlers.
Turn Instant Public /seo-checker Scans into Full Project Audits
Run a 500ms check on /seo-checker to benchmark SSR health (0–100), then sync those live signals directly into /seo/actions inside your project.
4. Frequently Asked Questions
What is No-JS / SSR parity and why does it matter for GPTBot and PerplexityBot?
SSR parity measures how much of your visible page copy and semantic structure arrives in the initial server HTML response without running client JavaScript. AI retrieval bots frequently parse raw HTML without full browser JS execution, so missing SSR copy means missing AI citations.
Why does the checker inspect OAI-SearchBot separately from GPTBot?
OpenAI uses distinct user agents for model training (GPTBot), ChatGPT Search indexing (OAI-SearchBot), and live user-triggered browsing (ChatGPT-User). Checking all seven bots ensures you do not accidentally block ChatGPT or Perplexity search referrals.
What causes a 0 or low Semantic & Schema score even when the site looks fine in Chrome?
Sites built with client-side-only rendering, missing <main> landmarks, duplicate or missing H1s, or JSON-LD injected solely via client JavaScript fail initial-HTML inspection until server-rendered properly.
Can I run a free No-JS AI Crawler test without creating a project?
Yes. The public scanner at /seo-checker tests live HTTPS status, 7-bot robots.txt rules, SSR parity, canonicals, JSON-LD, and security headers immediately.
Related Solutions & Guides
- Track & Grow Your Brand Visibility Across AI Answer Engines (AEO)
- Audit Your Website Across 35 Quality Rater & E-E-A-T Dials
- Canonical Tag, Meta Robots & Search Indexation Conflict Auditor
- All Agency Solutions Directory (`/solutions`)
- What Can Agency Connect To? (Complete Connector Directory)
- Agency Security & Trust Center (AES-256-GCM Credential Vault)
Shaun Anderson
Shaun Anderson, AKA @hobo_web, is the Lead AI Architect behind Agency (thisisagency.ai). Shaun has over 25 years of experience in forensic SEO auditing. He was a primary investigator of the Google Search Leak of 2024, the writer of The Hobo SEO Quadrilogy, and a Keynote speaker at SMX Paris 2026.
Sign Up for Agency Updates
Subscribe for product updates, new AI workflows, and SEO research. Unsubscribe anytime.
