Agency_

Solutions/AI Crawlers & Technical SSR
No-JS SSR & 7-Bot Audit

Check What GPTBot, ClaudeBot & PerplexityBot Actually See Without JavaScript

Most AI retrieval bots—including GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot—do not execute heavy client-side JavaScript bundles. If your pages rely on client hydration or hidden React streaming fallbacks, answer engines read an empty shell. Agency verifies server-rendered HTML parity, bot access rules, JSON-LD entities, and security headers across 7 crawlers.

By Shaun Anderson•AI Specialists: James AI & Shaun AI•Route: /evaluation/ai-crawler
Native Multi-Agent Workspace Engine
AES-256-GCM Encrypted VaultManaged by James AI & Shaun AI

Audit No-JS Server Rendering & 7 AI Bot Rules

Inspect raw non-JS word counts, <main> landmark parity, robots.txt bot rules, JSON-LD blocks, and security headers inside /evaluation/ai-crawler—or test any URL publicly on /seo-checker.

1Enter your URL in the free /seo-checker or run a full Quality Rater scan in /rater.
2Inspect the 7-bot access matrix (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Googlebot).
3Review No-JS SSR word parity and fix any client-only rendering or robots.txt blocks.
⚡Included in Every Rater Scan & Free Public Snapshot at /seo-checker
Start in Under 60 Seconds
Choose your plan & launch AI Crawler & SSR Visibility Auditor inside your workspace
View Plans & Launch AI Crawler & SSR Visibility AuditorSign In & Open AI Crawler & SSR Visibility Auditor →

1. Why Answer Engines & AI Search Bots Silently Miss Client-Rendered Websites

Modern React, Vue, and Next.js applications often score cleanly in desktop browsers while returning nearly empty HTML in the initial HTTP 200 response. Unlike evergreen headless Chromium renderers, AI grounding and search bots frequently fetch initial HTML without running deferred client JavaScript—and misconfigured streaming boundaries can leave key copy inside hidden templates.

Agency’s AI Crawler & No-JS Visibility Auditor (/evaluation/ai-crawler) simulates an unrendered bot fetch against your live URLs, measures exact visible server words inside <main>, flags empty client shells and accidental Disallow directives across all 7 major search and AI bots, and scores JSON-LD structured data and security headers on a clear 0–100 health scale.

7-Bot Robots.txt & Access Validation

Tests live robots.txt and meta crawler directives individually for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, and Googlebot.

No-JS Visible Word & <main> SSR Parity

Measures raw server-rendered words and primary <main> landmark copy before JavaScript executes so you catch CSR blindspots and React Suspense streaming traps early.

JSON-LD Structured Data Health Check

Verifies valid server-delivered Organization, WebSite, WebPage, Article, and BreadcrumbList JSON-LD blocks rather than JS-injected schema that bots never parse.

HTTPS, Canonicals & Security Header Check

Checks HTTP 200 latency, self-referencing rel="canonical", single H1 structure, HSTS, CSP, and X-Frame-Options headers alongside initial script tag bloat.


2. Step-by-Step: How to Verify AI Crawler & SSR Readiness

Confirm both search engines and AI citation crawlers can parse your pages in three steps:

Step 1: Test Live Robots.txt Rules Across All 7 Crawlers

Bot Access

Inspect /evaluation/ai-crawler (or test any live URL free on /seo-checker) to see whether CDN WAF defaults or robots.txt wildcards block OpenAI, Anthropic, or Perplexity retrieval bots.

Step 2: Verify Initial HTML Word Count & <main> Copy Without JavaScript

SSR Parity

Compare server-delivered visible copy against full rendered text to ensure pricing tables, core product explanations, and editorial articles render 100% server-side.

Flags hidden streaming containers, empty React root divs, and missing canonicals.

Step 3: Fix Blockers in /seo/actions or /website/auto-fixer

Code Repair

Unlock the full 95-point deterministic technical action queue and deploy SSR/JSON-LD improvements directly across your Next.js, WordPress, or static stack.


3. Four AI Crawler & Rendering Workflows in Agency

Workflow 1 • Pre-Launch JS Framework Audit

Catch Client-Side Hydration Traps Before Launching Next.js or React Builds

Confirm key commercial landing pages deliver complete semantic headings, body copy, and JSON-LD in raw HTML rather than loading spinners.

Workflow 2 • Selective AI Bot Governance

Separate Training Crawlers from Live AI Search Citation Bots

Audit robots.txt nuance so you intentionally allow OAI-SearchBot, ChatGPT-User, and PerplexityBot for live citations while controlling training crawlers.

Workflow 3 • WAF & CDN Whitelisting

Detect Stealth 403s Blocking AI Retrieval Agents

Identify security rules that return 200 OK to desktop browsers while serving challenge pages or blocks to automated AI and SEO crawlers.

Workflow 4 • Free Lead-Magnet Snapshot to Workspace Sync

Turn Instant Public /seo-checker Scans into Full Project Audits

Run a 500ms check on /seo-checker to benchmark SSR health (0–100), then sync those live signals directly into /seo/actions inside your project.


4. Frequently Asked Questions

What is No-JS / SSR parity and why does it matter for GPTBot and PerplexityBot?

SSR parity measures how much of your visible page copy and semantic structure arrives in the initial server HTML response without running client JavaScript. AI retrieval bots frequently parse raw HTML without full browser JS execution, so missing SSR copy means missing AI citations.

Why does the checker inspect OAI-SearchBot separately from GPTBot?

OpenAI uses distinct user agents for model training (GPTBot), ChatGPT Search indexing (OAI-SearchBot), and live user-triggered browsing (ChatGPT-User). Checking all seven bots ensures you do not accidentally block ChatGPT or Perplexity search referrals.

What causes a 0 or low Semantic & Schema score even when the site looks fine in Chrome?

Sites built with client-side-only rendering, missing <main> landmarks, duplicate or missing H1s, or JSON-LD injected solely via client JavaScript fail initial-HTML inspection until server-rendered properly.

Can I run a free No-JS AI Crawler test without creating a project?

Yes. The public scanner at /seo-checker tests live HTTPS status, 7-bot robots.txt rules, SSR parity, canonicals, JSON-LD, and security headers immediately.


Related Solutions & Guides

Shaun Anderson

Shaun Anderson

Shaun Anderson, AKA @hobo_web, is the Lead AI Architect behind Agency (thisisagency.ai). Shaun has over 25 years of experience in forensic SEO auditing. He was a primary investigator of the Google Search Leak of 2024, the writer of The Hobo SEO Quadrilogy, and a Keynote speaker at SMX Paris 2026.

Disclosure: I use generative AI when specifically writing about my own experiences, ideas, stories, concepts, tools, tool documentation or research, across the Hobo Web and Agency platforms. My tool of choice for this process is Agency—the publishing architecture I created and licensed to Candlestickoperations Ltd—using Google Gemini Pro (2.5 / 3) Deep Research and Reasoning and later models (as default). I have been writing about Google Search for 2 decades. This content is a direct result of cutting-edge HITL AI content production, using a process I designed myself and described in my 2025 ebook, Strategic AI SEO. Also note: This content has not been created to “pass off” as human, or “write as Shaun Anderson” (me). It is raw, factually correct content. Agency can produce content with certain styles, but it is not my aim to fool anyone with it. This is the state of AI content in 2026, an example of the default Agency creative output. All content was conceived, directed, edited, fact-checked and verified as correct by me (and is under constant development). This article is amongst the first “beta-test” of Agency, the autonomous AI-powered Agency as presented in my March 2026 SMX Paris keynote presentation. Edited by Shaun Anderson AKA Hobo. Corrections welcome. Read my AI policy.

Sign Up for Agency Updates

Subscribe for product updates, new AI workflows, and SEO research. Unsubscribe anytime.

Agency (Thisisagency.ai) is in public beta. Agency is an AI workplace simulation. Agency was conceived, bootstrapped, and architected solo by Shaun Anderson AKA Hobo (@hobo_web). “This Is Agency” and “Agency” are trading names of Candlestickoperations Ltd. Registered in England and Wales, Company No. 17503419. Registered Office: 3rd Floor, 86-90 Paul Street, London, England, EC2A 4NE.
All team members are AI entities powered by Google Gemini technology (by default). When API are available, Agency is a multi-provider, multi-model, multi-modal, multi agent system.