Parse
IndexPricing
Sign in
  1. Research
Research
Noindex, JS walls, and canonical mismatches on 378,392 AI-cited pages

Noindex, JS walls, and canonical mismatches on 378,392 AI-cited pages

Published Nov 20, 2026

On this page
  • Direct fetch failures
  • Robots directives
  • Engine directive split
  • Canonical hygiene
  • 17 source types
  • Category outliers
  • Latency distribution
  • Study scope
  • What to do next
  • How we measured
  • Sources
  • More like this
  • Direct fetch failures
  • Robots directives
  • Engine directive split
  • Canonical hygiene
  • 17 source types
  • Category outliers
  • Latency distribution
  • Study scope
  • What to do next
  • How we measured
  • Sources
  • More like this

Dimitry ApollonskyFounder, Parse

DataWhat reviewed means

Do technical SEO guardrails like noindex tags, cross-domain canonicals, JavaScript rendering walls, and bot protection stop AI search engines from citing a page?

We inspected November 1 to November 19, 2026 HTTP crawl checks across 378,392 unique pages actively cited by organic ChatGPT Search, Google AI Mode, and Perplexity answers, plus parsed HTML meta robots directives and canonical link tags across 329,519 HTML pages.

Across 378,392 actively AI-cited pages crawled in November 2026, 16.99% (64,291 pages) failed simple direct HTTP retrieval and required browser rendering, TLS impersonation, or proxies, including 15,030 explicit JS-blocked pages and 6,269 bot/CAPTCHA walls. Among 329,519 parsed HTML pages, AI engines still cited 2,392 pages tagged noindex, 5,332 tagged noarchive, 169 tagged noai/noimageai, and 7,533 (2.29%) carrying a cross-domain canonical tag.

16.99% of actively AI-cited pages fail clean direct HTTP retrieval

Across 378,392 actively AI-cited URLs inspected in November 2026, 280,395 succeeded cleanly on a direct HTTP fetch, while 64,291 (16.99%) failed direct HTTP retrieval and required browser rendering, TLS impersonation, or proxy paths. That friction includes 15,030 explicit JavaScript-required walls (3.97%), 6,269 bot or CAPTCHA walls (1.66%), and 213 soft-404 homepage redirects.

Perplexity's cited pages are cleanest to crawl directly at 83.88% (10.72% non-direct), compared with 74.93% on ChatGPT Search (16.66% non-direct) and 72.95% on Google AI Mode (17.49% non-direct, with the highest JS-blocked rate at 4.63%).

Failed simple direct HTTP fetch
16.99%

64,291 of 378,392 actively cited pages

Needed browser rendering, TLS impersonation, or proxy fetch

Explicit JS-blocked pages
15,030

3.97% JS-blocked · 1.66% bot/CAPTCHA walls (6,269)

Plus 213 soft-404 redirects to site homepages

Cited pages tagged noindex
2,392

0.73% of 329,519 HTML pages

Plus 5,332 noarchive (1.62%) and 169 noai/noimageai

Cross-domain canonical mismatches
7,533

2.29% cross-domain · 5.57% different same-domain path

34,515 pages (10.47%) lacked any canonical tag

Direct HTTP success and crawl barrier rates across AI engines

  • ChatGPT Search
  • Google AI Mode
  • Perplexity
  • Clean direct HTTP
  • ChatGPT Search74.93%228,958
  • Google AI Mode72.95%63,860
  • Perplexity83.88%4,892
  • Non-direct fetch needed
  • ChatGPT Search16.66%50,916
  • Google AI Mode17.49%15,309
  • Perplexity10.72%625
  • JS-blocked
  • ChatGPT Search3.63%11,106
  • Google AI Mode4.63%4,049
  • Perplexity2.88%168
  • Bot or CAPTCHA wall
  • ChatGPT Search1.74%5,329
  • Google AI Mode1.21%1,060
  • Perplexity1.35%79
  • 0%50%100%
378,392 unique cited pages inspected November 1 to November 19, 2026 (ChatGPT Search: 305,570; Google AI Mode: 87,538; Perplexity: 5,832).

Takeaway

Plain HTML scrapers fail on roughly one in six AI-cited URLs; Google AI Mode cites the highest share of JS-gated pages (4.63%), while Perplexity sticks closest to clean direct HTTP (83.88%).
  • Ahrefs: 67% of ChatGPT's top 1,000 citations are off-limits to marketers

AI engines still cite 2,392 pages tagged noindex, 5,332 tagged noarchive, and 169 tagged noai

HTML meta robots tags do not guarantee exclusion from AI search citations. Out of 329,519 parsed HTML pages actively cited in November 2026, 2,392 pages (0.73%) carried a meta noindex directive, 5,332 (1.62%) carried noarchive, and 169 carried noai or noimageai directives.

Another 213 cited pages acted as soft-404 redirects back to a root homepage on re-crawl. Because AI engines blend search-index snippets, live retrieval, and partner feeds, a page-level meta directive on its own can lag or get bypassed.

Cited pages carrying noindex
2,392

0.73% of 329,519 parsed HTML pages

Cited pages carrying noarchive
5,332

1.62% of 329,519 parsed HTML pages

Cited pages carrying noai/noimageai
169

70 on ChatGPT · 99 on Google AI Mode · 3 on Perplexity

Soft-404 homepage redirects
213

192 on ChatGPT · 21 on Google AI Mode · 3 on Perplexity

Takeaway

A meta noindex or noai tag inside HTML does not cleanly suppress active AI citations; verify enforcement at robots.txt and server access layers.
  • Do generative AI assistants respect robots.txt? Tracing web access beyond visible answers
  • Configuring robots.txt for AI crawlers

Perplexity cites noindex (1.23%) and noarchive (4.34%) at 2x to 3.2x ChatGPT rates, while ChatGPT leads canonical mismatches

The three engines diverge cleanly on HTML directive tolerance. Perplexity cited noindex pages at 1.23% (66 of 5,371 HTML pages), double Google AI Mode's 0.61% (490 of 79,824), and cited noarchive pages at 4.34% (233 pages), 3.2x ChatGPT Search's 1.37% (3,618 of 263,838).

On canonical tags, ChatGPT Search goes the other way: 2.62% of its cited HTML pages (6,912 URLs) carried a cross-domain canonical, more than 3.3x Google AI Mode's 0.8% (640 of 79,824), and 6.09% (16,071 URLs) pointed canonical to a different path on the same domain vs 2.97% on Google AI Mode.

Meta robots directives and canonical mismatches by AI engine

Missing canonical (%) · 3 results

  • ChatGPT Search10.6%
  • Google AI Mode9.5%
  • Perplexity4.88%
  • 0%5%10%15%
Explore the data (3 rows)
ChatGPT Search263,8380.71.3710.62.626.09
Google AI Mode79,8240.612.349.50.82.97
Perplexity5,3711.234.344.881.062.81
329,519 unique cited HTML pages inspected November 1 to November 19, 2026 (overlapping URLs counted per engine).

Takeaway

Google AI Mode stays closest to classic search canonicalization (0.8% cross-domain canonicals), while ChatGPT Search cites non-canonical URLs and syndicated mirrors far more often.

10.47% of cited HTML pages lack a canonical tag and 7.86% point canonical elsewhere

Across all 329,519 unique cited HTML pages, 34,515 pages (10.47%) lacked a rel='canonical' link tag entirely. Another 18,363 pages (5.57%) carried a canonical pointing to a different path on the same domain, and 7,533 (2.29%) pointed canonical to a completely different domain.

That means canonical mismatches do not stop ChatGPT Search from citing parameter variants, locale URLs, or syndicated pages. Clean self-referencing canonicals matter for attribution hygiene so your analytics and AI citations consolidate onto the URL you intend.

Canonical tag anomalies across 329,519 AI-cited HTML pages

  • Missing canonical tag10.47%34,515 pages
  • Same-domain different-path canonical5.57%18,363 pages
  • Cross-domain canonical tag2.29%7,533 pages
  • 0%5%10%15%
Parsed HTML canonical link tags vs normalized cited URLs, November 1 to November 19, 2026.

Takeaway

AI engines regularly cite non-canonical URLs; keep self-referencing canonicals consistent if you want clean URL-level reporting.

44.09% of cited ecommerce pages require non-direct fetch and 21.69% of social pages hit JS walls

Crawl resistance varies dramatically by publisher category across the 17 source types with at least 1,000 crawled pages. Ecommerce leads non-direct fetching at 44.09% of 44,021 cited pages (alongside 4.36% JS-blocked, 2.28% bot walls, and 2.76% noindex, the highest noindex rate of any category). Social platforms follow at 33.5% non-direct fetch and a category-high 21.69% explicit JS walls.

By contrast, company blogs (7.25% non-direct, 0.27% JS-blocked), SaaS marketing sites (8.24% non-direct, 0.68% JS-blocked), and editorial publishers (8.91% non-direct, 1.22% JS-blocked) are overwhelmingly clean to retrieve directly.

Crawl barriers and directive rates across 17 domain source types

Share of cited pages requiring browser/impersonated/proxy fetch, hitting explicit JS walls, or hitting bot/CAPTCHA walls.

Non-direct fetch (%) · Top 10 of 17

  • ecommerce44.09%
  • social33.5%
  • community27.06%
  • documentation17.61%
  • general16.81%
  • directory16.66%
  • news15.95%
  • academic15.4%
  • government14.3%
  • health12.37%
  • 0%20%40%60%
Explore the data (17 rows)
ecommerce44,02144.094.362.28
social10,02333.521.690.09
community1,85927.064.412.42
documentation30,72317.610.90.59
general85,69016.818.992.12
directory11,81116.662.573.14
news10,09015.958.712.66
academic1,16215.44.35.85
government13,65514.31.292.36
health4,72312.370.422.24
finance10,18011.753.532.47
education5,86311.211.042.64
developer6,14910.290.410.49
reference27,0299.920.511.53
editorial19,0908.911.221.95
saas89,9928.240.680.76
blog5,5307.250.270.33
Source categories with at least 1,000 unique crawled cited pages, November 1 to November 19, 2026.

Takeaway

Ecommerce (44.09% non-direct, 2.76% noindex) and social (21.69% JS-blocked) generate the heaviest technical audit traps, while SaaS and blog pages remain cleanest.

Academic sites tag 42.06% of cited pages noarchive, while docs (5.22%) lead cross-domain canonicals

Two source-type outliers explain much of the directive tail. Academic domains carry noarchive on 42.06% of parsed HTML pages (plus the highest bot-wall rate at 5.85%), followed by social (9.84%), news (6.96%), government (6.67%), and community sites (6.2%).

Meanwhile, documentation (5.22%) and developer sites (5.02%) post the highest cross-domain canonical rates, reflecting multi-domain product docs, versioned SDK mirrors, and localized sub-properties.

Share of cited HTML pages tagged noarchive across top source types

  • Academic42.06%
  • Social9.84%
  • News6.96%
  • Government6.67%
  • Community6.2%
  • Education3.73%
  • General web2.67%
  • Editorial1.92%
  • SaaS0.12%
  • Blog0.04%
  • 0%20%40%60%
Parsed HTML pages within each source category, November 1 to November 19, 2026.

Takeaway

Expect academic, news, and government citations to restrict archiving and direct scraping far more often than commercial SaaS or blog pages.
  • Cloudflare: From Googlebot to GPTBot

Non-direct browser and proxy crawls nearly triple p90 tail latency (20,107 ms vs 7,382 ms)

Whenever a page blocks clean HTTP requests and forces headless browser rendering, TLS impersonation, or proxy fallback, latency stretches sharply at the tail. Across 311,345 direct HTTP fetches, median latency was 5,731 ms and p90 was 7,382 ms.

Across the 64,291 non-direct fetches, median latency registered 3,589 ms while p90 jumped to 20,107 ms, nearly triple the direct HTTP tail. Meanwhile, the 2,756 pages fetched via structured API endpoints settled in 940 ms median and 1,652 ms at p90.

Fetch latency by retrieval path: median vs p90 (milliseconds)

  • Median latency
  • p90 latency
  • Direct HTTP
  • Median latency5,731 ms
  • p90 latency7,382 ms
  • Non-direct fetch
  • Median latency3,589 ms
  • p90 latency20,107 ms
  • API fetch
  • Median latency940 ms
  • p90 latency1,652 ms
  • 010,00020,00030,000
All 378,392 unique crawled pages across direct (311,345), non-direct (64,291), and API (2,756) crawl tiers, November 1 to November 19, 2026.

Takeaway

Heavy client-side rendering and WAF challenges push 90th-percentile fetch times past 20 seconds, slowing both retrieval pipelines and automated citation audits.

Study scope and counting rules

This study covers 378,392 unique pages cited by organic ChatGPT Search (305,570), Google AI Mode (87,538), and Perplexity (5,832) answers whose latest inspection ran between November 1 and November 19, 2026. Of those crawled URLs, 329,519 yielded parsed HTML metadata (263,838 on ChatGPT Search, 79,824 on Google AI Mode, 5,371 on Perplexity). Because an identical URL can appear on more than one engine, per-engine totals overlap above the unique count.

Fetch mode, JS-required blocks, bot/CAPTCHA walls, soft-404 homepage redirects, and latency reflect Parse's November HTTP fetch logs rather than what a first-party engine bot observed at the moment an answer was generated. Robots directives (noindex, noarchive, nofollow, nosnippet, noai/noimageai) are parsed from HTML meta tags and do not separately track robots.txt rules or HTTP X-Robots-Tag headers.

What to do with these numbers

Audit your own cited and target URLs with plain direct HTTP checks alongside browser rendering so you catch silent JS walls, bot-challenge interstitials, and soft-404 homepage redirects before assuming a citation is healthy. Nearly 44.09% of cited ecommerce pages and 21.69% of social pages trip non-direct or JS barriers that simple fetchers miss.

If your goal is to keep internal filters or staging URLs out of AI answers, don't rely on an HTML meta noindex tag alone: 2,392 noindex pages still surfaced in active citations. Pair server-level access control or explicit OAI-SearchBot rules in robots.txt with self-referencing canonicals across documentation and syndicated pages.

Technical citation hygiene checklist

Soft-404 redirects back to homepage213 cited URLsReturn 301 redirects to the true replacement guide instead of dumping retired slugs onto /.
HTML meta noindex / noai leaking into citations2,392 noindex; 169 noai pages citedBlock unwanted paths via robots.txt or auth rather than relying only on HTML meta directives.
Cross-domain or missing canonical tags10.47% missing; 5.22% cross-domain on docsEmit exact self-referencing canonical URLs on primary docs and canonicalize mirrors properly.
Client-side JS or bot walls blocking direct HTTP16.99% overall; 44.09% ecommercePre-render product, pricing, and spec HTML server-side and whitelist verified AI search bots.

Takeaway

Serve key product and docs pages as clean server-rendered HTML with explicit canonicals, and enforce exclusions upstream of HTML meta tags.
  • Audit cited pages and content accessibility
  • Inspect live cited URLs directly via Parse MCP

How we measured

We analyzed 378,392 unique AI-cited pages with November 1 to November 19, 2026 HTTP crawl checks across organic ChatGPT Search (305,570 pages), Google AI Mode (87,538 pages), and Perplexity (5,832 pages) answers. Of those 378,392 crawled pages, 329,519 yielded parsed HTML metadata covering meta robots directives and canonical link tags.

AI-cited pages failing clean direct HTTP fetch
16.99%

64,291 of 378,392 pages required browser rendering, impersonation, or proxies

Cited pages returning explicit JS-required blocks
15,030

3.97% of crawled pages, alongside 6,269 bot/CAPTCHA walls (1.66%)

Cited HTML pages tagged noindex
2,392

0.73% of 329,519 HTML pages, alongside 5,332 tagged noarchive (1.62%)

Cited HTML pages with cross-domain canonicals
7,533

2.29% pointed canonical to another domain; 34,515 (10.47%) had none

Get the data

Full study dataset (CSV)All 26 rows across engine crawl barriers, meta robots and canonicals, 17 domain source types, and fetch latency tiers.

Sources

  1. Ahrefs: 67% of ChatGPT's top 1,000 citations are off-limits to marketers (opens in a new tab) · accessed November 20, 2026
  2. Semrush: Blocked from AI Search in Site Audit (opens in a new tab) · accessed November 20, 2026
  3. Cloudflare: From Googlebot to GPTBot (opens in a new tab) · accessed November 20, 2026
  4. Do generative AI assistants respect robots.txt? Tracing web access beyond visible answers (opens in a new tab) · accessed November 20, 2026

More like this

How often do AI-cited pages block access?
Only 1.6% of checked AI-cited pages blocked direct access afterward; 97.5% returned a normal response.
How often do AI citations lead to broken pages?
Only 0.67% of 152,313 AI-cited pages returned 404 or 410, with ChatGPT Search breaking 4.7x more often than Google AI Mode.
Where do AI citation links lead?
Some citation links redirected or failed. Redirects were much more common than missing pages in our checks.
Do tracking parameters inflate AI citation counts?
Tracking links split one page into multiple URLs. Stripping common tracking tags cut ChatGPT Search's distinct cited-page count by 9.2%.
JSON-LD schema, author bylines, and content age on 329,519 AI-cited pages
70.86% of 329,519 AI-cited HTML pages carry JSON-LD schema, 30.86% expose an author byline, and dated pages show a 74.5-day median age.
HTML token bloat vs. Markdown twins and llms.txt across 300 AI-cited sites
46.00% of 300 top AI-cited sites serve /llms.txt and 15.00% honor Accept: text/markdown, cutting median page weight 25.41x across 62 paired pages while 19.67% fail plain HTTP fetch.
Why AI agents fail to read SaaS pricing and changelog pages
47.3% of 74 active top-SaaS /pricing endpoints fail plain-HTML delivery (14.9% WAF, 17.6% sales-gated, 9.5% JS-only, 5.4% partial JS) with 2.88% text-to-HTML.
For agents: this page as Markdown, https://parse.gl/research/crawler-barriers-noindex-and-canonicals-on-cited-pages.md. JSON, MCP · free for the last 90 days, checks from $0.01
Open in ChatGPT (opens in a new tab) Open in Claude (opens in a new tab)

This page

As Markdown: https://parse.gl/research/crawler-barriers-noindex-and-canonicals-on-cited-pages.md
As CSV: https://parse.gl/research/crawler-barriers-noindex-and-canonicals-on-cited-pages/dataset.csv
Reuse: Figures and tables on this page are free to republish with attribution: a link to https://parse.gl/research/crawler-barriers-noindex-and-canonicals-on-cited-pages beside them. A standard followed link (dofollow) is preferred.

Read and search

Any brand's shares and rivals, with 95% ranges: GET https://agent.parse.gl/v1/tools/check_brand?subject=%3Cbrand-domain%3E
Sample answers in a market: GET https://agent.parse.gl/v1/tools/search_answers?args=%7B%22filter%22%3A%7B%22market%22%3A%5B%22%3Cmkt_id%3E%22%5D%7D%7D
Cited pages in a market: GET https://agent.parse.gl/v1/tools/search_cited_pages?args=%7B%22filter%22%3A%7B%22market%22%3A%5B%22%3Cmkt_id%3E%22%5D%7D%7D
Search the index: GET https://parse.gl/api/public/search?q=<words>

Connect

Connect an assistant: https://agent.parse.gl/mcp

An agent can save work before signup: POST https://agent.parse.gl/v1/workspaces

What it costs

No account: Public figures and up to 5 examples per list, about 120 brands a day per address.
Free account: $5 in credit, your brand's first check (20 buyer questions) and a free weekly check.
After your first top-up: Exports and deeper search results. The first top-up starts at $5.
Live checks: $0.01 per question per engine ($0.05 on Gemini and Claude).
Full public answers and pages: 25 free per day, then $0.005 each.
Public answers older than 90 days: $0.001 each the first time your workspace counts or lists one, then yours.

Prices: https://parse.gl/pricing.md

Map of the site: https://parse.gl/llms.txt

Inspect crawlability, directives, and canonicals across the pages AI cites

Start free in your browser.
Parse

Index

  • Brands
  • Markets
  • Topics
  • How we measure

For agents

  • Docs
  • Connect
  • Workflows
  • llms.txt

Parse

  • Use cases
  • Pricing
  • Research
  • Blog
  • Compare

Company

  • About
  • Contact
  • Corrections

© 2026 Parse, Inc. All rights reserved.

Privacy PolicyTerms of Service