# Noindex, JS walls, and canonical mismatches on 378,392 AI-cited pages

> Across 378,392 actively AI-cited pages crawled in November 2026, 16.99% (64,291 pages) failed simple direct HTTP retrieval and required browser rendering, TLS impersonation, or proxies, including 15,030 explicit JS-blocked pages and 6,269 bot/CAPTCHA walls. Among 329,519 parsed HTML pages, AI engines still cited 2,392 pages tagged noindex, 5,332 tagged noarchive, 169 tagged noai/noimageai, and 7,533 (2.29%) carrying a cross-domain canonical tag.

HTML: https://parse.gl/research/crawler-barriers-noindex-and-canonicals-on-cited-pages · Site index: https://parse.gl/llms.txt · Published: 2026-11-20

Do technical SEO guardrails like noindex tags, cross-domain canonicals, JavaScript rendering walls, and bot protection stop AI search engines from citing a page?

We inspected November 1 to November 19, 2026 HTTP crawl checks across 378,392 unique pages actively cited by organic ChatGPT Search, Google AI Mode, and Perplexity answers, plus parsed HTML meta robots directives and canonical link tags across 329,519 HTML pages.

## Key figures

- **AI-cited pages failing clean direct HTTP fetch:** 16.99% (64,291 of 378,392 pages required browser rendering, impersonation, or proxies)
- **Cited pages returning explicit JS-required blocks:** 15,030 (3.97% of crawled pages, alongside 6,269 bot/CAPTCHA walls (1.66%))
- **Cited HTML pages tagged noindex:** 2,392 (0.73% of 329,519 HTML pages, alongside 5,332 tagged noarchive (1.62%))
- **Cited HTML pages with cross-domain canonicals:** 7,533 (2.29% pointed canonical to another domain; 34,515 (10.47%) had none)

## 16.99% of actively AI-cited pages fail clean direct HTTP retrieval

Across 378,392 actively AI-cited URLs inspected in November 2026, 280,395 succeeded cleanly on a direct HTTP fetch, while 64,291 (16.99%) failed direct HTTP retrieval and required browser rendering, TLS impersonation, or proxy paths. That friction includes 15,030 explicit JavaScript-required walls (3.97%), 6,269 bot or CAPTCHA walls (1.66%), and 213 soft-404 homepage redirects.

Perplexity's cited pages are cleanest to crawl directly at 83.88% (10.72% non-direct), compared with 74.93% on ChatGPT Search (16.66% non-direct) and 72.95% on Google AI Mode (17.49% non-direct, with the highest JS-blocked rate at 4.63%).

**Direct HTTP success and crawl barrier rates across AI engines**

| Item | Value |
| --- | --- |
| Clean direct HTTP: ChatGPT Search | 74.93% (228,958) |
| Clean direct HTTP: Google AI Mode | 72.95% (63,860) |
| Clean direct HTTP: Perplexity | 83.88% (4,892) |
| Non-direct fetch needed: ChatGPT Search | 16.66% (50,916) |
| Non-direct fetch needed: Google AI Mode | 17.49% (15,309) |
| Non-direct fetch needed: Perplexity | 10.72% (625) |
| JS-blocked: ChatGPT Search | 3.63% (11,106) |
| JS-blocked: Google AI Mode | 4.63% (4,049) |
| JS-blocked: Perplexity | 2.88% (168) |
| Bot or CAPTCHA wall: ChatGPT Search | 1.74% (5,329) |
| Bot or CAPTCHA wall: Google AI Mode | 1.21% (1,060) |
| Bot or CAPTCHA wall: Perplexity | 1.35% (79) |

_378,392 unique cited pages inspected November 1 to November 19, 2026 (ChatGPT Search: 305,570; Google AI Mode: 87,538; Perplexity: 5,832)._

> Plain HTML scrapers fail on roughly one in six AI-cited URLs; Google AI Mode cites the highest share of JS-gated pages (4.63%), while Perplexity sticks closest to clean direct HTTP (83.88%).

## AI engines still cite 2,392 pages tagged noindex, 5,332 tagged noarchive, and 169 tagged noai

HTML meta robots tags do not guarantee exclusion from AI search citations. Out of 329,519 parsed HTML pages actively cited in November 2026, 2,392 pages (0.73%) carried a meta noindex directive, 5,332 (1.62%) carried noarchive, and 169 carried noai or noimageai directives.

Another 213 cited pages acted as soft-404 redirects back to a root homepage on re-crawl. Because AI engines blend search-index snippets, live retrieval, and partner feeds, a page-level meta directive on its own can lag or get bypassed.

- **Cited pages carrying noindex:** 2,392
- **Cited pages carrying noarchive:** 5,332
- **Cited pages carrying noai/noimageai:** 169
- **Soft-404 homepage redirects:** 213

> A meta noindex or noai tag inside HTML does not cleanly suppress active AI citations; verify enforcement at robots.txt and server access layers.

Related: [Configuring robots.txt for AI crawlers](https://parse.gl/blog/robots-txt-ai-crawlers-configuration)

## Perplexity cites noindex (1.23%) and noarchive (4.34%) at 2x to 3.2x ChatGPT rates, while ChatGPT leads canonical mismatches

The three engines diverge cleanly on HTML directive tolerance. Perplexity cited noindex pages at 1.23% (66 of 5,371 HTML pages), double Google AI Mode's 0.61% (490 of 79,824), and cited noarchive pages at 4.34% (233 pages), 3.2x ChatGPT Search's 1.37% (3,618 of 263,838).

On canonical tags, ChatGPT Search goes the other way: 2.62% of its cited HTML pages (6,912 URLs) carried a cross-domain canonical, more than 3.3x Google AI Mode's 0.8% (640 of 79,824), and 6.09% (16,071 URLs) pointed canonical to a different path on the same domain vs 2.97% on Google AI Mode.

**Meta robots directives and canonical mismatches by AI engine**

| Engine | HTML pages | noindex (%) | noarchive (%) | Missing canonical (%) | Cross-domain canonical (%) | Same-domain diff path (%) |
| --- | --- | --- | --- | --- | --- | --- |
| ChatGPT Search | 263838 | 0.7 | 1.37 | 10.6 | 2.62 | 6.09 |
| Google AI Mode | 79824 | 0.61 | 2.34 | 9.5 | 0.8 | 2.97 |
| Perplexity | 5371 | 1.23 | 4.34 | 4.88 | 1.06 | 2.81 |

_329,519 unique cited HTML pages inspected November 1 to November 19, 2026 (overlapping URLs counted per engine)._

> Google AI Mode stays closest to classic search canonicalization (0.8% cross-domain canonicals), while ChatGPT Search cites non-canonical URLs and syndicated mirrors far more often.

## 10.47% of cited HTML pages lack a canonical tag and 7.86% point canonical elsewhere

Across all 329,519 unique cited HTML pages, 34,515 pages (10.47%) lacked a rel='canonical' link tag entirely. Another 18,363 pages (5.57%) carried a canonical pointing to a different path on the same domain, and 7,533 (2.29%) pointed canonical to a completely different domain.

That means canonical mismatches do not stop ChatGPT Search from citing parameter variants, locale URLs, or syndicated pages. Clean self-referencing canonicals matter for attribution hygiene so your analytics and AI citations consolidate onto the URL you intend.

**Canonical tag anomalies across 329,519 AI-cited HTML pages**

| Item | Value |
| --- | --- |
| Missing canonical tag | 10.47% (34,515 pages) |
| Same-domain different-path canonical | 5.57% (18,363 pages) |
| Cross-domain canonical tag | 2.29% (7,533 pages) |

_Parsed HTML canonical link tags vs normalized cited URLs, November 1 to November 19, 2026._

> AI engines regularly cite non-canonical URLs; keep self-referencing canonicals consistent if you want clean URL-level reporting.

## 44.09% of cited ecommerce pages require non-direct fetch and 21.69% of social pages hit JS walls

Crawl resistance varies dramatically by publisher category across the 17 source types with at least 1,000 crawled pages. Ecommerce leads non-direct fetching at 44.09% of 44,021 cited pages (alongside 4.36% JS-blocked, 2.28% bot walls, and 2.76% noindex, the highest noindex rate of any category). Social platforms follow at 33.5% non-direct fetch and a category-high 21.69% explicit JS walls.

By contrast, company blogs (7.25% non-direct, 0.27% JS-blocked), SaaS marketing sites (8.24% non-direct, 0.68% JS-blocked), and editorial publishers (8.91% non-direct, 1.22% JS-blocked) are overwhelmingly clean to retrieve directly.

**Crawl barriers and directive rates across 17 domain source types**

| Source type | Crawled pages | Non-direct fetch (%) | JS-blocked (%) | Bot/CAPTCHA (%) |
| --- | --- | --- | --- | --- |
| ecommerce | 44021 | 44.09 | 4.36 | 2.28 |
| social | 10023 | 33.5 | 21.69 | 0.09 |
| community | 1859 | 27.06 | 4.41 | 2.42 |
| documentation | 30723 | 17.61 | 0.9 | 0.59 |
| general | 85690 | 16.81 | 8.99 | 2.12 |
| directory | 11811 | 16.66 | 2.57 | 3.14 |
| news | 10090 | 15.95 | 8.71 | 2.66 |
| academic | 1162 | 15.4 | 4.3 | 5.85 |
| government | 13655 | 14.3 | 1.29 | 2.36 |
| health | 4723 | 12.37 | 0.42 | 2.24 |
| finance | 10180 | 11.75 | 3.53 | 2.47 |
| education | 5863 | 11.21 | 1.04 | 2.64 |
| developer | 6149 | 10.29 | 0.41 | 0.49 |
| reference | 27029 | 9.92 | 0.51 | 1.53 |
| editorial | 19090 | 8.91 | 1.22 | 1.95 |
| saas | 89992 | 8.24 | 0.68 | 0.76 |
| blog | 5530 | 7.25 | 0.27 | 0.33 |

_Source categories with at least 1,000 unique crawled cited pages, November 1 to November 19, 2026._

> Ecommerce (44.09% non-direct, 2.76% noindex) and social (21.69% JS-blocked) generate the heaviest technical audit traps, while SaaS and blog pages remain cleanest.

## Academic sites tag 42.06% of cited pages noarchive, while docs (5.22%) lead cross-domain canonicals

Two source-type outliers explain much of the directive tail. Academic domains carry noarchive on 42.06% of parsed HTML pages (plus the highest bot-wall rate at 5.85%), followed by social (9.84%), news (6.96%), government (6.67%), and community sites (6.2%).

Meanwhile, documentation (5.22%) and developer sites (5.02%) post the highest cross-domain canonical rates, reflecting multi-domain product docs, versioned SDK mirrors, and localized sub-properties.

**Share of cited HTML pages tagged noarchive across top source types**

| Item | Value |
| --- | --- |
| Academic | 42.06% |
| Social | 9.84% |
| News | 6.96% |
| Government | 6.67% |
| Community | 6.2% |
| Education | 3.73% |
| General web | 2.67% |
| Editorial | 1.92% |
| SaaS | 0.12% |
| Blog | 0.04% |

_Parsed HTML pages within each source category, November 1 to November 19, 2026._

> Expect academic, news, and government citations to restrict archiving and direct scraping far more often than commercial SaaS or blog pages.

## Non-direct browser and proxy crawls nearly triple p90 tail latency (20,107 ms vs 7,382 ms)

Whenever a page blocks clean HTTP requests and forces headless browser rendering, TLS impersonation, or proxy fallback, latency stretches sharply at the tail. Across 311,345 direct HTTP fetches, median latency was 5,731 ms and p90 was 7,382 ms.

Across the 64,291 non-direct fetches, median latency registered 3,589 ms while p90 jumped to 20,107 ms, nearly triple the direct HTTP tail. Meanwhile, the 2,756 pages fetched via structured API endpoints settled in 940 ms median and 1,652 ms at p90.

**Fetch latency by retrieval path: median vs p90 (milliseconds)**

| Item | Value |
| --- | --- |
| Direct HTTP: Median latency | 5,731 ms |
| Direct HTTP: p90 latency | 7,382 ms |
| Non-direct fetch: Median latency | 3,589 ms |
| Non-direct fetch: p90 latency | 20,107 ms |
| API fetch: Median latency | 940 ms |
| API fetch: p90 latency | 1,652 ms |

_All 378,392 unique crawled pages across direct (311,345), non-direct (64,291), and API (2,756) crawl tiers, November 1 to November 19, 2026._

> Heavy client-side rendering and WAF challenges push 90th-percentile fetch times past 20 seconds, slowing both retrieval pipelines and automated citation audits.

## Study scope and counting rules

This study covers 378,392 unique pages cited by organic ChatGPT Search (305,570), Google AI Mode (87,538), and Perplexity (5,832) answers whose latest inspection ran between November 1 and November 19, 2026. Of those crawled URLs, 329,519 yielded parsed HTML metadata (263,838 on ChatGPT Search, 79,824 on Google AI Mode, 5,371 on Perplexity). Because an identical URL can appear on more than one engine, per-engine totals overlap above the unique count.

Fetch mode, JS-required blocks, bot/CAPTCHA walls, soft-404 homepage redirects, and latency reflect Parse's November HTTP fetch logs rather than what a first-party engine bot observed at the moment an answer was generated. Robots directives (noindex, noarchive, nofollow, nosnippet, noai/noimageai) are parsed from HTML meta tags and do not separately track robots.txt rules or HTTP X-Robots-Tag headers.

## What to do with these numbers

Audit your own cited and target URLs with plain direct HTTP checks alongside browser rendering so you catch silent JS walls, bot-challenge interstitials, and soft-404 homepage redirects before assuming a citation is healthy. Nearly 44.09% of cited ecommerce pages and 21.69% of social pages trip non-direct or JS barriers that simple fetchers miss.

If your goal is to keep internal filters or staging URLs out of AI answers, don't rely on an HTML meta noindex tag alone: 2,392 noindex pages still surfaced in active citations. Pair server-level access control or explicit OAI-SearchBot rules in robots.txt with self-referencing canonicals across documentation and syndicated pages.

**Technical citation hygiene checklist**

| Observed issue | Study benchmark | Engineering action |
| --- | --- | --- |
| Client-side JS or bot walls blocking direct HTTP | 16.99% overall; 44.09% ecommerce | Pre-render product, pricing, and spec HTML server-side and whitelist verified AI search bots. |
| HTML meta noindex / noai leaking into citations | 2,392 noindex; 169 noai pages cited | Block unwanted paths via robots.txt or auth rather than relying only on HTML meta directives. |
| Cross-domain or missing canonical tags | 10.47% missing; 5.22% cross-domain on docs | Emit exact self-referencing canonical URLs on primary docs and canonicalize mirrors properly. |
| Soft-404 redirects back to homepage | 213 cited URLs | Return 301 redirects to the true replacement guide instead of dumping retired slugs onto /. |

> Serve key product and docs pages as clean server-rendered HTML with explicit canonicals, and enforce exclusions upstream of HTML meta tags.

Related: [Audit cited pages and content accessibility](https://parse.gl/use-cases/plan-content) · [Inspect live cited URLs directly via Parse MCP](https://parse.gl/mcp)

## Method

We analyzed 378,392 unique AI-cited pages with November 1 to November 19, 2026 HTTP crawl checks across organic ChatGPT Search (305,570 pages), Google AI Mode (87,538 pages), and Perplexity (5,832 pages) answers. Of those 378,392 crawled pages, 329,519 yielded parsed HTML metadata covering meta robots directives and canonical link tags.

## Data

- [Full study dataset (CSV)](https://parse.gl/research/crawler-barriers-noindex-and-canonicals-on-cited-pages/dataset.csv): All 26 rows across engine crawl barriers, meta robots and canonicals, 17 domain source types, and fetch latency tiers.

## Sources

- [Ahrefs: 67% of ChatGPT's top 1,000 citations are off-limits to marketers](https://ahrefs.com/blog/chatgpts-most-cited-pages/)
- [Semrush: Blocked from AI Search in Site Audit](https://www.semrush.com/kb/1571-blocked-from-ai-search-site-audit)
- [Cloudflare: From Googlebot to GPTBot](https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/)
- [Do generative AI assistants respect robots.txt? Tracing web access beyond visible answers](https://arxiv.org/abs/2607.14447)

## For agents

- [Study dataset (CSV)](https://parse.gl/research/crawler-barriers-noindex-and-canonicals-on-cited-pages/dataset.csv)
- Any brand's shares and rivals, with 95% ranges: `https://agent.parse.gl/v1/tools/check_brand?subject=%3Cbrand-domain%3E`
- Sample answers in a market: `https://agent.parse.gl/v1/tools/search_answers?args=%7B%22filter%22%3A%7B%22market%22%3A%5B%22%3Cmkt_id%3E%22%5D%7D%7D`
- Cited pages in a market: `https://agent.parse.gl/v1/tools/search_cited_pages?args=%7B%22filter%22%3A%7B%22market%22%3A%5B%22%3Cmkt_id%3E%22%5D%7D%7D`
- Search the index: `https://parse.gl/api/public/search?q=<words>`

Reuse: Figures and tables on this page are free to republish with attribution: a link to https://parse.gl/research/crawler-barriers-noindex-and-canonicals-on-cited-pages beside them. A standard followed link (dofollow) is preferred.

- No account: Public figures and up to 5 examples per list, about 120 brands a day per address.
- Free account: $5 in credit, your brand's first check (20 buyer questions) and a free weekly check.
- After your first top-up: Exports and deeper search results. The first top-up starts at $5.
- Live checks: $0.01 per question per engine ($0.05 on Gemini and Claude).
- Full public answers and pages: 25 free per day, then $0.005 each.
- Public answers older than 90 days: $0.001 each the first time your workspace counts or lists one, then yours.
- An agent can save work before signup: `POST https://agent.parse.gl/v1/workspaces`
- Connect an assistant: `https://agent.parse.gl/mcp`

[Prices](https://parse.gl/pricing.md) · [Map of the site](https://parse.gl/llms.txt)
