Parse
IndexPricing
Sign in
  1. Research
Research
HTML token bloat vs. Markdown twins and llms.txt across 300 AI-cited sites

HTML token bloat vs. Markdown twins and llms.txt across 300 AI-cited sites

Published Nov 20, 2026

On this page
  • llms.txt and Markdown twins
  • Category splits & Shopify UCP
  • MCP & .well-known gap
  • HTML token bloat
  • Native Markdown savings
  • Plain HTTP failures
  • Chromium JS recovery
  • WAF blocks vs robots.txt
  • Study scope
  • What to do next
  • How we measured
  • Sources
  • More like this
  • llms.txt and Markdown twins
  • Category splits & Shopify UCP
  • MCP & .well-known gap
  • HTML token bloat
  • Native Markdown savings
  • Plain HTTP failures
  • Chromium JS recovery
  • WAF blocks vs robots.txt
  • Study scope
  • What to do next
  • How we measured
  • Sources
  • More like this

Dimitry ApollonskyFounder, Parse

DataWhat reviewed means

How much of the web that AI search engines cite is readable without headless browsers or HTML boilerplate stripping, and how far have /llms.txt, Markdown twins, and agent discovery endpoints spread across top-cited domains?

We extracted the 300 most-cited hosts and each host's top-cited URL from organic ChatGPT Search, Google AI Mode, and Perplexity answers recorded September 1 through November 19, 2026 (90 Developer/Docs, 90 B2B SaaS, 60 Editorial/Media, and 60 E-commerce), then probed every host live on November 20, 2026 for /llms.txt, /llms-full.txt, Accept: text/markdown responses, .md twins, /.well-known/mcp manifests, robots.txt AI rules, OAI-SearchBot access, HTML token bloat, and headless Chromium rendering across 49 browser-checked pages alongside 13 named reference twin pages.

Across the 300 most-cited Developer/Docs, B2B SaaS, Editorial/Media, and E-commerce domains in Parse's September to November 2026 index, 46.00% serve a real /llms.txt and 15.00% honor Accept: text/markdown content negotiation. Serving native Markdown cuts median page weight 25.41x across 62 paired cited pages, while 19.67% of top-cited URLs fail plain HTTP retrieval and 23.67% reject requests using the OAI-SearchBot User-Agent from a non-OpenAI benchmark IP.

46.00% of top AI-cited hosts serve /llms.txt, while 15.00% honor Accept: text/markdown

Across the 300 most-cited web hosts in Parse's index, 138 (46.00%) serve a valid, non-HTML /llms.txt index at the cited host, and 151 (50.33%) serve one on either the cited host or its parent root domain. Another 49 hosts (16.33%) publish a consolidated /llms-full.txt file.

Page-level Markdown delivery lags directory-level /llms.txt adoption by roughly 3-to-1. Exactly 45 of 300 hosts (15.00%) honor Accept: text/markdown HTTP content negotiation (31 on the homepage and 45 on the top-cited URL), 46 of 300 (15.33%) expose a live .md URL twin on their top-cited page, 63 of 300 (21.00%) support at least one of those two mechanisms (28 support both), and 28 of 300 (9.33%) declare HTML alternate Markdown link tags in the document head.

Serve /llms.txt on the cited host
46.00%

138 of 300 top-cited sites (151 including root domains)

62.22% B2B SaaS · 54.44% Developer/Docs · 30.00% E-commerce

Median HTML-to-Markdown token savings
25.41x

62 native Markdown pairs · 24.97x HTML bloat across 238 pages

Median 86,772.5 raw HTML tokens vs 2,850.5 visible text tokens

Honor Accept: text/markdown headers
15.00%

45 of 300 sites · 15.33% (46) serve .md URL twins

2.67% (8 of 300) publish /.well-known/mcp discovery JSON

Refuse benchmark OAI-SearchBot requests
23.67%

71 of 300 cited URLs (73 homepages) from test IP

vs 2.26% (6 of 265) blocking OAI-SearchBot in robots.txt

Machine-readable web standard adoption across 300 top AI-cited hosts (%)

  • /llms.txt (host)
  • /llms-full.txt
  • Accept text/markdown
  • .md URL twin
  • Developer/Docs
  • /llms.txt (host)54.44%49 of 90
  • /llms-full.txt25.56%23 of 90
  • Accept text/markdown28.89%26 of 90
  • .md URL twin34.44%31 of 90
  • B2B SaaS
  • /llms.txt (host)62.22%56 of 90
  • /llms-full.txt16.67%15 of 90
  • Accept text/markdown17.78%16 of 90
  • .md URL twin12.22%11 of 90
  • E-commerce
  • /llms.txt (host)30.00%18 of 60
  • /llms-full.txt13.33%8 of 60
  • Accept text/markdown3.33%2 of 60
  • .md URL twin0.00%0 of 60
  • Editorial/Media
  • /llms.txt (host)25.00%15 of 60
  • /llms-full.txt5.00%3 of 60
  • Accept text/markdown1.67%1 of 60
  • .md URL twin6.67%4 of 60
  • 0%20%40%60%80%
Live HTTP probes run November 20, 2026 across 90 Developer/Docs, 90 B2B SaaS, 60 Editorial/Media, and 60 E-commerce sites.

Takeaway

Nearly half of top AI-cited websites now expose /llms.txt, while only one in seven deliver native Markdown per page.
  • The /llms.txt specification (llmstxt.org)
  • Cloudflare: Introducing Markdown for Agents
  • How to make your site readable for AI agents

B2B SaaS leads /llms.txt at 62.22%, while Shopify templates push E-commerce ahead of Editorial/Media

Adoption splits sharply by site category. B2B SaaS leads cited-host /llms.txt adoption at 62.22% (56 of 90 hosts). Developer/Docs posts 54.44% at the cited host (49 of 90) and rises to 67.78% (61 of 90) when parent root domains are checked, because several support and docs subdomains inherit a root /llms.txt. Developer/Docs also leads page-level Markdown twins at 34.44% (31 of 90) and Accept: text/markdown at 28.89% (26 of 90).

E-commerce /llms.txt adoption reaches 30.00% (18 of 60), beating Editorial/Media at 25.00% (15 of 60). Platform defaults explain why: 7 Shopify merchants in the 60-store cohort (sunhomesaunas.com, americantall.com, christydawn.com, everlane.com, girlfriend.com, mudjeans.com, and matethelabel.com) expose Shopify's default Universal Commerce Protocol (UCP) Shop Skill /llms.txt. Outside those storefronts, E-commerce has 0.00% .md twins (0 of 60) and 3.33% Accept: text/markdown support (2 of 60).

Machine-readable endpoint adoption across 300 top AI-cited domains by category

/llms.txt host (%) · 5 results

  • B2B SaaS62.22%
  • Developer/Docs54.44%
  • All 300 cited sites46%
  • E-commerce30%
  • Editorial/Media25%
  • 0%20%40%60%80%
Explore the data (5 rows)
B2B SaaS9062.2262.2216.6717.7812.2211.11
Developer/Docs9054.4467.7825.5628.8934.4416.67
All 300 cited sites3004650.3316.331515.339.33
E-commerce60303013.333.3300
Editorial/Media602526.6751.676.675
Counts and percentages from the November 20, 2026 live HTTP benchmark across all 300 top-cited hosts.

Takeaway

Default platform rollouts like Shopify UCP can push an entire vertical past Editorial/Media on /llms.txt overnight, even when per-page Markdown stays near zero.
  • Universal Commerce Protocol (UCP) specification

26.33% of sites reference MCP inside /llms.txt, yet only 2.67% publish /.well-known/mcp manifests

Companies are already documenting Model Context Protocol (MCP) endpoints inside /llms.txt far faster than they ship formal well-known JSON discovery endpoints. Across the 300 audited sites, 79 hosts (26.33%) explicitly mention or link to MCP inside their /llms.txt file, led by Developer/Docs (40.00%, 36 of 90) and B2B SaaS (38.89%, 35 of 90), alongside 11.67% in E-commerce (7 of 60 via Shopify UCP templates) and 1.67% in Editorial/Media (1 of 60).

By contrast, only 8 of 300 hosts (2.67%) return a valid JSON MCP discovery manifest at /.well-known/mcp or /.well-known/mcp.json: cloudflare.com, signoz.io, monday.com, notion.com, attio.com, langfuse.com, sentry.io, and docs.langchain.com (3.33% of Developer/Docs and 5.56% of B2B SaaS). Only 4 of 300 hosts (1.33%: cloudflare.com, monday.com, attio.com, and docs.langchain.com) publish a valid Agent-to-Agent (A2A) card at /.well-known/agent-card.json or /.well-known/agent.json.

Agent and MCP discovery mechanisms across 300 top AI-cited sites

  • MCP referenced inside /llms.txt26.33%79 of 300
  • HTML alternate Markdown link tags9.33%28 of 300
  • Valid /.well-known/mcp or mcp.json2.67%8 of 300
  • Valid /.well-known/agent-card.json1.33%4 of 300
  • 0%10%20%30%
Share of hosts returning valid non-HTML, non-bot-challenge payloads at each discovery path (Nov 20, 2026).

Takeaway

Most agent discovery today happens through plain-text /llms.txt links rather than /.well-known/mcp JSON files.
  • Model Context Protocol specification
  • Agent2Agent (A2A) Protocol specification
  • How AI recommends MCP and agent-native tools
  • Connect your agent directly to Parse MCP

Clean cited HTML pages carry 24.97x token bloat, peaking at 78.62x in E-commerce

When an AI coding agent or retrieval pipeline pulls a raw HTML page without a Markdown twin, most of the context window goes to page chrome rather than visible content. Across the 238 top-cited URLs in our 300-site panel that returned a clean HTTP 200 HTML page (at least 200 visible characters and no bot challenge), median raw HTML payload measured 347,139.5 bytes (86,772.5 estimated tokens at 4 characters per token) versus just 11,456 bytes (2,850.5 tokens) of stripped visible text, a median bloat multiplier of 24.97x (and 108.84x at the 90th percentile).

Inlining accounts for most of that tax: clean pages carried a median of 54,927 bytes in script tags, 22,779 bytes in inline SVG elements, and 2,253 bytes in style blocks. Storefronts and SaaS marketing pages carry the steepest overhead: E-commerce reaches a median bloat multiplier of 78.62x (126,840 raw HTML tokens vs 2,277.5 visible tokens; 167.68x at p90), and B2B SaaS reaches 31.46x (92,890 vs 2,491 tokens; 99.95x at p90), compared with 13.51x in Editorial/Media (121,616 vs 6,885 tokens) and 13.50x in Developer/Docs (38,300 vs 2,456.5 tokens).

Raw HTML tokens vs stripped visible text tokens across 238 clean cited pages

Median bloat (x) · 5 results

  • E-commerce78.62x
  • B2B SaaS31.46x
  • All clean cited pages24.97x
  • Editorial/Media13.51x
  • Developer/Docs13.5x
  • 020406080
Explore the data (5 rows)
E-commerce32126,8402,277.578.62167.68
B2B SaaS8592,8902,49131.4699.95
All clean cited pages23886,772.52,850.524.97108.84
Editorial/Media43121,6166,88513.5174.48
Developer/Docs7838,3002,456.513.545.38
Includes all HTTP 200 non-challenge pages with at least 200 visible characters; token estimates use UTF-8 decoded characters divided by 4.

Takeaway

On median E-commerce and B2B SaaS citations, over 96% to 98% of raw HTML characters are scripts, inline SVGs, styles, and layout markup rather than readable text.

Native Markdown cuts cited-page weight 25.41x overall and up to 640x on SaaS homepages

Across the 62 HTTP 200 cited URLs in the 300-site panel that return native Markdown via Accept: text/markdown or a .md URL twin, median page weight falls 25.41x, from 80,389.5 raw HTML tokens to 2,323 native Markdown tokens. Savings reach a median of 38.09x across 18 B2B SaaS pairs, 35.85x across 2 E-commerce pairs, 22.55x across 38 Developer/Docs pairs, and 8.87x across 4 Editorial/Media pairs.

Named production deployments show the practical payoff: sentry.io drops 640.29x (156,231 HTML tokens to 244 Markdown tokens), cloudflare.com drops 446.82x (333,772 to 747), vercel.com drops 265.25x (150,134 to 566), supabase.com drops 258.95x (337,409 to 1,303), docs.datadoghq.com/watchdog drops 214.65x (197,689 to 921), linear.app drops 182.04x (316,378 to 1,738), and developer.avalara.com drops 1,893.88x (749,975 to 396) where raw HTML embedded huge serialized script payloads. In our 13-page named reference twin panel, Parse's own docs page (parse.gl/docs/mcp) cuts tokens 10.91x (51,376 to 4,711; 90.83% reduction), alongside Mintlify docs (61.22x), Anthropic docs (76.17x), Stripe tax docs (75.46x), and Linear docs (119.58x).

Exact HTML vs native Markdown token counts on paired Top-300 and reference pages

Highest token-reduction ratios among the 62 top-cited 300 pages serving native Markdown via Accept: text/markdown or .md twins.

Savings (x) · Top 10 of 12

  • developer.avalara.com1,893.88x
  • sentry.io640.29x
  • cloudflare.com446.82x
  • vercel.com265.25x
  • supabase.com258.95x
  • opentelemetry.io219.34x
  • docs.datadoghq.com214.65x
  • linear.app182.04x
  • openobserve.ai142.09x
  • asana.com140.35x
  • 05001,0001,5002,000
Explore the data (12 rows)
developer.avalara.comDeveloper/Docs749,975712,7633961,893.88
sentry.ioDeveloper/Docs156,2314,995244640.29
cloudflare.comB2B SaaS333,7721,815747446.82
vercel.comB2B SaaS150,1341,009566265.25
supabase.comB2B SaaS337,4092,3411,303258.95
opentelemetry.ioDeveloper/Docs101,7723,588464219.34
docs.datadoghq.comDeveloper/Docs197,68914,636921214.65
linear.appB2B SaaS316,3782,6901,738182.04
openobserve.aiB2B SaaS218,2493,7001,536142.09
asana.comB2B SaaS236,0612,6681,682140.35
developer.harness.ioDeveloper/Docs244,3631,9891,953125.12
freshworks.comB2B SaaS134,7231,0471,315102.45
Switch views between the top 300 paired Markdown savings and the 13-page Named Reference Twin panel (Nov 20, 2026).

Takeaway

Serving clean Markdown reduces page size by 90% to 99% on real production docs and SaaS pages while keeping core content intact.
  • Cloudflare: Introducing Markdown for Agents

19.67% of top AI-cited URLs fail plain HTTP retrieval, led by E-commerce at 43.33%

Plain HTTP GET requests using a standard Chrome User-Agent failed on 59 of the 300 top-cited URLs (19.67%). Explicit HTTP 403 blocks accounted for 41 failures, followed by 9 HTTP 200 pages serving JavaScript shells with fewer than 200 visible characters, 2 explicit bot-challenge interstitials (zoho.com and docs.pytorch.org), 2 HTTP 429 rate limits (chewy.com and wayfair.com), 2 HTTP 402 payment walls (travelandleisure.com and seriouseats.com), 1 HTTP 401 (reuters.com), 1 HTTP 400 (developers.meta.com), and 1 header-overflow failure (finance.yahoo.com).

Failure rates track industry stack choices closely: 43.33% of E-commerce citations (26 of 60) and 28.33% of Editorial/Media citations (17 of 60) failed plain retrieval, versus 13.33% in Developer/Docs (12 of 90) and only 4.44% in B2B SaaS (4 of 90).

Plain HTTP retrieval failure rate across 300 top AI-cited URLs

  • E-commerce43.33%26 of 60
  • Editorial/Media28.33%17 of 60
  • All 300 cited URLs19.67%59 of 300
  • Developer/Docs13.33%12 of 90
  • B2B SaaS4.44%4 of 90
  • 0%20%40%60%
Includes HTTP >= 400 errors, connection failures, bot-challenge pages, and HTTP 200 empty JS shells (<200 visible chars).

Takeaway

One in five top-cited URLs fails simple direct HTTP fetching, hitting nearly half of E-commerce citations.
  • Noindex, JS walls, and canonical mismatches on 378,392 AI-cited pages
  • How often do AI-cited pages block access?

Headless Chromium recovers 9 of 10 empty JS shells up to 473.2x, while server-rendered pages stay flat

To verify whether HTTP 200 thin responses were client-side JavaScript shells rather than short articles, we executed headless Playwright Chromium across all 10 reachable JS-shell suspects (<500 plain visible characters) alongside a stratified sample across all four categories (49 total Playwright measurements recorded, including 37 valid non-error stratified pages plus 2 stratified timeouts/protocol errors on eileenfisher.com and bestbuy.com).

On 9 of the 10 JS-shell suspects, rendering JavaScript restored full page text from under 150 plain characters to between 2,897 and 26,026 DOM characters: docs.pytorch.org jumped 473.20x (55 to 26,026 chars), pro.globalforestwatch.org jumped 218.74x (23 to 5,031), bubble.io jumped 202.42x (86 to 17,408), support.zoom.com jumped 153.29x (87 to 13,336), help.asana.com jumped 103.57x (70 to 7,250), developer.apple.com jumped 83.86x (37 to 3,103), siemens.com jumped 67.08x (131 to 8,788), wearpact.com jumped 21.46x (135 to 2,897), and nordstromrack.com moved from 0 plain chars to 6,276 rendered chars. Only help.draftkings.com stayed thin at 320 chars (2.69x). Across the 37 normal stratified pages, the median rendered-to-plain ratio was 0.88x (confirming plain HTTP already captured their main copy, except partial hydrations like rtings.com at 14.97x).

Playwright Chromium recovery across all 10 HTTP 200 JS-shell suspects

Chromium chars · 10 results

  • docs.pytorch.org26,026
  • bubble.io17,408
  • support.zoom.com13,336
  • siemens.com8,788
  • help.asana.com7,250
  • nordstromrack.com6,276
  • pro.globalforestwatch.org5,031
  • developer.apple.com3,103
  • wearpact.com2,897
  • help.draftkings.com320
  • 010,00020,00030,000
Explore the data (10 rows)
docs.pytorch.orgDeveloper/Docs5526,026473.20x
bubble.ioB2B SaaS8617,408202.42x
support.zoom.comDeveloper/Docs8713,336153.29x
siemens.comB2B SaaS1318,78867.08x
help.asana.comDeveloper/Docs707,250103.57x
nordstromrack.comE-commerce06,2760 to 6,276 chars
pro.globalforestwatch.orgDeveloper/Docs235,031218.74x
developer.apple.comDeveloper/Docs373,10383.86x
wearpact.comE-commerce1352,89721.46x
help.draftkings.comDeveloper/Docs1193202.69x
Compares plain HTTP stripped visible characters against rendered Playwright DOM innerText characters (Nov 20, 2026).

Takeaway

Client-side rendering hides entire help centers and docs pages from plain HTTP fetchers unless the site exposes an HTML SSR layer or a .md twin like developer.apple.com and docs.pytorch.org do.
  • Microsoft: Playwright MCP server repository

23.67% of cited URLs blocked OAI-SearchBot requests from a test IP, vs 2.26% disallowing it in robots.txt

Live HTTP requests sent with OpenAI's documented OAI-SearchBot/1.0 User-Agent from a non-OpenAI benchmark test IP were blocked on 23.67% of top-cited URLs (71 of 300: 36 HTTP 403, 21 challenge-403, 6 timeouts, 2 challenge-200, 2 HTTP 429, 1 challenge-429, 2 HTTP 402, and 1 HTTP 401) and 24.33% of homepages (73 of 300). Cited-URL blocks reached 46.67% in E-commerce (28 of 60) and 35.00% in Editorial/Media (21 of 60), versus 14.44% in B2B SaaS (13 of 90) and 10.00% in Developer/Docs (9 of 90).

Methodological boundary: because these test probes sent the OAI-SearchBot User-Agent from a non-OpenAI benchmark IP, edge WAF IP-verification and anti-spoofing rules contribute alongside pure User-Agent blocks. Production crawlers connecting from OpenAI's published IP ranges can pass IP-verified allowlists. Even with that boundary, comparing live WAF behavior against the 265 readable robots.txt files shows a 10.5x gap: only 6 of 265 hosts (2.26%) explicitly disallow OAI-SearchBot at /, compared with 16 (6.04%) disallowing GPTBot, 20 (7.55%) disallowing ClaudeBot, and 22 (8.30%) disallowing CCBot. Meanwhile, 2 Editorial/Media sites (travelandleisure.com and seriouseats.com) returned HTTP 402 Pay Per Crawl responses, 27 of 265 robots.txt files contained Content-Signal directives, and 3 hosts (freshworks.com, retool.com, theimagecdn.com) emitted Content-Signal HTTP headers.

Declared robots.txt AI rules vs live HTTP and OAI-SearchBot test-IP blocks

Explicit root-level Disallow: / and bot mentions across the 265 top-cited hosts with a valid non-HTML robots.txt.

Disallow: / (%) · 7 results

  • ccbot8.3%
  • claudebot7.55%
  • google-extended6.42%
  • gptbot6.04%
  • perplexitybot5.66%
  • chatgpt-user2.64%
  • oai-searchbot2.26%
  • 0%5%10%
Explore the data (7 rows)
ccbot44228.3
claudebot57207.55
google-extended51176.42
gptbot56166.04
perplexitybot52155.66
chatgpt-user4672.64
oai-searchbot4562.26
Switch views between robots.txt Disallow: / rates across 265 readable files and category-level HTTP block rates across all 300 sites.

Takeaway

WAF rules and IP-verification gates decide whether third-party agents and unverified crawlers can read your pages far more often than robots.txt does.
  • OpenAI: Overview of OpenAI crawlers (OAI-SearchBot and GPTBot)
  • Cloudflare: Introducing Pay Per Crawl
  • Cloudflare: Content Signals Policy
  • Do generative AI assistants respect robots.txt? Tracing web access beyond visible answers

Study scope, counting rules, and methodological boundaries

Target selection reflects Parse's observed organic citations from September 1 through November 19, 2026 across ChatGPT Search, Google AI Mode, and Perplexity. We selected the top 90 Developer/Docs, top 90 B2B SaaS, top 60 Editorial/Media, and top 60 E-commerce hosts deduplicated by root entity (keeping individual hosts on multi-tenant publishing platforms such as github.io, readthedocs.io, mintlify.app, and medium.com), along with each host's most-cited non-binary URL.

All HTTP probes and 49 headless Playwright Chromium verifications ran live on November 20, 2026 alongside 13 named reference twin URLs. Token estimates use UTF-8 decoded characters divided by 4. Clean HTML bloat covers the 238 top-cited URLs returning HTTP 200 non-challenge HTML with at least 200 visible characters; native Markdown savings cover the 62 paired top-cited URLs returning non-HTML Markdown via Accept: text/markdown or a .md twin. OAI-SearchBot HTTP block rates reflect probes sent with the documented OAI-SearchBot/1.0 User-Agent from a non-OpenAI benchmark test IP, so edge IP-verification checks contribute alongside User-Agent rules.

Engineering and SEO checklist for an agent-readable site

Start by ensuring every core product, pricing, and documentation URL renders readable server HTML without JavaScript hydration walls, and verify live HTTP status directly using Parse MCP's fetch_live_pages tool or plain curl checks alongside a browser render.

Next, publish a clean /llms.txt index at both your root domain and docs subdomain, serve native Markdown via Accept: text/markdown or .md twins with HTML alternate Markdown link tags to trim payload weight by ~25x, emit Content-Signal preferences, and publish /.well-known/mcp if you run a remote MCP server.

Agent-readability engineering checklist

WAF, robots.txt, and Content-Signal23.67% test-IP block vs 2.26% robots.txt DisallowAllow verified OAI-SearchBot IP ranges in your WAF, align robots.txt rules, and set explicit Content-Signal directives.
Server-rendered core HTML19.67% plain fetch fail; 9 of 10 JS shells emptyPre-render product, pricing, and documentation copy in initial HTTP 200 HTML so simple HTTP agents never hit empty JS shells.
Accept: text/markdown and .md twins15.00% Accept: text/markdown; 25.41x token cutServe clean Markdown on Accept: text/markdown or .md URL twins and declare HTML alternate Markdown link tags in the document head.
/llms.txt and /llms-full.txt46.00% host /llms.txt; 62.22% B2B SaaSPublish a curated /llms.txt table of contents linking directly to canonical Markdown guides and any MCP endpoint.
/.well-known/mcp and A2A cards2.67% /.well-known/mcp; 1.33% agent cardsPublish structured JSON discovery metadata at /.well-known/mcp.json alongside human-readable /llms.txt links.

Takeaway

Pair server-rendered HTML with /llms.txt and native Markdown twins so both search crawlers and context-constrained coding agents read your site cleanly.
  • Plan machine-readable content updates in Parse
  • Audit live cited URLs via Parse MCP (fetch_live_pages)
  • How to make your site readable for AI agents
  • How AI recommends MCP and agent-native tools
  • Noindex, JS walls, and canonical mismatches on 378,392 AI-cited pages
  • GitHub repos, package registries, and docs architecture in AI search

How we measured

We analyzed 300 top AI-cited hosts and their #1 most-cited URLs selected from September 1 to November 19, 2026 organic citations across ChatGPT Search, Google AI Mode, and Perplexity (90 Developer/Docs, 90 B2B SaaS, 60 Editorial/Media, and 60 E-commerce), probed live via HTTP on November 20, 2026 alongside 13 named reference twin URLs and 49 local Playwright Chromium browser verifications (yielding 238 clean HTTP 200 HTML pages and 62 native Markdown pairs inside the 300-site panel).

Median token cut from raw HTML to native Markdown
25.41x

Across 62 paired cited pages (80,389.5 HTML vs 2,323 Markdown tokens)

Top 300 cited hosts serving /llms.txt
46.00%

138 of 300 at cited host; 50.33% (151) including root domains

Top 300 sites honoring Accept: text/markdown
15.00%

45 of 300 overall; 28.89% in Developer/Docs vs 1.67% in Editorial/Media

Cited URLs failing plain HTTP fetch
19.67%

59 of 300 overall, peaking at 43.33% in E-commerce (26 of 60)

Get the data

Full study dataset (CSV)All 318 data rows: 5 category summaries, all 300 top AI-cited hosts with live endpoint measurements, and 13 named reference twin pages.

Sources

  1. The /llms.txt specification (llmstxt.org) (opens in a new tab) · accessed November 20, 2026
  2. Cloudflare: Introducing Markdown for Agents (opens in a new tab) · accessed November 20, 2026
  3. Cloudflare: Introducing Pay Per Crawl (opens in a new tab) · accessed November 20, 2026
  4. Cloudflare: Content Signals Policy (opens in a new tab) · accessed November 20, 2026
  5. Universal Commerce Protocol (UCP) specification (opens in a new tab) · accessed November 20, 2026
  6. Model Context Protocol specification (opens in a new tab) · accessed November 20, 2026
  7. Agent2Agent (A2A) Protocol specification (opens in a new tab) · accessed November 20, 2026
  8. OpenAI: Overview of OpenAI crawlers (OAI-SearchBot and GPTBot) (opens in a new tab) · accessed November 20, 2026
  9. Microsoft: Playwright MCP server repository (opens in a new tab) · accessed November 20, 2026
  10. Do generative AI assistants respect robots.txt? Tracing web access beyond visible answers (opens in a new tab) · accessed November 20, 2026

More like this

How AI recommends MCP and agent-native tools across 911,413 answers
94.96% of 6,292 MCP mentions across 911,413 answers happened unprompted (surging +127.12% on ChatGPT Search), while narrow agent/API topics flip #1 55.00% of the time.
Noindex, JS walls, and canonical mismatches on 378,392 AI-cited pages
16.99% of 378,392 AI-cited pages failed simple direct HTTP fetch, while 2,392 cited HTML pages carried noindex and 7,533 carried cross-domain canonicals.
GitHub repos, package registries, and docs architecture in AI search
88.55% of 41,065 GitHub citations hit repo roots or Markdown docs, while GitHub + own-site backing lifts #1 win rates 34.5% on ChatGPT Search.
How often do AI-cited pages block access?
Only 1.6% of checked AI-cited pages blocked direct access afterward; 97.5% returned a normal response.
JSON-LD schema, author bylines, and content age on 329,519 AI-cited pages
70.86% of 329,519 AI-cited HTML pages carry JSON-LD schema, 30.86% expose an author byline, and dated pages show a 74.5-day median age.
How long are the pages AI cites?
ChatGPT Search cited pages had a median length of 1,000 words vs 1,368 on Google AI Mode in Aug to Nov 2026, led by 3,098-word reviews and 1,026-word vendor sites.
For agents: this page as Markdown, https://parse.gl/research/agent-readability-html-vs-markdown-and-llms-txt.md. JSON, MCP · free for the last 90 days, $0.01 a check
Open in ChatGPT (opens in a new tab) Open in Claude (opens in a new tab)

This page

As Markdown: https://parse.gl/research/agent-readability-html-vs-markdown-and-llms-txt.md
As CSV: https://parse.gl/research/agent-readability-html-vs-markdown-and-llms-txt/dataset.csv
Reuse: Figures and tables on this page are free to republish with attribution: a link to https://parse.gl/research/agent-readability-html-vs-markdown-and-llms-txt beside them. A standard followed link (dofollow) is preferred.

Read and search

Any brand's shares and rivals, with 95% ranges: GET https://agent.parse.gl/v1/tools/check_brand?subject=%3Cbrand-domain%3E
Sample answers in a market: GET https://agent.parse.gl/v1/tools/search_answers?args=%7B%22filter%22%3A%7B%22market%22%3A%5B%22%3Cmkt_id%3E%22%5D%7D%7D
Cited pages in a market: GET https://agent.parse.gl/v1/tools/search_cited_pages?args=%7B%22filter%22%3A%7B%22market%22%3A%5B%22%3Cmkt_id%3E%22%5D%7D%7D
Search the index: GET https://parse.gl/api/public/search?q=<words>

Connect

Connect an assistant: https://agent.parse.gl/mcp

An agent can save work before signup: POST https://agent.parse.gl/v1/workspaces

What it costs

No account: Public figures and up to 5 examples per list, about 120 brands a day per address.
Free account: $5 in credit, your brand's first check (20 buyer questions) and a free weekly check.
After your first top-up: Exports and deeper search results. The first top-up starts at $5.
Live checks: $0.01 per question per engine.
Full public answers and pages: 25 free per day, then $0.005 each.
Public answers older than 90 days: $0.001 each the first time your workspace counts or lists one, then yours.

Prices: https://parse.gl/pricing.md

Map of the site: https://parse.gl/llms.txt

Inspect agent readability, Markdown twins, and crawl barriers across your cited pages

Start free in your browser.
Parse

Index

  • Brands
  • Markets
  • Topics
  • How we measure

For agents

  • Docs
  • Connect
  • Workflows
  • llms.txt

Parse

  • Use cases
  • Pricing
  • Research
  • Blog
  • Compare

Company

  • About
  • Contact
  • Corrections

© 2026 Parse, Inc. All rights reserved.

Privacy PolicyTerms of Service