How much of the web that AI search engines cite is readable without headless browsers or HTML boilerplate stripping, and how far have /llms.txt, Markdown twins, and agent discovery endpoints spread across top-cited domains?
We extracted the 300 most-cited hosts and each host's top-cited URL from organic ChatGPT Search, Google AI Mode, and Perplexity answers recorded September 1 through November 19, 2026 (90 Developer/Docs, 90 B2B SaaS, 60 Editorial/Media, and 60 E-commerce), then probed every host live on November 20, 2026 for /llms.txt, /llms-full.txt, Accept: text/markdown responses, .md twins, /.well-known/mcp manifests, robots.txt AI rules, OAI-SearchBot access, HTML token bloat, and headless Chromium rendering across 49 browser-checked pages alongside 13 named reference twin pages.
Across the 300 most-cited Developer/Docs, B2B SaaS, Editorial/Media, and E-commerce domains in Parse's September to November 2026 index, 46.00% serve a real /llms.txt and 15.00% honor Accept: text/markdown content negotiation. Serving native Markdown cuts median page weight 25.41x across 62 paired cited pages, while 19.67% of top-cited URLs fail plain HTTP retrieval and 23.67% reject requests using the OAI-SearchBot User-Agent from a non-OpenAI benchmark IP.
46.00% of top AI-cited hosts serve /llms.txt, while 15.00% honor Accept: text/markdown
Across the 300 most-cited web hosts in Parse's index, 138 (46.00%) serve a valid, non-HTML /llms.txt index at the cited host, and 151 (50.33%) serve one on either the cited host or its parent root domain. Another 49 hosts (16.33%) publish a consolidated /llms-full.txt file.
Page-level Markdown delivery lags directory-level /llms.txt adoption by roughly 3-to-1. Exactly 45 of 300 hosts (15.00%) honor Accept: text/markdown HTTP content negotiation (31 on the homepage and 45 on the top-cited URL), 46 of 300 (15.33%) expose a live .md URL twin on their top-cited page, 63 of 300 (21.00%) support at least one of those two mechanisms (28 support both), and 28 of 300 (9.33%) declare HTML alternate Markdown link tags in the document head.
- Serve /llms.txt on the cited host
- 46.00%
138 of 300 top-cited sites (151 including root domains)
62.22% B2B SaaS · 54.44% Developer/Docs · 30.00% E-commerce
- Median HTML-to-Markdown token savings
- 25.41x
62 native Markdown pairs · 24.97x HTML bloat across 238 pages
Median 86,772.5 raw HTML tokens vs 2,850.5 visible text tokens
- Honor Accept: text/markdown headers
- 15.00%
45 of 300 sites · 15.33% (46) serve .md URL twins
2.67% (8 of 300) publish /.well-known/mcp discovery JSON
- Refuse benchmark OAI-SearchBot requests
- 23.67%
71 of 300 cited URLs (73 homepages) from test IP
vs 2.26% (6 of 265) blocking OAI-SearchBot in robots.txt
Machine-readable web standard adoption across 300 top AI-cited hosts (%)
- /llms.txt (host)
- /llms-full.txt
- Accept text/markdown
- .md URL twin
- Developer/Docs
- /llms.txt (host)54.44%49 of 90
- /llms-full.txt25.56%23 of 90
- Accept text/markdown28.89%26 of 90
- .md URL twin34.44%31 of 90
- B2B SaaS
- /llms.txt (host)62.22%56 of 90
- /llms-full.txt16.67%15 of 90
- Accept text/markdown17.78%16 of 90
- .md URL twin12.22%11 of 90
- E-commerce
- /llms.txt (host)30.00%18 of 60
- /llms-full.txt13.33%8 of 60
- Accept text/markdown3.33%2 of 60
- .md URL twin0.00%0 of 60
- Editorial/Media
- /llms.txt (host)25.00%15 of 60
- /llms-full.txt5.00%3 of 60
- Accept text/markdown1.67%1 of 60
- .md URL twin6.67%4 of 60
Takeaway
B2B SaaS leads /llms.txt at 62.22%, while Shopify templates push E-commerce ahead of Editorial/Media
Adoption splits sharply by site category. B2B SaaS leads cited-host /llms.txt adoption at 62.22% (56 of 90 hosts). Developer/Docs posts 54.44% at the cited host (49 of 90) and rises to 67.78% (61 of 90) when parent root domains are checked, because several support and docs subdomains inherit a root /llms.txt. Developer/Docs also leads page-level Markdown twins at 34.44% (31 of 90) and Accept: text/markdown at 28.89% (26 of 90).
E-commerce /llms.txt adoption reaches 30.00% (18 of 60), beating Editorial/Media at 25.00% (15 of 60). Platform defaults explain why: 7 Shopify merchants in the 60-store cohort (sunhomesaunas.com, americantall.com, christydawn.com, everlane.com, girlfriend.com, mudjeans.com, and matethelabel.com) expose Shopify's default Universal Commerce Protocol (UCP) Shop Skill /llms.txt. Outside those storefronts, E-commerce has 0.00% .md twins (0 of 60) and 3.33% Accept: text/markdown support (2 of 60).
Machine-readable endpoint adoption across 300 top AI-cited domains by category
/llms.txt host (%) · 5 results
- B2B SaaS62.22%
- Developer/Docs54.44%
- All 300 cited sites46%
- E-commerce30%
- Editorial/Media25%
Explore the data (5 rows)
| B2B SaaS | 90 | 62.22 | 62.22 | 16.67 | 17.78 | 12.22 | 11.11 |
| Developer/Docs | 90 | 54.44 | 67.78 | 25.56 | 28.89 | 34.44 | 16.67 |
| All 300 cited sites | 300 | 46 | 50.33 | 16.33 | 15 | 15.33 | 9.33 |
| E-commerce | 60 | 30 | 30 | 13.33 | 3.33 | 0 | 0 |
| Editorial/Media | 60 | 25 | 26.67 | 5 | 1.67 | 6.67 | 5 |
Takeaway
26.33% of sites reference MCP inside /llms.txt, yet only 2.67% publish /.well-known/mcp manifests
Companies are already documenting Model Context Protocol (MCP) endpoints inside /llms.txt far faster than they ship formal well-known JSON discovery endpoints. Across the 300 audited sites, 79 hosts (26.33%) explicitly mention or link to MCP inside their /llms.txt file, led by Developer/Docs (40.00%, 36 of 90) and B2B SaaS (38.89%, 35 of 90), alongside 11.67% in E-commerce (7 of 60 via Shopify UCP templates) and 1.67% in Editorial/Media (1 of 60).
By contrast, only 8 of 300 hosts (2.67%) return a valid JSON MCP discovery manifest at /.well-known/mcp or /.well-known/mcp.json: cloudflare.com, signoz.io, monday.com, notion.com, attio.com, langfuse.com, sentry.io, and docs.langchain.com (3.33% of Developer/Docs and 5.56% of B2B SaaS). Only 4 of 300 hosts (1.33%: cloudflare.com, monday.com, attio.com, and docs.langchain.com) publish a valid Agent-to-Agent (A2A) card at /.well-known/agent-card.json or /.well-known/agent.json.
Agent and MCP discovery mechanisms across 300 top AI-cited sites
- MCP referenced inside /llms.txt26.33%79 of 300
- HTML alternate Markdown link tags9.33%28 of 300
- Valid /.well-known/mcp or mcp.json2.67%8 of 300
- Valid /.well-known/agent-card.json1.33%4 of 300
Takeaway
Clean cited HTML pages carry 24.97x token bloat, peaking at 78.62x in E-commerce
When an AI coding agent or retrieval pipeline pulls a raw HTML page without a Markdown twin, most of the context window goes to page chrome rather than visible content. Across the 238 top-cited URLs in our 300-site panel that returned a clean HTTP 200 HTML page (at least 200 visible characters and no bot challenge), median raw HTML payload measured 347,139.5 bytes (86,772.5 estimated tokens at 4 characters per token) versus just 11,456 bytes (2,850.5 tokens) of stripped visible text, a median bloat multiplier of 24.97x (and 108.84x at the 90th percentile).
Inlining accounts for most of that tax: clean pages carried a median of 54,927 bytes in script tags, 22,779 bytes in inline SVG elements, and 2,253 bytes in style blocks. Storefronts and SaaS marketing pages carry the steepest overhead: E-commerce reaches a median bloat multiplier of 78.62x (126,840 raw HTML tokens vs 2,277.5 visible tokens; 167.68x at p90), and B2B SaaS reaches 31.46x (92,890 vs 2,491 tokens; 99.95x at p90), compared with 13.51x in Editorial/Media (121,616 vs 6,885 tokens) and 13.50x in Developer/Docs (38,300 vs 2,456.5 tokens).
Raw HTML tokens vs stripped visible text tokens across 238 clean cited pages
Median bloat (x) · 5 results
- E-commerce78.62x
- B2B SaaS31.46x
- All clean cited pages24.97x
- Editorial/Media13.51x
- Developer/Docs13.5x
Explore the data (5 rows)
| E-commerce | 32 | 126,840 | 2,277.5 | 78.62 | 167.68 |
| B2B SaaS | 85 | 92,890 | 2,491 | 31.46 | 99.95 |
| All clean cited pages | 238 | 86,772.5 | 2,850.5 | 24.97 | 108.84 |
| Editorial/Media | 43 | 121,616 | 6,885 | 13.51 | 74.48 |
| Developer/Docs | 78 | 38,300 | 2,456.5 | 13.5 | 45.38 |
Takeaway
Native Markdown cuts cited-page weight 25.41x overall and up to 640x on SaaS homepages
Across the 62 HTTP 200 cited URLs in the 300-site panel that return native Markdown via Accept: text/markdown or a .md URL twin, median page weight falls 25.41x, from 80,389.5 raw HTML tokens to 2,323 native Markdown tokens. Savings reach a median of 38.09x across 18 B2B SaaS pairs, 35.85x across 2 E-commerce pairs, 22.55x across 38 Developer/Docs pairs, and 8.87x across 4 Editorial/Media pairs.
Named production deployments show the practical payoff: sentry.io drops 640.29x (156,231 HTML tokens to 244 Markdown tokens), cloudflare.com drops 446.82x (333,772 to 747), vercel.com drops 265.25x (150,134 to 566), supabase.com drops 258.95x (337,409 to 1,303), docs.datadoghq.com/watchdog drops 214.65x (197,689 to 921), linear.app drops 182.04x (316,378 to 1,738), and developer.avalara.com drops 1,893.88x (749,975 to 396) where raw HTML embedded huge serialized script payloads. In our 13-page named reference twin panel, Parse's own docs page (parse.gl/docs/mcp) cuts tokens 10.91x (51,376 to 4,711; 90.83% reduction), alongside Mintlify docs (61.22x), Anthropic docs (76.17x), Stripe tax docs (75.46x), and Linear docs (119.58x).
Exact HTML vs native Markdown token counts on paired Top-300 and reference pages
Highest token-reduction ratios among the 62 top-cited 300 pages serving native Markdown via Accept: text/markdown or .md twins.
Savings (x) · Top 10 of 12
developer.avalara.com1,893.88x
sentry.io640.29x
cloudflare.com446.82x
vercel.com265.25x
supabase.com258.95x
opentelemetry.io219.34x
docs.datadoghq.com214.65x
linear.app182.04x
openobserve.ai142.09x
asana.com140.35x
Explore the data (12 rows)
| Developer/Docs | 749,975 | 712,763 | 396 | 1,893.88 | |
| Developer/Docs | 156,231 | 4,995 | 244 | 640.29 | |
| B2B SaaS | 333,772 | 1,815 | 747 | 446.82 | |
| B2B SaaS | 150,134 | 1,009 | 566 | 265.25 | |
| B2B SaaS | 337,409 | 2,341 | 1,303 | 258.95 | |
| Developer/Docs | 101,772 | 3,588 | 464 | 219.34 | |
| Developer/Docs | 197,689 | 14,636 | 921 | 214.65 | |
| B2B SaaS | 316,378 | 2,690 | 1,738 | 182.04 | |
| B2B SaaS | 218,249 | 3,700 | 1,536 | 142.09 | |
| B2B SaaS | 236,061 | 2,668 | 1,682 | 140.35 | |
| Developer/Docs | 244,363 | 1,989 | 1,953 | 125.12 | |
| B2B SaaS | 134,723 | 1,047 | 1,315 | 102.45 |
Takeaway
19.67% of top AI-cited URLs fail plain HTTP retrieval, led by E-commerce at 43.33%
Plain HTTP GET requests using a standard Chrome User-Agent failed on 59 of the 300 top-cited URLs (19.67%). Explicit HTTP 403 blocks accounted for 41 failures, followed by 9 HTTP 200 pages serving JavaScript shells with fewer than 200 visible characters, 2 explicit bot-challenge interstitials (zoho.com and docs.pytorch.org), 2 HTTP 429 rate limits (chewy.com and wayfair.com), 2 HTTP 402 payment walls (travelandleisure.com and seriouseats.com), 1 HTTP 401 (reuters.com), 1 HTTP 400 (developers.meta.com), and 1 header-overflow failure (finance.yahoo.com).
Failure rates track industry stack choices closely: 43.33% of E-commerce citations (26 of 60) and 28.33% of Editorial/Media citations (17 of 60) failed plain retrieval, versus 13.33% in Developer/Docs (12 of 90) and only 4.44% in B2B SaaS (4 of 90).
Plain HTTP retrieval failure rate across 300 top AI-cited URLs
- E-commerce43.33%26 of 60
- Editorial/Media28.33%17 of 60
- All 300 cited URLs19.67%59 of 300
- Developer/Docs13.33%12 of 90
- B2B SaaS4.44%4 of 90
Takeaway
Headless Chromium recovers 9 of 10 empty JS shells up to 473.2x, while server-rendered pages stay flat
To verify whether HTTP 200 thin responses were client-side JavaScript shells rather than short articles, we executed headless Playwright Chromium across all 10 reachable JS-shell suspects (<500 plain visible characters) alongside a stratified sample across all four categories (49 total Playwright measurements recorded, including 37 valid non-error stratified pages plus 2 stratified timeouts/protocol errors on eileenfisher.com and bestbuy.com).
On 9 of the 10 JS-shell suspects, rendering JavaScript restored full page text from under 150 plain characters to between 2,897 and 26,026 DOM characters: docs.pytorch.org jumped 473.20x (55 to 26,026 chars), pro.globalforestwatch.org jumped 218.74x (23 to 5,031), bubble.io jumped 202.42x (86 to 17,408), support.zoom.com jumped 153.29x (87 to 13,336), help.asana.com jumped 103.57x (70 to 7,250), developer.apple.com jumped 83.86x (37 to 3,103), siemens.com jumped 67.08x (131 to 8,788), wearpact.com jumped 21.46x (135 to 2,897), and nordstromrack.com moved from 0 plain chars to 6,276 rendered chars. Only help.draftkings.com stayed thin at 320 chars (2.69x). Across the 37 normal stratified pages, the median rendered-to-plain ratio was 0.88x (confirming plain HTTP already captured their main copy, except partial hydrations like rtings.com at 14.97x).
Playwright Chromium recovery across all 10 HTTP 200 JS-shell suspects
Chromium chars · 10 results
docs.pytorch.org26,026
bubble.io17,408
support.zoom.com13,336
siemens.com8,788
help.asana.com7,250
nordstromrack.com6,276
pro.globalforestwatch.org5,031
developer.apple.com3,103
wearpact.com2,897
help.draftkings.com320
Explore the data (10 rows)
| Developer/Docs | 55 | 26,026 | 473.20x | |
| B2B SaaS | 86 | 17,408 | 202.42x | |
| Developer/Docs | 87 | 13,336 | 153.29x | |
| B2B SaaS | 131 | 8,788 | 67.08x | |
| Developer/Docs | 70 | 7,250 | 103.57x | |
| E-commerce | 0 | 6,276 | 0 to 6,276 chars | |
| Developer/Docs | 23 | 5,031 | 218.74x | |
| Developer/Docs | 37 | 3,103 | 83.86x | |
| E-commerce | 135 | 2,897 | 21.46x | |
| Developer/Docs | 119 | 320 | 2.69x |
Takeaway
23.67% of cited URLs blocked OAI-SearchBot requests from a test IP, vs 2.26% disallowing it in robots.txt
Live HTTP requests sent with OpenAI's documented OAI-SearchBot/1.0 User-Agent from a non-OpenAI benchmark test IP were blocked on 23.67% of top-cited URLs (71 of 300: 36 HTTP 403, 21 challenge-403, 6 timeouts, 2 challenge-200, 2 HTTP 429, 1 challenge-429, 2 HTTP 402, and 1 HTTP 401) and 24.33% of homepages (73 of 300). Cited-URL blocks reached 46.67% in E-commerce (28 of 60) and 35.00% in Editorial/Media (21 of 60), versus 14.44% in B2B SaaS (13 of 90) and 10.00% in Developer/Docs (9 of 90).
Methodological boundary: because these test probes sent the OAI-SearchBot User-Agent from a non-OpenAI benchmark IP, edge WAF IP-verification and anti-spoofing rules contribute alongside pure User-Agent blocks. Production crawlers connecting from OpenAI's published IP ranges can pass IP-verified allowlists. Even with that boundary, comparing live WAF behavior against the 265 readable robots.txt files shows a 10.5x gap: only 6 of 265 hosts (2.26%) explicitly disallow OAI-SearchBot at /, compared with 16 (6.04%) disallowing GPTBot, 20 (7.55%) disallowing ClaudeBot, and 22 (8.30%) disallowing CCBot. Meanwhile, 2 Editorial/Media sites (travelandleisure.com and seriouseats.com) returned HTTP 402 Pay Per Crawl responses, 27 of 265 robots.txt files contained Content-Signal directives, and 3 hosts (freshworks.com, retool.com, theimagecdn.com) emitted Content-Signal HTTP headers.
Declared robots.txt AI rules vs live HTTP and OAI-SearchBot test-IP blocks
Explicit root-level Disallow: / and bot mentions across the 265 top-cited hosts with a valid non-HTML robots.txt.
Disallow: / (%) · 7 results
- ccbot8.3%
- claudebot7.55%
- google-extended6.42%
- gptbot6.04%
- perplexitybot5.66%
- chatgpt-user2.64%
- oai-searchbot2.26%
Explore the data (7 rows)
| ccbot | 44 | 22 | 8.3 |
| claudebot | 57 | 20 | 7.55 |
| google-extended | 51 | 17 | 6.42 |
| gptbot | 56 | 16 | 6.04 |
| perplexitybot | 52 | 15 | 5.66 |
| chatgpt-user | 46 | 7 | 2.64 |
| oai-searchbot | 45 | 6 | 2.26 |
Takeaway
Study scope, counting rules, and methodological boundaries
Target selection reflects Parse's observed organic citations from September 1 through November 19, 2026 across ChatGPT Search, Google AI Mode, and Perplexity. We selected the top 90 Developer/Docs, top 90 B2B SaaS, top 60 Editorial/Media, and top 60 E-commerce hosts deduplicated by root entity (keeping individual hosts on multi-tenant publishing platforms such as github.io, readthedocs.io, mintlify.app, and medium.com), along with each host's most-cited non-binary URL.
All HTTP probes and 49 headless Playwright Chromium verifications ran live on November 20, 2026 alongside 13 named reference twin URLs. Token estimates use UTF-8 decoded characters divided by 4. Clean HTML bloat covers the 238 top-cited URLs returning HTTP 200 non-challenge HTML with at least 200 visible characters; native Markdown savings cover the 62 paired top-cited URLs returning non-HTML Markdown via Accept: text/markdown or a .md twin. OAI-SearchBot HTTP block rates reflect probes sent with the documented OAI-SearchBot/1.0 User-Agent from a non-OpenAI benchmark test IP, so edge IP-verification checks contribute alongside User-Agent rules.
Engineering and SEO checklist for an agent-readable site
Start by ensuring every core product, pricing, and documentation URL renders readable server HTML without JavaScript hydration walls, and verify live HTTP status directly using Parse MCP's fetch_live_pages tool or plain curl checks alongside a browser render.
Next, publish a clean /llms.txt index at both your root domain and docs subdomain, serve native Markdown via Accept: text/markdown or .md twins with HTML alternate Markdown link tags to trim payload weight by ~25x, emit Content-Signal preferences, and publish /.well-known/mcp if you run a remote MCP server.
Agent-readability engineering checklist
| WAF, robots.txt, and Content-Signal | 23.67% test-IP block vs 2.26% robots.txt Disallow | Allow verified OAI-SearchBot IP ranges in your WAF, align robots.txt rules, and set explicit Content-Signal directives. |
| Server-rendered core HTML | 19.67% plain fetch fail; 9 of 10 JS shells empty | Pre-render product, pricing, and documentation copy in initial HTTP 200 HTML so simple HTTP agents never hit empty JS shells. |
| Accept: text/markdown and .md twins | 15.00% Accept: text/markdown; 25.41x token cut | Serve clean Markdown on Accept: text/markdown or .md URL twins and declare HTML alternate Markdown link tags in the document head. |
| /llms.txt and /llms-full.txt | 46.00% host /llms.txt; 62.22% B2B SaaS | Publish a curated /llms.txt table of contents linking directly to canonical Markdown guides and any MCP endpoint. |
| /.well-known/mcp and A2A cards | 2.67% /.well-known/mcp; 1.33% agent cards | Publish structured JSON discovery metadata at /.well-known/mcp.json alongside human-readable /llms.txt links. |
Takeaway
- Plan machine-readable content updates in Parse
- Audit live cited URLs via Parse MCP (fetch_live_pages)
- How to make your site readable for AI agents
- How AI recommends MCP and agent-native tools
- Noindex, JS walls, and canonical mismatches on 378,392 AI-cited pages
- GitHub repos, package registries, and docs architecture in AI search
How we measured
We analyzed 300 top AI-cited hosts and their #1 most-cited URLs selected from September 1 to November 19, 2026 organic citations across ChatGPT Search, Google AI Mode, and Perplexity (90 Developer/Docs, 90 B2B SaaS, 60 Editorial/Media, and 60 E-commerce), probed live via HTTP on November 20, 2026 alongside 13 named reference twin URLs and 49 local Playwright Chromium browser verifications (yielding 238 clean HTTP 200 HTML pages and 62 native Markdown pairs inside the 300-site panel).
- Median token cut from raw HTML to native Markdown
- 25.41x
Across 62 paired cited pages (80,389.5 HTML vs 2,323 Markdown tokens)
- Top 300 cited hosts serving /llms.txt
- 46.00%
138 of 300 at cited host; 50.33% (151) including root domains
- Top 300 sites honoring Accept: text/markdown
- 15.00%
45 of 300 overall; 28.89% in Developer/Docs vs 1.67% in Editorial/Media
- Cited URLs failing plain HTTP fetch
- 19.67%
59 of 300 overall, peaking at 43.33% in E-commerce (26 of 60)
Get the data
Sources
- The /llms.txt specification (llmstxt.org) (opens in a new tab) · accessed November 20, 2026
- Cloudflare: Introducing Markdown for Agents (opens in a new tab) · accessed November 20, 2026
- Cloudflare: Introducing Pay Per Crawl (opens in a new tab) · accessed November 20, 2026
- Cloudflare: Content Signals Policy (opens in a new tab) · accessed November 20, 2026
- Universal Commerce Protocol (UCP) specification (opens in a new tab) · accessed November 20, 2026
- Model Context Protocol specification (opens in a new tab) · accessed November 20, 2026
- Agent2Agent (A2A) Protocol specification (opens in a new tab) · accessed November 20, 2026
- OpenAI: Overview of OpenAI crawlers (OAI-SearchBot and GPTBot) (opens in a new tab) · accessed November 20, 2026
- Microsoft: Playwright MCP server repository (opens in a new tab) · accessed November 20, 2026
- Do generative AI assistants respect robots.txt? Tracing web access beyond visible answers (opens in a new tab) · accessed November 20, 2026