Most websites hand AI assistants a 347 KB wall of JavaScript bundles, inline SVGs, and navigation chrome just to deliver 2,851 tokens of actual copy. Across 62 top AI-cited pages serving native Markdown twins in Parse's November 20, 2026 audit of 300 top-cited domains, offering native Markdown shrank token weight by a median of 25.41× from 80,389.5 raw HTML tokens to 2,323 Markdown tokens (while the 238 clean HTML pages overall carried an 86,772.5-token HTML median).
As we reported in our earlier /llms.txt verdict, /llms.txt is still not a passive search ranking signal on ChatGPT Search or Google AI Mode. Yet our live 300-domain agent readability study shows that 46.00% of top-cited hosts (50.33% including root domains) now ship /llms.txt and 15.00% negotiate Accept: text/markdown, because coding assistants and MCP agents break on 19.67% of plain HTTP HTML fetches. Parse indexes 853,414 active brands across 1,774 markets, backed by 744,460 counted-engine answers (378,499 ChatGPT Search and 365,961 Google AI Mode) and 4,053,878 citations across 73,847 buyer questions in our 90-day public MCP snapshot (August 23 to November 20, 2026).
Median token reduction from raw HTML (80,389.5 tokens) to native Markdown (2,323 tokens) across 62 paired cited pages
Of 300 top-cited hosts serving a valid /llms.txt (50.33% including root domains; 62.22% in B2B SaaS)
Of top 300 sites honoring Accept: text/markdown (15.33% serve .md twins; 9.33% link alternate Markdown)
Of 300 top-cited URLs failing plain HTTP fetch (rising to 43.33% in E-commerce)
How 300 top AI-cited sites compare on llms.txt, Markdown twins, and agent discovery
We pulled the 300 most-cited web hosts across ChatGPT Search, Google AI Mode, and Perplexity from September 1 to November 19, 2026 (90 Developer/Docs, 90 B2B SaaS, 60 Editorial/Media, and 60 E-commerce) and probed each host and its #1 most-cited URL live on November 20, 2026.
The table below measures adoption of /llms.txt, /llms-full.txt, Accept: text/markdown content negotiation, .md twin URLs, alternate Markdown link tags, MCP mentions inside /llms.txt, and /.well-known/mcp* JSON manifests across all four categories.
| Site category | /llms.txt host (host or root) | /llms-full.txt | Accept: text/markdown | .md twin URL | Alternate MD link | MCP in /llms.txt | /.well-known/mcp* |
|---|---|---|---|---|---|---|---|
Developer/Docs (n=90) | 54.44% (67.78%) | 25.56% | 28.89% | 34.44% | 16.67% | 40.00% | 3.33% |
B2B SaaS (n=90) | 62.22% (62.22%) | 16.67% | 17.78% | 12.22% | 11.11% | 38.89% | 5.56% |
Editorial/Media (n=60) | 25.00% (26.67%) | 5.00% | 1.67% | 6.67% | 5.00% | 1.67% | 0.00% |
E-commerce (n=60) | 30.00% (30.00%) | 13.33% | 3.33% | 0.00% | 0.00% | 11.67% | 0.00% |
| All 300 top-cited | 46.00% (50.33%) | 16.33% | 15.00% | 15.33% | 9.33% | 26.33% | 2.67% |
Two operational takeaways jump straight out of that matrix:
- Platform defaults push adoption faster than manual SEO. E-commerce beat Editorial/Media on
/llms.txt(30.00%vs25.00%) and hit11.67%(7of60) MCP mentions inside/llms.txtbecause 7 Shopify merchants automatically expose Shopify's defaultucp.dev/Shop skill/llms.txttemplate. Similarly, Mintlify, ReadMe, and Cloudflare defaults lift Developer/Docs to34.44%.mdtwins and28.89%Accept: text/markdown. - Teams describe MCP in prose but skip machine discovery. Across the
300sites, 26.33% (79of300, including40.00%of Developer/Docs and38.89%of B2B SaaS) mention MCP inside/llms.txt, yet only 2.67% (8of300:cloudflare.com,signoz.io,monday.com,notion.com,attio.com,langfuse.com,sentry.io, anddocs.langchain.com) serve a valid JSON manifest at/.well-known/mcp*, and just 1.33% (4of300:cloudflare.com,monday.com,attio.com,docs.langchain.com) expose an A2A Agent Card.
How much HTML bloat costs AI agents in raw tokens
When an agent fetches a normal web page without a Markdown twin, it doesn't just download your article. Across 238 clean HTTP 200 cited HTML pages in our benchmark, median raw HTML size reached 347,139.5 bytes (86,772.5 estimated tokens at characters / 4), including 54,927 bytes of inline/script tags (<script>) and 22,779 bytes of vector markup (<svg>) compared with just 11,456 bytes (2,850.5 tokens) of stripped visible text.
The bloat multiplier (raw HTML tokens / visible text tokens) exposes why coding assistants burn context windows fast on marketing pages, and how sharply native Markdown endpoints cut waste.
| Category | Clean HTML pages (n) | Median raw HTML tokens | Median visible text tokens | Median bloat multiplier (p90) | Native MD pairs (n) | Median HTML-to-MD savings |
|---|---|---|---|---|---|---|
| Developer/Docs | 78 | 38,300 | 2,456.5 | 13.50× (45.38×) | 38 | 22.55× |
| B2B SaaS | 85 | 92,890 | 2,491.0 | 31.46× (99.95×) | 18 | 38.09× |
| Editorial/Media | 43 | 121,616 | 6,885.0 | 13.51× (74.48×) | 4 | 8.87× |
| E-commerce | 32 | 126,840 | 2,277.5 | 78.62× (167.68×) | 2 | 35.85× |
| All clean pages | 238 | 86,772.5 | 2,850.5 | 24.97× (108.84×) | 62 | 25.41× |
Inspecting individual sites from our 62 Top-300 native Markdown pairs and 13-page Named Reference Twin panel shows the scale of token savings in production:
| Tested URL | Accept: text/markdown / .md twin | Raw HTML tokens | Native Markdown tokens | Token reduction multiplier |
|---|---|---|---|---|
sentry.io/ | Yes / Yes | 156,231 | 244 | 640.29× |
www.cloudflare.com/ | Yes / Yes | 333,772 | 747 | 446.82× |
vercel.com/ | Yes / No | 150,134 | 566 | 265.25× |
supabase.com/ | Yes / Yes | 337,409 | 1,303 | 258.95× |
docs.datadoghq.com/watchdog | Yes / Yes | 197,689 | 921 | 214.65× |
linear.app/ | Yes / Yes | 316,378 | 1,738 | 182.04× |
linear.app/docs | Yes / Yes | 139,433 | 1,166 | 119.58× |
vercel.com/docs | Yes / Yes | 228,461 | 2,482 | 92.05× |
docs.anthropic.com/en/docs/welcome | Yes / Yes | 109,303 | 1,435 | 76.17× |
docs.stripe.com/tax | Yes / Yes | 82,475 | 1,093 | 75.46× |
mintlify.com/docs | Yes / No | 52,158 | 852 | 61.22× |
parse.gl/docs/mcp | Yes / Yes | 51,376 | 4,711 | 10.91× (90.83% saved) |
parse.gl/ai | Yes / Yes | 5,526 | 3,967 | 1.39× (28.21% saved) |
Notice two distinct patterns in that benchmark. Heavy marketing homepages (sentry.io, cloudflare.com, supabase.com, linear.app) drop 182× to 640× when served as native Markdown, whereas developer.avalara.com/products/crossborder showed an extreme 1,893.88× outlier (749,975 raw HTML tokens vs 396 .md tokens). Even on Parse's intentionally lean server-rendered /ai page (5,526 HTML tokens), Markdown trims another 28.21%, while Mintlify-powered parse.gl/docs/mcp shrinks 90.83% (51,376 to 4,711 tokens).
Why plain HTTP fetch fails on 19.67% of cited URLs (and what OAI-SearchBot sees)
Token bloat isn't the worst failure mode. Across all 300 top-cited URLs, 59 (19.67%) failed a plain HTTP fetch completely (43.33% in E-commerce, 28.33% in Editorial/Media, 13.33% in Developer/Docs, and 4.44% in B2B SaaS). This matches our broader crawl telemetry in crawler barriers on cited pages, where 16.99% of 378,392 cited URLs failed simple direct HTTP requests, and our SaaS pricing crawl audit, where 47.3% of active top-SaaS /pricing endpoints hid or blocked prices from plain HTTP fetchers.
Another stealth failure hides behind HTTP 200 status codes: client-rendered JavaScript shells. When we ran local Playwright Chromium across 10 HTTP-200 cited pages whose plain HTTP body yielded fewer than 150 visible characters (<38 tokens), 9 of 10 expanded to 2,897-26,026 rendered characters:
docs.pytorch.org/docs/stable/profiler.htmljumped 473.20× from55plain characters (14tokens) to26,026rendered characters (6,507tokens).bubble.io/jumped 202.42× from86plain characters to17,408rendered characters (4,352tokens).support.zoom.com/hc/en/article?id=zm_kb&sysparm_article=KB0069029jumped 153.29× from87to13,336characters.help.asana.com/s/article/top-asana-integrationsjumped 103.57× from70to7,250characters.developer.apple.com/documentation/arkitjumped 83.86× from37to3,103characters, whilenordstromrack.com/went from0plain characters to6,276rendered characters.
At the same time, WAF bot-filtering behaves very differently from robots.txt. Across 265 valid robots.txt files on the 300 hosts, only 2.26% (6 of 265) explicitly disallowed OAI-SearchBot (6.04% disallowed GPTBot, 7.55% ClaudeBot, 8.30% CCBot, while 27 carried Content-Signal directives and 3 sent Content-Signal HTTP headers). Yet 23.67% (71 of 300) of cited URLs blocked HTTP requests carrying the OAI-SearchBot/1.0 User-Agent from our benchmark test IP (46.67% in E-commerce, 35.00% in Editorial/Media, 14.44% in B2B SaaS, 10.00% in Developer/Docs), alongside 2 explicit HTTP 402 pay-per-crawl responses (travelandleisure.com and seriouseats.com).
Our 23.67% block rate measured HTTP responses to OpenAI's documented OAI-SearchBot/1.0 User-Agent sent from a standard benchmark IP. Production OpenAI search crawlers operating from verified OpenAI IP ranges can pass edge allowlists that block unverified IPs, while third-party coding agents, MCP web fetchers, and custom scrapers hit those exact plain-HTTP walls.
Review our robots.txt AI crawler configuration guide and AI crawler diagnostic tools comparison if your analytics show flat crawl logs despite open robots.txt rules.
Step-by-step: ship llms.txt, .md twins, and Accept: text/markdown
Copying Parse's production architecture (https://parse.gl/llms.txt) takes one afternoon and works across Next.js, Cloudflare Workers, Nginx, or documentation engines like Mintlify.
Fix SSR HTML and WAF allowlists first. Ensure core product, docs, and /pricing pages render full specs in plain HTTP 200 HTML without client-only hydration walls, and verify verified search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) aren't challenged by aggressive CDN rules.
Serve .md twins and Accept: text/markdown negotiation. Map /pricing.md and Accept: text/markdown on key marketing, blog, and research routes to strip layout chrome while preserving canonical links and Vary: Accept cache separation.
Publish a job-grouped /llms.txt index. Open with an H1 and one-sentence blockquote, organize canonical .md links by agent job (Evaluate, Start, Operate, Research), and add /llms-full.txt for docs hubs.
Expose discovery tags and /.well-known/mcp. Link the Markdown twin in HTML headers and publish /.well-known/mcp.json if you operate a remote MCP server so agents discover endpoints programmatically.
Here is the exact HTTP header contract Parse's Markdown endpoints return so CDNs never serve raw Markdown to human browsers while keeping search equity tied to the HTML URL:
HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept
Link: <https://parse.gl/pricing>; rel="canonical"
Cache-Control: public, max-age=300, s-maxage=3600
When serving HTML, advertise the twin in <head> with an alternate link tag so parsers find the clean Markdown endpoint in one hop without guessing URL rules:
<link rel="alternate" type="text/markdown" href="https://parse.gl/pricing.md" />
Notice one subtle caching rule in Parse's live routing: /index.md, /ai.md, /pricing.md, /auth.md, /blog/<slug>.md, and /research/<slug>.md all serve clean Markdown directly, while Accept: text/markdown negotiation runs selectively on static reference routes (/index, /ai, /pricing, /auth) rather than personalized routes so edge caches never leak a Markdown payload onto cached HTML URLs. Pair that with Vary: Accept on every negotiated response.
How to verify your site with four curl checks
Don't guess whether your CDN or Next.js middleware handles agent requests cleanly. Run these four curl probes against your live domain (shown here against parse.gl and your target /pricing route):
# 1. Confirm /llms.txt returns HTTP 200 text/plain or text/markdown (never HTML 404 fallback)
curl -i -sS https://parse.gl/llms.txt | head -n 20
# 2. Verify Accept: text/markdown content negotiation + Vary: Accept cache header
curl -i -sS -H "Accept: text/markdown" https://parse.gl/pricing | head -n 20
# 3. Verify explicit .md twin + canonical Link header
curl -i -sS https://parse.gl/pricing.md | head -n 20
# 4. Compare plain HTTP byte delivery when identifying as OAI-SearchBot
curl -i -sS -A "OAI-SearchBot/1.0" https://yourdomain.com/pricing | head -n 20
Watch out for two traps in step 1 and step 4: SPAs frequently return HTTP 200 HTML shell pages at /llms.txt instead of 404 or Markdown (our audit specifically filtered out HTML impostors), and a 403 or sub-5 KB body on step 4 signals an immediate WAF or JS wall blocking plain HTTP agents.
Run this workflow with your agent
Before rebuilding templates blind, connect Claude Code, Cursor, Codex, or ChatGPT to Parse MCP (https://agent.parse.gl/mcp) and inspect which pages on your domain AI engines already cite versus where fetch barriers break extraction. Index discovery (check_brand, search_cited_pages) and logging notes (log_work, list_work) are free; fetch_live_pages quotes at $0.01 per page inside your $5 signup credit and monthly budget cap.
Run check_brand for "<our domain>" and search_cited_pages to pull our top 10
owned URLs currently cited across ChatGPT Search and Google AI Mode.
Quote and call fetch_live_pages on those 5 highest-cited URLs plus our /pricing
and /docs landing pages to check whether plain HTTP extraction recovers full text.
Save a prioritized /llms.txt, .md twin, and SSR fix checklist via log_work.
Once your agent logs the baseline via log_work, review it across sessions with list_work, wire the matching playbooks from our Claude Code and Cursor MCP workflows guide, or compare extraction-friendly stacks in our 14 marketing MCP servers benchmark. Explore content planning workflows and impact tracking to measure what changes after you ship.
Want to find third-party pages AI already cites that leave out your brand? find pages to pitch with Parse.