# HTML token bloat vs. Markdown twins and llms.txt across 300 AI-cited sites

> Across the 300 most-cited Developer/Docs, B2B SaaS, Editorial/Media, and E-commerce domains in Parse's September to November 2026 index, 46.00% serve a real /llms.txt and 15.00% honor Accept: text/markdown content negotiation. Serving native Markdown cuts median page weight 25.41x across 62 paired cited pages, while 19.67% of top-cited URLs fail plain HTTP retrieval and 23.67% reject requests using the OAI-SearchBot User-Agent from a non-OpenAI benchmark IP.

HTML: https://parse.gl/research/agent-readability-html-vs-markdown-and-llms-txt · Site index: https://parse.gl/llms.txt · Published: 2026-11-20

How much of the web that AI search engines cite is readable without headless browsers or HTML boilerplate stripping, and how far have /llms.txt, Markdown twins, and agent discovery endpoints spread across top-cited domains?

We extracted the 300 most-cited hosts and each host's top-cited URL from organic ChatGPT Search, Google AI Mode, and Perplexity answers recorded September 1 through November 19, 2026 (90 Developer/Docs, 90 B2B SaaS, 60 Editorial/Media, and 60 E-commerce), then probed every host live on November 20, 2026 for /llms.txt, /llms-full.txt, Accept: text/markdown responses, .md twins, /.well-known/mcp manifests, robots.txt AI rules, OAI-SearchBot access, HTML token bloat, and headless Chromium rendering across 49 browser-checked pages alongside 13 named reference twin pages.

## Key figures

- **Median token cut from raw HTML to native Markdown:** 25.41x (Across 62 paired cited pages (80,389.5 HTML vs 2,323 Markdown tokens))
- **Top 300 cited hosts serving /llms.txt:** 46.00% (138 of 300 at cited host; 50.33% (151) including root domains)
- **Top 300 sites honoring Accept: text/markdown:** 15.00% (45 of 300 overall; 28.89% in Developer/Docs vs 1.67% in Editorial/Media)
- **Cited URLs failing plain HTTP fetch:** 19.67% (59 of 300 overall, peaking at 43.33% in E-commerce (26 of 60))

## 46.00% of top AI-cited hosts serve /llms.txt, while 15.00% honor Accept: text/markdown

Across the 300 most-cited web hosts in Parse's index, 138 (46.00%) serve a valid, non-HTML /llms.txt index at the cited host, and 151 (50.33%) serve one on either the cited host or its parent root domain. Another 49 hosts (16.33%) publish a consolidated /llms-full.txt file.

Page-level Markdown delivery lags directory-level /llms.txt adoption by roughly 3-to-1. Exactly 45 of 300 hosts (15.00%) honor Accept: text/markdown HTTP content negotiation (31 on the homepage and 45 on the top-cited URL), 46 of 300 (15.33%) expose a live .md URL twin on their top-cited page, 63 of 300 (21.00%) support at least one of those two mechanisms (28 support both), and 28 of 300 (9.33%) declare HTML alternate Markdown link tags in the document head.

**Machine-readable web standard adoption across 300 top AI-cited hosts (%)**

| Item | Value |
| --- | --- |
| Developer/Docs: /llms.txt (host) | 54.44% (49 of 90) |
| Developer/Docs: /llms-full.txt | 25.56% (23 of 90) |
| Developer/Docs: Accept text/markdown | 28.89% (26 of 90) |
| Developer/Docs: .md URL twin | 34.44% (31 of 90) |
| B2B SaaS: /llms.txt (host) | 62.22% (56 of 90) |
| B2B SaaS: /llms-full.txt | 16.67% (15 of 90) |
| B2B SaaS: Accept text/markdown | 17.78% (16 of 90) |
| B2B SaaS: .md URL twin | 12.22% (11 of 90) |
| E-commerce: /llms.txt (host) | 30.00% (18 of 60) |
| E-commerce: /llms-full.txt | 13.33% (8 of 60) |
| E-commerce: Accept text/markdown | 3.33% (2 of 60) |
| E-commerce: .md URL twin | 0.00% (0 of 60) |
| Editorial/Media: /llms.txt (host) | 25.00% (15 of 60) |
| Editorial/Media: /llms-full.txt | 5.00% (3 of 60) |
| Editorial/Media: Accept text/markdown | 1.67% (1 of 60) |
| Editorial/Media: .md URL twin | 6.67% (4 of 60) |

_Live HTTP probes run November 20, 2026 across 90 Developer/Docs, 90 B2B SaaS, 60 Editorial/Media, and 60 E-commerce sites._

> Nearly half of top AI-cited websites now expose /llms.txt, while only one in seven deliver native Markdown per page.

Related: [How to make your site readable for AI agents](https://parse.gl/blog/make-your-site-readable-for-ai-agents)

## B2B SaaS leads /llms.txt at 62.22%, while Shopify templates push E-commerce ahead of Editorial/Media

Adoption splits sharply by site category. B2B SaaS leads cited-host /llms.txt adoption at 62.22% (56 of 90 hosts). Developer/Docs posts 54.44% at the cited host (49 of 90) and rises to 67.78% (61 of 90) when parent root domains are checked, because several support and docs subdomains inherit a root /llms.txt. Developer/Docs also leads page-level Markdown twins at 34.44% (31 of 90) and Accept: text/markdown at 28.89% (26 of 90).

E-commerce /llms.txt adoption reaches 30.00% (18 of 60), beating Editorial/Media at 25.00% (15 of 60). Platform defaults explain why: 7 Shopify merchants in the 60-store cohort (sunhomesaunas.com, americantall.com, christydawn.com, everlane.com, girlfriend.com, mudjeans.com, and matethelabel.com) expose Shopify's default Universal Commerce Protocol (UCP) Shop Skill /llms.txt. Outside those storefronts, E-commerce has 0.00% .md twins (0 of 60) and 3.33% Accept: text/markdown support (2 of 60).

**Machine-readable endpoint adoption across 300 top AI-cited domains by category**

| Category | Sites | /llms.txt host (%) | /llms.txt host or root (%) | /llms-full.txt (%) | Accept Markdown (%) | .md twin (%) | Alt link tag (%) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| All 300 cited sites | 300 | 46 | 50.33 | 16.33 | 15 | 15.33 | 9.33 |
| B2B SaaS | 90 | 62.22 | 62.22 | 16.67 | 17.78 | 12.22 | 11.11 |
| Developer/Docs | 90 | 54.44 | 67.78 | 25.56 | 28.89 | 34.44 | 16.67 |
| E-commerce | 60 | 30 | 30 | 13.33 | 3.33 | 0 | 0 |
| Editorial/Media | 60 | 25 | 26.67 | 5 | 1.67 | 6.67 | 5 |

_Counts and percentages from the November 20, 2026 live HTTP benchmark across all 300 top-cited hosts._

> Default platform rollouts like Shopify UCP can push an entire vertical past Editorial/Media on /llms.txt overnight, even when per-page Markdown stays near zero.

## 26.33% of sites reference MCP inside /llms.txt, yet only 2.67% publish /.well-known/mcp manifests

Companies are already documenting Model Context Protocol (MCP) endpoints inside /llms.txt far faster than they ship formal well-known JSON discovery endpoints. Across the 300 audited sites, 79 hosts (26.33%) explicitly mention or link to MCP inside their /llms.txt file, led by Developer/Docs (40.00%, 36 of 90) and B2B SaaS (38.89%, 35 of 90), alongside 11.67% in E-commerce (7 of 60 via Shopify UCP templates) and 1.67% in Editorial/Media (1 of 60).

By contrast, only 8 of 300 hosts (2.67%) return a valid JSON MCP discovery manifest at /.well-known/mcp or /.well-known/mcp.json: cloudflare.com, signoz.io, monday.com, notion.com, attio.com, langfuse.com, sentry.io, and docs.langchain.com (3.33% of Developer/Docs and 5.56% of B2B SaaS). Only 4 of 300 hosts (1.33%: cloudflare.com, monday.com, attio.com, and docs.langchain.com) publish a valid Agent-to-Agent (A2A) card at /.well-known/agent-card.json or /.well-known/agent.json.

**Agent and MCP discovery mechanisms across 300 top AI-cited sites**

| Item | Value |
| --- | --- |
| MCP referenced inside /llms.txt | 26.33% (79 of 300) |
| HTML alternate Markdown link tags | 9.33% (28 of 300) |
| Valid /.well-known/mcp or mcp.json | 2.67% (8 of 300) |
| Valid /.well-known/agent-card.json | 1.33% (4 of 300) |

_Share of hosts returning valid non-HTML, non-bot-challenge payloads at each discovery path (Nov 20, 2026)._

> Most agent discovery today happens through plain-text /llms.txt links rather than /.well-known/mcp JSON files.

Related: [How AI recommends MCP and agent-native tools](https://parse.gl/research/how-ai-recommends-mcp-and-agent-native-tools) · [Connect your agent directly to Parse MCP](https://parse.gl/mcp)

## Clean cited HTML pages carry 24.97x token bloat, peaking at 78.62x in E-commerce

When an AI coding agent or retrieval pipeline pulls a raw HTML page without a Markdown twin, most of the context window goes to page chrome rather than visible content. Across the 238 top-cited URLs in our 300-site panel that returned a clean HTTP 200 HTML page (at least 200 visible characters and no bot challenge), median raw HTML payload measured 347,139.5 bytes (86,772.5 estimated tokens at 4 characters per token) versus just 11,456 bytes (2,850.5 tokens) of stripped visible text, a median bloat multiplier of 24.97x (and 108.84x at the 90th percentile).

Inlining accounts for most of that tax: clean pages carried a median of 54,927 bytes in script tags, 22,779 bytes in inline SVG elements, and 2,253 bytes in style blocks. Storefronts and SaaS marketing pages carry the steepest overhead: E-commerce reaches a median bloat multiplier of 78.62x (126,840 raw HTML tokens vs 2,277.5 visible tokens; 167.68x at p90), and B2B SaaS reaches 31.46x (92,890 vs 2,491 tokens; 99.95x at p90), compared with 13.51x in Editorial/Media (121,616 vs 6,885 tokens) and 13.50x in Developer/Docs (38,300 vs 2,456.5 tokens).

**Raw HTML tokens vs stripped visible text tokens across 238 clean cited pages**

| Category | Clean pages | Median raw HTML tokens | Median visible tokens | Median bloat (x) | p90 bloat (x) |
| --- | --- | --- | --- | --- | --- |
| All clean cited pages | 238 | 86772.5 | 2850.5 | 24.97 | 108.84 |
| E-commerce | 32 | 126840 | 2277.5 | 78.62 | 167.68 |
| B2B SaaS | 85 | 92890 | 2491 | 31.46 | 99.95 |
| Editorial/Media | 43 | 121616 | 6885 | 13.51 | 74.48 |
| Developer/Docs | 78 | 38300 | 2456.5 | 13.5 | 45.38 |

_Includes all HTTP 200 non-challenge pages with at least 200 visible characters; token estimates use UTF-8 decoded characters divided by 4._

> On median E-commerce and B2B SaaS citations, over 96% to 98% of raw HTML characters are scripts, inline SVGs, styles, and layout markup rather than readable text.

## Native Markdown cuts cited-page weight 25.41x overall and up to 640x on SaaS homepages

Across the 62 HTTP 200 cited URLs in the 300-site panel that return native Markdown via Accept: text/markdown or a .md URL twin, median page weight falls 25.41x, from 80,389.5 raw HTML tokens to 2,323 native Markdown tokens. Savings reach a median of 38.09x across 18 B2B SaaS pairs, 35.85x across 2 E-commerce pairs, 22.55x across 38 Developer/Docs pairs, and 8.87x across 4 Editorial/Media pairs.

Named production deployments show the practical payoff: sentry.io drops 640.29x (156,231 HTML tokens to 244 Markdown tokens), cloudflare.com drops 446.82x (333,772 to 747), vercel.com drops 265.25x (150,134 to 566), supabase.com drops 258.95x (337,409 to 1,303), docs.datadoghq.com/watchdog drops 214.65x (197,689 to 921), linear.app drops 182.04x (316,378 to 1,738), and developer.avalara.com drops 1,893.88x (749,975 to 396) where raw HTML embedded huge serialized script payloads. In our 13-page named reference twin panel, Parse's own docs page (parse.gl/docs/mcp) cuts tokens 10.91x (51,376 to 4,711; 90.83% reduction), alongside Mintlify docs (61.22x), Anthropic docs (76.17x), Stripe tax docs (75.46x), and Linear docs (119.58x).

**Exact HTML vs native Markdown token counts on paired Top-300 and reference pages**

| Host | Category | Raw HTML tokens | Visible tokens | Native MD tokens | Savings (x) |
| --- | --- | --- | --- | --- | --- |
| developer.avalara.com | Developer/Docs | 749975 | 712763 | 396 | 1893.88 |
| sentry.io | Developer/Docs | 156231 | 4995 | 244 | 640.29 |
| cloudflare.com | B2B SaaS | 333772 | 1815 | 747 | 446.82 |
| vercel.com | B2B SaaS | 150134 | 1009 | 566 | 265.25 |
| supabase.com | B2B SaaS | 337409 | 2341 | 1303 | 258.95 |
| opentelemetry.io | Developer/Docs | 101772 | 3588 | 464 | 219.34 |
| docs.datadoghq.com | Developer/Docs | 197689 | 14636 | 921 | 214.65 |
| linear.app | B2B SaaS | 316378 | 2690 | 1738 | 182.04 |
| openobserve.ai | B2B SaaS | 218249 | 3700 | 1536 | 142.09 |
| asana.com | B2B SaaS | 236061 | 2668 | 1682 | 140.35 |
| developer.harness.io | Developer/Docs | 244363 | 1989 | 1953 | 125.12 |
| freshworks.com | B2B SaaS | 134723 | 1047 | 1315 | 102.45 |

_Switch views between the top 300 paired Markdown savings and the 13-page Named Reference Twin panel (Nov 20, 2026)._

> Serving clean Markdown reduces page size by 90% to 99% on real production docs and SaaS pages while keeping core content intact.

## 19.67% of top AI-cited URLs fail plain HTTP retrieval, led by E-commerce at 43.33%

Plain HTTP GET requests using a standard Chrome User-Agent failed on 59 of the 300 top-cited URLs (19.67%). Explicit HTTP 403 blocks accounted for 41 failures, followed by 9 HTTP 200 pages serving JavaScript shells with fewer than 200 visible characters, 2 explicit bot-challenge interstitials (zoho.com and docs.pytorch.org), 2 HTTP 429 rate limits (chewy.com and wayfair.com), 2 HTTP 402 payment walls (travelandleisure.com and seriouseats.com), 1 HTTP 401 (reuters.com), 1 HTTP 400 (developers.meta.com), and 1 header-overflow failure (finance.yahoo.com).

Failure rates track industry stack choices closely: 43.33% of E-commerce citations (26 of 60) and 28.33% of Editorial/Media citations (17 of 60) failed plain retrieval, versus 13.33% in Developer/Docs (12 of 90) and only 4.44% in B2B SaaS (4 of 90).

**Plain HTTP retrieval failure rate across 300 top AI-cited URLs**

| Item | Value |
| --- | --- |
| E-commerce | 43.33% (26 of 60) |
| Editorial/Media | 28.33% (17 of 60) |
| All 300 cited URLs | 19.67% (59 of 300) |
| Developer/Docs | 13.33% (12 of 90) |
| B2B SaaS | 4.44% (4 of 90) |

_Includes HTTP >= 400 errors, connection failures, bot-challenge pages, and HTTP 200 empty JS shells (<200 visible chars)._

> One in five top-cited URLs fails simple direct HTTP fetching, hitting nearly half of E-commerce citations.

Related: [Noindex, JS walls, and canonical mismatches on 378,392 AI-cited pages](https://parse.gl/research/crawler-barriers-noindex-and-canonicals-on-cited-pages) · [How often do AI-cited pages block access?](https://parse.gl/research/how-often-ai-citations-block-access)

## Headless Chromium recovers 9 of 10 empty JS shells up to 473.2x, while server-rendered pages stay flat

To verify whether HTTP 200 thin responses were client-side JavaScript shells rather than short articles, we executed headless Playwright Chromium across all 10 reachable JS-shell suspects (<500 plain visible characters) alongside a stratified sample across all four categories (49 total Playwright measurements recorded, including 37 valid non-error stratified pages plus 2 stratified timeouts/protocol errors on eileenfisher.com and bestbuy.com).

On 9 of the 10 JS-shell suspects, rendering JavaScript restored full page text from under 150 plain characters to between 2,897 and 26,026 DOM characters: docs.pytorch.org jumped 473.20x (55 to 26,026 chars), pro.globalforestwatch.org jumped 218.74x (23 to 5,031), bubble.io jumped 202.42x (86 to 17,408), support.zoom.com jumped 153.29x (87 to 13,336), help.asana.com jumped 103.57x (70 to 7,250), developer.apple.com jumped 83.86x (37 to 3,103), siemens.com jumped 67.08x (131 to 8,788), wearpact.com jumped 21.46x (135 to 2,897), and nordstromrack.com moved from 0 plain chars to 6,276 rendered chars. Only help.draftkings.com stayed thin at 320 chars (2.69x). Across the 37 normal stratified pages, the median rendered-to-plain ratio was 0.88x (confirming plain HTTP already captured their main copy, except partial hydrations like rtings.com at 14.97x).

**Playwright Chromium recovery across all 10 HTTP 200 JS-shell suspects**

| Host | Category | Plain HTTP chars | Chromium chars | Render gain |
| --- | --- | --- | --- | --- |
| docs.pytorch.org | Developer/Docs | 55 | 26026 | 473.20x |
| bubble.io | B2B SaaS | 86 | 17408 | 202.42x |
| support.zoom.com | Developer/Docs | 87 | 13336 | 153.29x |
| siemens.com | B2B SaaS | 131 | 8788 | 67.08x |
| help.asana.com | Developer/Docs | 70 | 7250 | 103.57x |
| nordstromrack.com | E-commerce | 0 | 6276 | 0 to 6,276 chars |
| pro.globalforestwatch.org | Developer/Docs | 23 | 5031 | 218.74x |
| developer.apple.com | Developer/Docs | 37 | 3103 | 83.86x |
| wearpact.com | E-commerce | 135 | 2897 | 21.46x |
| help.draftkings.com | Developer/Docs | 119 | 320 | 2.69x |

_Compares plain HTTP stripped visible characters against rendered Playwright DOM innerText characters (Nov 20, 2026)._

> Client-side rendering hides entire help centers and docs pages from plain HTTP fetchers unless the site exposes an HTML SSR layer or a .md twin like developer.apple.com and docs.pytorch.org do.

## 23.67% of cited URLs blocked OAI-SearchBot requests from a test IP, vs 2.26% disallowing it in robots.txt

Live HTTP requests sent with OpenAI's documented OAI-SearchBot/1.0 User-Agent from a non-OpenAI benchmark test IP were blocked on 23.67% of top-cited URLs (71 of 300: 36 HTTP 403, 21 challenge-403, 6 timeouts, 2 challenge-200, 2 HTTP 429, 1 challenge-429, 2 HTTP 402, and 1 HTTP 401) and 24.33% of homepages (73 of 300). Cited-URL blocks reached 46.67% in E-commerce (28 of 60) and 35.00% in Editorial/Media (21 of 60), versus 14.44% in B2B SaaS (13 of 90) and 10.00% in Developer/Docs (9 of 90).

Methodological boundary: because these test probes sent the OAI-SearchBot User-Agent from a non-OpenAI benchmark IP, edge WAF IP-verification and anti-spoofing rules contribute alongside pure User-Agent blocks. Production crawlers connecting from OpenAI's published IP ranges can pass IP-verified allowlists. Even with that boundary, comparing live WAF behavior against the 265 readable robots.txt files shows a 10.5x gap: only 6 of 265 hosts (2.26%) explicitly disallow OAI-SearchBot at /, compared with 16 (6.04%) disallowing GPTBot, 20 (7.55%) disallowing ClaudeBot, and 22 (8.30%) disallowing CCBot. Meanwhile, 2 Editorial/Media sites (travelandleisure.com and seriouseats.com) returned HTTP 402 Pay Per Crawl responses, 27 of 265 robots.txt files contained Content-Signal directives, and 3 hosts (freshworks.com, retool.com, theimagecdn.com) emitted Content-Signal HTTP headers.

**Declared robots.txt AI rules vs live HTTP and OAI-SearchBot test-IP blocks**

| Crawler | Mentioned in robots.txt | Disallow: / count | Disallow: / (%) |
| --- | --- | --- | --- |
| ccbot | 44 | 22 | 8.3 |
| claudebot | 57 | 20 | 7.55 |
| google-extended | 51 | 17 | 6.42 |
| gptbot | 56 | 16 | 6.04 |
| perplexitybot | 52 | 15 | 5.66 |
| chatgpt-user | 46 | 7 | 2.64 |
| oai-searchbot | 45 | 6 | 2.26 |

_Switch views between robots.txt Disallow: / rates across 265 readable files and category-level HTTP block rates across all 300 sites._

> WAF rules and IP-verification gates decide whether third-party agents and unverified crawlers can read your pages far more often than robots.txt does.

## Study scope, counting rules, and methodological boundaries

Target selection reflects Parse's observed organic citations from September 1 through November 19, 2026 across ChatGPT Search, Google AI Mode, and Perplexity. We selected the top 90 Developer/Docs, top 90 B2B SaaS, top 60 Editorial/Media, and top 60 E-commerce hosts deduplicated by root entity (keeping individual hosts on multi-tenant publishing platforms such as github.io, readthedocs.io, mintlify.app, and medium.com), along with each host's most-cited non-binary URL.

All HTTP probes and 49 headless Playwright Chromium verifications ran live on November 20, 2026 alongside 13 named reference twin URLs. Token estimates use UTF-8 decoded characters divided by 4. Clean HTML bloat covers the 238 top-cited URLs returning HTTP 200 non-challenge HTML with at least 200 visible characters; native Markdown savings cover the 62 paired top-cited URLs returning non-HTML Markdown via Accept: text/markdown or a .md twin. OAI-SearchBot HTTP block rates reflect probes sent with the documented OAI-SearchBot/1.0 User-Agent from a non-OpenAI benchmark test IP, so edge IP-verification checks contribute alongside User-Agent rules.

## Engineering and SEO checklist for an agent-readable site

Start by ensuring every core product, pricing, and documentation URL renders readable server HTML without JavaScript hydration walls, and verify live HTTP status directly using Parse MCP's fetch_live_pages tool or plain curl checks alongside a browser render.

Next, publish a clean /llms.txt index at both your root domain and docs subdomain, serve native Markdown via Accept: text/markdown or .md twins with HTML alternate Markdown link tags to trim payload weight by ~25x, emit Content-Signal preferences, and publish /.well-known/mcp if you run a remote MCP server.

**Agent-readability engineering checklist**

| Surface | Study benchmark | Engineering action |
| --- | --- | --- |
| Server-rendered core HTML | 19.67% plain fetch fail; 9 of 10 JS shells empty | Pre-render product, pricing, and documentation copy in initial HTTP 200 HTML so simple HTTP agents never hit empty JS shells. |
| /llms.txt and /llms-full.txt | 46.00% host /llms.txt; 62.22% B2B SaaS | Publish a curated /llms.txt table of contents linking directly to canonical Markdown guides and any MCP endpoint. |
| Accept: text/markdown and .md twins | 15.00% Accept: text/markdown; 25.41x token cut | Serve clean Markdown on Accept: text/markdown or .md URL twins and declare HTML alternate Markdown link tags in the document head. |
| /.well-known/mcp and A2A cards | 2.67% /.well-known/mcp; 1.33% agent cards | Publish structured JSON discovery metadata at /.well-known/mcp.json alongside human-readable /llms.txt links. |
| WAF, robots.txt, and Content-Signal | 23.67% test-IP block vs 2.26% robots.txt Disallow | Allow verified OAI-SearchBot IP ranges in your WAF, align robots.txt rules, and set explicit Content-Signal directives. |

> Pair server-rendered HTML with /llms.txt and native Markdown twins so both search crawlers and context-constrained coding agents read your site cleanly.

Related: [Plan machine-readable content updates in Parse](https://parse.gl/use-cases/plan-content) · [Audit live cited URLs via Parse MCP (fetch_live_pages)](https://parse.gl/mcp) · [How to make your site readable for AI agents](https://parse.gl/blog/make-your-site-readable-for-ai-agents) · [How AI recommends MCP and agent-native tools](https://parse.gl/research/how-ai-recommends-mcp-and-agent-native-tools) · [Noindex, JS walls, and canonical mismatches on 378,392 AI-cited pages](https://parse.gl/research/crawler-barriers-noindex-and-canonicals-on-cited-pages) · [GitHub repos, package registries, and docs architecture in AI search](https://parse.gl/research/github-repos-and-docs-architecture-in-ai-search)

## Method

We analyzed 300 top AI-cited hosts and their #1 most-cited URLs selected from September 1 to November 19, 2026 organic citations across ChatGPT Search, Google AI Mode, and Perplexity (90 Developer/Docs, 90 B2B SaaS, 60 Editorial/Media, and 60 E-commerce), probed live via HTTP on November 20, 2026 alongside 13 named reference twin URLs and 49 local Playwright Chromium browser verifications (yielding 238 clean HTTP 200 HTML pages and 62 native Markdown pairs inside the 300-site panel).

## Data

- [Full study dataset (CSV)](https://parse.gl/research/agent-readability-html-vs-markdown-and-llms-txt/dataset.csv): All 318 data rows: 5 category summaries, all 300 top AI-cited hosts with live endpoint measurements, and 13 named reference twin pages.

## Sources

- [The /llms.txt specification (llmstxt.org)](https://llmstxt.org/)
- [Cloudflare: Introducing Markdown for Agents](https://blog.cloudflare.com/markdown-for-agents/)
- [Cloudflare: Introducing Pay Per Crawl](https://blog.cloudflare.com/introducing-pay-per-crawl/)
- [Cloudflare: Content Signals Policy](https://blog.cloudflare.com/content-signals-policy/)
- [Universal Commerce Protocol (UCP) specification](https://ucp.dev/)
- [Model Context Protocol specification](https://modelcontextprotocol.io/docs/2026-07-28/getting-started/intro)
- [Agent2Agent (A2A) Protocol specification](https://a2a-protocol.org/latest/)
- [OpenAI: Overview of OpenAI crawlers (OAI-SearchBot and GPTBot)](https://developers.openai.com/api/docs/bots)
- [Microsoft: Playwright MCP server repository](https://github.com/microsoft/playwright-mcp)
- [Do generative AI assistants respect robots.txt? Tracing web access beyond visible answers](https://arxiv.org/abs/2607.14447)

## For agents

- [Study dataset (CSV)](https://parse.gl/research/agent-readability-html-vs-markdown-and-llms-txt/dataset.csv)
- Any brand's shares and rivals, with 95% ranges: `https://agent.parse.gl/v1/tools/check_brand?subject=%3Cbrand-domain%3E`
- Sample answers in a market: `https://agent.parse.gl/v1/tools/search_answers?args=%7B%22filter%22%3A%7B%22market%22%3A%5B%22%3Cmkt_id%3E%22%5D%7D%7D`
- Cited pages in a market: `https://agent.parse.gl/v1/tools/search_cited_pages?args=%7B%22filter%22%3A%7B%22market%22%3A%5B%22%3Cmkt_id%3E%22%5D%7D%7D`
- Search the index: `https://parse.gl/api/public/search?q=<words>`

Reuse: Figures and tables on this page are free to republish with attribution: a link to https://parse.gl/research/agent-readability-html-vs-markdown-and-llms-txt beside them. A standard followed link (dofollow) is preferred.

- No account: Public figures and up to 5 examples per list, about 120 brands a day per address.
- Free account: $5 in credit, your brand's first check (20 buyer questions) and a free weekly check.
- After your first top-up: Exports and deeper search results. The first top-up starts at $5.
- Live checks: $0.01 per question per engine ($0.05 on Gemini and Claude).
- Full public answers and pages: 25 free per day, then $0.005 each.
- Public answers older than 90 days: $0.001 each the first time your workspace counts or lists one, then yours.
- An agent can save work before signup: `POST https://agent.parse.gl/v1/workspaces`
- Connect an assistant: `https://agent.parse.gl/mcp`

[Prices](https://parse.gl/pricing.md) · [Map of the site](https://parse.gl/llms.txt)
