Loading articles

Put your agents to work growing your AI visibility.

Research, act and build on what your team has tried, with Parse keeping the shared record.

Start free in your browser.
Parse

Index

  • Brands
  • Markets
  • Topics
  • How we measure

For agents

  • Docs
  • Connect
  • Workflows
  • llms.txt

Parse

  • Use cases
  • Pricing
  • Research
  • Blog
  • Compare

Company

  • About
  • Contact
  • Corrections

© 2026 Parse, Inc. All rights reserved.

Privacy PolicyTerms of Service
Parse
IndexPricing
Sign in
  1. Blog
Blog
Make your site readable for AI agents: 25.4× lighter pages with Markdown and llms.txt

Make your site readable for AI agents: 25.4× lighter pages with Markdown and llms.txt

Published Nov 20, 2026Updated Nov 21, 2026

On this page
  • How 300 top AI-cited sites compare on llms.txt, Markdown twins, and agent discovery
  • How much HTML bloat costs AI agents in raw tokens
  • Why plain HTTP fetch fails on 19.67% of cited URLs (and what OAI-SearchBot sees)
  • Step-by-step: ship llms.txt, .md twins, and Accept: text/markdown
  • How to verify your site with four curl checks
  • Run this workflow with your agent
  • Sources
  • More in Citation strategy
  • How 300 top AI-cited sites compare on llms.txt, Markdown twins, and agent discovery
  • How much HTML bloat costs AI agents in raw tokens
  • Why plain HTTP fetch fails on 19.67% of cited URLs (and what OAI-SearchBot sees)
  • Step-by-step: ship llms.txt, .md twins, and Accept: text/markdown
  • How to verify your site with four curl checks
  • Run this workflow with your agent
  • Sources
  • More in Citation strategy

Reviewed by

Dimitry ApollonskyFounder, Parse

What reviewed means

Most websites hand AI assistants a 347 KB wall of JavaScript bundles, inline SVGs, and navigation chrome just to deliver 2,851 tokens of actual copy. Across 62 top AI-cited pages serving native Markdown twins in Parse's November 20, 2026 audit of 300 top-cited domains, offering native Markdown shrank token weight by a median of 25.41× from 80,389.5 raw HTML tokens to 2,323 Markdown tokens (while the 238 clean HTML pages overall carried an 86,772.5-token HTML median).

As we reported in our earlier /llms.txt verdict, /llms.txt is still not a passive search ranking signal on ChatGPT Search or Google AI Mode. Yet our live 300-domain agent readability study shows that 46.00% of top-cited hosts (50.33% including root domains) now ship /llms.txt and 15.00% negotiate Accept: text/markdown, because coding assistants and MCP agents break on 19.67% of plain HTTP HTML fetches. Parse indexes 853,414 active brands across 1,774 markets, backed by 744,460 counted-engine answers (378,499 ChatGPT Search and 365,961 Google AI Mode) and 4,053,878 citations across 73,847 buyer questions in our 90-day public MCP snapshot (August 23 to November 20, 2026).

Median token reduction from raw HTML (80,389.5 tokens) to native Markdown (2,323 tokens) across 62 paired cited pages

25.41×
Parse Live Audit, Nov 2026

Of 300 top-cited hosts serving a valid /llms.txt (50.33% including root domains; 62.22% in B2B SaaS)

46.00%
Parse Live Audit, Nov 2026

Of top 300 sites honoring Accept: text/markdown (15.33% serve .md twins; 9.33% link alternate Markdown)

15.00%
Parse Live Audit, Nov 2026

Of 300 top-cited URLs failing plain HTTP fetch (rising to 43.33% in E-commerce)

19.67%
Parse Live Audit, Nov 2026

How 300 top AI-cited sites compare on llms.txt, Markdown twins, and agent discovery

We pulled the 300 most-cited web hosts across ChatGPT Search, Google AI Mode, and Perplexity from September 1 to November 19, 2026 (90 Developer/Docs, 90 B2B SaaS, 60 Editorial/Media, and 60 E-commerce) and probed each host and its #1 most-cited URL live on November 20, 2026.

The table below measures adoption of /llms.txt, /llms-full.txt, Accept: text/markdown content negotiation, .md twin URLs, alternate Markdown link tags, MCP mentions inside /llms.txt, and /.well-known/mcp* JSON manifests across all four categories.

Site category/llms.txt host (host or root)/llms-full.txtAccept: text/markdown.md twin URLAlternate MD linkMCP in /llms.txt/.well-known/mcp*
Developer/Docs (n=90)54.44% (67.78%)25.56%28.89%34.44%16.67%40.00%3.33%
B2B SaaS (n=90)62.22% (62.22%)16.67%17.78%12.22%11.11%38.89%5.56%
Editorial/Media (n=60)25.00% (26.67%)5.00%1.67%6.67%5.00%1.67%0.00%
E-commerce (n=60)30.00% (30.00%)13.33%3.33%0.00%0.00%11.67%0.00%
All 300 top-cited46.00% (50.33%)16.33%15.00%15.33%9.33%26.33%2.67%

Two operational takeaways jump straight out of that matrix:

  • Platform defaults push adoption faster than manual SEO. E-commerce beat Editorial/Media on /llms.txt (30.00% vs 25.00%) and hit 11.67% (7 of 60) MCP mentions inside /llms.txt because 7 Shopify merchants automatically expose Shopify's default ucp.dev / Shop skill /llms.txt template. Similarly, Mintlify, ReadMe, and Cloudflare defaults lift Developer/Docs to 34.44% .md twins and 28.89% Accept: text/markdown.
  • Teams describe MCP in prose but skip machine discovery. Across the 300 sites, 26.33% (79 of 300, including 40.00% of Developer/Docs and 38.89% of B2B SaaS) mention MCP inside /llms.txt, yet only 2.67% (8 of 300: cloudflare.com, signoz.io, monday.com, notion.com, attio.com, langfuse.com, sentry.io, and docs.langchain.com) serve a valid JSON manifest at /.well-known/mcp*, and just 1.33% (4 of 300: cloudflare.com, monday.com, attio.com, docs.langchain.com) expose an A2A Agent Card.

How much HTML bloat costs AI agents in raw tokens

When an agent fetches a normal web page without a Markdown twin, it doesn't just download your article. Across 238 clean HTTP 200 cited HTML pages in our benchmark, median raw HTML size reached 347,139.5 bytes (86,772.5 estimated tokens at characters / 4), including 54,927 bytes of inline/script tags (<script>) and 22,779 bytes of vector markup (<svg>) compared with just 11,456 bytes (2,850.5 tokens) of stripped visible text.

The bloat multiplier (raw HTML tokens / visible text tokens) exposes why coding assistants burn context windows fast on marketing pages, and how sharply native Markdown endpoints cut waste.

CategoryClean HTML pages (n)Median raw HTML tokensMedian visible text tokensMedian bloat multiplier (p90)Native MD pairs (n)Median HTML-to-MD savings
Developer/Docs7838,3002,456.513.50× (45.38×)3822.55×
B2B SaaS8592,8902,491.031.46× (99.95×)1838.09×
Editorial/Media43121,6166,885.013.51× (74.48×)48.87×
E-commerce32126,8402,277.578.62× (167.68×)235.85×
All clean pages23886,772.52,850.524.97× (108.84×)6225.41×

Inspecting individual sites from our 62 Top-300 native Markdown pairs and 13-page Named Reference Twin panel shows the scale of token savings in production:

Tested URLAccept: text/markdown / .md twinRaw HTML tokensNative Markdown tokensToken reduction multiplier
sentry.io/Yes / Yes156,231244640.29×
www.cloudflare.com/Yes / Yes333,772747446.82×
vercel.com/Yes / No150,134566265.25×
supabase.com/Yes / Yes337,4091,303258.95×
docs.datadoghq.com/watchdogYes / Yes197,689921214.65×
linear.app/Yes / Yes316,3781,738182.04×
linear.app/docsYes / Yes139,4331,166119.58×
vercel.com/docsYes / Yes228,4612,48292.05×
docs.anthropic.com/en/docs/welcomeYes / Yes109,3031,43576.17×
docs.stripe.com/taxYes / Yes82,4751,09375.46×
mintlify.com/docsYes / No52,15885261.22×
parse.gl/docs/mcpYes / Yes51,3764,71110.91× (90.83% saved)
parse.gl/aiYes / Yes5,5263,9671.39× (28.21% saved)

Notice two distinct patterns in that benchmark. Heavy marketing homepages (sentry.io, cloudflare.com, supabase.com, linear.app) drop 182× to 640× when served as native Markdown, whereas developer.avalara.com/products/crossborder showed an extreme 1,893.88× outlier (749,975 raw HTML tokens vs 396 .md tokens). Even on Parse's intentionally lean server-rendered /ai page (5,526 HTML tokens), Markdown trims another 28.21%, while Mintlify-powered parse.gl/docs/mcp shrinks 90.83% (51,376 to 4,711 tokens).

Why plain HTTP fetch fails on 19.67% of cited URLs (and what OAI-SearchBot sees)

Token bloat isn't the worst failure mode. Across all 300 top-cited URLs, 59 (19.67%) failed a plain HTTP fetch completely (43.33% in E-commerce, 28.33% in Editorial/Media, 13.33% in Developer/Docs, and 4.44% in B2B SaaS). This matches our broader crawl telemetry in crawler barriers on cited pages, where 16.99% of 378,392 cited URLs failed simple direct HTTP requests, and our SaaS pricing crawl audit, where 47.3% of active top-SaaS /pricing endpoints hid or blocked prices from plain HTTP fetchers.

Another stealth failure hides behind HTTP 200 status codes: client-rendered JavaScript shells. When we ran local Playwright Chromium across 10 HTTP-200 cited pages whose plain HTTP body yielded fewer than 150 visible characters (<38 tokens), 9 of 10 expanded to 2,897-26,026 rendered characters:

  • docs.pytorch.org/docs/stable/profiler.html jumped 473.20× from 55 plain characters (14 tokens) to 26,026 rendered characters (6,507 tokens).
  • bubble.io/ jumped 202.42× from 86 plain characters to 17,408 rendered characters (4,352 tokens).
  • support.zoom.com/hc/en/article?id=zm_kb&sysparm_article=KB0069029 jumped 153.29× from 87 to 13,336 characters.
  • help.asana.com/s/article/top-asana-integrations jumped 103.57× from 70 to 7,250 characters.
  • developer.apple.com/documentation/arkit jumped 83.86× from 37 to 3,103 characters, while nordstromrack.com/ went from 0 plain characters to 6,276 rendered characters.

At the same time, WAF bot-filtering behaves very differently from robots.txt. Across 265 valid robots.txt files on the 300 hosts, only 2.26% (6 of 265) explicitly disallowed OAI-SearchBot (6.04% disallowed GPTBot, 7.55% ClaudeBot, 8.30% CCBot, while 27 carried Content-Signal directives and 3 sent Content-Signal HTTP headers). Yet 23.67% (71 of 300) of cited URLs blocked HTTP requests carrying the OAI-SearchBot/1.0 User-Agent from our benchmark test IP (46.67% in E-commerce, 35.00% in Editorial/Media, 14.44% in B2B SaaS, 10.00% in Developer/Docs), alongside 2 explicit HTTP 402 pay-per-crawl responses (travelandleisure.com and seriouseats.com).

note

Our 23.67% block rate measured HTTP responses to OpenAI's documented OAI-SearchBot/1.0 User-Agent sent from a standard benchmark IP. Production OpenAI search crawlers operating from verified OpenAI IP ranges can pass edge allowlists that block unverified IPs, while third-party coding agents, MCP web fetchers, and custom scrapers hit those exact plain-HTTP walls.

Review our robots.txt AI crawler configuration guide and AI crawler diagnostic tools comparison if your analytics show flat crawl logs despite open robots.txt rules.

Step-by-step: ship llms.txt, .md twins, and Accept: text/markdown

Copying Parse's production architecture (https://parse.gl/llms.txt) takes one afternoon and works across Next.js, Cloudflare Workers, Nginx, or documentation engines like Mintlify.

Step 1 · 30 min

Fix SSR HTML and WAF allowlists first. Ensure core product, docs, and /pricing pages render full specs in plain HTTP 200 HTML without client-only hydration walls, and verify verified search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) aren't challenged by aggressive CDN rules.

Step 2 · 60 min

Serve .md twins and Accept: text/markdown negotiation. Map /pricing.md and Accept: text/markdown on key marketing, blog, and research routes to strip layout chrome while preserving canonical links and Vary: Accept cache separation.

Step 3 · 20 min

Publish a job-grouped /llms.txt index. Open with an H1 and one-sentence blockquote, organize canonical .md links by agent job (Evaluate, Start, Operate, Research), and add /llms-full.txt for docs hubs.

Step 4 · 15 min

Expose discovery tags and /.well-known/mcp. Link the Markdown twin in HTML headers and publish /.well-known/mcp.json if you operate a remote MCP server so agents discover endpoints programmatically.

Here is the exact HTTP header contract Parse's Markdown endpoints return so CDNs never serve raw Markdown to human browsers while keeping search equity tied to the HTML URL:

HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept
Link: <https://parse.gl/pricing>; rel="canonical"
Cache-Control: public, max-age=300, s-maxage=3600

When serving HTML, advertise the twin in <head> with an alternate link tag so parsers find the clean Markdown endpoint in one hop without guessing URL rules:

<link rel="alternate" type="text/markdown" href="https://parse.gl/pricing.md" />

Notice one subtle caching rule in Parse's live routing: /index.md, /ai.md, /pricing.md, /auth.md, /blog/<slug>.md, and /research/<slug>.md all serve clean Markdown directly, while Accept: text/markdown negotiation runs selectively on static reference routes (/index, /ai, /pricing, /auth) rather than personalized routes so edge caches never leak a Markdown payload onto cached HTML URLs. Pair that with Vary: Accept on every negotiated response.

How to verify your site with four curl checks

Don't guess whether your CDN or Next.js middleware handles agent requests cleanly. Run these four curl probes against your live domain (shown here against parse.gl and your target /pricing route):

# 1. Confirm /llms.txt returns HTTP 200 text/plain or text/markdown (never HTML 404 fallback)
curl -i -sS https://parse.gl/llms.txt | head -n 20

# 2. Verify Accept: text/markdown content negotiation + Vary: Accept cache header
curl -i -sS -H "Accept: text/markdown" https://parse.gl/pricing | head -n 20

# 3. Verify explicit .md twin + canonical Link header
curl -i -sS https://parse.gl/pricing.md | head -n 20

# 4. Compare plain HTTP byte delivery when identifying as OAI-SearchBot
curl -i -sS -A "OAI-SearchBot/1.0" https://yourdomain.com/pricing | head -n 20

Watch out for two traps in step 1 and step 4: SPAs frequently return HTTP 200 HTML shell pages at /llms.txt instead of 404 or Markdown (our audit specifically filtered out HTML impostors), and a 403 or sub-5 KB body on step 4 signals an immediate WAF or JS wall blocking plain HTTP agents.

Run this workflow with your agent

Before rebuilding templates blind, connect Claude Code, Cursor, Codex, or ChatGPT to Parse MCP (https://agent.parse.gl/mcp) and inspect which pages on your domain AI engines already cite versus where fetch barriers break extraction. Index discovery (check_brand, search_cited_pages) and logging notes (log_work, list_work) are free; fetch_live_pages quotes at $0.01 per page inside your $5 signup credit and monthly budget cap.

Run check_brand for "<our domain>" and search_cited_pages to pull our top 10
owned URLs currently cited across ChatGPT Search and Google AI Mode.
Quote and call fetch_live_pages on those 5 highest-cited URLs plus our /pricing
and /docs landing pages to check whether plain HTTP extraction recovers full text.
Save a prioritized /llms.txt, .md twin, and SSR fix checklist via log_work.

Once your agent logs the baseline via log_work, review it across sessions with list_work, wire the matching playbooks from our Claude Code and Cursor MCP workflows guide, or compare extraction-friendly stacks in our 14 marketing MCP servers benchmark. Explore content planning workflows and impact tracking to measure what changes after you ship.

Want to find third-party pages AI already cites that leave out your brand? find pages to pitch with Parse.

Sources

  1. HTML Token Bloat vs. Markdown Twins and llms.txt: Live Audit of 300 Top AI-Cited Domains (Parse) (opens in a new tab) · accessed November 20, 2026
  2. Crawler barriers, noindex, and canonicals on AI-cited pages (Parse) (opens in a new tab) · accessed November 20, 2026
  3. Why AI Agents Fail to Read SaaS Pricing and Changelog Pages (Parse) (opens in a new tab) · accessed November 20, 2026
  4. The /llms.txt file specification (opens in a new tab) · accessed November 20, 2026
  5. Parse live llms.txt site index (opens in a new tab) · accessed November 20, 2026
  6. Parse live llms-full.txt combined documentation (opens in a new tab) · accessed November 20, 2026
  7. Stripe documentation in Markdown (opens in a new tab) · accessed November 20, 2026
  8. Overview of OpenAI crawlers (opens in a new tab) · accessed November 20, 2026

More in Citation strategy

  • Product docs as an AI acquisition channel: architecture, engines, and GTM
  • GitHub and DevRel for AI visibility: the 41,065-citation playbook
  • YouTube creator partnerships for AI search: the 93,598-citation playbook

Run this check for your brand with your agent.

Research, act and build on what your team has tried, with Parse keeping the shared record.

Start free in your browser.
For agents: this page as Markdown, https://parse.gl/blog/make-your-site-readable-for-ai-agents.md. JSON, MCP · free for the last 90 days, checks from $0.01
Open in ChatGPT (opens in a new tab) Open in Claude (opens in a new tab)

This page

As Markdown: https://parse.gl/blog/make-your-site-readable-for-ai-agents.md
Reuse: Figures and tables on this page are free to republish with attribution: a link to https://parse.gl/blog/make-your-site-readable-for-ai-agents beside them. A standard followed link (dofollow) is preferred.

Read and search

Any brand's shares and rivals, with 95% ranges: GET https://agent.parse.gl/v1/tools/check_brand?subject=%3Cbrand-domain%3E
Sample answers in a market: GET https://agent.parse.gl/v1/tools/search_answers?args=%7B%22filter%22%3A%7B%22market%22%3A%5B%22%3Cmkt_id%3E%22%5D%7D%7D
Cited pages in a market: GET https://agent.parse.gl/v1/tools/search_cited_pages?args=%7B%22filter%22%3A%7B%22market%22%3A%5B%22%3Cmkt_id%3E%22%5D%7D%7D
Search the index: GET https://parse.gl/api/public/search?q=<words>

Connect

Connect an assistant: https://agent.parse.gl/mcp

An agent can save work before signup: POST https://agent.parse.gl/v1/workspaces

What it costs

No account: Public figures and up to 5 examples per list, about 120 brands a day per address.
Free account: $5 in credit, your brand's first check (20 buyer questions) and a free weekly check.
After your first top-up: Exports and deeper search results. The first top-up starts at $5.
Live checks: $0.01 per question per engine ($0.05 on Gemini and Claude).
Full public answers and pages: 25 free per day, then $0.005 each.
Public answers older than 90 days: $0.001 each the first time your workspace counts or lists one, then yours.

Prices: https://parse.gl/pricing.md

Map of the site: https://parse.gl/llms.txt