# Make your site readable for AI agents: 25.4× lighter pages with Markdown and llms.txt

> In Parse's 300-domain audit, raw HTML bloats to 86,773 tokens while native Markdown cuts weight 25.41x. See adoption stats and exact .md implementation steps.

HTML: https://parse.gl/blog/make-your-site-readable-for-ai-agents · Site index: https://parse.gl/llms.txt · Published: 2026-11-20 · Updated: 2026-11-21

Most websites hand AI assistants a `347 KB` wall of JavaScript bundles, inline SVGs, and navigation chrome just to deliver `2,851` tokens of actual copy. Across `62` top AI-cited pages serving native Markdown twins in Parse's November 20, 2026 audit of **300** top-cited domains, offering native Markdown shrank token weight by a median of **25.41×** from `80,389.5` raw HTML tokens to `2,323` Markdown tokens (while the `238` clean HTML pages overall carried an `86,772.5`-token HTML median).

As we reported in [our earlier /llms.txt verdict](https://parse.gl/blog/llms-txt-ai-visibility), `/llms.txt` is still not a passive search ranking signal on ChatGPT Search or Google AI Mode. Yet our [live 300-domain agent readability study](https://parse.gl/research/agent-readability-html-vs-markdown-and-llms-txt) shows that **46.00%** of top-cited hosts (`50.33%` including root domains) now ship `/llms.txt` and **15.00%** negotiate `Accept: text/markdown`, because coding assistants and MCP agents break on `19.67%` of plain HTTP HTML fetches. Parse indexes **853,414 active brands** across **1,774 markets**, backed by **744,460** counted-engine answers (`378,499` ChatGPT Search and `365,961` Google AI Mode) and **4,053,878** citations across **73,847** buyer questions in our 90-day public MCP snapshot (August 23 to November 20, 2026).

- **Median token reduction from raw HTML (80,389.5 tokens) to native Markdown (2,323 tokens) across 62 paired cited pages:** 25.41× (Parse Live Audit, Nov 2026)
- **Of 300 top-cited hosts serving a valid /llms.txt (50.33% including root domains; 62.22% in B2B SaaS):** 46.00% (Parse Live Audit, Nov 2026)
- **Of top 300 sites honoring Accept: text/markdown (15.33% serve .md twins; 9.33% link alternate Markdown):** 15.00% (Parse Live Audit, Nov 2026)
- **Of 300 top-cited URLs failing plain HTTP fetch (rising to 43.33% in E-commerce):** 19.67% (Parse Live Audit, Nov 2026)

## How 300 top AI-cited sites compare on llms.txt, Markdown twins, and agent discovery

We pulled the `300` most-cited web hosts across ChatGPT Search, Google AI Mode, and Perplexity from September 1 to November 19, 2026 (`90` Developer/Docs, `90` B2B SaaS, `60` Editorial/Media, and `60` E-commerce) and probed each host and its `#1` most-cited URL live on November 20, 2026.

The table below measures adoption of `/llms.txt`, `/llms-full.txt`, `Accept: text/markdown` content negotiation, `.md` twin URLs, alternate Markdown link tags, MCP mentions inside `/llms.txt`, and `/.well-known/mcp*` JSON manifests across all four categories.

| Site category                | `/llms.txt` host (host or root) | `/llms-full.txt` | `Accept: text/markdown` | `.md` twin URL | Alternate MD link | MCP in `/llms.txt` | `/.well-known/mcp*` |
| ---------------------------- | ------------------------------: | ---------------: | ----------------------: | -------------: | ----------------: | -----------------: | ------------------: |
| **Developer/Docs** (`n=90`)  |               54.44% (`67.78%`) |           25.56% |                  28.89% |         34.44% |            16.67% |             40.00% |               3.33% |
| **B2B SaaS** (`n=90`)        |               62.22% (`62.22%`) |           16.67% |                  17.78% |         12.22% |            11.11% |             38.89% |               5.56% |
| **Editorial/Media** (`n=60`) |               25.00% (`26.67%`) |            5.00% |                   1.67% |          6.67% |             5.00% |              1.67% |               0.00% |
| **E-commerce** (`n=60`)      |               30.00% (`30.00%`) |           13.33% |                   3.33% |          0.00% |             0.00% |             11.67% |               0.00% |
| **All 300 top-cited**        |           **46.00% (`50.33%`)** |       **16.33%** |              **15.00%** |     **15.33%** |         **9.33%** |         **26.33%** |           **2.67%** |

Two operational takeaways jump straight out of that matrix:

- **Platform defaults push adoption faster than manual SEO.** E-commerce beat Editorial/Media on `/llms.txt` (`30.00%` vs `25.00%`) and hit `11.67%` (`7` of `60`) MCP mentions inside `/llms.txt` because **7 Shopify merchants** automatically expose Shopify's default `ucp.dev` / `Shop skill` `/llms.txt` template. Similarly, Mintlify, ReadMe, and Cloudflare defaults lift Developer/Docs to `34.44%` `.md` twins and `28.89%` `Accept: text/markdown`.
- **Teams describe MCP in prose but skip machine discovery.** Across the `300` sites, **26.33%** (`79` of `300`, including `40.00%` of Developer/Docs and `38.89%` of B2B SaaS) mention MCP inside `/llms.txt`, yet only **2.67%** (`8` of `300`: `cloudflare.com`, `signoz.io`, `monday.com`, `notion.com`, `attio.com`, `langfuse.com`, `sentry.io`, and `docs.langchain.com`) serve a valid JSON manifest at `/.well-known/mcp*`, and just **1.33%** (`4` of `300`: `cloudflare.com`, `monday.com`, `attio.com`, `docs.langchain.com`) expose an A2A Agent Card.

## How much HTML bloat costs AI agents in raw tokens

When an agent fetches a normal web page without a Markdown twin, it doesn't just download your article. Across `238` clean HTTP `200` cited HTML pages in our benchmark, median raw HTML size reached **347,139.5 bytes** (`86,772.5` estimated tokens at `characters / 4`), including **54,927 bytes** of inline/script tags (`<script>`) and **22,779 bytes** of vector markup (`<svg>`) compared with just **11,456 bytes** (`2,850.5` tokens) of stripped visible text.

The bloat multiplier (`raw HTML tokens / visible text tokens`) exposes why coding assistants burn context windows fast on marketing pages, and how sharply native Markdown endpoints cut waste.

| Category            | Clean HTML pages (`n`) | Median raw HTML tokens | Median visible text tokens | Median bloat multiplier (p90) | Native MD pairs (`n`) | Median HTML-to-MD savings |
| ------------------- | ---------------------: | ---------------------: | -------------------------: | ----------------------------: | --------------------: | ------------------------: |
| **Developer/Docs**  |                     78 |                 38,300 |                    2,456.5 |             13.50× (`45.38×`) |                    38 |                **22.55×** |
| **B2B SaaS**        |                     85 |                 92,890 |                    2,491.0 |             31.46× (`99.95×`) |                    18 |                **38.09×** |
| **Editorial/Media** |                     43 |                121,616 |                    6,885.0 |             13.51× (`74.48×`) |                     4 |                 **8.87×** |
| **E-commerce**      |                     32 |                126,840 |                    2,277.5 |            78.62× (`167.68×`) |                     2 |                **35.85×** |
| **All clean pages** |                **238** |           **86,772.5** |                **2,850.5** |        **24.97× (`108.84×`)** |                **62** |                **25.41×** |

Inspecting individual sites from our `62` Top-300 native Markdown pairs and `13`-page Named Reference Twin panel shows the scale of token savings in production:

| Tested URL                           | `Accept: text/markdown` / `.md` twin | Raw HTML tokens | Native Markdown tokens |  Token reduction multiplier |
| ------------------------------------ | ------------------------------------ | --------------: | ---------------------: | --------------------------: |
| `sentry.io/`                         | Yes / Yes                            |         156,231 |                    244 |                 **640.29×** |
| `www.cloudflare.com/`                | Yes / Yes                            |         333,772 |                    747 |                 **446.82×** |
| `vercel.com/`                        | Yes / No                             |         150,134 |                    566 |                 **265.25×** |
| `supabase.com/`                      | Yes / Yes                            |         337,409 |                  1,303 |                 **258.95×** |
| `docs.datadoghq.com/watchdog`        | Yes / Yes                            |         197,689 |                    921 |                 **214.65×** |
| `linear.app/`                        | Yes / Yes                            |         316,378 |                  1,738 |                 **182.04×** |
| `linear.app/docs`                    | Yes / Yes                            |         139,433 |                  1,166 |                 **119.58×** |
| `vercel.com/docs`                    | Yes / Yes                            |         228,461 |                  2,482 |                  **92.05×** |
| `docs.anthropic.com/en/docs/welcome` | Yes / Yes                            |         109,303 |                  1,435 |                  **76.17×** |
| `docs.stripe.com/tax`                | Yes / Yes                            |          82,475 |                  1,093 |                  **75.46×** |
| `mintlify.com/docs`                  | Yes / No                             |          52,158 |                    852 |                  **61.22×** |
| `parse.gl/docs/mcp`                  | Yes / Yes                            |          51,376 |                  4,711 | **10.91×** (`90.83%` saved) |
| `parse.gl/ai`                        | Yes / Yes                            |           5,526 |                  3,967 |  **1.39×** (`28.21%` saved) |

Notice two distinct patterns in that benchmark. Heavy marketing homepages (`sentry.io`, `cloudflare.com`, `supabase.com`, `linear.app`) drop **182× to 640×** when served as native Markdown, whereas `developer.avalara.com/products/crossborder` showed an extreme `1,893.88×` outlier (`749,975` raw HTML tokens vs `396` `.md` tokens). Even on Parse's intentionally lean server-rendered `/ai` page (`5,526` HTML tokens), Markdown trims another `28.21%`, while Mintlify-powered `parse.gl/docs/mcp` shrinks `90.83%` (`51,376` to `4,711` tokens).

## Why plain HTTP fetch fails on 19.67% of cited URLs (and what OAI-SearchBot sees)

Token bloat isn't the worst failure mode. Across all `300` top-cited URLs, **59 (`19.67%`) failed a plain HTTP fetch** completely (`43.33%` in E-commerce, `28.33%` in Editorial/Media, `13.33%` in Developer/Docs, and `4.44%` in B2B SaaS). This matches our broader crawl telemetry in [crawler barriers on cited pages](https://parse.gl/research/crawler-barriers-noindex-and-canonicals-on-cited-pages), where `16.99%` of `378,392` cited URLs failed simple direct HTTP requests, and [our SaaS pricing crawl audit](https://parse.gl/blog/why-ai-agents-cannot-read-your-pricing-page), where `47.3%` of active top-SaaS `/pricing` endpoints hid or blocked prices from plain HTTP fetchers.

Another stealth failure hides behind HTTP `200` status codes: client-rendered JavaScript shells. When we ran local Playwright Chromium across `10` HTTP-200 cited pages whose plain HTTP body yielded fewer than `150` visible characters (`<38` tokens), **9 of 10 expanded to `2,897-26,026` rendered characters**:

- `docs.pytorch.org/docs/stable/profiler.html` jumped **473.20×** from `55` plain characters (`14` tokens) to `26,026` rendered characters (`6,507` tokens).
- `bubble.io/` jumped **202.42×** from `86` plain characters to `17,408` rendered characters (`4,352` tokens).
- `support.zoom.com/hc/en/article?id=zm_kb&sysparm_article=KB0069029` jumped **153.29×** from `87` to `13,336` characters.
- `help.asana.com/s/article/top-asana-integrations` jumped **103.57×** from `70` to `7,250` characters.
- `developer.apple.com/documentation/arkit` jumped **83.86×** from `37` to `3,103` characters, while `nordstromrack.com/` went from `0` plain characters to `6,276` rendered characters.

At the same time, WAF bot-filtering behaves very differently from `robots.txt`. Across `265` valid `robots.txt` files on the `300` hosts, only **2.26%** (`6` of `265`) explicitly disallowed `OAI-SearchBot` (`6.04%` disallowed `GPTBot`, `7.55%` `ClaudeBot`, `8.30%` `CCBot`, while `27` carried `Content-Signal` directives and `3` sent `Content-Signal` HTTP headers). Yet **23.67%** (`71` of `300`) of cited URLs blocked HTTP requests carrying the `OAI-SearchBot/1.0` User-Agent from our benchmark test IP (`46.67%` in E-commerce, `35.00%` in Editorial/Media, `14.44%` in B2B SaaS, `10.00%` in Developer/Docs), alongside `2` explicit HTTP `402` pay-per-crawl responses (`travelandleisure.com` and `seriouseats.com`).

**Read the OAI-SearchBot block metric honestly**
Our `23.67%` block rate measured HTTP responses to OpenAI's documented `OAI-SearchBot/1.0` User-Agent sent from a standard benchmark IP. Production OpenAI search crawlers operating from verified OpenAI IP ranges can pass edge allowlists that block unverified IPs, while third-party coding agents, MCP web fetchers, and custom scrapers hit those exact plain-HTTP walls.

Review our [robots.txt AI crawler configuration guide](https://parse.gl/blog/robots-txt-ai-crawlers-configuration) and [AI crawler diagnostic tools comparison](https://parse.gl/blog/best-ai-crawler-diagnostic-tools) if your analytics show flat crawl logs despite open `robots.txt` rules.

## Step-by-step: ship llms.txt, .md twins, and Accept: text/markdown

Copying Parse's production architecture (`https://parse.gl/llms.txt`) takes one afternoon and works across Next.js, Cloudflare Workers, Nginx, or documentation engines like Mintlify.

**30 min**
**Fix SSR HTML and WAF allowlists first.** Ensure core product, docs, and `/pricing` pages render full specs in plain HTTP `200` HTML without client-only hydration walls, and verify verified search bots (`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`) aren't challenged by aggressive CDN rules.
**60 min**
**Serve `.md` twins and `Accept: text/markdown` negotiation.** Map `/pricing.md` and `Accept: text/markdown` on key marketing, blog, and research routes to strip layout chrome while preserving canonical links and `Vary: Accept` cache separation.
**20 min**
**Publish a job-grouped `/llms.txt` index.** Open with an H1 and one-sentence blockquote, organize canonical `.md` links by agent job (`Evaluate`, `Start`, `Operate`, `Research`), and add `/llms-full.txt` for docs hubs.
**15 min**
**Expose discovery tags and `/.well-known/mcp`.** Link the Markdown twin in HTML headers and publish `/.well-known/mcp.json` if you operate a remote MCP server so agents discover endpoints programmatically.

Here is the exact HTTP header contract Parse's Markdown endpoints return so CDNs never serve raw Markdown to human browsers while keeping search equity tied to the HTML URL:

```http
HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept
Link: <https://parse.gl/pricing>; rel="canonical"
Cache-Control: public, max-age=300, s-maxage=3600
```

When serving HTML, advertise the twin in `<head>` with an alternate link tag so parsers find the clean Markdown endpoint in one hop without guessing URL rules:

```html
<link rel="alternate" type="text/markdown" href="https://parse.gl/pricing.md" />
```

Notice one subtle caching rule in Parse's live routing: `/index.md`, `/ai.md`, `/pricing.md`, `/auth.md`, `/blog/<slug>.md`, and `/research/<slug>.md` all serve clean Markdown directly, while `Accept: text/markdown` negotiation runs selectively on static reference routes (`/index`, `/ai`, `/pricing`, `/auth`) rather than personalized routes so edge caches never leak a Markdown payload onto cached HTML URLs. Pair that with `Vary: Accept` on every negotiated response.

## How to verify your site with four curl checks

Don't guess whether your CDN or Next.js middleware handles agent requests cleanly. Run these four `curl` probes against your live domain (shown here against `parse.gl` and your target `/pricing` route):

```bash
# 1. Confirm /llms.txt returns HTTP 200 text/plain or text/markdown (never HTML 404 fallback)
curl -i -sS https://parse.gl/llms.txt | head -n 20

# 2. Verify Accept: text/markdown content negotiation + Vary: Accept cache header
curl -i -sS -H "Accept: text/markdown" https://parse.gl/pricing | head -n 20

# 3. Verify explicit .md twin + canonical Link header
curl -i -sS https://parse.gl/pricing.md | head -n 20

# 4. Compare plain HTTP byte delivery when identifying as OAI-SearchBot
curl -i -sS -A "OAI-SearchBot/1.0" https://yourdomain.com/pricing | head -n 20
```

Watch out for two traps in step 1 and step 4: SPAs frequently return HTTP `200` HTML shell pages at `/llms.txt` instead of `404` or Markdown (our audit specifically filtered out HTML impostors), and a `403` or sub-`5 KB` body on step 4 signals an immediate WAF or JS wall blocking plain HTTP agents.

## Run this workflow with your agent

Before rebuilding templates blind, connect Claude Code, Cursor, Codex, or ChatGPT to [Parse MCP](https://parse.gl/mcp) (`https://agent.parse.gl/mcp`) and inspect which pages on your domain AI engines already cite versus where fetch barriers break extraction. Index discovery (`check_brand`, `search_cited_pages`) and logging notes (`log_work`, `list_work`) are free; `fetch_live_pages` quotes at `$0.01` per page inside your `$5` signup credit and monthly budget cap.

```text
Run check_brand for "<our domain>" and search_cited_pages to pull our top 10
owned URLs currently cited across ChatGPT Search and Google AI Mode.
Quote and call fetch_live_pages on those 5 highest-cited URLs plus our /pricing
and /docs landing pages to check whether plain HTTP extraction recovers full text.
Save a prioritized /llms.txt, .md twin, and SSR fix checklist via log_work.
```

Once your agent logs the baseline via `log_work`, review it across sessions with `list_work`, wire the matching playbooks from our [Claude Code and Cursor MCP workflows guide](https://parse.gl/blog/claude-code-and-cursor-ai-visibility-workflows), or compare extraction-friendly stacks in our [14 marketing MCP servers benchmark](https://parse.gl/blog/best-mcp-servers-for-marketing-and-seo). Explore [content planning workflows](https://parse.gl/use-cases/plan-content) and [impact tracking](https://parse.gl/use-cases/measure-impact) to measure what changes after you ship.

## Sources

- [HTML Token Bloat vs. Markdown Twins and llms.txt: Live Audit of 300 Top AI-Cited Domains (Parse)](https://parse.gl/research/agent-readability-html-vs-markdown-and-llms-txt)
- [Crawler barriers, noindex, and canonicals on AI-cited pages (Parse)](https://parse.gl/research/crawler-barriers-noindex-and-canonicals-on-cited-pages)
- [Why AI Agents Fail to Read SaaS Pricing and Changelog Pages (Parse)](https://parse.gl/research/why-ai-agents-fail-to-read-pricing-and-changelog-pages)
- [The /llms.txt file specification](https://llmstxt.org/)
- [Parse live llms.txt site index](https://parse.gl/llms.txt)
- [Parse live llms-full.txt combined documentation](https://parse.gl/llms-full.txt)
- [Stripe documentation in Markdown](https://stripe.com/blog/markdown-docs)
- [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)

## For agents

- Any brand's shares and rivals, with 95% ranges: `https://agent.parse.gl/v1/tools/check_brand?subject=%3Cbrand-domain%3E`
- Sample answers in a market: `https://agent.parse.gl/v1/tools/search_answers?args=%7B%22filter%22%3A%7B%22market%22%3A%5B%22%3Cmkt_id%3E%22%5D%7D%7D`
- Cited pages in a market: `https://agent.parse.gl/v1/tools/search_cited_pages?args=%7B%22filter%22%3A%7B%22market%22%3A%5B%22%3Cmkt_id%3E%22%5D%7D%7D`
- Search the index: `https://parse.gl/api/public/search?q=<words>`

Reuse: Figures and tables on this page are free to republish with attribution: a link to https://parse.gl/blog/make-your-site-readable-for-ai-agents beside them. A standard followed link (dofollow) is preferred.

- No account: Public figures and up to 5 examples per list, about 120 brands a day per address.
- Free account: $5 in credit, your brand's first check (20 buyer questions) and a free weekly check.
- After your first top-up: Exports and deeper search results. The first top-up starts at $5.
- Live checks: $0.01 per question per engine ($0.05 on Gemini and Claude).
- Full public answers and pages: 25 free per day, then $0.005 each.
- Public answers older than 90 days: $0.001 each the first time your workspace counts or lists one, then yours.
- An agent can save work before signup: `POST https://agent.parse.gl/v1/workspaces`
- Connect an assistant: `https://agent.parse.gl/mcp`

[Prices](https://parse.gl/pricing.md) · [Map of the site](https://parse.gl/llms.txt)
