AIScan cover: Markdown saves tokens, mostly against HTML. A Markdown chip and the figure 13.4% of the HTML's tokens over a bar motif, from 130 aiscan.site posts measured 22 Nov 2026.
AIScan cover: Markdown saves tokens, mostly against HTML. A Markdown chip and the figure 13.4% of the HTML's tokens over a bar motif, from 130 aiscan.site posts measured 22 Nov 2026.
AI Readiness

How Much Smaller Is the Markdown Copy? 130 Posts Measured in Bytes and Tokens

We measured 130 posts: the Markdown copy is 13.4% of the HTML's tokens, close to Cloudflare's 80% claim, but only 11% smaller than the page's plain text.

22 Nov 2026 18 min read
#Markdown#content negotiation#tokens#AI agents

This guide covers C1 · Content, C3 · Content.

Table of contents

The Markdown copy of an aiscan.site blog post is a median 15.0% of the HTML page's size and 13.4% of its tokens, measured across all 130 live posts on 22 November 2026. That is close to Cloudflare's widely repeated "80% fewer tokens", but it only holds against raw HTML. Against the plain text an agent would pull out of the page anyway, the clean Markdown copy saves about 11%, and the Markdown our site hands to a client that asks for it with Accept: text/markdown is 45% larger than that text.

Quick summary

QuestionWhat we measured on 130 posts, 22 Nov 2026Median
How big is the HTML page?bytes as served, decompressed123.6 KB, 35,306 tokens
How big is the .md copy?/blog/<slug>.md18.6 KB (15.0%), 4,815 tokens (13.4%)
How big is the negotiated Markdown?same URL, Accept: text/markdown29.4 KB (23.9%), 7,679 tokens (21.9%)
What does it save on the wire?gzip transfer size, .md vs HTML7.4 KB vs 30.7 KB (24.6%)
What does it save against extracted text?.md tokens vs the page's visible text88.7% of the text, so about 11% fewer
Where do the HTML bytes go?share of the decompressed pagearticle 48.4%, hydration payload 21.8%, inline SVG 7.2%, JSON-LD 5.6%
Do two tokenizers agree?o200k_base vs cl100k_basewithin 1% on every post

The short version for a site owner: serve Markdown, because it is smaller in every measure, but strip the navigation, footer and banner out of it. The size win comes from what you leave out, and a converter that keeps your site chrome gives most of it back.

Why we measured this

Cloudflare's launch post for its Markdown for Agents feature, published 12 February 2026, gives one number. According to Cloudflare's blog, that post "takes 16,180 tokens in HTML and 3,150 tokens when converted to markdown", which it calls an 80% reduction. The figure has been repeated in trade press coverage since then, always as Cloudflare's single measurement of its own page. We could not find a second measurement on a different site published anywhere.

We have the material to make one. Every post on this blog exists in three forms: the HTML page a browser gets, a Markdown copy at the same path with .md appended, and a negotiated Markdown response at the original URL when a client sends Accept: text/markdown. Check C1 in AIScan grades whether that last request works, and our content dimension docs explain how. RFC 9110 calls this proactive negotiation and describes it, in its own words, as preferences "sent by the user agent in a request to encourage an algorithm located at the server to select the preferred representation". So the question is concrete: how much smaller is each copy, in bytes, on the wire, and in tokens, and is "80%" the right number to plan around?

It matters for two reasons. An agent with a context window pays for every token it reads, so a site that is cheap to read is a site an agent can read more of. And if most of the saving is something a good HTML-to-text step already gets, the case for maintaining a Markdown copy rests on structure and accuracy, which is a different argument.

How we measured

The population is every live post in our own blog list on 22 November 2026, read from the publishing API: 184 rows, minus 54 that are 301 redirects into hub posts, leaves 130 posts. None was marked noindex. The measurement ran between about 03:10 and 03:40 UTC.

For each post the script made four requests to aiscan.site:

  1. the HTML page with Accept-Encoding: gzip, recording the transfer size and the decompressed size;
  2. the .md copy the same way;
  3. the HTML page with Accept: text/markdown, the negotiated response C1 tests;
  4. nothing else: no third-party site was fetched, and no host other than ours is named.

Tokens were counted with OpenAI's tiktoken 0.14.0 on two encodings, o200k_base (used by GPT-4o and later) and cl100k_base (GPT-4 and GPT-3.5). Claude, Gemini and Llama tokenizers are not in that package, so their counts will differ; the ratios between copies should move much less than the absolute numbers, because the two OpenAI encodings agreed within 1% on every post.

"Visible text" is the HTML with script, style, svg and template elements removed, tags stripped and whitespace collapsed. "Article text" is the same step applied only to the page's <article> element. Both stand in for what a reasonable HTML-to-text step in an agent would produce. Real agents vary: some feed raw HTML to the model, most run a readability-style extractor first.

Every number below is a median across the 130 posts unless the table says otherwise. Every request returned 200. Every .md copy and every negotiated response came back as text/markdown; charset=utf-8, and the negotiated responses all carried Vary: Accept.

Result 1: in raw bytes, the Markdown copy is 15% of the page

Copy (130 posts)Median size10th percentile90th percentileShare of HTML (median)
HTML page, decompressed123.6 KB92.8 KB177.8 KB100%
.md copy18.6 KB10.1 KB30.0 KB15.0% (range 9.3 to 21.3%)
Negotiated Markdown29.4 KB19.8 KB44.1 KB23.9% (range 19.6 to 29.1%)

The .md copy is a median 85% smaller than the HTML. That agrees with the first 30-post sample we took on 19 November (15.6%) and with Cloudflare's one page. On this measure the "80%" claim holds, and for our clean copy it is slightly conservative.

The range matters more than the median. The smallest page in the set, a short explainer on robots.txt syntax, is 78.2 KB of HTML and 7.5 KB of Markdown, 9.6%. The largest, our setup hub, is 415.1 KB and 88.2 KB, 21.3%. Short posts shrink more, and the reason is in Result 4.

Result 2: on the wire, the gap closes to about 75%

Nobody downloads 123.6 KB for a blog post. Our edge serves gzip, and HTML compresses far better than Markdown, because markup repeats itself: the same class names, the same wrapper elements, the same attribute patterns hundreds of times.

Copy (130 posts)Gzip transfer size, medianCompressed toShare of HTML transfer
HTML page30.7 KB24.4% of its raw size100%
.md copy7.4 KB40.0% of its raw size24.6%
Negotiated Markdown10.4 KB35.4% of its raw size34.4%

So the clean copy saves about 75% of transfer, not 85%, and the negotiated copy about 66%. Across the whole archive, the 130 HTML pages are 17.33 MB decompressed and 4.20 MB on the wire; the .md copies are 2.68 MB and 1.05 MB.

A side finding about our own edge: a request that offers only Accept-Encoding: br gets the page uncompressed (272,270 bytes for the Markdown hub, where gzip delivers about 55 KB). Browsers offer gzip alongside Brotli, so they receive gzip, and no visitor is affected. But the site serves no Brotli at all, and a client that asks only for Brotli pays full price. Bytes on the wire do not count toward an agent's context window, though. Tokens do, and tokens are computed on the decompressed text.

Result 3: in tokens, raw HTML is the wrong baseline

Here is the comparison that decides whether Markdown copies are worth maintaining.

What the model reads (130 posts)Median tokens, o200k_basevs raw HTMLvs visible text
Raw HTML page35,306100%671%
Visible text of the whole page5,25914.9%100%
Article text only4,83413.7%91.9%
.md copy4,81513.4%88.7%
Negotiated Markdown7,67921.9%144.8%

Against raw HTML, the .md copy uses 86.6% fewer tokens, which beats Cloudflare's 80%. But a model almost never reads a page that way. Strip the scripts and tags, keep the words, and you are already at 5,259 tokens. The clean Markdown copy saves a median 11% on top of that, and it was smaller than the page's visible text on 128 of 130 posts. Against the article element alone the saving is about 4%, and the .md copy was smaller on 104 of 130.

So most of the famous 80% is the difference between raw HTML and any text at all. A Markdown copy keeps the structure (headings, tables, code fences, link targets) that a flat text extraction loses, and it is accurate in a way a heuristic extractor is not, because it comes from the source rather than a guess about which <div> holds the article. Those are the real reasons to serve it. "It is five times cheaper than the page" is true only for an agent that would otherwise have read the raw markup.

The negotiated copy is the surprise. At 7,679 tokens it costs 45% more than the plain visible text and a median 1.6 times the .md copy of the same post (range 1.2 to 2.2), and it exceeded the visible-text count on all 130. Only 22 of the 130 negotiated responses came in under 20% of the HTML tokens. The clean .md copy did that on all 130.

Why our negotiated Markdown is bigger: it converts the whole page

We split the negotiated responses of 20 randomly chosen posts at their headings to see where the extra 2,958 tokens (median, all 130 posts) come from.

Part of the negotiated response (20 posts)Median tokensShare of the response
Before the title: meta block, announcement banner, site navigation7968.9%
The article body6,372about 67%
The FAQ block7829.1%
After the FAQ: author box, related guides, footer links1,09713.2%

The article body alone came to 1.07 times the .md copy, so the converter's rendering of the article is close to the source. The rest is site chrome: every page starts with a line about AIScan being in Claude's connector directory, the full navigation menu with its platform list, and ends with the author box, three related-guide cards (with raw <h3> tags left inside the link text), a second copy of the table of contents, and the footer. A median negotiated response carries 95.5 Markdown links, most of them the same on every page.

Cloudflare's Markdown for Agents documentation, fetched from developers.cloudflare.com on 22 November 2026, describes the opposite behaviour for its edge converter, in its own words: "Non-content elements (such as headers, footers, navigation, scripts, and styles) are stripped during pre-processing." According to the same documentation, the converted response also gets Vary: Accept "so that caches store separate variants for Markdown and HTML", which ours does too. Our negotiation is our site's own, not Cloudflare's, and it does not strip the chrome. We have filed that as a site issue, and the fix is in the next section. Until it ships, an agent that sends Accept: text/markdown to aiscan.site gets the right content type and passes C1, but pays roughly 3,000 extra tokens per post for menus it did not ask for.

The FAQ block is a fair exception. The .md copies leave out the FAQ entirely on all 130 posts, because the FAQ lives in a separate field and the .md route renders only the body. The negotiated copy includes it, which is the right call: those are the questions the post answers most directly. A good Markdown copy has the article, the FAQ, and nothing else.

One post, three copies: the Markdown setup guide

Medians hide what an individual page looks like, so here is one post in full: our per-stack guide to serving Markdown, the hub for this topic and one of the larger posts in the set.

Copy of markdown-for-ai-agents, 22 Nov 2026DecompressedGzip on the wireTokens, o200k_base
HTML page272,270 bytes55,346 bytes78,607
Visible text of the pagenot servednot served12,172
Article text onlynot servednot served11,471
.md copy38,200 bytes13,454 bytes10,247
Negotiated Markdown62,577 bytes20,134 bytes16,618

The .md copy here is 14.0% of the HTML in bytes and 13.0% in tokens, close to the median. The guide carries five stacks' worth of configuration in 20 fenced code blocks and 10 tables, which is where the Markdown copy earns its keep beyond size. A flat text extraction turns a table into a run of cell values with no row boundaries, and turns an nginx map block into words with the indentation gone. The .md copy keeps both exactly as written, because it is the source the page was built from. An agent asked to copy that config into a server has a much better chance with the .md copy than with 12,172 tokens of flattened text, even though the token counts are within 16% of each other.

The negotiated copy of the same post is 1.6 times the .md copy, the median ratio. Its first 44 lines are the meta block, the banner and the navigation, before the post's title appears. That is the part a reader of this post should look for on their own site: a Markdown response that opens with your menu.

The .md copy also carries front matter the HTML page does not show as text: title, slug, published and updated dates, author, category, tags, description and canonical URL, a median 637 bytes across the 130 posts. That block is cheap and useful. It tells an agent when the page was last changed and where its canonical lives, which a model reading extracted text would have to guess. It is one of the few places where the Markdown copy carries more information than the page, not less.

Result 4: the fixed template cost, and why short posts shrink more

Where do 123.6 KB of HTML go, if the article is about a fifth of that as Markdown?

Part of the HTML page (130 posts)Median bytesMedian share of the page
<article> element (body, tables, FAQ, figures as markup)59.8 KB48.4%
Hydration payload (data-tsr-stream-part scripts)26.9 KB21.8%
Inline SVG (icons, logo)8.8 KB7.2%
JSON-LD structured data6.6 KB5.6%
Other inline scripts3.3 KB2.7%
Head, header, navigation, footer, sidebar, class namesthe rest20.9%

The shares are medians, so they do not add up to exactly 100%. Two of these lines are a property of our stack. The site is a Lovable app on TanStack Start, and the hydration payload is the serialised page data the browser needs to make the server-rendered HTML interactive. It carries the post's data again, as script, and it grows with the post. A static site with no client framework has no such payload.

A straight-line fit across the 130 posts makes the fixed part visible: HTML bytes ≈ 45.4 KB + 4.26 × Markdown bytes (R² 0.945). In tokens, HTML ≈ 13,516 + 4.68 × Markdown tokens. Every post pays about 45 KB, or 13,500 tokens, of template before its first word, and then about four bytes of HTML per byte of Markdown on top.

Posts split into thirds by .md sizeMedian .md size.md as share of HTML bytes.md as share of HTML tokens
Smallest 4311.8 KB11.8%10.6%
Middle 4418.6 KB15.1%13.5%
Largest 4327.4 KB17.5%15.7%

That fixed cost is why short posts shrink more, and why a percentage measured on one page does not transfer. Cloudflare's blog post page is heavy with its own navigation and tag lists; ours carries a hydration payload. A lean static page with little template, measured the same way, would be expected to show a smaller saving. We did not measure one here, so treat that as arithmetic from the fit, not a result.

What this means if you serve Markdown to agents

The numbers point to a short list of decisions, in order of how much they move:

DecisionEffect on what an agent reads, from our 130 postsEffort
Serve any text copy at all instead of raw HTMLabout 86% fewer tokens than raw HTMLdepends on stack
Strip navigation, banners, author boxes and footers from itabout 2,958 fewer tokens per post (1.6× down to 1×)a converter setting or one template change
Include the FAQ in the Markdown copyabout 780 more tokens, the post's direct answersone template change
Generate from source instead of converting HTMLabout 7% fewer body tokens (1.07× to 1×) and exact tables and codeneeds access to the source
Turn on gzip or Brotliabout 60% less transfer, no token changeusually a toggle

The first line is what Check C1 grades. The second is the one most sites with a converter will get wrong without noticing, because the response still has the right content type and still passes.

How to check your own site

The quickest way is a scan. Run npx aiscan-cli yoursite.com, or paste the URL at aiscan.site, free and with no account. Check C1 reports whether your server answers Accept: text/markdown with a Markdown response, and the evidence line shows which route matched. Check C3 (structured HTML) and check E3 (server-rendered content) cover the HTML side: if E3 fails, an agent reading your HTML sees the template and little of the article, and no Markdown copy fixes that on its own. C2 covers llms.txt, which is where you would list your Markdown copies; our llms.txt generator writes one.

What a scan cannot tell you is what is inside the Markdown. C1 checks the content type, not whether the response is the article or the article plus your menu. For that, measure it. If you would rather check by hand, this is the script we ran, trimmed to the parts that matter. It needs Python 3 and pip install tiktoken, and it reads a file of URLs, one per line.

import sys, re, gzip, html, urllib.request, tiktoken
enc = tiktoken.get_encoding("o200k_base")
UA = "Mozilla/5.0 (compatible; size-check)"

def fetch(url, accept="text/html"):
    req = urllib.request.Request(url, headers={
        "User-Agent": UA, "Accept": accept, "Accept-Encoding": "gzip"})
    r = urllib.request.urlopen(req, timeout=60)
    wire = r.read()
    raw = gzip.decompress(wire) if r.headers.get("Content-Encoding") == "gzip" else wire
    return len(wire), raw.decode("utf8", "replace"), r.headers.get("Content-Type", "")

def visible_text(page):
    page = re.sub(r"<(script|style|svg|template)[\s\S]*?</\1>", " ", page)
    page = re.sub(r"<[^>]+>", " ", page)
    return re.sub(r"\s+", " ", html.unescape(page)).strip()

def tokens(s):
    return len(enc.encode(s, disallowed_special=()))

print("url\thtml_tok\ttext_tok\tmd_tok\tneg_tok\tneg_type")
for url in open(sys.argv[1]).read().split():
    _, page, _ = fetch(url)
    md_tok = "-"
    try:
        _, md, _ = fetch(url.rstrip("/") + ".md")
        md_tok = tokens(md)
    except Exception:
        pass
    _, neg, neg_type = fetch(url, accept="text/markdown")
    print(f"{url}\t{tokens(page)}\t{tokens(visible_text(page))}"
          f"\t{md_tok}\t{tokens(neg)}\t{neg_type}")

Read the output this way. If neg_type is text/html, your server does not negotiate and C1 will fail; the setup guide for every stack fixes that. If neg_tok is larger than text_tok, your Markdown carries more than the page's words, almost always the site chrome, and the converter needs to drop the header, navigation and footer. If md_tok is close to text_tok, your Markdown copy is clean, and its value is structure and accuracy rather than size.

Limits of this measurement

  • One site, one template. All 130 posts share our layout, our hydration payload and our converter. The ratios are ours. A site with a lighter template will see a smaller saving, and a site with heavier chrome a larger one.
  • One tokenizer family. o200k_base and cl100k_base agreed within 1%, but both are OpenAI's. Anthropic, Google and Meta models tokenize differently; absolute counts will move, and the ratios between copies should move less.
  • "Visible text" is our approximation of an agent's extraction step. Real extractors keep or drop different things, and some agents send raw HTML to the model.
  • The negotiated split is from 20 posts, chosen at random with a fixed seed. The size and token figures for the negotiated copy cover all 130.
  • Content, not caching. We measured what a fresh request gets. Edge caching, conditional requests and HTTP/2 header compression all change real transfer costs and were not modelled.

What we are changing on our own site

Two things follow from this for aiscan.site. The negotiated Markdown should drop the banner, navigation, author box, related guides and footer, and keep the article and its FAQ, which on the 20-post sample cuts the response by about a fifth, to roughly 6,000 tokens for a median post. And the .md copies should include the FAQ block, since the negotiated route already does. Both are filed with the site team. We will re-run this script once they ship and add the result here, dated.

For the wider question of whether agents and search engines use these copies at all, two companion pieces cover what we found: who serves Markdown twins outside documentation sites, and whether Google indexes llms.txt and .md copies (it did not index any of ours). If you are on Cloudflare and want the edge to do the conversion, the Cloudflare Markdown for Agents rule walks through the switch and its plan limits. The format itself is defined by convention rather than a standard, but llmstxt.org is where the .md copy convention is written down, and content negotiation by Accept is RFC 9110, section 12 (both verified on 22 November 2026).

Next step

Scan your site at aiscan.site or with npx aiscan-cli yoursite.com and read three rows: C1 for Markdown negotiation, C3 for structured HTML and E3 for server-rendered content. If C1 passes, run the script above on five of your pages and compare neg_tok with text_tok. If C1 fails, start with the per-stack setup guide. Every other fix we have written up is in the AIScan guides.

Frequently asked questions

How much smaller is a Markdown copy than the HTML page?

On our 130 blog posts, measured 22 November 2026, the .md copy was a median 15.0% of the HTML in bytes and 13.4% in tokens, so about 85% smaller. On the wire with gzip the gap narrows: 7.4 KB against 30.7 KB, about 75% smaller. Your ratio depends on how heavy your page template is.

Is Cloudflare's 80% fewer tokens figure accurate?

It is accurate for what it measures: one Cloudflare blog post, 16,180 tokens of HTML against 3,150 of Markdown. Our own posts show 86.6% fewer tokens against raw HTML. Against the page's visible text, which is what most agents extract before reading, a clean Markdown copy saves only about 11%.

Why is my Accept: text/markdown response bigger than my .md file?

Your converter is probably keeping the site chrome. On aiscan.site the negotiated response was a median 1.6 times the .md copy because it includes the banner, navigation, author box, related links and footer. Cloudflare's own converter strips headers, footers and navigation; a home-grown one often does not.

Why does my Markdown copy have more tokens than the page text?

Markdown keeps link URLs, table pipes and code fences that a text extraction drops, and a converted copy may include menus. Our clean .md copies were smaller than the visible text on 128 of 130 posts, while the negotiated copies were larger on all 130. Compare the two counts with the script in the post.

Why does a client asking only for Brotli get an uncompressed page?

Because the server does not offer Brotli and falls back to no compression rather than gzip. On aiscan.site a br-only request got 272,270 bytes where gzip sends about 55 KB. Browsers send gzip as well, so they are unaffected; agents that send only br are not.

My C1 check passes, so is my Markdown fine?

Not necessarily. C1 checks that Accept: text/markdown returns a text/markdown response, not what is inside it. Run the script in the post and compare neg_tok with text_tok: if the Markdown is larger than the page's words, strip the navigation and footer from it.

Do token counts differ between AI models?

Yes. We counted with OpenAI's o200k_base and cl100k_base encodings, which agreed within 1% on every post. Claude, Gemini and Llama tokenizers give different absolute numbers, but the ratio between the HTML and the Markdown copy should move much less than the counts themselves.

Should the Markdown copy include the FAQ?

Yes. The FAQ holds the post's most direct answers. Our .md copies left it out on all 130 posts because it lives in a separate field, while the negotiated copy included it at a median 782 tokens. A good Markdown copy carries the article and the FAQ and nothing else.

M Asif Rahman

About the author

Entrepreneur, curious about AI

I have been building companies for a long time, and lately most of my curiosity goes to AI. AIScan is a small personal project I built on a flight from Dubai to Narita. I use it to understand how ready the web is for AI agents, and I share what the scans teach me.

Related guides

Browse every guide by check and platform