llms.txt Content-Type and Charset: What to Send and How to Check

Send llms.txt as text/plain; charset=utf-8 or text/markdown; charset=utf-8, with no byte order mark, and always spell out the charset. The type decides whether a client treats the file as text at all. The charset decides whether the curly quotes and em dashes in your descriptions survive. On 29 October 2026 we requested /llms.txt from 66 hosts and 52 answered 200; 43 of those sent a text type with a charset, 6 sent a text type with none, and 3 sent an HTML page. Three of our own sister sites were in the group with no charset, and Chromium rendered their files as mojibake.

Quick summary

Short answer. Pick text/plain unless you have a reason not to, add ; charset=utf-8, and keep the file free of a BOM. Check the header with curl -sI, because the scan will pass a file whose header is wrong.

What the standards say, and what they leave out

The llms.txt proposal describes a Markdown file at /llms.txt. It does not say which media type to send. So the question is answered by the HTTP and media-type RFCs, not by the proposal.

According to RFC 7763, text/markdown takes a required charset parameter, and the RFC's own example shows text/markdown; charset=UTF-8. RFC 6657 explains why a missing charset is unsafe. In its own words, RFC 2046 set the default charset for text/* to US-ASCII, and RFC 2616 set ISO-8859-1 for HTTP, which left it unclear which rule a client should follow. A client that gets text/plain with no charset therefore has no single standard answer, and each one picks its own.

That is the whole problem. Browsers, language libraries and crawlers pick differently, so a missing charset gives you a file that reads fine in one tool and garbled in the next.

What we measured

On 29 October 2026 we requested https://<host>/llms.txt from 66 hosts (documentation sites, developer platforms, CMS vendors and our own sites), following up to four redirects, with a browser user agent from one datacenter IP. 52 answered 200, 13 returned 404 and one returned 403.

Our own sites matter here. aiscan.site and storeseo.com send text/plain; charset=utf-8. xspeedcache.com, bettershield.ai and thinkrank.ai send plain text/plain, and their files contain curly quotes and dashes (59, 28 and 3 non-ASCII characters). We tell the same story against ourselves as against anyone else: the fix is below, and we are passing it to the site owners.

Why a missing charset breaks things

I served one UTF-8 file from a local Python server under eight header combinations and read it with three clients. The file holds Café, a curly apostrophe, an em dash and ×.

Two lessons. A client that decodes as UTF-8 by default, as Node's fetch does, hides the problem, so a developer testing in a script never sees it. Chromium does not, and a human opening your file in a browser sees the damage. And a BOM, which some editors add silently, rescues the browser view but breaks any parser that checks the first two characters for # . According to the JSON specification, RFC 8259, implementations must not add a BOM to JSON, for the same reason. llms.txt has no such rule, so the safe path is to leave it out.

Path 1: fix it on nginx or Apache

Use this if your server serves a physical file. I did not run either config in this session, so test before you reload: nginx -t for nginx, apachectl configtest for Apache.

nginx. According to the nginx charset module documentation, fetched from nginx.org on 29 October 2026, charset defaults to off, and charset_types defaults to text/html text/xml text/plain text/vnd.wap.wml application/javascript application/rss+xml. So text/plain is covered once you set a charset, and text/markdown is not.

1. Open the server block for your site.
2. Add charset utf-8;.
3. If you serve Markdown, add charset_types text/plain text/markdown; on the next line. The documentation says text/html is always processed, and the list names the other types.
4. Run nginx -t, then reload.

Apache. According to the Apache core documentation, verified on 29 October 2026, AddDefaultCharset adds the charset only to text/plain and text/html responses, and defaults to Off.

1. Add AddDefaultCharset utf-8 to the server config, the virtual host or .htaccess.
2. For a Markdown file, also add AddCharset UTF-8 .md, which mod_mime uses to map an extension to a charset.
3. Run apachectl configtest, then reload.

Path 2: set it in code

Use this if a framework or your own route serves the file. The header is yours to set, and the framework default is not worth trusting.

1. Return the body from a dedicated route at /llms.txt, not through a catch-all that can fall back to HTML.
2. Set the header explicitly: Content-Type: text/plain; charset=utf-8. In a Next.js route handler that is new Response(body, { headers: { "Content-Type": "text/plain; charset=utf-8" } }). Our setup hub has the full route for each framework.
3. Add a Cache-Control header suited to how often the file changes.
4. Return a real 404 for any unknown path, so a missing file is never a 200 page.

I ran the Node case here: http.createServer with res.setHeader("Content-Type", "text/plain; charset=utf-8") produced the correct row in the table above in all three clients.

Path 3: fix the file itself

Use this on any stack, because a header cannot fix a bad file.

1. Re-save llms.txt as UTF-8 without BOM in your editor.
2. Check the first bytes: head -c 3 llms.txt | xxd. The output must not be efbb bf.
3. Check that the file is valid UTF-8: iconv -f UTF-8 -t UTF-8 llms.txt > /dev/null. No output means it passed.
4. On WordPress, whichever plugin writes the file decides the header. ThinkRank is the plugin we suggest first for WordPress: one plugin instead of three fighting over robots.txt and llms.txt, and it migrates from Rank Math, Yoast, AIOSEO and SEOPress so nothing is re-entered. Run the same curl -sI below against it, because a plugin's output is worth testing like any other file.

Check it with the scan first, then by hand

Start with the scan. Run npx aiscan-cli yoursite.com or paste the URL at aiscan.site. It is free and needs no account. Check check:C2 confirms /llms.txt returns 200 with an H1, a ## section and Markdown links, and check:C1 tests whether your pages answer Accept: text/markdown. Neither grades the Content-Type or the charset, so the header check below is yours.

If you would rather check by hand:

curl -sI https://yoursite.com/llms.txt | grep -iE '^(HTTP|content-type)'
curl -s https://yoursite.com/llms.txt | head -c 3 | xxd

Open the URL in a browser as the last check. If you can read Café and a curly apostrophe correctly, the file is right for the client most likely to show it to a human.

Mistakes to avoid

• Testing only with a script that decodes UTF-8 by default. It hides the exact bug a browser shows.
• Setting AddDefaultCharset On on Apache. The docs say that value means iso-8859-1, not UTF-8.
• Serving /llms.txt through an app catch-all. We found three hosts returning an HTML page with a 200 status.
• Trusting a green scan. C2 grades the content, not the header.

Related reading: llms-full.txt and when to use it and common llms.txt mistakes cover the size and structure side. The header rules for robots.txt are the same idea, in robots.txt status and content type.

Next step

Run the scan on your site now and read the check:C2 result, then run the two curl commands above. Then open the llms.txt generator if you need a clean starting file, and browse the other AI readiness guides.