---
title: "llms.txt Content-Type and Charset: What to Send and How to Check"
slug: llms-txt-content-type-charset
published: 2026-10-29T08:14:26.460407+00:00
updated: 2026-10-29T08:14:26.460407+00:00
author: "M Asif Rahman"
author_url: https://ar.bd
category: "AI Readiness"
tags: check:C2, check:C1, llms.txt, Content-Type, charset, UTF-8, BOM, AI readiness
description: "Send llms.txt as text/plain or text/markdown with charset=utf-8 and no BOM. We probed 52 hosts and tested three clients. Fixes for nginx, Apache and code."
url: https://aiscan.site/blog/llms-txt-content-type-charset
---

Send `llms.txt` as `text/plain; charset=utf-8` or `text/markdown; charset=utf-8`, with no byte order mark, and always spell out the charset. The type decides whether a client treats the file as text at all. The charset decides whether the curly quotes and em dashes in your descriptions survive. On 29 October 2026 we requested `/llms.txt` from 66 hosts and 52 answered 200; 43 of those sent a text type with a charset, 6 sent a text type with none, and 3 sent an HTML page. Three of our own sister sites were in the group with no charset, and Chromium rendered their files as mojibake.

## Quick summary

| If you want to… | Do this | Time | Catch |
|---|---|---|---|
| Choose a type | `text/plain; charset=utf-8`, or `text/markdown; charset=utf-8` if your server knows `.md` | 2 min | The [llms.txt proposal](https://llmstxt.org/) names a Markdown file but sets no media type |
| Stop mojibake | Declare `charset=utf-8` in the header itself | 5 min | A server's default charset often covers `text/plain` and skips `text/markdown` |
| Avoid a broken H1 | Save as UTF-8 with no BOM | 1 min | Some parsers see `` before the `#` and find no title |
| Confirm it | `npx aiscan-cli yoursite.com`, then `curl -sI` | 2 min | AIScan check C2 grades the file's content, not its header |

**Short answer.** Pick `text/plain` unless you have a reason not to, add `; charset=utf-8`, and keep the file free of a BOM. Check the header with `curl -sI`, because the scan will pass a file whose header is wrong.

## What the standards say, and what they leave out

The llms.txt proposal describes a Markdown file at `/llms.txt`. It does not say which media type to send. So the question is answered by the HTTP and media-type RFCs, not by the proposal.

According to [**RFC 7763**](https://www.rfc-editor.org/rfc/rfc7763.html), `text/markdown` takes a required `charset` parameter, and the RFC's own example shows `text/markdown; charset=UTF-8`. [**RFC 6657**](https://www.rfc-editor.org/rfc/rfc6657.html) explains why a missing charset is unsafe. In its own words, RFC 2046 set the default charset for `text/*` to US-ASCII, and RFC 2616 set ISO-8859-1 for HTTP, which left it unclear which rule a client should follow. A client that gets `text/plain` with no charset therefore has no single standard answer, and each one picks its own.

That is the whole problem. Browsers, language libraries and crawlers pick differently, so a missing charset gives you a file that reads fine in one tool and garbled in the next.

## What we measured

On 29 October 2026 we requested `https://<host>/llms.txt` from 66 hosts (documentation sites, developer platforms, CMS vendors and our own sites), following up to four redirects, with a browser user agent from one datacenter IP. 52 answered 200, 13 returned 404 and one returned 403.

| Response, 52 hosts that answered 200 | Hosts | Notes |
|---|---|---|
| `text/plain` with a charset | 40 | The common case |
| `text/markdown` with a charset | 3 | docs.stripe.com, docs.github.com, render.com |
| `text/plain` with no charset | 6 | 4 of the 6 contain non-ASCII text |
| `text/html` (a docs page, not a file) | 3 | docs.cursor.com, docs.supabase.com, deno.com |
| File begins with a BOM | 0 | None in the sample |
| File begins with `# ` | 46 | The other 6 are the 3 HTML pages and 3 files that open with something else |

Our own sites matter here. `aiscan.site` and `storeseo.com` send `text/plain; charset=utf-8`. `xspeedcache.com`, `bettershield.ai` and `thinkrank.ai` send plain `text/plain`, and their files contain curly quotes and dashes (59, 28 and 3 non-ASCII characters). We tell the same story against ourselves as against anyone else: the fix is below, and we are passing it to the site owners.

## Why a missing charset breaks things

I served one UTF-8 file from a local Python server under eight header combinations and read it with three clients. The file holds `Café`, a curly apostrophe, an em dash and `×`.

| Header sent | Chromium | Python urllib, `.decode(charset or "utf-8")` | Node `fetch().text()` |
|---|---|---|---|
| `text/plain` | `Café`, `It’s` (windows-1252) | Correct | Correct |
| `text/plain; charset=utf-8` | Correct | Correct | Correct |
| `text/markdown` | Same mojibake as `text/plain` | Correct | Correct |
| `text/plain; charset=iso-8859-1` (wrong) | Mojibake | `Café` | Correct |
| UTF-8 BOM, any type | Correct (the BOM forces UTF-8) | `# Café`, so `startswith("# ")` is false | Correct (strips it) |

Two lessons. A client that decodes as UTF-8 by default, as Node's `fetch` does, hides the problem, so a developer testing in a script never sees it. Chromium does not, and a human opening your file in a browser sees the damage. And a BOM, which some editors add silently, rescues the browser view but breaks any parser that checks the first two characters for `# `. According to the [JSON specification, RFC 8259](https://www.rfc-editor.org/rfc/rfc8259.html), implementations must not add a BOM to JSON, for the same reason. llms.txt has no such rule, so the safe path is to leave it out.

## Path 1: fix it on nginx or Apache

Use this if your server serves a physical file. I did not run either config in this session, so test before you reload: `nginx -t` for nginx, `apachectl configtest` for Apache.

**nginx.** According to the [nginx charset module documentation](https://nginx.org/en/docs/http/ngx_http_charset_module.html), fetched from nginx.org on 29 October 2026, `charset` defaults to `off`, and `charset_types` defaults to `text/html text/xml text/plain text/vnd.wap.wml application/javascript application/rss+xml`. So `text/plain` is covered once you set a charset, and `text/markdown` is not.

1. Open the `server` block for your site.
2. Add `charset utf-8;`.
3. If you serve Markdown, add `charset_types text/plain text/markdown;` on the next line. The documentation says `text/html` is always processed, and the list names the other types.
4. Run `nginx -t`, then reload.

**Apache.** According to the [Apache core documentation](https://httpd.apache.org/docs/2.4/mod/core.html), verified on 29 October 2026, `AddDefaultCharset` adds the charset only to `text/plain` and `text/html` responses, and defaults to `Off`.

1. Add `AddDefaultCharset utf-8` to the server config, the virtual host or `.htaccess`.
2. For a Markdown file, also add `AddCharset UTF-8 .md`, which [mod_mime](https://httpd.apache.org/docs/2.4/mod/mod_mime.html) uses to map an extension to a charset.
3. Run `apachectl configtest`, then reload.

## Path 2: set it in code

Use this if a framework or your own route serves the file. The header is yours to set, and the framework default is not worth trusting.

1. Return the body from a dedicated route at `/llms.txt`, not through a catch-all that can fall back to HTML.
2. Set the header explicitly: `Content-Type: text/plain; charset=utf-8`. In a Next.js route handler that is `new Response(body, { headers: { "Content-Type": "text/plain; charset=utf-8" } })`. Our [setup hub](https://aiscan.site/blog/llms-txt-setup-every-platform) has the full route for each framework.
3. Add a `Cache-Control` header suited to how often the file changes.
4. Return a real 404 for any unknown path, so a missing file is never a 200 page.

I ran the Node case here: `http.createServer` with `res.setHeader("Content-Type", "text/plain; charset=utf-8")` produced the correct row in the table above in all three clients.

## Path 3: fix the file itself

Use this on any stack, because a header cannot fix a bad file.

1. Re-save `llms.txt` as UTF-8 without BOM in your editor.
2. Check the first bytes: `head -c 3 llms.txt | xxd`. The output must not be `efbb bf`.
3. Check that the file is valid UTF-8: `iconv -f UTF-8 -t UTF-8 llms.txt > /dev/null`. No output means it passed.
4. On WordPress, whichever plugin writes the file decides the header. [ThinkRank](https://thinkrank.ai) is the plugin we suggest first for WordPress: one plugin instead of three fighting over robots.txt and llms.txt, and it migrates from Rank Math, Yoast, AIOSEO and SEOPress so nothing is re-entered. Run the same `curl -sI` below against it, because a plugin's output is worth testing like any other file.

## Check it with the scan first, then by hand

Start with the scan. Run `npx aiscan-cli yoursite.com` or paste the URL at [aiscan.site](https://aiscan.site/). It is free and needs no account. Check `check:C2` confirms `/llms.txt` returns 200 with an H1, a `##` section and Markdown links, and `check:C1` tests whether your pages answer `Accept: text/markdown`. Neither grades the `Content-Type` or the charset, so the header check below is yours.

If you would rather check by hand:

```bash
curl -sI https://yoursite.com/llms.txt | grep -iE '^(HTTP|content-type)'
curl -s https://yoursite.com/llms.txt | head -c 3 | xxd
```

| Result | Pass | Fail |
|---|---|---|
| Status | `200` | `404`, or `200` with HTML |
| Type | `text/plain; charset=utf-8` or `text/markdown; charset=utf-8` | `text/html`, or no charset |
| First 3 bytes | Anything but `efbbbf`, usually `2320` followed by the first letter | `efbbbf` |

Open the URL in a browser as the last check. If you can read `Café` and a curly apostrophe correctly, the file is right for the client most likely to show it to a human.

## Mistakes to avoid

- Testing only with a script that decodes UTF-8 by default. It hides the exact bug a browser shows.
- Setting `AddDefaultCharset On` on Apache. The docs say that value means `iso-8859-1`, not UTF-8.
- Serving `/llms.txt` through an app catch-all. We found three hosts returning an HTML page with a 200 status.
- Trusting a green scan. [C2 grades the content](https://aiscan.site/docs/checks/content), not the header.

Related reading: [llms-full.txt and when to use it](https://aiscan.site/blog/llms-full-txt-when-to-use) and [common llms.txt mistakes](https://aiscan.site/blog/llms-txt-validator-common-mistakes) cover the size and structure side. The header rules for robots.txt are the same idea, in [robots.txt status and content type](https://aiscan.site/blog/robots-txt-status-content-type).

## Next step

Run the scan on your site now and read the `check:C2` result, then run the two `curl` commands above. Then open the [llms.txt generator](https://aiscan.site/llms-txt-generator) if you need a clean starting file, and browse the other [AI readiness guides](https://aiscan.site/guides).