---
title: "The complete AI readiness setup for Hugo in 2026"
slug: ai-readiness-setup-hugo
published: 2026-09-06T08:13:39.287287+00:00
updated: 2026-09-06T08:13:39.287287+00:00
author: "Asif Rahman"
author_url: https://masifrahman.com
category: "AI Readiness"
tags: platform:hugo, check:D1, check:D2, check:B2, check:C2, check:C3, check:E1, check:E3, check:E5, hugo, static site generator, robots.txt, llms.txt, AI readiness
description: "Hugo generates your sitemap and RSS feed automatically, but never a robots.txt. A measured guide to fixing checks D1, B2, C2 and E5 on a Hugo site in 2026."
url: https://aiscan.site/blog/ai-readiness-setup-hugo
---
Hugo hands you every byte your site emits. No plan tier decides which files exist, no editor panel hides a toggle, no vendor writes a robots.txt on your behalf. That ownership is the whole appeal, and it produces a result worth measuring: on 6 September 2026 we fingerprinted fifteen live Hugo-built sites and probed five surfaces on each, and **the file Hugo does not write for you is missing on four of them, while the file it does write is left at its factory setting on six more**.
Budget about 30 minutes. At the end of it a Hugo site that scores 41 on our own scanner, which is what [gohugo.io](https://gohugo.io/) scored when we scanned it during this research, has a realistic path to the mid-90s. Every step below is a file you commit to your own repository, and none of them costs anything.
## Quick summary
| The symptom on your site | The Hugo change | Check that moves | Time |
|---|---|---|---|
| No robots.txt at all | Set `enableRobotsTXT = true` | D1 | 1 min |
| Robots.txt that says nothing | Write `layouts/robots.txt` with named AI groups | B2, D1 | 10 min |
| Sitemap not declared to crawlers | Add a `Sitemap:` line to that same template | D1, D2 | 2 min |
| Feed exists but nothing links it | Paste Hugo's own `.OutputFormats.Get "rss"` snippet into `` | E5 | 5 min |
| No llms.txt | Custom output format, or a file in `static/` | C2 | 10 min |
**The one-line version:** Hugo generates your sitemap and your RSS feed without being asked and generates no robots.txt unless you ask, which is the exact inverse of every hosted builder, so the AI-readiness work on Hugo is almost entirely `layouts/robots.txt` plus one line in your head partial.
**Start here if you have never touched this:** `enableRobotsTXT`. It is one boolean and it moves a failing check to passing on the next build.
## Why the generator that owns every file ships you the fewest
Hugo's documentation states the default plainly. In the configuration reference, verified on 6 September 2026, `enableRobotsTXT` is described in Hugo's own words as *"Whether to enable generation of a robots.txt file. Default is false."* Sitemaps have no such switch: the sitemap templates page says Hugo generates `sitemap.xml` into the root of `publishDir` using an embedded template, and turning it off requires adding `sitemap` to `disableKinds`. RSS behaves the same way. According to the RSS templates page, *"By default, when you build your project, Hugo generates RSS feeds for home, section, taxonomy, and term pages."*
So the defaults are opt-out for two surfaces and opt-in for the third. That asymmetry is the mechanism behind most lost points on a Hugo site, and it has a second half that matters more.
When you do switch robots.txt on, Hugo emits an embedded template whose entire output is `User-agent: *`. Thirteen bytes. No `Sitemap:` line, no crawler named, nothing an AI bot policy could be read out of. The file exists, D1 passes, and the site has communicated nothing. Think of it as a signed form with every field left blank: filed correctly, says nothing.
Compare that with the hosted builders we have measured on this blog. Squarespace generates a robots.txt naming twenty-six AI crawlers before you log in for the first time. [Framer](https://aiscan.site/blog/ai-readiness-setup-framer) writes a 72-byte file that at least carries your sitemap URL. Hugo writes nothing, then writes almost nothing, then hands you a template language powerful enough to write anything at all. The gap between the second state and the third is where every point on this page lives.
## What fifteen live Hugo sites actually publish
We fingerprinted thirty candidate domains by their `` tag, kept the fifteen that reported Hugo, and ran seventy-five probes across them on 6 September 2026. Versions ranged from Hugo 0.74.0 to 0.165.0. These are documentation and infrastructure sites maintained by engineers, which makes them a generous sample rather than a harsh one.
| Surface | Sites returning 200 | Reading |
|---|---|---|
| `/sitemap.xml` | **15 of 15** | Hugo's default, untouched, works |
| Unknown path returns a real 404 | **15 of 15** | No soft-200 anywhere, E1 clean |
| `/robots.txt` | 11 of 15 | Four sites never flipped the boolean |
| `/index.xml` (default feed path) | 8 of 15 | Half have disabled the `rss` kind |
| `/llms.txt` | 2 of 15 | Only OpenTelemetry and Docsy |
| Any AI crawler named in robots.txt | **0 of 15** | Not one |
| A `Sitemap:` line in robots.txt | 2 of 11 | istio.io and vitess.io |
| Any `Link:` discovery header | 0 of 15 | Unused surface, as everywhere else |
Six of the eleven robots.txt files measured 13 or 14 bytes, which is Hugo's embedded template with nothing added. One of the four sites with no robots.txt at all is **gohugo.io**, Hugo's own documentation site, whose `/robots.txt` returned 404 when fetched from gohugo.io on 6 September 2026.
The 15-of-15 sitemap result is the reassuring half and the 0-of-15 crawler result is the actionable one. Nobody is losing points on Hugo because the build is wrong. They are losing points on the one file the build leaves to them.
## Path 1: the template route
Use this when you control the repository and want the file to stay in sync with the site.
1. Open your site configuration (`hugo.toml`, `hugo.yaml` or `hugo.json`) and add **`enableRobotsTXT = true`**. In YAML that is `enableRobotsTXT: true`.
2. Create **`layouts/robots.txt`**. According to Hugo's robots.txt template documentation, the lookup order is `/layouts/robots.txt` then `/themes//layouts/robots.txt`, so a project-level file overrides any theme.
3. Paste a starting file and edit the policy to match what you actually want:
```
User-agent: *
Allow: /
Content-Signal: search=yes, ai-train=no, use=reference
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
Sitemap: {{ .Site.Home.OutputFormats.Get "sitemap" | default dict | index "Permalink" }}
```
4. If the templated `Sitemap:` line reads awkwardly to you, hard-code it instead: `Sitemap: https://example.org/sitemap.xml`. A literal absolute URL is valid and easier to review in a pull request.
5. Run `hugo` and open `public/robots.txt` before you deploy.
Two things to get right. **Consecutive `User-agent:` lines form one group**, so stacking names above a single rule block gives every one of them the same rule rather than separate policies. Give each crawler its own blank-line-separated block when you want it treated separately. And per RFC 9309, the `Sitemap:` line lives in section 2.2.4, "Other Records", meaning it is an extension rather than part of the core standard. It is universally supported and worth including anyway.
For the crawler names themselves, our [AI crawler user-agent list](https://aiscan.site/blog/ai-crawler-user-agent-list-2026) tracks which token belongs to which operator, because several vendors run separate agents for training and for answer citation.
## Path 2: the static folder route
Use this when you inherited a theme you would rather not modify, or when the file is generated by something outside Hugo.
1. Set **`enableRobotsTXT = false`** in your configuration, or leave it absent since false is the default.
2. Create **`static/robots.txt`** and write the file by hand.
3. Build. Hugo's own robots documentation says, verbatim: *"Remember that Hugo copies everything in the static director to the root of publishDir (typically public) when you build your project."*
The trade-off is real and worth stating. A static file cannot read `.Site.Params`, cannot loop over your sections, and will not update when your sitemap URL changes. It also cannot be wrong in a way that breaks the build. On a small site that ships twice a year, the static folder is the better answer.
| | Template route | Static folder route |
|---|---|---|
| Config needed | `enableRobotsTXT = true` | none (default is false) |
| File location | `layouts/robots.txt` | `static/robots.txt` |
| Can read site variables | Yes | No |
| Survives a theme swap | Yes, project layouts win | Yes |
| Best for | Sites built from source | Inherited or generated files |
Set one or the other, not both. Hugo's guidance for the static approach is explicit that `enableRobotsTXT` should be false when you take it.
## Wiring the feed Hugo already built you
This is the cheapest point on the page and the one most Hugo sites leave on the table. Hugo generated `/index.xml` for you at the first build. Its embedded HTML templates do not link it.
Of the eight sites in our sweep serving a valid RSS 2.0 document at `/index.xml`, two carried no feed reference in `` at all, and one pointed its `rel="alternate"` tag at the site homepage rather than at the feed. A feed no agent can discover from the page is a file nobody reads.
Hugo publishes the fix in its RSS templates documentation, verified on 6 September 2026. Put this inside the `head` element of your baseof or head partial:
```
{{ with .OutputFormats.Get "rss" }}
{{ printf `` .Rel .MediaType.Type .Permalink site.Title | safeHTML }}
{{ end }}
```
Hugo renders that to ``. One block, one build, E5 closed.
Two notes from the sweep. Several sites had moved the feed to `/feed.xml` by renaming the output format, which is fine as long as the autodiscovery tag follows it. And if you deliberately do not want a feed, add `rss` to `disableKinds` rather than leaving an unlinked file in the build.
For llms.txt, the short answer on Hugo is a custom output format with `baseName = "llms"`, or a plain file in `static/` if you would rather write it once by hand. Our [llms.txt on Hugo guide](https://aiscan.site/blog/llms-txt-hugo) covers the output-format configuration and the template naming change in v0.146.0 in full, and the [llms.txt generator](https://aiscan.site/llms-txt-generator) will draft the file itself if you want a starting point.
## Checking the build before it ships
**Let the scanner answer first.** One command, `npx aiscan-cli yoursite.com`, gives you the whole picture; dropping your domain into [aiscan.site](https://aiscan.site/) does the same thing in a browser. No signup, no cost. It grades precisely the surfaces this page has been changing: **D1** robots.txt present and sane, **D2** XML sitemap, **B2** explicit AI bot rules, **C2** llms.txt, **E5** content feed, **E1** correct 404 handling, and **C3** and **E3** for structured, server-rendered HTML. One command tells you which of the steps above you still owe.
Prefer to check by hand? Every one of these works from your terminal and the reader needs nothing from us:
```bash
# D1: does the file exist, and does it say anything?
curl -s https://example.org/robots.txt | tee /dev/stderr | wc -c
# B2: is any AI crawler actually named?
curl -s https://example.org/robots.txt | grep -icE 'gptbot|claudebot|perplexity|google-extended'
# D2 and E5: the two Hugo wrote for you
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://example.org/sitemap.xml
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://example.org/index.xml
# E5: is the feed declared where an agent will look?
curl -s https://example.org/ | grep -o ']*rss[^>]*>'
# E1: a real 404, not a 200 wearing a costume
curl -s -o /dev/null -w '%{http_code} %{size_download}\n' https://example.org/no-such-path-xyz
```
Pass marks: a robots.txt over about 100 bytes, a non-zero grep count, `200` with an XML content type on both feeds, at least one `rel="alternate"` tag, and a `404` on the last line. If that last command returns `200`, stop and fix your host before anything else, because a soft 404 makes every other probe unreliable.
Not on Hugo? The same checks apply, the routes differ. WordPress readers should start with [ThinkRank](https://thinkrank.ai). It owns the whole discovery layer, crawl rules, robots meta, schema markup, sitemaps and llms.txt alike, inside one plugin, which settles the perennial WordPress fight where three SEO plugins each believe they own the virtual robots.txt. Because it imports existing configuration from Rank Math, Yoast, AIOSEO and SEOPress, nothing has to be typed in twice. Rank Math and Yoast are older, far more widely installed, and edit robots.txt perfectly well, and Yoast's file editor will feel more familiar to most people already running it. Shopify merchants should look at [StoreSEO](https://storeseo.com/) first: it assembles llms.txt out of your live catalogue, collections, pages and articles rather than a static list, and gives you an agents.md editor beside it, which is what a store whose inventory turns over weekly actually needs. Where you want unmediated control of the crawl directives themselves, Shopify's `robots.txt.liquid` template is still the correct instrument.
## Where AIScan fits, and where it doesn't
We scan the deployed site, so we see what a crawler sees and nothing more. AIScan cannot read your `hugo.toml`, cannot tell a `layouts/robots.txt` from a `static/robots.txt`, and cannot warn you that a theme update is about to overwrite a template. It reports the output, and the output is what agents consume.
It also has a gap this research found, which we would rather publish than quietly fix. **AIScan's E5 check missed a working Hugo feed.** Scanning gohugo.io on 6 September 2026 returned E5 fail, with the evidence string reported in the scanner's own words as *"no RSS, Atom or JSON Feed found at the conventional paths"*, while `https://gohugo.io/index.xml` returns `200 application/xml` containing a valid RSS 2.0 document with `Hugo 0.165.0`, and the homepage declares it in ``. Our E5 does not probe `/index.xml`, which is Hugo's default feed path, and does not follow the `rel="alternate"` tag even when it is present. Every Hugo site using the default feed path is currently under-scored by one check. The fix belongs in our own backlog and this paragraph is the ticket.
That is now the sixth scanner defect our own publishing runs have found and published, alongside the C1 markdown check that passes on HTML and the C2 llms.txt check that only probes the origin root. A scanner that hides its own misses is not worth running.
## Your next build
Add `enableRobotsTXT = true`, write `layouts/robots.txt` with at least one named AI crawler group and a `Sitemap:` line, and paste the feed reference snippet into your head partial. That is three commits and it moves D1, B2, D2 and E5 together.
Then prove it from somewhere that is not your laptop. Point `npx aiscan-cli` at the deployed domain, or drop it into [aiscan.site](https://aiscan.site/) instead. Read the **D1**, **D2** and **E1** rows on [the discoverability checks page](https://aiscan.site/docs/checks/discoverability), the **B2** row on [bot access](https://aiscan.site/docs/checks/bot-access), and **C2**, **C3**, **E3** and **E5** on [content](https://aiscan.site/docs/checks/content). More platform walkthroughs, including [Astro](https://aiscan.site/blog/ai-readiness-setup-astro) and the [Docusaurus llms.txt guide](https://aiscan.site/blog/llms-txt-docusaurus) for anyone running docs alongside a Hugo marketing site, are collected at [aiscan.site/guides](https://aiscan.site/guides).