The complete AI readiness setup for Hugo in 2026

Hugo hands you every byte your site emits. No plan tier decides which files exist, no editor panel hides a toggle, no vendor writes a robots.txt on your behalf. That ownership is the whole appeal, and it produces a result worth measuring: on 6 September 2026 we fingerprinted fifteen live Hugo-built sites and probed five surfaces on each, and the file Hugo does not write for you is missing on four of them, while the file it does write is left at its factory setting on six more.

Budget about 30 minutes. At the end of it a Hugo site that scores 41 on our own scanner, which is what gohugo.io scored when we scanned it during this research, has a realistic path to the mid-90s. Every step below is a file you commit to your own repository, and none of them costs anything.

Quick summary

The one-line version: Hugo generates your sitemap and your RSS feed without being asked and generates no robots.txt unless you ask, which is the exact inverse of every hosted builder, so the AI-readiness work on Hugo is almost entirely layouts/robots.txt plus one line in your head partial.

Start here if you have never touched this: enableRobotsTXT. It is one boolean and it moves a failing check to passing on the next build.

Why the generator that owns every file ships you the fewest

Hugo's documentation states the default plainly. In the configuration reference, verified on 6 September 2026, enableRobotsTXT is described in Hugo's own words as "Whether to enable generation of a robots.txt file. Default is false." Sitemaps have no such switch: the sitemap templates page says Hugo generates sitemap.xml into the root of publishDir using an embedded template, and turning it off requires adding sitemap to disableKinds. RSS behaves the same way. According to the RSS templates page, "By default, when you build your project, Hugo generates RSS feeds for home, section, taxonomy, and term pages."

So the defaults are opt-out for two surfaces and opt-in for the third. That asymmetry is the mechanism behind most lost points on a Hugo site, and it has a second half that matters more.

When you do switch robots.txt on, Hugo emits an embedded template whose entire output is User-agent: *. Thirteen bytes. No Sitemap: line, no crawler named, nothing an AI bot policy could be read out of. The file exists, D1 passes, and the site has communicated nothing. Think of it as a signed form with every field left blank: filed correctly, says nothing.

Compare that with the hosted builders we have measured on this blog. Squarespace generates a robots.txt naming twenty-six AI crawlers before you log in for the first time. Framer writes a 72-byte file that at least carries your sitemap URL. Hugo writes nothing, then writes almost nothing, then hands you a template language powerful enough to write anything at all. The gap between the second state and the third is where every point on this page lives.

What fifteen live Hugo sites actually publish

We fingerprinted thirty candidate domains by their <meta name="generator"> tag, kept the fifteen that reported Hugo, and ran seventy-five probes across them on 6 September 2026. Versions ranged from Hugo 0.74.0 to 0.165.0. These are documentation and infrastructure sites maintained by engineers, which makes them a generous sample rather than a harsh one.

Six of the eleven robots.txt files measured 13 or 14 bytes, which is Hugo's embedded template with nothing added. One of the four sites with no robots.txt at all is gohugo.io, Hugo's own documentation site, whose /robots.txt returned 404 when fetched from gohugo.io on 6 September 2026.

The 15-of-15 sitemap result is the reassuring half and the 0-of-15 crawler result is the actionable one. Nobody is losing points on Hugo because the build is wrong. They are losing points on the one file the build leaves to them.

Path 1: the template route

Use this when you control the repository and want the file to stay in sync with the site.

1. Open your site configuration (hugo.toml, hugo.yaml or hugo.json) and add enableRobotsTXT = true. In YAML that is enableRobotsTXT: true.
2. Create layouts/robots.txt. According to Hugo's robots.txt template documentation, the lookup order is /layouts/robots.txt then /themes/<THEME>/layouts/robots.txt, so a project-level file overrides any theme.
3. Paste a starting file and edit the policy to match what you actually want:

User-agent: *
Allow: /
Content-Signal: search=yes, ai-train=no, use=reference

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: {{ .Site.Home.OutputFormats.Get "sitemap" | default dict | index "Permalink" }}

1. If the templated Sitemap: line reads awkwardly to you, hard-code it instead: Sitemap: https://example.org/sitemap.xml. A literal absolute URL is valid and easier to review in a pull request.
2. Run hugo and open public/robots.txt before you deploy.

Two things to get right. Consecutive User-agent: lines form one group, so stacking names above a single rule block gives every one of them the same rule rather than separate policies. Give each crawler its own blank-line-separated block when you want it treated separately. And per RFC 9309, the Sitemap: line lives in section 2.2.4, "Other Records", meaning it is an extension rather than part of the core standard. It is universally supported and worth including anyway.

For the crawler names themselves, our AI crawler user-agent list tracks which token belongs to which operator, because several vendors run separate agents for training and for answer citation.

Path 2: the static folder route

Use this when you inherited a theme you would rather not modify, or when the file is generated by something outside Hugo.

1. Set enableRobotsTXT = false in your configuration, or leave it absent since false is the default.
2. Create static/robots.txt and write the file by hand.
3. Build. Hugo's own robots documentation says, verbatim: "Remember that Hugo copies everything in the static director to the root of publishDir (typically public) when you build your project."

The trade-off is real and worth stating. A static file cannot read .Site.Params, cannot loop over your sections, and will not update when your sitemap URL changes. It also cannot be wrong in a way that breaks the build. On a small site that ships twice a year, the static folder is the better answer.

Set one or the other, not both. Hugo's guidance for the static approach is explicit that enableRobotsTXT should be false when you take it.

Wiring the feed Hugo already built you

This is the cheapest point on the page and the one most Hugo sites leave on the table. Hugo generated /index.xml for you at the first build. Its embedded HTML templates do not link it.

Of the eight sites in our sweep serving a valid RSS 2.0 document at /index.xml, two carried no feed reference in <head> at all, and one pointed its rel="alternate" tag at the site homepage rather than at the feed. A feed no agent can discover from the page is a file nobody reads.

Hugo publishes the fix in its RSS templates documentation, verified on 6 September 2026. Put this inside the head element of your baseof or head partial:

{{ with .OutputFormats.Get "rss" }}
  {{ printf `<link rel=%q type=%q href=%q title=%q>` .Rel .MediaType.Type .Permalink site.Title | safeHTML }}
{{ end }}

Hugo renders that to <link rel="alternate" type="application/rss+xml" href="https://example.org/index.xml" title="ABC Widgets">. One block, one build, E5 closed.

Two notes from the sweep. Several sites had moved the feed to /feed.xml by renaming the output format, which is fine as long as the autodiscovery tag follows it. And if you deliberately do not want a feed, add rss to disableKinds rather than leaving an unlinked file in the build.

For llms.txt, the short answer on Hugo is a custom output format with baseName = "llms", or a plain file in static/ if you would rather write it once by hand. Our llms.txt on Hugo guide covers the output-format configuration and the template naming change in v0.146.0 in full, and the llms.txt generator will draft the file itself if you want a starting point.

Checking the build before it ships

Let the scanner answer first. One command, npx aiscan-cli yoursite.com, gives you the whole picture; dropping your domain into aiscan.site does the same thing in a browser. No signup, no cost. It grades precisely the surfaces this page has been changing: D1 robots.txt present and sane, D2 XML sitemap, B2 explicit AI bot rules, C2 llms.txt, E5 content feed, E1 correct 404 handling, and C3 and E3 for structured, server-rendered HTML. One command tells you which of the steps above you still owe.

Prefer to check by hand? Every one of these works from your terminal and the reader needs nothing from us:

# D1: does the file exist, and does it say anything?
curl -s https://example.org/robots.txt | tee /dev/stderr | wc -c

# B2: is any AI crawler actually named?
curl -s https://example.org/robots.txt | grep -icE 'gptbot|claudebot|perplexity|google-extended'

# D2 and E5: the two Hugo wrote for you
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://example.org/sitemap.xml
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://example.org/index.xml

# E5: is the feed declared where an agent will look?
curl -s https://example.org/ | grep -o '<link[^>]*rss[^>]*>'

# E1: a real 404, not a 200 wearing a costume
curl -s -o /dev/null -w '%{http_code} %{size_download}\n' https://example.org/no-such-path-xyz

Pass marks: a robots.txt over about 100 bytes, a non-zero grep count, 200 with an XML content type on both feeds, at least one rel="alternate" tag, and a 404 on the last line. If that last command returns 200, stop and fix your host before anything else, because a soft 404 makes every other probe unreliable.

Not on Hugo? The same checks apply, the routes differ. WordPress readers should start with ThinkRank. It owns the whole discovery layer, crawl rules, robots meta, schema markup, sitemaps and llms.txt alike, inside one plugin, which settles the perennial WordPress fight where three SEO plugins each believe they own the virtual robots.txt. Because it imports existing configuration from Rank Math, Yoast, AIOSEO and SEOPress, nothing has to be typed in twice. Rank Math and Yoast are older, far more widely installed, and edit robots.txt perfectly well, and Yoast's file editor will feel more familiar to most people already running it. Shopify merchants should look at StoreSEO first: it assembles llms.txt out of your live catalogue, collections, pages and articles rather than a static list, and gives you an agents.md editor beside it, which is what a store whose inventory turns over weekly actually needs. Where you want unmediated control of the crawl directives themselves, Shopify's robots.txt.liquid template is still the correct instrument.

Where AIScan fits, and where it doesn't

We scan the deployed site, so we see what a crawler sees and nothing more. AIScan cannot read your hugo.toml, cannot tell a layouts/robots.txt from a static/robots.txt, and cannot warn you that a theme update is about to overwrite a template. It reports the output, and the output is what agents consume.

It also has a gap this research found, which we would rather publish than quietly fix. AIScan's E5 check missed a working Hugo feed. Scanning gohugo.io on 6 September 2026 returned E5 fail, with the evidence string reported in the scanner's own words as "no RSS, Atom or JSON Feed found at the conventional paths", while https://gohugo.io/index.xml returns 200 application/xml containing a valid RSS 2.0 document with <generator>Hugo 0.165.0</generator>, and the homepage declares it in <head>. Our E5 does not probe /index.xml, which is Hugo's default feed path, and does not follow the rel="alternate" tag even when it is present. Every Hugo site using the default feed path is currently under-scored by one check. The fix belongs in our own backlog and this paragraph is the ticket.

That is now the sixth scanner defect our own publishing runs have found and published, alongside the C1 markdown check that passes on HTML and the C2 llms.txt check that only probes the origin root. A scanner that hides its own misses is not worth running.

Your next build

Add enableRobotsTXT = true, write layouts/robots.txt with at least one named AI crawler group and a Sitemap: line, and paste the feed reference snippet into your head partial. That is three commits and it moves D1, B2, D2 and E5 together.

Then prove it from somewhere that is not your laptop. Point npx aiscan-cli at the deployed domain, or drop it into aiscan.site instead. Read the D1, D2 and E1 rows on the discoverability checks page, the B2 row on bot access, and C2, C3, E3 and E5 on content. More platform walkthroughs, including Astro and the Docusaurus llms.txt guide for anyone running docs alongside a Hugo marketing site, are collected at aiscan.site/guides.