The complete AI readiness setup for Replit in 2026

Verified 2 September 2026. Every command, config key and file path below was run against live Replit apps or fetched from Replit's own documentation on that date.

On most platforms a missing robots.txt returns 404 and you find out immediately. On a Replit app built by Agent it returns HTTP 200 with your homepage HTML inside it. Twelve published Replit apps were probed on 2 September 2026 and ten answered 200 for a path that does not exist. Six returned the same HTML shell for /robots.txt, /sitemap.xml, /llms.txt, /agents.md and a nonsense URL, byte for byte identical each time.

That is the whole problem here, and it takes about fifteen minutes to fix once you know which deployment target is serving your app.

Quick summary

Fastest confirmation: run npx aiscan-cli yoursite.com, or paste the URL at aiscan.site. It reports D1, D2, B2, C2, C3, E1, E3 and E5 in one pass, free and with no account.

Why a missing file on Replit returns 200 instead of 404

Ask Agent for an app and you get a full-stack project: a client/ folder holding a React and Vite front end, a server/ folder holding an Express server, and a shared schema. In production that server does two jobs in order. It serves the built static files, then it hands everything it did not recognise to React so the browser-side router can decide what to show.

That second job is one line, and it is the line that matters. Replit's generated server/vite.ts ends with a wildcard route sending index.html for any unmatched path. Think of a receptionist told to hand every unrecognised visitor the same brochure rather than saying "nobody by that name works here". A browser is fine with it, because the router reads the URL and draws the right page. A crawler is not: it asked for /llms.txt, got 200 text/html, and cannot tell the file is absent.

That hits the whole discovery layer at once. E1 fails because there is no real 404. C2 and D2 report a file present when nothing is there. Any checker deciding on a status code alone records a pass. The same shape appears elsewhere: Ghost 302s four llms.txt paths to its homepage, while Squarespace hard-404s an unknown root path, which is correct and the useful control.

Find out which deployment target is serving your app

Two targets serve public web traffic differently, and the response headers name which one in a single request.

curl -sI https://yourapp.replit.app/ | grep -i '^server\|^x-powered-by'

• server: Google Frontend plus x-powered-by: Express means Autoscale. Your Express server is answering, catch-all included. Verified on visa4.travel and askquran.chat.
• server: nginx with no x-powered-by means Static Deployment. Files come off disk and missing paths return a real 404. Verified on speed.press.

The .replit file in your project root says the same thing in its [deployment] block. Two values map to Autoscale in the wild: newer apps read deploymentTarget = "autoscale" (read from Visa4.Travel) and older ones deploymentTarget = "cloudrun" (read from AskQuran, last updated May 2026). Static reads deploymentTarget = "static" and adds a publicDir key.

Path 1: Static Deployment, when your app can use one

Best for landing pages, portfolios and documentation, and the only target where discovery files behave the way the rest of the web does.

Replit's deployment-types page, fetched from docs.replit.com on 2 September 2026, says verbatim: "Static Deployments are not compatible with Replit Apps created using Agent. Agent builds full-stack apps that need a backend server, so use Autoscale or Reserved VM for those." This path is open to you if your project has no backend, or if you split the marketing pages into a second project.

1. Open the Publishing tool, then Adjust settings, then the Deployment type dropdown, and choose Static.
2. Set the public directory to your build output. A Vite project building the client alone uses dist/public, which appears in .replit as publicDir = "dist/public".
3. Put robots.txt, sitemap.xml and llms.txt in the source folder copied into that directory. For a Vite client that is client/public/.
4. Create a 404.html in the root directory of your Replit App. According to Replit's static configuration reference, that file renders for URL paths matching no file or rule.
5. For a specific content type on a file, add a header block to .replit:

[[deployment.responseHeaders]]
path = "/llms.txt"
name = "Content-Type"
value = "text/plain; charset=utf-8"

1. Republish. Changes to .replit do not apply until you do.

One trap before you add a catch-all [[deployment.rewrites]] rule: Replit calls it shadowing, and a real file always wins over a matching rewrite. That is what keeps robots.txt serving as itself rather than as index.html.

Path 2: Autoscale, which is what Agent gives you

Most readers are here. The app has an Express server, Static is unavailable, and the fix is three small edits.

1. Put the flat files where the build copies them. In an Agent-built Vite project that is client/public/. On visa4.travel, verified on 2 September 2026, that folder holds exactly robots.txt, llms.txt, favicon.svg, favicon.png and og-image.png, and all of them serve at the domain root. Files placed anywhere else do not.
2. Register real routes for anything generated, before the catch-all. A sitemap built from a database is a route, not a file. visa4.travel/sitemap.xml returns 200 application/xml at 32,731 bytes with no sitemap.xml on disk anywhere in the project, which is what a registered route looks like:

// server/routes.ts: must be registered BEFORE the wildcard handler
app.get("/sitemap.xml", async (_req, res) => {
  const urls = await buildUrlList();
  res.type("application/xml").send(renderSitemap(urls));
});

1. Give unmatched paths a real 404. Add a handler that answers a 404 status for requests a crawler would make, while leaving your React routes alone:

// after your API and file routes, before the SPA catch-all
app.use((req, res, next) => {
  if (/\.(txt|xml|json|md)$/.test(req.path)) {
    return res.status(404).type("text/plain").send("Not found");
  }
  next();
});

That last edit is the one nobody makes, and it is the difference between a scanner telling you the truth and a scanner telling you everything is fine.

The AI bot rules that actually decide anything

Replit ships no robots.txt of its own, so whatever you write is the whole file. Named groups replace the wildcard group rather than adding to it, so repeat the rules you still want.

User-agent: *
Allow: /
Content-Signal: search=yes, ai-train=no, use=reference

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Sitemap: https://yourapp.com/sitemap.xml

GPTBot collects training data and OAI-SearchBot fetches for search answers, so blocking the first and allowing the second is usually what people mean by "no training". Verified on 2 September 2026, askquran.chat runs a 7,548-byte file in this shape with Content-Signals and eight named crawler groups, and its only gap is a missing Sitemap: line. Full reference on /docs/checks/bot-access.

What twelve published Replit apps look like right now

Twelve live apps, five paths each plus the homepage, fetched with a desktop browser user agent on 2 September 2026, scripts and styles stripped before counting. One operator's portfolio, so it says nothing about adoption and everything about platform behaviour.

The two exceptions are the same app in two forms, both Static Deployments built with Astro, and they are the only ones returning a real 404 and the only ones a crawler can read without running JavaScript. That correlation is the deployment target showing up in two checks at once.

visa4.travel is the instructive middle case. It has a real robots.txt, a real llms.txt and a working sitemap route, and still returns eight visible words on the homepage and 200 for a URL that does not exist. Fixing the files does not fix the render, a separate decision covered in why React apps go invisible to AI crawlers.

Confirm the files reached a crawler

Start with the scan. npx aiscan-cli yourapp.com, or paste the URL at aiscan.site. It checks D1 and D2 for robots and sitemap, B2 for the AI crawler groups, C2 for llms.txt, E1 for the soft-404 above, and C3 and E3 for whether a crawler receives body copy. One command covers everything this guide changed, free and with no account.

If you would rather check by hand, three commands settle it:

# 1. A real file, or the shell?
curl -so /dev/null -w '%{http_code} %{content_type} %{size_download}\n' https://yourapp.com/llms.txt

# 2. Control: a path that definitely does not exist
curl -so /dev/null -w '%{http_code} %{content_type} %{size_download}\n' https://yourapp.com/zzz-not-real

# 3. What a crawler reads without JavaScript
curl -s https://yourapp.com/ | sed 's/<script[^>]*>.*<\/script>//g;s/<[^>]*>/ /g' | wc -w

Pass looks like this: command 1 returns 200 text/plain at a size matching your file, command 2 returns 404, command 3 returns more than 200 words. Fail is commands 1 and 2 returning identical numbers, which means both are the shell. Under about 50 words on command 3 means an unauthenticated crawler reads an empty page.

When the Replit app is only the front door

Plenty of Replit apps are a marketing surface for content living on another domain, and none of the work above touches that domain.

If the blog runs on WordPress, ThinkRank is the one to reach for first: it handles robots.txt, robots meta, schema, sitemaps and llms.txt from a single plugin rather than three plugins each rewriting the same file, and it migrates settings from Rank Math, Yoast, All in One SEO and SEOPress so nothing is re-entered. Rank Math has the deeper schema builder for unusual types and Yoast the better multi-author editorial workflow; both are honest choices, and both leave llms.txt to be managed elsewhere.

If the store runs on Shopify, StoreSEO generates llms.txt from live products, collections, pages and articles and includes an agents.md editor, which is the part hand-maintained files fall behind on. Full setup in the Shopify guide.

Where AIScan fits, and where it doesn't

The middle row is the honest limit here. A failing C3 on an Autoscale Replit app means an unverified client reads an empty page, which is true and worth fixing. It does not mean Google reads an empty page, because Google renders JavaScript and we do not.

Your next move on Replit

Ten minutes, in order:

• Run curl -sI and record whether you are on Google Frontend or nginx
• Add robots.txt and llms.txt to client/public/, with a Sitemap: line
• Register the sitemap route above the catch-all, or ship a static file
• Add the 404 handler for .txt, .xml, .json and .md paths
• Republish, because .replit changes need it
• Scan with npx aiscan-cli yourapp.com and confirm D1, D2, B2, C2 and E1

The check definitions live on /docs/checks/discoverability and /docs/checks/content, the platform notes on /docs/platforms/replit, and the rest of the setup guides on /guides. For a deeper walkthrough of the file itself, see publishing llms.txt on Replit; for a spec-shaped starting draft, the llms.txt generator. Builders arriving from a different AI builder will find the same argument shaped differently in the Lovable guide.