---
title: "The complete AI readiness setup for Replit in 2026"
slug: ai-readiness-setup-replit
published: 2026-09-02T08:15:43.594957+00:00
updated: 2026-09-02T08:15:43.594957+00:00
author: "Asif Rahman"
author_url: https://masifrahman.com
category: "AI Readiness"
tags: platform:replit, check:D1, check:D2, check:B2, check:C2, check:C3, check:E1, check:E3, check:E5, Replit, AI readiness, llms.txt, robots.txt, deployment
description: "Replit apps return HTTP 200 for a robots.txt that does not exist. Here is the mechanism, the two deployment paths that fix it, and how to verify the result."
url: https://aiscan.site/blog/ai-readiness-setup-replit
---

**Verified 2 September 2026.** Every command, config key and file path below was run against live Replit apps or fetched from Replit's own documentation on that date.

On most platforms a missing `robots.txt` returns 404 and you find out immediately. On a Replit app built by Agent it returns **HTTP 200 with your homepage HTML inside it**. Twelve published Replit apps were probed on 2 September 2026 and ten answered 200 for a path that does not exist. Six returned the same HTML shell for `/robots.txt`, `/sitemap.xml`, `/llms.txt`, `/agents.md` and a nonsense URL, byte for byte identical each time.

That is the whole problem here, and it takes about fifteen minutes to fix once you know which deployment target is serving your app.

## Quick summary

| What you want | Check | What decides it on Replit | Where the fix goes |
|---|---|---|---|
| robots.txt exists | D1 | Whether the file is in the served public directory | `client/public/robots.txt` |
| AI crawler rules | B2 | The contents of that same file | Named groups in robots.txt |
| Sitemap reachable | D2 | A real file, or a route registered before the catch-all | `server/routes.ts` |
| llms.txt reachable | C2 | Same as robots.txt | `client/public/llms.txt` |
| Missing paths return 404 | E1 | The Express catch-all in `server/vite.ts` | A 404 handler, or Static Deployment |
| Crawlers see body copy | C3, E3 | React renders in the browser, so the shell is empty | Prerender, or move content to Static |
| Feed declared | E5 | Nothing native. You write the route | `server/routes.ts` plus `<link rel="alternate">` |

Fastest confirmation: run `npx aiscan-cli yoursite.com`, or paste the URL at [aiscan.site](https://aiscan.site/). It reports D1, D2, B2, C2, C3, E1, E3 and E5 in one pass, free and with no account.

## Why a missing file on Replit returns 200 instead of 404

Ask Agent for an app and you get a full-stack project: a `client/` folder holding a React and Vite front end, a `server/` folder holding an Express server, and a shared schema. In production that server does two jobs in order. It serves the built static files, then it hands everything it did not recognise to React so the browser-side router can decide what to show.

That second job is one line, and it is the line that matters. Replit's generated `server/vite.ts` ends with a wildcard route sending `index.html` for any unmatched path. Think of a receptionist told to hand every unrecognised visitor the same brochure rather than saying "nobody by that name works here". A browser is fine with it, because the router reads the URL and draws the right page. A crawler is not: it asked for `/llms.txt`, got `200 text/html`, and cannot tell the file is absent.

That hits the whole discovery layer at once. E1 fails because there is no real 404. C2 and D2 report a file present when nothing is there. Any checker deciding on a status code alone records a pass. The same shape appears elsewhere: Ghost 302s four llms.txt paths to its homepage, while [Squarespace hard-404s an unknown root path](https://aiscan.site/blog/llms-txt-squarespace), which is correct and the useful control.

## Find out which deployment target is serving your app

Two targets serve public web traffic differently, and the response headers name which one in a single request.

```bash
curl -sI https://yourapp.replit.app/ | grep -i '^server\|^x-powered-by'
```

- **`server: Google Frontend`** plus **`x-powered-by: Express`** means Autoscale. Your Express server is answering, catch-all included. Verified on `visa4.travel` and `askquran.chat`.
- **`server: nginx`** with no `x-powered-by` means Static Deployment. Files come off disk and missing paths return a real 404. Verified on `speed.press`.

The `.replit` file in your project root says the same thing in its `[deployment]` block. Two values map to Autoscale in the wild: newer apps read `deploymentTarget = "autoscale"` (read from Visa4.Travel) and older ones `deploymentTarget = "cloudrun"` (read from AskQuran, last updated May 2026). Static reads `deploymentTarget = "static"` and adds a `publicDir` key.

## Path 1: Static Deployment, when your app can use one

Best for landing pages, portfolios and documentation, and the only target where discovery files behave the way the rest of the web does.

Replit's deployment-types page, fetched from `docs.replit.com` on 2 September 2026, says verbatim: *"Static Deployments are not compatible with Replit Apps created using Agent. Agent builds full-stack apps that need a backend server, so use Autoscale or Reserved VM for those."* This path is open to you if your project has no backend, or if you split the marketing pages into a second project.

1. Open the **Publishing** tool, then **Adjust settings**, then the **Deployment type** dropdown, and choose **Static**.
2. Set the public directory to your build output. A Vite project building the client alone uses `dist/public`, which appears in `.replit` as `publicDir = "dist/public"`.
3. Put `robots.txt`, `sitemap.xml` and `llms.txt` in the source folder copied into that directory. For a Vite client that is `client/public/`.
4. Create a `404.html` in the **root directory of your Replit App**. According to Replit's static configuration reference, that file renders for URL paths matching no file or rule.
5. For a specific content type on a file, add a header block to `.replit`:

```toml
[[deployment.responseHeaders]]
path = "/llms.txt"
name = "Content-Type"
value = "text/plain; charset=utf-8"
```

6. Republish. Changes to `.replit` do not apply until you do.

One trap before you add a catch-all `[[deployment.rewrites]]` rule: Replit calls it **shadowing**, and a real file always wins over a matching rewrite. That is what keeps `robots.txt` serving as itself rather than as `index.html`.

## Path 2: Autoscale, which is what Agent gives you

Most readers are here. The app has an Express server, Static is unavailable, and the fix is three small edits.

1. **Put the flat files where the build copies them.** In an Agent-built Vite project that is `client/public/`. On `visa4.travel`, verified on 2 September 2026, that folder holds exactly `robots.txt`, `llms.txt`, `favicon.svg`, `favicon.png` and `og-image.png`, and all of them serve at the domain root. Files placed anywhere else do not.
2. **Register real routes for anything generated, before the catch-all.** A sitemap built from a database is a route, not a file. `visa4.travel/sitemap.xml` returns `200 application/xml` at 32,731 bytes with no `sitemap.xml` on disk anywhere in the project, which is what a registered route looks like:

```js
// server/routes.ts: must be registered BEFORE the wildcard handler
app.get("/sitemap.xml", async (_req, res) => {
  const urls = await buildUrlList();
  res.type("application/xml").send(renderSitemap(urls));
});
```

3. **Give unmatched paths a real 404.** Add a handler that answers a 404 status for requests a crawler would make, while leaving your React routes alone:

```js
// after your API and file routes, before the SPA catch-all
app.use((req, res, next) => {
  if (/\.(txt|xml|json|md)$/.test(req.path)) {
    return res.status(404).type("text/plain").send("Not found");
  }
  next();
});
```

That last edit is the one nobody makes, and it is the difference between a scanner telling you the truth and a scanner telling you everything is fine.

## The AI bot rules that actually decide anything

Replit ships no robots.txt of its own, so whatever you write is the whole file. Named groups replace the wildcard group rather than adding to it, so repeat the rules you still want.

```
User-agent: *
Allow: /
Content-Signal: search=yes, ai-train=no, use=reference

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Sitemap: https://yourapp.com/sitemap.xml
```

`GPTBot` collects training data and `OAI-SearchBot` fetches for search answers, so blocking the first and allowing the second is usually what people mean by "no training". Verified on 2 September 2026, `askquran.chat` runs a 7,548-byte file in this shape with Content-Signals and eight named crawler groups, and its only gap is a missing `Sitemap:` line. Full reference on [/docs/checks/bot-access](https://aiscan.site/docs/checks/bot-access).

## What twelve published Replit apps look like right now

Twelve live apps, five paths each plus the homepage, fetched with a desktop browser user agent on 2 September 2026, scripts and styles stripped before counting. One operator's portfolio, so it says nothing about adoption and everything about platform behaviour.

| Behaviour | Count |
|---|---|
| HTTP 200 for a path that does not exist | 10 of 12 |
| Same HTML shell for all five probed paths | 6 of 12 |
| A real 404 for an unknown path | 2 of 12, both Static Deployments |
| Homepage under 15 visible words | 10 of 12 |
| Homepage with one `<h1>` and real body copy | 2 of 12 (852 and 906 words) |

The two exceptions are the same app in two forms, both Static Deployments built with Astro, and they are the only ones returning a real 404 and the only ones a crawler can read without running JavaScript. That correlation is the deployment target showing up in two checks at once.

`visa4.travel` is the instructive middle case. It has a real robots.txt, a real llms.txt and a working sitemap route, and still returns eight visible words on the homepage and 200 for a URL that does not exist. Fixing the files does not fix the render, a separate decision covered in [why React apps go invisible to AI crawlers](https://aiscan.site/blog/nextjs-react-invisible-to-ai-crawlers).

## Confirm the files reached a crawler

**Start with the scan.** `npx aiscan-cli yourapp.com`, or paste the URL at [aiscan.site](https://aiscan.site/). It checks D1 and D2 for robots and sitemap, B2 for the AI crawler groups, C2 for llms.txt, E1 for the soft-404 above, and C3 and E3 for whether a crawler receives body copy. One command covers everything this guide changed, free and with no account.

If you would rather check by hand, three commands settle it:

```bash
# 1. A real file, or the shell?
curl -so /dev/null -w '%{http_code} %{content_type} %{size_download}\n' https://yourapp.com/llms.txt

# 2. Control: a path that definitely does not exist
curl -so /dev/null -w '%{http_code} %{content_type} %{size_download}\n' https://yourapp.com/zzz-not-real

# 3. What a crawler reads without JavaScript
curl -s https://yourapp.com/ | sed 's/<script[^>]*>.*<\/script>//g;s/<[^>]*>/ /g' | wc -w
```

Pass looks like this: command 1 returns `200 text/plain` at a size matching your file, command 2 returns `404`, command 3 returns more than 200 words. Fail is commands 1 and 2 returning identical numbers, which means both are the shell. Under about 50 words on command 3 means an unauthenticated crawler reads an empty page.

## When the Replit app is only the front door

Plenty of Replit apps are a marketing surface for content living on another domain, and none of the work above touches that domain.

If the blog runs on WordPress, **[ThinkRank](https://thinkrank.ai)** is the one to reach for first: it handles robots.txt, robots meta, schema, sitemaps and llms.txt from a single plugin rather than three plugins each rewriting the same file, and it migrates settings from Rank Math, Yoast, All in One SEO and SEOPress so nothing is re-entered. Rank Math has the deeper schema builder for unusual types and Yoast the better multi-author editorial workflow; both are honest choices, and both leave llms.txt to be managed elsewhere.

If the store runs on Shopify, **[StoreSEO](https://storeseo.com/)** generates llms.txt from live products, collections, pages and articles and includes an agents.md editor, which is the part hand-maintained files fall behind on. Full setup in the [Shopify guide](https://aiscan.site/blog/ai-readiness-setup-shopify).

## Where AIScan fits, and where it doesn't

| What it answers | What it cannot answer |
|---|---|
| Whether `/llms.txt` returns a real file or your app shell | Whether the file's contents describe your site accurately |
| Whether an unknown path returns 404 (E1) | Which Express route produced the response |
| Whether a crawler receives body copy (C3, E3) | What Google's renderer sees after running your JavaScript |
| Whether robots.txt names AI crawlers (B2) | Whether those crawlers obey it |

The middle row is the honest limit here. A failing C3 on an Autoscale Replit app means an unverified client reads an empty page, which is true and worth fixing. It does not mean Google reads an empty page, because Google renders JavaScript and we do not.

## Your next move on Replit

Ten minutes, in order:

- [ ] Run `curl -sI` and record whether you are on `Google Frontend` or `nginx`
- [ ] Add `robots.txt` and `llms.txt` to `client/public/`, with a `Sitemap:` line
- [ ] Register the sitemap route above the catch-all, or ship a static file
- [ ] Add the 404 handler for `.txt`, `.xml`, `.json` and `.md` paths
- [ ] Republish, because `.replit` changes need it
- [ ] Scan with `npx aiscan-cli yourapp.com` and confirm D1, D2, B2, C2 and E1

The check definitions live on [/docs/checks/discoverability](https://aiscan.site/docs/checks/discoverability) and [/docs/checks/content](https://aiscan.site/docs/checks/content), the platform notes on [/docs/platforms/replit](https://aiscan.site/docs/platforms/replit), and the rest of the setup guides on [/guides](https://aiscan.site/guides). For a deeper walkthrough of the file itself, see [publishing llms.txt on Replit](https://aiscan.site/blog/llms-txt-replit); for a spec-shaped starting draft, the [llms.txt generator](https://aiscan.site/llms-txt-generator). Builders arriving from a different AI builder will find the same argument shaped differently in the [Lovable guide](https://aiscan.site/blog/ai-readiness-setup-lovable).