--- title: "Fix soft 404s on Astro, Docusaurus, Lovable, Next.js, Webflow and Wix" slug: fix-soft-404s-for-ai-agents published: 2026-10-27T12:27:20.827526+00:00 updated: 2026-10-27T12:27:20.827526+00:00 author: "M Asif Rahman" author_url: https://ar.bd category: "AI Readiness" tags: check:E1, platform:astro, platform:docusaurus, platform:lovable, platform:nextjs, platform:webflow, platform:wix, soft 404, 404, AI readiness description: "Fix soft 404s on Astro, Docusaurus, Lovable, Next.js, Webflow and Wix. Real 404 status, a body agents can follow, and the curl that proves AIScan check E1." url: https://aiscan.site/blog/fix-soft-404s-for-ai-agents --- A soft 404 is a page that says "not found" while the status line says 200. On the six platforms below, the default behaviour ranges from a correct 404 with a dead-end body (Webflow, Wix, Docusaurus build) to a 200 for every made-up URL (Lovable on React + Vite, Docusaurus `npm start`, an Astro or Next.js route that finds no data). AIScan check **E1** (Correct 404 handling) grades both the status code and the body, so each platform needs its own fix and the same `curl` to prove it. ## TL;DR - **The test is one command.** `curl -s -o /dev/null -w '%{http_code}\n' https://yoursite.com/this-page-does-not-exist-9f3a` must print `404`, and the body must hold at least two links and the site name. - **Test a dynamic URL too.** `/blog/zzz` exercises the route you fixed. A random path often hits the framework's own handler and passes while the soft 404 hides in your template. - **Never redirect dead URLs to the homepage.** A 301 to `/` ends in a 200 page that does not match the request, which [Google's status code guidance](https://developers.google.com/crawling/docs/troubleshooting/http-status-codes) treats as a soft 404. - **Fastest check of all:** `npx aiscan-cli yoursite.com`, then read **E1, Correct 404 handling** under Discoverability. ## Contents | Platform | Jump to | Default behaviour on a missing URL | |---|---|---| | Astro | [#astro](#astro) | 404 for unmatched URLs, **200** when a dynamic route finds no data | | Docusaurus | [#docusaurus](#docusaurus) | **200** under `npm start`, 404 from the production build, no links in the body | | Lovable | [#lovable](#lovable) | **200** on React + Vite, 404 on TanStack Start | | Next.js | [#next-js](#next-js) | 404, but **200** when `loading.tsx` streams first | | Webflow | [#webflow](#webflow) | 404, stock page has no links; reserved `/404` returns 200 | | Wix | [#wix](#wix) | 404, script-built shell with no links | ## What every platform has in common E1 reads the status line first and the body second. Full credit needs a `404` and a body that names the site and links somewhere useful. [RFC 9110 defines 404](https://www.rfc-editor.org/rfc/rfc9110.html#name-404-not-found) as exactly the "this resource does not exist" signal, and an agent that reads only the status line treats a 200 as a real page. The same four steps apply everywhere, so each section below only says what is different: 1. **Return the status.** Only a server (or the host) can send 404. A client-side router cannot. 2. **Give the body a way out.** One heading, one sentence naming the site, and plain text links to the homepage, your main sections and the sitemap. 3. **Check the layer in front.** A CDN or proxy rule that swaps a 404 for a friendly 200 hides a correct origin. Test the public domain, not `localhost` or the preview address. 4. **Verify by hand.** The commands are in [the shared check](#verify-the-fix-on-any-platform) at the end. ## What I measured on each platform First-party probes, `curl` with a made-up path on live sites or local builds, 10 to 15 October 2026. | Platform (version or sample) | Date | Made-up URL returns | Body | |---|---|---|---| | Astro 7.3.8, `@astrojs/node` 11.1.7, four route patterns | 11 Oct | 404 for three patterns, **200** for the route that prints "not found" | Your `404.astro` only when you rewrite | | Docusaurus 3.10.2, `classic` site, 5 URLs | 14 Oct | **200** under `npm start` (same 1,873-byte shell), **404** after `build` and `serve` | 11 links, none inside `
` | | Lovable, 3 published sites | 13 Oct | **200** on both React + Vite sites, **404** on the TanStack Start site | Vite: empty `
`, 0 links. TanStack: 1 link | | Next.js 16.4.0, App Router, 2 dynamic routes | 10 Oct | 404, 404, and **200** for the segment with `loading.tsx` | `not-found.tsx` content | | Webflow, 2 published sites | 12 Oct | 404 on both; `/404` itself **200** on the designed-page site | Stock page: 906 bytes, 0 links | | Wix, 9 sites (5 on `wixsite.com`, 4 on custom domains) | 15 Oct | 404 on all nine | 2.2 to 2.5 KB script shell, 0 links, `noindex` | ## Astro Astro sends 200 unless you change it. A missing database row or an empty `getEntry()` result is not an error to Astro, so a template that prints "Post not found" is a successful response. [Astro's pages guide](https://docs.astro.build/en/basics/astro-pages/) covers the file for unmatched URLs: a `404.astro` or `404.md` in `src/pages` builds to `404.html`. It says nothing about routes that match a pattern and then find no data, and that gap is where the soft 404 lives. Tested on 11 October 2026. **Step 1: add `src/pages/404.astro`** with a heading, a sentence naming the site and links to home, blog and `/sitemap-index.xml`. **Step 2: reproduce it.** I built a server-rendered project (`output: 'server'`, Node adapter) with one lookup route written four ways: | Route | Miss handling | Status | Body | |---|---|---|---| | a | Prints "Post not found" | **200** | Not-found text, soft 404 | | b | `Astro.response.status = 404` | 404 | Not-found text | | c | `return Astro.rewrite('/404')` | 404 | Your 404 page | | d | `return new Response(null, { status: 404 })` | 404 | Empty | Route a is the failure. Routes b to d fix the status, but only c also serves the recovery links, so it is the one that passes both halves of E1. **Step 3: fix the dynamic route.** In `src/pages/blog/[slug].astro`, put the check before any other work: ```astro --- const { slug } = Astro.params; const post = posts.find((p) => p.slug === slug); if (!post) return Astro.rewrite('/404'); --- ``` `Astro.rewrite()` keeps the visitor on the original URL and renders the other page's content. **The trap in Astro's own routing guide.** The [routing guide](https://docs.astro.build/en/guides/routing/) shows a catch-all route that ends a miss with `return Astro.redirect("/404")`. I tested that exact pattern. The missing URL answers **302** with `location: /404`, and a client that follows it ends on a 404 at a different address. Crawlers that do not follow redirects record a 302. Use `Astro.rewrite('/404')`. **Static builds and hosts.** A fully static site has no dynamic miss, but the host must serve `404.html` with a 404 status. Both the dev server and `astro preview` did that for me, so a 200 on your live site comes from the host. On Cloudflare Workers, [Astro's deploy guide](https://docs.astro.build/en/guides/deploy/cloudflare/) says you must set `not_found_handling` to serve a custom 404 page: ```json { "assets": { "directory": "./dist", "not_found_handling": "404-page" } } ``` Cloudflare's `single-page-application` value serves `/index.html` for unknown navigation requests, which is a soft 404 for every bad URL. **Astro edge cases.** Prerendered routes with `getStaticPaths()` only build the paths you return, so other URLs get the plain 404 page. Middleware that rewrites or catches errors can override a status, so test through the live domain. For the rest of the Astro setup, see [AI readiness setup on Astro](https://aiscan.site/blog/ai-readiness-setup-guide#astro). ## Docusaurus Docusaurus builds a real `404.html`, and the place you test it hides it. Run `npm start`, type a made-up URL and you get a friendly page with a **200** status. The production build and each host behave differently. Tested on 14 October 2026 with Docusaurus 3.10.2, a fresh `classic` site and Node 22.22.0. **Why `npm start` returns 200 for everything.** The dev server hands every path to the client-side router. I started `docusaurus start` on port 3200 and requested five URLs that do not exist, including `/zzz-nope` and `/docs/zzz`. Every one returned `200 text/html` with the same 1,873-byte shell. Nothing about your site is broken; you are testing the wrong server. **Step 1: test the production build.** ```bash npm run build npm run serve -- --port 3100 curl -s -o /dev/null -w '%{http_code} %{content_type}\n' localhost:3100/zzz-nope curl -s -o /dev/null -w '%{http_code}\n' localhost:3100/docs/zzz ``` Both returned `404` with the body of `build/404.html`. A missing `/img/zzz.png` and a missing `/assets/js/zzz.js` also got `404`. If your own site returns `200` here, the cause is in your config. If it returns 404 locally and 200 live, the cause is your host (Step 3). **Step 2: give the 404 page something to follow.** The default page says only that you should "contact the owner of the site that linked you to the original URL". In my build the whole page held 11 links, all in the navbar and footer, and none inside `
`. Docusaurus marks `NotFound/Content` as unsafe to swizzle, so the eject needs `--danger`: ```bash npm run swizzle @docusaurus/theme-classic NotFound/Content -- --eject --danger ``` That writes `src/theme/NotFound/Content/index.js`. Replace its body with real links: ```jsx import React from 'react'; import clsx from 'clsx'; import Link from '@docusaurus/Link'; import Heading from '@theme/Heading'; export default function NotFoundContent({className}) { return (
Page not found

This page is not on example.com. Try one of these instead:

); } ``` After `npm run build`, `build/404.html` had one `

` and three links inside `
`, and `/zzz` still answered `404`. *Why not `src/pages/404.js`?* It works and replaces the built-in page, but Docusaurus treats it as an ordinary page. The build produced `build/404/index.html` and `sitemap.xml` listed `/404`. With the swizzle the sitemap held no 404 entry. A sitemap that advertises your error page is the opposite of what E1 confirms. **Step 3: check what your host does with a miss.** Docusaurus only emits `404.html`; the host decides whether to use it. - **GitHub Pages** serves a `404.html` at the root of the publishing source. A made-up path on `facebook.github.io` returned `404 text/html` on 14 October. - **Cloudflare Pages** looks for the closest `404.html` up the directory tree. [Its docs](https://developers.cloudflare.com/pages/configuration/serving-pages/) add that without a top-level `404.html`, Pages assumes a single-page app and serves the root for any unknown path. Docusaurus always emits that file, so keep it in the output directory. - **Netlify** needs no rule for a `404.html`. [Its redirect docs](https://docs.netlify.com/manage/routing/redirects/redirect-options/) describe the automatic custom 404 page. The trap is a rule such as `/* /index.html 200`, the single-page-app recipe, which turns every bad URL into a 200. I did not deploy to Netlify; this comes from its documentation. `docusaurus.io` itself passes: `/zzz-nope-9f3a` and `/docs/zzz-nope` both returned `404 text/html`. **Docusaurus edge cases.** Trailing slashes did not change the result (`/docs/zzz/` returned `404`). With `baseUrl` set to a repo name on GitHub Pages, URLs outside that folder never reach your `404.html`. See also [AI readiness setup on Docusaurus](https://aiscan.site/blog/ai-readiness-setup-guide#docusaurus). ## Lovable Lovable can return a real 404 or a soft 200 for the same missing URL, depending on the stack the project was built on. I probed live Lovable sites on 13 October 2026: a project on the newer TanStack Start template answered 404, and two older React + Vite projects answered **200** for a path that does not exist. | Site stack | `/zz-nope-123` | `/zz-nope-123.md` | |---|---|---| | React + Vite (two sites) | **200**, ~3.3 to 3.7 KB, `
`, 0 links | 404, `text/plain`, 9 bytes | | TanStack Start (one site) | **404**, `text/html`, ~12.9 KB, rendered "Page not found" | 404 | The Vite sites show why this is a soft 404: the server does not know which routes exist, so it sends the same `index.html` for every path and lets React Router decide in the browser. Note the `.md` row. The platform answers that one itself, so a probe with only a `.md` path would wrongly tell you the site is fine. Lovable's own docs draw the same line: new apps created from 13 May 2026 use server-side rendering, while older apps serve pre-rendered copies only to verified search and AI crawlers. **Step 1: find out which stack you are on.** Ask the project chat "What stack is this project on?", which Lovable's upgrade documentation suggests. Or view source on the live site: an empty `
` means React + Vite. **Step 2: migrate a Vite project to TanStack Start.** Only a server can set a status code, so this is the real fix. Lovable documents three ways in: 1. In the project chat, type `/`, choose **Migrate to TanStack Start** and confirm. 2. Open project settings, go to **General > Project actions** and select **Migrate**. 3. Ask Lovable in plain words to upgrade the app to TanStack Start. Lovable states that most upgrades cost 10 to 35 credits, that the live site keeps serving the old version until you publish again, and that you can revert from version history. Test in the preview, then click **Publish**. If the migrate option is missing, the project is probably on TanStack Start already. **Step 3: give the 404 page links, and cover missing records.** On TanStack Start the router throws a not-found error for any URL that matches no route, and the status follows. What is left is the body. On the site I probed, the 404 page had exactly **one** link, to `/`. [TanStack Router's documentation](https://tanstack.com/router/latest/docs/framework/react/guide/not-found-errors) lets you set one component for the whole app through `defaultNotFoundComponent`: ```tsx const router = createRouter({ defaultNotFoundComponent: () => (

Page not found

Home · Pricing · Blog · Sitemap

), }) ``` Dynamic routes need one more rule. For a record that does not exist, such as `/posts/missing-slug`, throw `notFound()` in the route's `loader`. That call turns a missing record into a not-found response instead of an empty page. **Step 4: if you must stay on React + Vite.** A static host cannot send a 404 for a client-side route. [Google's JavaScript SEO guide](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) names two options for single-page apps. The redirect to a `/not-found` URL that returns 404 does not work here, because every path returns 200. That leaves adding `` when the route is missing: ```tsx useEffect(() => { const meta = document.createElement("meta"); meta.name = "robots"; meta.content = "noindex"; document.head.appendChild(meta); return () => meta.remove(); }, []); ``` This keeps Googlebot from indexing the empty page. It does not change the status code, so E1 still reports a failure. Migrating is the only complete fix. **Lovable edge cases.** Several `lovable.app` sites I tried answer with a 302 to their own domain, so probe the domain visitors use. Lovable warns that some browser-only libraries can break server rendering after the upgrade, so open every page in the preview first. Check one logged-out visit to a private URL after migrating. See also [AI readiness setup on Lovable](https://aiscan.site/blog/ai-readiness-setup-guide#lovable). ## Next.js Next.js has a built-in 404 path, and it works until a `loading.tsx` file sits in the same route segment. Then a missing record returns **HTTP 200** with a "not found" page and a `noindex` tag. Everything here was measured on Next.js 16.4.0 (App Router) on 10 October 2026. **Why the status goes wrong.** HTTP sends the status line before the body. [The `not-found.js` reference](https://nextjs.org/docs/app/api-reference/file-conventions/not-found) says that before streaming starts Next.js returns a `404`, and after streaming starts the status stays `200` while the not-found UI renders. A `loading.tsx` file wraps the page in a Suspense boundary, so the shell streams first and the 200 is already on the wire when `notFound()` throws. The [`loading.js` reference](https://nextjs.org/docs/app/api-reference/file-conventions/loading#status-codes) adds that the `noindex` tag keeps these pages out of search results, which helps Google but does nothing for an agent that never reads the HTML head. **Step 1: reproduce it.** Two dynamic routes call `notFound()` for unknown IDs; only `app/shop/[id]/` has a `loading.tsx`: ```text app/not-found.tsx app/blog/[slug]/page.tsx # notFound() for unknown slug app/shop/[id]/page.tsx # notFound() for unknown id app/shop/[id]/loading.tsx ``` After `npx next build` and `npx next start`: ```bash curl -s -o /dev/null -w '%{http_code}\n' localhost:3000/nowhere # 404 curl -s -o /dev/null -w '%{http_code}\n' localhost:3000/blog/nope # 404 curl -s -o /dev/null -w '%{http_code}\n' localhost:3000/shop/nope # 200 <- soft 404 ``` The `/shop/` miss returns 200 and still carries ``. **Step 2: write a not-found page agents can use.** In `app/not-found.tsx`, render a heading, a sentence naming the site and a list of `Link` components to home, blog and `/sitemap.xml`. It is a Server Component by default, so the links appear in the raw HTML, and Next.js renders it for both unmatched URLs and `notFound()` calls. **Step 3: pick the fix for the 200.** - **Branch A: drop `loading.tsx`.** Delete `app/shop/[id]/loading.tsx`. In my test the sibling `/blog/[slug]` route, which had none, returned a true 404. - **Branch B: keep the loading state.** Decide before streaming begins, which the `loading.js` reference points to [Proxy](https://nextjs.org/docs/app/api-reference/file-conventions/proxy) for. Create `proxy.ts` in the project root, next to `app/`: ```ts import { NextResponse, type NextRequest } from 'next/server' const known = new Set(['1', '2']) // your real IDs or a fast lookup export function proxy(req: NextRequest) { const id = req.nextUrl.pathname.split('/')[2] if (!known.has(id)) { return NextResponse.rewrite(new URL('/not-a-route', req.url)) } } export const config = { matcher: '/shop/:id' } ``` The rewrite sends the request to a path no route matches, so Next.js serves `app/not-found.tsx` with a real 404. Keep the lookup cheap, because the docs warn against fetching full content in Proxy. In Next.js 16 the file is `proxy.ts` and the export is `proxy`; the build output still labels it "Middleware". After this change `/shop/nope` returned `404` with the recovery links, and `/shop/1` still returned `200` with its loading state. **Next.js edge cases.** A `notFound()` call nobody awaits throws where nothing catches it, and no not-found UI renders. Re-run the `/shop/nope` check after adding any new `loading.tsx`. For why agents treat a 200 error page as content, read [the soft-404 discovery problem](https://aiscan.site/blog/soft-404-agent-discovery-problem), and for the content-negotiation side of a good 404 body, [correct 404 handling for AI agents](https://aiscan.site/blog/correct-404-for-ai-agents). The rest of the setup is in [AI readiness setup on Next.js](https://aiscan.site/blog/ai-readiness-setup-guide#next-js) and the [Next.js platform page](https://aiscan.site/docs/platforms/nextjs). ## Webflow Webflow already does the hard part: ask a published site for a URL that does not exist and the server answers with a real 404. What it does not do is make that page useful. Tested on 12 October 2026 against two live Webflow-hosted sites, one with a designed 404 page and one still on the platform's stock error page. A made-up path and a made-up `.md` path both returned **404** on both, so Webflow does not soft-200 one extension and hard-404 another. The difference is the body: | Site | Status for `/zz-nonexist-123` | Body | |---|---|---| | Stock page (`cloneable.webflow.io`) | 404 | 906 bytes, one `

`, **0 links** | | Designed page | 404 | ~50 KB, navigation and footer links | **Step 1: design a 404 page that links somewhere.** Webflow's help article on the 404 error page says it sits under **Utility pages** in the Pages panel. In the Designer: click **Pages**, expand **Utility pages**, click **404**, add a heading, one sentence naming your site, and plain links to the homepage, main sections and sitemap, then **Publish**. Use static elements (a Navbar component, a text link or a Link block). The same article states you cannot add Collection lists to a 404 page, because it has to work even if the CMS is down, so hard-code the links. The slug is reserved at `/404` and cannot be changed, and the page cannot be saved as a draft. **Step 2: keep `/404` itself out of the index.** On the designed-page site, `curl -I https://yoursite.com/404` returned **200**, while every other unknown URL returned 404. The reserved slug is a real page, so it answers like one, and a crawler that finds a link to `/404` sees a 200 page that says "not found". Open the 404 page, go to **Page settings > SEO settings**, switch **Sitemap indexing** off and publish. Webflow documents that this adds `` to the page, and that it works only if the auto-generated sitemap is enabled. On the live site I tested, `/404` was not listed in `sitemap.xml`. **Step 3: redirect URLs that used to exist.** Webflow's redirect documentation requires a paid Site plan, or a site in a Workspace with a paid plan. Go to **Site settings > Publishing > 301 redirects**, enter the old path, enter the new path in **Redirect to path**, click **Add redirect path** and publish. To redirect an existing static page, Webflow says you must first delete it, save it as a draft or change its slug. When you change a slug in Page settings, tick **Permanently redirect the old URL to this new one**. Do not use one wildcard rule to push every unknown URL to the homepage: a redirect answers 301, so a missing page would stop returning 404 at all. **Webflow edge cases.** A reverse proxy or CDN in front of Webflow can replace the 404 with its own 200 page, so test the public domain and not only the `.webflow.io` address. Webflow notes that 301 redirects do not apply to localised slugs, so `/old-url` and `/es/old-url` need separate rules. A CMS item must be archived, deleted, unpublished or re-slugged before a redirect rule for its URL applies. Webflow's help center returned 403 to my automated fetch on 27 October 2026, so the click paths above are as I recorded them on 12 October. See also [AI readiness setup on Webflow](https://aiscan.site/blog/ai-readiness-setup-guide#webflow). ## Wix Wix gets the status code right and leaves the page empty. I requested a made-up path on nine Wix sites, five on `wixsite.com` addresses and four on custom domains, on 15 October 2026. All nine answered `404 text/html`, between 2,204 and 2,505 bytes, with `` and the title "404 Error: Page Not Found". Inside the body there is one `
` and a script that draws the message. There were zero `` tags in any of them, so a crawler or agent that does not run scripts finds nothing to follow. AIScan scored one of those sites (a Wix manufacturer's site) as a pass on E1, with the evidence line `HTTP 404, 2206 bytes`. The status is right; what the default page cannot give a script-less reader is a way out. None of the nine sites had a custom 404 page, so I could not measure one. Step 1 builds it and Step 3 is the test. **Step 1: add a 404 page with links.** Per [Wix's help article](https://support.wix.com/en/article/wix-editor-customizing-your-404-error-page), a custom 404 page needs a premium plan. In the Editor: click **Pages & Menu**, **Add Menu Item**, **New Page**, **(404) Error Page**, hover a template and click **Add Page**. Edit the text, add your site menu and links to the sections visitors look for, then **Publish**. The page URL always contains `error404`, though you can change the title in Page Settings. Name the site in the sentence and use text links, not only buttons or an image. **Step 2: do not redirect dead URLs to the homepage.** The Wix **URL Redirect Manager** creates 301 redirects, and it is easy to point every retired URL at `/`. Redirect an old URL to its closest replacement, and when there is none, leave it to 404. Limits from [Wix's redirect article](https://support.wix.com/en/article/setting-up-a-301-redirect-for-a-page-on-your-site): - 301 redirects only work with a **custom domain**; a free `wixsite.com` address cannot have them. - You cannot create a redirect whose source is your homepage. - A site can hold at most 5,000 redirects. - Wix creates a redirect automatically when you change an Editor page slug. **Blog pages are the exception**: change a blog slug and you must add the 301 yourself, or the old URL 404s. **Step 3: check the layers in front of Wix, then verify.** A CDN or proxy in front of a custom domain can replace a `404` with a friendly `200`. Run the shared check below against the live domain. On the default Wix page I measured a link count of `0`. If the status prints `200` after you add your page, Wix support is the next stop. I found no live Wix site with a custom page to measure, so the command is the test, not my word. See also [AI readiness setup on Wix](https://aiscan.site/blog/ai-readiness-setup-guide#wix). ## Verify the fix on any platform **With AIScan (default):** run `npx aiscan-cli yoursite.com`, open **Discoverability** and find **E1, Correct 404 handling**. It should read pass, with the 404 body counted. The [Discoverability checks page](https://aiscan.site/docs/checks/discoverability) explains the grading. **By hand with curl:** ```bash curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://yoursite.com/this-page-does-not-exist-9f3a curl -s https://yoursite.com/this-page-does-not-exist-9f3a | grep -a -c '