---
title: "Duplicate www and apex Hosts: 38.2% of 696 Sites Fail the Canonical Check"
slug: canonical-host-https-corpus
published: 2026-10-07T03:16:09.265044+00:00
updated: 2026-10-07T03:16:09.265044+00:00
author: "AIScan Content Team"
author_url: https://aiscan.site/about
category: "AI Readiness"
tags: check:E4, check:D1, check:D2, corpus study, canonical host, HTTPS, redirects, HSTS
description: "696 sites measured on canonical host and HTTPS: 266 serve a duplicate www copy, 105 skip the HTTP upgrade. Counts by platform, plus the redirect fix. (E4)"
url: https://aiscan.site/blog/canonical-host-https-corpus
---

On 7 October 2026 we queried the AIScan database for check E4, the canonical host and HTTPS check, across **696 distinct hosts**. 405 pass. The other 291 are partial, and not one fails outright. The partials share a single cause: **266 hosts (38.2%) answer on both the bare domain and the `www.` host and serve a full copy from each**, and 105 hosts (15.1%) leave plain `http://` without an upgrade to HTTPS. An agent that follows a link to either duplicate gets a valid page. It also gets a second URL for every page on the site, and nothing in the response says which one is the real one.

This post is the adoption count behind that number, broken down by redirect behaviour, HSTS and platform, with the limits of the measurement stated before the tables. It then walks through the fix for nginx, Apache, a CDN rule and WordPress, and shows how to verify it with [AIScan](https://aiscan.site/) (`check:E4`) or by hand with `curl`.

## Quick summary

| Question | Answer from 696 hosts | What to do |
|---|---|---|
| How many sites pass E4? | 405 (58.2%) | Keep the redirect rules you have |
| How many get partial credit? | 291 (41.8%), zero hard fails | Fix the two causes below |
| Most common problem | Both apex and `www.` serve their own copy: 266 hosts (38.2%) | Pick one host, 301 the other to it |
| Second problem | `http://` does not upgrade to HTTPS: 105 hosts (15.1%) | Add a site-wide 301 from HTTP to HTTPS |
| Worst case | HTTP stays on HTTP **and** the two hosts duplicate: 83 hosts (11.9%) | Both fixes, in that order |
| Does HSTS change the verdict? | No. 187 hosts pass without it | Add it after the redirects, not instead of them |
| Platform with the most duplicates (100+ hosts) | Hosts with no detected platform (51.3%), then Next.js (41.3%) | Check your host or CDN redirect settings |

**Expert recommendation:** [Google's canonicalization documentation](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls) lists redirects first among the methods it uses, calling them "a strong signal that the target of the redirect should become canonical". It tells site owners to pick one of the URL variants as canonical and redirect the others to it. E4 checks exactly that setup.

**Best for a one-site owner:** one redirect rule from the host you do not use to the host you do.
**Best for a WordPress owner:** confirm the Site Address under Settings, General, then test both hosts once.
**Best for a platform team:** put the redirect at the CDN or load balancer so every app behind it inherits it.

## What E4 actually checks

E4 is one of the five discoverability checks, alongside D1, D2, D3 and E1, and it carries a weight of 4 in the 2026.08.2 rubric. The evidence strings it writes follow one fixed template, which makes them easy to count:

`final origin <URL> · http:// <upgrades to https | does NOT upgrade to https | unreachable> · <host> <redirects to the canonical host | serves its own copy (duplicate origin) | unreachable (fine)> · <HSTS set | no HSTS header>`

Four facts per scan, then. Which origin the request lands on after redirects, whether plain HTTP is upgraded, what the *other* host (`www.` when you scanned the apex, the bare domain when you scanned `www.`) does, and whether the final response carries a `Strict-Transport-Security` header.

The scoring is what the data shows it to be, not what we assumed when we wrote the check. Across the corpus the check awards 4 of 4 to 405 hosts, 3 of 4 to 205 hosts and 2 of 4 to 86 hosts. Grouping the evidence strings by status gives a clean mapping: a pass means HTTP upgrades and the other host redirects (or does not exist). A duplicate origin with a working HTTPS upgrade earns 3. A missing HTTPS upgrade, usually together with a duplicate origin, earns 2. HSTS appears in the evidence but does not move the score.

### Why a crawler or agent cares about this at all

An agent that fetches a page does not know your preferred hostname. It knows the URL it was handed, by a search result, a link, a sitemap or a training-data snapshot. If `https://example.com/pricing` and `https://www.example.com/pricing` both return 200 with the same content, any index that stores pages by URL now holds two entries for one page. Citation counts split. A retrieval system that deduplicates by exact URL keeps both. A `rel="canonical"` tag in the HTML can paper over this for search engines that read it, but an agent that fetches and summarises one page often never looks at it.

Google's documentation describes the same problem for its own index. [Google Search Central](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls) says it prefers HTTPS pages over equivalent HTTP pages as canonical, with exceptions such as an invalid certificate or insecure dependencies, and recommends using permanent redirects to tell Google that a redirected URL is a worse version than its target. Search engines have years of heuristics for resolving duplicates. A newer agent that reads a page once has far fewer.

## Method and limits, before any table

**Population.** Every scan in the AIScan production database taken under rubric version 2026.08.2, reduced to **one latest scan per host** (hosts compared case-insensitively with `www.` and the path removed), with `aiscan.site` excluded. That leaves 696 hosts. No individual host is named in this post.

**What the population is.** These are sites that someone chose to scan with AIScan, or that we scanned while researching earlier posts. It is not a random sample of the web and it skews toward people who already care about agent readiness, which probably *flatters* the pass rate. Treat 58.2% as a ceiling for the ordinary web.

**What the probe sees.** E4 runs from AIScan's scanner, from one network location, at the time of the scan. A redirect that depends on country, user agent or cookie is outside what it observes. The scan date differs per host, so a site that fixed its redirects last week and has not been rescanned still shows up as partial.

**What the evidence does not record.** The evidence string has no redirect hop count and no status code for the intermediate responses. We had planned to report the average number of hops an agent spends before the first byte. The data cannot support that, so we are not reporting it. Counting hops would need a fresh probe, which is a different study.

**A quirk we found while reading the strings.** For hosts that are already subdomains, the probe still tries a `www.` prefix on them (a subdomain of a larger site shows up in the data with a `www.` variant that serves its own copy). Some of the 266 duplicate-origin verdicts are therefore hosts for which no `www.` variant was ever meant to exist. We cannot separate them with the stored evidence, so the 38.2% figure includes them. We flag it as a product issue below.

**A word on causation.** Where a table compares average scores between groups, it is an association on observational data. Sites that handle redirects well also tend to handle many other things well.

## The result: two questions, four boxes

Every host answers two independent questions. Does plain HTTP upgrade to HTTPS? And does the other hostname redirect to the canonical one, or serve its own copy? Crossing them gives four boxes (plus a few unreachable cases):

| | Other host redirects (or absent) | Other host serves its own copy |
|---|---|---|
| **HTTP upgrades to HTTPS** | **403 hosts**, E4 pass, 4 of 4 | **180 hosts**, partial, 3 of 4 |
| **HTTP does not upgrade** | **20 hosts**, partial | **83 hosts**, partial, 2 of 4 |

The remaining 10 hosts have one side unreachable. Two of them are passes because the missing `www.` host is treated as fine.

Three things stand out.

**1. The duplicate origin is the main problem, not HTTPS.** 266 hosts have a duplicate origin against 105 with no HTTPS upgrade. Most operators have been told for years to get HTTPS working, and the data agrees: 585 of 696 hosts (84.1%) upgrade HTTP to HTTPS. Far fewer finished the second half of the job, which is making the two hostnames collapse into one.

**2. The two failures cluster.** Of the 105 hosts with no HTTPS upgrade, 83 also have a duplicate origin. That is 79% of them. A site without a redirect from HTTP is usually a site where nobody has written any redirect rule at all, or the redirect lives in a place the scanner cannot reach, such as a client-side script.

**3. HSTS is a separate decision.** Of the 696 hosts, 319 (45.8%) send an HSTS header. Among passes the rate is 218 of 405 (53.8%), among partials 101 of 291 (34.7%). Because 187 hosts pass E4 without HSTS, the check clearly does not require it. We cover why you may still want it in the fix section.

### The evidence strings, counted

The 22 templates collapse into the following. This is the part you can reproduce by running the query shown at the end.

| Evidence pattern (host replaced by H) | Hosts | Status |
|---|---|---|
| HTTP upgrades · www redirects to canonical · no HSTS | 151 | pass |
| HTTP upgrades · www redirects to canonical · HSTS set | 141 | pass |
| HTTP upgrades · www serves own copy · no HSTS | 93 | partial |
| HTTP upgrades · bare domain redirects to canonical · HSTS set | 76 | pass |
| HTTP upgrades · www serves own copy · HSTS set | 74 | partial |
| HTTP does NOT upgrade · www serves own copy · no HSTS | 59 | partial |
| HTTP upgrades · bare domain redirects to canonical · no HSTS | 35 | pass |
| HTTP does NOT upgrade · www serves own copy · HSTS set | 15 | partial |
| All other combinations (14 patterns) | 52 | mixed |

The last row is mostly small combinations, for example HTTP unreachable, or no upgrade with a working redirect between hosts.

One pattern in that table deserves a second look: 15 hosts send an HSTS header on the final HTTPS response while plain HTTP still does not redirect. That combination looks odd, and it is not a contradiction. The header is only honoured when it arrives over HTTPS, so a browser that has visited the HTTPS site once will upgrade by itself, while every first-time visitor, crawler and agent still gets the plain-HTTP response.

## Does your platform change the odds?

The platform column comes from AIScan's own detection, so a quarter of the corpus has none. The sample sizes below are uneven, so the small rows are indicative only.

| Detected platform | Hosts | Duplicate origin | Share | No HTTPS upgrade | Share |
|---|---|---|---|---|---|
| None detected | 261 | 134 | 51.3% | 71 | 27.2% |
| WordPress | 185 | 29 | 15.7% | 7 | 3.8% |
| Next.js | 160 | 66 | 41.3% | 17 | 10.6% |
| Astro | 34 | 12 | 35.3% | 4 | 11.8% |
| Shopify | 17 | 2 | 11.8% | 0 | 0% |
| Lovable | 11 | 6 | 54.5% | 0 | 0% |

Two readings fit the data.

**Managed or CMS platforms tend to handle it for you.** WordPress normally redirects to the Site Address you set, and Shopify lets a store mark one domain as primary. Their duplicate rates, 15.7% and 11.8%, are the lowest in the table. A platform that makes one hostname the primary by default produces fewer duplicates than a platform that asks you to decide.

**Framework deployments leave it to the host.** A Next.js or Astro site serves whatever hostnames its host maps to it. If you add both `example.com` and `www.example.com` as domains and never mark one as the redirect, both answer. That is the default state, not a mistake in the code, which is why 41.3% of Next.js hosts and 35.3% of Astro hosts land in the duplicate-origin column. The fix lives in the hosting dashboard or the edge, and the framework itself rarely needs to change.

The "none detected" row has the worst numbers on both axes, and the 27.2% with no HTTPS upgrade is over seven times the WordPress rate. We can only guess what is in that group (hand-built sites, static hosts, unusual stacks), so we leave it at the observation.

## What the score gap looks like

The mean overall score of hosts that pass E4 is **63.6**. For hosts with a partial E4 it is **44.0**. A gap of 19.6 points looks like a finding, and it is mostly not one.

Platform composition explains much of it. The no-platform-detected group is the largest and has the lowest scores, and it also has the most duplicate origins. Within one platform the gap changes size and sometimes direction: among Shopify hosts the mean for passes and partials is almost identical (66.6 against 67.0), while among Astro hosts it is 82.0 against 40.8, from small samples.

The honest summary is that E4 is a **hygiene signal that travels with other signals**. Fixing E4 alone will earn the points E4 carries, up to 2 of its 4, and it will not drag the other checks up behind it. For the ranked view of which checks move the overall score most, see [the agent-readiness fix ranking](https://aiscan.site/blog/agent-readiness-highest-impact-fix). E4 is cheap, though: it is a few lines of server configuration, once, with no content changes.

## Why a duplicate host matters more for agents than for people

A person types one address and never sees the other. An agent meets both. Four routes lead to a duplicate:

1. **Inbound links.** Other sites link to whichever form they typed. Both forms accumulate links.
2. **Your own sitemap.** [Sitemap files list absolute URLs](https://aiscan.site/blog/sitemap-reality-corpus-study). If your sitemap uses the apex and a theme or plugin writes `www.` links inside the pages, a crawler sees both.
3. **Training and retrieval snapshots.** A model built from a crawl in which both forms appeared may recall either one, and a citation then points at whichever it stored.
4. **Redirect-blind fetchers.** Some tools fetch a URL once and stop at the first 200. When that 200 comes from the host you did not intend, there is no second chance to correct it.

None of these requires the duplicate to be malicious or even visible. They only require that both hostnames answer, and 266 of 696 hosts in this corpus have that setup today.

### What HSTS adds, and what it cannot

The [HSTS specification, RFC 6797](https://www.rfc-editor.org/rfc/rfc6797), says a user agent must ignore any `Strict-Transport-Security` header that arrives over insecure transport. That one rule produces the limit we saw in the 15 hosts above: HSTS is a memory held by browsers that have already made a secure request. It protects repeat visitors. It does nothing for a crawler or agent whose first and only request is `http://`, because that request is answered in the clear, and a server that does not redirect it hands back the plain-HTTP page.

So the order is fixed. **The redirect handles the first request. HSTS handles the second and later requests from browsers.** Rolling it out with a short `max-age` first is the cautious way to do it.

## The fix, in the order that avoids breaking things

The goal is one canonical origin: one scheme and one host. Pick it first. Either the apex (`https://example.com`) or `www` works. The choice matters less than making it once and keeping it. If you already have inbound links and indexed pages on one form, keep that form.

### Step 1: make sure both hostnames and both schemes have a valid certificate

A redirect from a host with a certificate error never reaches the browser or agent as a redirect. It fails the TLS handshake first. Confirm that the certificate covers the apex and `www`, or that your CDN terminates TLS for both. This is the step skipped most often when a site "only uses the apex".

### Step 2: add the redirects

**nginx**, with the apex as canonical:

```nginx
server {
    listen 80;
    listen [::]:80;
    server_name example.com www.example.com;
    return 301 https://example.com$request_uri;
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name www.example.com;
    # ssl_certificate lines for the www host go here
    return 301 https://example.com$request_uri;
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name example.com;
    # your real site configuration
}
```

The first block sends HTTP for both hosts straight to the final origin in one hop. A common mistake is to redirect `http://www` to `http://example.com` and then redirect that to HTTPS, which works but spends two hops.

**Apache** (`.htaccess` or a virtual host with `mod_rewrite`):

```apache
RewriteEngine On
RewriteCond %{HTTPS} off [OR]
RewriteCond %{HTTP_HOST} ^www\. [NC]
RewriteRule ^ https://example.com%{REQUEST_URI} [L,R=301]
```

**A CDN or hosting redirect rule.** Most hosts and CDNs have a "redirect www to apex" or "redirect apex to www" toggle in the domain settings, plus an "always use HTTPS" setting. Turn on both, and make sure the rule is set to permanent (301 or 308) rather than temporary. If your DNS and your host disagree about which hostname is primary, the host setting wins for what visitors see.

**WordPress.** Under Settings, General, set the WordPress Address and the Site Address to the exact canonical form, scheme included. WordPress will redirect the other host to that address on its own. If the site still serves both, the redirect is usually being cached upstream or overridden by a server rule, so check the layer in front of PHP. On the head side, one plugin should own the canonical tag and the sitemap. [ThinkRank](https://thinkrank.ai) is the plugin we point WordPress readers to for those: it replaces the several plugins that otherwise fight over robots, sitemap and schema output, and it migrates settings from Rank Math, Yoast, AIOSEO and SEOPress. It cannot add a server redirect, so treat it as the second half of the job, not the first.

### Step 3: use permanent redirects and a single hop

Google's documentation tells site owners to use permanent redirects for duplicates and says server-side redirects take effect quickest. Use 301 or 308. Keep the hop count at one wherever you can: HTTP on either host should land directly on the final HTTPS origin, not wander through an intermediate hostname first.

### Step 4: only then add HSTS

```
Strict-Transport-Security: max-age=300
```

Start with a five-minute `max-age`, confirm nothing breaks, then raise it. Adding `includeSubDomains` commits every subdomain you own to HTTPS, including ones you forgot about, so check the subdomain list before you add it. Setting a year-long `max-age` before the redirects work is how sites lock themselves out of their own staging hosts.

### Step 5: align the signals that point at the host

After the redirect works, make everything else agree with it: the `rel="canonical"` tag, the URLs in your sitemap, the URLs in structured data, and the absolute links your templates generate. A redirect fixes what happens when someone arrives on the wrong host. These fix why they keep arriving there.

## Verify it: the scan first, `curl` second

**Run AIScan.** `npx aiscan-cli yourdomain.com`, or paste the URL at [aiscan.site](https://aiscan.site/). It is free and needs no account. The report names check **E4** (canonical host and HTTPS) and its evidence line has the template shown earlier, so you can read your own result directly: both "upgrades to https" and "redirects to the canonical host" mean a pass. The neighbouring discoverability checks, D1, D2, D3 and E1, are in the same report, and their pages are collected under [discoverability in the docs](https://aiscan.site/docs/checks/discoverability).

**What AIScan cannot see.** It probes from one location, so it will not catch a redirect that only misfires for a particular country or a logged-in session. It also does not record the number of hops. Use `curl` for those two.

**If you would rather check by hand**, three commands cover the whole check:

```bash
# 1. Does plain HTTP upgrade, and in how many hops?
curl -sIL -o /dev/null -w 'final=%{url_effective} hops=%{num_redirects} code=%{http_code}\n' http://example.com/

# 2. Does the other host collapse into the canonical one?
curl -sIL -o /dev/null -w 'final=%{url_effective} hops=%{num_redirects} code=%{http_code}\n' https://www.example.com/

# 3. What does the first redirect actually return?
curl -sI http://www.example.com/ | head -n 5
```

Run all three against both hosts. You want the same `final=` URL every time, `hops=1` at most, and `301` or `308` on the first response in command 3. A `302` or `307` means the redirect is temporary.

## Common mistakes

| Mistake | What you see | Fix |
|---|---|---|
| Redirect loop between apex and `www` | `curl` reports "too many redirects"; the browser shows ERR_TOO_MANY_REDIRECTS | One rule only: the CDN and the origin must not redirect in opposite directions |
| Certificate covers apex only | `curl: (60)` on `https://www.` | Add `www` to the certificate, or let the CDN terminate TLS |
| HTTP redirects to the wrong host first | `hops=2` or more | Send HTTP for both hosts straight to the final origin |
| Temporary redirect left from setup | `302` in the first response | Change the rule to 301 or 308 |
| Redirect drops the path | Every old URL lands on the homepage | Pass `$request_uri` or `%{REQUEST_URI}` through |
| HSTS added with `includeSubDomains` too early | Staging or legacy subdomains stop loading | Remove the directive, wait for the `max-age` to expire, then add it back with a plan |
| Canonical tag disagrees with the redirect | Tag says `www`, server sends visitors to apex | Make both point at the same origin |

## What this means for the product

Reading 696 evidence strings turned up three issues in E4 itself. We list them because publishing the limits of our own check is part of reporting the data.

1. **The evidence omits hops and intermediate status codes.** That is why the hop-count analysis above is missing. Recording the chain would also let a transient error be told apart from an absent redirect.
2. **Subdomains get a `www.` probe.** A subdomain of a larger site should not be marked down for failing to serve a `www.` prefix that nobody uses. The 266 figure includes these.
3. **The unreachable-host wording says "fine".** Two passes rest on a host being unreachable. That is a reasonable reading for a `www.` that does not exist, and a poor one for a timeout.

None of these changes the main finding. The 291 partials mostly come from real duplicate origins, and the redirect rules above fix them. We will rerun the numbers after the probe is updated.

## Reproduce the numbers

The queries below run against the AIScan scans table. They aggregate in SQL and never select rows, which is why no host appears in this post.

```sql
with latest as (
  select distinct on (lower(split_part(regexp_replace(url,'^https?://(www\.)?',''),'/',1)))
         url, platform, score, result
  from scans
  where rubric_version = '2026.08.2' and url not ilike '%aiscan.site%'
  order by lower(split_part(regexp_replace(url,'^https?://(www\.)?',''),'/',1)), created_at desc
),
e4 as (
  select platform, score, c->>'status' s, c->>'evidence' ev
  from latest, jsonb_array_elements(result->'checks') c
  where c->>'id' = 'E4'
)
select count(*) n,
       count(*) filter (where s = 'pass') pass,
       count(*) filter (where s = 'partial') partial,
       count(*) filter (where ev ilike '%duplicate origin%') duplicate_origin,
       count(*) filter (where ev ilike '%does NOT upgrade%') no_https_upgrade,
       count(*) filter (where ev ilike '%HSTS set%') hsts
from e4;
```

## Your next action

Scan your domain with [AIScan](https://aiscan.site/) and read the **E4** line. If it says "serves its own copy (duplicate origin)", add the redirect from the hostname you do not use. If it says "does NOT upgrade to https", add the site-wide HTTP-to-HTTPS 301 first. Rescan, then add HSTS with a short `max-age`. The rest of the discoverability family, D1, D2, D3 and E1, is covered in the [discoverability checks](https://aiscan.site/docs/checks/discoverability), and every guide is collected at [/guides](https://aiscan.site/guides). For the wider picture of where sites stand today, see [the state of AI agent readiness in 2026](https://aiscan.site/blog/state-of-ai-agent-readiness-2026).