Duplicate www and apex Hosts: 38.2% of 696 Sites Fail the Canonical Check

On 7 October 2026 we queried the AIScan database for check E4, the canonical host and HTTPS check, across 696 distinct hosts. 405 pass. The other 291 are partial, and not one fails outright. The partials share a single cause: 266 hosts (38.2%) answer on both the bare domain and the www. host and serve a full copy from each, and 105 hosts (15.1%) leave plain http:// without an upgrade to HTTPS. An agent that follows a link to either duplicate gets a valid page. It also gets a second URL for every page on the site, and nothing in the response says which one is the real one.

This post is the adoption count behind that number, broken down by redirect behaviour, HSTS and platform, with the limits of the measurement stated before the tables. It then walks through the fix for nginx, Apache, a CDN rule and WordPress, and shows how to verify it with AIScan (check:E4) or by hand with curl.

Quick summary

Expert recommendation: Google's canonicalization documentation lists redirects first among the methods it uses, calling them "a strong signal that the target of the redirect should become canonical". It tells site owners to pick one of the URL variants as canonical and redirect the others to it. E4 checks exactly that setup.

Best for a one-site owner: one redirect rule from the host you do not use to the host you do. Best for a WordPress owner: confirm the Site Address under Settings, General, then test both hosts once. Best for a platform team: put the redirect at the CDN or load balancer so every app behind it inherits it.

What E4 actually checks

E4 is one of the five discoverability checks, alongside D1, D2, D3 and E1, and it carries a weight of 4 in the 2026.08.2 rubric. The evidence strings it writes follow one fixed template, which makes them easy to count:

final origin <URL> · http:// <upgrades to https | does NOT upgrade to https | unreachable> · <host> <redirects to the canonical host | serves its own copy (duplicate origin) | unreachable (fine)> · <HSTS set | no HSTS header>

Four facts per scan, then. Which origin the request lands on after redirects, whether plain HTTP is upgraded, what the other host (www. when you scanned the apex, the bare domain when you scanned www.) does, and whether the final response carries a Strict-Transport-Security header.

The scoring is what the data shows it to be, not what we assumed when we wrote the check. Across the corpus the check awards 4 of 4 to 405 hosts, 3 of 4 to 205 hosts and 2 of 4 to 86 hosts. Grouping the evidence strings by status gives a clean mapping: a pass means HTTP upgrades and the other host redirects (or does not exist). A duplicate origin with a working HTTPS upgrade earns 3. A missing HTTPS upgrade, usually together with a duplicate origin, earns 2. HSTS appears in the evidence but does not move the score.

Why a crawler or agent cares about this at all

An agent that fetches a page does not know your preferred hostname. It knows the URL it was handed, by a search result, a link, a sitemap or a training-data snapshot. If https://example.com/pricing and https://www.example.com/pricing both return 200 with the same content, any index that stores pages by URL now holds two entries for one page. Citation counts split. A retrieval system that deduplicates by exact URL keeps both. A rel="canonical" tag in the HTML can paper over this for search engines that read it, but an agent that fetches and summarises one page often never looks at it.

Google's documentation describes the same problem for its own index. Google Search Central says it prefers HTTPS pages over equivalent HTTP pages as canonical, with exceptions such as an invalid certificate or insecure dependencies, and recommends using permanent redirects to tell Google that a redirected URL is a worse version than its target. Search engines have years of heuristics for resolving duplicates. A newer agent that reads a page once has far fewer.

Method and limits, before any table

Population. Every scan in the AIScan production database taken under rubric version 2026.08.2, reduced to one latest scan per host (hosts compared case-insensitively with www. and the path removed), with aiscan.site excluded. That leaves 696 hosts. No individual host is named in this post.

What the population is. These are sites that someone chose to scan with AIScan, or that we scanned while researching earlier posts. It is not a random sample of the web and it skews toward people who already care about agent readiness, which probably flatters the pass rate. Treat 58.2% as a ceiling for the ordinary web.

What the probe sees. E4 runs from AIScan's scanner, from one network location, at the time of the scan. A redirect that depends on country, user agent or cookie is outside what it observes. The scan date differs per host, so a site that fixed its redirects last week and has not been rescanned still shows up as partial.

What the evidence does not record. The evidence string has no redirect hop count and no status code for the intermediate responses. We had planned to report the average number of hops an agent spends before the first byte. The data cannot support that, so we are not reporting it. Counting hops would need a fresh probe, which is a different study.

A quirk we found while reading the strings. For hosts that are already subdomains, the probe still tries a www. prefix on them (a subdomain of a larger site shows up in the data with a www. variant that serves its own copy). Some of the 266 duplicate-origin verdicts are therefore hosts for which no www. variant was ever meant to exist. We cannot separate them with the stored evidence, so the 38.2% figure includes them. We flag it as a product issue below.

A word on causation. Where a table compares average scores between groups, it is an association on observational data. Sites that handle redirects well also tend to handle many other things well.

The result: two questions, four boxes

Every host answers two independent questions. Does plain HTTP upgrade to HTTPS? And does the other hostname redirect to the canonical one, or serve its own copy? Crossing them gives four boxes (plus a few unreachable cases):

The remaining 10 hosts have one side unreachable. Two of them are passes because the missing www. host is treated as fine.

Three things stand out.

1. The duplicate origin is the main problem, not HTTPS. 266 hosts have a duplicate origin against 105 with no HTTPS upgrade. Most operators have been told for years to get HTTPS working, and the data agrees: 585 of 696 hosts (84.1%) upgrade HTTP to HTTPS. Far fewer finished the second half of the job, which is making the two hostnames collapse into one.

2. The two failures cluster. Of the 105 hosts with no HTTPS upgrade, 83 also have a duplicate origin. That is 79% of them. A site without a redirect from HTTP is usually a site where nobody has written any redirect rule at all, or the redirect lives in a place the scanner cannot reach, such as a client-side script.

3. HSTS is a separate decision. Of the 696 hosts, 319 (45.8%) send an HSTS header. Among passes the rate is 218 of 405 (53.8%), among partials 101 of 291 (34.7%). Because 187 hosts pass E4 without HSTS, the check clearly does not require it. We cover why you may still want it in the fix section.

The evidence strings, counted

The 22 templates collapse into the following. This is the part you can reproduce by running the query shown at the end.

The last row is mostly small combinations, for example HTTP unreachable, or no upgrade with a working redirect between hosts.

One pattern in that table deserves a second look: 15 hosts send an HSTS header on the final HTTPS response while plain HTTP still does not redirect. That combination looks odd, and it is not a contradiction. The header is only honoured when it arrives over HTTPS, so a browser that has visited the HTTPS site once will upgrade by itself, while every first-time visitor, crawler and agent still gets the plain-HTTP response.

Does your platform change the odds?

The platform column comes from AIScan's own detection, so a quarter of the corpus has none. The sample sizes below are uneven, so the small rows are indicative only.

Two readings fit the data.

Managed or CMS platforms tend to handle it for you. WordPress normally redirects to the Site Address you set, and Shopify lets a store mark one domain as primary. Their duplicate rates, 15.7% and 11.8%, are the lowest in the table. A platform that makes one hostname the primary by default produces fewer duplicates than a platform that asks you to decide.

Framework deployments leave it to the host. A Next.js or Astro site serves whatever hostnames its host maps to it. If you add both example.com and www.example.com as domains and never mark one as the redirect, both answer. That is the default state, not a mistake in the code, which is why 41.3% of Next.js hosts and 35.3% of Astro hosts land in the duplicate-origin column. The fix lives in the hosting dashboard or the edge, and the framework itself rarely needs to change.

The "none detected" row has the worst numbers on both axes, and the 27.2% with no HTTPS upgrade is over seven times the WordPress rate. We can only guess what is in that group (hand-built sites, static hosts, unusual stacks), so we leave it at the observation.

What the score gap looks like

The mean overall score of hosts that pass E4 is 63.6. For hosts with a partial E4 it is 44.0. A gap of 19.6 points looks like a finding, and it is mostly not one.

Platform composition explains much of it. The no-platform-detected group is the largest and has the lowest scores, and it also has the most duplicate origins. Within one platform the gap changes size and sometimes direction: among Shopify hosts the mean for passes and partials is almost identical (66.6 against 67.0), while among Astro hosts it is 82.0 against 40.8, from small samples.

The honest summary is that E4 is a hygiene signal that travels with other signals. Fixing E4 alone will earn the points E4 carries, up to 2 of its 4, and it will not drag the other checks up behind it. For the ranked view of which checks move the overall score most, see the agent-readiness fix ranking. E4 is cheap, though: it is a few lines of server configuration, once, with no content changes.

Why a duplicate host matters more for agents than for people

A person types one address and never sees the other. An agent meets both. Four routes lead to a duplicate:

1. Inbound links. Other sites link to whichever form they typed. Both forms accumulate links.
2. Your own sitemap. Sitemap files list absolute URLs. If your sitemap uses the apex and a theme or plugin writes www. links inside the pages, a crawler sees both.
3. Training and retrieval snapshots. A model built from a crawl in which both forms appeared may recall either one, and a citation then points at whichever it stored.
4. Redirect-blind fetchers. Some tools fetch a URL once and stop at the first 200. When that 200 comes from the host you did not intend, there is no second chance to correct it.

None of these requires the duplicate to be malicious or even visible. They only require that both hostnames answer, and 266 of 696 hosts in this corpus have that setup today.

What HSTS adds, and what it cannot

The HSTS specification, RFC 6797, says a user agent must ignore any Strict-Transport-Security header that arrives over insecure transport. That one rule produces the limit we saw in the 15 hosts above: HSTS is a memory held by browsers that have already made a secure request. It protects repeat visitors. It does nothing for a crawler or agent whose first and only request is http://, because that request is answered in the clear, and a server that does not redirect it hands back the plain-HTTP page.

So the order is fixed. The redirect handles the first request. HSTS handles the second and later requests from browsers. Rolling it out with a short max-age first is the cautious way to do it.

The fix, in the order that avoids breaking things

The goal is one canonical origin: one scheme and one host. Pick it first. Either the apex (https://example.com) or www works. The choice matters less than making it once and keeping it. If you already have inbound links and indexed pages on one form, keep that form.

Step 1: make sure both hostnames and both schemes have a valid certificate

A redirect from a host with a certificate error never reaches the browser or agent as a redirect. It fails the TLS handshake first. Confirm that the certificate covers the apex and www, or that your CDN terminates TLS for both. This is the step skipped most often when a site "only uses the apex".

Step 2: add the redirects

nginx, with the apex as canonical:

server {
    listen 80;
    listen [::]:80;
    server_name example.com www.example.com;
    return 301 https://example.com$request_uri;
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name www.example.com;
    # ssl_certificate lines for the www host go here
    return 301 https://example.com$request_uri;
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name example.com;
    # your real site configuration
}

The first block sends HTTP for both hosts straight to the final origin in one hop. A common mistake is to redirect http://www to http://example.com and then redirect that to HTTPS, which works but spends two hops.

Apache (.htaccess or a virtual host with mod_rewrite):

RewriteEngine On
RewriteCond %{HTTPS} off [OR]
RewriteCond %{HTTP_HOST} ^www\. [NC]
RewriteRule ^ https://example.com%{REQUEST_URI} [L,R=301]

A CDN or hosting redirect rule. Most hosts and CDNs have a "redirect www to apex" or "redirect apex to www" toggle in the domain settings, plus an "always use HTTPS" setting. Turn on both, and make sure the rule is set to permanent (301 or 308) rather than temporary. If your DNS and your host disagree about which hostname is primary, the host setting wins for what visitors see.

WordPress. Under Settings, General, set the WordPress Address and the Site Address to the exact canonical form, scheme included. WordPress will redirect the other host to that address on its own. If the site still serves both, the redirect is usually being cached upstream or overridden by a server rule, so check the layer in front of PHP. On the head side, one plugin should own the canonical tag and the sitemap. ThinkRank is the plugin we point WordPress readers to for those: it replaces the several plugins that otherwise fight over robots, sitemap and schema output, and it migrates settings from Rank Math, Yoast, AIOSEO and SEOPress. It cannot add a server redirect, so treat it as the second half of the job, not the first.

Step 3: use permanent redirects and a single hop

Google's documentation tells site owners to use permanent redirects for duplicates and says server-side redirects take effect quickest. Use 301 or 308. Keep the hop count at one wherever you can: HTTP on either host should land directly on the final HTTPS origin, not wander through an intermediate hostname first.

Step 4: only then add HSTS

Strict-Transport-Security: max-age=300

Start with a five-minute max-age, confirm nothing breaks, then raise it. Adding includeSubDomains commits every subdomain you own to HTTPS, including ones you forgot about, so check the subdomain list before you add it. Setting a year-long max-age before the redirects work is how sites lock themselves out of their own staging hosts.

Step 5: align the signals that point at the host

After the redirect works, make everything else agree with it: the rel="canonical" tag, the URLs in your sitemap, the URLs in structured data, and the absolute links your templates generate. A redirect fixes what happens when someone arrives on the wrong host. These fix why they keep arriving there.

Verify it: the scan first, curl second

Run AIScan. npx aiscan-cli yourdomain.com, or paste the URL at aiscan.site. It is free and needs no account. The report names check E4 (canonical host and HTTPS) and its evidence line has the template shown earlier, so you can read your own result directly: both "upgrades to https" and "redirects to the canonical host" mean a pass. The neighbouring discoverability checks, D1, D2, D3 and E1, are in the same report, and their pages are collected under discoverability in the docs.

What AIScan cannot see. It probes from one location, so it will not catch a redirect that only misfires for a particular country or a logged-in session. It also does not record the number of hops. Use curl for those two.

If you would rather check by hand, three commands cover the whole check:

# 1. Does plain HTTP upgrade, and in how many hops?
curl -sIL -o /dev/null -w 'final=%{url_effective} hops=%{num_redirects} code=%{http_code}\n' http://example.com/

# 2. Does the other host collapse into the canonical one?
curl -sIL -o /dev/null -w 'final=%{url_effective} hops=%{num_redirects} code=%{http_code}\n' https://www.example.com/

# 3. What does the first redirect actually return?
curl -sI http://www.example.com/ | head -n 5

Run all three against both hosts. You want the same final= URL every time, hops=1 at most, and 301 or 308 on the first response in command 3. A 302 or 307 means the redirect is temporary.

Common mistakes

What this means for the product

Reading 696 evidence strings turned up three issues in E4 itself. We list them because publishing the limits of our own check is part of reporting the data.

1. The evidence omits hops and intermediate status codes. That is why the hop-count analysis above is missing. Recording the chain would also let a transient error be told apart from an absent redirect.
2. Subdomains get a www. probe. A subdomain of a larger site should not be marked down for failing to serve a www. prefix that nobody uses. The 266 figure includes these.
3. The unreachable-host wording says "fine". Two passes rest on a host being unreachable. That is a reasonable reading for a www. that does not exist, and a poor one for a timeout.

None of these changes the main finding. The 291 partials mostly come from real duplicate origins, and the redirect rules above fix them. We will rerun the numbers after the probe is updated.

Reproduce the numbers

The queries below run against the AIScan scans table. They aggregate in SQL and never select rows, which is why no host appears in this post.

with latest as (
  select distinct on (lower(split_part(regexp_replace(url,'^https?://(www\.)?',''),'/',1)))
         url, platform, score, result
  from scans
  where rubric_version = '2026.08.2' and url not ilike '%aiscan.site%'
  order by lower(split_part(regexp_replace(url,'^https?://(www\.)?',''),'/',1)), created_at desc
),
e4 as (
  select platform, score, c->>'status' s, c->>'evidence' ev
  from latest, jsonb_array_elements(result->'checks') c
  where c->>'id' = 'E4'
)
select count(*) n,
       count(*) filter (where s = 'pass') pass,
       count(*) filter (where s = 'partial') partial,
       count(*) filter (where ev ilike '%duplicate origin%') duplicate_origin,
       count(*) filter (where ev ilike '%does NOT upgrade%') no_https_upgrade,
       count(*) filter (where ev ilike '%HSTS set%') hsts
from e4;

Your next action

Scan your domain with AIScan and read the E4 line. If it says "serves its own copy (duplicate origin)", add the redirect from the hostname you do not use. If it says "does NOT upgrade to https", add the site-wide HTTP-to-HTTPS 301 first. Rescan, then add HSTS with a short max-age. The rest of the discoverability family, D1, D2, D3 and E1, is covered in the discoverability checks, and every guide is collected at /guides. For the wider picture of where sites stand today, see the state of AI agent readiness in 2026.