Duplicate URLs: Why One Page Gets Many Addresses

2026-11-02 · 994 words

A page does not have one address. It has as many addresses as clients can construct that resolve to it, and each of those is a separate key in every cache, log, analytics report and access rule that has ever seen it. Most of the confusion people blame on "caching" or "duplicate content" starts here.

The specification is precise about which differences matter and which are cosmetic. Practice is less tidy, because servers, proxies and clients each normalise at different points — or not at all.

Six differences, and what each one does

Trailing slash. example.com/about and example.com/about/ are different URLs as far as any cache or log is concerned. Whether they are the same resource is decided by the server: it can redirect one to the other, serve the same content at both, or 404 one of them. There is no rule in the spec that settles it, only the server's behaviour.

Case in the scheme and host. HTTPS://Example.COM/x and https://example.com/x are equivalent, and hosts are case-insensitive by definition. Case in the path is not: /About and /about may be two different pages, so normalisation must stop at the host boundary.

Default port. https://example.com:443/x and https://example.com/x are the same address. The port rule is scheme-dependent — 443 for https, 80 for http — which is why a normaliser that strips "any port" breaks legitimate non-default ports.

Percent-encoding case. %2F and %2f are the same octet: hex digits are case-insensitive inside a triplet. Your string comparison is not, so two URLs that a browser treats as identical can hash differently in your cache.

Over-encoding. %41 is A, and A never needed encoding. Percent-encoding an unreserved character produces a second spelling of the same URL. RFC 3986 requires normalisation to decode those back, which matters because the same rule says a %2E that decodes to . must not be allowed to change the path structure.

Query order and repetition. ?a=1&b=2 and ?b=2&a=1 are considered equivalent in the abstract, but no spec requires a server to treat them so, and most do not. Reordering parameters is safe for a cache and unsafe for a signature — there is no universally correct answer, only a decision you make and then apply everywhere. Repeating a parameter (?a=1&a=2) is a third thing again: the spec says nothing, so each framework picks a rule, and "first wins" versus "last wins" is a real source of bugs.

Where the duplication actually comes from

Most duplicate URLs are not user error. They are produced by the software:

• A link generated with a slash and a form action posted without one.
• An email template that hardcoded http:// from an era when the site was not yet on TLS.
• A crawler that found %2F in a sitemap and the plain slash in a menu.
• A tracking script appending parameters in a different order on every page load.
• A canonical tag pointing at one spelling while the internal links use another.

That last one is the expensive one. A canonical tag is a hint about which spelling you prefer; it does not stop the other spellings from being crawled, cached or counted. If your internal links disagree with your canonical, you have told a search engine which form you want while simultaneously producing the other.

The three layers, and which one owns what

The server owns the mapping. A redirect from the non-canonical form to the canonical one is the only mechanism that actually removes the extra key from caches and browsers. A 301 or 308 is permanent, a 302 or 307 is temporary; the choice affects whether the old form keeps being requested.

The page owns the preference. rel="canonical" says which address you want indexed when several return the same content. It is advisory. It works best when it is consistent with the redirect, and worst when it contradicts it.

The application owns the signature. If you sign URLs — for uploads, for CDN access, for password-reset links — normalisation must happen before signing, and every component must normalise identically. Signing a reordered query string produces a signature that fails on a server that sorted it, and the error message will say "invalid signature", not "you sorted the parameters".

A practical rule set

1. Pick one form per resource and write it down. Trailing slash or not, www or not, lowercase paths or not — decide once, and treat deviations as bugs.
2. Redirect every other form, at the edge. One 308 rule beats fifty canonical tags. Our own site ends up doing exactly this: /x answers with a redirect to /x/, and /x.html with a redirect to /x. The cost is visible in server logs as a steady trickle of 308s from clients that only ever heard the shorter spelling.
3. Normalise before you compare or sign. Decode unreserved characters, uppercase your triplets, drop the default port — then hash. If you must support reordered queries, sort them at the point of signing and at the point of verification.
4. Do not rely on search engines to merge anything quickly. They do merge, eventually, and slowly. In the meantime the extra spellings accumulate impressions, split click-through, and make your own reporting wrong.

What it looks like when it goes wrong

The symptom is rarely an error page. It is a report where the same article has two rows, one with fifty impressions and one with thirty-five, and a total that matches neither. Or a cache with a 4% hit rate on a page that is requested constantly. Or an access rule that blocks /admin/ while /admin is served happily.

The same discipline shows up in smaller places. When a token arrives truncated at an ampersand, the cause is an unencoded separator — the subject of which characters actually need encoding. When a Base64 value breaks after a URL round trip, the cause is the + that became a space — covered in URL encoding versus Base64, along with the URL-safe alphabet that replaces + and /. Different symptoms, one root cause: the same bytes written two ways.

Normalisation is the discipline of deciding which way, once, and then making every layer agree.