If you have spotted “Duplicate without user-selected canonical” in the Pages report in Google Search Console, the fix usually comes down to one of three things you can change from your hosting account: the robots.txt file, a redirect, or a canonical sent by the server. This article explains which one fits which situation, how to apply each on a site hosted with us, and when it is worth handing over to our support team.
What the warning means
Google found the same content at more than one URL, found no canonical tag saying which one is the primary version, and chose one itself. Google’s choice is not always a good one. In the video, Google picked a page on a completely different website as the canonical for one of the site’s own URLs. That is why this status is worth acting on rather than leaving for Google to sort out.
Before you change anything
- Use the domain property in Search Console rather than an http or www property. It combines every version of the domain, including subdomains, so nothing is hidden.
- Open a few of the affected URLs in a browser and see what they actually are: a page, a PDF, a feed, a script.
- Run URL Inspection on each and note the “Google-selected canonical”.
- Decide whether the URL should be in Google at all. If it should not, block it. If it should, decide which copy is the real one.
Option 1: block the path in robots.txt
Use this for URLs with no search value: a plugin’s template folder, a script directory, WordPress feed URLs. When there is no physical robots.txt file, WordPress generates a virtual one. A robots.txt file placed in the site’s root folder (the folder that holds wp-config.php on a standard install) replaces it. Yoast and Rank Math both include a robots.txt editor that writes this file for you, or you can upload it through File Manager or SFTP.
User-agent: *
Disallow: /dap/
Disallow: /feed/
Disallow: /*/feed/Swap the example paths for the folders you found. After saving, load yourdomain.com/robots.txt in a private window to confirm the live file has changed. If a page cache or Cloudflare is serving an older copy, purge it.
Option 2: remove or redirect the duplicate
When the content is genuine and people search for it, keep one URL. Typical cases are a PDF that the CMS serves at two different paths, or a deleted page that now returns a 404 but is still being crawled. Delete the extra copy, or add a 301 redirect to the version Google selected (or the most relevant live page). A redirect plugin works, or you can add a line to .htaccess:
Redirect 301 /assets/uploads/floor-plan.pdf https://yourdomain.com/assets/floor-plan.pdfTake a copy of .htaccess before editing it. A typo in that file can stop the whole site loading with a 500 error, and restoring the copy fixes it straight away. Test the old URL afterwards: it should land on the new one in a single hop.
Option 3: send a canonical in the HTTP header
A canonical tag normally goes in a page’s HTML head. PDFs and other files have no head, so the server has to send the canonical as a Link header instead. On sites using .htaccess, a block like this does it (adjust the filename and URL):
<Files "colour-palette.pdf">
Header set Link "<https://yourdomain.com/files/colour-palette.pdf>; rel=\"canonical\""
</Files>Check it in your browser’s developer tools: open the Network tab, load the file and look under Response Headers for the Link line. This may not clear the warning immediately, but it gives Google a clear instruction, and the URL should move to a different status once it is recrawled.
Why fixed URLs linger in the report
Blocked URLs move to “Blocked by robots.txt” in the Pages report rather than disappearing. Google stops crawling them over time, but it can take several months, even up to a year, before they drop off. That is expected and does not mean the fix has failed.
When to open a ticket
Open a ticket from the client area if the duplicate URLs point to a script or folder you do not recognise. Indexed scripts can expose things that should not be public, so we would rather take a look. The same goes for a canonical header you would like added without editing .htaccess yourself, or redirects that do not seem to take effect. Include:
- A handful of affected URLs, or an export from the Pages report.
- The Google-selected canonical for each, from URL Inspection.
- What you have already changed, and when.
- Whether the domain runs through Cloudflare.
One exception: if the duplicates live on a domain run by a third party, such as tracking links from an email marketing service, neither you nor we can change that robots.txt. Ask the provider to block crawling on their side.