Canonical Tags, Robots.txt and Sitemaps, Explained for Business Owners

Canonical tags, robots.txt and sitemaps come up in almost every SEO audit, and they’re often set up wrong. Each one has a specific job. Use the wrong tool and you can hide pages you want found, or let Google pick a messy URL you never meant to show.

Here’s what each does, in plain English, and when to use it.

Why duplicate URLs happen

To you, your home page is one page. To a search engine, these can be four different URLs showing the same content:

• http://example.com/
• https://example.com/
• https://www.example.com/
• https://www.example.com/index.html

Add tracking parameters (?utm_source=facebook), filter and sort options on product listings, printer-friendly versions and trailing-slash variations, and a small site can generate dozens of duplicates without anyone noticing.

Google usually copes by picking one version as the “canonical” and showing that in results. The problem is that it may not pick the one you want, and signals like links can end up split between versions. Canonicalization is how you tell Google which version you prefer.

The tools, from strongest to weakest

Google describes several ways to indicate a preferred URL. They differ in strength.

301 redirects: strongest

A permanent (301) redirect sends visitors and search engines from one URL to another. Use it when the duplicate shouldn’t exist as a separate page at all, for example:

• http to https
• non-www to www (or the reverse)
• old page URLs after a redesign
• two near-identical pages you’ve merged into one

Redirects are a strong signal that the target is the preferred URL. If you’re rebuilding your site, a complete redirect map is the single most important protection for your existing rankings. Our website redesign SEO checklist covers it in detail.

rel=canonical: strong

A canonical tag is a line in a page’s HTML head that says “the main version of this page lives at this URL”:

<link rel="canonical" href="https://www.example.com/services/drain-cleaning/" />

Use it when duplicates need to stay accessible to people but you want one version treated as the main one. A filtered product list, or the same article reachable through two category paths, are typical cases.

Good practice: every indexable page has a canonical tag pointing to itself (a “self-referencing canonical”), using the exact URL format you’ve chosen: https, the right host and a consistent trailing slash.

XML sitemaps: weak

A sitemap lists the URLs you want search engines to know about. Listing a URL in your sitemap is a hint that it’s your preferred version, but only a weak one. Sitemaps help Google discover pages, especially new ones or pages with few internal links. They don’t override redirects or canonical tags.

Keep your sitemap clean: only canonical, indexable URLs that return a 200 status. No redirected URLs, no noindexed pages, no duplicates. Submit it in Google Search Console.

Robots.txt: crawling, not indexing

Robots.txt is a small text file at the root of your site that tells crawlers which paths they may or may not crawl. It’s useful for keeping bots out of areas that waste crawl effort, such as internal search results or an admin area.

What it does not do is reliably keep a page out of Google’s index. If other pages link to a URL blocked in robots.txt, Google can still index that URL without crawling it, often with no description.

Google also advises against using robots.txt for canonicalization. If you block a duplicate URL, Google can’t crawl it to see its canonical tag or redirect, so it can’t consolidate the signals properly.

Noindex: keeping a page out of results

To keep a page out of search results, use a noindex robots meta tag (or the equivalent HTTP header):

<meta name="robots" content="noindex" />

Google has to be able to crawl the page to see this tag, so don’t also block it in robots.txt. Typical noindex candidates: thank-you pages, internal search results, thin tag archives, staging copies and draft content.

Which one should you use?

Situation	Use
Old URL replaced by a new one	301 redirect
http, non-www or index.html versions	301 redirect to your chosen version
Duplicate that must stay visible to users	rel=canonical to the main version
Page that shouldn’t appear in search	noindex (and keep it crawlable)
Area bots shouldn’t waste time crawling	robots.txt disallow
Helping Google find important pages	XML sitemap + internal links

Common mistakes on small business sites

Mixed host and protocol. The site loads on both www and non-www, or http and https, with no redirects. Fix with site-wide 301s to one version.

Trailing-slash duplicates. /contact and /contact/ both load the same page. Choose one style and redirect the other.

Canonicals pointing to the home page. Some themes and plugins set every page’s canonical to the home page, which tells Google to ignore your service pages. Check a few pages’ source code.

Noindex left on after launch. Developers often noindex a site during building and forget to remove it. If your new site isn’t showing up, check this first. Our guide to why a website isn’t showing on Google lists the other usual suspects.

Blocking CSS and JavaScript. If robots.txt blocks the files that render your pages, Google may not see your content properly.

Sitemaps full of junk. Redirected, deleted or noindexed URLs listed in the sitemap send mixed signals.

How to check your own site

Type your domain into a browser in each http/https and www/non-www combination and confirm they all end up at one address. Use Search Console’s URL Inspection tool on key pages to see which canonical Google selected. Read your robots.txt at yourdomain.com/robots.txt. Our free audit preview checks several of these on a page instantly.

For a full crawl of every URL, redirect and canonical, plus a prioritised list of fixes, see our SEO audit service.