Why should I avoid duplicate pages?
What are duplicate pages?
Duplicate pages are pages with nearly or completely identical text content that belong to the same site but have different URLs.
For example, a home page can have multiple site addresses:
https://example.com/;https://www.example.com/;https://example.com/index;https://example.com/index.html;https://example.com/?utm_source=link&utm_medium=source-example&utm_campaign=partner-offer.
For pages with matching text, the indexing bot creates a group of duplicates. It then selects one page from this group to be displayed in search results. Occasionally, the bot may change its choice to another duplicate.
Why should I avoid duplicate pages?
- The bot indexes multiple pages instead of one. Crawling duplicate pages wastes time as well as the resources of your site and Yandex Search.
- It may take longer to index new pages.
- Duplicate pages may compete with each other in search results.
- The indexing bot can consider a duplicate and exclude from search results a landing page that is important for your site.
Why are there duplicate pages?
Duplicate pages can emerge due to:
- Specific features of your content management system (CMS). For example, page URLs may have or have not a trailing slash (
/). - Web server settings that make site pages accessible over HTTP or HTTPS and with or without the
wwwprefix. - Adding GET parameters to links, such as tracking UTM tags used by advertising systems.
- The same page appearing in different site sections under different URLs.
In Yandex Webmaster, can I check which pages Yandex Search considers duplicates?
To get a list of duplicates, use the Indexing → Searchable pages tool: open the Excluded pages tab, find the Status column, and apply the Duplicate filter. For more details, click the three dots.
To see if a specific page is a duplicate, insert its address in the URL filter.
To find duplicates emerged due to adding GET parameters to links, run diagnostics: Website optimization → Site diagnostics. Information about duplicates will appear in the critical issues section.
In addition, Yandex Webmaster flags these issues on the Summary page.
Learn more:
How to remove duplicate pages from Yandex Search?
- Set up redirects: from alternate site addresses to the primary one, and from duplicates to the desired page.
- In the page code, specify which of the duplicate pages you want to include in search results using the
rel="canonical"attribute. - Use the robots.txt file to prevent the duplicates from being indexed.
- Prevent the duplicates from being indexed by adding the
noindexrule to therobotsmeta tag in the page code.
Learn more:
- How to get rid of duplicate pages
- What is a site move, and how do you do it?
- Primary and alternate site addresses
Ungrouping
The owner of a site that has subdomains and often appears at the top of search results may request to reclassify their domain as a web portal through Yandex Webmaster. To do this, you have to provide a description of the services on the subdomains and their owners.
Learn more: