A 200-page site with half of it remaining invisible on Google, we’ve all encountered this scenario. The problem rarely lies in the content: it’s the sitemap that guides (or misguides) the crawlers. Organizing this file correctly changes the game regarding the speed of page discovery and, by extension, the site’s organic SEO.
XML Sitemap: a diagnostic file before being a list of URLs
The sitemap is often treated as a simple inventory of pages. In practice, it’s a diagnostic tool. The Sitemaps report in Google Search Console displays the last read of the file, the number of detected URLs, and processing errors. A “success” mention means that the file has been correctly parsed, not that all URLs are indexed.
This distinction is fundamental. Google treats the sitemap as a discovery signal and not as an automatic indexing request. One can list a perfectly valid URL that Google will decide not to index, due to lack of relevance or because of duplicate content. Therefore, before adding pages to the file, we check that they deserve to be included.
To delve deeper into the structure and best practices surrounding this file, the expertsmarketing.fr page details the accepted formats and common errors to prioritize fixing.
Sorting URLs in the sitemap: what to keep, what to remove
The common reflex is to generate an automatic sitemap via a plugin (Yoast, Rank Math, All in One SEO) and leave it at that. The result: tag pages, empty category archives, URLs with sorting or pagination parameters, all mixed in with strategic pages. Google’s crawler wastes time on unnecessary URLs and slows down the exploration of important pages.
Before submitting the file, we filter based on concrete criteria:
- Each URL must return an HTTP 200 code. Redirects (301, 302) and errors (404, 410) have no place in the sitemap.
- The page must be indexable: no noindex tag in the head, no blockage in robots.txt.
- The content must be unique. A product page offered in six color variants with the same text clutters the file without adding value.
- Orphan pages (without internal links) deserve either a link from the site structure or removal from the sitemap if they are no longer useful.
On an e-commerce site or a portal with several hundred pages, this sorting can sometimes reduce the sitemap by half. The gain is direct: the crawl budget focuses on high SEO potential content.
Sitemap index and thematic segmentation
When the site exceeds a few dozen pages, we gain readability (for us and for Google) by segmenting the sitemap. The sitemaps.org protocol allows for an index file that points to several child sitemaps. We can break it down by content type: one file for blog articles, another for service pages, a third for product sheets.
A segmented sitemap facilitates diagnostics in Search Console. If the report shows a low indexing rate on the “blog” file but correct on the “services” file, we know exactly where to look for the problem. With a monolithic sitemap, this granularity disappears.
For sites that publish high-value videos or images (portfolios, tutorials), specialized sitemaps exist. The video sitemap, for example, enriches search results with thumbnails. Feedback varies on the actual impact of these specialized files depending on the site size, but for a site with rich visual content, the effort remains limited for a potential visibility gain.
Submission and declaration of the sitemap: still valid methods
A point often overlooked: the old Google sitemap ping endpoint was removed in 2023. Guides that still recommend a URL like google.com/ping?sitemap=... are outdated. Continuing to use it does not cause a visible error, but the call has no effect anymore.
The currently recognized methods for declaring a sitemap are three:
- Direct submission via the Google Search Console interface, in the “Sitemaps” section.
- The Search Console API, useful for sites managed by technical teams or automation tools.
- Declaration in the robots.txt file with the
Sitemap:directive followed by the full URL of the file. This method has the advantage of also working for Bing and other engines.
We generally combine submission via Search Console and declaration in robots.txt. The redundancy poses no problem and covers the case where a secondary engine only consults robots.txt.
Update frequency and lastmod tag
The lastmod tag in the XML sitemap indicates the last modification date of a page. Google uses it as a hint to prioritize crawling of recently updated content. The trap: some CMS update this date with every save, even without any real content change. The signal then loses all reliability.
Update lastmod only when the content actually changes remains the rule. An artificial date does not speed up indexing. It may, on the contrary, prompt Googlebot to revisit the page, find nothing new, and ultimately space out its visits.
For the frequency of regeneration of the file itself, a dynamic sitemap (generated on-the-fly or recalculated daily) is suitable for sites that publish often. A showcase site updated once a month does not need a sitemap recalculated every hour.
The sitemap does not replace a solid internal linking structure, quality content, or a coherent site structure. It is a complementary technical lever. When well organized, filtered, and submitted correctly, it provides search engines with exactly what they need to crawl the site without wasting resources.
