An XML sitemap is useless if it contains URLs that Google should not index. We regularly observe bloated sitemap files filled with 301 redirects, noindex pages, or duplicated URL parameters, which dilutes the crawl budget instead of concentrating it. Optimizing a site’s navigation through its sitemap starts with cleaning this file to leave only canonical, indexable, and up-to-date URLs.
Reliability of lastmod: the only XML signal that Google still uses
The <priority> and <changefreq> tags are ignored by Google. We recommend removing them from the file to lighten its reading. The only XML parameter that truly influences the crawler’s behavior is <lastmod> with an actual modification date.
A reliable lastmod date means it corresponds to a significant change in the content, structured data, or internal links of the page. Automatically regenerating the sitemap every night by updating all dates does not constitute a relevant signal. Google compares the announced date to the actual content of the page: if nothing has changed, trust in the file decreases overall.
Bing goes further and uses lastmod as a freshness signal in its search systems, including for AI-assisted experiences. Therefore, a sitemap with reliable dates improves visibility on both major engines. To observe how a site concretely structures its URLs in a sitemap file, the Soupir sitemap page illustrates a breakdown by thematic sections well.
XML Sitemap and Crawl Budget: What to Exclude
The first reflex to adopt is to cross-reference the sitemap content with the indexing coverage in Google Search Console. URLs that return errors (404, soft 404, redirects) or have a status of “crawled, not indexed” have no place in the file.
Here are the categories of URLs to systematically exclude:
- Redirected pages (301 or 302): the sitemap should point to the final destination, not the source URL of the redirect
- URLs with sorting, filtering, or pagination parameters that generate duplicated content without the correct canonical tag
- Pages protected by a
noindexor blocked in robots.txt, as their presence in the sitemap sends a contradictory signal to the crawler - Obsolete resources (out-of-stock product sheets, unpublished articles) that no longer return useful content
A sitemap should only contain indexable and up-to-date URLs. Each unnecessary entry consumes crawl budget without return. On a site with several thousand pages, this hygiene makes a measurable difference in the speed of indexing new content.
Sitemap Index and Segmentation by Content Type
Beyond a certain volume of pages, a single file becomes difficult to maintain and analyze. The sitemap protocol specification allows index files that reference multiple child sitemaps. We recommend segmenting by content type rather than alphabetically or by creation date.
An effective breakdown separates editorial pages (articles, guides), product or service sheets, category pages, and media (images, videos). This segmentation allows for quick identification of which subset is causing issues in Search Console. If the indexing rate of product sheets drops, diagnosis can be done in seconds instead of sifting through a monolithic file.
Specialized sitemaps for images and videos deserve particular attention. An image sitemap enriched with <image:loc> and <image:title> tags facilitates indexing in Google Images. For video, the <video:thumbnail_loc>, <video:title>, and <video:description> tags allow for appearing in rich results. These specialized sitemaps are underutilized on the majority of sites we audit.
IndexNow and Sitemap Submission: Combining Both Approaches
Submitting your sitemap via Google Search Console and Bing Webmaster Tools remains the reference method for the initial discovery of URLs. The declaration in the robots.txt file (Sitemap: https://example.com/sitemap.xml) ensures that any compliant crawler finds the file without additional configuration.
For Bing, the IndexNow protocol complements the sitemap during one-off changes. Instead of waiting for the crawler to revisit the sitemap, IndexNow notifies the engine in real-time that a URL has been added, modified, or deleted. This notification should not be sent repeatedly for unchanged pages, or else credibility with the system may be lost.
The combination works as follows: the XML sitemap covers the entire site and serves as a safety net for large-scale discovery, while IndexNow accelerates the consideration of urgent updates. Google has not adopted IndexNow, but its indexing API offers a similar mechanism for eligible content types.
HTML Sitemap and Internal Linking: Two Complementary Tools
An HTML sitemap visible to visitors does not replace the XML file, but it strengthens internal linking. Each link in an HTML sitemap transmits PageRank to deeper pages that the main navigation does not always cover.
For this page to be useful and not just a flat directory, it must reflect the actual hierarchy of the site. Grouping links by category, with a depth level limited to two or three levels, provides both the visitor and the crawler with a readable map of the structure.
A sitemap, whether XML or HTML, is not a static document. The update frequency should follow the site’s publication rhythm. An outdated XML file that references dead URLs degrades the crawler’s trust in the entire domain. Maintaining this consistency between the sitemap and the actual state of the site remains the primary condition for the tool to truly serve navigation and indexing.
