Sitemap
The Sitemap section shows the sitemap tree WebPixie discovers for the site and cross-checks it against the Link Crawler results.
Through this section, users can review:
- The sitemap tree discovered for the site
- Every URL declared across the site's sitemap files
- Whether crawled internal links are actually listed in the sitemap
Some features and limits mentioned on this page vary by plan. Visit our Pricing page to review the plans.
How Sitemap Discovery Works
WebPixie automatically discovers a site's sitemap tree by reading the Sitemap: directive in robots.txt. From there, it walks any sitemap indexes down to their leaf sitemap files.
Nested (recursive) sitemap indexes — a sitemap index that points to other sitemap indexes instead of directly to leaf files — are not supported and are flagged as such.
Supported Formats
Both XML and plain text sitemap files are supported, including gzip-compressed responses.
Catalogued Data
For each leaf sitemap file, WebPixie catalogs every declared URL, along with its lastmod, changefreq, and priority values when the sitemap provides them.
Processing Limits
The standard sitemap protocol itself limits a single sitemap file to 50,000 URLs and 50 MB uncompressed, a sitemap index to 50,000 sitemaps, and each URL entry to under 2,048 characters.
On top of the protocol limits, WebPixie applies its own fixed processing caps, the same for every plan:
- Up to 10,000 URLs read per sitemap file
- Up to 100 sitemap files processed per site, per discovery cycle
- Up to 50,000 sitemap URLs catalogued per site in total
A sitemap file larger than 50 MB is skipped. If a site's sitemaps exceed WebPixie's file or URL processing caps, this is flagged so you know the catalog may be incomplete.
Check Frequency
Sitemap discovery runs daily, as part of the site's daily snapshot cycle. Individual sitemap files are re-fetched daily when they support cache validation (ETag or Last-Modified); otherwise, a file may be left as-is for up to about a week between fetches.
Cross-Check with the Link Crawler
Once Sitemap Monitoring has completed a full discovery cycle without errors, WebPixie cross-checks its results against the Link Crawler: any crawled internal link that is not listed in the sitemap is flagged with a "Not found in sitemap" tag, visible on the Links page.