Why your sitemap and your crawl disagree

A sitemap and a crawl disagree because they measure two different things. A sitemap is a declaration, a list someone or something generated and published at a point in time. A crawl is an observation, what actually gets reached by following links right now. Those two can only match on the day the sitemap was generated. Every page shipped or removed after that point pulls them apart, and most sites ship or remove pages every week.

Two different mismatches, not one

"Sitemap and crawl disagree" hides two distinct problems, and treating them as one is where most orphan-page advice goes generic:

• Crawled but not in the sitemap: a page is reachable by following internal links, but the sitemap generator never picked it up. This usually means a new page shipped after the last sitemap regeneration, or a page type the generator was never configured to include. Search engines that rely on the sitemap for discovery may take longer to find it, or miss it if nothing else links to it strongly.
• In the sitemap but never crawled: a URL is declared in the sitemap, but nothing on the site actually links to it. This is the classic definition of an orphan page: it exists, it is declared as canonical content, and a visitor or a crawler following normal navigation will never land on it. It survives only because the sitemap generator still knows about it, often from a database record that never got an internal link.

These need different fixes. The first is a sitemap-generation lag, usually solved by triggering the generator, or checking why it excluded the page. The second is a navigation problem: the page needs an internal link from somewhere a visitor or crawler would actually reach, not just a sitemap entry.

Why this is a drift problem, not an audit problem

The standard advice treats this as a one-time technique: crawl your site, diff the URL list against your sitemap, fix what does not match. That works the day you run it. It says nothing about the week after, when three new blog posts ship, an old product page gets pulled, and a sitemap index that regenerates on its own schedule quietly falls a few pages further out of sync.

A sitemap file with an ETag or Last-Modified header can be re-checked daily with a cheap conditional request. One without those headers has to be re-fetched in full to know if anything changed, which is why most ongoing coverage tools fall back to about a week between checks for those files. Either way, the gap is not a one-time defect to close, it is a rate: pages accumulate on one side or the other of the mismatch continuously, and the only question is how long they sit there before anyone notices.

What ongoing coverage tracking actually involves

WebPixie’s own Sitemap Monitoring discovers a site’s sitemap tree from the Sitemap: directive in robots.txt, walks index files down to their leaf sitemaps, and catalogs every declared URL. That discovery cycle re-runs daily as part of the site’s snapshot, using a conditional GET on any file that supports cache validators, and falling back to roughly weekly re-fetches for files that do not.

Once the first full discovery cycle completes, the catalog gets cross-checked against the Link Crawler’s results: any internal link the crawler reaches that is missing from the sitemap gets flagged. That catches the first kind of mismatch, crawled but undeclared, automatically and on a schedule instead of as a one-time diff.

A structural limit worth knowing before you rely on any of this

Sitemap indexes cannot nest. A sitemap index file can list individual sitemap files, but it cannot list another sitemap index, and search engines themselves do not support that structure. A generator that produces a nested or recursive index is producing something outside the sitemap protocol, not an edge case a smarter parser can work around. If a coverage tool flags an index as unsupported, the fix is flattening the structure so the top-level index points directly at sitemap files, which is also what keeps the sitemap discoverable to search engines in the first place.

When the gap actually costs you something

The mismatch is not equally urgent everywhere. A rough sense of what matters:

1. A new, high-value page missing from the sitemap for days after launch: this delays discovery for exactly the content most worth finding quickly, a launch, a pricing change, a new product line.
2. An orphan page that used to rank: a page search engines indexed while it still had internal links, now surviving only in the sitemap after the links were removed, is bleeding the value it once had with no path back for a new visitor to find it.
3. A stale sitemap entry for a page that was deleted entirely: lower urgency, mostly a cleanliness issue, though a sitemap full of dead entries can slow how quickly a crawler works through the URLs that do matter.