You see why a page might drop out of search or AI answers
WebPixie reads the signals that decide whether search engines such as Google and Bing, and AI assistants such as ChatGPT and Claude, can find and index your pages: robots.txt, llms.txt, canonical, hreflang and link coverage.
Indexability signals across WebPixie
You can review the live signals surfaced by TXT Files, Main Page, Link Crawler, and Sitemap Monitoring.
These are indexability-related signals from live feature views, not a single overall indexable or blocked verdict.
One robots.txt line can hide a page from search and AI
WebPixie surfaces indexability-related signals across the relevant feature views. TXT Files validates robots.txt and llms.txt line by line, Main Page surfaces canonical, alternate, and header signals, Link Crawler labels crawled URLs that robots.txt disallows, and Sitemap Monitoring can flag crawled internal links missing from your sitemap.
A single accidental noindex tag or one restrictive robots.txt line can quietly remove a page from search results with no error to warn you. WebPixie surfaces these signals in the relevant views, so you can review a noindex directive, a Disallow rule, or a conflicting canonical or hreflang before it turns into a long debugging session.
Coverage goes past classic SEO. WebPixie also validates llms.txt, so you can review the guidance you publish for AI tools as that traffic grows.
You can review accidental noindex and robots blocks
One wrong directive can drop a page from search
A noindex meta tag left on after a redesign, or a Disallow line added to robots.txt during a deploy, can remove a page from search results without any error. WebPixie surfaces indexability signals across the relevant views, so you can review the directive where that signal is collected instead of treating a single label as the diagnosis. These signals show up in the analysis, not as an alert. If you want an alert when a release adds noindex, you can add "noindex" as a keyword that must not appear on your homepage uptime monitor.
Conflicting canonical and hreflang signals
Canonical and hreflang tell engines which version wins
When a canonical tag points to the wrong URL, or hreflang alternates do not reciprocate, engines may favor a version you did not intend or split signals across near-identical URLs. WebPixie surfaces canonical and hreflang signals from the relevant page and link analyses, so you can review which URL engines are being asked to treat as the original.
You see which AI crawlers your robots.txt lets in
GPTBot, ClaudeBot, Google-Extended, PerplexityBot and Content-Signal
AI assistants such as ChatGPT and Claude reach your pages through their own crawlers. WebPixie reads your robots.txt and shows which known AI crawlers, such as GPTBot, ClaudeBot, Google-Extended and PerplexityBot, have explicit rules, and which Content-Signal directives you set. It also checks that llms.txt is reachable and validates it line by line.
WebPixie confirms your robots.txt, sitemap, and link rel signals
robots.txt sitemap reference, sitemap coverage, rel nofollow, ugc, sponsored
WebPixie checks whether a sitemap is referenced in your robots.txt and reads the rel attributes on your links, including nofollow, ugc, and sponsored. On paid plans, Sitemap Monitoring crawls the declared sitemap tree and flags crawled internal links missing from your sitemap, so sitemap coverage gaps are visible alongside link-level signals.
You can review your indexability signals in about a minute
Free plan, no credit card.
Everything you need to monitor a website. In one workspace.
A quick look at other WebPixie features.
Why teams choose WebPixie for indexability
Setup takes about a minute
No agent to install and no access to your server needed. Once you enter a domain, WebPixie reads your robots.txt, llms.txt and homepage signals from its own servers.
Your whole site in one workspace
Indexability sits next to uptime, SSL, DNS, domain, and link health, with one dashboard across all of them.
Search and AI coverage together
Search indexing and AI-engine access signals appear in the same workspace, so you can review both without switching tools.
Frequently Asked Questions
Common questions about the indexability checker.
The Indexability Checker covers signals that affect whether search engines such as Google and Bing and AI assistants such as ChatGPT and Claude can discover, crawl, and index your important pages. TXT Files validates robots.txt and llms.txt, Main Page surfaces canonical and alternate signals, Link Crawler contributes page-level robots access signals, and Sitemap Monitoring contributes sitemap coverage signals. Each signal can be reviewed in the feature view where it is collected, alongside the Link Crawler and Main Page Analyzer details that provide its context.
Uptime affects SEO because search engines can only crawl and index pages that respond, and Google documents that repeated server errors slow down its crawling. A short outage is unlikely to change rankings on its own. When a site keeps returning server errors or timing out, Google crawls it less often, and URLs that keep failing can eventually drop out of the index. Uptime monitoring checks the pages you choose, such as landing pages, product pages and pages that bring in organic traffic, and emails your team when one times out, returns an unexpected status code or loses the content you expect. Network errors are retried before an alert, and on paid plans a timeout is checked again from the monitor's other selected locations, so a single blip does not wake anyone up. Alerts go by email on every plan, with Slack and webhooks on higher plans, and on paid plans each outage is kept as an incident with its duration. For the search side, the Indexability Checker shows what your robots.txt allows, and the Link Crawler lists internal links that return errors.
Yes. The Indexability Checker validates and analyzes robots.txt and llms.txt line by line, giving each file a valid or invalid result and flagging syntax it cannot parse. This lives in the TXT Files view, where you can see how a directive is interpreted instead of guessing whether a rule is written correctly. It also checks whether a sitemap is referenced in robots.txt, so a missing reference is visible during review. On the Main Page view, WebPixie surfaces the canonical URL and alternates as supporting visibility signals rather than a standalone indexing verdict. Indexability signals contribute to the WebPixie Site Score without requiring you to inspect each technical finding manually. For link-level crawl health, it pairs with the Link Crawler, and you can compare plan limits on the pricing page.
They answer different questions. The Link Crawler is about link health: starting from your homepage it follows discovered public internal links and reports each checked link's status code, response time, redirect status, and whether it loads successfully, while also listing the outbound links each page points to without checking them. The Indexability Checker is about crawl signals: it validates robots.txt and llms.txt, reports robots.txt sitemap-reference presence, uses canonical and alternate signals from the Main Page view, and includes crawler-derived robots-disallowed tags. A link can be healthy yet blocked from search engines, and a page can be visible to crawlers yet link to broken destinations, so the two signals do not replace each other. The crawler deliberately ignores robots.txt while following links, because it must fetch a page to confirm it works; search eligibility is reviewed through indexability signals instead. You can use both and compare plan limits on the pricing page.
The Indexability Checker only reports whether a Sitemap: directive is present in your robots.txt, a presence check. Sitemap Monitoring goes further: it actually fetches the declared sitemap tree, walks supported sitemap indexes down to leaf files, and catalogs declared URLs with lastmod, changefreq, and priority when present. It also cross-checks sitemap coverage against the Link Crawler, flagging crawled internal links that are missing from your sitemap. Sitemap Monitoring is available on Starter and above, while the robots.txt presence check is included on every plan; you can compare plans on the pricing page.
Yes. The Indexability Checker reads your robots.txt line by line and shows which known AI crawlers, such as GPTBot, ClaudeBot, Google-Extended and PerplexityBot, have explicit rules there. It also lists any Content-Signal directives, such as search, ai-train or ai-input, so you can confirm that your file says what your team decided. WebPixie validates your llms.txt file in the same view, and the Main Page view notes whether your homepage returns Markdown when a client asks for it, which some AI tools prefer. These are signals from your own files; WebPixie does not report how often any crawler visits. The Link Crawler adds a tag for pages your robots.txt disallows, and the Main Page Analyzer shows the canonical URL of your homepage.
Ready to review your indexability signals?
Free plan, no credit card. robots.txt, llms.txt and homepage signals in one view.
From the blog









