TXT Files

The TXT Files section allows you to view text-based configuration files located in the root directory of the website.

These files typically include configurations such as:

WebPixie automatically detects these files and allows you to view their contents directly.

Robots.txt

The robots.txt file contains rules that determine how search engine bots should crawl your website.

With this file, site owners can control:

Specifies which search engine bot the rules apply to.

Example:

User-agent: *

This means the rules apply to all bots.

Indicates which pages should not be accessed by search engines.

Example:

Disallow: /admin
Disallow: /login

These rules prevent the specified pages from being crawled.

You can use multiple Disallow directives to block different sections of your site from search engines.

WebPixie automatically detects the robots.txt file and displays its contents in the analysis screen.

This allows users to:

quickly review crawling rules
spot misconfigurations
identify critical SEO issues

Reviewing your robots.txt regularly helps ensure your site is indexed correctly and avoids accidental blocking of important pages.