Is your website agent-ready? A practical checklist

A website is agent-ready when AI agents such as browsing assistants and coding tools can discover its content, understand what it offers, and in some cases act on it, without a human clicking through every page first.

That definition sounds simple, but the checklists circulating right now blur it. Most of them list ten or fifteen items, from robots.txt to obscure DNS records, with no indication that some of these are decade-old web standards and others are a single company's proposal from three months ago. Treating both as equally urgent wastes engineering time. This piece sorts the checklist by how mature each standard actually is, shows the syntax for each one, and says plainly which condition makes it worth doing.

What "agent-ready" actually means

AI agent readiness became a live topic once free scanners started grading sites against emerging agent-discovery standards. isitagentready.com is one example, and running your own domain through it gives you a quick check. A low score is not the same as "broken", though: most of the point deductions come from standards that are still drafts, sometimes backed by exactly one vendor.

In practice, agent readiness covers five areas: can an agent find your content (discoverability), can it read that content cleanly (content accessibility), does your site tell bots what they are and are not allowed to do (bot access control), can an agent discover your APIs and how to authenticate with them (protocol discovery), and, if you sell things, can an agent complete a purchase (commerce). The rest of this post walks through each area.

Why standards maturity matters here

Every item below falls into one of three tiers. Tier 1 is a ratified, widely deployed standard you should already have. Tier 2 is a standard with a real specification and multiple implementers, worth adopting now even though the ecosystem is still forming. Tier 3 is a proposal, often from a single company, that may or may not become the standard everyone converges on. Building for a Tier 3 spec is a bet, not a requirement.

Tier 1: standards you should already have

robots.txt

robots.txt is defined in RFC 9309, ratified in 2022 after decades of de facto use. It tells any crawler, search engine or AI agent, which paths it may fetch. A minimal file that allows crawling by default looks like this:

User-agent: *
Allow: /

If you want to opt a specific AI crawler out while leaving the wildcard rule untouched for everyone else, add a named block instead of editing it, for example to block GPTBot only:

User-agent: GPTBot
Disallow: /

Do this when you have an actual reason to treat one crawler differently, such as keeping a specific model out of training data while still allowing search indexing. If you have no such reason, the wildcard rule already covers AI crawlers and there is nothing to add. Our earlier look at blocking AI crawlers covers the tradeoffs of allowing versus disallowing them by name.

sitemap.xml

A sitemap lists your indexable URLs in one machine-readable file. Reference it from robots.txt with a Sitemap: directive so a crawler can find it without guessing the path:

Sitemap: https://example.com/sitemap.xml

This has been standard practice since 2005 and every major crawler, human-built or agentic, expects it. Do it once your site has more than a handful of pages; for a single-page site there is little for a sitemap to index. For a side-by-side of what robots.txt, sitemaps, and llms.txt each do and don't do, see our robots.txt vs llms.txt vs sitemap.xml comparison.

WebPixie's indexability checker validates robots.txt and confirms whether it references your sitemap, which covers this entire tier in one pass if you already monitor your site with it.

Tier 2: worth adopting now

Link response headers for agent discovery

Link response headers, standardized in RFC 8288, let your homepage point agents to useful resources without them having to guess file paths. For example, a header like this advertises an API catalog directly in the HTTP response, before an agent even parses the page body:

Link: </.well-known/api-catalog>; rel="api-catalog"

Add this once you have something worth pointing to, an API catalog, an OpenAPI spec, or documentation. Verify it landed with curl -I https://yoursite.com/ | grep -i link; the header should show up in the response before you rely on any agent picking it up.

Content Signals in robots.txt

Content Signals extend robots.txt with a Content-Signal directive so you can state your preferences for how crawled content gets used, separately from whether it can be crawled at all. A line such as this says a page can be indexed for search and read at query time, but not used to train models:

Content-Signal: ai-train=no, search=yes, ai-input=no

Add this once you have an actual opinion on the three values. If you have not thought about AI training rights for your content, the directive has nothing meaningful to declare yet and skipping it is fine for now. The spec lives at contentsignals.org and is backed by an IETF draft.

Web Bot Auth

Web Bot Auth lets a bot cryptographically sign its requests, using RFC 9421 (HTTP Message Signatures), so a server can verify which agent is actually asking instead of trusting a spoofable User-Agent string that anyone can fake. The work now sits under a chartered IETF working group, though the signing and key-directory specs are still individual drafts rather than adopted working group documents. Google signs a partial, self-described experimental subset of its crawler traffic this way, and Cloudflare has built verification into its bot-management product, which is why it sits in Tier 2 rather than Tier 3. But adoption elsewhere is thin, and it is more work than the other items in this tier: you need a key pair per bot identity, request signing on outbound crawler traffic if you operate a bot, and a directory endpoint serving your public keys if you operate a site being crawled.

That effort is worth it when you need to grant a verified agent something a spoofable identity should not get, such as bypassing a rate limit or a paywall for a specific known crawler. If your only goal is basic bot identification, the robots.txt rules above already cover that case, and Web Bot Auth is safe to defer.

API catalogs and OAuth discovery

If your product exposes a public API, an API catalog at /.well-known/api-catalog, defined in RFC 9727, lists it in a machine-readable format so an agent does not have to find your docs page first:

{
  "linkset": [
    {
      "anchor": "https://api.example.com/v1",
      "service-desc": [{ "href": "https://example.com/openapi.json" }],
      "service-doc": [{ "href": "https://example.com/docs/api" }]
    }
  ]
}

And if that API requires authentication, OAuth Protected Resource Metadata (RFC 9728) tells an agent which authorization server issues tokens for it and which scopes are available, so it can request access without a human reading your auth docs first. Neither is worth building if you have no public API to begin with.

Tier 3: emerging, worth watching

Everything in this section is a real proposal with working implementations, but none of them has settled as the way things will be done. Adopting one now means being an early adopter of that specific vendor's bet, not of an agreed standard.

• MCP Server Card: a JSON manifest at a well-known path describing a Model Context Protocol server, still moving through the MCP specification process
• Agent Skills index: a discovery file listing reusable "skills" a site exposes to agents, proposed by Cloudflare and not yet adopted broadly
• WebMCP: a browser API that lets a page register tools an in-browser agent can call directly, currently an early web platform proposal
• llms.txt: a markdown file at the domain root meant to hand LLMs a curated map of a site, but a 137,000-site study found 97% of published files are never fetched; Google has since repositioned it as part of agentic browsing alongside WebMCP rather than a search signal, see our dedicated look at whether it works
• DNS for AI Discovery (DNS-AID): DNS records that advertise agent endpoints under a domain, accepted as a Linux Foundation project in mid-2026 but with no meaningful adoption yet
• Markdown content negotiation: returning a markdown version of a page when a client sends Accept: text/markdown, a convention popularized by Cloudflare rather than an open standard
• auth.md: a proposed file describing how agents can register for API access, put forward by WorkOS

None of these are wrong bets. They solve real problems agent builders are running into. But shipping all of them today means re-implementing each one again if the ecosystem converges on a different shape, which is common in the first year or two of any new web standard. Revisit this tier when a specific agent integration you actually use asks for one of them by name, not because a scanner flagged it.

Commerce protocols: only if you sell things

A handful of emerging protocols, including x402, MPP, UCP, and ACP, let an agent complete a payment or a purchase on a user's behalf without a human filling out a checkout form. If your site has no e-commerce or paid transactions, these do not apply to you and skipping them will not lower any score that matters. If you do sell online, treat this category the same as Tier 3: worth prototyping, not worth betting your checkout flow on a single protocol yet.

How to prioritize if you are starting from zero

A rough order that matches effort to payoff for most sites:

1. Confirm robots.txt and sitemap.xml exist and are valid, and that the sitemap is referenced from robots.txt. Check with curl -I https://yoursite.com/robots.txt and the same for /sitemap.xml; both should return 200.
2. Decide your stance on AI crawlers by name (allow, block, or leave to the wildcard rule) and encode it explicitly instead of relying on defaults.
3. Add a Link header pointing to your API catalog if you have a public API worth advertising, then confirm it with curl -I against your homepage.
4. Add Content-Signal directives if you have an opinion on AI training versus search indexing versus live retrieval.
5. Revisit Tier 3 items only once you have a concrete agent integration that needs one of them, not because a scanner flagged it.

This is also roughly the order in which the checks contribute real traffic or real risk reduction. A missing robots.txt entry can quietly misroute every crawler that visits. A missing MCP Server Card affects nothing until an agent specifically looks for one, which for most sites is rare today.

Checking where you actually stand

Running a scanner such as isitagentready.com is a fast way to see the full list applied to your domain, but read the result as a map, not a grade. A site scoring low because it lacks a Tier 3 DNS record is in a very different position from one that lacks a valid robots.txt file, even if both show up as failed checks on the same page.

If you already track robots.txt and sitemap health for SEO reasons, most of Tier 1 is covered by the same monitoring. Our indexability checker validates both line by line and flags a missing sitemap reference, which is a common Tier 1 gap in practice. It also validates llms.txt alongside them, which is worth flagging: the checker treats it as a core signal because it is a commonly requested one, not because the underlying standard has matured past Tier 3 here. For the newer Tier 2 items like Link headers and Content Signals, there is no widely adopted monitoring yet, so a manual check every few months is reasonable until tooling catches up.