Web Bot Auth explained: what RFC 9421 signing proves

Web Bot Auth lets a bot cryptographically sign its HTTP requests, so a server can verify the request actually came from the identity it claims, instead of trusting a User-Agent string that anyone can type into a header. It uses RFC 9421 (HTTP Message Signatures) as its signing mechanism, and it is under active IETF development, not yet a finished standard.

Why a User-Agent string is not proof of anything

Anything a client sends in a request header is a claim, not evidence. User-Agent: GPTBot proves nothing about who is actually asking; any script can send that string. Our earlier look at blocking AI crawlers covers robots.txt rules that tell crawlers what they are allowed to do, but allowed and authenticated are different questions. robots.txt is a posted sign; it relies on the crawler choosing to read and follow it. Web Bot Auth is closer to a signed ID: it lets a server check whether a request was cryptographically signed by the private key of the identity it claims to be, independent of whether that identity chooses to behave.

How the signing actually works

A signing agent adds three things to its request: a Signature-Agent header identifying where its public keys can be found, a Signature-Input header describing what was signed and with which key, and a Signature header carrying the signature itself. The spec requires signing at least the request's @authority or @target-uri component, plus signature metadata: creation time, expiration, key ID, and a tag value of web-bot-auth that tells a verifier this signature is for bot identification specifically, not some other use of RFC 9421 signing.

A verifier that does not already recognize the key ID fetches it from wherever Signature-Agent points, most commonly a well-known directory URL:

Signature-Agent: sig1="https://example.com/.well-known/http-message-signatures-directory";type=directory

That directory is itself a real, fetchable endpoint. Google publishes one for its crawler at agent.bot.goog, and querying it shows the mechanism working end to end, including the directory response signing itself:

$ curl -sD - https://agent.bot.goog/.well-known/http-message-signatures-directory
HTTP/2 200
content-type: application/http-message-signatures-directory+json
signature-input: g=("@authority";req);created=...;keyid="DYiMjA";alg="ed25519";tag="http-message-signatures-directory"
signature: g=:6xnKVPFI9pXivHBottuKTkQdK43JRmOk4J2iIPhCGo8Sc9Vtm3APKsSxBBFAk0NrAwBQvhpv+EfyC2H1b1L9Ag==:

{"keys":[{"alg":"EdDSA","crv":"Ed25519","key_ops":["verify"],"kid":"DYiMjA","kty":"OKP","use":"sig","x":"0cFRtAeGOa46EsjXFVKw1o_pMPFBGq4ReeznYs_RBrs"}, ...]}

The directory itself carries a different tag, http-message-signatures-directory, distinguishing "here is proof this key list is authentic" from the web-bot-auth tag used on the actual crawl requests that reference it. A verifier matches the key ID on an incoming request against a key in this list, confirms the signature, and only then treats the identity claim as backed by something other than a text string.

Current spec status

The work sits under a chartered IETF working group, "Web Bot Authentication" (webbotauth), with milestones for a standards-track signing spec and a companion bot-info spec by April 2026, and an operational best-practices document by August 2026. As of this writing, the actual specifications, the signing protocol and the key-directory format, remain individual drafts rather than documents formally adopted by the working group. A chartered working group with active milestones is a stronger signal than an individual draft with no group behind it, but it is still short of a published RFC.

Who actually signs requests right now

Primary-source adoption is thinner than most coverage of this topic suggests. Google signs a partial, self-described experimental subset of its crawler traffic and explicitly tells sites to keep IP and reverse-DNS verification as a fallback during rollout. Cloudflare has built verification into its bot-management product, but names no confirmed external adopter beyond its own tooling in that announcement. AWS has a dated, named Web Bot Auth preview in Bedrock AgentCore for its own agent browser, and Akamai states that signature verification is already supported in its bot-management product.

Several marketing posts claim considerably more than this, including broad adoption by Anthropic, OpenAI, and Perplexity, and one claim that a "W3C spec" was finalized. Neither claim holds up: this is an IETF effort, not a W3C one, and no primary source from any of those three companies confirming signed traffic exists as of this writing. Treat any adoption claim that does not link to the vendor's own documentation as marketing, not fact.

What the spec still leaves unresolved

Signing is voluntary, and the mechanism has no answer for a crawler that does not participate. Cloudflare's own announcement states that when a signature is missing or fails verification, its system falls back to the same legacy bot-detection it used before, meaning an unsigned request is treated exactly as it would have been without Web Bot Auth existing at all. That is not a hypothetical edge case; it is the default path for the overwhelming majority of crawler traffic today, since almost nothing signs yet.

There is a second, less technical tension worth naming: a small number of large platforms, mainly Cloudflare and Google, are shaping how bot identity verification works for a large share of the web's traffic. That is not automatically a problem. Standards need someone to build the first implementation, but it is worth watching whether smaller crawlers and independent verifiers get a real seat at the table as the working group's milestones come due, or whether the practical standard ends up being whatever the two largest implementers ship.

Should you implement this now

If you operate a crawler or an agent that fetches other people's sites, signing your own requests costs you a key pair and a directory endpoint, and it lets sites that check for it treat your traffic as verified rather than guessed at. That is worth doing once the spec direction is stable enough to commit engineering time to.

If you operate a site being crawled, checking signatures today mostly protects you against nothing, since almost no crawler that would spoof its identity is signing yet, and the ones that do sign, Google and a handful of others, are not the traffic you needed cryptographic proof for in the first place. It is reasonable to wait until verification is built into your existing bot-management tooling rather than hand-rolling signature checks against a protocol still moving through working group review.