The Ultimate Guide to Web Bot Auth

Web Bot Auth (often abbreviated WBA) is an emerging IETF standard that lets bots, crawlers, and AI agents prove who they are by cryptographically signing their HTTP requests. The server checks the signature against a public key the bot publishes at a well-known URL. No shared secrets, no IP allowlists, no trusting a User-Agent string that anyone can type.

This guide is the long version of the story: where the standard came from, what is and is not an RFC, who signs and who verifies today, and how to implement it on either side. If you want the two-minute version first, read What is Web Bot Auth? and come back.

A note on names: “Web Bot Auth” is the project name used by the IETF working group and the ecosystem. On the wire it appears as the web-bot-auth tag inside an RFC 9421 HTTP Message Signature. You will also see “HTTP Message Signatures for Bots”, the title of the architecture draft. All three refer to the same mechanism.

The problem it solves

For twenty years, sites identified well-behaved bots two ways: the User-Agent string and the source IP. Googlebot says it is Googlebot, and you can reverse-DNS its IP to confirm. Both methods are now failing at once.

User-Agent strings are plain text. Any scraper can claim to be Googlebot, and many do, precisely because sites allowlist that string. IP verification held up better, but it assumes a bot runs on stable, publishable infrastructure. AI agents do not. They run in serverless functions, shared cloud egress pools, and increasingly on the user’s own machine or a hosted browser, where the IP says nothing about who is driving. Vercel’s engineers put it plainly when they added verification support: signature checks prove authenticity “regardless of network origin.”

The result is an arms race nobody enjoys. Sites that cannot tell a legitimate agent from an impersonator block both, then soften the block with CAPTCHAs, which agents fail, which pushes agent vendors toward stealth, which makes sites block harder. The Cloudflare and Perplexity dispute of August 2025 was this dynamic playing out in public.

Web Bot Auth replaces “guess from the metadata” with “verify a signature.” Only the holder of the private key can produce a valid signature, so identity becomes something a site can check, not something it has to believe.

How the protocol works

Three pieces, all standard HTTP.

1. The bot publishes keys. The operator generates a keypair (Ed25519 in every deployment today) and publishes the public key as a JSON Web Key Set at a fixed path on a domain they control:

https://example.com/.well-known/http-message-signatures-directory

2. The bot signs each request. Every outbound request carries an RFC 9421 signature covering, at minimum, the request’s authority (or its full target URI), with a tag=web-bot-auth parameter so verifiers know which scheme they are looking at. A Signature-Agent header tells the verifier where the keys live. A real signed request, here from OpenAI’s agent, looks like this:

Signature-Agent: "https://chatgpt.com"
Signature-Input: sig1=("@authority" "signature-agent")
  ;created=1735689600;expires=1735693200
  ;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U"
  ;tag="web-bot-auth";nonce="e8N7S2MFd..."
Signature: sig1=:jdq0SqOwHdyHr9...:

The keyid is a thumbprint of the signing key. The nonce defends against replay. created and expires bound the signature’s lifetime.

3. The server verifies. It reads Signature-Agent, fetches the directory (caching it), finds the key whose thumbprint matches keyid, and checks the signature bytes. If the math works, the request provably came from whoever controls that directory. Castle’s writeup of verifying OpenAI Operator requests walks through this step by step.

That is the whole mechanism. The details, which components must be covered, how thumbprints are computed, what the directory must serve, are exactly what our checker tests, and every check links to a reference page explaining the rule it enforces.

The standard: what is an RFC and what is not

Search results around “web bot auth rfc” deserve a precise answer, because the situation confuses people.

Already RFCs (the foundations): RFC 9421 defines HTTP Message Signatures, the generic signing mechanism. RFC 7638 defines the key thumbprints used as keyid. RFC 8941 defines the structured header syntax. RFC 8615 defines well-known URIs. All published, all stable.

Not yet RFCs (Web Bot Auth itself): the profile that applies those RFCs to bot traffic is specified in Internet-Drafts:

• draft-meunier-web-bot-auth-architecture, the core document. Version 05 (March 2026) is authored by Thibault Meunier (Cloudflare) and Sandor Major (Google). This is the draft people mean when they search “draft meunier”.
• draft-meunier-http-message-signatures-directory, the key directory format: the well-known path, the media type, key requirements, and the recommended self-signing of the directory response.
• draft-meunier-webbotauth-registry, the Signature Agent Card (machine-readable metadata about an agent) and registry formats.
• draft-nottingham-webbotauth-use-cases, Mark Nottingham’s catalogue of the problems the group is solving.

Around these sit satellite drafts that show where the work is heading: Gary Illyes (Google) wrote one on crawler best practices and one defining a JSON format for publishing crawler IP ranges, Eric Rescorla drafted anonymous bot authentication for rate-limiting without identification, and there are early documents on responsible data collection and semantic anchors.

The working group: the IETF chartered the WebBotAuth working group in early 2026, chaired by David Schinazi and Rifaat Shekh-Yusef, with Mike Bishop as responsible Area Director. The charter scopes it to cryptographically authenticating automated HTTP clients: search crawlers, archivers, AI training crawlers, and agents. Its milestones target standards-track specifications for the authentication mechanism and a best-current-practice operations document during 2026. To follow along, the datatracker page lists every document and meeting.

The draft versions this guide (and our checker) are verified against are pinned in the Last verified box below.

Timeline: from Operator to working group

• Jan 2025: OpenAI ships Operator. Its browsing requests carry RFC 9421 signatures, the first large-scale deployment of what becomes Web Bot Auth.
• May 2025: Cloudflare publishes “Forget IPs: using cryptography to verify bot and agent traffic”, submits the first drafts, and starts accepting message signatures in its Verified Bots program.
• Jul 2025: the proposal gets an IETF airing at IETF 123, with use cases presented by Vercel among others.
• Aug 2025: Cloudflare launches signed agents with partners including Browserbase and Anchor Browser. Vercel ships verification support (Aug 12). The same month, Cloudflare publicly accuses Perplexity of stealth crawling, the clearest demonstration yet of why verifiable identity matters.
• Sep 2025: Stytch adds Web Bot Auth support for agent verification (Sep 23).
• Oct 2025: AWS Bedrock AgentCore Browser starts signing in preview to reduce CAPTCHAs for its agents (Oct 30). The registry draft lands.
• Nov 2025: Akamai announces support across its bot management products. Mark Nottingham files the request that leads to an IETF working group.
• Jan 2026: DataDome ships agent-trust verification built on Web Bot Auth.
• Early 2026: the IETF charters the WebBotAuth working group.
• Feb 2026: Cloudflare and AWS publish an open registry format for discovering agent keys.
• Mar 2026: architecture draft -05 is published, now co-authored by Cloudflare and Google.
• May 2026: Shopify starts rate-limiting unsigned bots across storefronts (May 7). Google begins signing a subset of its Google-Agent traffic, experimentally.

Eighteen months from one product shipping signatures to a chartered standards group with the largest CDNs, clouds, and commerce platforms on board. By web-standards clocks, that is fast.

Who supports Web Bot Auth today

Two sides to track: agents that sign, and platforms that verify. Vendor links go to official documentation.

Agents and platforms that sign

Who	What signs	Status	In production since	Source
OpenAI	ChatGPT agent (Operator) browsing requests, Signature-Agent: https://chatgpt.com	Production	Jan 2025	OpenAI Help Center
Google	A subset of Google-Agent fetcher traffic, keys at agent.bot.goog	Experimental	May 2026	Google Search Central
AWS	Bedrock AgentCore Browser signs every request when enabled	Preview	Oct 2025	AWS docs
Browserbase	Hosted browser infrastructure signs for the agents it runs	Production	Aug 2025	Browserbase blog
Anchor Browser	Verified browser agents on its platform	Production	Aug 2025	Cloudflare signed agents post

Dates reflect our last verification pass. See Last verified.

Platforms and sites that verify

Who	What it does with signatures	Status	In production since	Source
Cloudflare	Verified Bots accepts Web Bot Auth; signed agents are a distinct, end-user-directed category sites can allow or block	Production	May 2025 (verified bots), Aug 2025 (signed agents)	Cloudflare docs
Akamai	Edge verification feeding bot classification in App & API Protector	Production	Nov 2025	Akamai blog
Vercel	Bot directory verified by IP, reverse DNS, and Web Bot Auth signatures	Production	Aug 2025	Vercel changelog
HUMAN	Verifies signed agent traffic against published directories	Production	Oct 2025	AWS launch post naming verifying vendors
DataDome	Agent-trust verification through the customer journey	Production	Jan 2026	DataDome changelog
Stytch	Agent verification without pre-registration	Production	Sep 2025	Stytch blog
Shopify	Strictest rate limits for unsigned bots storewide; merchants can authorize their own crawlers with signing keys; checkout stays gated regardless	Production	May 2026	Shopify changelog

Dates reflect our last verification pass. See Last verified.

Notably absent from the signing table: Anthropic. ClaudeBot, Claude-User, and Claude-SearchBot identified themselves by User-Agent and robots.txt compliance only, as of our verification date. More on what that means for Claude-based agents below.

Registries and agent directories

Verification answers “does this request come from the holder of these keys?” The harder question is the next one: which key directories should a site trust in the first place? Anyone can publish a directory; publishing one does not make a bot welcome.

Three discovery mechanisms exist today:

1. The Signature-Agent header points at the directory of the agent that signed. This is self-declared identity: cryptographically solid, but it only proves the agent controls some domain, not that the agent is reputable.
2. The Signature Agent Card (registry draft) is machine-readable metadata about the agent: operator name and contacts, declared purpose, robots.txt product token, rate expectations, where the keys and IP ranges live. The draft establishes an IANA registry for its fields. Our agent card article covers it field by field.
3. Registries, curated lists mapping known agents to their cards and directories. Cloudflare and AWS published an open registry format in February 2026 so that any party can host one and verifiers can consume them interchangeably. Cloudflare’s own signed-agents list and Vercel’s bot directory are effectively registries with admission criteria.

The open question, who runs the registries that matter and who gets in, is a governance debate more than a technical one. It is covered honestly in Limitations.

Implementing Web Bot Auth

If you operate a bot or agent

The path from zero to signed requests:

1. Generate an Ed25519 keypair. Every deployed verifier speaks Ed25519. Keep the private key in your secret store; it never leaves your infrastructure.
2. Publish the public key as a JWKS at /.well-known/http-message-signatures-directory on a domain that clearly belongs to your organization, with the media type application/http-message-signatures-directory+json. Serving it from a small handler (rather than a static file) lets you also self-sign the response, which the directory draft recommends.
3. Sign your outbound requests. The reference libraries do the wire format for you: web-bot-auth on npm (TypeScript) or the web-bot-auth Rust crate. Cover @authority, send a Signature-Agent header pointing at your directory, include a nonce, and keep expiry short (minutes, well under the 24-hour recommendation).
4. Publish a Signature Agent Card so site operators can find out who you are and how you behave. Optional, but it is the difference between “a valid signature” and “a known, accountable agent”.
5. Test it. Run your domain through the checker for the directory side, and paste a captured signed request into the verifier for the signature side. Both grade against the current drafts and link every finding to an explanation.

If you run a website

You rarely need to build verification yourself:

• Behind a CDN or bot manager: Cloudflare, Akamai, Vercel, HUMAN, and DataDome verify signatures for you (links in the table above) and expose the result as policy input: allow verified bots, challenge unknown automation, block impersonators. If you are on one of these platforms, Web Bot Auth is a setting, not a project.
• Verifying at your origin: the cloudflare/web-bot-auth repo ships a Caddy plugin (Go) and Cloudflare Workers examples; Gary Illyes maintains an Apache module (C). The verification logic is: parse Signature-Input, fetch and cache the directory named by Signature-Agent, match the keyid thumbprint, verify the bytes, then apply your policy.
• Policy for unsigned traffic stays yours. Today most sites treat signatures as a positive signal (verified bots get through cleanly) rather than requiring them. Shopify is the first large platform to attach real cost to not signing; expect more of that as agent traffic grows.

Signing from your own agent (a Claude example)

A frequent question with no official answer yet: what about Claude? Anthropic’s own crawlers do not sign with Web Bot Auth as of our verification date. But if you build an agent yourself, on the Claude Agent SDK, with MCP tools, or any other stack, the signing side is yours, and it is small.

The pattern: wrap your agent’s outbound HTTP with a signer. Generate a keypair, publish the directory on your domain, then sign each request before it leaves:

import { signatureHeaders, generateNonce, SIGNATURE_AGENT_HEADER } from 'web-bot-auth';
import { Ed25519Signer } from 'web-bot-auth/crypto';

const signer = await Ed25519Signer.fromJWK(privateJwk);
const headers = { [SIGNATURE_AGENT_HEADER]: '"https://your-agent.example"' };
const now = new Date();
const sig = await signatureHeaders(
  { method: 'GET', url, headers },
  signer,
  { created: now, expires: new Date(now.getTime() + 300_000), nonce: generateNonce() },
);
// attach { ...headers, ...sig } to the outbound request

This site is the case study: the checker’s own outbound fetches are signed exactly this way, with the directory at webbotauth.net/.well-known/http-message-signatures-directory and the agent card linked from it. When our bot fetches your directory to grade it, you can verify us right back on /verify. An agent that signs its requests gets recognized by every verifier in the table above, which increasingly means fewer CAPTCHAs and fewer silent blocks.

The open-source ecosystem

The reference implementation lives at cloudflare/web-bot-auth (Apache-2.0), a monorepo holding:

• TypeScript packages: web-bot-auth (the bot profile), http-message-sig (generic RFC 9421), and jsonwebkey-thumbprint (RFC 7638).
• Rust crates: web-bot-auth and http-signature-directory, a directory validation tool.
• Working examples: a Chrome extension that signs from the browser, Cloudflare Workers for signing and verifying, a Caddy plugin, and a directory-hosting example.
• The official JSON test vectors that implementations (including this checker) test against.

Beyond the monorepo, the architecture draft’s implementations appendix lists a Puppeteer signing example by Stytch, a PHP Guzzle middleware, and the Apache verification module mentioned above. The draft sources themselves are developed openly at thibmeu/http-message-signatures-directory.

Live directories you can inspect right now: chatgpt.com and this site’s own.

Limitations and open questions

A neutral guide owes you the unsolved parts.

Identity is not intent. A valid signature proves which operator sent a request. It says nothing about whether that operator respects robots.txt, hammers your origin, or trains on your content. Accountability still depends on knowing who the operator is (the card, the registries) and on their behavior matching their declarations.

Registry governance is unsettled. When Cloudflare launched signed agents, critics argued that requiring registration with one company to get reliable web access looks like a tollbooth, however well-intentioned. The open registry format is a real answer (anyone can run one), but admission criteria, dispute handling, and revocation across registries are unwritten. This is the part of the ecosystem most likely to generate friction.

Replay protection is operator work. Signatures carry a nonce and a validity window, but nothing in the protocol stops a captured signature from being replayed within its window unless the verifier tracks nonces. The drafts leave that, deliberately, to implementations. Our replay advisory explains the trade-off.

Key management is the usual hard part. Rotation, compromise response, and the question of how long verifiers may cache a directory are all live topics in the working group. The directory draft’s validity fields and the recommended response self-signature are partial answers.

Privacy cuts both ways. Per-operator keys make agent traffic linkable by design; that is the feature. For cases where a bot should prove “I am rate-limited and accountable” without revealing which bot it is, Eric Rescorla’s anonymous authentication draft explores blind-signature schemes. Early days.

And robots.txt still applies. Web Bot Auth does not change what a crawler may fetch. It changes whether the “who” in your access rules means anything.

Resources

Type	Resource	What it is
Spec	Architecture draft	The core document: how agents sign and origins verify
Spec	Directory draft	The key directory: well-known path, media type, response self-signature
Spec	Registry draft	The Signature Agent Card and registry formats
Spec	Use cases draft	The problem catalogue the working group works from
Spec	WebBotAuth WG	Charter, documents, milestones, mailing list
Spec	RFC 9421	HTTP Message Signatures, the published foundation
Vendor	Cloudflare	Verification reference, verified bots, signed agents
Vendor	Akamai	Bot management support announcement
Vendor	Vercel	Bot management documentation
Vendor	AWS	AgentCore Browser signing guide
Vendor	Google	Google-Agent signing documentation
Vendor	Shopify	Rate limits and crawler access keys
Code	cloudflare/web-bot-auth	Reference monorepo: TypeScript + Rust libraries, examples, test vectors
Code	web-bot-auth on npm	The TypeScript signing and verification package
Code	Apache module	Origin-side verification for Apache
Code	Draft sources	Where the drafts are developed in the open
This site	Check a domain	Grade a published directory A to F
This site	Verify a signature	Check captured request headers
This site	The short primer	The two-minute version of this guide
This site	How grading works	Verdicts, scores, and capping rules
This site	The agent card	The metadata document, field by field