The Ultimate Guide to Web Bot Auth

Web Bot Auth (often abbreviated WBA) is an emerging IETF standard that lets bots, crawlers, and AI agents prove who they are by cryptographically signing their HTTP requests. The server checks the signature against a public key the bot publishes at a well-known URL. No shared secrets, no IP allowlists, no trusting a User-Agent string that anyone can type.

This guide is the long version of the story: where the standard came from, what is and is not an RFC, who signs and who verifies today, and how to implement it on either side. If you want the two-minute version first, read What is Web Bot Auth? and come back.

A note on names: “Web Bot Auth” is the project name used by the IETF working group and the ecosystem. On the wire it appears as the web-bot-auth tag inside an RFC 9421 HTTP Message Signature. You will also see “HTTP Message Signatures for automated traffic”, the title of the protocol draft, and its short form “HTTP Message Signatures for Bots”. All of these refer to the same mechanism.

The problem it solves

For twenty years, sites identified well-behaved bots two ways: the User-Agent string and the source IP. Googlebot says it is Googlebot, and you can reverse-DNS its IP to confirm. Both methods are now failing at once.

User-Agent strings are plain text. Any scraper can claim to be Googlebot, and many do, precisely because sites allowlist that string. IP verification held up better, but it assumes a bot runs on stable, publishable infrastructure. AI agents do not. They run in serverless functions, shared cloud egress pools, and increasingly on the user’s own machine or a hosted browser, where the IP says nothing about who is driving. Vercel’s engineers put it plainly when they added verification support: signature checks prove authenticity “regardless of network origin.”

The result is an arms race nobody enjoys. Sites that cannot tell a legitimate agent from an impersonator block both, then soften the block with CAPTCHAs, which agents fail, which pushes agent vendors toward stealth, which makes sites block harder. The Cloudflare and Perplexity dispute of August 2025 was this dynamic playing out in public.

Web Bot Auth replaces “guess from the metadata” with “verify a signature.” Only the holder of the private key can produce a valid signature, so identity becomes something a site can check, not something it has to believe.

How the protocol works

Three pieces, all standard HTTP.

1. The bot publishes keys. The operator generates a keypair (Ed25519 in every deployment today) and publishes the public key as a JSON Web Key Set at a fixed path on a domain they control:

https://example.com/.well-known/http-message-signatures-directory

2. The bot signs each request. Every outbound request carries an RFC 9421 signature and a Signature-Agent header that tells the verifier where the keys live. The signature covers, at minimum, the request’s authority (or its full target URI) and the Signature-Agent header, with a tag=web-bot-auth parameter so verifiers know which scheme they are looking at. A signed request looks like this:

Signature-Agent: sig1="https://signer.example.com"
Signature-Input: sig1=("@authority" "signature-agent";key="sig1")
  ;created=1735689600;expires=1735693200
  ;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U"
  ;tag="web-bot-auth"
Signature: sig1=:<base64 Ed25519 signature>:

Signature-Agent is a structured-field dictionary. Its member, keyed to the signature label (sig1), holds an origin; the keys are at that origin’s well-known path. The header is required on every signed request, and the signature must cover the member. Deployments still send the older bare-string form, Signature-Agent: "https://signer.example.com", covered as plain "signature-agent". The draft requires signers to send the dictionary form and allows verifiers to accept the string form. Signers face a conflict today: Cloudflare’s verifier and Fingerprint’s document that they verify only the string form and reject the dictionary form.

The keyid is a thumbprint of the signing key. created and expires bound the signature’s lifetime. A nonce parameter is optional.

3. The server verifies. It reads Signature-Agent, fetches the directory at that origin’s well-known path (caching it, and without following redirects), finds the key whose thumbprint matches keyid, and checks the signature bytes. If the math works, the request provably came from whoever controls that directory, and the verifier can attribute it to that URL. Castle’s writeup of verifying OpenAI Operator requests walks through this step by step.

That is the whole mechanism. The details, which components must be covered, how thumbprints are computed, what the directory must serve, are exactly what our checker tests, and every check links to a reference page explaining the rule it enforces.

The standard: what is an RFC and what is not

Search results around “web bot auth rfc” deserve a precise answer, because the situation confuses people.

Already RFCs (the foundations): RFC 9421 defines HTTP Message Signatures, the generic signing mechanism. RFC 7638 defines the key thumbprints used as keyid. RFC 8941 defines the structured header syntax. RFC 8615 defines well-known URIs. All published, all stable.

Not yet RFCs (Web Bot Auth itself): the profile that applies those RFCs to bot traffic is specified in Internet-Drafts:

• draft-ietf-webbotauth-httpsig-protocol, the core document: how agents sign, the Signature-Agent header, the key directory at the well-known path, and the recommended self-signature of the directory response. The working group adopted it as version 00, dated 1 September 2026, authored by Thibault Meunier (Cloudflare) and Sandor Major (Google).
• draft-meunier-webbotauth-registry, the Signature Agent Card (machine-readable metadata about an agent) and registry formats. Version 03 (June 2026) makes the card an OAuth Client ID Metadata Document with the bot-specific fields in one web_bot_auth object.
• draft-nottingham-webbotauth-use-cases, Mark Nottingham’s catalogue of the problems the group is solving.

Older links and search results (“draft meunier”) use several earlier names. The protocol draft was draft-meunier-web-bot-auth-architecture up to version 05 (March 2026), then draft-meunier-webbotauth-httpsig-protocol (versions 00 to 02), which the working group adopted unchanged as draft-ietf-webbotauth-httpsig-protocol-00. It has also absorbed the separate key directory draft, draft-meunier-http-message-signatures-directory (later renamed draft-meunier-webbotauth-httpsig-directory): the directory rules are now section 5.5 and Appendix B of the protocol draft.

Around these sit satellite drafts that show where the work is heading: Gary Illyes (Google) wrote one on crawler best practices and one defining a JSON format for publishing crawler IP ranges, Eric Rescorla drafted anonymous bot authentication for rate-limiting without identification, and there are early documents on responsible data collection and semantic anchors.

The working group: the IETF chartered the WebBotAuth working group in early 2026, chaired by David Schinazi and Rifaat Shekh-Yusef, with Mike Bishop as responsible Area Director. The charter scopes it to cryptographically authenticating automated HTTP clients: search crawlers, archivers, AI training crawlers, and agents. Its milestones target standards-track specifications for the authentication mechanism and a best-current-practice operations document during 2026. To follow along, the datatracker page lists every document and meeting.

The draft versions this guide (and our checker) are verified against are pinned in the Last verified box below.

Timeline: from Operator to working group

• Jan 2025: OpenAI ships Operator. Its browsing requests carry RFC 9421 signatures, the first large-scale deployment of what becomes Web Bot Auth.
• May 2025: Cloudflare publishes “Forget IPs: using cryptography to verify bot and agent traffic”, submits the first drafts, and starts accepting message signatures in its Verified Bots program.
• Jul 2025: the proposal gets an IETF airing at IETF 123, with use cases presented by Vercel among others.
• Aug 2025: Cloudflare launches signed agents with partners including Browserbase and Anchor Browser. Vercel ships verification support (Aug 12). The same month, Cloudflare publicly accuses Perplexity of stealth crawling, the clearest demonstration yet of why verifiable identity matters.
• Sep 2025: Stytch adds Web Bot Auth support for agent verification (Sep 23).
• Oct 2025: AWS Bedrock AgentCore Browser starts signing in preview to reduce CAPTCHAs for its agents (Oct 30). The registry draft lands.
• Nov 2025: Akamai announces support across its bot management products. Mark Nottingham files the request that leads to an IETF working group.
• Jan 2026: DataDome ships agent-trust verification built on Web Bot Auth.
• Early 2026: the IETF charters the WebBotAuth working group.
• Feb 2026: Cloudflare and AWS publish an open registry format for discovering agent keys.
• Mar 2026: architecture draft -05 is published, now co-authored by Cloudflare and Google.
• May 2026: Shopify starts rate-limiting unsigned bots across storefronts (May 7). Google begins signing a subset of its Google-Agent traffic, experimentally.
• Jun 2026: registry draft -03 recasts the Signature Agent Card as an OAuth Client ID Metadata Document (Jun 26).
• Sep 2026: the working group adopts the protocol draft as draft-ietf-webbotauth-httpsig-protocol-00 (Sep 1). Compared with architecture-05, it makes the Signature-Agent header mandatory on every signed request (it was RECOMMENDED), requires signers to use the dictionary form, and contains the key directory rules.

Eighteen months from one product shipping signatures to a chartered standards group with the largest CDNs, clouds, and commerce platforms on board. By web-standards clocks, that is fast.

Who supports Web Bot Auth today

Two sides to track: agents that sign, and platforms that verify. Vendor links go to official documentation.

Agents and platforms that sign

Who	What signs	Status	In production since	Source
OpenAI	ChatGPT agent (Operator) browsing requests, Signature-Agent: https://chatgpt.com	Production	Jan 2025	OpenAI Help Center
Google	A subset of Google-Agent fetcher traffic, keys at agent.bot.goog	Experimental	May 2026	Google Search Central
AWS	Bedrock AgentCore Browser signs every request when enabled	Preview	Oct 2025	AWS docs
Browserbase	Hosted browser infrastructure signs for the agents it runs	Production	Aug 2025	Browserbase blog
Anchor Browser	Verified browser agents on its platform	Production	Aug 2025	Cloudflare signed agents post

Dates reflect our last verification of the vendor tables. See Last verified.

Platforms and sites that verify

Who	What it does with signatures	Status	In production since	Source
Cloudflare	Verified Bots accepts Web Bot Auth; signed agents are a distinct, end-user-directed category sites can allow or block	Production	May 2025 (verified bots), Aug 2025 (signed agents)	Cloudflare docs
Akamai	Edge verification feeding bot classification in App & API Protector	Production	Nov 2025	Akamai blog
Vercel	Bot directory verified by IP, reverse DNS, and Web Bot Auth signatures	Production	Aug 2025	Vercel changelog
HUMAN	Verifies signed agent traffic against published directories	Production	Oct 2025	AWS launch post naming verifying vendors
DataDome	Agent-trust verification through the customer journey	Production	Jan 2026	DataDome changelog
Stytch	Agent verification without pre-registration	Production	Sep 2025	Stytch blog
Shopify	Strictest rate limits for unsigned bots storewide; merchants can authorize their own crawlers with signing keys; checkout stays gated regardless	Production	May 2026	Shopify changelog

Dates reflect our last verification of the vendor tables. See Last verified.

Notably absent from the signing table: Anthropic. ClaudeBot, Claude-User, and Claude-SearchBot identified themselves by User-Agent and robots.txt compliance only, as of our last verification of the vendor tables. More on what that means for Claude-based agents below.

Registries and agent directories

Verification answers “does this request come from the holder of these keys?” The harder question is the next one: which key directories should a site trust in the first place? Anyone can publish a directory; publishing one does not make a bot welcome.

Three discovery mechanisms exist today:

1. The Signature-Agent header, sent with every signed request, names where the signer’s keys live: by default an origin whose well-known path serves the directory, or, with a type parameter, a JWK Set URL (type=jwks_uri) or a Signature Agent Card (type=cimd). This is self-declared identity: cryptographically solid, but it only proves the agent controls that URL, not that the agent is reputable.
2. The Signature Agent Card (registry draft) is machine-readable metadata about the agent: operator name and contacts, declared purpose, robots.txt product token, rate expectations, where the keys and IP ranges live. Since version 03 it is an OAuth Client ID Metadata Document, served at its own client_id URL. The draft establishes an IANA registry for its fields. Our agent card article covers it field by field.
3. Registries, curated lists mapping known agents to their cards and directories. Cloudflare and AWS published an open registry format in February 2026 so that any party can host one and verifiers can consume them interchangeably. Cloudflare’s own signed-agents list and Vercel’s bot directory are effectively registries with admission criteria.

The open question, who runs the registries that matter and who gets in, is a governance debate more than a technical one. It is covered honestly in Limitations.

Implementing Web Bot Auth

If you operate a bot or agent

The path from zero to signed requests:

1. Generate an Ed25519 keypair. Every deployed verifier speaks Ed25519. Keep the private key in your secret store; it never leaves your infrastructure.

2. Publish the public key as a JWKS at /.well-known/http-message-signatures-directory on a domain that clearly belongs to your organization. The protocol draft sets these rules for it:

   • Serve it over HTTPS with the media type application/http-message-signatures-directory+json.
   • Answer directly with status 200. Verifiers do not follow redirects, so a redirect from the apex to www loses them.
   • Leave kid out, or set it to the key’s RFC 7638 thumbprint.
   • Never publish a test key, such as the RFC 9421 example keys.

   Serving it from a small handler (rather than a static file) lets you also self-sign the response over "@authority";req and content-digest, which the draft recommends.

3. Sign your outbound requests. Send a Signature-Agent header with your origin as the value, and cover it together with @authority. Pick the form by the verifiers you need to reach:

   • Dictionary form, as the draft requires: Signature-Agent: sig1="https://<your-origin>", covered as ("@authority" "signature-agent";key="sig1"). Verifiers that accept only the string form, Cloudflare and Fingerprint among them, reject it.
   • Legacy string form: Signature-Agent: "https://<your-origin>", covered as ("@authority" "signature-agent"). Those verifiers accept it; the draft allows verifiers to accept it but asks signers for the dictionary form, and the checker warns on it.

   Keep expiry short: minutes, well under the 24-hour recommendation. A nonce is optional, because the draft adds no nonce requirement of its own. The reference libraries are web-bot-auth on npm (TypeScript) and the web-bot-auth Rust crate. The example below builds the signature with Web Crypto alone.

4. Publish a Signature Agent Card so site operators can find out who you are and how you behave. In the registry-03 shape it is an OAuth client metadata document whose client_id is its own URL, with the bot-specific fields in a web_bot_auth object. Optional, but it is the difference between “a valid signature” and “a known, accountable agent”.

5. Test it. Run your domain through the checker for the directory side, and paste a captured signed request into the verifier for the signature side. Both grade against the current drafts and link every finding to an explanation.

If you run a website

You rarely need to build verification yourself:

• Behind a CDN or bot manager: Cloudflare, Akamai, Vercel, HUMAN, and DataDome verify signatures for you (links in the table above) and expose the result as policy input: allow verified bots, challenge unknown automation, block impersonators. If you are on one of these platforms, Web Bot Auth is a setting, not a project.
• Verifying at your origin: the cloudflare/web-bot-auth repo ships a Caddy plugin (Go) and Cloudflare Workers examples; Gary Illyes maintains an Apache module (C). The verification logic, in order:
  1. Parse Signature-Input.
  2. Resolve the Signature-Agent member the signature covers. For the default type, fetch the directory at that origin’s well-known path, accept only a direct 200, and cache it.
  3. Match the keyid thumbprint.
  4. Verify the bytes.
  5. Apply your policy.
• Policy for unsigned traffic stays yours. Today most sites treat signatures as a positive signal (verified bots get through cleanly) rather than requiring them. Shopify is the first large platform to attach real cost to not signing; expect more of that as agent traffic grows.

Signing from your own agent (a Claude example)

A frequent question with no official answer yet: what about Claude? Anthropic’s own crawlers do not sign with Web Bot Auth as of our verification date. But if you build an agent yourself, on the Claude Agent SDK, with MCP tools, or any other stack, the signing side is yours, and it is small.

The pattern: wrap your agent’s outbound HTTP with a signer. Generate a keypair, publish the directory on your domain, then sign each request before it leaves. The example below builds the RFC 9421 signature base by hand and signs it with Web Crypto, so it depends on no library. It was tested on Node.js 24; any runtime whose Web Crypto supports Ed25519 can run it. privateJwk is your Ed25519 private key as a JWK.

const ORIGIN = 'https://your-agent.example'; // serves /.well-known/http-message-signatures-directory
const LABEL = 'sig1';

const utf8 = (s) => new TextEncoder().encode(s);
const b64 = (bytes) => btoa(String.fromCharCode(...bytes));
const b64url = (bytes) => b64(bytes).replace(/\+/g, '-').replace(/\//g, '_').replace(/=+$/, '');

// keyid: the RFC 7638 thumbprint of the public key (members in this order, no spaces)
const keyid = b64url(new Uint8Array(await crypto.subtle.digest('SHA-256',
  utf8(`{"crv":"Ed25519","kty":"OKP","x":"${privateJwk.x}"}`))));
const key = await crypto.subtle.importKey('jwk',
  { kty: 'OKP', crv: 'Ed25519', x: privateJwk.x, d: privateJwk.d },
  { name: 'Ed25519' }, false, ['sign']);

async function signedHeaders(url) {
  const created = Math.floor(Date.now() / 1000);
  const params = `("@authority" "signature-agent";key="${LABEL}")`
    + `;created=${created};expires=${created + 300}`
    + `;keyid="${keyid}";alg="ed25519";tag="web-bot-auth"`;
  const base = [
    `"@authority": ${new URL(url).host}`,
    `"signature-agent";key="${LABEL}": "${ORIGIN}"`,
    `"@signature-params": ${params}`,
  ].join('\n');
  const sig = new Uint8Array(await crypto.subtle.sign('Ed25519', key, utf8(base)));
  return {
    'Signature-Agent': `${LABEL}="${ORIGIN}"`,
    'Signature-Input': `${LABEL}=${params}`,
    'Signature': `${LABEL}=:${b64(sig)}:`,
  };
}
// attach await signedHeaders(url) to the outbound request

The second line of the base carries the Signature-Agent member value exactly as sent, quotes included. For the legacy string form, which Cloudflare and Fingerprint require today, change three things:

1. Send the header value as "https://your-agent.example", without the sig1= key.
2. Cover ("@authority" "signature-agent"), without ;key=.
3. Make the second base line "signature-agent": "https://your-agent.example".

If you prefer the web-bot-auth npm package: its 0.1.x signatureHeaders helper covers the whole signature-agent header by default, which is the legacy form, and version 0.2.0 changed the package’s API. Check what the version you install puts into Signature-Input before relying on its defaults.

This site is the case study: the checker’s own outbound fetches are signed this way in the legacy string form, Signature-Agent: "https://webbotauth.net" covered as ("@authority" "signature-agent"), because Cloudflare and Fingerprint verify only that form. The directory is at webbotauth.net/.well-known/http-message-signatures-directory, and the agent card linked from it. When our bot fetches your directory to grade it, you can verify us right back on /verify. An agent that signs its requests gets recognized by every verifier in the table above, which increasingly means fewer CAPTCHAs and fewer silent blocks.

The open-source ecosystem

The reference implementation lives at cloudflare/web-bot-auth (Apache-2.0), a monorepo holding:

• TypeScript packages: web-bot-auth (the bot profile), http-message-sig (generic RFC 9421), and jsonwebkey-thumbprint (RFC 7638).
• Rust crates: web-bot-auth and http-signature-directory, a directory validation tool.
• Working examples: a Chrome extension that signs from the browser, Cloudflare Workers for signing and verifying, a Caddy plugin, and a directory-hosting example.
• The official JSON test vectors that implementations (including this checker) test against.

Beyond the monorepo, the protocol draft’s implementations appendix (Appendix F) lists a Puppeteer signing example by Stytch, a PHP Guzzle middleware, and the Apache verification module mentioned above. The draft sources themselves are developed openly at thibmeu/http-message-signatures-directory.

Live directories you can inspect right now: chatgpt.com and this site’s own.

Limitations and open questions

A neutral guide owes you the unsolved parts.

Identity is not intent. A valid signature proves which operator sent a request. It says nothing about whether that operator respects robots.txt, hammers your origin, or trains on your content. Accountability still depends on knowing who the operator is (the card, the registries) and on their behavior matching their declarations.

Registry governance is unsettled. When Cloudflare launched signed agents, critics argued that requiring registration with one company to get reliable web access looks like a tollbooth, however well-intentioned. The open registry format is a real answer (anyone can run one), but admission criteria, dispute handling, and revocation across registries are unwritten. This is the part of the ecosystem most likely to generate friction.

Replay protection is operator work. A signature stays valid until it expires, and a signature that covers only @authority verifies against any method, path, or body sent to that authority until then. Nothing in the protocol stops a captured signature from being replayed within its window unless the verifier tracks nonces. The protocol draft adds no nonce requirement of its own and leaves replay protection to RFC 9421 and to verifier policy. Our replay note explains the trade-off.

Key management is the usual hard part. Rotation, compromise response, and the question of how long verifiers may cache a directory are all live topics in the working group. The optional nbf and exp key fields, the rotation procedure in the protocol draft (section 5.5.2), and the recommended response self-signature are partial answers. The draft defines no revocation mechanism.

Privacy cuts both ways. Per-operator keys make agent traffic linkable by design; that is the feature. For cases where a bot should prove “I am rate-limited and accountable” without revealing which bot it is, Eric Rescorla’s anonymous authentication draft explores blind-signature schemes. Early days.

And robots.txt still applies. Web Bot Auth does not change what a crawler may fetch. It changes whether the “who” in your access rules means anything.

Resources

Type	Resource	What it is
Spec	Protocol draft	The core document: how agents sign, the Signature-Agent header, the key directory, and how origins verify
Spec	Registry draft	The Signature Agent Card and registry formats
Spec	Use cases draft	The problem catalogue the working group works from
Spec	WebBotAuth WG	Charter, documents, milestones, mailing list
Spec	RFC 9421	HTTP Message Signatures, the published foundation
Vendor	Cloudflare	Verification reference, verified bots, signed agents
Vendor	Akamai	Bot management support announcement
Vendor	Vercel	Bot management documentation
Vendor	Amazon Web Services	AgentCore Browser signing guide
Vendor	Google	Google-Agent signing documentation
Vendor	Shopify	Rate limits and crawler access keys
Code	cloudflare/web-bot-auth	Reference monorepo: TypeScript + Rust libraries, examples, test vectors
Code	web-bot-auth on npm	The TypeScript signing and verification package
Code	Apache module	Origin-side verification for Apache
Code	Draft sources	Where the drafts are developed in the open
This site	Check a domain	Grade a published directory A to F
This site	Verify a signature	Check captured request headers
This site	The short primer	The two-minute version of this guide
This site	How grading works	Verdicts, scores, and capping rules
This site	The agent card	The metadata document, field by field