What is Web Bot Auth?

Web Bot Auth is a standard that lets automated clients (bots, crawlers, and AI agents) prove who they are by signing their requests with a cryptographic key, instead of a User-Agent string that anyone can fake.

How it works

A bot operator generates an Ed25519 key pair and publishes the public key, as a JWK, at a well-known URL on their domain:

/.well-known/http-message-signatures-directory

When the bot makes a request, it signs that request using RFC 9421 HTTP Message Signatures. The signature covers at least the request’s authority (or its full target URI), usually a few more components, and carries a tag=web-bot-auth parameter so verifiers know which scheme it is.

To check it, the server (or something in front of it) fetches the bot’s directory, finds the key whose RFC 7638 thumbprint matches the signature’s keyid, and verifies the Ed25519 signature. If it verifies, the request really came from whoever holds that key, and there is no need to trust the User-Agent.

Why this matters

User-Agent strings are trivial to forge. Web Bot Auth moves identity onto a cryptographic proof: only the holder of the private key can produce a valid signature. That lets site operators tell real bots from impersonators and set per-bot access rules based on identity they can actually verify.

Key components

• JWKS directory - the JSON Web Key Set at /.well-known/http-message-signatures-directory, holding the bot’s Ed25519 public key(s).
• RFC 9421 signature - the HTTP Message Signature on each request, made with the matching private key.
• Agent Card (optional) - a metadata document, linked from the directory, describing the bot’s purpose, contacts, and rate expectations. This checker reads it as advisory information.

Who uses it

As of June 2026, Google has started signing some of its crawler traffic with Web Bot Auth. A subset of requests from its Google-Agent fetcher carry a Signature-Agent header pointing at https://agent.bot.goog, where Google publishes its key directory, so site operators can verify that traffic cryptographically instead of guessing from a User-Agent or IP range. It is still experimental, and Google recommends keeping your usual IP and DNS checks as a fallback. For the current details, see Google’s Web Bot Auth documentation.

What this checker does

It fetches a domain’s published directory and checks it against the spec: are the keys well-formed? Do the kid values match their RFC 7638 thumbprints? Is any private key material exposed? Paste in request headers and it also verifies the RFC 9421 signature. You get back a structured report with a letter grade.

See how grading works for the full scoring reference.

This page is the short version. For the full picture, the standard’s history, the IETF status, who signs and who verifies today, and how to implement it on either side, see The Ultimate Guide to Web Bot Auth.