robots.txt (RFC 9309) compliance

What this check verifies

The Signature Agent Card is optional metadata describing the agent behind a Web Bot Auth deployment, defined in the web-bot-auth registry draft (draft-meunier-webbotauth-registry). It is separate from the JWKS directory and has no fixed well-known path.

This check looks for an rfc9309-compliance member on the card. It passes when the member is present, regardless of its value, and warns when it is absent. The field states whether the agent honours robots.txt as defined in the Robots Exclusion Protocol, RFC 9309.

This check is advisory. A missing declaration lowers the score slightly and raises a warning, but it never caps the grade and never changes the verdict. A directory with valid keys is VALID even with no card at all.

Why it matters

robots.txt is how a site tells crawlers which paths they may fetch. Declaring RFC 9309 compliance on the card states up front whether the agent reads and follows those rules, which is one of the first things a site operator wants to know about an automated visitor. The declaration is a stated commitment, not a guarantee of behaviour, but it tells the operator what to expect and what to hold the agent to.

How to fix it

Add an rfc9309-compliance member that states whether the agent follows robots.txt:

{ "rfc9309-compliance": true }

Set it to reflect what the agent actually does. If it does honour robots.txt, say so; if it does not, declaring that is more useful than leaving the field out.

References

• RFC 9309 defines the Robots Exclusion Protocol (robots.txt).
• The web-bot-auth registry draft (draft-meunier-webbotauth-registry) defines the Signature Agent Card and its rfc9309-compliance field.
• What is a Signature Agent Card? explains the card and how it differs from an A2A or MCP agent card.
• How grading works explains why card checks are advisory.