A standard, a registry and a public record
Readable and usable by machines. Proven.
Machine Readiness defines what it means for a website to be readable by AI crawlers, usable by AI agents, and verifiably permitted for each use. It keeps the public record that goes with the definition: the registry of AI crawlers, the permission check for the people who fetch pages, and the standard itself.
Maintained by the makers of Crawl Readiness. Nothing is sold on this site, and nothing here promises that an AI will mention anyone.
What the top 1,000 sites look like
of reachable top sites refuse an AI crawler their robots.txt allows. The policy says yes; the CDN says no.
block GPTBot in robots.txt and serve it the page anyway. A block that is not enforced is not a block.
sites with forms has labelled them so an AI agent can use them. Readable is common. Usable is not.
From the October 2026 study of the Tranco top 1,000, re-run monthly. Data and method published under CC BY 4.0.
What lives here
The standard
Four parts: readable, access, usable, proven. Each is a checklist a site either meets or does not, measured from outside on a date.
Read version 1.0 →The crawler registry
Every AI crawler token we know, who runs it and what it is for: search, answers, training, or fetching on a user's behalf. Human-readable, and as JSON under CC BY 4.0.
Open the registry →The permission check
For the people who build with pages: send a page and a purpose, get back what every published signal says, what the server did when asked, and a signed receipt.
Read the API →Verify pages
A public page per site with its latest verified result and its policy statement, the place a Machine Ready badge will point to. Coming.
Not yet published.Who it is for
Site owners and agencies
Check a site against the standard for free, get the fixes, keep it that way. That work lives on Crawl Readiness.
People who build with pages
RAG and agent products, scrapers, dataset vendors. The permission check reads robots.txt, Content-Signal, Content-Usage, RSL, the TDM reservation, ai.txt and the page's own directives so you do not have to.
Researchers and tool makers
The registry as JSON and the monthly study, both under CC BY 4.0, with the reproduce-it script beside the data.