Skip to content
AskableHQ
Launch

AI crawler directory

Which websites block AI crawlers?

We read the robots.txt of 1,278 popular websites to see which AI crawlers they allow, restrict or block — and whether they publish llms.txt. Updated Oct 2, 2026.
Open to AI crawlers
73%
939 sites
Blocks AI training, allows AI answers
10%
129 sites
Blocks AI answer engines
0%
2 sites
Blocks AI training and AI answers
16%
208 sites

Share of sites blocking each AI crawler

Orange: crawlers that power AI answers. Grey: crawlers for AI training. Only full blocks (Disallow: /) are counted.

19% of these sites publish an llms.txt file (245 sites).

Most popular sites

Browse all sites A–Z

Methodology

  • Sites: the most popular domains in the Tranco list (list ID Y83YG), one domain per brand, excluding infrastructure/CDN hosts and adult sites.
  • We fetched each site's /robots.txt and /llms.txt once, identifying as AskableHQ. Sites without a readable robots.txt aren't included.
  • Blocked means the rules that apply to that crawler (its own group, or * if it has none) disallow /. Some paths restricted means only parts of the site are disallowed.
  • robots.txt is a public request, not enforcement: sites may also block bots at the firewall, which this data doesn't capture. Crawler purposes follow each vendor's own documentation.
  • Each page shows the date its data was checked. We re-check the directory regularly.

Should you block GPTBot? Our guide →