Does grok.com block AI crawlers?

Can AI assistants cite grok.com?

• ChatGPT search: its search crawler is blocked
• Claude: its search crawlers may read the site
• Perplexity: its search crawler is blocked

grok.com opts out of AI training by OpenAI, Anthropic, Google, Apple. Training crawlers and answer crawlers are separate: blocking training doesn't by itself remove a site from AI answers.

AI crawler access, bot by bot

How grok.com compares

• 19% of the 1,278 popular sites we checked block GPTBot; grok.com is one of them.
• 18% of the 1,278 popular sites we checked block ClaudeBot; grok.com is one of them.
• 14% of the 1,278 popular sites we checked block PerplexityBot; grok.com is one of them.
• Among 175 sites of similar popularity, 25% block at least one AI crawler.

The robots.txt rules that apply to AI crawlers

User-agent: *
Allow: /
Allow: /chat$
Allow: /chat/$
Allow: /c$
Allow: /c/$
Disallow: /rest
Disallow: /share-links
Disallow: /c/
Disallow: /chat/
Disallow: /project/*
Disallow: /_worker/*

User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Allow: /$
Allow: /chat$
Allow: /highlights
Allow: /imagine
Allow: /plans$
Allow: /plans/business$
Allow: /project$
Allow: /automations$
Allow: /supergrok/imagine$
Disallow: /

Excerpt: only groups for all user-agents (*) or named AI crawlers, up to 40 rules per group.

Sites with a similar AI policy

• huntress.io · 12 blocked
• cell.com · 8 blocked
• system-monitor.com · 12 blocked
• newscientist.com · 12 blocked
• ksapisrv.com · 12 blocked
• franceinfo.fr · 12 blocked
• salesforce-scrt.com · 12 blocked
• tmdb.org · 12 blocked

All sites that are “blocks ai training and ai answers” →