Does github.com block AI crawlers?

Can AI assistants cite github.com?

• ChatGPT search: its search crawlers may read the site
• Claude: its search crawlers may read the site
• Perplexity: its search crawlers may read the site

github.com opts out of AI training by ByteDance. Training crawlers and answer crawlers are separate: blocking training doesn't by itself remove a site from AI answers.

AI crawler access, bot by bot

How github.com compares

• 19% of the 1,278 popular sites we checked block GPTBot; github.com does not.
• 18% of the 1,278 popular sites we checked block ClaudeBot; github.com does not.
• 14% of the 1,278 popular sites we checked block PerplexityBot; github.com does not.
• Among 100 sites of similar popularity, 38% block at least one AI crawler.

The robots.txt rules that apply to AI crawlers

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: PerplexityBot
Allow: /$
Allow: /about
Allow: /customer-stories
Allow: /enterprise
Allow: /features
Allow: /marketplace
Allow: /mcp
Allow: /newsroom
Allow: /pricing
Allow: /resources
Allow: /security
Allow: /solutions
Allow: /topics/download*
Allow: /topics/comments*
Allow: /marketplace/actions/download*
Disallow: /*/*/pulse
Disallow: /*/*/projects
Disallow: /*/*/forks
Disallow: /*/*/issues/new
Disallow: /*/*/milestones/new
Disallow: /*/*/issues/search
Disallow: /*/*/commits/
Disallow: /*/*/branches
Disallow: /*/*/contributors
Disallow: /*/*/tags
Disallow: /*/*/stargazers
Disallow: /*/*/watchers
Disallow: /*/*/network
Disallow: /*/*/graphs
Disallow: /*/*/compare
Disallow: /*/tree/
Disallow: /gist/
Disallow: /*/download
Disallow: /*/revisions
Disallow: /*/commits/*?author
Disallow: /*/commits/*?path
Disallow: /*/comments
Disallow: /*/archive/
Disallow: /*/blame/
Disallow: /*/raw/

User-agent: Bytespider
Disallow: /

User-agent: *
Allow: /*?tab=achievements&achievement=*
Allow: /marketplace/*?tab=readme$
Allow: /marketplace/*?tab=transparency$
Allow: /topics/download*
Allow: /topics/comments*
Allow: /marketplace/actions/download*
Disallow: /*/*/pulse
Disallow: /*/*/projects
Disallow: /*/*/forks
Disallow: /*/*/issues/new
Disallow: /*/*/milestones/new
Disallow: /*/*/issues/search
Disallow: /*/*/commits/
Disallow: /*/*/branches
Disallow: /*/*/contributors
Disallow: /*/*/tags
Disallow: /*/*/stargazers
Disallow: /*/*/watchers
Disallow: /*/*/network
Disallow: /*/*/graphs
Disallow: /*/*/compare
Disallow: /*/tree/
Disallow: /gist/
Disallow: /*/download
Disallow: /*/revisions
Disallow: /*/commits/*?author
Disallow: /*/commits/*?path
Disallow: /*/comments
Disallow: /*/archive/
Disallow: /*/blame/
Disallow: /*/raw/
Disallow: /*/cache/
Disallow: /.git/
Disallow: */.git/
Disallow: /*.git$
Disallow: /search/advanced
Disallow: /search$
Disallow: /*.atom$
Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*

Excerpt: only groups for all user-agents (*) or named AI crawlers, up to 40 rules per group.

Sites with a similar AI policy

• netflix.com · 4 blocked
• vimeo.com · 5 blocked
• snapchat.com · 7 blocked
• unity3d.com · 1 blocked
• gravatar.com · 4 blocked
• flickr.com · 7 blocked
• medium.com · 5 blocked
• canva.com · 6 blocked

All sites that are “blocks ai training, allows ai answers” →