Does sas.com block AI crawlers?

Can AI assistants cite sas.com?

• ChatGPT search: its search crawlers may read the site
• Claude: its search crawlers may read the site
• Perplexity: its search crawlers may read the site

sas.com opts out of AI training by Apple, Meta. Training crawlers and answer crawlers are separate: blocking training doesn't by itself remove a site from AI answers.

AI crawler access, bot by bot

How sas.com compares

• 19% of the 1,278 popular sites we checked block GPTBot; sas.com does not.
• 18% of the 1,278 popular sites we checked block ClaudeBot; sas.com does not.
• 14% of the 1,278 popular sites we checked block PerplexityBot; sas.com does not.
• Among 178 sites of similar popularity, 26% block at least one AI crawler.

The robots.txt rules that apply to AI crawlers

User-agent: Googlebot
User-agent: Bingbot
User-agent: Duckduckbot
User-agent: Slurp
User-agent: Baiduspider
User-agent: YandexBot
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: Moz
User-agent: MJ12Bot
User-agent: facebookexternalhit
User-agent: Twitterbot
User-agent: LinkedInBot
User-agent: Slackbot
User-agent: Pinterestbot
User-agent: GPTBot
User-agent: CCBot
User-agent: PerplexityBot
User-agent: Bytespider
User-agent: Amazonbot
User-agent: Applebot
User-agent: SiteimproveBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Google-Extended
Disallow: /content/sascom/
Disallow: /apps/
Disallow: /ctx/
Disallow: /mv/
Disallow: /adminapps1/
Disallow: /adminapps2/
Disallow: /adminapps3/
Disallow: /adminapps4/
Disallow: /analystconf/
Disallow: /bin/f2f.cgi
Disallow: /bin/broker2
Disallow: /events/sasforuminternational/schedulebuilder/
Disallow: /bin/broker
Disallow: /forums/
Disallow: /images/email/
Disallow: /onlinedoc/
Disallow: /usergroups/sasforuminternational/call/
Disallow: /usergroups/sasforuminternational/proceedings/
Disallow: /usergroups/seugi/call/
Disallow: /search/
Disallow: /reg/
Disallow: /store/
Disallow: /contract/
Disallow: /pstest/
Disallow: /offices/europe/germany/css/
Disallow: /offices/europe/germany/curated/
Disallow: /offices/europe/germany/download/
Disallow: /offices/europe/germany/email/
Disallow: /offices/europe/germany/forms/
Disallow: /offices/europe/germany/includes/
Disallow: /offices/europe/germany/internal_use/
Disallow: /offices/europe/germany/kampagnen/ressources/download/pdf/
Disallow: /offices/europe/germany/popup/
Disallow: /offices/europe/germany/surveys/
Disallow: /offices/europe/germany/commerzbank/
Disallow: /reg/gen/de/commerzbank_sas
Disallow: /offices/europe/slovakia/sig/
Disallow: /offices/europe/greece/sig/
Disallow: /offices/europe/czech/sig/
Disallow: /offices/europe/france/pdf/

User-agent: *
Disallow: /

Excerpt: only groups for all user-agents (*) or named AI crawlers, up to 40 rules per group.

Sites with a similar AI policy

• theregister.com · 7 blocked
• loom.com · 1 blocked
• n-able.com · 1 blocked
• tagesschau.de · 7 blocked
• cbssports.com · 1 blocked
• crunchbase.com · 7 blocked
• starmakerstudios.com · 3 blocked
• nicovideo.jp · 1 blocked

All sites that are “blocks ai training, allows ai answers” →