Does sagepub.com block AI crawlers?

Can AI assistants cite sagepub.com?

• ChatGPT search: its search crawlers may read the site
• Claude: its search crawlers may read the site
• Perplexity: its search crawlers may read the site

sagepub.com opts out of AI training by Common Crawl, ByteDance, Meta. Training crawlers and answer crawlers are separate: blocking training doesn't by itself remove a site from AI answers.

AI crawler access, bot by bot

How sagepub.com compares

• 19% of the 1,278 popular sites we checked block GPTBot; sagepub.com does not.
• 18% of the 1,278 popular sites we checked block ClaudeBot; sagepub.com does not.
• 14% of the 1,278 popular sites we checked block PerplexityBot; sagepub.com does not.
• Among 195 sites of similar popularity, 29% block at least one AI crawler.

The robots.txt rules that apply to AI crawlers

User-agent: *
Allow: /misc/*.css$
Allow: /misc/*.css?
Allow: /misc/*.js$
Allow: /misc/*.js?
Allow: /misc/*.gif
Allow: /misc/*.jpg
Allow: /misc/*.jpeg
Allow: /misc/*.png
Allow: /modules/*.css$
Allow: /modules/*.css?
Allow: /modules/*.js$
Allow: /modules/*.js?
Allow: /modules/*.gif
Allow: /modules/*.jpg
Allow: /modules/*.jpeg
Allow: /modules/*.png
Allow: /profiles/*.css$
Allow: /profiles/*.css?
Allow: /profiles/*.js$
Allow: /profiles/*.js?
Allow: /profiles/*.gif
Allow: /profiles/*.jpg
Allow: /profiles/*.jpeg
Allow: /profiles/*.png
Allow: /themes/*.css$
Allow: /themes/*.css?
Allow: /themes/*.js$
Allow: /themes/*.js?
Allow: /themes/*.gif
Allow: /themes/*.jpg
Allow: /themes/*.jpeg
Allow: /themes/*.png

User-agent: *
Disallow: /sitefinity/
Disallow: /Sitefinity/
Disallow: /misc/
Disallow: /modules/
Disallow: /profiles/
Disallow: /scripts/
Disallow: /themes/
Disallow: /CHANGELOG.txt
Disallow: /cron.php
Disallow: /INSTALL.mysql.txt
Disallow: /INSTALL.pgsql.txt
Disallow: /INSTALL.sqlite.txt
Disallow: /install.php
Disallow: /INSTALL.txt
Disallow: /LICENSE.txt
Disallow: /MAINTAINERS.txt
Disallow: /update.php
Disallow: /UPGRADE.txt
Disallow: /xmlrpc.php
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /filter/tips/
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register/
Disallow: /user/password/
Disallow: /user/login/
Disallow: /user/logout/
Disallow: /?q=admin/
Disallow: /?q=comment/reply/
Disallow: /?q=filter/tips/
Disallow: /?q=node/add/
Disallow: /?q=search/
Disallow: /?q=user/password/
Disallow: /?q=user/register/
Disallow: /?q=user/login/
Disallow: /?q=user/logout/

User-agent: Applebot-Extended
Disallow: 

User-agent: Google-Extended
Disallow: 

User-agent: GPTBot
Disallow: 

User-agent: ChatGPT-User
Disallow: 

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: anthropic-ai
Disallow: 

User-agent: ClaudeBot
Disallow: 

User-agent: meta-externalagent
Disallow: /

Excerpt: only groups for all user-agents (*) or named AI crawlers, up to 40 rules per group.

Sites with a similar AI policy

• cbsnews.com · 1 blocked
• tripadvisor.com · 6 blocked
• bandcamp.com · 6 blocked
• airbnb.com · 3 blocked
• tandfonline.com · 1 blocked
• slideshare.net · 3 blocked
• zillow.com · 2 blocked
• behance.net · 7 blocked

All sites that are “blocks ai training, allows ai answers” →