Does wsj.com block AI crawlers?

Can AI assistants cite wsj.com?

• ChatGPT search: its search crawlers may read the site
• Claude: its search crawler is blocked
• Perplexity: its search crawler is blocked

wsj.com opts out of AI training by Anthropic, Google, Apple, Common Crawl, ByteDance, Meta. Training crawlers and answer crawlers are separate: blocking training doesn't by itself remove a site from AI answers.

AI crawler access, bot by bot

How wsj.com compares

• 19% of the 1,278 popular sites we checked block GPTBot; wsj.com does not.
• 18% of the 1,278 popular sites we checked block ClaudeBot; wsj.com is one of them.
• 14% of the 1,278 popular sites we checked block PerplexityBot; wsj.com is one of them.
• Among 193 sites of similar popularity, 30% block at least one AI crawler.

The robots.txt rules that apply to AI crawlers

User-agent: *
Disallow: /

User-agent: googlebot
User-agent: googlebot-image
User-agent: GoogleOther
User-agent: Googlebot-Video
User-agent: Google-InspectionTool
User-agent: AdsBot-Google
User-agent: AdsBot-Google-Mobile
User-agent: AdsBot-Google-Mobile-Apps
User-agent: Storebot-Google
User-agent: google-read-aloud
User-agent: bingbot
User-agent: msnbot
User-agent: bingpreview
User-agent: slurp
User-agent: yahoo
User-agent: baiduspider
User-agent: Pinterestbot
User-agent: Yeti
User-agent: MojeekBot
User-agent: 360Spider
User-agent: google-cloudvertexbot
User-agent: duckduckbot
User-agent: Applebot
User-agent: flipboard
User-agent: qwantbot
User-agent: SeznamBot
User-agent: proximic
User-agent: admantx
User-agent: thetradedesk
User-agent: outbrain
User-agent: ias_crawler
User-agent: AmazonAdBot
User-agent: pubmatic
User-agent: smartologybot
User-agent: parselybot
User-agent: BomboraBot
User-agent: IllumaBot/1.0
User-agent: Publication-Access-for-Facebook
User-agent: Wallie
User-agent: mediapartners-google
User-agent: Screaming Frog SEO Spider
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: SimilarWebBot
User-agent: SISTRIX
User-agent: botify
User-agent: Chrome-Lighthouse
User-agent: ChatGPT-User
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: facebookexternalhit
User-agent: facebot
User-agent: twitterbot
User-agent: linkedinbot
User-agent: snapchat
User-agent: sentry
User-agent: Iframely
User-agent: Vocabtracker
User-agent: EpvzCrawl6194680250
User-agent: Citoid
User-agent: ZoteroTranslationServer
Allow: /
Disallow: /article_email/*
Disallow: /user/*
Disallow: /pdf/documents/*
Disallow: /login/*
Disallow: /acct/*
Disallow: /msgcenter/*
Disallow: /setup/*
Disallow: /marketing/*
Disallow: /public/article/*
Disallow: /public/resources/documents/*
Disallow: /public/search/
Disallow: /public/search*
Disallow: /search*
Disallow: /public/page/wsj-x-marketing.html
Disallow: /public/page/news-media-marketing.html
Disallow: /public/page/0_0_WP_RT_MARKETING.html
Disallow: /news/articles/SB2*
Disallow: /news/articles/SB3*
Disallow: /news/articles/SB4*
Disallow: /articles/SB2*
Disallow: /articles/SB3*
Disallow: /articles/SB4*
Disallow: /article/AP*
Disallow: /article/BT-CO*
Disallow: /article/DN-CO*
Disallow: /article/PR-CO*
Disallow: /article/HUG*
Disallow: /video/search/*
Disallow: /articles/BT-CO*
Disallow: /articles/DN-CO*
Disallow: /articles/PR-CO*
Disallow: /news/articles/BT-CO*
Disallow: /news/articles/DN-CO*
Disallow: /news/articles/PR-CO*
Disallow: /catchup/*
Disallow: /articles/the-meaning-behind-juneteenth-11592413234
Disallow: /emailservice/*
Disallow: /emailsignup/*
Disallow: /insetsrv/v1/*

Excerpt: only groups for all user-agents (*) or named AI crawlers, up to 40 rules per group.

Sites with a similar AI policy

• bloomberg.com · 12 blocked
• espn.com · 5 blocked
• washingtonpost.com · 6 blocked
• unsplash.com · 3 blocked
• naver.com · 12 blocked
• claude.ai · 4 blocked
• skybridge.click · 12 blocked
• weibo.com · 12 blocked

All sites that are “blocks ai training and ai answers” →