Does thesun.co.uk block AI crawlers?

Can AI assistants cite thesun.co.uk?

• ChatGPT search: its search crawlers may read the site
• Claude: its search crawler is blocked
• Perplexity: its search crawler is blocked

thesun.co.uk opts out of AI training by Anthropic, Google, Apple, Common Crawl, ByteDance, Meta. Training crawlers and answer crawlers are separate: blocking training doesn't by itself remove a site from AI answers.

AI crawler access, bot by bot

How thesun.co.uk compares

• 19% of the 1,278 popular sites we checked block GPTBot; thesun.co.uk does not.
• 18% of the 1,278 popular sites we checked block ClaudeBot; thesun.co.uk is one of them.
• 14% of the 1,278 popular sites we checked block PerplexityBot; thesun.co.uk is one of them.
• Among 168 sites of similar popularity, 22% block at least one AI crawler.

The robots.txt rules that apply to AI crawlers

User-agent: *
Disallow: /

User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-Video
User-agent: Googlebot-News
User-agent: Google-InspectionTool
User-agent: GoogleOther
User-agent: APIs-Google
User-agent: Mediapartners-Google
User-agent: DuckDuckBot
User-agent: Slurp
User-agent: Twitterbot
User-agent: Bingbot
User-agent: adidxbot
User-agent: MicrosoftPreview
User-agent: Applebot
User-agent: AdsBot-Google
User-agent: AmazonAdBot
User-agent: facebookexternalhit
User-agent: Yahoo Ad Monitoring
User-agent: parse.ly scraper
User-agent: Screaming Frog SEO Spider
User-agent: SemrushBot
User-agent: OnCrawl
User-agent: Chrome-Lighthouse
User-agent: Slackbot-LinkExpanding
User-agent: Slack-ImgProxy
User-agent: Slackbot
User-agent: Snap URL Preview Service
User-agent: proximic
User-agent: GumGum-Bot
User-agent: TTD-Content
User-agent: Pubmatic
User-agent: ias_crawler
User-agent: outbrain
User-agent: Leikibot
User-agent: weborama-fetcher
User-agent: AHC/2.1
User-agent: SirdataBot
User-agent: peer39_crawler/1.0
User-agent: EchoboxBot/1.0
User-agent: CriteoBot/0.1
User-agent: AdDefend Site Context Crawler/1.0
User-agent: SmartologyBot
User-agent: Dianomi
User-agent: ChatGPT-User
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: snews
Allow: /
Allow: /ads.txt
Allow: /wp-admin/admin-ajax.php
Disallow: /search/
Disallow: /horoscope/
Disallow: /*?x=
Disallow: /*?action=
Disallow: /*&action=
Disallow: /*?adid=
Disallow: /*&adid=
Disallow: /*?bitrate=
Disallow: /*&bitrate=
Disallow: /*?channel=
Disallow: /*&channel=
Disallow: /*?clipid=
Disallow: /*&clipid=
Disallow: /*?contentCollection=
Disallow: /*&contentCollection=
Disallow: /*?contentId=
Disallow: /*&contentId=
Disallow: /*?ei=
Disallow: /*&ei=
Disallow: /*?format=
Disallow: /*&format=
Disallow: /*?height=
Disallow: /*&height=
Disallow: /*?iframe=
Disallow: /*&iframe=
Disallow: /*?mediaId=
Disallow: /*&mediaId=
Disallow: /*?module=
Disallow: /*&module=
Disallow: /*?monthnum=
Disallow: /*&monthnum=
Disallow: /*?pgtype=
Disallow: /*&pgtype=
Disallow: /*?preview=
Disallow: /*&preview=
Disallow: /*?priority=
Disallow: /*&priority=

Excerpt: only groups for all user-agents (*) or named AI crawlers, up to 40 rules per group.

Sites with a similar AI policy

• mashable.com · 11 blocked
• openssl.org · 12 blocked
• mparticle.com · 10 blocked
• variety.com · 10 blocked
• mirror.co.uk · 7 blocked
• adsco.re · 12 blocked
• salesforceliveagent.com · 12 blocked
• ign.com · 3 blocked

All sites that are “blocks ai training and ai answers” →