Which websites block ClaudeBot?
ClaudeBot is Anthropic's crawler that collects content that may be used to train Anthropic's models. Of the 1,278 popular websites we checked, 234 (18%) block it completely and 836 restrict some paths. 248 sites name it explicitly in robots.txt.
How to block or allow ClaudeBot
# block User-agent: ClaudeBot Disallow: / # allow User-agent: ClaudeBot Allow: /
ClaudeBot is a training crawler: blocking it opts out of model training but doesn't by itself remove you from AI answers. Check your robots.txt.
Sites that block ClaudeBot (234)
- instagram.com
- mail.ru
- twitter.com
- amazon.com
- whatsapp.net
- x.com
- pinterest.com
- tiktok.com
- yahoo.com
- myfritz.net
- msn.com
- baidu.com
- vimeo.com
- reddit.com
- registrar-servers.com
- snapchat.com
- nytimes.com
- flickr.com
- soundcloud.com
- medium.com
- cnn.com
- theguardian.com
- ebay.com
- t-online.de
- forbes.com
- bbc.com
- canva.com
- meraki.com
- imdb.com
- launchpad.net
- reuters.com
- nature.com
- weibo.com
- skybridge.click
- naver.com
- washingtonpost.com
- bloomberg.com
- wsj.com
- unsplash.com
- businessinsider.com
- aol.com
- pixabay.com
- cnbc.com
- statista.com
- sohu.com
- quora.com
- scribd.com
- behance.net
- slideshare.net
- ft.com
- wired.com
- uol.com.br
- tripadvisor.com
- usatoday.com
- threads.com
- imgsmail.ru
- telegraph.co.uk
- sharethrough.com
- techcrunch.com
- elpais.com
- yelp.com
- dailymail.com
- hihonorcloud.com
- lemonde.fr
- fb.com
- pinimg.com
- bandcamp.com
- quantserve.com
- airbnb.com
- apnews.com
- ted.com
- people.com
- adform.net
- lefigaro.fr
- academia.edu
- pexels.com
- daum.net
- figma.com
- nbcnews.com
- latimes.com
- mercadolivre.com.br
- genius.com
- mercadolibre.com.ar
- mailchi.mp
- webmd.com
- corriere.it
- btloader.com
- investopedia.com
- theatlantic.com
- letterboxd.com
- as.com
- huffpost.com
- xiaohongshu.com
- aftonbladet.se
- cookpad.com
- gsmarena.com
- nypost.com
- economist.com
- theconversation.com
- netcraze.io
- ad.nl
- nu.nl
- namu.wiki
- indiamart.com
- vnexpress.net
- vg.no
- fmkorea.com
- dcinside.com
- pandora.com
- gazzetta.it
- amap.com
- hln.be
- kakao.com
- mundodeportivo.com
- abc.net.au
- litnet.com
- themoviedb.org
- ladepeche.fr
- mydramalist.com
- spiegel.de
- actu.fr
- justdial.com
- jstor.org
- sapo.pt
- dagbladet.no
- ilmeteo.it
- iltalehti.fi
- eenadu.net
- tjk.org
- aka.ms
- meteociel.fr
- keirin.jp
- shutterstock.com
- usnews.com
- vinted.fr
- qcloud.com
- newsweek.com
- newyorker.com
- merkur.de
- zdnet.com
- rocketmoney.dev
- adrta.com
- marketwatch.com
- liquipedia.net
- meb.gov.tr
- iqiyi.com
- wikihow.com
- blackhub.team
- lnkd.in
- strava.com
- sedoparking.com
- clevertap-prod.com
- secureserver.net
- salesforceliveagent.com
- adsco.re
- mashable.com
- thesun.co.uk
- openssl.org
- medicalnewstoday.com
- mparticle.com
- variety.com
- mirror.co.uk
- photobucket.com
- sendbird.com
- instacart.com
- congress.gov
- inc.com
- co.com
- guardian.co.uk
- rollingstone.com
- nba.com
- redditmedia.com
- slate.com
- tidal.com
- express.co.uk
- hollywoodreporter.com
- news.com.au
- rockstargames.com
- zeit.de
- expressapisv2.net
- smh.com.au
- sfgate.com
- pxf.io
- kuaishou.com
- leparisien.fr
- thehill.com
- tmdb.org
- grok.com
- huntress.io
- cell.com
- system-monitor.com
- newscientist.com
- ksapisrv.com
- franceinfo.fr
- salesforce-scrt.com
- sueddeutsche.de
- glassdoor.com
- idnes.cz
- jwpltx.com
- chicagotribune.com
- prezi.com
- osu.edu
- imageshack.us
- medscape.com
- nrk.no
- france24.com
- thetimes.co.uk
- starmakerstudios.com
- yumpu.com
- hellotalk8.com
- theregister.com
- tagesschau.de
- newsbreak.com
- yr.no
- kahoot.it
- crunchbase.com
- vogue.com
- myanimelist.net
- metro.co.uk
- siteforce.com
- unep.org
- allrecipes.com
- nzherald.co.nz
- zalando.de
- nate.com
- tagesspiegel.de
- permutive.app
- cuny.edu
- nymag.com
- ebsco.com
- deadline.com
- vietnam.vn
- orf.at
- ansa.it
Popular sites that allow ClaudeBot
What is ClaudeBot?
ClaudeBot is operated by Anthropic. According to Anthropic's documentation, it collects content that may be used to train Anthropic's models.
How many websites block ClaudeBot?
234 of the 1278 popular websites in our directory (18%) block it completely in robots.txt.
Should I block ClaudeBot?
It depends on whether you're comfortable with your content being used for AI training. Blocking it doesn't remove you from AI answers, which use separate crawlers.