Which websites block Claude-SearchBot?
Claude-SearchBot is Anthropic's crawler that improves Claude's search results. Of the 1,278 popular websites we checked, 125 (10%) block it completely and 935 restrict some paths. 123 sites name it explicitly in robots.txt.
How to block or allow Claude-SearchBot
# block User-agent: Claude-SearchBot Disallow: / # allow User-agent: Claude-SearchBot Allow: /
Blocking Claude-SearchBot can keep your pages out of Anthropic's AI answers and citations. Check your robots.txt.
Sites that block Claude-SearchBot (125)
- facebook.com
- instagram.com
- mail.ru
- twitter.com
- amazon.com
- x.com
- pinterest.com
- tiktok.com
- myfritz.net
- msn.com
- baidu.com
- reddit.com
- registrar-servers.com
- nytimes.com
- cnn.com
- theguardian.com
- meraki.com
- imdb.com
- launchpad.net
- reuters.com
- weibo.com
- skybridge.click
- naver.com
- bloomberg.com
- wsj.com
- unsplash.com
- sohu.com
- quora.com
- wired.com
- uol.com.br
- usatoday.com
- threads.com
- imgsmail.ru
- telegraph.co.uk
- sharethrough.com
- elpais.com
- yelp.com
- dailymail.com
- hihonorcloud.com
- lemonde.fr
- fb.com
- pinimg.com
- quantserve.com
- apnews.com
- people.com
- adform.net
- lefigaro.fr
- daum.net
- nbcnews.com
- mailchi.mp
- btloader.com
- investopedia.com
- theatlantic.com
- as.com
- huffpost.com
- xiaohongshu.com
- aftonbladet.se
- gsmarena.com
- nypost.com
- economist.com
- theconversation.com
- netcraze.io
- namu.wiki
- vg.no
- amap.com
- kakao.com
- themoviedb.org
- spiegel.de
- actu.fr
- sapo.pt
- iltalehti.fi
- eenadu.net
- tjk.org
- aka.ms
- keirin.jp
- qcloud.com
- newyorker.com
- zdnet.com
- rocketmoney.dev
- adrta.com
- marketwatch.com
- wikihow.com
- blackhub.team
- sedoparking.com
- clevertap-prod.com
- secureserver.net
- salesforceliveagent.com
- adsco.re
- mashable.com
- thesun.co.uk
- openssl.org
- instacart.com
- congress.gov
- co.com
- guardian.co.uk
- redditmedia.com
- news.com.au
- expressapisv2.net
- pxf.io
- kuaishou.com
- tmdb.org
- huntress.io
- system-monitor.com
- newscientist.com
- ksapisrv.com
- franceinfo.fr
- salesforce-scrt.com
- jwpltx.com
- chicagotribune.com
- osu.edu
- imageshack.us
- nrk.no
- france24.com
- thetimes.co.uk
- hellotalk8.com
- kahoot.it
- vogue.com
- metro.co.uk
- siteforce.com
- allrecipes.com
- nzherald.co.nz
- nate.com
- permutive.app
- cuny.edu
- nymag.com
Popular sites that allow Claude-SearchBot
What is Claude-SearchBot?
Claude-SearchBot is operated by Anthropic. According to Anthropic's documentation, it improves Claude's search results.
How many websites block Claude-SearchBot?
125 of the 1278 popular websites in our directory (10%) block it completely in robots.txt.
Should I block Claude-SearchBot?
Only if you don't want Anthropic's AI answers to read or cite your pages. Most businesses that want to be recommended by AI assistants allow it.