Which websites block CCBot?
CCBot is Common Crawl's crawler that builds the Common Crawl dataset that many AI models train on. Of the 1,278 popular websites we checked, 265 (21%) block it completely and 813 restrict some paths. 246 sites name it explicitly in robots.txt.
How to block or allow CCBot
# block User-agent: CCBot Disallow: / # allow User-agent: CCBot Allow: /
CCBot is a training crawler: blocking it opts out of model training but doesn't by itself remove you from AI answers. Check your robots.txt.
Sites that block CCBot (265)
- facebook.com
- instagram.com
- mail.ru
- twitter.com
- amazon.com
- netflix.com
- x.com
- pinterest.com
- tiktok.com
- yahoo.com
- myfritz.net
- msn.com
- baidu.com
- chatgpt.com
- vimeo.com
- reddit.com
- amazonvideo.com
- registrar-servers.com
- snapchat.com
- gravatar.com
- nytimes.com
- flickr.com
- soundcloud.com
- cnn.com
- theguardian.com
- ebay.com
- t-online.de
- forbes.com
- bbc.com
- canva.com
- who.int
- imdb.com
- forter.com
- launchpad.net
- reuters.com
- nature.com
- weibo.com
- weather.com
- skybridge.click
- naver.com
- washingtonpost.com
- bloomberg.com
- wsj.com
- calendly.com
- espn.com
- globo.com
- businessinsider.com
- aol.com
- pixabay.com
- goodreads.com
- cnbc.com
- statista.com
- sohu.com
- quora.com
- scribd.com
- behance.net
- slideshare.net
- ft.com
- wired.com
- uol.com.br
- tripadvisor.com
- usatoday.com
- threads.com
- imgsmail.ru
- telegraph.co.uk
- sharethrough.com
- techcrunch.com
- sagepub.com
- primevideo.com
- elpais.com
- yelp.com
- dailymail.com
- hihonorcloud.com
- lemonde.fr
- fb.com
- pinimg.com
- bandcamp.com
- quantserve.com
- bild.de
- apnews.com
- ted.com
- welt.de
- people.com
- adform.net
- lefigaro.fr
- academia.edu
- pexels.com
- daum.net
- figma.com
- hbr.org
- nbcnews.com
- genius.com
- mailchi.mp
- repubblica.it
- webmd.com
- corriere.it
- btloader.com
- investopedia.com
- elmundo.es
- flightradar24.com
- flashscore.com
- theatlantic.com
- letterboxd.com
- themeforest.net
- n-tv.de
- pbs.org
- archiveofourown.org
- as.com
- huffpost.com
- marca.com
- xiaohongshu.com
- skysports.com
- aftonbladet.se
- cookpad.com
- gsmarena.com
- nypost.com
- economist.com
- theconversation.com
- netcraze.io
- ad.nl
- nu.nl
- namu.wiki
- vnexpress.net
- vg.no
- fmkorea.com
- dcinside.com
- rightmove.co.uk
- pandora.com
- gazzetta.it
- hln.be
- kakao.com
- abc.net.au
- themoviedb.org
- ladepeche.fr
- mydramalist.com
- spiegel.de
- actu.fr
- jstor.org
- sapo.pt
- nikkansports.com
- dagbladet.no
- ilmeteo.it
- iltalehti.fi
- eenadu.net
- tjk.org
- aka.ms
- keirin.jp
- shutterstock.com
- usnews.com
- hostinger.com
- vinted.fr
- qcloud.com
- newsweek.com
- newyorker.com
- ixl.com
- merkur.de
- zdnet.com
- rocketmoney.dev
- hola.com
- adrta.com
- marketwatch.com
- meb.gov.tr
- sciencemag.org
- wikihow.com
- oreilly.com
- euronews.com
- blackhub.team
- diretta.it
- lnkd.in
- sedoparking.com
- clevertap-prod.com
- science.org
- secureserver.net
- salesforceliveagent.com
- gizmodo.com
- adsco.re
- mashable.com
- thesun.co.uk
- fliphtml5.com
- openssl.org
- medicalnewstoday.com
- mparticle.com
- variety.com
- mirror.co.uk
- photobucket.com
- sendbird.com
- instacart.com
- gameanalytics.com
- congress.gov
- inc.com
- co.com
- mos.ru
- guardian.co.uk
- nejm.org
- rollingstone.com
- nba.com
- redditmedia.com
- slate.com
- abcnews.com
- tidal.com
- express.co.uk
- hollywoodreporter.com
- news.com.au
- rockstargames.com
- zeit.de
- expressapisv2.net
- smh.com.au
- sfgate.com
- pxf.io
- kuaishou.com
- leparisien.fr
- thehill.com
- tmdb.org
- envato.com
- aarp.org
- huntress.io
- cell.com
- system-monitor.com
- newscientist.com
- ksapisrv.com
- franceinfo.fr
- salesforce-scrt.com
- glassdoor.com
- venturebeat.com
- jwpltx.com
- chicagotribune.com
- osu.edu
- imageshack.us
- medscape.com
- nrk.no
- ctvnews.ca
- telstra.com
- france24.com
- thetimes.co.uk
- yumpu.com
- hellotalk8.com
- theregister.com
- n-able.com
- tagesschau.de
- newsbreak.com
- yr.no
- kahoot.it
- crunchbase.com
- vogue.com
- myanimelist.net
- metro.co.uk
- siteforce.com
- unep.org
- allrecipes.com
- nzherald.co.nz
- geeksforgeeks.org
- worldcat.org
- flipboard.com
- dictionary.com
- nate.com
- tagesspiegel.de
- permutive.app
- cuny.edu
- 123rf.com
- nymag.com
- ebsco.com
- deadline.com
- vietnam.vn
- orf.at
- ansa.it
Popular sites that allow CCBot
What is CCBot?
CCBot is operated by Common Crawl. According to Common Crawl's documentation, it builds the Common Crawl dataset that many AI models train on.
How many websites block CCBot?
265 of the 1278 popular websites in our directory (21%) block it completely in robots.txt.
Should I block CCBot?
It depends on whether you're comfortable with your content being used for AI training. Blocking it doesn't remove you from AI answers, which use separate crawlers.