Does cnet.com block AI crawlers?

Can AI assistants cite cnet.com?

• ChatGPT search: its search crawlers may read the site
• Claude: its search crawlers may read the site
• Perplexity: its search crawlers may read the site

cnet.com doesn't opt out of AI model training through robots.txt. Training crawlers and answer crawlers are separate: blocking training doesn't by itself remove a site from AI answers.

AI crawler access, bot by bot

How cnet.com compares

• 19% of the 1,278 popular sites we checked block GPTBot; cnet.com does not.
• 18% of the 1,278 popular sites we checked block ClaudeBot; cnet.com does not.
• 14% of the 1,278 popular sites we checked block PerplexityBot; cnet.com does not.
• Among 195 sites of similar popularity, 29% block at least one AI crawler.

The robots.txt rules that apply to AI crawlers

User-agent: *
Disallow: */*max-price=*
Disallow: */*min-price=*
Disallow: */Xhr*
Disallow: */account-information/*
Disallow: */ad/ad-cookie/*
Disallow: */ajax/
Disallow: */alert/*
Disallow: */cnet_global_app_redirect.xml
Disallow: */component/*
Disallow: */edit/
Disallow: */edit/change-email/*
Disallow: */edit/change-password/*
Disallow: */email/editor/*
Disallow: */email/member/*
Disallow: */email/newsletter/*
Disallow: */error/*
Disallow: */ext/make/ingest/*
Disallow: */feature/*
Disallow: */finder/*/*/?*filter=*_*_*_*_*
Disallow: */finder/*/?*filter=*_*_*_*_*
Disallow: */forums/*
Disallow: */g00/*
Disallow: */informacion-de-mi-cuenta/*
Disallow: */members/*
Disallow: */products/*?*filter=*
Disallow: */roadshow/finder/*
Disallow: */search/*
Disallow: */user/*
Disallow: */videos/manifest/*
Disallow: */videos/secure/*
Disallow: */videos/selector/*
Disallow: */wrap-function*
Disallow: */wrap-xhr*
Disallow: */xhr*
Disallow: *?sort=popularity*
Disallow: *?sort=price*
Disallow: *?sort=user-rating*
Disallow: *hub-preview-version=*
Disallow: /.well-known/*
Disallow: /api/editionurlgenerator/

User-agent: AddSearchBot
User-agent: AI2Bot
User-agent: Ai2Bot-DeepResearchEval
User-agent: Ai2Bot-Dolma
User-agent: AIWebIndex
User-agent: amazon-QBusiness
User-agent: Amazonbot
User-agent: AmazonBuyForMe
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: Anchor Browser
User-agent: anthropic-ai
User-agent: ApifyBot
User-agent: ApifyWebsiteContentCrawler
User-agent: Applebot
User-agent: Applebot-extended
User-agent: atlassian-bot
User-agent: AutoRAG
User-agent: AwarioSmartBot
User-agent: AzureAI-SearchBot
User-agent: Big Sur AI
User-agent: bigsur.ai
User-agent: Brandwatch
User-agent: Bravebot
User-agent: Brightbot
User-agent: Bytespider
User-agent: CCBot
User-agent: Channel3Bot
User-agent: ChatGLM-Spider
User-agent: ChatGPT-User
User-agent: Claude-Code
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Claude-Web
User-agent: ClaudeBot
User-agent: Code
User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: Cotoyogi
User-agent: Crawl4AI
User-agent: Cursor
User-agent: Datenbank Crawler
User-agent: Datenbank-Crawler
User-agent: DeepSeekBot
User-agent: Devin
User-agent: Diffbot
User-agent: Direqt Anomura
User-agent: DuckAssistBot
User-agent: Echobot Bot
User-agent: ExaBot
User-agent: FacebookBot
User-agent: Factset_spyderbot
User-agent: FirecrawlAgent
User-agent: GeistHaus-PageFetcher
User-agent: GPTBot
User-agent: iAsk
User-agent: iAskBot
User-agent: iaskspider
User-agent: ICC Crawler
User-agent: ICC-Crawler
User-agent: ImageSiftBot
User-agent: imageSpider
User-agent: kagi-fetcher
User-agent: Kangaroo Bot
User-agent: Kangaroo-Bot
User-agent: Kimi-User
User-agent: KlaviyoAIBot
User-agent: Kunato
User-agent: laion-huggingface-processor
User-agent: LCC
User-agent: LINER Bot
User-agent: LinerBot
User-agent: LinkupBot
User-agent: Manus-User
User-agent: meta-externalagent
User-agent: meta-externalfetcher
User-agent: meta-webindexer
User-agent: mistral.ai
User-agent: MistralAI-User
User-agent: netEstate Imprint Crawler
User-agent: NovaAct
User-agent: Novellum
User-agent: Novellum AI Crawl
User-agent: OAI-SearchBot
User-agent: omgili
User-agent: opencode
User-agent: PanguBot
User-agent: PeopleInc-DCipher-Scraper/1.1.0
User-agent: Perplexity-User
User-agent: PerplexityBot
User-agent: PetalBot
User-agent: PhindBot
User-agent: Poggio-Citations
User-agent: QualifiedBot
User-agent: SBIntuitionsBot
User-agent: SeekrBot
User-agent: SemrushBot-OCOB
User-agent: SemrushBotSwa
User-agent: Shap-User
User-agent: ShapBot
User-agent: Spider
User-agent: TavilyBot
User-agent: TerraCotta
User-agent: Timpibot
User-agent: TongyiBot
User-agent: Trae
User-agent: TwinAgent
User-agent: UseAI
User-agent: Velen Crawler
User-agent: VelenPublicWebCrawler
User-agent: Webzio-Extended
User-agent: Wrtn
User-agent: YiyanBot
User-agent: YouBot
User-agent: ZanistaBot
Allow: /home/internet/do-you-need-symmetrical-internet-speeds-what-to-know-before-you-upgrade/
Allow: /home/internet/dont-buy-internet-until-you-check-the-cost-per-mbps/
Allow: /home/internet/equipment-for-fast-home-internet/
Allow: /home/internet/fix-wi-fi-dead-spots-how-to-get-internet-throughout-your-home/
Allow: /home/internet/gigabit-internet-is-it-worth-splurging-for-a-faster-internet-plan/
Allow: /home/internet/how-to-spot-hidden-fees-and-save-money/
Allow: /home/internet/internet-data-caps-explained-avoid-extra-charges-and-make-most-of-your-internet-plan/
Allow: /home/internet/is-your-internet-connection-delivering-on-the-speeds-you-pay-for-heres-how-to-tell/
Allow: /home/internet/lag-free-internet-how-to-find-the-right-internet-speed-for-a-family-of-4-or-more/
Allow: /home/internet/what-are-the-different-internet-connection-types/
Allow: /home/internet/why-your-gigabit-internet-plan-might-be-overkill/
Allow: /home/internet/xfinity-vs-att-which-internet-provider-is-best/
Allow: /home/internet/xfinity-vs-verizon-which-internet-provider-is-best/
Allow: /home/internet/your-internet-bill-might-be-booby-trapped-heres-what-to-do/
Allow: /news/ai-automation-changing-how-we-file-taxes/
Allow: /paid-content/cash-app-families-guide-how-to-set-up-a-teen-or-child-account-step-by-step/
Allow: /paid-content/cash-app-for-teens-safety-and-parental-controls/
Allow: /paid-content/cash-apps-new-kids-debit-card-what-parents-need-to-know/
Allow: /paid-content/cricket-wireless-offers-reliable-prepaid-nationwide-5g/
Allow: /paid-content/dji-builds-on-12-years-of-handheld-imaging-innovation-with-the-dji-osmo-360-ii/
Allow: /paid-content/dji-osmo-pocket-4p-is-a-powerful-tool-for-independent-filmmakers/
Allow: /paid-content/enjoy-unlimited-wi-fi-and-predictable-pricing-with-on-your-home-internet/
Allow: /paid-content/from-tacos-to-tvs-doordash-delivers-your-game-day-essentials/
Allow: /paid-content/galaxy-s26-camera-system-the-biggest-questions-answered-before-you-buy/
Allow: /paid-content/get-an-unlimited-5g-phone-plan-for-25-per-month-with-metro-by-t-mobile/
Allow: /paid-content/get-everything-you-need-for-the-world-cup-delivered-with-doordash/
Allow: /paid-content/h-r-blocks-local-tax-pros-help-you-navigate-complex-returns/
Allow: /paid-content/how-cash-app-builds-savings-habits-for-kids-and-teens/
Allow: /paid-content/how-cash-app-sponsorship-works-adding-removing-and-managing-parental-controls/
Allow: /paid-content/how-families-get-more-with-t-mobile/
Allow: /paid-content/how-fast-should-your-home-internet-be-for-browsing-and-streaming-t-mobile/
Allow: /paid-content/how-galaxy-ai-camera-tools-help-you-take-better-photos-automatically-on-the-s25/
Allow: /paid-content/how-h-r-block-helps-freelancers-and-gig-workers-stay-ahead-of-tax-season/
Allow: /paid-content/how-h-r-block-helps-you-maximize-freelance-and-gig-worker-write-offs/
Allow: /paid-content/how-h-r-block-simplifies-taxes-for-first-time-filers/
Allow: /paid-content/how-h-r-blocks-experts-can-help-you-avoid-common-filing-mistakes/
Allow: /paid-content/how-samsungs-galaxy-ai-boosts-gaming-performance/
Allow: /paid-content/how-sling-is-disrupting-the-old-live-tv-streaming-model/
Allow: /paid-content/how-to-choose-a-reliable-home-internet-provider-in-2026/
Allow: /paid-content/how-to-choose-the-right-dell-ai-laptop-for-your-workflow/

Excerpt: only groups for all user-agents (*) or named AI crawlers, up to 40 rules per group.

Sites with a similar AI policy

• independent.co.uk · 0 blocked
• uber.com · 0 blocked
• itch.io · 0 blocked
• fontawesome.com · 0 blocked
• asus.com · 0 blocked
• ya.ru · 0 blocked
• merriam-webster.com · 0 blocked
• seznam.cz · 0 blocked

All sites that are “open to ai crawlers” →