Does gazzetta.it block AI crawlers?

Can AI assistants cite gazzetta.it?

• ChatGPT search: its search crawlers may read the site
• Claude: its search crawlers may read the site
• Perplexity: its search crawlers may read the site

gazzetta.it opts out of AI training by OpenAI, Anthropic, Google, Apple, Common Crawl, Meta. Training crawlers and answer crawlers are separate: blocking training doesn't by itself remove a site from AI answers.

AI crawler access, bot by bot

How gazzetta.it compares

• 19% of the 1,278 popular sites we checked block GPTBot; gazzetta.it is one of them.
• 18% of the 1,278 popular sites we checked block ClaudeBot; gazzetta.it is one of them.
• 14% of the 1,278 popular sites we checked block PerplexityBot; gazzetta.it does not.
• Among 198 sites of similar popularity, 29% block at least one AI crawler.

The robots.txt rules that apply to AI crawlers

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: *
Disallow: /_Template/
Disallow: /gazzastar/
Disallow: /prova/
Disallow: /tmp/
Disallow: /test/
Disallow: /doc/
Disallow: /test_akamai/
Disallow: /test_space/
Disallow: /test_tline/
Disallow: /dilatuagaz/
Disallow: /community/utenti/
Disallow: /cms_sport_data/speciali/risultati_classifiche/calcio/giocatori/includes/
Disallow: /ajaxsc/
Disallow: /cf/
Disallow: /commento/
Disallow: /dyn/
Disallow: /haberler/
Disallow: /passport/
Disallow: /openxlink.shtml?*
Disallow: /temporeale/flash/radiocorsa/
Disallow: /sportlocale-risultati/
Disallow: /speciali/automobilismo/2007/
Disallow: /speciali/automobilismo/2006/
Disallow: /matchcenter/app/
Disallow: /codici-sconto/user/*
Disallow: /codici-sconto/ajax/*
Disallow: /codici-sconto/retailer/index/hiw
Disallow: /codici-sconto/clickout/*
Disallow: /codici-sconto/gotoshop/*
Disallow: /codici-sconto/show/
Disallow: /*undefined
Disallow: /*(none)
Disallow: /_templates/
Disallow: *_commenti.shtml
Disallow: /gallery/
Disallow: /gallery-includes/
Disallow: /primi_piani/
Disallow: /avviso/
Disallow: /foto_del_giorno/
Disallow: /premium/plus/

Excerpt: only groups for all user-agents (*) or named AI crawlers, up to 40 rules per group.

Sites with a similar AI policy

• pandora.com · 7 blocked
• rightmove.co.uk · 2 blocked
• coursera.org · 1 blocked
• mundodeportivo.com · 1 blocked
• livescore.com · 1 blocked
• leboncoin.fr · 1 blocked
• litnet.com · 1 blocked
• vnexpress.net · 6 blocked

All sites that are “blocks ai training, allows ai answers” →