Does netangels.ru block AI crawlers?

Can AI assistants cite netangels.ru?

• ChatGPT search: its search crawlers may read the site
• Claude: its search crawlers may read the site
• Perplexity: its search crawlers may read the site

netangels.ru doesn't opt out of AI model training through robots.txt. Training crawlers and answer crawlers are separate: blocking training doesn't by itself remove a site from AI answers.

AI crawler access, bot by bot

How netangels.ru compares

• 19% of the 1,278 popular sites we checked block GPTBot; netangels.ru does not.
• 18% of the 1,278 popular sites we checked block ClaudeBot; netangels.ru does not.
• 14% of the 1,278 popular sites we checked block PerplexityBot; netangels.ru does not.
• Among 196 sites of similar popularity, 30% block at least one AI crawler.

The robots.txt rules that apply to AI crawlers

User-agent: *
Disallow: */register/
Disallow: */order/
Disallow: */auth/
Disallow: /out/
Disallow: /static/
Disallow: /support/hosting-old/
Disallow: *?what=*
Disallow: */?p_ref=*
Disallow: *?q=*
Disallow: *?search=*
Disallow: *?modal=*
Disallow: *?__proto__*
Disallow: *?constructor*
Disallow: *?rest_route=*
Disallow: *?_action=*
Disallow: *?email=*
Disallow: *?ignorethis=*
Disallow: *?KEY1*
Disallow: *?K1*
Disallow: *?%D0%B5%D1%83=*
Disallow: *?_id=*
Disallow: *?bx_sender_conversion_id=*
Disallow: *?2VSb5wRQEs3*
Disallow: *?2VSb5wUNDRj*
Disallow: *?2VSb5wdG97o*
Disallow: *?yprqee=*
Allow: /cloud/?utm_
Allow: /cloud/?gclid
Allow: /cloud/?yclid
Allow: /cloud/?from
Allow: /cloud/?openstat
Disallow: /cloud/?*

Excerpt: only groups for all user-agents (*) or named AI crawlers, up to 40 rules per group.

Sites with a similar AI policy

• tp-link.com · 0 blocked
• stanford.edu · 0 blocked
• issuu.com · 0 blocked
• nvidia.com · 0 blocked
• zendesk.com · 0 blocked
• datadoghq.com · 0 blocked
• nasa.gov · 0 blocked
• plesk.com · 0 blocked

All sites that are “open to ai crawlers” →