Does mos.ru block AI crawlers?

Can AI assistants cite mos.ru?

• ChatGPT search: its search crawler is blocked
• Claude: its search crawlers may read the site
• Perplexity: its search crawlers may read the site

mos.ru opts out of AI training by OpenAI, Common Crawl, ByteDance. Training crawlers and answer crawlers are separate: blocking training doesn't by itself remove a site from AI answers.

AI crawler access, bot by bot

How mos.ru compares

• 19% of the 1,278 popular sites we checked block GPTBot; mos.ru is one of them.
• 18% of the 1,278 popular sites we checked block ClaudeBot; mos.ru does not.
• 14% of the 1,278 popular sites we checked block PerplexityBot; mos.ru does not.
• Among 198 sites of similar popularity, 20% block at least one AI crawler.

The robots.txt rules that apply to AI crawlers

User-agent: anthropic-ai
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: *
Allow: */event/337206257/
Allow: */event/337208257/
Allow: */event/337215257/
Allow: */event/337219257/
Allow: */event/337224257/
Allow: */event/337226257/
Allow: */event/337239257/
Allow: */event/337240257/
Allow: */event/337241257/
Allow: */event/127001257/
Disallow: /?*
Disallow: */*&action=*
Disallow: */*&print=*
Disallow: */*/district=*
Disallow: */*/pgu=*
Disallow: */*?action=*
Disallow: */*?print=*
Disallow: */*?utm_source=*
Disallow: */*ADD_TO_COMPARE_LIST*
Disallow: */*adfox*
Disallow: */*auth=*
Disallow: */*back_url_admin=*
Disallow: */*back_url=*
Disallow: */*BACK_URL=*
Disallow: */*backurl=*
Disallow: */*BACKURL=*
Disallow: */*bitrix_include_areas=*
Disallow: */*change_password=*
Disallow: */*clear_cache_session=*
Disallow: */*clear_cache=*
Disallow: */*forgot_password=*
Disallow: */*login=*
Disallow: */*logout=*
Disallow: */*ORDER_BY*
Disallow: */*PAGEN*
Disallow: */*print_course=*
Disallow: */*register=*
Disallow: */*search=*
Disallow: */*show_include_exec_time=*
Disallow: */*show_page_exec_time=*

Excerpt: only groups for all user-agents (*) or named AI crawlers, up to 40 rules per group.

Sites with a similar AI policy

• guardian.co.uk · 8 blocked
• nejm.org · 5 blocked
• co.com · 12 blocked
• rollingstone.com · 10 blocked
• inc.com · 6 blocked
• nba.com · 7 blocked
• redditmedia.com · 12 blocked
• congress.gov · 11 blocked

All sites that are “blocks ai training and ai answers” →