Does hse.ru block AI crawlers?

Can AI assistants cite hse.ru?

• ChatGPT search: its search crawlers may read the site
• Claude: its search crawlers may read the site
• Perplexity: its search crawlers may read the site

hse.ru opts out of AI training by OpenAI. Training crawlers and answer crawlers are separate: blocking training doesn't by itself remove a site from AI answers.

AI crawler access, bot by bot

How hse.ru compares

• 19% of the 1,278 popular sites we checked block GPTBot; hse.ru is one of them.
• 18% of the 1,278 popular sites we checked block ClaudeBot; hse.ru does not.
• 14% of the 1,278 popular sites we checked block PerplexityBot; hse.ru does not.
• Among 110 sites of similar popularity, 25% block at least one AI crawler.

The robots.txt rules that apply to AI crawlers

User-agent: *
Disallow: */mmo/diplomas/*
Disallow: */db/
Disallow: */search/
Disallow: */captcha/
Disallow: /*empty.html
Disallow: */reg
Disallow: */user/
Disallow: */adm/
Disallow: */text/image
Disallow: */co-auth/
Disallow: */schoolstat/
Disallow: */pubs/lib/ajax/
Disallow: */mirror/pubs/lib/ajax/
Disallow: */lib/ajax/
Disallow: */mirror/lib/ajax/
Disallow: */ajax/
Disallow: */announcement/new/
Disallow: */data/xf/
Disallow: /pubs/?category
Disallow: */expresspolls/poll/
Disallow: /video/*?*
Disallow: */news/archive/*
Disallow: /?_r*
Disallow: /org/persons/?elib=*
Disallow: /n/api/counter
Disallow: /en/docs/
Disallow: /api/ep-passport/schedule/*
Disallow: *?_ga*
Disallow: *?search_person*
Disallow: */n/web/playground*
Disallow: */n/web/playground/*
Disallow: */?utm_source=
Disallow: */openedu/api/*
Disallow: /*print=
Disallow: /*v=
Disallow: /*continueFlag=
Disallow: /*kw=
Disallow: /*tab=
Disallow: /*_anchor=
Disallow: /*r=

User-agent: GPTBot
Disallow: /

Excerpt: only groups for all user-agents (*) or named AI crawlers, up to 40 rules per group.

Sites with a similar AI policy

• vietnam.vn · 6 blocked
• ebsco.com · 7 blocked
• 123rf.com · 3 blocked
• flightaware.com · 1 blocked
• postman.com · 1 blocked
• tagesspiegel.de · 6 blocked
• razorpay.com · 1 blocked
• dictionary.com · 3 blocked

All sites that are “blocks ai training, allows ai answers” →