Does dns-shop.ru block AI crawlers?
Open to AI crawlers
dns-shop.ru doesn't block any of the major AI crawlers we track in its robots.txt.
Checked from dns-shop.ru/robots.txt. Robots rules change; the live file is the source of truth.
- AI crawlers blocked
- 0 of 12
- AI bots named in robots.txt
- 0
- llms.txt
- Published
- Popularity
- Tranco top 2,000 (#1,624)
Can AI assistants cite dns-shop.ru?
- ChatGPT search: its search crawlers may read the site
- Claude: its search crawlers may read the site
- Perplexity: its search crawlers may read the site
dns-shop.ru doesn't opt out of AI model training through robots.txt. Training crawlers and answer crawlers are separate: blocking training doesn't by itself remove a site from AI answers.
AI crawler access, bot by bot
| Crawler | Used for | Status on dns-shop.ru |
|---|---|---|
| GPTBotOpenAI | AI training | Some paths restricted |
| OAI-SearchBotOpenAI | AI answers | Some paths restricted |
| ChatGPT-UserOpenAI | AI answers | Some paths restricted |
| ClaudeBotAnthropic | AI training | Some paths restricted |
| Claude-SearchBotAnthropic | AI answers | Some paths restricted |
| Claude-UserAnthropic | AI answers | Some paths restricted |
| PerplexityBotPerplexity | AI answers | Some paths restricted |
| Google-ExtendedGoogle | AI training | Some paths restricted |
| Applebot-ExtendedApple | AI training | Some paths restricted |
| CCBotCommon Crawl | AI training | Some paths restricted |
| BytespiderByteDance | AI training | Some paths restricted |
| Meta-ExternalAgentMeta | AI training | Some paths restricted |
How dns-shop.ru compares
- 19% of the 1,278 popular sites we checked block GPTBot; dns-shop.ru does not.
- 18% of the 1,278 popular sites we checked block ClaudeBot; dns-shop.ru does not.
- 14% of the 1,278 popular sites we checked block PerplexityBot; dns-shop.ru does not.
- Among 151 sites of similar popularity, 18% block at least one AI crawler.
The robots.txt rules that apply to AI crawlers
User-agent: * Allow: / Allow: /*?city Allow: /*?page= Allow: /*utm_source=google Allow: /assets/*.css Allow: /assets/*.js Allow: /catalog/markdown/$ Allow: /configurator/$ Allow: /custompc/$ Allow: /custompc/user-pc/$ Allow: /product/accessories/* Allow: /product/analog/* Allow: /product/characteristics/* Allow: /product/opinion/* Allow: /product/service-rating/* Allow: /product/similar-design/* Disallow: /*.woff2 Disallow: /*? Disallow: /actionMarketing/ Disallow: /actionsOnTV/ Disallow: /cart/ Disallow: /catalog/actions/ Disallow: /catalog/collection/* Disallow: /catalog/markdown/* Disallow: /check-invoice/ Disallow: /checkout/ Disallow: /configurator/* Disallow: /custompc/ Disallow: /custompc/user-pc/configuration/* Disallow: /mobile-app/ Disallow: /opinion-page/ Disallow: /order/ Disallow: /preorder/ Disallow: /product/*/*/description Disallow: /profile/feedback/ Disallow: /profile/mail-change/ Disallow: /search/ Disallow: /similar-design/ Disallow: /survey/checkout-order/* Disallow: /survey/delivery/*
Excerpt: only groups for all user-agents (*) or named AI crawlers, up to 40 rules per group.
Sites with a similar AI policy
Does dns-shop.ru block GPTBot?
No. As of Oct 2, 2026, dns-shop.ru's robots.txt doesn't block GPTBot, though some paths are restricted for all crawlers.
Can ChatGPT search show results from dns-shop.ru?
Its robots.txt doesn't block OAI-SearchBot or ChatGPT-User, so ChatGPT's search crawlers may read it. Whether pages are actually shown depends on ChatGPT.
Does dns-shop.ru have an llms.txt file?
Yes — dns-shop.ru/llms.txt was found when we checked.
Data: dns-shop.ru's public robots.txt and llms.txt, fetched by AskableHQ. Popularity rank from the Tranco list (ID Y83YG). Methodology.