Skip to content
AskableHQ
Launch

Does nu.nl block AI crawlers?

Blocks AI training and AI answers

nu.nl blocks both AI training crawlers and AI answer crawlers in its robots.txt.

Checked from nu.nl/robots.txt. Robots rules change; the live file is the source of truth.

AI crawlers blocked
9 of 12
AI bots named in robots.txt
10
llms.txt
Not found
Popularity
Tranco top 2,000 (#1,183)

Can AI assistants cite nu.nl?

  • ChatGPT search: its search crawler is blocked
  • Claude: its search crawlers may read the site
  • Perplexity: its search crawler is blocked

nu.nl opts out of AI training by OpenAI, Anthropic, Google, Apple, Common Crawl, ByteDance. Training crawlers and answer crawlers are separate: blocking training doesn't by itself remove a site from AI answers.

AI crawler access, bot by bot

CrawlerUsed forStatus on nu.nl
GPTBotOpenAIAI trainingBlocked
OAI-SearchBotOpenAIAI answersBlocked
ChatGPT-UserOpenAIAI answersBlocked
ClaudeBotAnthropicAI trainingBlocked
Claude-SearchBotAnthropicAI answersSome paths restricted
Claude-UserAnthropicAI answersSome paths restricted
PerplexityBotPerplexityAI answersBlocked
Google-ExtendedGoogleAI trainingBlocked
Applebot-ExtendedAppleAI trainingBlocked
CCBotCommon CrawlAI trainingBlocked
BytespiderByteDanceAI trainingBlocked
Meta-ExternalAgentMetaAI trainingSome paths restricted

How nu.nl compares

  • 19% of the 1,278 popular sites we checked block GPTBot; nu.nl is one of them.
  • 18% of the 1,278 popular sites we checked block ClaudeBot; nu.nl is one of them.
  • 14% of the 1,278 popular sites we checked block PerplexityBot; nu.nl is one of them.
  • Among 210 sites of similar popularity, 30% block at least one AI crawler.

The robots.txt rules that apply to AI crawlers

User-agent: *
Disallow: /api/
Disallow: /blocks/
Disallow: /component/
Disallow: /components/
Disallow: /zoeken
Disallow: /*redirect=1*

User-agent: AhrefsBot
User-agent: Amazonbot
User-agent: anthropic-ai
User-agent: Applebot-Extended
User-agent: Bytespider
User-agent: CCBot
User-agent: ChatGPT-User
User-agent: Claude
User-agent: Claude-Web
User-agent: ClaudeBot
User-agent: coccocbot-web
User-agent: cohere-ai
User-agent: DeepSeekBot
User-agent: Diffbot
User-agent: Exabot
User-agent: Google-Extended
User-agent: GPTBot
User-agent: Meta-ExternalFetcher
User-agent: MistralAI-User
User-agent: MistralAI-User/1.0
User-agent: MJ12bot
User-agent: OAI-SearchBot
User-agent: Perplexity-User
User-agent: PerplexityBot
User-agent: PetalBot
User-agent: proximic
User-agent: SemrushBot
User-agent: SemrushBot-OCOB
User-agent: SemrushBot-SWA
User-agent: Timpibot
User-agent: Yandex
User-agent: YandexAdditional
User-agent: YandexAdditionalBot
User-agent: YandexBot
User-agent: YandexImages
User-agent: YandexRenderBot
User-agent: YouBot
Disallow: /

Excerpt: only groups for all user-agents (*) or named AI crawlers, up to 40 rules per group.

Sites with a similar AI policy

All sites that are “blocks ai training and ai answers” →

Does nu.nl block GPTBot?

Yes. As of Oct 2, 2026, nu.nl's robots.txt disallows GPTBot, OpenAI's training crawler.

Can ChatGPT search show results from nu.nl?

Its robots.txt blocks a ChatGPT search crawler, so ChatGPT search is unlikely to read or cite those pages.

Does nu.nl have an llms.txt file?

We didn't find a valid llms.txt at nu.nl/llms.txt when we checked.

Data: nu.nl's public robots.txt and llms.txt, fetched by AskableHQ. Popularity rank from the Tranco list (ID Y83YG). Methodology.