Skip to content
AskableHQ
Launch

Does rfc-editor.org block AI crawlers?

Open to AI crawlers

rfc-editor.org doesn't block any of the major AI crawlers we track in its robots.txt.

Checked from rfc-editor.org/robots.txt. Robots rules change; the live file is the source of truth.

AI crawlers blocked
0 of 12
AI bots named in robots.txt
0
llms.txt
Not found
Popularity
Tranco top 1,000 (#989)

Can AI assistants cite rfc-editor.org?

  • ChatGPT search: its search crawlers may read the site
  • Claude: its search crawlers may read the site
  • Perplexity: its search crawlers may read the site

rfc-editor.org doesn't opt out of AI model training through robots.txt. Training crawlers and answer crawlers are separate: blocking training doesn't by itself remove a site from AI answers.

AI crawler access, bot by bot

CrawlerUsed forStatus on rfc-editor.org
GPTBotOpenAIAI trainingSome paths restricted
OAI-SearchBotOpenAIAI answersSome paths restricted
ChatGPT-UserOpenAIAI answersSome paths restricted
ClaudeBotAnthropicAI trainingSome paths restricted
Claude-SearchBotAnthropicAI answersSome paths restricted
Claude-UserAnthropicAI answersSome paths restricted
PerplexityBotPerplexityAI answersSome paths restricted
Google-ExtendedGoogleAI trainingSome paths restricted
Applebot-ExtendedAppleAI trainingSome paths restricted
CCBotCommon CrawlAI trainingSome paths restricted
BytespiderByteDanceAI trainingSome paths restricted
Meta-ExternalAgentMetaAI trainingSome paths restricted

How rfc-editor.org compares

  • 19% of the 1,278 popular sites we checked block GPTBot; rfc-editor.org does not.
  • 18% of the 1,278 popular sites we checked block ClaudeBot; rfc-editor.org does not.
  • 14% of the 1,278 popular sites we checked block PerplexityBot; rfc-editor.org does not.
  • Among 196 sites of similar popularity, 32% block at least one AI crawler.

The robots.txt rules that apply to AI crawlers

User-agent: *
Disallow: /rfc/authors/
Disallow: /rfc/rerendered/
Disallow: /rfc/v3test/

Excerpt: only groups for all user-agents (*) or named AI crawlers, up to 40 rules per group.

Sites with a similar AI policy

All sites that are “open to ai crawlers” →

Does rfc-editor.org block GPTBot?

No. As of Oct 2, 2026, rfc-editor.org's robots.txt doesn't block GPTBot, though some paths are restricted for all crawlers.

Can ChatGPT search show results from rfc-editor.org?

Its robots.txt doesn't block OAI-SearchBot or ChatGPT-User, so ChatGPT's search crawlers may read it. Whether pages are actually shown depends on ChatGPT.

Does rfc-editor.org have an llms.txt file?

We didn't find a valid llms.txt at rfc-editor.org/llms.txt when we checked.

Data: rfc-editor.org's public robots.txt and llms.txt, fetched by AskableHQ. Popularity rank from the Tranco list (ID Y83YG). Methodology.