Does pexels.com block AI crawlers?

Can AI assistants cite pexels.com?

• ChatGPT search: its search crawlers may read the site
• Claude: its search crawlers may read the site
• Perplexity: its search crawler is blocked

pexels.com opts out of AI training by OpenAI, Anthropic, Google, Apple, Common Crawl, ByteDance. Training crawlers and answer crawlers are separate: blocking training doesn't by itself remove a site from AI answers.

AI crawler access, bot by bot

How pexels.com compares

• 19% of the 1,278 popular sites we checked block GPTBot; pexels.com is one of them.
• 18% of the 1,278 popular sites we checked block ClaudeBot; pexels.com is one of them.
• 14% of the 1,278 popular sites we checked block PerplexityBot; pexels.com is one of them.
• Among 199 sites of similar popularity, 30% block at least one AI crawler.

The robots.txt rules that apply to AI crawlers

User-agent: *
Disallow: */likes
Disallow: *orientation=*
Disallow: *size=*
Disallow: *color=*
Disallow: *people_count=*
Disallow: *people_age=*
Disallow: *max_width=*
Disallow: *min_width=*
Disallow: *max_height=*
Disallow: *min_height=*
Disallow: *date_from=*
Disallow: *min_duration=*
Disallow: *max_duration=*
Disallow: *frame_rate=*
Disallow: *resolution_name=*
Disallow: *q=*
Disallow: *page=21
Disallow: *page=22
Disallow: *page=23
Disallow: *page=24
Disallow: *page=25
Disallow: */users/*
Disallow: /@*/followers/*
Disallow: /*/@*/followers/*
Disallow: /@*/following/*
Disallow: /*/@*/following/*
Allow: /sitemaps/
Disallow: */download/*
Disallow: */donate/*
Disallow: */report/*
Disallow: */api/v*
Disallow: /*ｃｏｍ
Disallow: /*ｖｉｐ
Disallow: /*ｃｃ
Disallow: /*.com
Disallow: /*.vip
Disallow: /*.cc
Disallow: /*.tw
Disallow: /*.hk
Disallow: /*.co

User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /

User-agent: Applebot-Extended
User-agent: GPTBot
User-agent: Google-Extended
User-agent: Bytespider
User-agent: ClaudeBot
User-agent: CCBot
User-agent: PerplexityBot
Disallow: /

Excerpt: only groups for all user-agents (*) or named AI crawlers, up to 40 rules per group.

Sites with a similar AI policy

• daum.net · 12 blocked
• lefigaro.fr · 8 blocked
• adform.net · 12 blocked
• people.com · 9 blocked
• nbcnews.com · 12 blocked
• welt.de · 4 blocked
• apnews.com · 7 blocked
• mercadolivre.com.br · 5 blocked

All sites that are “blocks ai training and ai answers” →