Does glassdoor.com block AI crawlers?

Can AI assistants cite glassdoor.com?

• ChatGPT search: its search crawlers may read the site
• Claude: its search crawlers may read the site
• Perplexity: its search crawlers may read the site

glassdoor.com opts out of AI training by OpenAI, Anthropic, Google, Apple, Common Crawl, ByteDance. Training crawlers and answer crawlers are separate: blocking training doesn't by itself remove a site from AI answers.

AI crawler access, bot by bot

How glassdoor.com compares

• 19% of the 1,278 popular sites we checked block GPTBot; glassdoor.com is one of them.
• 18% of the 1,278 popular sites we checked block ClaudeBot; glassdoor.com is one of them.
• 14% of the 1,278 popular sites we checked block PerplexityBot; glassdoor.com does not.
• Among 171 sites of similar popularity, 24% block at least one AI crawler.

The robots.txt rules that apply to AI crawlers

User-agent: *
Disallow: /*?*hostSite=*
Disallow: /1347171559/
Disallow: /about/board/
Disallow: /about/contact/
Disallow: /about/faq/
Disallow: /about/forCareerCenters/
Disallow: /about/forLibraries/
Disallow: /about/forStudents/
Disallow: /about/guidelines/
Disallow: /about/index/
Disallow: /about/jobs/
Disallow: /about/learn/
Disallow: /about/overview/
Disallow: /about/privacy/
Disallow: /about/syndicationCenter/
Disallow: /about/team/
Disallow: /about/terms/
Disallow: /about/widgetTerms/
Disallow: /ajax/
Disallow: /abtest
Disallow: /browse/
Disallow: /chat
Disallow: /Compare/choose
Disallow: /employerinfo/
Disallow: /employerInfo/
Disallow: /Explore/browse-companies
Disallow: /home/
Disallow: /integrations/facebook/glassdoor/eep
Disallow: /jobview/
Disallow: /legal/
Disallow: /lists/
Disallow: /more/
Disallow: /partner/
Disallow: /partner-center/
Disallow: /partners/company/
Disallow: /partners/insights/
Disallow: /partners/jobs/
Disallow: /partners/reports/
Disallow: /partners/resumeView
Disallow: /partners/settings/

User-agent: GPTBot
User-agent: Google-Extended
User-agent: Amazonbot
User-agent: anthropic-ai
User-agent: GoogleOther
User-agent: Claude-Web
User-agent: ClaudeBot
User-agent: Perplexity
User-agent: Cohere
User-agent: cohere-ai
User-agent: Applebot-Extended
User-agent: Google-CloudVertexBot
Disallow: /
Allow: /blog/
Allow: /Award/
Allow: /About/
Allow: /employers/
Disallow: /employers/ec
Disallow: /employers/engagement/api/
Disallow: /employers/enhanced/billing_*

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

Excerpt: only groups for all user-agents (*) or named AI crawlers, up to 40 rules per group.

Sites with a similar AI policy

• realtor.com · 1 blocked
• mastodon.social · 1 blocked
• venturebeat.com · 3 blocked
• prezi.com · 2 blocked
• aarp.org · 2 blocked
• telstra.com · 2 blocked
• envato.com · 1 blocked
• livescience.com · 2 blocked

All sites that are “blocks ai training, allows ai answers” →