Does rightmove.co.uk block AI crawlers?

Can AI assistants cite rightmove.co.uk?

• ChatGPT search: its search crawlers may read the site
• Claude: its search crawlers may read the site
• Perplexity: its search crawlers may read the site

rightmove.co.uk opts out of AI training by OpenAI, Common Crawl. Training crawlers and answer crawlers are separate: blocking training doesn't by itself remove a site from AI answers.

AI crawler access, bot by bot

How rightmove.co.uk compares

• 19% of the 1,278 popular sites we checked block GPTBot; rightmove.co.uk is one of them.
• 18% of the 1,278 popular sites we checked block ClaudeBot; rightmove.co.uk does not.
• 14% of the 1,278 popular sites we checked block PerplexityBot; rightmove.co.uk does not.
• Among 199 sites of similar popularity, 29% block at least one AI crawler.

The robots.txt rules that apply to AI crawlers

User-agent: *
Disallow: /*/fullscreen/image-gallery.html*
Disallow: /*/fullscreen/view-*
Disallow: /*/mortgage-calculator/*
Disallow: /*/save-search.html*
Disallow: /addtoshortlist.html*
Disallow: /api/*
Disallow: /branchfirstviewlog
Disallow: /branchfirstviewlog*
Disallow: /change-alert-frequency.html*
Disallow: /commercial-property-for-sale/contactBranch.html*
Disallow: /commercial-property-for-sale/draw-a-search.html?*
Disallow: /commercial-property-for-sale/full-description/*
Disallow: /commercial-property-for-sale/map-view/*
Disallow: /commercial-property-for-sale/map.html?*
Disallow: /commercial-property-for-sale/photos/*
Disallow: /commercial-property-for-sale/street-view/*
Disallow: /commercial-property-to-let/contactBranch.html*
Disallow: /commercial-property-to-let/draw-a-search.html?*
Disallow: /commercial-property-to-let/full-description/*
Disallow: /commercial-property-to-let/map-view/*
Disallow: /commercial-property-to-let/map.html?*
Disallow: /commercial-property-to-let/photos/*
Disallow: /commercial-property-to-let/street-view/*
Disallow: /contact-form/viewing*
Disallow: /contactBranch.html*
Disallow: /draw-a-search.html?*
Disallow: /feedback/feedback.html*
Disallow: /home-ideas/advertiser/redirect.html*
Disallow: /home-ideas/collection.html*
Disallow: /home-ideas/collections.html*
Disallow: /home-ideas/detail.html*
Disallow: /home-ideas/photo.html*
Disallow: /home-ideas/profile.html*
Disallow: /home-ideas/search.html*
Disallow: /home-ideas/search/*
Disallow: /home-ideas/tagging.html*
Disallow: /home-services/broadband/results*
Disallow: /house-prices/detailMatching.html*
Disallow: /house-prices/details/*
Disallow: /house-value.html?searchLocation*

User-agent: GPTBot
Disallow: /
Allow: /mortgages/
Allow: /mortgages/mortgage-in-principle
Allow: /mortgages/calculators/mortgage-calculator/
Allow: /mortgages/calculators/remortgage-calculator/
Disallow: /mortgages/mortgage-in-principle?*

User-agent: CCbot
Disallow: /

Excerpt: only groups for all user-agents (*) or named AI crawlers, up to 40 rules per group.

Sites with a similar AI policy

• coursera.org · 1 blocked
• pandora.com · 7 blocked
• gazzetta.it · 6 blocked
• leboncoin.fr · 1 blocked
• vnexpress.net · 6 blocked
• indiamart.com · 2 blocked
• mundodeportivo.com · 1 blocked
• livescore.com · 1 blocked

All sites that are “blocks ai training, allows ai answers” →