Does instacart.com block AI crawlers?

Can AI assistants cite instacart.com?

• ChatGPT search: its search crawlers may read the site
• Claude: its search crawler is blocked
• Perplexity: its search crawler is blocked

instacart.com opts out of AI training by OpenAI, Anthropic, Google, Apple, Common Crawl, ByteDance, Meta. Training crawlers and answer crawlers are separate: blocking training doesn't by itself remove a site from AI answers.

AI crawler access, bot by bot

How instacart.com compares

• 19% of the 1,278 popular sites we checked block GPTBot; instacart.com is one of them.
• 18% of the 1,278 popular sites we checked block ClaudeBot; instacart.com is one of them.
• 14% of the 1,278 popular sites we checked block PerplexityBot; instacart.com is one of them.
• Among 197 sites of similar popularity, 19% block at least one AI crawler.

The robots.txt rules that apply to AI crawlers

User-agent: OAI-SearchBot
Allow: /
Disallow: /accounts/unlock
Disallow: /api/
Disallow: /ex/
Disallow: /orders/
Disallow: /store/orders/
Disallow: /redeem_coupon_token
Disallow: /store/order_changes/
Disallow: /store/*/browse_departments*
Disallow: /store/*/ng/l/*
Disallow: /store/referrals*
Disallow: /store/item*
Disallow: /store/meals/*
Disallow: /store/pickup/status/*
Disallow: /store/*/buy_it_again*
Disallow: /store/*/l/*
Disallow: /store/*/meals/*
Disallow: /widgets/*
Disallow: /client-side-redirect/*
Disallow: /track/*
Disallow: /affiliate-program*
Disallow: /family/invite/*
Disallow: /family/accept/*
Disallow: /rate/orders/*
Disallow: /store_manager_admin
Disallow: /rest/v8/
Disallow: /login?next=
Disallow: /*?*interaction_id=
Disallow: /*?*item_detail_product_id=
Disallow: /store/*/collections/*?*sisid=
Disallow: /store/*/*?*xrs_id=

User-agent: PerplexityBot
Disallow: /

User-agent: ChatGPT-User
Allow: /
Allow: /store/brands/l/pads-tampons-liners-wipes
Disallow: /accounts/unlock
Disallow: /api/
Disallow: /ex/
Disallow: /orders/
Disallow: /store/orders/
Disallow: /redeem_coupon_token
Disallow: /store/order_changes/
Disallow: /store/*/browse_departments*
Disallow: /store/*/ng/l/*
Disallow: /store/referrals*
Disallow: /store/item*
Disallow: /store/meals/*
Disallow: /store/pickup/status/*
Disallow: /store/*/buy_it_again*
Disallow: /store/*/l/*
Disallow: /store/*/meals/*
Disallow: /widgets/*
Disallow: /client-side-redirect/*
Disallow: /track/*
Disallow: /affiliate-program*
Disallow: /family/invite/*
Disallow: /family/accept/*
Disallow: /rate/orders/*
Disallow: /store_manager_admin
Disallow: /rest/v8/
Disallow: /login?next=
Disallow: /*?*interaction_id=
Disallow: /*?*item_detail_product_id=
Disallow: /store/*/collections/*?*sisid=
Disallow: /store/*/*?*xrs_id=

User-agent: meta-externalagent
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: *
Disallow: /

Excerpt: only groups for all user-agents (*) or named AI crawlers, up to 40 rules per group.

Sites with a similar AI policy

• umd.edu · 3 blocked
• congress.gov · 11 blocked
• inc.com · 6 blocked
• co.com · 12 blocked
• mos.ru · 4 blocked
• guardian.co.uk · 8 blocked
• nejm.org · 5 blocked
• rollingstone.com · 10 blocked

All sites that are “blocks ai training and ai answers” →