Does tandfonline.com block AI crawlers?

Can AI assistants cite tandfonline.com?

• ChatGPT search: its search crawlers may read the site
• Claude: its search crawlers may read the site
• Perplexity: its search crawlers may read the site

tandfonline.com opts out of AI training by OpenAI. Training crawlers and answer crawlers are separate: blocking training doesn't by itself remove a site from AI answers.

AI crawler access, bot by bot

How tandfonline.com compares

• 19% of the 1,278 popular sites we checked block GPTBot; tandfonline.com is one of them.
• 18% of the 1,278 popular sites we checked block ClaudeBot; tandfonline.com does not.
• 14% of the 1,278 popular sites we checked block PerplexityBot; tandfonline.com does not.
• Among 204 sites of similar popularity, 29% block at least one AI crawler.

The robots.txt rules that apply to AI crawlers

User-agent: *
Disallow: /action
Disallow: /help
Disallow: /search
Disallow: /feedback
Disallow: /rss
Disallow: /page/account-confirmation-thanks
Disallow: /media
Disallow: /medical-research
Disallow: /servlet/linkout
Disallow: /na101/
Disallow: /na101v1/
Disallow: /na102/
Disallow: /doi/mlt/
Disallow: /topic
Disallow: /author/
Disallow: /doi/metrics/
Disallow: /keyword/
Disallow: /action/showPublications*ConceptID=*
Disallow: /action/showPublications*alphabetRange=*
Disallow: /doi/epdf/*
Disallow: /pricing/*
Disallow: /doi/epub/*
Disallow: /authored-by/
Disallow: /history/
Allow: /action/showPublications
Allow: /action/showXml
Allow: /action/showCoverImage
Allow: /action/showOpenGraphCoverImage
Allow: /action/showGraphicalAbstractImage
Allow: /doi/cover-img
Allow: /action/journalInformation*aimsScope*
Allow: /action/journalInformation*editorialBoard*
Allow: /action/journalInformation*journalMetrics*
Allow: /topic/articlecollection
Allow: /topic/sections
Allow: /action/journalInformation*societyInformation*
Allow: /journals/*/about-this-journal
Allow: /subjects/
Allow: /action/showAxaArticles?journalCode=*
Allow: /.well-known/tdmrep.json

User-agent: GPTBot
Disallow: /

Excerpt: only groups for all user-agents (*) or named AI crawlers, up to 40 rules per group.

Sites with a similar AI policy

• slideshare.net · 3 blocked
• tripadvisor.com · 6 blocked
• behance.net · 7 blocked
• scribd.com · 3 blocked
• sagepub.com · 3 blocked
• goodreads.com · 2 blocked
• pixabay.com · 6 blocked
• aol.com · 5 blocked

All sites that are “blocks ai training, allows ai answers” →