Does usatoday.com block AI crawlers?
Blocks AI training and AI answers
usatoday.com blocks both AI training crawlers and AI answer crawlers in its robots.txt.
Checked from usatoday.com/robots.txt. Robots rules change; the live file is the source of truth.
- AI crawlers blocked
- 12 of 12
- AI bots named in robots.txt
- 13
- llms.txt
- Not found
- Popularity
- Tranco top 1,000 (#598)
Can AI assistants cite usatoday.com?
- ChatGPT search: its search crawler is blocked
- Claude: its search crawler is blocked
- Perplexity: its search crawler is blocked
usatoday.com opts out of AI training by OpenAI, Anthropic, Google, Apple, Common Crawl, ByteDance, Meta. Training crawlers and answer crawlers are separate: blocking training doesn't by itself remove a site from AI answers.
AI crawler access, bot by bot
| Crawler | Used for | Status on usatoday.com |
|---|---|---|
| GPTBotOpenAI | AI training | Blocked |
| OAI-SearchBotOpenAI | AI answers | Blocked |
| ChatGPT-UserOpenAI | AI answers | Blocked |
| ClaudeBotAnthropic | AI training | Blocked |
| Claude-SearchBotAnthropic | AI answers | Blocked |
| Claude-UserAnthropic | AI answers | Blocked |
| PerplexityBotPerplexity | AI answers | Blocked |
| Google-ExtendedGoogle | AI training | Blocked |
| Applebot-ExtendedApple | AI training | Blocked |
| CCBotCommon Crawl | AI training | Blocked |
| BytespiderByteDance | AI training | Blocked |
| Meta-ExternalAgentMeta | AI training | Blocked |
How usatoday.com compares
- 19% of the 1,278 popular sites we checked block GPTBot; usatoday.com is one of them.
- 18% of the 1,278 popular sites we checked block ClaudeBot; usatoday.com is one of them.
- 14% of the 1,278 popular sites we checked block PerplexityBot; usatoday.com is one of them.
- Among 202 sites of similar popularity, 28% block at least one AI crawler.
The robots.txt rules that apply to AI crawlers
User-agent: anthropic-ai Disallow: / User-agent: ClaudeBot Disallow: / User-agent: ChatGPT-User Disallow: / Allow: /branded-story/ Allow: /contributor-content/ Allow: /story/branded-story/ Allow: /story/special/contributor-content/ Allow: /money/money-management/ Allow: /shopping/ Allow: /story/shopping/ Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ User-agent: GPTBot Disallow: / Allow: /branded-story/ Allow: /contributor-content/ Allow: /story/branded-story/ Allow: /story/special/contributor-content/ Allow: /money/money-management/ Allow: /shopping/ Allow: /story/shopping/ Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / Allow: /branded-story/ Allow: /contributor-content/ Allow: /story/branded-story/ Allow: /story/special/contributor-content/ Allow: /money/money-management/ Allow: /shopping/ Allow: /story/shopping/ Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ User-agent: PerplexityBot Disallow: / Allow: /branded-story/ Allow: /contributor-content/ Allow: /story/branded-story/ Allow: /story/special/contributor-content/ Allow: /money/money-management/ Allow: /shopping/ Allow: /story/shopping/ Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ User-agent: Applebot-Extended Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: meta-externalagent Disallow: / User-agent: OAI-SearchBot Disallow: / Allow: /branded-story/ Allow: /contributor-content/ Allow: /story/branded-story/ Allow: /story/special/contributor-content/ Allow: /money/money-management/ Allow: /shopping/ Allow: /story/shopping/ Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ User-agent: Bytespider Disallow: / User-agent: Claude-User Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: * Disallow: /errors Disallow: /interactive/ Disallow: /userauth/ Disallow: /ugc/ Disallow: /feeds/ Disallow: /services/ Disallow: /facebook/ Disallow: /version-info/ Disallow: /longform/draft/ Disallow: /story/draft/ Disallow: /topic/*/smart/ Disallow: /search Disallow: /module-showcase/ Disallow: /newsletter/ Disallow: /blended-newsletter/ Disallow: /story/nletter/ Disallow: /sports/services/photos/ Disallow: /optimus Disallow: /ux-train Disallow: /story/advisory/ Disallow: /.cam-tangent/ Disallow: /pbd/ Disallow: /gciaf/ Disallow: /gcdn/gciaf/ Disallow: /dcc/ Disallow: /gcdn/dcc/ Disallow: /dc/ Disallow: /gcdn/dc/ Disallow: /dcjs/ Disallow: /gcdn/dcjs/ Disallow: /content-queries/ Disallow: /zxc/ Disallow: /money/homefront/ Disallow: /exp-cruise Disallow: /exp-las-vegas2 Disallow: /exp-faw Disallow: /exp-caribbean Disallow: /exp-beach Disallow: /exp-cruise2 Disallow: /yourtake
Excerpt: only groups for all user-agents (*) or named AI crawlers, up to 40 rules per group.
Sites with a similar AI policy
Does usatoday.com block GPTBot?
Yes. As of Oct 2, 2026, usatoday.com's robots.txt disallows GPTBot, OpenAI's training crawler.
Can ChatGPT search show results from usatoday.com?
Its robots.txt blocks a ChatGPT search crawler, so ChatGPT search is unlikely to read or cite those pages.
Does usatoday.com have an llms.txt file?
We didn't find a valid llms.txt at usatoday.com/llms.txt when we checked.
Data: usatoday.com's public robots.txt and llms.txt, fetched by AskableHQ. Popularity rank from the Tranco list (ID Y83YG). Methodology.