What is AI crawler?
A bot that fetches pages for AI training or AI answers.
AI crawlers are bots operated by AI companies — such as GPTBot (training) and OAI-SearchBot (ChatGPT search) from OpenAI, ClaudeBot (training) and Claude-SearchBot (search) from Anthropic, PerplexityBot, and the Google-Extended token for Gemini — that fetch web pages for model training or real-time answers. They can be allowed or blocked individually in robots.txt.
Free toolRobots.txt AI Crawler Checker — See which AI bots your robots.txt blocks