What is robots.txt?

A file telling crawlers which paths they may access.

robots.txt is a plain-text file at the root of a site that tells web crawlers which URLs they may or may not request, using User-agent, Allow and Disallow rules, plus an optional Sitemap line. It is advisory: well-behaved crawlers follow it.