A file on a web site in the root directory of a website that is used to control which spiders have access to which pages within a website. When a spider or robot connects to a website, it checks for the presence of a robot.txt. Only spiders that adhere to the Robots Exclusion Standard will obey a robots.txt command file.