Rule: -- Sid: 1857 -- Summary: This event is generated when a client is requesting the file "robot.txt" from a web server. -- Impact: Information Disclosure. This file may contain data that could provide an attacker with information that could assist in an attack on the server. -- Detailed Information: In the early days of the web, when search engines first began indexing sites, it was often desirable to tell the indexing programs, referred to as robots, not to index certain parts of a site. A standarized method of accomplishing this was created; by placing a file called "robot.txt" or "robots.txt" in the root of your web site which search engines could read and which would tell them what parts of your site you did not want indexed. However, this file can also be very valuable to potential attackers if it contains information such as restricted directories, cgi-bin locations, etc. -- Affected Systems: Any web site that uses this method to communicate with robots. -- Attack Scenarios: An attacker can read the "robot.txt" file and use any sensitive data in it to profile your site in preparation for an attack. -- Ease of Attack: Simple. No exploit software required. Any browser can request a copy of "robot.txt" from the server. -- False Positives: Many. Most automated search engine indexing programs still request this file prior to crawling through a web site. -- False Negatives: None known. -- Corrective Action: Ensure that your "robot.txt" file, if you need one, does not contain any sensitive data. -- Contributors: Sourcefire Research Team Brian Caswell Snort documentation contributed by Kevin Peuhkurinen -- Additional References: --
