User-agent: * Disallow: /styles.css Disallow: /style/ Disallow: /pic/ # Tells Scanning Robots Where They Are Not Welcome # --------------------- # User-agent: * applies to all robots, can also specify by name # Disallow: / disallow indexing of all pages. # If this matches first part of requested path: forget it! # NOTE: "Allow" tag is not officially endorsed # --------------------- # example # User-agent: infoseek # Disallow: /cgi-bin/ # --------------------- # http://www.searchengineworld.com/cgi-bin/robotcheck.cgi - Validation # http://www.jimtools.com/ go to Search Engine Stuff - Keywords Density