#file to control web spiders, robots, etc. and what #they may index on our site. User-agent: * Disallow: /