User-agent: Slurp #Yahoo Crawl-delay: 4 User-agent: MSNBot #MSN Crawl-delay: 4 User-agent: Googlebot #Google Crawl-delay: 4 User-agent: * #Disallow: / Disallow: /301Translator/ Disallow: /_layouts/backup/ Disallow: /aspnet_client/ Disallow: /cart/ Disallow: /CGI-BIN/ Disallow: /custom/ Disallow: /database/ Disallow: /featured/backup/ Disallow: /homepage/backup/ Disallow: /images/temp_images/ Disallow: /images/type_headers/ Disallow: /leela/ Disallow: /product/backup/ Disallow: /search/ Disallow: /tank/ Request-rate: 1/3 # maximum rate is one page every 3 seconds Visit-time: 0000-0545 # only visit between midnight and 5:45 AM #---------------------------------------------------------------------------------------------------------# # Removed Disallow All restriction 02.13.07 HH BHA # Modified and implemented 02.10.07 HH BHA # Updated 11.09.2006 HH BHA # Updated 11.01.2006 HH BHA