SMF® Unofficial Help Forum

2.0.* Support => 2.0.* Support => Topic started by: shubh on May 29, 2012, 09:22:29 AM

Title: google ignored (robots)
Post by: shubh on May 29, 2012, 09:22:29 AM
Google ignored this in robots.txt

Quote
Crawl-delay: 5 

  Rule ignored by Googlebot
Title: Re: google ignored (robots)
Post by: Illori on May 30, 2012, 05:58:37 AM
looks like google does not follow that. you need to read http://support.google.com/webmasters/bin/answer.py?hl=en&answer=48620