Matt Cutts: Gadgets, Google, and SEO � New robots.txt tool: "Sitemaps team just introduced a new robots.txt tool into Sitemaps. The robots.txt file is one of the easiest things for a webmaster to make a mistake on. Brett Tabke�s Search Engine World has a great robots.txt tutorial and even a robots.txt validator...
Part of the problem is that mucking around with robots.txt files is pretty rare; once you get it right once, you usually never have to think about the file again. Another issue is that if you get the file wrong, it can have a large impact on your site, so most people don’t mess with their robots.txt file very often. Finally, each search engine has slightly different extra options that they support. For example, Google permits wildcards (*) and the “Allow:” directive...
the robots.txt checker from the Sitemaps team is that it lets you take a robots.txt file out for a test drive and see how the real Googlebot would handle a file. Want to play with wildcards to allow all files except for ‘*.gif’? Go for it. Want to experiment with upper vs. lower case? Answer: upper vs. lower case doesn’t matter. Want to check whether hyphens matter for Google? Go wild. Answer: we’ll accept “UserAgent” or “User-Agent”, but we’ll remind you that the hyphenated version is the correct version.
The best part is that you can test a robots.txt file without risking anything by doing it on your live site..."
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment