WebProWorld :: Did Google Unleash Additional Googlebots?

WebProWorld :: Did Google Unleash Additional Googlebots?
Quotes:

"Apparently, Google has begun using another spider in their scanning and indexing of web sites...

Brett Tabke posted an interesting thought concerning Google's extensive crawling, "looks like "panic" based spidering… as if an index needs to be rebuilt from the ground up in a short time period (aka: the old index didn't work)." Another member believed these scans are apart of the PR re-calculation for the next PageRank update. Another poster, idoc, had also had an intriguing look at Google actions:

"I expect a lot of cloaking and redirect sites will be dropped soon from these new bot IPs and this crawl. It's what I had in mind in the post about hijacks when I said I think Google is on it. They have been asking for file paths and filenames with extensions I have never used before. I am hopeful anyway...

As it stands, the reasons behind Google's scanning efforts are unknown. The only things that are certain is they are using more than one crawler and that at least one of them performs a complete site scan. Is Google repopulating their index, or are they hunting out cloaked/doorway pages? Or are they finally getting around to doing another PR update? Like so many others have said, time will tell."

"quote]do you think it has something to do with clearing out doorway/cloaked pages?[/quote]

I think this makes the most sense. When you think about the fact that Google has reached a limit to the number of pages it can hold (4,285,199,774 since August 2003 I believe) and has had to drop outdated or obsolete pages from its cache, it would only seem natural that they would like to stop what they (Google) term SE spam.
There is a good article on this at http://www.searchengineethics.com/cloaking.htm if anyone is interested."

And on redirects: "sending the proper redirect code to this new Googlebot or should I be sending a 301 redirect or a 404 error? I'm afraid that this new Bot might see what I'm doing as spammy even though it is obviously there to help the users."