 |
 | Home | Contact Us | FAQ | Search & Site Map | Link to Us |
 | Sign In | Join | Other 45 Sites in Network |
| HomeDiscussion GroupsGeneralPHPASPPerlColdFusionFlashHTML, CSS, ScriptsBrowsers |
Webmaster Forum / General / Web Design / December 2005Reducing googlebot crawl rate
Further to my queries regarding the rate at which Googlebot spiders, I have received the following official information from Google: "our robots do not currently recognize the Crawl-delay command [in robots.txt] but we are able to change the rate by your email request." Regards Matt -- The Probert Encyclopaedia - Beyond Britannica http://www.probertencyclopaedia.com > "our robots do not currently recognize the Crawl-delay command [in > robots.txt] but we are able to change the rate by your email request." An actual response from Google? Isn't that pretty much unheard of?  SignatureBlinky Killfiling all posts from Google Groups Details: http://blinkynet.net/comp/uip5.html > Further to my queries regarding the rate at which Googlebot spiders, I > have received the following official information from Google: > > "our robots do not currently recognize the Crawl-delay command [in > robots.txt] but we are able to change the rate by your email request." That's interesting, bespoke crawling. Potential administrative nightmare with the number of sites they list. SignatureCharles Sweeney http://CharlesSweeney.com >> Further to my queries regarding the rate at which Googlebot spiders, I >> have received the following official information from Google: [quoted text clipped - 4 lines] >That's interesting, bespoke crawling. Potential administrative nightmare >with the number of sites they list. Only if a lot of people email asking them to change it. Having a field for crawl rate in their site database is pretty straightforward, especially since their system does crawl different sites at different rates - my guess would be that normally, this is determined by whatever algorithms decide on how important a site is. And, given that such a mechanism exists, a manual override for the automated rate control isn't that much more complex. What would be more useful, of course, is for their spider to read the Crawl-delay line in robots.txt and update the database from that. If enough people email them to ask for a manual rate change, that might just encourage them to implement it :-) Mark Signaturehttp://www.OrangeHedgehog.com - RSS feeds and Google Adsense tools "My light will shine so brightly it will blind you" >>That's interesting, bespoke crawling. Potential administrative >>nightmare with the number of sites they list. > Only if a lot of people email asking them to change it. Absolutely. SignatureCharles Sweeney http://CharlesSweeney.com >>>That's interesting, bespoke crawling. Potential administrative >>>nightmare with the number of sites they list. > >> Only if a lot of people email asking them to change it. > >Absolutely. I suspect, though, that the number of requests to have crawling slowed down will be small enough to be manageable. Most people - especially those who have already swallowed the SEO snake oil - would prefer crawling to be speeded up. But the answer to that request is always going to be simply "no". Mark Signaturehttp://www.MineOfUseless.info - everything you never needed to know! "And so we're told this is the golden age" >>>>That's interesting, bespoke crawling. Potential administrative >>>>nightmare with the number of sites they list. [quoted text clipped - 10 lines] > > Mark I would think most people have no idea at what rate their site is crawled, or what problems it could cause. SignatureBrian Wakem Email: http://homepage.ntlworld.com/b.wakem/myemail.png |
Reply to this Message |
 Sign In
 Join
 My Latest Posts My Monitored Threads My Blog My Photo Gallery My Profile My Homepage Start New Thread Enable EMail Alerts Rate this Thread
|
©2010 Advenet LLC Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.