February 2, 201412 yr Has anybody else been hit badly by google bot? Just come back from hol to find site down due to bandwidth exceeded. Stats show gbot hit me for 8.6gb yesterday alone! Just one site affected and I have upped the bandwidth for it. Any suggestions on how to deal with this?
February 2, 201412 yr Have you got a robots.txt file on the site? If so it's worth disallowing bot access to folders like images and other assets, images in particular can use a lot of data transfer if you have lots of them.
February 2, 201412 yr Author Yes I have the following (it's a wordpress site plus an affiliate's shop using datafeeds)... User-agent: BaiduMobaiderUser-agent: BaiduImagespiderUser-agent: BaiduspiderUser-agent: Baiduspider-imageUser-agent: Baiduspider-videoUser-agent: Baiduspider-newsUser-agent: Baiduspider-favoUser-agent: Baiduspider-cproUser-agent: Baiduspider-adsDisallow: /User-agent: *Disallow: /wp-content/Disallow: /wp-admin/Disallow: /wp-includes/Disallow: /homeandgardenshop/categories.phpDisallow: /homeandgardenshop/merchant.phpDisallow: /homeandgardenshop/brands.phpDisallow: /homeandgardenshop/reviews.phpDisallow: /homeandgardenshop/admin/Disallow: /homeandgardenshop/search.phpDisallow: /homeandgardenshop/jump.php They've hit me again so site down again
February 2, 201412 yr ok, there's lots of areas where the bot can get stuck in a loop, or start sucking in endless data (calendars are notorious because you can have URL's for every date going into the future ad-infinitum, resulting in the bot crawling over and over again.). Some links that may help. https://groups.google.com/forum/#!topic/Google_Webmaster_Help-Indexing/pP4vyWXsMcM https://support.google.com/webmasters/answer/76401?hl=en http://www.bdtips.com/read/?article=what-to-do-when-googlebot-eats-up-huge-bandwidth Edited February 2, 201412 yr by BlueDreamer
Create an account or sign in to comment