Contact Us

Welcome to ineedhits Blog

Welcome to the ineedhits Search Engine Marketing blog, where we share the latest search engine and online marketing news, releases, industry trends and great DIY tips and advice.

Sunday, December 14, 2008

Google Gives You Free Reign on Site Crawling

Posted by @ 7:17 pm

The Googlebot is Google’s mighty algorithm which determines how much to crawl each site it visits. Now Google have given you more power to control this process.

The crawl rate is the component of the Googlebot which determines the time the Googlebot spends on crawling your site. Obviously, Google would like to thoroughly crawl your site (so that searchers are given the best results in the SERP pages), however it also needs to take into consideration that it should create little impact on your server’s bandwidth.

There is a default crawl setting; however some sites may have more specific needs. It’s in these situations that the new changes are useful.

Pooja Shah from the Google Webmaster Tools Team explains more:

“For a vast majority of sites, it’s probably best to choose the “Let Google determine my crawl rate” option, which is the default. However, if you’re an advanced user or if you’re facing bandwidth issues with your server, you can customize your crawl rate to the speed most optimal for your web server(s). The custom crawl rate option allows you to provide Googlebot insight to the maximum number of requests per second and the number of seconds between requests that you feel are best for your environment.”

Shah outlines more on the changes here, including a warning that if you do customize your crawl rate then it will only be in effect for 90 days, after which it will reset to Google’s recommended value.

There are a number of factors that determine the rate that Google sets as your site’s crawl rate. I would suggest that if you wish to customize your crawl rate, then you do so with care.

Courtney Mills Courtney is an online marketing and communications specialist at ineedhits - a leading search engine marketing firm with over 16 years experience. Courtney has been living and breathing online marketing for over 5 years. She specializes in web and communication marketing, while providing news and opinion to online marketing communities.

View Courtney Mills's profile

Discussion (1 - comment)

Sorry to say but Google sucks, even in their ” solutions” section if you have an error in the promotingtool section, the solution is not understandable adn if you implantvthe suggested code to let google read the page or site than 2 eerors appear coz the bot can not read the code they gave as solution….better use Cuil, the searc engine that has been founded by the original google founders;0

By Peter - December 16, 2008

Add Your Comments


Keep up to date with the latest from our blogs.

Subscribe to all blog posts

The Newsletter

  • New Posts
  • Popular
  • Comments


More in Search Engines, Search News (892 of 1797 articles)

These days, corporate blogging is a great way to connect with your customers and participate in discussions about your products ...