Contact Us

Welcome to ineedhits Blog

Welcome to the ineedhits Search Engine Marketing blog, where we share the latest search engine and online marketing news, releases, industry trends and great DIY tips and advice.

Sunday, March 30, 2008

Google Adds DIY Robots.txt creator to Webmaster Tools

Posted by @ 1:31 am

Google has added yet another handy feature to the Google Webmaster Tools. Now website owners can easily create Robots.txt files and control which web pages search engines see and “don’t” see on their sites.

For those of you unfamiliar with Robots.txt files, Wikipedia explains them:

A robots.txt file on a website will function as a request that specified robots ignore specified files or directories in their search. This might be, for example, out of a preference for privacy from search engine results, or the belief that the content of the selected directories might be misleading or irrelevant to the categorization of the site as a whole, or out of a desire that an application only operate on certain data.

Now if you’re not overly confident creating your robots.txt files from scratch, you can simply create one using your Google Webmaster Tools.

As the Google release post suggests:

“It’s designed to give you an easy and interactive way to build a robots.txt file. It can be as simple as entering the files and directories you don’t want crawled by any robots.”

The tool allows users to either create broad exclusion rules which apply to all robots and crawlers or specific rules for particular robots. See the screenshots below.

Once you’ve created the robots.txt file using the tool, all you have to do is upload it to the top level directory (root) of your website and voila – you’re all done.

As the release post highlights, not all robots obey the conventions of the Robots.txt protocol, but rest assured most of the major search engines do, which is where most of your website crawling will come from.

Rene LeMerle Rene is the marketing manager of - a global search engine marketing company. He also leads the marketing for - a web 2.0 style community for online and digital marketers. Rene has been in the industry since 1997 with much of that time spent helping businesses embrace the best of the internet and digital world.

View Rene LeMerle's profile

Discussion (Not Started)

Add Your Comments


Keep up to date with the latest from our blogs.

Subscribe to all blog posts

The Newsletter

  • New Posts
  • Popular
  • Comments


More in Search News (1102 of 1797 articles)

A new report out by Nielsen Online shared a common tale that Google again dominated the share of searches in ...