Google has added yet another handy feature to the Google Webmaster Tools. Now website owners can easily create Robots.txt files and control which web pages search engines see and “don’t” see on their sites.

For those of you unfamiliar with Robots.txt files, Wikipedia explains them:

A robots.txt file on a website will function as a request that specified robots ignore specified files or directories in their search. This might be, for example, out of a preference for privacy from search engine results, or the belief that the content of the selected directories might be misleading or irrelevant to the categorization of the site as a whole, or out of a desire that an application only operate on certain data.

Now if you’re not overly confident creating your robots.txt files from scratch, you can simply create one using your Google Webmaster Tools.

As the Google release post suggests:

“It’s designed to give you an easy and interactive way to build a robots.txt file. It can be as simple as entering the files and directories you don’t want crawled by any robots.”

The tool allows users to either create broad exclusion rules which apply to all robots and crawlers or specific rules for particular robots. See the screenshots below.

Once you’ve created the robots.txt file using the tool, all you have to do is upload it to the top level directory (root) of your website and voila – you’re all done.

As the release post highlights, not all robots obey the conventions of the Robots.txt protocol, but rest assured most of the major search engines do, which is where most of your website crawling will come from.

Related posts:

• Google Webmaster Tools on Your Homepage
• DIY Tip: Make Sitemaps Work for Your Website
• Google Tips to Avoid Duplicate Content Penalties
• Search Engine Technology:: The Three Kinds of Search
• ineedhits and Hitwise Form Strategic Alliance to Bring World-Class Search Marketing Tools to Small Businesses