Contact Us
 
 

Welcome to ineedhits Blog

Welcome to the ineedhits Search Engine Marketing blog, where we share the latest search engine and online marketing news, releases, industry trends and great DIY tips and advice.



Monday, October 12, 2009

Why Google Can Index Your Robots.txt Blocked URLs

Posted by @ 5:34 pm
0
  •  

Have you ever noticed a URL showing up in search results, even after you’ve blocked it using Robots.txt? It’s a common query according to Google, so they’ve created a video (below) to explain how this can occur.

In the video, Matt explains that while Google will not crawl content blocked by robots.txt, they may still display a link to this page (with an empty title, description etc.) if it’s the most relevant result to the query.

For example, if www.nasa.gov blocked all search engines via robots.txt, Google may still display this ‘uncrawled URL’ in their results because there’s a huge amount of people linking to this page with targeted anchor text. The search engine listing may look something like this:

nasa-image

You’ll notice that this listing does not have a title and description, because Google was blocked from crawling the page by robots.txt.

If you want to completely remove your content from Google search results you have a couple of other options:

  • Use a NOINDEX meta tag at the top of your page
  • Use the URL removal tool within your Google webmaster account

Hopefully this clears up some confusion about robots.txt and blocked content. If you have any other questions let us know via the blog comments below!



Matthew Elshaw Matt is a marketing professional at ineedhits.com, an international search marketing firm. Matt's passion for online marketing began at university and has proved invaluable in steering product development and marketing initiatives at the company. Matt is a regular contributor to the ineedhits search marketing blog.

View Matthew Elshaw's profile






Discussion (Not Started)


Add Your Comments







SUBSCRIBE

Keep up to date with the latest from our blogs.

Subscribe to all blog posts

The Newsletter
BROWSE OUR POSTS




Sponsored Listings

PR0 to PR6! Dramatic TRAFFIC increase and proven results

Ignite your digital marketing with Perth's proven SEO specialists

Handbags | Bags | Bags Online

Mobile Search Advertising $9.95

  • New Posts
  • Popular
  • Comments


Jobthread



More in Tips & Advice (279 of 480 articles)


Earlier this week, Google announced the launch of the Google Merchant Center, which allows business owners to submit any product-type ...