Contact Us

Welcome to ineedhits Blog

Welcome to the ineedhits Search Engine Marketing blog, where we share the latest search engine and online marketing news, releases, industry trends and great DIY tips and advice.

Monday, October 12, 2009

Why Google Can Index Your Robots.txt Blocked URLs

Posted by @ 5:34 pm

Have you ever noticed a URL showing up in search results, even after you’ve blocked it using Robots.txt? It’s a common query according to Google, so they’ve created a video (below) to explain how this can occur.

In the video, Matt explains that while Google will not crawl content blocked by robots.txt, they may still display a link to this page (with an empty title, description etc.) if it’s the most relevant result to the query.

For example, if blocked all search engines via robots.txt, Google may still display this ‘uncrawled URL’ in their results because there’s a huge amount of people linking to this page with targeted anchor text. The search engine listing may look something like this:


You’ll notice that this listing does not have a title and description, because Google was blocked from crawling the page by robots.txt.

If you want to completely remove your content from Google search results you have a couple of other options:

  • Use a NOINDEX meta tag at the top of your page
  • Use the URL removal tool within your Google webmaster account

Hopefully this clears up some confusion about robots.txt and blocked content. If you have any other questions let us know via the blog comments below!

Matthew Elshaw Matt is a marketing professional at, an international search marketing firm. Matt's passion for online marketing began at university and has proved invaluable in steering product development and marketing initiatives at the company. Matt is a regular contributor to the ineedhits search marketing blog.

View Matthew Elshaw's profile

Discussion (Not Started)

Add Your Comments


Keep up to date with the latest from our blogs.

Subscribe to all blog posts

The Newsletter

  • New Posts
  • Popular
  • Comments


More in Tips & Advice (279 of 480 articles)

Earlier this week, Google announced the launch of the Google Merchant Center, which allows business owners to submit any product-type ...