Have you ever noticed a URL showing up in search results, even after you’ve blocked it using Robots.txt? It’s a common query according to Google, so they’ve created a video (below) to explain how this can occur.

In the video, Matt explains that while Google will not crawl content blocked by robots.txt, they may still display a link to this page (with an empty title, description etc.) if it’s the most relevant result to the query.

For example, if www.nasa.gov blocked all search engines via robots.txt, Google may still display this ‘uncrawled URL’ in their results because there’s a huge amount of people linking to this page with targeted anchor text. The search engine listing may look something like this:

You’ll notice that this listing does not have a title and description, because Google was blocked from crawling the page by robots.txt.

If you want to completely remove your content from Google search results you have a couple of other options:

• Use a NOINDEX meta tag at the top of your page
• Use the URL removal tool within your Google webmaster account

Hopefully this clears up some confusion about robots.txt and blocked content. If you have any other questions let us know via the blog comments below!

Related posts:

• Google’s Tips on How to Write Perfect Meta Description Tags
• Google Advice: Don’t Obsess About Keyword Position in URLs
• How to Ensure You Are Always Included in Google
• Claiming a Top Google Local Spot
• Google Tip: How to Unrank Your Website!