Have you ever noticed a URL showing up in search results, even after you’ve blocked it using Robots.txt? It’s a common query according to Google, so they’ve created a video (below) to explain how this can occur.
In the video, Matt explains that while Google will not crawl content blocked by robots.txt, they may still display a link to this page (with an empty title, description etc.) if it’s the most relevant result to the query.
For example, if www.nasa.gov blocked all search engines via robots.txt, Google may still display this ‘uncrawled URL’ in their results because there’s a huge amount of people linking to this page with targeted anchor text. The search engine listing may look something like this:
You’ll notice that this listing does not have a title and description, because Google was blocked from crawling the page by robots.txt.
If you want to completely remove your content from Google search results you have a couple of other options:
Use a NOINDEX meta tag at the top of your page
Use the URL removal tool within your Google webmaster account
Hopefully this clears up some confusion about robots.txt and blocked content. If you have any other questions let us know via the blog comments below!
Matt is a marketing professional at ineedhits.com, an international search marketing firm. Matt's passion for online marketing began at university and has proved invaluable in steering product development and marketing initiatives at the company. Matt is a regular contributor to the ineedhits search marketing blog.