According to a post on WebProNews, Microsoft has admitted a problem with Bing whereby pages marked as NoIndex or NoFollow are being accessed incorrectly.
NoIndex/NoFollow are Meta Tags that can be used to advise search engine spiders to avoid indexing a web page. There are a number of reasons why you might want to use these tags such as avoiding duplicate content or hiding more private details.
The tags are currently supported by other major search engines like Google and Yahoo, so the issue appears to be impacting Bing only. Here’s what Brett Yount from Bing’s webmaster team had to say:
This is a known issue we are working quickly to resolve. If you have pages you would like permanently removed from our index, please send me a mail to firstname.lastname@example.org with your domain name and “MSNBot ignoring robots tags” in the subject line. Please also include the URLs in the body of the message. You may use an * wildcard for any directories such as:
Normally, I would request that you fill out a content removal request, however, since this is a problem on our side, I’ll do the leg work for you.
If you’ve got pages using these tags I’d recommend checking them in Bing and taking appropriate action if they’ve been accessed incorrectly. If you’re after other ways to prevent spiders from indexing your content check out the other methods supported by the Robots exclusion standard.