ronwidhaBing's twitter search doesn't honor robots.txt
[caption id="attachment_257" align="aligncenter" width="659" caption="Twitter robots"]
[/caption]
Edited 12.08 PM:
Jeroen has kindly pointed out to me that I was looking at the wrong robots file. Instead of www.twitter.com/robots.txt, I should have looked at twitter.com/robots.txt which does show that Twitter wants their content to get indexed and doesn't say anything about Bing's position on robots file. Thanks!
[caption id="attachment_251" align="aligncenter" width="657" caption="Twitter robots file"]
[/caption]
A few minutes ago, Bing - Microsoft's own search engine announce that it's providing a taste of real time search by introducing '@' keyword to its search term list. By typing in @ronaldwidha for example you will see my Twitter timeline. However at the time of writing this post, Microsoft has only just started to index thousands of more prominent Twitterers.
What's interesting to me is the fact that Twitter doesn't want its contents to get indexed. As you can see from Twitter's robots file above, they are disallowing any crawlers to index the site. I guess that doesn't stop Bing from doing so.
I'll be checking Bing closely today to try out the feature as soon as the roll out is completed.
As you probably know twitter uses the non-www URL's as main domain. You're showing the www. version of robots.txt. Now check https://twitter.com/robots.txt and you'll see that Twitter does want it's content to get indexed.