This is one of a series of blog posts that I'm going to publish in preparation for my TechEd session next month in Dubai: this one is in relation to my oData session ‘Open Data for the Open Web’.

Wouldn't it be nice?

One problem that we see in today is information lives in silos: GetGlue knows some of the movie I like, so is Flixter. Hunch knows about more. Facebook knows about my personal friends and LinkedIn knows about my colleagues. Wouldn't it be great if I was able to create simple queries on top of all these different information, slice & dice, sort them out in a way that I want it to be?

The technology doesn't even make it any easier to exchange and control data. Just look at the various ways of retreiving a friends list. Facebook, LinkedIn, Twitter, Live, Orkut all have different APIs and demands us to use different ways of retrieving data. On top of that the data format is also quite different to one another even though the structure are similar.

More than one way to skin the cat

There are a few solutions out there to solve the problem: one is the Open Data Protocol (OData) . It is trying to solve some part of the problem by establihing a protocol on how one should query and update data. It is based on the web standards like XML, AtomPub and JSON.

Another offering came from Yahoo! with their Open Data Table and YQL. Unlike OData which is just a protocol specification,  Yahoo! came up with a hosted service offering which essentially allow us to write query in a sql-like syntax on top of multiple services. If only YQL also support querying on top of OData feeds too (not sure if someone can make a YQL community open data table for generic OData feed).

Challenges

There's going to be always challenges. Let's start the obvious one, security. I want to have full authentication and authorization to who and what can access my data. One part of the problem is solved with  OpenID and oAuth. But we will definitely need something more powerful for sure in the near future.  I want to be able to control which doctors can have access to my medical records just by flicking options on/off as I do with my Facebook account. Heck, I want to be able to pick which people on Faceook and grant access rights to my x-ray results.