Yahoo Query Language (YQL) searching the web like a database

I remember a short while ago listening to a Ted talk by Tim Berners Lee. The father of the internet as we know it. he was explaining that now we have mass adoption and the web has become part of our lives it's getting ready for its next stage of evolution. That being the web as data. Essentially we're all getting more and more used to all types of info being available on the web. This most obvious in that this is the first place most of us look for anything.

Well I was really surprised and excited to see that Yahoo have already taken a lead in this area with their YQL (Yahoo Query Language) project. I'm literally discovering it as I type so I can't say how good it is. I can just talk about the concept and wonder whether there are other solutions like this out there.

So what's it for?
From a quick play with the 'console' I can see that it's essentially a tool for ajax enabled web pages to get info easily and in a standard format. so it's very much about providing a database over the web with a language much like the standard 'SQL' database language.

What can you search?
That's where I'm not so clear. I was hoping I could just search the whole of yahoos catalogue of sites. A big ask but that's what I really want and expect. Surely it would just be another way of getting into their search results. Maybe I'm thinking about it wrong. Maybe you can already do this. I think what I was hoping for was to be able to search the web as though it were a massive database.

Now I feel a bit disappointed. I kept looking for this and found it's a little abstracted, if I interpret it right, but you've got to use open datatables to search stuff on other web sites. You create a data table definition for the site or data you care about. This effectively tells YQL or whatever how to interpret the page or objects you're interested in. Once you've done that you load this into YQL and tell it to 'use' it as a reference. That's the overview I took from it.

How would I use it?
From that basic overview I'm still really impressed. I'd really just love to use the data in yahoos search results but that doesn't seem possible. Probably a ton of good reasons but that's what I want. anywho the idea of having a structured way of getting data from web sites that I want to query without having to build a whole engine to do this for me. Just right a few translation files and query like a database. That sounds like a dream for me.

So there we have it. Don't know if there are other providers that do similar but this seems like a significant evolutionary leap in the web. I hope it lasts given Microsofts deal with Yahoo. But for now I'm suitably impressed and excited. Can't wait to get on and start using it.