110% Easier Web Scraping With The Yahoo Query Language Library

I've been examining and evaluating web scraping frameworks for Python over the last month and have found a few really good ones. The issue that I've been having though is they require too much time and effort for most people who simply want to perform quick and dirty scraping operations for small personal projects.

Today while surfing dzone I found a library that I think is way better for the needs of the majority of script writers; Yahoo Query Language.

"The Yahoo! Query Language is an expressive SQL-like language that lets you query, filter, and join data across Web services. With YQL, apps run faster with fewer lines of code and a smaller network footprint." -from the official site

In addition to being able to use a SQL-esque language for querying pages/services you can also use XPath selectors in the query statements to make it even easier to grab the data that you want. And as always, you can use firebug to retrieve the page element's XPath.

You can get started working with YQL in several ways:

• Use the YQL web console
• Import yql into the python REPL and have a go at it

Here are some links to get you started in developing web scraping utilities using YQL:

• Python YQL Library
• The Main Project Page for YQL
• Official YQL FAQ
• Two-Minute Quick Start Tutorial

I'll post some code samples sometime today most likely to show off what I've been up to this morning with YQL.