W-Shadow.com

If you’ve ever tried to write a program that fetches search results from Google, you’ll no doubt be familiar with the excrutiating annoyances of parsing the results and getting blocked periodically. Run a couple hundred queries in a row and bam! - your script is banned until proven innocent by entering an captcha. Even that would provide only a short reprieve, as you’d soon get blocked again.

Luckily there’s an official Google search API that will let you avoid that hassle. In this post you’ll find an example PHP script and a (mainly) SEO-oriented review of the API.

Using the AJAX API in PHP

I must confess that until yesterday I didn’t know you could use the Google AJAX search API in languages other than JavaScript. The documentation didn’t even mention the possibility when the API was first released. Well, it does now, and PHP is among the supported languages. Oh, the joy.

The API is already pretty well documented, so I won’t waste your time with another lengthy tutorial. Instead, here’s a simple example of how you could use it in PHP :

/**
* google_search_api()
* Query Google AJAX Search API
*
* @param array $args URL arguments. For most endpoints only "q" (query) is required.  
* @param string $referer Referer to use in the HTTP header (must be valid).
* @param string $endpoint API endpoint. Defaults to 'web' (web search).
* @return object or NULL on failure
*/
function google_search_api($args, $referer = 'http://localhost/test/', $endpoint = 'web'){
$url = "http://ajax.googleapis.com/ajax/services/search/".$endpoint;
 
if ( !array_key_exists('v', $args) )
$args['v'] = '1.0';
 
$url .= '?'.http_build_query($args, '', '&');
 
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// note that the referer *must* be set
curl_setopt($ch, CURLOPT_REFERER, $referer);
$body = curl_exec($ch);
curl_close($ch);
//decode and return the response
return json_decode($body);
}
 
$rez = google_search_api(array(
'q' => 'antique shoes',
));
 
print_r($rez);

That’s it for the programming part.

So should we really throw away our lovingly crafted SERP scrapers and embrace the “official” API? Perhaps not. There are some peculiar things I’ve noticed after trying out the new API.

The Good

Lets start with the positive aspects. First, it looks like you can indeed safely use the API without getting blocked - I successfully ran about 1800 API queries in ~2 hours. Due to my crappy connection I was unable to test how it would behave if you turn it up to eleven and send hundreds of requests per second, but the rate limiter is definitely more lenient on API users than on plain SERP scrapers. This is a major plus for people who don’t like throttling their software to one request per minute or hunting for working proxies to get around bans.

The API also makes it easy to parse the results. All queries return JSON-encoded data, so you just json_decode() it and go. No need to invent complicated regexps that must be rewritten every time Google changes the HTML structure of the search results page.

The Bad

Of course, with a cliche megacorporation like Google it’s never all fun and games. You can only get 8 search results at a time, and no more than 64 results in total for any particular keyword. Whether this is a problem depends on what you intend to do with the API, but it’s certainly an unpleasant limitation.

The really peculiar - nay, insidious - thing is how the search results returned by the API differ from normal SERPs. A site that is #10 in a normal Google search may suddenly turn up as #1 in the API results. The typical #5 result may be moved to the second page. Basically, the API results look like they’ve been shuffled around a bit - the same URLs are returned but in slightly different order. Also, the “estimated result count” provided by the API is consistently much lower than what a normal search shows. All this makes the API useless for rank checking and similar SEO applications.

According to my tests you can’t just write off these discrepancies as a sideffect of geo-targeting.

It Depends

Overall, the API is either great or it kind-of sucks, depending on what you want to do with it.

At the risk of sounding like a conspiracy theorist, I must say the API seems to be cleverly engineered to be useful for “normal” purposes and somewhat useless for SEO. After all, only SEO workers really need accurate ranking data and more than 64 results per keyword phrase. Typical search engine users rarely move beyond the first page of results, so the limitations don’t hurt them. The various mashup makers that cater to the common user are also unaffected. It’s only the SEOs (and the rare academic researcher) that would be dissatisfied with the imposed constraints.

Of course, I’m sure you can still imagine a few interesting uses for the API

As I discovered just a few minutes ago, Google Image Search has a feature that lets you restrict your search only to images that contain faces. However, it looks like the algorithm still has a few kinks to be worked out… Take a look at the first result of this image search query :

I shall not comment on the (fairly cliched) thought process that lead to the particular search query visible in the screenshot

December has just passed, so here’s and update on my experimental charity project. According to my stats, the Tree Link received nine links, which means I have to sponsor nine new trees. That’s not a lot, but it’s a start. And, since it’s New Year (more or less), I’ll throw in some additional $ and make it a nice round number… in fact, in hopes of fostering a bit more activity for this idea, lets make it really nice 1000 trees donation

I’ve donated $100 (=1000 new trees) to the Trees For The Future charity. Here’s the screenshot of the confirmation page (with some sensitive personal info smudged out) :

And here’s the full page without all the browser stuff :

When inserting an image into a post, WordPress lets you choose an alignment (left/right/center/none) and even add a caption. This usually works well when viewing the post on the blog itself, but will often look messed-up in the RSS feed. Basically, the image becomes unaligned and breaks the intended [...] Continue Reading…

As the year is coming to an end, perhaps it’s time for a different kind of “Top X” post. Instead of a list of new WP plugins to try out, here are the top plugin authors - the brilliant people who create outstanding tools and share them with the [...] Continue Reading…