Get Google Search Results With PHP – Google AJAX API And The SEO Perspective

If you’ve ever tried to write a program that fetches search results from Google, you’ll no doubt be familiar with the excrutiating annoyances of parsing the results and getting blocked periodically. Run a couple hundred queries in a row and bam! – your script is banned until proven innocent by entering an captcha. Even that would provide only a short reprieve, as you’d soon get blocked again.

Luckily there’s an official Google search API that will let you avoid that hassle. In this post you’ll find an example PHP script and a (mainly) SEO-oriented review of the API.

Using the AJAX API in PHP

I must confess that until yesterday I didn’t know you could use the Google AJAX search API in languages other than JavaScript. The documentation didn’t even mention the possibility when the API was first released. Well, it does now, and PHP is among the supported languages. Oh, the joy.

The API is already pretty well documented, so I won’t waste your time with another lengthy tutorial. Instead, here’s a simple example of how you could use it in PHP :

/**
 * google_search_api()
 * Query Google AJAX Search API
 *
 * @param array $args URL arguments. For most endpoints only "q" (query) is required.  
 * @param string $referer Referer to use in the HTTP header (must be valid).
 * @param string $endpoint API endpoint. Defaults to 'web' (web search).
 * @return object or NULL on failure
 */
function google_search_api($args, $referer = 'http://localhost/test/', $endpoint = 'web'){
	$url = "http://ajax.googleapis.com/ajax/services/search/".$endpoint;
	
	if ( !array_key_exists('v', $args) )
		$args['v'] = '1.0';
	
	$url .= '?'.http_build_query($args, '', '&');
	
	$ch = curl_init();
	curl_setopt($ch, CURLOPT_URL, $url);
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
	// note that the referer *must* be set
	curl_setopt($ch, CURLOPT_REFERER, $referer);
	$body = curl_exec($ch);
	curl_close($ch);
	//decode and return the response
	return json_decode($body);
}

$rez = google_search_api(array(
		'q' => 'antique shoes',
 ));

print_r($rez);

That’s it for the programming part.

So should we really throw away our lovingly crafted SERP scrapers and embrace the “official” API? Perhaps not. There are some peculiar things I’ve noticed after trying out the new API.

The Good

Lets start with the positive aspects. First, it looks like you can indeed safely use the API without getting blocked – I successfully ran about 1800 API queries in ~2 hours. Due to my crappy connection I was unable to test how it would behave if you turn it up to eleven and send hundreds of requests per second, but the rate limiter is definitely more lenient on API users than on plain SERP scrapers. This is a major plus for people who don’t like throttling their software to one request per minute or hunting for working proxies to get around bans.

The API also makes it easy to parse the results. All queries return JSON-encoded data, so you just json_decode() it and go. No need to invent complicated regexps that must be rewritten every time Google changes the HTML structure of the search results page.

The Bad

Of course, with a cliche megacorporation like Google it’s never all fun and games. You can only get 8 search results at a time, and no more than 64 results in total for any particular keyword. Whether this is a problem depends on what you intend to do with the API, but it’s certainly an unpleasant limitation.

The really peculiar – nay, insidious – thing is how the search results returned by the API differ from normal SERPs. A site that is #10 in a normal Google search may suddenly turn up as #1 in the API results. The typical #5 result may be moved to the second page. Basically, the API results look like they’ve been shuffled around a bit – the same URLs are returned but in slightly different order. Also, the “estimated result count” provided by the API is consistently much lower than what a normal search shows. All this makes the API useless for rank checking and similar SEO applications.

According to my tests you can’t just write off these discrepancies as a sideffect of geo-targeting.

It Depends

Overall, the API is either great or it kind-of sucks, depending on what you want to do with it.

At the risk of sounding like a conspiracy theorist, I must say the API seems to be cleverly engineered to be useful for “normal” purposes and somewhat useless for SEO. After all, only SEO workers really need accurate ranking data and more than 64 results per keyword phrase. Typical search engine users rarely move beyond the first page of results, so the limitations don’t hurt them. The various mashup makers that cater to the common user are also unaffected. It’s only the SEOs (and the rare academic researcher) that would be dissatisfied with the imposed constraints.

Of course, I’m sure you can still imagine a few interesting uses for the API ;)

Related posts :

82 Responses to “Get Google Search Results With PHP – Google AJAX API And The SEO Perspective”

  1. anjali says:

    hello,
    can any one tell me how can get more rankings with this api.i want to get upto 500 rankings for a keyword.pls help me in this..
    thanks in advance.

  2. White Shadow says:

    You can’t. This API is very limited. If you want to get 500 rankings you’ll have to write your own parser that extracts the rankings from normal search results pages.

  3. […] sprzedaży.Powtarzaj za mną: Dywersyfikacja dochodów jest ważna! Potem pobawiłem się chwilę skryptem, który pobiera wyniki z wyszukiwarki google. Radość, gdy kilkoma linijkami kodu możesz wyświetlić pasujące obrazy z wyszukiwarki google […]

  4. Jeff Zhang says:

    hi,
    are sure the difference between google ajax api and web browser is only the order and url format ? I test some keywords for my own site, it seems the difference is not only order. Some url can not been retrieved by google ajax api but I can find them through web browser
    Could you tell me what keywords you use for testing ?

  5. White Shadow says:

    Well, I didn’t say those were the only differences – they’re just the ones that are most immediately obvious. There are others – for example, the normal search results use something called Universal Search, while the API is purely web search (AFAIK).

  6. Indonesia News Reader says:

    I was apply these tips… its great and safe… Thanks and Peace.

  7. But I wanna more clear about.. declaring the the result on the web page and I want the linkable url. Peace

  8. breakaway says:

    Hey… it’s only returning 4 results, would you know why?

  9. White Shadow says:

    Yes, it does that. Check the API docs, they’ll tell you how to increase the number of results per query.

  10. Nguyen says:

    As the author said”64 results in total for any particular keyword”

    How can I get more than 64 results?

  11. White Shadow says:

    The API won’t give you more results no matter how nicely you ask. You need to write a scraper instead.

  12. Rich Carlson says:

    Hey… I am trying to do the site:URL search in the google ajax API with PHP using your script. I tried URLencode of the search string, but still nothing.

    Any tips?

  13. White Shadow says:

    URLencode shouldn’t be necessary as http_build_query() does that automatically.

    Try echo’ing the generated $url to see if it’s encoding the arguments correctly.

  14. Rich Carlson says:

    http://ajax.googleapis.com/ajax/services/search/web?q=site%3Alifehacker.com%2F5599508%2Fuse-a-camera-to-help-see-your-clutter-trouble-spots&v=1.0&rsz=large

    is the output with:
    site:lifehacker.com/5599508/use-a-camera-to-help-see-your-clutter-trouble-spots
    as the keyword to search.

    I get no results, even though when I do the site: search on Google, the results show up.

    I am wanting a way to check many URLs to see if they are indexed or not.

  15. White Shadow says:

    Did the URL actually contain the

    &

    entities, or is that a side-effect of my comment form?

  16. Rich Carlson says:

    &v=1.0&rsz=large shows as the actual character on the 7 key, not the HTML code for it…. those show even if the keyword isnt a site:www.url.com

  17. White Shadow says:

    Well, in that case I have no idea what’s wrong.

    Echo $body before the “return json_decode($body);” line and see if there are any error messages in the returned JSON?

  18. Rich Carlson says:

    Thanks in advance for all of your help, I definitely appreciate it…

    Odd development… that specific URL started working with no change at all in the code…. another verified indexed URL is showing nothing, much like the previous one.

    The echo $body is here: http://pastebin.com/rY33WYxr

  19. White Shadow says:

    Perhaps the servers that handle AJAX queries don’t use the same index as “normal” Google servers, so it takes a while until results start showing up.

    Also, I don’t see anything wrong with that JSON response – except that there are no results, of course.

  20. Rich Carlson says:

    Alright, just wanting to make sure I was not screwing something up. So I will move forward with the script and simply keeping that in mind… thanks!

Leave a Reply