On the other hand, it’s blindingly obvious that this approach doesn’t quite mesh with another frequently repeated truism : “People don’t care about YOU”. You don’t need to be a jagged cynic to realise that this is true – for example, everyone knows that you can’t just blog about what you had for breakfast and expect to get thousands of visitors. Likewise, selling any random thing that you think is cool won’t work either – you need to do market research first and determine if there is any demand for your product.

So one has to wonder – what does “be yourself” really mean, and does it have any practical value?

“Be Like Me”

People who sincerely believe in the “be yourself” mantra are usually operating under the assumption that since it worked well for them, it will also work for you. This is a very common mistake that we all make from time to time. We tend to assume other people are mostly like ourselves, so things that are easy for us should also be easy for everyone else. For example, as a self-confessed geek, I’m constantly surprised at how “normal” people can make the dumbest mistakes when dealing with computers.

As it turns out, when someone says “be yourself”, what they actually mean is “be like me“.

The “be yourself” advice is only really useful when the advice-giver elaborates on what qualities in particular they want you to develop. Perhaps they think that being yourself means being more honest or more outspoken. Or maybe what they really mean is you should be ready to take risks and more confident about your choices. Either way, there needs to be something specific to go on. “Be yourself”  by itself is meaningless, a null operator.

Even if they don’t specify what exactly they mean by “being yourself”, you can sometimes figure it out on your own. The trick is to determine which aspects of their personality or environment are the ones critical to their success. If you manage that, you can then imitate those aspects. For example, you can check what skills they have and try to develop the same skills. You can find out what kind of people they hang out with and adjust your social circle accordingly. You can even see how they dress and adopt the same style.

Other Benefits of Being True To Yourself

While I think that people who advise you to “be yourself” are usually doing it for the wrong reasons, there can also be some advantages to embracing your own interests and personality :

• For one, it can be beneficial to say (or write) what’s really on your mind, even your position is controversial. People will tend to notice the passion and sincerity of your arguments even if they disagree with you. In the long run, this can help you develop a reputation for being honest.
• Another advantage is the confidence boost that you get when dealing with topics that you know and like. It’s much easier to talk confidently about things you personally find interesting. Similarly, if you manage to find a job in a field that you’re really passionate about, you will definitely feel more confident about yourself than if you had to slog through your day at a soulless cubicle farm. And confidence, as we know, is attractive.
• Finally, it’s just more fun.

Don’t take the above as indication that I think being yourself is a great idea for everyone. Ultimately, being yourself is the second step. The first step is to be somewhat attractive (in the broad sense), either naturally or by imitating someone else. Then you can use the “be yourself” shtick to emphasize that attractiveness, to play to your strengths and reap the emergent benefits listed above.

Image credit : cobrasoft @ sxc.hu

And we’re skipping merrily along the edge, safe in the knowledge that our cuddly teddy-bear of “individual action” will make all the bad things go away.

This is the mental image I get when I think about our response(s) to problems like climate change and existential risk. On the one hand, we have this big impending catastrophe that can potentially cripple the entire civilization and wipe out untold numbers of species. On the other hand, we have various suggestions on how we can “save the planet”. Like, for example, buying energy-efficient lightbulbs or reusable shopping bags.

There are two main issues with these calls for “individual action”.

1. Assuming Total Cooperation

Many of the articles and even scientific papers that advocate specific ways to lead an environment-friendly lifestyle usually contain a typical statement that goes something like this : “If everybody chose to do X then the amount of [carbon emissions/wasted electricity/hungry puppies] would be decreased by [a huge number]!”

The key word here is “everybody”. It is completely unrealistic to assume that everyone will willingly embrace a lifestyle change that demands a personal sacrifice (which is almost always required for anything environment-related). Four words : Tragedy of the Commons. So most changes proposed in the above manner give people unreasonable expectations about their efficacy because the expected payoff, the impressive number of saved kilowatts or fed mouths is unachievable in the real world.

Of course, you could argue that even small contributions help. That’s certainly a fair point, but when the problem in question is something like climate change and involves critical tipping points and feedback loops, a half-assed effort just isn’t good enough to avoid falling over the edge. Unfortunately, the tragedy of the commons ensures that half-assed is the only voluntary effort our society is capable of.

2. Warm and Fuzzy and Insignificant

Individual action is great for making people feel good, making them feel they’re doing their part and making a difference. Buying “green” stuff also serves as an excellent status symbol and provides a way to show off your selfless nature and environmental awareness (bought a horribly expensive plugin-in hybrid? Wow, you’re the best hero of everything forever!). However, we tend to clinically overestimate the global impact of actions that have “close and personal” effects on us.

As this slightly crazy article illustrates, individual humans only account for a tiny percentage of carbon emissions and other kinds of waste. Most of the negative environmental impact is caused by industrial sources. So even if you somehow manage to do the impossible and convince everyone to live as sustainably possible it will only lead to relatively small direct benefits.

Again, one might claim that if a lot of people go green and “vote with their wallets” then all the big companies will be forced to adopt greener practices. This may actually be a more effective strategy/argument than simply trying to save the planet directly by reducing your own emissions, but I’m still not convinced it’s possible to get a sufficient number of people to do it to have the required impact.

In Favor of Overkill

So what is one to do in this lamentable situation? It’s unrealistic to expect that humanity will carry out the current (comparatively mild) recommendations of our climate scientists in time. The governments will just dawdle around and implement a bunch of half-assed “compromises” that please voters but fail to prevent a catastrophe. Personally, I’m in favor of skipping that step and going straight to the radical, high-risk/high-payoff, long-term strategies :

• Geoengineering. I’m not advocating deploying something like stratospheric sulfur aerosols right away, but I think more resources should be allocated to the relevant fields research so that we have ready-to-go geonengineering solutions available when the need arises.
• Nanotechnology – sufficiently advanced nanotech (and molecular manufacturing in particular) has the potential to be one of those “solve everything” technologies. It could virtually eliminate material scarcity and make pretty much any technological process more efficient, thus drastically reducing environmental impact. Well, aside the rogue nanoparticle pollutants that can easily cross the blood-brain barrier and kill you. A minor inconvenience.
• Artificial Intelligence – unable to solve the environmental challenges on your own? Make something that thinks smarter and faster than any human! Hell, if Vinge’s predicted timescale is correct then we needn’t worry about climate change at all – the post-Singularity humanity of 2030 would probably find the environmental challenges trivial (so I worry about runaway AI’s instead).
• Space travel – if all else fails we could just run away.

This concludes my first-ever rant on climate change (well, except that one time). Next up : why gay marriage is a human right, the benefits of Linux vs. Windows, and a discussion of the most widespread political movements

However, sometimes reviewers get carried away by their own superlative eloquence and start claiming that a specific site will top everything else and become the only worthwhile way for people to communicate in the online world. Why, I’ve read such claims about FriendFeed just a short time period ago.

Why We Need Both Twitter and Books

Communication is a diverse beast. You might want to contact someone in private or send out a message to a large number of people. You might want a realtime conversation or a long letter or article that you can read at your leisure. Sometimes you might want the intimate in-person experience, and other times a bland just-the-facts report would be enough.

There’s a wide spectrum of possible situations and no single technology covers them all (yet). What’s more, we don’t really always know what medium would be best in each particular case. For example, “videophones” were perceived as an obvious next step for remote communication but now that we actually have the technology it turns out we don’t actually want them that much, at least not for everyday use. We’re still exploring the space of possible solutions. It’s obviously much too early to say that a given technology/service/application is perfect – even for one specific purpose.

The Spectrum

If you’re wondering how the spectrum would actually look like, imagine a multidimensional graph with axes that represent the main parameters of various communication mediums, like average message size, number of people sending/receiving, delivery speed and how long the message stays relevant. Then plot all various the technologies and services – from body language to 4chan – on the graph and animate it along the timeline of our civilization. It would probably make for an interesting presentation.

Due to lack of suitable software/skill I was forced to limit myself to two dimensions (cardinality and message relevance time/TTL) and no animation. Here’s my attempt :

I’d love to point out some of the apparent trends and correlations, but they’ve already been discussed to death by every two-bit futurist out there. Lets just say we’re still only getting started with new mediums.

It’s a slow news month!

At first it might seem like the answer is braindead-simple – users with lots followers are obviously more popular (on Twitter) than those who have just a small bunch of people looking at their updates. Indeed, this is the approach taken by the recently established Twitter directory WeFollow – users and tags are sorted by the number of followers, and so far it seems to work out okay. However, as you can probably guess, I’m about to say that this simplistic approach is wrong and suggest a more “advanced” way to measure Twitter popularity

When we determine popularity based on the follower count we treat each follower as a single vote, and in good democratic fashion we consider all votes equal. But the voting analogy doesn’t really hold. The real value that’s being passed around in the Twitter-verse isn’t votes – it’s attention. If you follow someone then you’re going to devote a portion of your time and attention to read their updates. And attention is a scarce resource. The more users you follow the less of your attention each of their updates will get.

We need to take this into account when counting the “votes”. Fortunately, somebody has already worked out the math for us :

Agalmos was first conceived for the Twitter platform, where people can “follow” other people’s updates, and the number of users one is following is assumed to be inversely proportional to the attention each feed is receiving. Then we use information entropy. Look up Wikipedia.

Behold the Twitter/RSS/”subscribing model” formula to determine agalmos:

Source : Agalmos – an information measure for agalmic economics

I’ve written a little script that calculates the “attention value” of a Twitter account using this function. Due to API limitations it only analyzes the first 50 followers and then uses the average value to estimate the total, but I think it works well enough as a proof-of-concept. You can try it out yourself here.

To put the numbers it reports in perspective, here’s what the script says about my two accounts :

white_shadow

• 86 followers – mostly normal users
• 5.93 follower value

antiprivacy

• 193 followers – mostly spammers and people with auto-follow scripts
• 7.47 follower value

It’s not perfect, but judging from this singular, anecdotal example, I think it’s clear that an attention-based metric works better than directly comparing the number of followers

Well, this post is actually my attempt to answer a burning question : “How can we download stuff for free and ensure that musicians, actors, programmers and other people who create stuff still get paid?”. It’s a longish, wistful rant on intellectual property and conflicts between scarcity-based economy and post-scarcity society (The Web), with a few constructive suggestions tacked on in the end. Comments are welcome.

The Problem

What do the following examples have in common :

• Lawsuits against filesharers and sites that enable filesharing. Case in point : The Pirate Bay trial.
• Fledging attempts to make the Internet fully monitored, supposedly to stop copyright infringement.
• People who write great OSS/freeware tools still need to have a “day job” to pay the bills.
• Ridiculously cumbersome DRM.
• Buying something through an online content delivery service can cost more than buying the same thing in a physical store. Case in point : Steam.
• The fact that the owner of 4chan is in debt.

Any ideas? Well, here’s mine : all of these problems are caused by a conflict between scarcity and post-scarcity.

We’re currently at an unique point in time, a time when there still exists a scarcity of many resources but a few resources are becoming effectively free (disk space, computing power, bandwidth… well, almost). This leads to a conflict of intuitions. On the one hand, people realize that copying a file or loading a website costs practically nothing (uses non-scarce resources), so they don’t want to pay for downloaded movies/music/software or pay for using a site like 4chan or Wikipedia.

On the other hand, the producers of value – actors, musicians, programmers, webmasters – have to expend scarce resources to create stuff, even if the end-result is stored in a non-scarce medium. People still need equipment like computers, music instruments, brushes, filming sets, video cameras and other tools of their craft. They also need food, shelter, clothes, transportation, time and perhaps web hosting. You can download [The.Dark.Knight[2008]DvDrip-aXXo.avi] for free, but it cost millions to produce.

In other words, creators need a way to obtain scarce resources in exchange for non-scarce resources. They’ve spent time and money to make something people like and they feel they should get something tangible in return (also, they want resources to make more stuff in the future). Of course, this is intuitively unacceptable to consumers who wonder “Why should I pay for stuff I can get for free?”.

And so it goes.

Bad Solutions

So far, all the attempts to deal with this conflict have only lead to partial solutions at best and crazy abolish-human-rights projects at worst.

• Artificial scarcity. Yeah, I’m looking at you, DRM. Trying to turn a non-scarce resource into a scarce one seems downright criminal when you consider that this thing we call “progress” normally leads to less scarcity, not more. Also, I’m sure you’re aware of the fact that DRM has so far proven ineffective at deterring piracy and might even push more people towards using torrents as it’s more convenient than dealing with idiotic file protection schemes.
• Selling merchandise and tickets for live performances. This is a fine practice in it self, but it only works for certain niches and levels of popularity. A band like Sonata Arctica can probably make a decent amount of cash on t-shirts and tickets, but I doubt anyone would pay to see the author of the WordPress plugin WP Super Cache programming “live” (no offense Donncha ).
• Get a job. Of course, the programmers, writers, musicians and everyone else could just get a “normal” job and use that income to support their art, right? That works in theory, but this solution creates more problems than it solves. First, you have to find a job that doesn’t suck, which is no easy feat. Second, you’d have less time to spend on the thing you actually want to do. And finally, there’s one drawback that most people fail to consider – if your day job is in the same field as your “art” (e.g. a programmer working on commercial software to support his open-source projects), chances are it will be subject to the same scarcity vs. non-scarcity problem. Getting a job only shifts the problem to the employer, not eliminate it.
• Advertising. This is another partial solution that only works for certain niches. How would product placement work in music, and would you really want it to? The truth is people hate ads and can go to great lengths to get rid of them. A website with AdSense on it might get away with it for a while (though a lot of geeks seem to think in an ideal world everybody would be using Adblock), but adware is usually mercilessly eradicated and commercial breaks in streamed videos are seen as a deadly sin. In summary : advertising works, but it’s not the best solution.

A Better Solution?

Any proposed solution needs to answer two basic questions :

• Where does the money come from? As much as we’d like that, we can’t really get everything for free. So even if we invent a perfect system for dealing with postscarcity in a world that still depends largely on a scarcity-based economy, somebody will have to pay = provide the required scarce resources to the people producing non-scarce stuff. This could be either the consumers, or advertising income.
• Who gets the money, and how much? This is a problem of resource allocation. You’ve probably heard various rumors that record companies bag most of the income and hand only pennies to the artists. Another example would be mediocre videogames that cost as much as good ones, and the fact that you have to pay up front for movie tickets. Ideally, what you pay should be commensurate with how valuable the product is to you.

My proposal : Whuffie + real money

Imagine a membership service called “PostRep” (from post-scarcity and reputation). This service takes a small subscription fee, say, $20-$30/month. This is where the money comes from (advertising might also be used – more on that later). In return, every member of the system gets the ability to hand out and receive reputation points. You could also call it Whuffie, karma, upvotes, tokens or whatever else you think is cool and marketable. Team accounts can also be created. Each team has to decide on their own how to redistribute points received by a team account as the system doesn’t impose any particular algorithm there.

You can hand out as many points as you like – there is no limit – and you can do it manually and/or automatically. For example, you could give 100 points to the programmer who wrote an open-source program you use, and your music player could automatically give 1 point to the author(s) of each track you listen to. Downloading stuff via the PostRep system would also automatically assign a (configurable) amount of reputation to the creator of the file. This should be fairly easy to integrate with existing rating systems, so that when you rate a movie on IMDB, you also automatically send a proportional amount of points to the team who made it.

What (else) is in it for me?

As a member of this imaginary system, you get free and easy access to the non-scarce values created by any member of the PostRep online system. Music, movies, software, books, whatever else that can be expressed and transmitted digitally. No DRM and other bullshit. Creating a system that actually works would be a huge engineering challenge, but it’s definitely feasible (see also : Steam and similar services).

As a side-effect of the reputation system you can also get a very flexible rating scheme for digital content and even other people. Hell, you could even implement the “right-handed vs. left-handed Whuffie” idea and see how much reputation points where assigned by people you like vs. the people you dislike. Of course, the amounts would need to be normalized based on the total number of points handed out by each individual, but that would be pretty easy from the engineering perspective.

Where’s the money?

So what happens to your subscription fee? Simple – each month, PostRep redistributes the funds to other members proportionally to the amount of rep. points/Whuffie/whatever you gave them. This also means that you get a smaller or larger sum as well, depending on how much rep you got for your work. Popular artists, OSS programmers and the owners of popular websites could even make a decent income.

Advertising

A more controversial approach would be to make the system partially or wholly ad-supported. Marketers would jump on the possibility to access data on per-user ratings and point transactions, as it would give them the ability to create finely targeted advertising. However, as I mentioned before, advertising isn’t generally well-regarded by users. Making it opt-in might help.

And where’s the catch?…

Of course, there are several disadvantages to this idea, like political feasibility and how the PostRep earnings of popular artists would compare to their current income. I’ve also been (intentionally) vague on how exactly the reputation points would be calculated and normalized. However, I won’t go into details on that today – this post is already long enough as it is Feel free to post any criticisms and suggestions in the comment section.

Hmm, I really need to find some illustrative pictures next time.

#1 Lack of focus

Pretty much every blogging-related guide begins with “pick a niche”. Having a focused blog has many advantages :

• You get more RSS subscribers. People like to know what to expect and don’t like sifting through (subjectively) irrelevant fluff. If your blog is focused on widgets, you’ll get RSS subscribers who are interested in widgets. If your blog has a few posts about widgets, but is otherwise all over the place, only people who share several of your interests are likely to subscribe. To put it otherwise, there’s a lot of people who care about politics. On the other hand, there are significantly less people who care about politics and underwater kickboxing.
• It helps with SEO. If you have a site focused on a single topic, you can put your main keyword(s) in the domain name and use relevant category names. You can also get links from other niche blogs and niche directories. On the more obscure end, semantically related content lets you reap the benefits of latent semantic indexing.
• (More) relevant AdSense ads. In my experience, generalist sites are more likely to show mismatched contextual ads than niche sites. It’s especially noticeable when I publish a new post – the ads are usually based on something I’ve written about in the past (e.g. converting video files) and may be completely irrelevant to the actual post. Of course, the results usually get better after Google analyzes the actual post. Sometimes.
• If you try hard enough, you can become an authority blogger (or at least a contender) in your field and achieve SUCCESS!!™. Becoming a recognized authority is a challenging enough goal in itself, but for a generalist blogger it’s a nearly impossible one.

All that said, this blog has no focus. The recent skew towards WordPress and blogging in general is purely incidental. You’re just as likely to see posts about … copyright issues and transhumanism, or maybe my numerous cats. The reason is simple – if I had a niche-specific site, it would get updated twice per month at best.

#2 Irregular posting frequency

Speaking of which, a regular update schedule is another thing that’s missing on this blog. Opinions differ on how often you should post, but the general consensus is that one should stick to the same posting frequency.

The idea is, again, that people like to know what to expect. There’s also an individual “comfort zone” of posting frequency for each reader – you need to post often enough to keep them interested, but not so often that they feel overloaded with information. Obviously, this is not something where you can easily find the optimal solution – every subscriber will have different preferences. However, if you vary your posting rate too much you risk losing your existing readers.

Compared to the early days, I’ve improved a lot in this respect. When I started out in 2006, I might post every day for a while and then suddenly disappear for several weeks. Now I usually stick with writing 2-3 posts every week. However, the intense desire for procrastination still gets me from time to time, and I end up ranting about something after midnight (which probably hurts the quality of prose a little bit).

Sidenote : It might be interesting to see/gather stats on posting frequency vs. popularity vs. topic.

#3 Personality issues

You probably don’t even know if I’m male or female – it’s certainly not mentioned anywhere on this site (I checked). Eh, I don’t even have an “About” page!

Well, should I? On the one hand, Nobody Cares About You. On the other, people are more likely to trust you (and your opinion) if they know there’s a real person behind the site. It also helps if you have some relevant credentials to show off and a cute/professional photo. Conversely, an anonymous schmuck posting under a nickname doesn’t impress credibility.

Nevertheless, an unhealthy mix of cynicism and nonconformism has so far prevented me from creating a public bio. I’m okay with revealing personal information (for an extreme example, see : my full browsing history (updated in real time)), but doing it to knowingly manipulate people’s opinions just seems kinda uncool. On the other hand, all communication is manipulation, so I expect I’ll eventually create an “About” page of some fashion.

There’s also another aspect to this “being a personality” stuff. The acclaimed masters of blogging say that your writing should be personal. That is, you should let your personality show through your writing and have something that differentiates you from the hordes of other bloggers.

I say that, at least in this general form, the suggestion is bullshit.

It’s not about having a personality. As any experienced marketer would tell you, it’s actually about being likable and/or someone the reader can identify with. As a result, the general advice to “just be yourself” really only works when you’re a naturally attractive person.

And I’m a pretty weird, antisocial guy with about 0.6 friends. Does it show?

#4 Neglecting social networking

Again, almost every guide to blogging, SEO and online marketing strongly recommends to leave comments on related blogs, participate in relevant forums and use social media sites (Digg, Youtube, etc) to the fullest extent. As you might have guessed from the previous section, this is not something that I do. In fact, I’ve posted about two comments on other blogs in the last 12 months. The lurker is me.

Of course, you don’t have to fall into the same trap. Go out and start expanding your social network right away. I’m quite serious.

Why social networking? (warning : some cliches follow)

• From the SEO perspective, comments on other sites help drive visitors towards your own site. Also, if you keep in touch with other bloggers, they’re more likely to mention you (and your site) in one of their posts.
• It’s the age of the global crowd. Internet users congregate on social media sites, so that’s where you have to be if you want to get their attention.
• Every comment, guest post, YouTube video and social bookmark you create is another entryway to your blog. In addition to the direct benefits of promotion, this Web 2.0 actitivty also offers the advantage of diversification.
• Reputation economy is upon us. More (positive) contacts = better reputation.

Final words

I read several posts about blogging mistakes prior to writing my own. What I found interesting were the implicit assumptions about what it means to be a good blogger. Learning what one considers a mistake can tell you a lot about the actual goals and interests of that person.

There was some talk about technical issues (platform choice, RSS feeds, etc), a bunch of posts about self-discipline (getting ideas, posting frequency), and a huge amount of marketing stuff (SEO, social networking, branding). Consequently, a good blogger = a technically savvy, skilled marketer specializing in self-promotion.

Of course, traffic and popularity are ultimately means to an end. Therefore, blogging is also only a way to achieve something – to get your message out, regardless of whether the message is a sales pitch or an idealistic rant about the environment. So this is one more mistake some bloggers make – to forget why they blog and concentrate on the act of blogging itself.

Kind of like the rant above, hmm?

There are two main things we dislike about ads :

• They try to make you buy shit you don’t need.
• They are distracting. They try to grab your attention and steer it away from the content you’re currently viewing or the task you’re trying to accomplish.

Now consider what would happen if advertising was carefully targeted towards the interests, hobbies and vital stats (age, sex, location and so on) of each individual. Sounds good at first – there’d be no more idiotic offers to “enlarge your reproductive organ” if you’re female, no more dubious “speed up your computer” ads for people with a shred of common sense, etc. Instead, we’d see relevant and – dare we hope – interesting advertising.

And therein lays the rub. As humans, we’re very good at filtering out irrelevant and insignificant details. On a crowded street, we only notice the bypassers that are somehow unusual or out of place – weirdly dressed, very attractive, armed, or old friends from school.

The current state of advertising is somewhat like that street – there’s a myriad of people (ads), but only a few are interesting enough (or sufficiently obnoxious) to really catch your attention.

With individually targeted advertising, the street is just as crowded. The difference is that everybody is unusually good-looking, shares your interests, knows you intimately and (still) likes you. And all your highschool friends are there, too.

And they’re all trying to sell you something.

Targeted, relevant advertising will attract your attention more easily and you will waste more time viewing ads than you did before. Ads that promote things that you might actually like will be more effective at making you buy shit you don’t need (if this seems counterintuitive remember that like ≠ need). Also, better targeting will enable more effective consumer manipulation.

At least there are less armed robbers in the crowd, eh?

Image by Kyle Wegner

What’s Wrong With Existing RSS Readers?

The current RSS reader applications are far from perfect. In fact, they’re little more than dumb aggregators. There are three main issues that they don’t address very well (or at all) :

• Huge amounts of information
  Manual categorization and a search function will only get you so far. It’s still easy to be overwhelmed if you subscribe to hundreds of feeds.
• Interestingness
  The user has to manually sift through mountains of data to find the salient tidbits. Most current applications don’t deal with this aspect at all.
• Timely-ness
  Some items are suitable to read at your own time, with a cup of tea in your hand. Other items, like news about a major event in your industry or a great deal at an auction site, should be seen immediately. Most existing RSS readers have no concept of “priority” or “importance”.

There’s also the whole “OMG social web!” aspect that’s generally lacking from current implementations (Google Reader being a partial exception), but I won’t deal with that here.

To The Shining Future

The web as a whole is slowly moving limping towards greater usability, but at the moment most sites are trying to take the easy way out by outsourcing “crowdsourcing” all the complex stuff (e.g. collaborative filtering and ranking). That makes sense as an initial approximation, but ultimately you can’t get very far by averaging opinions.

Lets assume we have time and money to spend on actual usability R&D, and possibly a weak AI handy. What improvements could we make to existing RSS readers?

• Use icons and colors to identify sources and represent common concepts or topics. Assign them automatically, possibly multiple icons per item. Make the glyphs degrade gracefully – use basic shapes recognizable at a glance. This will make “scanning” a list easier, not to mention how useful it will be when we all have retina displays and use HUD-type interfaces.
• Rewrite texts to use familiar words for easy comprehension and unfamiliar terms for newsness value. Determine familiar terms by examining existing communications : email, RSS, search history…
• Use machine learning to determine which items are more likely to be judged as interesting by the user. Possible criteria : how long the user spent reading the item (relative to the length of the text), whether they clicked any links, etc. Current RSS filters must be configured manually. Make them smart and adaptive instead (we already have adaptive spam filters). Reduce the time required to teach/personalize the filter by seeding it with settings used by people with similar interests. See the previous item for a hint on how to find them.
• Experiment with other modalities. Instead of plain text, try : shape, color, movement, structure (hierarchy), sound. Haptics, when they become available (sometime in the next decade most likely).
• Offer intelligent summaries, or at least personalized keyword extraction (topic detection). In the current framework, automatically categorize/label new items based on how the user has categorized existing items in the past.
• Remember that clients are part of the cloud. Are your servers too feeble to handle all of the above? Some personalization and filtering can be done offline, and that’s a good thing – you shouldn’t need to give away all of your info to online megacorps.

Luckily there’s an official Google search API that will let you avoid that hassle. In this post you’ll find an example PHP script and a (mainly) SEO-oriented review of the API.

Using the AJAX API in PHP

I must confess that until yesterday I didn’t know you could use the Google AJAX search API in languages other than JavaScript. The documentation didn’t even mention the possibility when the API was first released. Well, it does now, and PHP is among the supported languages. Oh, the joy.

The API is already pretty well documented, so I won’t waste your time with another lengthy tutorial. Instead, here’s a simple example of how you could use it in PHP :

/**
 * google_search_api()
 * Query Google AJAX Search API
 *
 * @param array $args URL arguments. For most endpoints only "q" (query) is required.  
 * @param string $referer Referer to use in the HTTP header (must be valid).
 * @param string $endpoint API endpoint. Defaults to 'web' (web search).
 * @return object or NULL on failure
 */
function google_search_api($args, $referer = 'http://localhost/test/', $endpoint = 'web'){
	$url = "http://ajax.googleapis.com/ajax/services/search/".$endpoint;
 
	if ( !array_key_exists('v', $args) )
		$args['v'] = '1.0';
 
	$url .= '?'.http_build_query($args, '', '&');
 
	$ch = curl_init();
	curl_setopt($ch, CURLOPT_URL, $url);
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
	// note that the referer *must* be set
	curl_setopt($ch, CURLOPT_REFERER, $referer);
	$body = curl_exec($ch);
	curl_close($ch);
	//decode and return the response
	return json_decode($body);
}
 
$rez = google_search_api(array(
		'q' => 'antique shoes',
 ));
 
print_r($rez);

That’s it for the programming part.

So should we really throw away our lovingly crafted SERP scrapers and embrace the “official” API? Perhaps not. There are some peculiar things I’ve noticed after trying out the new API.

The Good

Lets start with the positive aspects. First, it looks like you can indeed safely use the API without getting blocked – I successfully ran about 1800 API queries in ~2 hours. Due to my crappy connection I was unable to test how it would behave if you turn it up to eleven and send hundreds of requests per second, but the rate limiter is definitely more lenient on API users than on plain SERP scrapers. This is a major plus for people who don’t like throttling their software to one request per minute or hunting for working proxies to get around bans.

The API also makes it easy to parse the results. All queries return JSON-encoded data, so you just json_decode() it and go. No need to invent complicated regexps that must be rewritten every time Google changes the HTML structure of the search results page.

The Bad

Of course, with a cliche megacorporation like Google it’s never all fun and games. You can only get 8 search results at a time, and no more than 64 results in total for any particular keyword. Whether this is a problem depends on what you intend to do with the API, but it’s certainly an unpleasant limitation.

The really peculiar – nay, insidious – thing is how the search results returned by the API differ from normal SERPs. A site that is #10 in a normal Google search may suddenly turn up as #1 in the API results. The typical #5 result may be moved to the second page. Basically, the API results look like they’ve been shuffled around a bit – the same URLs are returned but in slightly different order. Also, the “estimated result count” provided by the API is consistently much lower than what a normal search shows. All this makes the API useless for rank checking and similar SEO applications.

According to my tests you can’t just write off these discrepancies as a sideffect of geo-targeting.

It Depends

Overall, the API is either great or it kind-of sucks, depending on what you want to do with it.

At the risk of sounding like a conspiracy theorist, I must say the API seems to be cleverly engineered to be useful for “normal” purposes and somewhat useless for SEO. After all, only SEO workers really need accurate ranking data and more than 64 results per keyword phrase. Typical search engine users rarely move beyond the first page of results, so the limitations don’t hurt them. The various mashup makers that cater to the common user are also unaffected. It’s only the SEOs (and the rare academic researcher) that would be dissatisfied with the imposed constraints.

Of course, I’m sure you can still imagine a few interesting uses for the API