The custom update checker integrates closely with the upgrade system already built into WordPress, producing a seamless user experience. Observe :

An upgrade notice for a privately hosted plugin.

The 'Plugin Information' window with placeholder data.

(Excuse my marketing-speak. It’s a thing. On with the practicalities of the matter.)

Download

• Client library (requires PHP5 and WP 2.8 or later)
• Example plugin
• Example metadata file

Quick-start Guide

This section describes the quickest way to get automatic updates working for your plugin. Here’s what you’ll need to do: create a metadata file for your plugin, host it somewhere publicly accessible, and tell the update checker where to find it.

Lets start with the metadata. Copy the JSON code below into a new file and replace the placeholder values with your plugin’s info.

{
    "name" : "My Cool Plugin",
    "slug" : "my-cool-plugin",
    "download_url" : "http://example.com/plugins/my-cool-plugin.zip",
    "version" : "2.0",
    "author" : "John Smith",
    "sections" : {
        "description" : "Plugin description here. Basic HTML allowed."
    }
}

(This is the minimal amount data required to make automatic updates work. In most cases, you will probably want to add a couple more fields. See the metadata docs below for details.)

Most of the fields should be pretty self-explanatory, with one possible exception – the “slug”. WordPress expects all plugins that support automatic updates to have a unique textual identifier called the “slug”. Normally, slugs are assigned by the official plugin directory. For a private/commercial plugin that’s hosted elsewhere you’ll have to make something up. If unsure, just use the plugin’s file name without the “.php” extension (my-cool-plugin/my-cool-plugin.php becomes my-cool-plugin).

Upload the metadata file you just created to your web server. It doesn’t matter where exactly you put the file or how you name it. The important thing is for its URL to be accessible from wherever someone might install your plugin.

Next, copy the “update-checker” directory from the client library archive to your plugin’s directory. Then fire up your favourite code editor and add the following lines to the top of your plugin file:

require 'plugin-updates/plugin-update-checker.php';
$MyUpdateChecker = new PluginUpdateChecker(
    'http://example.com/path/to/metadata.json',
    __FILE__,
    'your-chosen-slug'
);

(If you followed my advice and used the plugin’s file name as the slug, you can omit the third parameter of the PluginUpdateChecker constructor.)

And that, believe it or not, is it.

The PluginUpdateChecker class will handle the rest. It’ll check the metadata file every 12 hours and, if it discovers that a new version has been released, twiddle the right bits in the undocumented WP API to make it show up as a standard upgrade notification in the “Plugins” tab. Assuming you’ve set up the download_url correctly, users will be able to install the update with a single click.

The rest of this post will be devoted to a more in-depth discussion of the update checker class and the metadata format.

The PluginUpdateChecker class

This class is the core of the update checker. It’s also the only part of the updater that you should need to deal with unless you decide to  extend the library yourself.

Class constructor

All configuration settings should be specified by passing them to the PluginUpdateChecker constructor. It takes the following parameters :

• $metadataUrl – The full URL of the plugin’s metadata file.
• $pluginFile – The fully qualified path to the plugin file. In most cases you can simply use the __FILE__ constant here.
• $slug – The plugin’s ‘slug’. If not specified, the filename part of $pluginFile (sans “.php”) will be used as the slug.
• $checkPeriod – How often to check for updates (in hours). Defaults to checking every 12 hours. Set to zero to disable automatic update checks.
• $optionName – Where to store book-keeping info about updates. Defaults to “external_updates-$slug”.

checkForUpdates()

Manually trigger an update check. This is especially useful when you’ve disabled automatic checks by setting $checkPeriod (above) to zero. This method takes no parameters and returns nothing.

addQueryArgFilter($callback)

Register a callback for filtering query arguments. Whenever the update checker needs to retrieve the metadata file, it will first run each filter callback and attach the query arguments that they return to the metadata URL. This lets you pass arbitrary data to the server hosting the metadata. For example, commercial plugins could use it to implement some kind of authorization scheme where only users that have the right “key” get automatic updates.

The callback function will be passed an associative array of query arguments and should return a modified array. By default, the update checker will add these arguments to the metadata URL:

• installed_version – set to the currently installed version of the plugin.
• checking_for_updates – set to 1 if checking for updates, absent otherwise (i.e. when loading data for the “Plugin Information” box).

This method takes one parameter – the callback function.

addHttpRequestArgFilter($callback)

Register a callback for filtering the various options passed to the built-in helper function wp_remote_get that the update checker uses to periodically download plugin metadata. The callback function should take one argument – an associative array of arguments – and return a modified array or arguments. See the WP documentation on wp_remote_get for details about what arguments are available and how they work.

This method takes one parameter – the callback function.

addResultFilter($callback)

Register a callback for filtering plugin info retrieved from the metadata URL.

The callback function should take two arguments. If the metadata was retrieved successfully, the first argument passed will be an instance of PluginInfo (see the source for a description of this class). Otherwise, it will be NULL. The second argument will be the corresponding return value of wp_remote_get (see WP docs for details). The callback function should return a new or modified instance of PluginInfo or NULL.

This method takes one parameter – the callback function.

Metadata format

The automatic update system uses a JSON-based file format to describe plugins.  Essentially, the entire file is one big JSON-encoded object (AKA hash-table or associative array). Each field – or array key – represents a piece of information about the latest version of the plugin. The full description of all available fields is here.

For the sake of simplicity, both general metadata and update-related information are stored in the same file. If this is undesirable, you can replace the plain JSON file with a script that checks for the presence of the the “checking_for_updates” query parameter and emits just the update-related fields if its set to “1″.

In practice, dbDelta is fragile, undocumented and prone to fail silently when something goes wrong. It can only parse queries tailored to its idiosyncratic requirements, doesn’t know how to drop obsolete columns, isn’t aware of character sets and collation, chokes on extra whitespace… Nasty, but evidently Good Enough™ for most plugins, even for WordPress itself. It’s still better than doing schema updates manually, and in most cases you can work around the limitations.

Then again, sometimes you can’t. Just a few days ago I ran into a situation that dbDelta just couldn’t handle (due to the aforementioned lack of collation awareness). This prompted me to refactor dbDelta into something more flexible. The result was the blcTableDelta class – still somewhat finicky with regards to query syntax, but much more flexible otherwise.

In the remainder of this post discuss how my version compares to the built-in dbDelta function, explain how to use it, and provide the source code.

Advantages

My version of dbDelta is by no means perfect – or, for that matter, even particularly good. That would require implementing a full-blown SQL parser, which is clearly overkill for a utility function that gets called only once or twice per install. Instead, it’s just Better Enough™. Here’s how blcTableDelta fares when compared to dbDelta :

• More lenient with whitespace – no more ridiculous requirements like “there must be exactly two spaces between the words PRIMARY KEY and the definition of your primary key”.
• Detects changes not just in column data type, but also in collation settings and NULL/NOT NULL/AUTO INCREMENT flags.
• Can optionally remove obsolete columns and indexes.
• Returns a complete query log, including execution status and any MySQL error messages.
• Doesn’t choke on “IF NOT EXISTS”.
• Has slightly better documentation.

The downside is that this class isn’t nearly as battle-tested as the native dbDelta function. It also doesn’t support INSERT queries.

Usage

Here’s a simple example of how to use the blcTableDelta class :

//Define the table structure you want to achieve.
$sql = <<<EOM
prefix}example_table` (
	`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
	`name` varchar(100) NOT NULL DEFAULT '',
	`another_field` text  COLLATE latin1_general_cs    not null,
 
	PRIMARY    KEY   ( `id` ),
	INDEX named_index(name)
);
EOM;
 
//Call the static delta() method
list($messages, $query_log) = blcTableDelta::delta($sql);
 
//Output the execution log
print_r($messages);
print_r($query_log);

The static delta() method can take up to four arguments :

• $queries - a string containing one or more CREATE TABLE queries, separated by semicolons.
• $execute – whether to apply the required schema changes (True) or just return the generated queries (False). Defaults to executing the queries right away.
• $drop_columns – whether to remove columns that are present in the current table but not in the corresponding CREATE TABLE query (True), or leave them alone (False). Defaults to removing the columns.
• $drop_indexes – same as the previous argument, except for indexes.

The function returns an array with two elements. The first element is a list of human-readable messages explaining what schema changes would need to be performed to update the database. The second returned value is the query log. Each item of the query log array is also an array, with up to three keys :

• “query” – the generated SQL query.
• “success” – a True/False flag indicating whether the query was executed successfully. Present only if the $execute argument was set to True.
• “error_message” – the MySQL error that occurred when executing the query. Only meaningful when “success” = False.

Source Code

The source code is too long to include in the post, so I’ve placed it in a separate .zip archive. Download it here. Just place the .php file found therein somewhere inside your plugin’s directory and include it before calling blcTableDelta::delta.

It got me wondering – could one write a program that converts any image into pure CSS & HTML?

As it turns out, that’s already been done. However, I was pretty sure it could be done better – despite limiting the maximum output resolution to 100×100, the CSS/HTML code produced by the aforementioned converter was truly humongous. Surely I could improve on that. Thus I went ahead and wrote my own image to CSS converter anyway.

The result, lovingly code-named “Pure CSS Perversion”, produces remarkably space-efficient* HTML and CSS code for a wide range of input images. You can try it out here.

*Only in the context of image-to-CSS conversion. It’s laughably inefficient when compared to most image file formats.

Awesome Features

• Highly perverse HTML/CSS coding
  Almost as space-efficient as storing your images in uncompressed bitmaps!
• Glorious 256 color palette
  Evokes fond memories of DOS and VGA.
  
• Dithering support
  Greatly improves image quality with only an enormous increase in file size.
  
• Cross-browser compatibility
  Tested in Firefox 3.6, Opera 10.53, Chrome 6.0, and even Internet Explorer 8.
• Supports JPG, PNG and GIF
  Also supports PCX, ARJ and ponies. Okay, not really.

Examples

Click a screenshot to go to the corresponding converter results page.

You can find more examples on the converter’s homepage.

Descent Into Optimization

For those interested in such things, here is a detailed explanation of how the converter works, including a tedious step-by-step account of how I arrived to the current algorithm. You can also download the source code.

1. Starting Simple
The simplest way to encode an image as HTML/CSS is to convert each image pixel to a separate <div> element 1×1 pixels in size and use an inline style attribute set the div’s background color to that of the pixel. Example :

<div class="css-image">
   <div class="row">
      <div style="background: #abcdef; width: 1px; height: 1px;"></div>
      <div style="background: #abcdef; width: 1px; height: 1px;"></div>
      <!-- more pixels -->
   </div>
   <div class="row">
      <div style="background: #abcdef; width: 1px; height: 1px;"></div>
      <div style="background: #abcdef; width: 1px; height: 1px;"></div>
      <!-- more pixels -->
   </div>
   <!-- more rows/scanlines -->

This produces good-looking results, but the amount of HTML code quickly balloons out of control. Even a modest 200×200 picture would result take up over 1 MB when encoded in this way, which is wasteful and can even crash some browsers.

2. Palletise and Cassify
The easiest way to reduce the file size of the “pure CSS” image is to convert it to a palette-based format and create a new CSS class for each palette color. This is what my first version of the image-to-CSS converter did.

<style>
div { width: 1px; height: 1px; float: left; }
.color1 { background: #abcdef; }
.color2 { background: #f00000; }
/*....*/
</style>
...
<div class="color1"></div><div class="color2"></div>

As you can see, I’ve also moved the rules that apply to all DIVs to a separate CSS block.

3. Run-Length Encoding
To further reduce the code size, we can detect continuous runs of the same color and encode each of them using only a single, wider <div>. To avoid explicitly writing out the width of each element, lets generate CSS classes for widths :

<style>
.width2 { width: 2px; }
.width3 { width: 3px; }
.width4 { width: 4px; }
.width5 { width: 5px; }
</style>
...
<div class="color1 width3"></div>

All <div>’s have a 1-pixel width by default, so we don’t need a “width1″ class.

4. Shorter Class Names
All those “colorA” and “widthB” class names are wastefully long. Lets shorten them to a single-letter prefix and a hex-encoded index.

<style>
.c1 { background: #abcdef; }
.c2 { background: #f00000; }
/* ... */
.w9 { width: 9px; }
.wa { width: 10px; }
.wb { width: 11px; }
</style>
<div class="c2a wf"></div><div class="c2 wa2"></div>...

5. Default Colors
We can squeeze out a few more bytes by determining the color that’s most frequently used in a given row and adding it as the background color to the container element. Then we can omit the color class from those elements that match the most common color.

<div class="c2">
    <div class="c1"></div>
    <div><!-- matches the row's background color --></div>
    <div class="c3"></div>
</div>

6. Perversion Begins
Hmm, look at all those byte-hogging <div> elements – each eating up 3 bytes on tag name alone. Plus, we’re forced to waste another 6 bytes on the completely useless closing tag! That will not do.

Luckily, the closing tags are optional for <p> and <li> elements. We can rework our HTML structure to make use of this.

<div class="css-image"><ul>
   <li>
      <p class="c1 w2">
      <p class="c3">
      <p class="cba w5">
      <!-- more pixels -->
   <li>
      <p class="c1 w2">
      <p class="c3">
      <p class="cba w5">
      <!-- more pixels -->
 
   <!-- more rows -->
<ul></div>

7. Going Classless
We’re also wasting 5 bytes/div on the “class” attribute, so lets get rid of it and use custom attributes instead.

<style>
p[c="1"],li[c="1"]{ background: #abcdef; }
p[c="2"],li[c="2"]{ background: #f00000; }
/*...*/
p[w="9"]{ width: 9px; }
p[w="a"]{ width: 10px; }
p[w="b"]{ width: 11px; }
</stlye>
<!-- ... -->
<p c="1"><p c="16" w="5"><p c="ab">

It may not validate, but it works. Who cares about validation anyway?

8. Even More Perversion
Well, that’s probably the limit of size optimization. We need to specify the background color and width somehow, so we can’t completely eliminate the attributes. We could remove the quotes from attribute values and format our output like “<p c=1 w=ab>”, but that’s it. Right?

Actually, there is a way to compress the output even more. We can use the attribute name itself to specify pixel color and width. Observe :

<style>
p[c1],li[c1]{ background: #abcdef; }
p[c2],li[c2]{ background: #f00000; }
/*...*/
p[w9]{ width: 9px; }
p[wa]{ width: 10px; }
p[wb]{ width: 11px; }
</stlye>
<!-- ... -->
<p c1><p c16 w5><p cab>

Yes, that actually works. It even works in Internet Explorer 8 (when in standards mode).

9. Huffman And Friends
In any given image, some colors are more common than others. For example, a picture of a lush countryside might have a lot of greens, whereas a beach photo would be predominantly blue and yellow. Our current algorithm is not aware of that – it simply enumerates all colors in sequential order, assigning attribute names as it goes. There’s no relation between the color and the codename it gets.

We can reduce output size by assigning shorter attribute names to more common colors. This is known as Huffman coding, and is one of the simplest algorithms used for lossless data compression.

The general idea is thus :

• Iterate over the image and record how many times each color is seen.
• Sort the list of colors by their frequency.
• Assign shorter names to more common colors, longer ones to rare ones.

In fact, we can use this idea to eliminate the color vs. width attribute distinction completely. Lets look at our HTML stucture again :

<p c1><p c16 w5><p cab><p ce2><p c1 w12><!-- ... -->

If we forget the “<p …>” wrappers for a moment…

c1 c16 w5 cab ce2 c1 w12

…it’s just a string of codes! Some more common, some less. We don’t need to know what each of them means to assign a optimally short and unique name to each one.

The actual implementation is somewhat complicated, but the HTML/CSS it produces is very space-efficient. This is what the output of my final algorithm looks like (line breaks inserted for clarity) :

<style>
p[a]{width:6px;}
p[b],li[b]{background:#dfdbda;}
p[c]{width:9px;}
p[d],li[d]{background:#dfe3e8;}
p[e],li[e]{background:#d4d5dc;}
/*...*/
</style>
<!-- ... -->
<div class="css-image"><ul><li y><p d f><p b c><p d a><p b xb><p o><p b _g><p e><p b l><!-- etc -->

Improve On That

Unbelievable though it may seem, this is still not the limit of size optimization for “pure CSS” images. For example, you could use variable-width right and left borders on the <p> element to encode additional pixels in a single element. This would decrease the HTML size at least by another 20%. You could also replace the primitive RLE algorithm I used in my converter with something more advanced. There might also be some interesting HTML hacks I haven’t even thought about.

I’ll leave that as an exercise to the reader

So how does one show different ads to different visitors? Normally you could use the excellent Who Sees Ads plugin, but that doesn’t work for my site. I’m using the WP-SuperCache caching plugin which is incompatible with Who Sees Ads.

Instead, I wrote a JavaScript function that can analyse the HTTP referrer to distinguish between four types of visitors :

• People who accessed your page directly, e.g. by typing in the URL (no referrer).
• People who clicked an internal link to access the page (referrer from the same domain).
• Search engine visitors; people who arrived via Google or Yahoo! (referrer matches one of the most popular search engines).
• External visitors. This is basically a catch-all category for everyone who found your link on a third-party site that’s not a search engine. For example, this would include external forums and web directories.

Here’s the script :

function get_referrer_type(){
	var ref = document.referrer;
 
	if ( ref.length == 0 ){
		return 'none';
	}
 
	function getHostname(str) {
		var re = /^(?:f|ht)tp(?:s)?\:\/\/([^\/]+)/im;
		var matches = str.match( re );
		if ( matches ){
			return matches[1].toString();
		}
		return '';
	}
	if ( getHostname(ref) == document.location.host ){
		return 'internal';
	}
 
	var SE = ['/search?', '.google.', 'web.info.com', 'search.', 
		'del.icio.us/search', 'soso.com', '/search/', '.yahoo.',
		'.ask.'];
	for ( var source in SE){
		if (ref.indexOf(SE[source]) !=-1) {
			return 'search';
		}
	}
 
	return 'external';
}

The function get_referrer_type() returns one of “none”, “internal”, “search” or “external” based on the referrer info.

And here’s the script that I used to show different  AdSense ads to different people :

function get_top_adsense_ad(){
	var default_slot = '1111111111';
	var code = '<'+'script type="text/javascript"><!--\ngoogle_ad_client = "pub-0000000000000000";google_ad_slot = "%ad_slot%";google_ad_width = 336;google_ad_height = 280;\n//--><'+'/script><'+'script type="text/javascript" src="http://pagead2.googlesyndication.com/pagead/show_ads.js"></'+'script>';
	var slots = {
		'none' : '2222222222',
		'internal' : '3333333333',
		'external' : '4444444444',
		'search' : '5555555555',
	};
 
	var ref_type = get_referrer_type();
	if ( typeof(slots[ref_type]) != 'undefined' ){
		code = code.replace('%ad_slot%', slots[ref_type]);
	} else {
		code = code.replace('%ad_slot%', default_slot);
	}
 
	return code;
}
 
document.write(get_top_adsense_ad());

Unlike get_referrer_type(), this script is optimized for my site/AdSense account. Don’t try using it without modification If you want to use it, replace “pub-XXXXXXXX”, the ad slot IDs and ad width/height settings with your own values (you can find them in your AdSense ad code).

As for the experiment, I’ll post the results sometime next week.

Below you’ll find a list of 12 great utility-style jQuery plugins. While invisible to the user, they can be invaluable to the developer.

1. Hotkeys

The HotKeys plugin makes it very easy to hook almost any key or key combination. Great for adding keyboard shortcuts to your web-app.

2. Mouse Wheel

Detect when the user scrolls the mouse wheel. Note : In theory, the plugin supports all major browsers. However, the current version may report inconsistent scroll deltas in different browsers.

3. XML to JSON

XML to JSON lets you convert XML documents to JSON. Handy if you find plain JS objects easier to use than the XML DOM, or if you’re building a mashup with heterogeneous data coming in from a bunch of APIs and you want to convert it all to JSON for simplicity.

4. URL Parser

jQuery URL parser processes URLs and provides you with easy access to info about their components – the protocol, host, path, query string, and so on. On top of that, it also has some neat extra features, like the ability access query parameter values by parameter name.

5. Taconite

With Taconite, you can update multiple DOM elements with a single AJAX request. This can considerably simplify your client-side code and helps cut down on bandwidth use.

6. AJAX Manager

Queue, order, abort, block, cache and otherwise manipulate your AJAX requests with the AJAX manager plugin.

7. Calculation

The jQuery Calculation Plugin lets you extract numeric data from HTML elements and apply a number of common mathematical operations to the results. It includes typical aggregation functions like sum(), avg(), min() and max() and also lets you define your own equations.

8. Freeze/Unfreeze Events

This mini-plugin can be used to temporarily stop all jQuery events assigned to a specific element (or a group of elements) from firing.

9. Cookie Handling

When it comes to cookies, there are two jQuery plugins that stand out :

• Cookie is simple and light-weight. Just the bare essentials  – you can read/write/delete cookies, and that’s it.
• Cookies is the advanced option. It can filter cookies using regexps, test if the browser is set to accept cookies, bind DOM element contents to cookies, and do automatic JSON serialization.

10. Metadata

The Metadata plugin provides a unified way to extract different formats of metadata from the DOM. It can extract metadata from classes, attributes, child elements or HTML5 data- attributes.

11. Navigation History For AJAX Pages

Add the coveted back-button support to your AJAX apps with the powerful Address plugin. There’s a number of other plugins that deal with #anchor-based navigation, but Address is a clear winner – it has the cleanest API and the best documentation.

12. Timers

jQuery Timers provides a more elegant, jQuery-flavoured way to deal with setTimetout() and setInterval(). You can apply timers to DOM elements, set the number of times a given timer will execute, create named timers, and more.

There needs to be a way to link to a page on Reddit and automatically apply Readability.

That seemed like an excellent idea to me. In case you haven’t heard about it yet, Readability is a bookmarklet that can take an article or blog post with even the most cluttered, ad-infested and eye-bleedingly ugly design and turn it into a clean and readable work of art. As a side-effect, it also removes most kinds of ads.

Of course, you could just install the bookmarklet for your own use and have that be the end of it. But if you regularly share links to interesting articles with friends, or post them to social bookmarking sites, it would sure be handy to have a way give others the benefit of Readability without forcing them to install anything.

So I went ahead and created a website that lets you do just that – MakeReadable.com. Go check it out.

Here’s how to use it :

1. Enter a URL of the page  that you want to make more readable. Pages with lots of text – like news articles or blog posts – work best.
2. Click the “Make Readable” button (or press Enter).
3. Wait a few seconds while the server processes the page and redirects you to a new URL that already has Readability applied.
4. Share the URL of the new page with your friends, etc.

An even easier way to use this service is to simply append the URL of the page you want processed after http://makereadable.com/, like this :

http://makereadable.com/example.com/someblog/article-123.htm

Behind the scenes, MakeReadable.com also performs several other optimizations, like normalizing the HTML and stripping out all other scripts and most CSS. Since Readability completely restyles the page anyway, this post-processing makes the resulting page smaller and faster-loading (in theory) without hurting its looks.

Thoughts? Feature suggestions? Leave a comment below.

Dust-Me Selectors

Dust-Me Selectors helps you find unused CSS selectors. You can use it to test an individual page, or let it spider your entire site. Results are presented in a nice color-coded report and can be exported as CSV.

Alternative : CSS Usage

Inline Code Finder for Firebug

Finds and highlights HTML elements with suspicious hacks like inline JavaScript events, inline styles and javascript: links. It also presents a summary report and calculates an overall score based on how many such elements were encountered on the page.

Window Resizer

Lets you quickly resize your browser window to a number of common screen sizes so that you can see how you page would look at a different display resolution.

Alternative : FireSizer

RESTClient

RESTClient is a very handy tool for web service developers, allowing you to directly test and analyze RESTful APIs. It also includes partial support for the WebDAV protocol.

JSONView

Lets you view JSON data directly in the browser and supports syntax highlighting and code folding. Overall, it’s another simple yet useful extension for developers working with WEB APIs.

Remove Cookie(s) for Site

Remove all cookies from the current site with a single click. If you need something more advanced, check out the alternatives below.

Alternatives : Add N Edit Cookies, FireCookie

CookieSwap

Lets you quickly swap all your cookies so that you can easily switch between different user accounts without logging in/out all the time. While this add-on is primarily advertised as a way to manage multiple webmail/Amazon/Google accounts, it can also be useful to web developers – e.g. for testing how your site behaves from the point of view of normal visitors vs. logged-in users, or users with different privilege levels.

TamperData

With TamperData, you can intercept HTTP(S) requests made by Firefox and view/modify their headers and POST data. All requests are logged and can also be exported as XML or displayed on a graph. Handy for security testing and catching certain breeds of elusive server-side bugs.

Alternatives : Modify Header, UrlParams

View Dependencies

Essentially a simpler alternative to YSlow, View Dependencies adds a new tab to the Page Info window, listing all the files that were loaded to display the current page.

HackBar

HackBar is a large collection of simple penetration testing tools, useful for security-auditing your site and testing it for common vulnerabilities. You can convert strings to hex and back, encrypt text with SHA-1, SHA-256 or ROT13 (if only just for fun), try a number of typical SQL injection/XSS tricks, manipulate POST data, spoof the referer, and much more.

Related : Access Me, XSS Me

iMacros for Firefox

“Whatever you do with Firefox, iMacros can automate it.” It can automatically fill forms, download & upload files, scrape web pages, memorize passwords, and so on. iMacros really makes tasks like web application testing, data mining and mashup creation much easier.

TableTools

Sort, filter or copy any HTML table. You can filter each column either by selecting a value from a drop-down menu or by searching for a specific keyword (regular expressions are also supported). TableTools will automatically recognize frequently used data types like numbers, timestamps and IP addresses, and sort/filter them intelligently.

SQLite Manager

Manage any SQLite database on your computer. This add-on has all the features you would expect from a full-featured DB manager – you can view the database structure, create, modify and search tables and views, manage indexes, export and import data, and manipulate pretty much any other aspect of the database.

Pencil

Make diagrams, build GUI prototypes and sketch website mockups with this awesome Firefox add-on!

Flashbug

A Firebug extension that logs all the SWF trace output to the Firebug console. Can be invaluable for debugging Flash-based applications and games.

FireLogger

Debug your server-side Python and PHP code from within the browser. After installing this Firebug extension and the appropriate Python/PHP library, you will be able to send logging information from your server-side scripts directly to the FireBug console.

Alternative : FirePHP

There are some (though very few they may be) edge case where regular expressions might work better, so I will discuss both approaches in this post.

Extracting Tags With DOM

PHP 5 comes with a usable DOM API built-in that you can use to parse and manipulate (X)HTML documents. For example, here’s how you could use it to extract all link URLs from a HTML file :

//Load the HTML page
$html = file_get_contents('page.htm');
//Create a new DOM document
$dom = new DOMDocument;
 
//Parse the HTML. The @ is used to suppress any parsing errors
//that will be thrown if the $html string isn't valid XHTML.
@$dom->loadHTML($html);
 
//Get all links. You could also use any other tag name here,
//like 'img' or 'table', to extract other tags.
$links = $dom->getElementsByTagName('a');
 
//Iterate over the extracted links and display their URLs
foreach ($links as $link){
	//Extract and show the "href" attribute. 
	echo $link->getAttribute('href'), '<br>';
}

In addition to getElementsByTagName() you can also use $dom->getElementById() to find tags with a specific id. For more complex tasks, like extracting deeply nested tags, XPath is probably the way to go. For example, to find all list items with the class “foo” containing links with the class “bar” and display the link URLs :

//Load the HTML page
$html = file_get_contents('page.htm');
//Parse it. Here we use loadHTML as a static method
//to parse the HTML and create the DOM object in one go.
@$dom = DOMDocument::loadHTML($html);
 
//Init the XPath object
$xpath = new DOMXpath($dom);
 
//Query the DOM
$links = $xpath->query( '//li[contains(@class, "foo")]//a[@class = "bar"]' );
 
//Display the results as in the previous example
foreach($links as $link){
	echo $link->getAttribute('href'), '<br>';
}

For more information about DOM and XPath see these resources :

• DOM API documentation
• W3Schools XPath tutorial
• XPath language specification

Honourable mention : Simple HTML DOM Parser is a popular alternative HTML parser for PHP 5 that lets you manipulate HTML pages with jQuery-like ease. However, I personally wouldn’t recommend using it if you care about your script’s performance, as in my tests Simple HTML DOM was about 30 times slower than DOMDocument.

Extracting Tags & Attributes With Regular Expressions

There are only two advantages to processing HTML with regular expressions – availability and edge-case performance. While most parsers require PHP 5 or later, regular expressions are available pretty much anywhere. Also, they are a little bit faster than real parsers when you need to extract something from a very large document (on the order of 400 KB or more). Still, in most cases you’re better off using the PHP DOM extension or even Simple HTML DOM, not messing with convoluted regexps.

That said, here’s a PHP function that can extract any HTML tags and their attributes from a given string :

/**
 * extract_tags()
 * Extract specific HTML tags and their attributes from a string.
 *
 * You can either specify one tag, an array of tag names, or a regular expression that matches the tag name(s). 
 * If multiple tags are specified you must also set the $selfclosing parameter and it must be the same for 
 * all specified tags (so you can't extract both normal and self-closing tags in one go).
 * 
 * The function returns a numerically indexed array of extracted tags. Each entry is an associative array
 * with these keys :
 * 	tag_name	- the name of the extracted tag, e.g. "a" or "img".
 *	offset		- the numberic offset of the first character of the tag within the HTML source.
 *	contents	- the inner HTML of the tag. This is always empty for self-closing tags.
 *	attributes	- a name -> value array of the tag's attributes, or an empty array if the tag has none.
 *	full_tag	- the entire matched tag, e.g. '<a href="http://example.com">example.com</a>'. This key 
 *		          will only be present if you set $return_the_entire_tag to true.	   
 *
 * @param string $html The HTML code to search for tags.
 * @param string|array $tag The tag(s) to extract.							 
 * @param bool $selfclosing	Whether the tag is self-closing or not. Setting it to null will force the script to try and make an educated guess. 
 * @param bool $return_the_entire_tag Return the entire matched tag in 'full_tag' key of the results array.  
 * @param string $charset The character set of the HTML code. Defaults to ISO-8859-1.
 *
 * @return array An array of extracted tags, or an empty array if no matching tags were found. 
 */
function extract_tags( $html, $tag, $selfclosing = null, $return_the_entire_tag = false, $charset = 'ISO-8859-1' ){
 
	if ( is_array($tag) ){
		$tag = implode('|', $tag);
	}
 
	//If the user didn't specify if $tag is a self-closing tag we try to auto-detect it
	//by checking against a list of known self-closing tags.
	$selfclosing_tags = array( 'area', 'base', 'basefont', 'br', 'hr', 'input', 'img', 'link', 'meta', 'col', 'param' );
	if ( is_null($selfclosing) ){
		$selfclosing = in_array( $tag, $selfclosing_tags );
	}
 
	//The regexp is different for normal and self-closing tags because I can't figure out 
	//how to make a sufficiently robust unified one.
	if ( $selfclosing ){
		$tag_pattern = 
			'@<(?P<tag>'.$tag.')			# <tag
			(?P<attributes>\s[^>]+)?		# attributes, if any
			\s*/?>					# /> or just >, being lenient here 
			@xsi';
	} else {
		$tag_pattern = 
			'@<(?P<tag>'.$tag.')			# <tag
			(?P<attributes>\s[^>]+)?		# attributes, if any
			\s*>					# >
			(?P<contents>.*?)			# tag contents
			</(?P=tag)>				# the closing </tag>
			@xsi';
	}
 
	$attribute_pattern = 
		'@
		(?P<name>\w+)							# attribute name
		\s*=\s*
		(
			(?P<quote>[\"\'])(?P<value_quoted>.*?)(?P=quote)	# a quoted value
			|							# or
			(?P<value_unquoted>[^\s"\']+?)(?:\s+|$)			# an unquoted value (terminated by whitespace or EOF) 
		)
		@xsi';
 
	//Find all tags 
	if ( !preg_match_all($tag_pattern, $html, $matches, PREG_SET_ORDER | PREG_OFFSET_CAPTURE ) ){
		//Return an empty array if we didn't find anything
		return array();
	}
 
	$tags = array();
	foreach ($matches as $match){
 
		//Parse tag attributes, if any
		$attributes = array();
		if ( !empty($match['attributes'][0]) ){ 
 
			if ( preg_match_all( $attribute_pattern, $match['attributes'][0], $attribute_data, PREG_SET_ORDER ) ){
				//Turn the attribute data into a name->value array
				foreach($attribute_data as $attr){
					if( !empty($attr['value_quoted']) ){
						$value = $attr['value_quoted'];
					} else if( !empty($attr['value_unquoted']) ){
						$value = $attr['value_unquoted'];
					} else {
						$value = '';
					}
 
					//Passing the value through html_entity_decode is handy when you want
					//to extract link URLs or something like that. You might want to remove
					//or modify this call if it doesn't fit your situation.
					$value = html_entity_decode( $value, ENT_QUOTES, $charset );
 
					$attributes[$attr['name']] = $value;
				}
			}
 
		}
 
		$tag = array(
			'tag_name' => $match['tag'][0],
			'offset' => $match[0][1], 
			'contents' => !empty($match['contents'])?$match['contents'][0]:'', //empty for self-closing tags
			'attributes' => $attributes, 
		);
		if ( $return_the_entire_tag ){
			$tag['full_tag'] = $match[0][0]; 			
		}
 
		$tags[] = $tag;
	}
 
	return $tags;
}

Usage examples

Extract all links and output their URLs :

$html = file_get_contents( 'example.html' );
$nodes = extract_tags( $html, 'a' );
foreach($nodes as $link){
	echo $link['attributes']['href'] , '<br>';
}

Extract all heading tags and output their text :

$nodes = extract_tags( $html, 'h\d+', false );
foreach($nodes as $node){
	echo strip_tags($link['contents']) , '<br>';
}

Extract meta tags :

$nodes = extract_tags( $html, 'meta' );

Extract bold and italicised text fragments :

$nodes = extract_tags( $html, array('b', 'strong', 'em', 'i') );
foreach($nodes as $node){
	echo strip_tags( $node['contents'] ), '<br>';
}

The function is pretty well documented, so check the source if anything is unclear. Of course, you can also leave a comment if you have any further questions or feedback.

Side-note : No, I haven’t been completely consumed by an inexorable desire to build fun graphical toys instead of practically useful software, why do you ask? In fact, there’ll be some WordPress-related announcements later this week…

Learning To Fly Fall

While I can’t claim intimate familiarity with the algorithms the aforementioned games use to simulate fire or explosions, it’s pretty easy to come up with a way to implement falling sand/flowing water. First, you need to figure out under what conditions something can fall/flow downwards. For a basic simulation this could be just a small set of very simple rules :

• For each entity that is affected by gravity
  • If there’s an empty space below it, move it down.
  • If there’s an empty space down and to the left, move it down and to the left.
  • If there’s an empty space down and to the right, move it down and to the right.

Next, you need to decide how to represent the simulation world and all the objects (like immovable walls, moving water particles et cetera) that it contains. One of the most common solutions is to use a two-dimensional array where each array element is an numeric ID of the material contained in that map tile. For example, 0 = air (or an empty square), 1 = wall, 2 = water, and so on.

In most falling sand games each array element would correspond to a single pixel on the game’s screen. However, in my example application I used 20×20 pixel tiles instead to make the underlying simulation behaviour easier to see.

If possible, use byte or char as the array element datatype. Using int’s might seem more straightforward, but it would also take up at least four times more memory and hurt performance. It’s also a good idea to make the array just a bit bigger than the actual map dimensions and create a single-tile “buffer zone” around the map so that you can avoid checking for boundary conditions during the simulation.

Finally, during the simulation step we just loop over the entire array and apply the simple rules I mentioned above to each tile that contains material that should be affected by gravity.

Getting It Right

If you go ahead and implement the above algorithm as written, you will get a simulation that kind-of works, but quickly grinds to a halt on big maps and sometimes behaves in unexpected ways. For example, you might see sand tiles that seem to “teleport” to the bottom of the map instead of falling gradually and water that always flows left after hitting a free-standing ground tile (even though there’s a perfectly suitable channel available on the right side). Lets see what we can do about that.

Keeping track of updated tiles

The “teleportation” behaviour can happen if our simulation algorithm doesn’t keep track of which tiles it has already moved around during the current simulation step. For example, first it loops over the top row of the map, finds a water tile A and moves it downwards one row. Then it loops over the second row, finds the tile A, moves it again, goes to the third row, moves the same water tile again… and so on, until the water tile A hits a solid tile or falls off the map.

To prevent this bug we need to somehow remember which tiles have been updated already. Since I wanted to keep the memory use low, I used the most-significant bit of each map array element to indicate whether that tile was last updated on an even-numbered simulation step (0) or an odd-numbered step (1). Then, at the start of each step, the algorithm calculates what the flag should be for tiles updated in that step :

update_mask = byte ( ( simulationFrameCount & (long)1 ) << 7 );
//this basically takes the modulo of frame-count / 2 and shifts the result 7 places to the left

…then sets the msb of each tile it updates to that value and automatically skips tiles that have it already set to that value.

Pseudo-random movement

When a falling particle hits a free-standing fixed tile on a 2D map it has two choices – it can either move down and left, or down and right. In the naive algorithm I discussed above it will always try to move left first, leading to weirdness. Obviously we want it to choose at random, but on a big map generating a new random number for each tile would be slow. Instead, lets use the same approach as for the “teleportation” problem and move it either diagonally left if the current simulation step is even-numbered and or diagonally right if it’s odd-numbered. This can still introduce some artifacts in the simulation behaviour, but they’re rarer and and significantly less noticeable.

Optimization

Speaking of optimization, we don’t really need to update each gravity-enabled tile every time. Many of them will eventually end up in a situation where they can’t move anywhere (e.g. water at the bottom of a puddle or sand resting on a flat surface), so it would be nice if we could quickly skip them and not waste our time trying to apply the simulation rules to these static tiles. This would help performance quite a bit.

The way I implemented is with the help with another bit-flag. If the simulation algorithm discovers that it can’t move a tile in any direction, it will set the tile’s “static” bit-flag. It will also skip any tiles that have the “static” bit-flag set.

There’s some additional book-keeping we need to do to make this work as intended. When an empty space is created somewhere in the map (e.g. by a water tile moving down) we need to re-activate any gravity-enabled tiles that are directly above the newly created hole, so that they can fall/flow down into that space.

Possible Improvements

The result is a pretty basic and plain simulation that is useful to explain the basic principle, but it wouldn’t really qualify as a full game. Here are some ideas you could use to improve it :

• Make water flow horizontally if there’s another water tile on top of it. Consider adding a simple pressure model [link to my prev. article].
• Implement fluid density. This one is easy – if the current tile is more dense than the tile below it, swap them.
• Add tile generators, e.g. a faucet that creates a water tile below itself on each simulation step (included in my demo application).
• To improve performance, use a quadtree to keep track of which regions of the map have interacting tiles and quickly skip inactive regions.
• Explore the source code of existing games and look for interesting ideas. Some open-source examples : wxSand, SDL Sand.

Live Demo & Source

Like my previous fluid simulation app, this one was also written in Processing. In addition to a working implementation of the algorithm discussed above it also lets you draw and erase water/ground/faucet tiles using your mouse, pan around the map with the WASD keys, zoom in/out with the mouse wheel and more. However, be warned that the map rendering code is not very optimized so the simulation will degenerate into a slideshow if you zoom all the way out, regardless of how fast your system is.

• Live demo  (requires Java)
• Source code

Big Words, or The Theoretical Part

Fluid dynamics is a complex topic. If you so much as look it up in Wikipedia, you will be immediately assaulted by integrals, differentials and other agents of mathematical insanity. To accurately model a fluid you would need a quite an in-depth understanding of physics and calculus. And while that’s certainly not something one couldn’t figure out given enough time, “proper” fluid simulation is still extremely rare in non-academic applications and games because it’s slow.

One way to create a faster (though less accurate) simulation is by using cellular automata to represent the water. A cellular automaton is basically a grid where each cell can be in one of a finite number of states, plus a set of rules that determine how a cell can change from one state to another. The rules are typically local – that is, they consider only the current cell and it’s direct neighbours when determining the new state. During each step of the simulation, you simply loop through the entire grid and apply the rules to each cell.

When simulating water or other fluids it can more appropriate to store a continuous value (e.g. water mass) in each cell, instead of using discrete states.

CA-Based Water Simulation

Before we go into details, lets take a look at the promised online demo. It will give you a good idea about what you can achieve with this approach :

(klicken Sie auf das Bild; Java required)

Alright, on to the code stuff. Lets try simulating the most common fluid – water.

The basic data structure

//Map dimensions
final int map_width = 16;
final int map_height = 16;
 
//Block types
final int AIR = 0;
final int GROUND = 1;
final int WATER = 2;
 
//Data structures
int[][] blocks = new int[map_width+2][map_height+2];
 
float[][] mass = new float[map_width+2][map_height+2],
          new_mass = new float[map_width+2][map_height+2];

I use two two-dimensional arrays to represent the simulation world. The blocks array defines a basic “map” where each cell can contain ground (solid, stops water flow), air (an empty cell, water can flow in) or water. Each water cell also has a corresponding entry in the mass array that defines how much water it contains. All water cells/blocks start out with one unit of water.

There’s also a third array – new_mass that is used to store intermediate mass values when running the simulation. Operating directly on the mass array is unadvisable because you would get various sideffects like water spreading at different speeds depending on the order in which you update the cells.

Overall algorithm

The main idea is to treat water as a slightly compressible liquid, as described in this article. In terms of implementation, this means that if there are two or more water cells stacked vertically, the bottom cells will be able to hold slightly more water than normal. This way we don’t need to explicitly track pressure to make the water level equalize in communicating vessels – we can just look at how much excess water a cell has, and move it upwards if required.

Lets set up the exact fluid properties as constants :

//Water properties
final float MaxMass = 1.0; //The normal, un-pressurized mass of a full water cell
final float MaxCompress = 0.02; //How much excess water a cell can store, compared to the cell above it
final float MinMass = 0.0001;  //Ignore cells that are almost dry

For example, if one cell contains 1.0 units of water, the cell below it should contain up to 1.02 units, the cell below that one should contain 1.04, the next one 1.06, and so on. There is one however special case that we need to consider to get a plausible simulation – what if the if the top cell contains less than MaxMass units of water? In my implementation the bottom cell will contain a proportionally smaller excess amount, not the usual mass_of_the_cell_above + MaxCompression units.

Based on the above, we can simulate water movement by looping over the mass array and applying these cellular automaton rules to each cell :

1. Take the mass of the current cell and the cell below it and figure out how much water the bottom cell should contain. If it has less than that, remove the corresponding amount from the current cell and add it to the bottom cell.
2. Check the cell to the left of this one. If it has less water, move over enough water to make both cells contain the same amount.
3. Do the same thing for the right neighbour.
4. Do the same thing as in step 1., but for the cell above the current one.

While doing this you also need to keep track of how much water the current cell still has remaining, or you could end up with negative-mass water blocks.

After the new mass of each cell has been calculated, you can safely copy over the new values to the mass array. You also need to update the blocks array so that cells that are now dry get marked as empty and vice versa.

Here’s the simulation code :

void simulate_compression(){
  float Flow = 0;
  float remaining_mass;
 
  //Calculate and apply flow for each block
  for (int x = 1; x <= map_width; x++){
     for(int y = 1; y <= map_height; y++){
       //Skip inert ground blocks
       if ( blocks[x][y] == GROUND) continue;
 
       //Custom push-only flow
       Flow = 0;
       remaining_mass = mass[x][y];
       if ( remaining_mass <= 0 ) continue;
 
       //The block below this one
       if ( (blocks[x][y-1] != GROUND) ){
         Flow = get_stable_state_b( remaining_mass + mass[x][y-1] ) - mass[x][y-1];
         if ( Flow > MinFlow ){
           Flow *= 0.5; //leads to smoother flow
         }
         Flow = constrain( Flow, 0, min(MaxSpeed, remaining_mass) );
 
         new_mass[x][y] -= Flow;
         new_mass[x][y-1] += Flow;   
         remaining_mass -= Flow;
       }
 
       if ( remaining_mass <= 0 ) continue;
 
       //Left
       if ( blocks[x-1][y] != GROUND ){
         //Equalize the amount of water in this block and it's neighbour
         Flow = (mass[x][y] - mass[x-1][y])/4;
         if ( Flow > MinFlow ){ Flow *= 0.5; }
         Flow = constrain(Flow, 0, remaining_mass);
 
         new_mass[x][y] -= Flow;
         new_mass[x-1][y] += Flow;
         remaining_mass -= Flow;
       }
 
       if ( remaining_mass <= 0 ) continue;
 
       //Right
       if ( blocks[x+1][y] != GROUND ){
         //Equalize the amount of water in this block and it's neighbour
         Flow = (mass[x][y] - mass[x+1][y])/4;
         if ( Flow > MinFlow ){ Flow *= 0.5; }
         Flow = constrain(Flow, 0, remaining_mass);
 
         new_mass[x][y] -= Flow;
         new_mass[x+1][y] += Flow;
         remaining_mass -= Flow;
       }
 
       if ( remaining_mass <= 0 ) continue;
 
       //Up. Only compressed water flows upwards.
       if ( blocks[x][y+1] != GROUND ){
         Flow = remaining_mass - get_stable_state_b( remaining_mass + mass[x][y+1] );
         if ( Flow > MinFlow ){ Flow *= 0.5; }
         Flow = constrain( Flow, 0, min(MaxSpeed, remaining_mass) );
 
         new_mass[x][y] -= Flow;
         new_mass[x][y+1] += Flow;   
         remaining_mass -= Flow;
       }
 
 
     }
  }
 
  //Copy the new mass values to the mass array
  for (int x = 0; x < map_width + 2; x++){
    for (int y = 0; y < map_height + 2; y++){
      mass[x][y] = new_mass[x][y];
    }
  }
 
  for (int x = 1; x <= map_width; x++){
     for(int y = 1; y <= map_height; y++){
       //Skip ground blocks
       if(blocks[x][y] == GROUND) continue;
       //Flag/unflag water blocks
       if (mass[x][y] > MinMass){
         blocks[x][y] = WATER;
       } else {
         blocks[x][y] = AIR;
       }
     }
  }
 
  //Remove any water that has left the map
  for (int x =0; x < map_width+2; x++){
    mass[x][0] = 0;
    mass[x][map_height+1] = 0;
  }
  for (int y = 1; y < map_height+1; y++){
    mass[0][y] = 0;
    mass[map_width+1][y] = 0;
  }
 
}

And here’s the function that calculates how to distribute a given amount of water between two vertically adjacent cells :

//Returns the amount of water that should be in the bottom cell.
float get_stable_state_b ( float total_mass ){
  if ( total_mass <= 1 ){
    return 1;
  } else if ( total_mass < 2*MaxMass + MaxCompress ){
    return (MaxMass*MaxMass + total_mass*MaxCompress)/(MaxMass + MaxCompress);
  } else {
    return (total_mass + MaxCompress)/2;
  }
}

Full source code

• Download the source (23 KB)
• You will need the Java-based Processing programming language to run the code.

I didn’t discuss how to render the water, or how to handle user interaction, but the source code also includes a simplistic implementation of those things.

Conclusion

CA-based algorithms have both advantages and disadvantages. On the one hand, they’re relatively fast and the simulation quality is often good enough for games and software toys. On the other, they can produce behaviour that is very unrealistic – for example, water falling into a basin will form a little “hill” and spread out slowly, instead of near-instantly as real water would. Whether this is an acceptable downside depends on your application.

For a different take on simulating fluid dynamics with cellular automata, check out this interview with the author of Dwarf Fortress.