Data Compression

	dog.png -	125 bytes.	Few but likely.
	noise.png -	246 bytes.	Many but unlikely.

As you walk around the world looking at all the dogs and things, the images that fall on your image sensor are much more likely to be selected from the relatively small set of interesting images with visual structure, than from the very large set of images that look like random noise.

This is because the world is sparse. Visual structure consists of repetition and redundancy, such as the large area of empty space around the dog, or the number of similar pixels lined up in a row to make the dog's leg.

Lossless data compression works by exploiting sparseness, the structure and redundancy of information, to produce a representation with less bits. A compression method expects a certain kind of structure in the input; it reduces the size of inputs that have that kind of structure, at the expense of increasing the size of inputs that do not. No compression method can reduce the size of all possible inputs.

Compression and intelligence are related, because both have to do with finding patterns and structure in information. It takes some intelligence to recognise that an image contains the patterns of a dog. It is sparseness that makes compression and intelligence possible.

Patterns and Causes

This is an image of a dog. You recognised it because you have seen dogs before, although likely not exactly the same as this one. In fact the image that falls on your retina will probably never be exactly the same twice, even if you spend the rest of your life looking only at dogs. And yet all these possible images of dogs have a certain recognisable dogness in common.

This particular dog image was formed on a 32x32 pixel image sensor, with 1 bit per pixel. There are 2¹⁰²⁴ distinct images that can be perceived by a sensor of those dimensions; in decimal that number has over 300 digits. But only a very small fraction of those possible images look like dogs, or anything else recognisable. Most of the possible images look like random noise.


An image of a dog is recognisable because the pixels are influenced indirectly by the presence of the dog. I say indirectly because the dog causes a head, body, legs and a tail to fall within the field of view. These dog-parts influence the number of photons reaching the various pixels on the image sensor, which in turn record light or dark pixels in the image.

People find it easy to recognise dogs, because the human neocortex is very good at learning patterns and remembering them. You weren't born knowing about dogs - your neocortex learned that from seeing lots of different images of them. And now your neocortex contains a hierarchical structure that records your memory of how dogs are made of dog-parts, and some essential features of the patterns in the appearance of dogs.

This blog is about how intelligence works, and how we might build computer systems that can recognise dogs and do other intelligent things like learning language.