Numbers That Barely Move
Most of what the world stores is neither words nor pictures. It is readings, a time and a number, over and over.
This is the weather in London, every hour for ten years. There are 87,672 rows, and each holds a time, a temperature, a humidity and an air pressure.
As text it comes to 2.76 MB.
gzip brings the text down to 603 kB, and xz to 363 kB.
They are matching runs of characters. They know nothing about clocks or thermometers.
The first move is to stop reading across. Stored row by row, a time sits beside a temperature beside a pressure, and no number resembles its neighbour.
Turn the table on its side. Now each column holds one kind of number, and each number looks very like the one above it.
Take one day of temperatures, counted in tenths of a degree.
Each bar shows how many bits that reading needs if it is written out whole.
Now keep the first reading whole, and after it keep only how much each one changed from the one before.
The numbers are small, and a small number needs few bits.
The clock does better still. Each reading comes exactly one hour after the last, so the change never changes. Take the difference of the differences and the whole column is noughts.
Ten years of times take 1.49 MB as text. Stored this way they take 695 B.
All four columns stored this way come to 184 kB. That is 15 times smaller than the text and half what xz managed, using nothing harder than subtraction.
Real systems work the same way. Parquet, a column format first released in 2013 and now in very wide use, brings this table to 262 kB.
In 2015 engineers at Facebook described the store that held the measurements from all their machines. It kept only how each reading differed from what was expected.
Ninety-six in every hundred of its timestamps took a single bit. A whole reading, time and value together, averaged 1.37 bytes.
There is a second prize. To find the hottest hour of 2019, a program needs one column and one year of it.
The rest of the file is never read, and whatever is not read costs nothing.
The last room holds the oldest data there is.
It had been copied for more than three billion years before anybody thought to compress it.