Predicting the Next Frame
A film is mostly the same picture again and again. The art is in saying only what changed.
Four seconds of film, 200 pictures. Uncompressed they come to 276 MB, and an hour would fill a quarter of a terabyte. No browser can play video in that form, so what you see here is a stand-in at very high quality, which takes 19.9 MB.
To the right of the line is the same film in 455 kB, which is 607 times smaller than the uncompressed pictures. Stop it and step through single frames to see what movement hides.
Two frames from a film of runners, a fraction of a second apart.
They are flicking back and forth so that you can see what moved.
Subtract one from the other. The ghost that is left is everything that changed.
This is what would have to be sent if a film were stored as differences and nothing more.
A codec does something cleverer first. It slides small blocks of the earlier frame about until they fit the later one.
Each line shows how far one block travelled.
Then it subtracts again. Much less is left over, and what is left is compressed like a still picture, with the patterns and rounding from the last room.
The directions for moving the blocks cost almost nothing to send.
You can see it in the file. Each step of this line is the size of one frame.
The tall step at the start is a whole picture, stored so that playback has somewhere to begin. The others that stand out, at regular intervals, are frames that many later ones will be guessed from, so they are kept with more care. Everything else is small corrections.
H.264 put video on the internet in 2003. AV1 followed in 2018 from an alliance of browser makers, chip makers and streaming services, and is free for anyone to use. The score up the side is VMAF, a measure built to agree with what viewers say.
To reach a VMAF of 85 on this film, H.264 needed 1.00 Mb/s and AV1 292 kb/s, which is 71 per cent less.
How much is saved depends on what was filmed. On the three films in this room it runs from 24 to 71 per cent.
Squeeze it yourself. The control engineers turn here is called CRF. A higher number means coarser rounding and a smaller file.
Knowing which is which makes differences easy to imagine. A blind test takes the knowledge away. X is secretly either A, the stand-in for the original, or B, AV1 CRF 38. Watch all three as often as you like, then say which one X is.
This version climbs. It begins with the smallest file and moves up each time you prove you can tell it from the original. It stops at the first file you cannot tell apart. That is your own limit, on your own equipment.
Starve a codec of bits and it shows you how it thinks. On the left is H.264 and on the right AV1, each allowed about 57.4 kB for the whole four seconds. H.264 breaks into squares. AV1 goes soft and keeps the scene.
The saving is paid for in arithmetic. The same film was encoded at three levels of effort, and the bars show how long each took.
To reach a VMAF of 85, the slowest setting needed 47 per cent fewer bits than the fastest, for 6 times the work.
Encoding can be done by a program, which may think for as long as it likes, or by a circuit built into a chip, which runs on almost no power and can never be improved after the day it is made.
These were measured on the computer that built this museum. To reach a VMAF of 85 on this film, its H.264 circuit needed 2.44 Mb/s where careful software needed 1.00 Mb/s, which is 2.4 times as many bits.
Every chip maker offers the same bargain, and what each generation of chips can encode is a fair history of the formats themselves.
AV1 reached all three makers of graphics cards within a single year.
For engineers, these are the other controls the encoder offers, changed one at a time while everything else is held still.
Choose a row to open that file in the viewer.
Text, sound, pictures and film are the four kinds of file everybody knows.
The next three rooms hold things you may never have thought of as compressed at all.