Fucking around with perceptual hashes

Why a normal hash gets you nowhere

The naive approach is to use any standard hash function that you would normally use to validate file equality. One of the most well-known examples is MD5, which is a cryptographic hash, which is a fancy way of saying it is easy to calculate but hard to reverse*.

Cryptographic hashes find byte-identical copies of files and nothing else. Hashes built for integrity checking are engineered for the avalanche effect: flip one input bit and every output bit flips with probability one half. Avalanche is exactly what you want when verifying a download or storing a secret, and exactly what you do not want when looking for pictures.

To demonstrate this, the above image was altered slightly: we did +1 to the red channel of the single pixel at the center. A change so small that on a typical image without a magnifying glass you cannot see it. MD5 outputs a 128 "digest". Here are the first 64 bits of the original and altered images' MD5 digest (as 16 characters of hex):

Any operation on the original image, such as a recompression, adjustment of the metadata, or the changing of a single pixel, will completely change the hash value.

Locality sensitive hashing

Hashes that don't display this avalanche behavior are called locality-sensitive hashes (LSH). It is still a hash function, i.e. it maps some arbitrary input domain to a fixed-size output domain, but by some measure of similarity the function clusters similar input.

Unlike a cryptographic hash, LSH's are not designed to be irreversible (though they may be).

We will introduce the perceptual hashing algorithm that I developed in the next steps, but as a preview example, here is that perceptual hash applied to three images. The first image is the original, the second is the same image "hue rotated", and the third is an unrelated photo.

Your image

Hue shifted 90°

distance

Unrelated photo

distance

Note that the second image has every pixel different from the original, something the md5 would definitely see as a change. But our perceptual hash treats it as almost the same image. Depending on your input image you should expect an exact match or at least a very small distance.

For the unrelated image we can see the same type of difference as we were getting before, depending on the visual similarity to the input image we would expect at least upwards of 10 bits difference, but more likely about half of the total 64 bits again.

The thing to note here is that distance between hashes suddenly carries meaning.

So how do you build a perceptual hash? Every bit is a yes/no question about the image. The art is picking questions whose answers survive "mutation". We will discuss mutation in a later chapter, but it can be any operation on the image that preserves the perceptual similarity: recompression, rotation, mirroring, resizing, color grading, contrast adjustment, and more.

Step 1: throw almost everything away

Scale down to 8x8. Fuck aspect ratio.

As it turns out, how you convert to grayscale matters. Luma weights:

64 pixels left, which is also our bit-budget. How can that now? What a wild coinkydink.

Your original image, compared to what the hash gets to work with:

Step 2: Median threshold

Take the median of the 64 pixels and emit one bit per pixel:

1 when the condition holds, 0 otherwise. Median from Thomee et al.^[1] Does a couple of things well. Brightness, gamma, and color shifts.

The bits are tied to pixel positions, so a mirrored copy scrambles them completely. See the shitty flip recall below. Also unrelated images might share this course layout. Imagine that many landscape fotographs might have the same general silhouette.

Step 3: DCT

The discrete cosine transform rewrites the 8×8 thumbnail as a weighted sum of 64 fixed cosine patterns. The weights are the coefficients:

TODO: Dit is waarschijnlijk beter als het meer simpel is. Hoop van die termen doen er niet echt toe hier.

Top-left is average brightness. Horizontal frequency increases to the right, vertical frequency increases downward.

One of the key observations of JPEG image compression is that typically speaking most images have most of their "energy" concentrated in the lower-frequency patterns. That is why the zigzag ordering is so effective: it effectively gives a MSB ordering of the coefficients. After that, JPEG uses a quantization table to reduce the higher-frequency components to less bits. For our case, we can just discard the LSBs.

Putting it back together

The proof is that the weighted patterns sum back to the picture. Add them one at a time, in zigzag order:

Animate the hash stops reading at 36

Note that the general shape of the picture comes quite quickly with the lower frequency components. The higher frequency components simply refine existing shapes and edges. That is why cutting the zigzag at 36 loses so little: the animation pauses there, and the second half barely changes the picture.

Step 4: 64 bits

Oké maar dat is dus nog steeds veel te veel data. 64 floats. ja doei. Dus we doen signs:

En de rest dan op volgorde: Is het volgende patroontje sterker dan de huidige?

36 + 28 = 64, qed. Dan nu het slimme. $\tilde{c}$ als normalisatie stap. We gaan altijd uit van dat het eerste patroon niet geinverteerd is. Als ie dat wel is draaien we hem alsnog om. Dus dan maakt het niet uit of de patroontjes omgedraaid zijn of niet.

Snippet:

let flip = dct[1].signum();              // sign of C(1,0)
for &i in ZIGZAG.iter().take(36) {
    let mut sign = dct[i].signum();
    if i % 2 != 0 { sign *= flip }       // odd horizontal frequency
    sign_mask = sign_mask << 1 | (sign < 0.0) as u64;
}
for pair in ZIGZAG[..=28].windows(2) {
    let bigger = dct[pair[1]].abs() > dct[pair[0]].abs();
    ordinal_mask = ordinal_mask << 1 | bigger as u64;
}
let hash = sign_mask << 28 | ordinal_mask;

Kim (2003)^[3] is de Cosine baseline in Thomee et al.^[1]

Dit is zo mogelijk het enige echt unieke aan mijn oplossing. whoop.

What we remember

We can reconstruct the original 8x8 image from these values. We store no values, only signs and cardinality. Still, we can make a rough approximation of the 8x8 starting point:

A ghost, but a recognizable one, out of 8 bytes. One quirk: because of the flip normalization the bits genuinely cannot tell left from right, so the ghost sometimes comes out mirrored.

Flip it back

Step 5: how pHash does it

pHash is the work of the pHash.org project, which also hosts an online demo in the same spirit as this page. The implementation here follows the widely used imagehash recipe.

Mutations

Distances of 4 or less are green: [motivate this with PR curves?]

The flip trick

Magnitudes never change, so ordinal bits are flip-invariant for free. Only the signs of odd horizontal frequencies flip, and they all flip together. So normalize them against one of their own:

Flip it

Bits that changed under the flip are red.

Retrieval

Comparing two is nice, but ranking is the real deal. Two questions: When do we declare two hashes to be the same image? And how do we collect that subset without comparing the query against every hash we have?

When are two images the same?

pick a threshold $t$ and call everything at distance $t$ or less the same image. your image against two groups: its own 16 mutated copies from the experiment suite, and the 1,111 unrelated pool images from the ranking demo.

threshold t = 4

The two groups keep a comfortable distance from each other. Copies near zero, strangers pile up just under half of the 64 bits*. In between the kingdom of mutated hits and the people's republic of unrelated images is the eehm.. the.. Federated Islands of false positives. When a mutation goes too far, the hash distance increases and the fingerprint gets banished from the kingdom and might end up in the republic of strangers. Set your threshold too high and you start catching some of these strangers. The tradeoff then is how serious you want to be about it: do you accept some false positives or would you rather get false negatives? It all depends on what you might use these copy-detectors for. If it is the first step in reducing an image set in order for the Big Guns to take over, you might be more willing to accept false positives. If you want to quickly see if a given image is already in your folder of holiday pictures, you might not mind a false negative or two, but would rather not have a false positive. The threshold is what makes this tradeoff, and the PR-curve is where it shows.

Skipping most of the work

The obvious retrieval algorithm is a linear scan: compare the query against all $N$ stored hashes, keep everything within $t$. At a million comparisons per second per thread that is genuinely fine for a while. But the cost grows with every stored image and is paid again on every query, and "compare against everything" should offend you a little when the answer is almost always "no".

The Burkhard-Keller tree^[6] fixes this for any metric distance, and Hamming distance is one. Pick any stored hash as the root. Every other hash goes into a subtree based on its distance to the root: all hashes at distance 7 from the root share subtree 7, and inside each subtree the same rule repeats. To search, compare the query to the root, giving some distance $d$. A match can only hide in a subtree whose label lies between $d-t$ and $d+t$, that is the triangle inequality doing its thing. Recurse into the surviving subtrees, ignore the rest forever.

radius t = 4 Animate the search

At small $t$ the search drops straight into the branch where the copies cluster and skips most stranger branches without computing a single distance in them. Raise $t$ and the band $d\pm t$ widens, fewer branches get pruned, and the search slowly degrades back into visiting everyone.

Does it scale?

The toy tree has 32 hashes, the experiment above produced 425,000 (25,000 images, 16 mutants each). Those collapse to 239,689 distinct hashes, identical images simply share one entry. Below, all of them sit in a BK-tree in your browser's memory, and your image queries it live.

radius t = 4

Query with one of the MIRFLICKR samples from the strip at the top and its mutants come right back. Query with Spook and nothing comes back, because my cat is not among the 25,000.

At $t=4$ the tree answers after touching a few percent of the hashes, roughly a 30× saving in comparisons. Now look at the stopwatch: the dumb scan is still competitive, and at wide radii it wins outright. Marching sequentially through memory doing one xor and one popcnt per hash is about the kindest thing you can do to a CPU, while the tree spends its savings on hopping through pointers. The comparisons saved only turn into time saved when the collection outgrows this demo by an order of magnitude or two, since the scan grows linearly and the visited slice of the tree does not. Also note how quickly the pruning decays in the curve: perceptual hashes cluster, so a wide radius keeps almost every branch alive. A BK-tree only earns its keep at small radii, which is conveniently the only place our threshold wants to be.

Do we expect it to be perfect?

No, and it does not have to be. The histogram showed copies that drift out of reach and the occasional stranger inside the radius, the precision-recall curves put numbers on both. If a wrong answer is expensive, treat the whole thing as a preselection filter: the hash plus BK-tree reduces 425,000 candidates to a handful in microseconds, and whatever heavyweight comparison you actually trust (full-resolution diffing, feature matching, a neural embedding, a human) only runs on that handful. A cheap filter in front of an expensive judge is a classic setup, and a 64-bit hash is about the cheapest filter there is.