diff --git a/web/index.html b/web/index.html index 2c6551b..32005d1 100644 --- a/web/index.html +++ b/web/index.html @@ -10,270 +10,19 @@
-

Small image descriptors for near-copy detection

+

Fucking around with perceptual hashes

Introduction

-

- This is quite a long setup towards playing around with small image fingerprints for similarity matching. - If you don't care about the background, feel free to skip straight to the demo! -

-

- I created a little framework for implementing small (64-bit) image fingerprints which can be used for quick image similarity or near-copy detection of images. - One novel descriptor I came up with is inspired by the JPEG algorithm and is based on the Discrete Cosine Transform. - It performs quite well. -

- This was originally going to be part of my Master's Thesis, which I never finished. - The general concepts and framework have been floating around in my head for quite a while now, - so I figured it was time to finish up the code and write up the results. -

-

- Originally I started my project in C++, but as it evolved and requirements changed the quality of the code quickly went down. - I recently started learning Rust, and figured it was the perfect language to reimplement my framework in. - As it turned out, I was right! -

-

-

-
-

Motivation

-

- TODO: Dit hele stuk schrappen? Hoop woorden om te zeggen dat je images visueel moet vergelijken. -

-

- Many computer vision applications don't process the image directly, but use some kind of representation of the image instead. - Usually this takes the form of some one-way function that calculates a vector of some dimensionality. - In modern machine learning this is called an embedding. - In computer vision we might consider something a feature, a descriptor or even a fingerprint, depending on the application. -

-

- The basic ideas are the same: We want to say something about the image, and to have the computer do some calculations with this image it is almost always easier to work with some (often smaller) representation of that image. - Even the biggest representations are usually at least an order of magnitude smaller than the original image, which makes running computations on them much faster. -

-

- Different tasks require larger representations of the image. - Object recognition, image classification, or even 3d image reconstruction require more precise input and thus more data to represent the image. - However, some simpler tasks might be accomplished with much less data. - One of these simpler applications is perceptual hashing. -

-

Image compare

-

- Let's imagine that we have two images, and we want to check if these two images are the same image. - Well, you might start by simply comparing data byte-for-byte, which works quite well for direct copies. - GNU diffutils even has a nice cmp tool that will do exactly that. - It will look through two files until it finds a byte that does not match between the two: -

-$ cp a.jpg a.copy.jpg
-$ cmp a.jpg a.copy.jpg && echo "Same" || echo "Not same"
-Same
-
-
-

- This works for our direct copy. - Now lets see how it handles two entirely different images: -

-
-$ cmp a.jpg b.jpg 
-a.jpg b.jpg differ: byte 14, line 1
-
-
-

- The two different images differ on byte 14, which is the byte in the JPEG header which defines the "density", or pixels per length of an image. - In this case, a.jpg does not have anything set, while b.jpg defines its density as pixels per inch. - What about two images that are the same, but reencoded? -

-
-$ convert c.png c.jpg && convert c.jpg d.png
-$ cmp c.png d.png 
-c.png d.png differ: byte 36, line 3
-
-
-

- In this case we convert a png image into a jpeg and then back into a png. - This introduces jpeg encoding artefacts into the image, which subtly alter the pixel values such that cmp no longer considers them the same image. -

-

- This gives us our first hints at why image comparisons are tricky. - Even when both files are jpeg images, the subtle differences in encoding already trip us up before we even get to the actual pixel information. - And even when both files are visually identical (and identically encoded), the subtle differences in pixel values still gets us. - -

-

- This might all seem a little silly so far. - Of course comparing images in this way does not make sense. - What we are looking for is a method that uses a more complex perception-based model to say if two images are the same. - Even more so, we are looking for a method to not only compare two given images, - but quickly calculate the nearest neighbours of a given image based on their similarity. -

Perceptual hashing

Locality-sensitive hashing

-

- I am going to assume you know what a hashing function is. - Different hashing functions have various properties that make them more or less suited to various purposes, - but one common feature is that they try to avoid collisions by having small changes in input correspond to large changes in the output. - When we want to avoid this behaviour we are doing something called locality-sensitive hashing (LSH). - In locality-sensitive hashing we are more likely to put similar data into the same bucket. -

-

- - In normal hashing we want to avoid the same output values so we call it a collision (bad!). - For LSH we want to encourage the same output values, so we say we "put it in a bucket" (good!). - -

-

- When we apply a LSH algorithm to an image, such that the locality-sensitivity is based around some perceptual qualities, we call it perceptual hashing. - A perceptual hashing algorithm generates a perceptual hash, which we often call a phash. - So what are the requirements of a good perceptual hashing algorithm? -

    -
  1. Reduces input size (i.e. generates a small phash)
    2. -
  2. Avoid collisions between different images.
    3. -
  3. High similarity between similar images.
    4. -
- We see that we have multiple objectives to optimize for, but we also see that two of our requirements depend on a subjective measure. -

-

Size

-

- For this project, we have imposed a limit of at most 64 bits (or 8 bytes) per phash. - This is of course somewhat of an arbitrary limit, - but more importantly, - because modern processors are very good at processing 64 bit integers this should also be a very performant fingerprint. -

-

- This takes care of our first requirement. -

Subjective image similarity

-

- Broadly speaking, we have to answer the question: when are two images the same? -

-

- Lets start by looking at some examples. -

-
- -
Close Up of a Sunflower in a Field of Sunflowers
-
-
- -
Sunflower against the sky
-
-
- These two pictures are, to a human observer, definitely similar, but also not the same. - The framing is similar, the subject is similar, the colors are different. - So we would like a good perceptual hashing algorithm to generate hashes for these images that are close, but not to put them in the same bucket. -

-

- TODO:Hier een beter voorbeeld neerzetten van een gemuteerde image ofzo -

-

-
- -
Full moon by dunc
-
-
- -
Vollmond nahe dem Zenit
-
-
- These are more similar. - We have the same subject, same coloring, very similar brightness and contrast values. - If we look a bit closer we see that the rotation is a bit different in both images, the left one is a bit sharper, and the right image has more "empty space" around the subject. -

-

- All things considered, these image should probably be classified as very similar by most perceptual hashing algorithms. - Whether or not these images should end up in the same bucket is honestly a matter for debate. -

Experimental setup

-

- Before we start cranking out algorithms, we would like to have some setup where we can try to objectively judge the performance of these algorithms. - We have tried to informally define what makes a good perceptual hash in the previous section, but lets try to enumerate some things our phash should be sensitive and insensitive to. -

Mutators

-

- If we have a dataset of images, we assume that all of the images in that dataset are different images. - We can then create copies of those images by applying mutations to those images in a controlled way, and injecting them into the dataset. - Then, we can check if the phash of an image matches with the phash of its mutated copies. -

-

- For picking some mutators, let's try to answer the question of image similarity from the perspective of copyright infringement. - For example, when uploading a video of copyrighted material to an online platform, - what mutation would someone apply to make sure that the content is still very much recognisable by humans, - but might slip by whatever simple automated copyright-detection system is in place? -

    -
  1. Horizontal flip
    2. -
  2. Brightness adjustment
    3. -
  3. Hue rotation
    4. -
  4. Blur
    5. -
  5. Sharpen
    6. -
  6. Watermark or logo added
    7. -
  7. Crop
    8. -
  8. Rotate and crop
    9. -
- TODO: Meer en betere mutators bedenken. Misschien ook niet demo los hebben maar gewoon integreren zodat je live voorbeelden hebt. -

Descriptor

-

- Descriptor is the term I use for a perceptual hashing algorithm. - This is because my first implementations were based on the Color Layout Descriptor, so I used the term descriptor. - A descriptor is not neccessarily the same as a perceptual hash, but for the purposes of this research, they can be. -

-

- Included with the descriptor is also a distance function. - For these small image descriptors the most common distance function is the hamming distance, which is to say the amount of bits that differs between two elements. -

Experiments

-

- I used the MIRFlickr dataset. - For each image, create its mutations and calculate the phashes with each algorithm. - All these phashes are then stored in a hash map, taking care to keep the mutated versions separated. - For quick nearest neighbour checking, a BK-tree is kept per descriptor with all phashes. -

-

- We can then use this collection of data to create a precision-recall curve. - We first set a threshold for maximum distance to consider two phashes "the same", which we can vary to create a PR-curve. - Then, for each threshold, we check this collection of neighbours per mutator to collect our statistics. - So, per threshold, per descriptor, per mutator, per unmutated image: -

- This gets us values for precision P and recall R per mutator per descriptor: - - P= - - Tp - - Tp - + - Fp - - - - , - - R= - - Tp - - Tp - + - Fn - - - -

-

- High precision corresponds to a low false positive rate, and high recall corresponds to a low false negative rate. - Ideally, both of these measures should be high for an accurate image similarity detector. -

-

Demo