From 9d0c9aab3e1c02c2047698cbf23a2b7c3ed554f6 Mon Sep 17 00:00:00 2001 From: Mark Hoekveen Date: Thu, 27 Jun 2024 10:33:43 +0200 Subject: [PATCH] text updates --- web/index.html | 149 ++++++++++++++++++++++++++++++++++++++++++++----- web/script.js | 37 +++++++----- web/style.css | 50 +++++++++++++---- 3 files changed, 198 insertions(+), 38 deletions(-) diff --git a/web/index.html b/web/index.html index 7141330..2c6551b 100644 --- a/web/index.html +++ b/web/index.html @@ -16,6 +16,27 @@ This is quite a long setup towards playing around with small image fingerprints for similarity matching. If you don't care about the background, feel free to skip straight to the demo!

+

+ I created a little framework for implementing small (64-bit) image fingerprints which can be used for quick image similarity or near-copy detection of images. + One novel descriptor I came up with is inspired by the JPEG algorithm and is based on the Discrete Cosine Transform. + It performs quite well. +

+ This was originally going to be part of my Master's Thesis, which I never finished. + The general concepts and framework have been floating around in my head for quite a while now, + so I figured it was time to finish up the code and write up the results. +

+

+ Originally I started my project in C++, but as it evolved and requirements changed the quality of the code quickly went down. + I recently started learning Rust, and figured it was the perfect language to reimplement my framework in. + As it turned out, I was right! +

+

+ +

+

Motivation

+

+ TODO: Dit hele stuk schrappen? Hoop woorden om te zeggen dat je images visueel moet vergelijken. +

Many computer vision applications don't process the image directly, but use some kind of representation of the image instead. Usually this takes the form of some one-way function that calculates a vector of some dimensionality. @@ -105,9 +126,11 @@ Not same file

- So what are the requirements of a good perceptual hash? + When we apply a LSH algorithm to an image, such that the locality-sensitivity is based around some perceptual qualities, we call it perceptual hashing. + A perceptual hashing algorithm generates a perceptual hash, which we often call a phash. + So what are the requirements of a good perceptual hashing algorithm?

    -
  1. Small (i.e. good dimensionality reduction)
    2. +
  2. Reduces input size (i.e. generates a small phash)
  3. Avoid collisions between different images.
  4. High similarity between similar images.
@@ -115,12 +138,13 @@ Not same file

Size

- I have decided to limit myself to fingerprints that are at most 64 bits (or 8 bytes) in size. - Firstly, this allows me to "compete" in a little happy place of small descriptors. - But more importantly, because modern processors are very good at processing 64 bit integers, this should also be a very performant fingerprint. + For this project, we have imposed a limit of at most 64 bits (or 8 bytes) per phash. + This is of course somewhat of an arbitrary limit, + but more importantly, + because modern processors are very good at processing 64 bit integers this should also be a very performant fingerprint.

- The good news is that this takes care of our first requirement. + This takes care of our first requirement.

Subjective image similarity

@@ -142,6 +166,8 @@ Not same file The framing is similar, the subject is similar, the colors are different. So we would like a good perceptual hashing algorithm to generate hashes for these images that are close, but not to put them in the same bucket.

+

+ TODO:Hier een beter voorbeeld neerzetten van een gemuteerde image ofzo

@@ -159,8 +185,96 @@ Not same file

All things considered, these image should probably be classified as very similar by most perceptual hashing algorithms. + Whether or not these images should end up in the same bucket is honestly a matter for debate.

+
+

Experimental setup

+

+ Before we start cranking out algorithms, we would like to have some setup where we can try to objectively judge the performance of these algorithms. + We have tried to informally define what makes a good perceptual hash in the previous section, but lets try to enumerate some things our phash should be sensitive and insensitive to. +

+

Mutators

+

+ If we have a dataset of images, we assume that all of the images in that dataset are different images. + We can then create copies of those images by applying mutations to those images in a controlled way, and injecting them into the dataset. + Then, we can check if the phash of an image matches with the phash of its mutated copies. +

+

+ For picking some mutators, let's try to answer the question of image similarity from the perspective of copyright infringement. + For example, when uploading a video of copyrighted material to an online platform, + what mutation would someone apply to make sure that the content is still very much recognisable by humans, + but might slip by whatever simple automated copyright-detection system is in place? +

    +
  1. Horizontal flip
    2. +
  2. Brightness adjustment
    3. +
  3. Hue rotation
    4. +
  4. Blur
    5. +
  5. Sharpen
    6. +
  6. Watermark or logo added
    7. +
  7. Crop
    8. +
  8. Rotate and crop
    9. +
+ TODO: Meer en betere mutators bedenken. Misschien ook niet demo los hebben maar gewoon integreren zodat je live voorbeelden hebt. +

+

Descriptor

+

+ Descriptor is the term I use for a perceptual hashing algorithm. + This is because my first implementations were based on the Color Layout Descriptor, so I used the term descriptor. + A descriptor is not neccessarily the same as a perceptual hash, but for the purposes of this research, they can be. +

+

+ Included with the descriptor is also a distance function. + For these small image descriptors the most common distance function is the hamming distance, which is to say the amount of bits that differs between two elements. +

+

Experiments

+

+ I used the MIRFlickr dataset. + For each image, create its mutations and calculate the phashes with each algorithm. + All these phashes are then stored in a hash map, taking care to keep the mutated versions separated. + For quick nearest neighbour checking, a BK-tree is kept per descriptor with all phashes. +

+

+ We can then use this collection of data to create a precision-recall curve. + We first set a threshold for maximum distance to consider two phashes "the same", which we can vary to create a PR-curve. + Then, for each threshold, we check this collection of neighbours per mutator to collect our statistics. + So, per threshold, per descriptor, per mutator, per unmutated image: +

+ This gets us values for precision P and recall R per mutator per descriptor: + + P= + + Tp + + Tp + + + Fp + + + + , + + R= + + Tp + + Tp + + + Fn + + + +

+

+ High precision corresponds to a low false positive rate, and high recall corresponds to a low false negative rate. + Ideally, both of these measures should be high for an accurate image similarity detector. +

+ +

Demo

Select an image to run this demo with. Don't worry, nothing will be sent to any server! All calculations are done in the browser.

@@ -173,22 +287,29 @@ Not same file
-
+

Resize

-

The first step is to size the image down. Since we are aiming for a 64-bit descriptor, the logical target size is 8 by 8 since that will give us 64 pixels to work with.

-
-
-
-
+

+ The first step is to size the image down, and remove all color information. + Converting the image to grayscale is done by simply averaging the pixels. + Since we are aiming for a 64-bit descriptor, the logical target size is 8 by 8 since that will give us 64 pixels to work with. +

It is obvious, but worth noting, that this action is destructive. We are throwing away a lot of information here, especially regarding the finer details. For now, this is a good thing.

- After all, we have to describe our image in only 64-bits, so throwing away information was inevitable. - If you squint your eyes the original image can be recognized. Sort of. + Here we use a nearest neighbour downscaling algorithm that does not preserve aspect ratio. + Press the button to resize your image:
+

+
+
+
+
+
+

DCT

What is DCT?

Frequency domain. Plaatje. Bla bla.

diff --git a/web/script.js b/web/script.js index fd776a0..971caf4 100644 --- a/web/script.js +++ b/web/script.js @@ -11,9 +11,20 @@ let resizedImageContainer = document.getElementById("image-resize"); let dctCoefficientContainer = document.getElementById("image-dct"); let formImage = document.getElementById("dctimage"); -let originalImage = document.createElement("img"); let resizedBuffer = null; +// Resets all containers, deletes images, etc +// Redefines the event handlers for the base image +function reset() { + //imageContainers.forEach((imageContainer) => { + for (let imageContainer of imageContainers) { + imageContainer.replaceChildren(); + } + resizedImageContainer.replaceChildren(); + dctCoefficientContainer.replaceChildren(); + document.getElementById("resize").classList.remove("resize"); +} + // Uses the resized buffer to get DCT coefficients function getDCT() { let buf = new Uint8Array(resizedBuffer) @@ -26,17 +37,6 @@ function getDCT() { } } -// Resets all containers, deletes images, etc -// Redefines the event handlers for the base image -function reset() { - //imageContainers.forEach((imageContainer) => { - for (let imageContainer of imageContainers) { - imageContainer.replaceChildren(); - } - resizedImageContainer.replaceChildren(); - dctCoefficientContainer.replaceChildren(); -} - // Result from the resize worker, means we get resized buffer resizeWorker.onmessage = function (e) { console.log(e.data); @@ -46,8 +46,17 @@ resizeWorker.onmessage = function (e) { new Blob([resizedBuffer], { type: 'image/png' }) ); resizedImageContainer.appendChild(resizedImage); + + // Set the width of the image explictly to help with the animation + let image = document.getElementById("resize-original").children[0]; + image.width = image.width; + image.height = image.height; } +document.getElementById("resize-button").addEventListener("click", function() { + document.getElementById("resize").classList.add("resize"); +}); + // User selected an image from disk. Start the demo. formImage.addEventListener("change", function() { reset(); @@ -65,5 +74,7 @@ formImage.addEventListener("change", function() { resizeWorker.postMessage(buf); }); } - document.getElementById("demo-cont").classList.add("visible"); + + // Show the rest of the demo. + document.getElementById("demo-resize").classList.add("visible"); }); \ No newline at end of file diff --git a/web/style.css b/web/style.css index 20aaa40..4dab206 100644 --- a/web/style.css +++ b/web/style.css @@ -71,14 +71,15 @@ code.block { img { max-width: 100%; - image-rendering: pixelated; } -#demo-cont { +/* Demo */ + +#demo-resize, #demo-dct { visibility: hidden; } -#demo-cont.visible { +#demo-resize.visible, #demo-dct.visible { visibility: visible; } @@ -87,37 +88,64 @@ img { min-width: 256px; } +/* Resize demo */ + +#resize { + position: relative; + height: 512px; + width: 100%; + transition: height 2s linear; +} + +#resize.resize { + height: 256px; +} + +#resize .image-original { + position: absolute; + width: 100%; + height: 512px; +} + #resize .image-original img { max-height: 50vh; min-height: 257px; - max-width: 50vw; + max-width: 100%; min-width: 257px; + /* height: 512px; */ } -#image-original img { +#resize-original { + position: relative; + height: 512px; +} + +#resize-original img { position: absolute; max-width: 90vw; - max-height: 90vh; + max-height: 512px; min-width: 257px; min-height: 257px; opacity: 1; transition: width 2s linear, height 2s linear, - filter 3s linear, - opacity 3s ease-in; + filter 3s linear 2s, + opacity 3s ease-in 5s; } -#image-original img.resize { +#resize.resize #resize-original img { width: 256px !important; height: 256px !important; + filter: grayscale(1) blur(10px); + opacity: 0; } -#image-original img.grayscale { +#resize-original img.grayscale { filter: grayscale(1) blur(10px); } -#image-original img.fade { +#resize-original img.fade { opacity: 0; }