mark/image-similarity
1
0
Fork 0
Code Pull Requests Packages Projects Releases Activity

text updates
continuous-integration/drone/push Build is passing
Details

Browse Source
This commit is contained in:
Mark Hoekveen
2024-06-27 10:33:43 +02:00
parent 5b0be40e2f
commit 9d0c9aab3e
3 changed files with 198 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files
web/index.html
+135 -14
View File
@@ -16,6 +16,27 @@
<em>This is quite a long setup towards playing around with small image fingerprints for similarity matching.
If you don't care about the background, feel free to skip straight to the <a href="#demo">demo</a>!</em>
</p>
<p>
I created a little framework for implementing small (64-bit) image fingerprints which can be used for quick image similarity or near-copy detection of images.
One novel descriptor I came up with is inspired by the JPEG algorithm and is based on the Discrete Cosine Transform.
<a href="#results">It performs quite well</a>.
<p>
This was originally going to be part of my Master's Thesis, which I never finished.
The general concepts and framework have been floating around in my head for quite a while now,
so I figured it was time to finish up the code and write up the results.
</p>
<p>
Originally I started my project in C++, but as it evolved and requirements changed the quality of the code quickly went down.
I recently started learning Rust, and figured it was the perfect language to reimplement my framework in.
As it turned out, I was right!
</p>
<p>
</article>
<article>
<h1>Motivation</h1>
<p>
<em><strong>TODO:</strong> Dit hele stuk schrappen? Hoop woorden om te zeggen dat je images visueel moet vergelijken.</em>
</p>
<p>
Many computer vision applications don't process the image directly, but use some kind of representation of the image instead.
Usually this takes the form of some one-way function that calculates a vector of some dimensionality.
@@ -105,9 +126,11 @@ Not same file
</em>
</p>
<p>
So what are the requirements of a good perceptual hash?
When we apply a LSH algorithm to an image, such that the locality-sensitivity is based around some perceptual qualities, we call it <strong>perceptual hashing</strong>.
A perceptual hashing algorithm generates a perceptual hash, which we often call a <strong>phash</strong>.
So what are the requirements of a good perceptual hashing algorithm?
<ol>
<li>Small (i.e. good dimensionality reduction)</li>
<li>Reduces input size (i.e. generates a small phash)</li>
<li>Avoid collisions between <strong>different</strong> images.</li>
<li>High similarity between <strong>similar</strong> images.</li>
</ol>
@@ -115,12 +138,13 @@ Not same file
</p>
<h3>Size</h3>
<p>
I have decided to limit myself to fingerprints that are <em>at most</em> 64 bits (or 8 bytes) in size.
Firstly, this allows me to "compete" in a little happy place of small descriptors.
But more importantly, because modern processors are very good at processing 64 bit integers, this should also be a very performant fingerprint.
For this project, we have imposed a limit of <em>at most</em> 64 bits (or 8 bytes) per phash.
This is of course somewhat of an arbitrary limit,
but more importantly,
because modern processors are very good at processing 64 bit integers this should also be a very performant fingerprint.
</p>
<p>
The good news is that this takes care of our first requirement.
This takes care of our first requirement.
</p>
<h3>Subjective image similarity</h3>
<p>
@@ -142,6 +166,8 @@ Not same file
The framing is similar, the subject is similar, the colors are different.
So we would like a good perceptual hashing algorithm to generate hashes for these images that are close, but not to put them in the same bucket.
</p>
<p>
<em><strong>TODO:</strong>Hier een beter voorbeeld neerzetten van een gemuteerde image ofzo</em>
<p>
<div class="similarity-example">
<figure>
@@ -159,8 +185,96 @@ Not same file
</p>
<p>
All things considered, these image should probably be classified as <em>very similar</em> by most perceptual hashing algorithms.
Whether or not these images should end up in the same bucket is honestly a matter for debate.
</p>
</article>
<article>
<h1>Experimental setup</h1>
<p>
Before we start cranking out algorithms, we would like to have some setup where we can try to objectively judge the performance of these algorithms.
We have tried to informally define what makes a good perceptual hash in the previous section, but lets try to enumerate some things our phash should be sensitive and insensitive to.
</p>
<h2>Mutators</h2>
<p>
If we have a dataset of images, we assume that all of the images in that dataset are different images.
We can then create copies of those images by applying mutations to those images in a controlled way, and injecting them into the dataset.
Then, we can check if the phash of an image matches with the phash of its mutated copies.
</p>
<p>
For picking some mutators, let's try to answer the question of image similarity from the perspective of copyright infringement.
For example, when uploading a video of copyrighted material to an online platform,
what mutation would someone apply to make sure that the content is still very much recognisable by humans,
but might slip by whatever simple automated copyright-detection system is in place?
<ol>
<li>Horizontal flip</li>
<li>Brightness adjustment</li>
<li>Hue rotation</li>
<li>Blur</li>
<li>Sharpen</li>
<li>Watermark or logo added</li>
<li>Crop</li>
<li>Rotate and crop</li>
</ol>
<em><strong>TODO</strong>: Meer en betere mutators bedenken. Misschien ook niet demo los hebben maar gewoon integreren zodat je live voorbeelden hebt.</em>
</p>
<h2>Descriptor</h2>
<p>
<strong>Descriptor</strong> is the term I use for a perceptual hashing algorithm.
This is because my first implementations were based on the Color Layout Descriptor, so I used the term descriptor.
A descriptor is not neccessarily the same as a perceptual hash, but for the purposes of this research, they can be.
</p>
<p>
Included with the descriptor is also a distance function.
For these small image descriptors the most common distance function is the hamming distance, which is to say the amount of bits that differs between two elements.
</p>
<h2>Experiments</h2>
<p>
I used the MIRFlickr dataset.
For each image, create its mutations and calculate the phashes with each algorithm.
All these phashes are then stored in a hash map, taking care to keep the mutated versions separated.
For quick nearest neighbour checking, a BK-tree is kept per descriptor with all phashes.
</p>
<p>
We can then use this collection of data to create a precision-recall curve.
We first set a threshold for maximum distance to consider two phashes "the same", which we can vary to create a PR-curve.
Then, for each threshold, we check this collection of neighbours per mutator to collect our statistics.
So, per threshold, per descriptor, per mutator, per unmutated image:
<ul>
<li>A true positive (<msub><mi>T</mi><mi>p</mi></msub>) is when a mutated base image is within threshold of the original image.</li>
<li>A false positive (<msub><mi>F</mi><mi>p</mi></msub>) is when a different mutated image is within threshold of the original image.</li>
<li>A false negative (<msub><mi>F</mi><mi>n</mi></msub>) is when the mutated base image is not a near neighbour of its original image.</li>
</ul>
This gets us values for precision P and recall R per mutator per descriptor:
<math><mrow>
<mi>P</mi><mo>=</mo>
<mfrac>
<msub><mi>T</mi><mi>p</mi></msub>
<mrow>
<msub><mi>T</mi><mi>p</mi></msub>
<mo>+</mo>
<msub><mi>F</mi><mi>p</mi></msub>
</mrow>
</mfrac>
</mrow></math>
,
<math><mrow>
<mi>R</mi><mo>=</mo>
<mfrac>
<msub><mi>T</mi><mi>p</mi></msub>
<mrow>
<msub><mi>T</mi><mi>p</mi></msub>
<mo>+</mo>
<msub><mi>F</mi><mi>n</mi></msub>
</mrow>
</mfrac>
</mrow></math>
</p>
<p>
High precision corresponds to a low false positive rate, and high recall corresponds to a low false negative rate.
Ideally, both of these measures should be high for an accurate image similarity detector.
</p>
</article>
<article id="demo">
<h1>Demo</h1>
<p>Select an image to run this demo with. Don't worry, nothing will be sent to any server! All calculations are done in the browser.</p>
@@ -173,22 +287,29 @@ Not same file
<div class="image-original"></div>
</div>
</article>
<article id="demo-cont">
<article id="demo-resize">
<h2>Resize</h2>
<p>The first step is to size the image down. Since we are aiming for a 64-bit descriptor, the logical target size is 8 by 8 since that will give us 64 pixels to work with.</p>
<div id="resize">
<div class="image-original" id="resize-original"></div>
<div id="image-resize"></div>
</div>
<p>
The first step is to size the image down, and remove all color information.
Converting the image to grayscale is done by simply averaging the pixels.
Since we are aiming for a 64-bit descriptor, the logical target size is 8 by 8 since that will give us 64 pixels to work with.
</p>
<p>
It is obvious, but worth noting, that this action is <em>destructive</em>.
We are throwing away a lot of information here, especially regarding the finer details.
For now, this is a good thing.
</p>
<p>
After all, we have to <em>describe</em> our image in only 64-bits, so throwing away information was inevitable.
If you squint your eyes the original image can be recognized. Sort of.
Here we use a nearest neighbour downscaling algorithm that does not preserve aspect ratio.
Press the button to resize your image: <br />
<button id="resize-button">Resize</button>
</p>
<div id="resize">
<div class="image-original" id="resize-original"></div>
<div id="image-resize"></div>
</div>
</article>
<article id="demo-dct">
<h2>DCT</h2>
<p>What is DCT?</p>
<p>Frequency domain. Plaatje. Bla bla.</p>
web/script.js
+24 -13
View File
@@ -11,9 +11,20 @@ let resizedImageContainer = document.getElementById("image-resize");
let dctCoefficientContainer = document.getElementById("image-dct");
let formImage = document.getElementById("dctimage");
let originalImage = document.createElement("img");
let resizedBuffer = null;
// Resets all containers, deletes images, etc
// Redefines the event handlers for the base image
function reset() {
//imageContainers.forEach((imageContainer) => {
for (let imageContainer of imageContainers) {
imageContainer.replaceChildren();
}
resizedImageContainer.replaceChildren();
dctCoefficientContainer.replaceChildren();
document.getElementById("resize").classList.remove("resize");
}
// Uses the resized buffer to get DCT coefficients
function getDCT() {
let buf = new Uint8Array(resizedBuffer)
@@ -26,17 +37,6 @@ function getDCT() {
}
}
// Resets all containers, deletes images, etc
// Redefines the event handlers for the base image
function reset() {
//imageContainers.forEach((imageContainer) => {
for (let imageContainer of imageContainers) {
imageContainer.replaceChildren();
}
resizedImageContainer.replaceChildren();
dctCoefficientContainer.replaceChildren();
}
// Result from the resize worker, means we get resized buffer
resizeWorker.onmessage = function (e) {
console.log(e.data);
@@ -46,8 +46,17 @@ resizeWorker.onmessage = function (e) {
new Blob([resizedBuffer], { type: 'image/png' })
);
resizedImageContainer.appendChild(resizedImage);
// Set the width of the image explictly to help with the animation
let image = document.getElementById("resize-original").children[0];
image.width = image.width;
image.height = image.height;
}
document.getElementById("resize-button").addEventListener("click", function() {
document.getElementById("resize").classList.add("resize");
});
// User selected an image from disk. Start the demo.
formImage.addEventListener("change", function() {
reset();
@@ -65,5 +74,7 @@ formImage.addEventListener("change", function() {
resizeWorker.postMessage(buf);
});
}
document.getElementById("demo-cont").classList.add("visible");
// Show the rest of the demo.
document.getElementById("demo-resize").classList.add("visible");
});
web/style.css
+39 -11
View File
@@ -71,14 +71,15 @@ code.block {
img {
max-width: 100%;
image-rendering: pixelated;
}
#demo-cont {
/* Demo */
#demo-resize, #demo-dct {
visibility: hidden;
}
#demo-cont.visible {
#demo-resize.visible, #demo-dct.visible {
visibility: visible;
}
@@ -87,37 +88,64 @@ img {
min-width: 256px;
}
/* Resize demo */
#resize {
position: relative;
height: 512px;
width: 100%;
transition: height 2s linear;
}
#resize.resize {
height: 256px;
}
#resize .image-original {
position: absolute;
width: 100%;
height: 512px;
}
#resize .image-original img {
max-height: 50vh;
min-height: 257px;
max-width: 50vw;
max-width: 100%;
min-width: 257px;
/* height: 512px; */
}
#image-original img {
#resize-original {
position: relative;
height: 512px;
}
#resize-original img {
position: absolute;
max-width: 90vw;
max-height: 90vh;
max-height: 512px;
min-width: 257px;
min-height: 257px;
opacity: 1;
transition:
width 2s linear,
height 2s linear,
filter 3s linear,
opacity 3s ease-in;
filter 3s linear 2s,
opacity 3s ease-in 5s;
}
#image-original img.resize {
#resize.resize #resize-original img {
width: 256px !important;
height: 256px !important;
filter: grayscale(1) blur(10px);
opacity: 0;
}
#image-original img.grayscale {
#resize-original img.grayscale {
filter: grayscale(1) blur(10px);
}
#image-original img.fade {
#resize-original img.fade {
opacity: 0;
}