text updates

web/index.html

@@ -16,6 +16,27 @@
           <em>This is quite a long setup towards playing around with small image fingerprints for similarity matching.
           If you don't care about the background, feel free to skip straight to the <a href="#demo">demo</a>!</em>
         </p>
+        <p>
+          I created a little framework for implementing small (64-bit) image fingerprints which can be used for quick image similarity or near-copy detection of images.
+          One novel descriptor I came up with is inspired by the JPEG algorithm and is based on the Discrete Cosine Transform.
+          <a href="#results">It performs quite well</a>.
+        <p>
+          This was originally going to be part of my Master's Thesis, which I never finished.
+          The general concepts and framework have been floating around in my head for quite a while now,
+          so I figured it was time to finish up the code and write up the results.
+        </p>
+        <p>
+          Originally I started my project in C++, but as it evolved and requirements changed the quality of the code quickly went down.
+          I recently started learning Rust, and figured it was the perfect language to reimplement my framework in.
+          As it turned out, I was right!
+        </p>
+        <p>
+      </article>
+      <article>
+        <h1>Motivation</h1>
+        <p>
+          <em><strong>TODO:</strong> Dit hele stuk schrappen? Hoop woorden om te zeggen dat je images visueel moet vergelijken.</em>
+        </p>
         <p>
           Many computer vision applications don't process the image directly, but use some kind of representation of the image instead.
           Usually this takes the form of some one-way function that calculates a vector of some dimensionality.
@@ -105,9 +126,11 @@ Not same file
           </em>
         </p>
         <p>
-          So what are the requirements of a good perceptual hash?
+          When we apply a LSH algorithm to an image, such that the locality-sensitivity is based around some perceptual qualities, we call it <strong>perceptual hashing</strong>.
+          A perceptual hashing algorithm generates a perceptual hash, which we often call a <strong>phash</strong>.
+          So what are the requirements of a good perceptual hashing algorithm?
           <ol>
-            <li>Small (i.e. good dimensionality reduction)</li>
+            <li>Reduces input size (i.e. generates a small phash)</li>
             <li>Avoid collisions between <strong>different</strong> images.</li>
             <li>High similarity between <strong>similar</strong> images.</li>
           </ol>
@@ -115,12 +138,13 @@ Not same file
         </p>
         <h3>Size</h3>
         <p>
-          I have decided to limit myself to fingerprints that are <em>at most</em> 64 bits (or 8 bytes) in size.
+          For this project, we have imposed a limit of <em>at most</em> 64 bits (or 8 bytes) per phash.
-          Firstly, this allows me to "compete" in a little happy place of small descriptors.
+          This is of course somewhat of an arbitrary limit,
-          But more importantly, because modern processors are very good at processing 64 bit integers, this should also be a very performant fingerprint.
+          but more importantly,
+          because modern processors are very good at processing 64 bit integers this should also be a very performant fingerprint.
         </p>
         <p>
-          The good news is that this takes care of our first requirement.
+          This takes care of our first requirement.
         </p>
         <h3>Subjective image similarity</h3>
         <p>
@@ -142,6 +166,8 @@ Not same file
           The framing is similar, the subject is similar, the colors are different.
           So we would like a good perceptual hashing algorithm to generate hashes for these images that are close, but not to put them in the same bucket.
         </p>
+        <p>
+          <em><strong>TODO:</strong>Hier een beter voorbeeld neerzetten van een gemuteerde image ofzo</em>
         <p>
           <div class="similarity-example">
             <figure>
@@ -159,8 +185,96 @@ Not same file
         </p>
         <p>
           All things considered, these image should probably be classified as <em>very similar</em> by most perceptual hashing algorithms.
+          Whether or not these images should end up in the same bucket is honestly a matter for debate.
         </p>
       </article>
+      <article>
+        <h1>Experimental setup</h1>
+        <p>
+          Before we start cranking out algorithms, we would like to have some setup where we can try to objectively judge the performance of these algorithms.
+          We have tried to informally define what makes a good perceptual hash in the previous section, but lets try to enumerate some things our phash should be sensitive and insensitive to.
+        </p>
+        <h2>Mutators</h2>
+        <p>
+          If we have a dataset of images, we assume that all of the images in that dataset are different images.
+          We can then create copies of those images by applying mutations to those images in a controlled way, and injecting them into the dataset.
+          Then, we can check if the phash of an image matches with the phash of its mutated copies.
+        </p>
+        <p>
+          For picking some mutators, let's try to answer the question of image similarity from the perspective of copyright infringement.
+          For example, when uploading a video of copyrighted material to an online platform,
+          what mutation would someone apply to make sure that the content is still very much recognisable by humans,
+          but might slip by whatever simple automated copyright-detection system is in place?
+          <ol>
+            <li>Horizontal flip</li>
+            <li>Brightness adjustment</li>
+            <li>Hue rotation</li>
+            <li>Blur</li>
+            <li>Sharpen</li>
+            <li>Watermark or logo added</li>
+            <li>Crop</li>
+            <li>Rotate and crop</li>
+          </ol>
+          <em><strong>TODO</strong>: Meer en betere mutators bedenken. Misschien ook niet demo los hebben maar gewoon integreren zodat je live voorbeelden hebt.</em>
+        </p>
+        <h2>Descriptor</h2>
+        <p>
+          <strong>Descriptor</strong> is the term I use for a perceptual hashing algorithm.
+          This is because my first implementations were based on the Color Layout Descriptor, so I used the term descriptor.
+          A descriptor is not neccessarily the same as a perceptual hash, but for the purposes of this research, they can be.
+        </p>
+        <p>
+          Included with the descriptor is also a distance function.
+          For these small image descriptors the most common distance function is the hamming distance, which is to say the amount of bits that differs between two elements.
+        </p>
+        <h2>Experiments</h2>
+        <p>
+          I used the MIRFlickr dataset.
+          For each image, create its mutations and calculate the phashes with each algorithm.
+          All these phashes are then stored in a hash map, taking care to keep the mutated versions separated.
+          For quick nearest neighbour checking, a BK-tree is kept per descriptor with all phashes.
+        </p>
+        <p>
+          We can then use this collection of data to create a precision-recall curve.
+          We first set a threshold for maximum distance to consider two phashes "the same", which we can vary to create a PR-curve.
+          Then, for each threshold, we check this collection of neighbours per mutator to collect our statistics.
+          So, per threshold, per descriptor, per mutator, per unmutated image:
+          <ul>
+            <li>A true positive (<msub><mi>T</mi><mi>p</mi></msub>) is when a mutated base image is within threshold of the original image.</li>
+            <li>A false positive (<msub><mi>F</mi><mi>p</mi></msub>) is when a different mutated image is within threshold of the original image.</li>
+            <li>A false negative (<msub><mi>F</mi><mi>n</mi></msub>) is when the mutated base image is not a near neighbour of its original image.</li>
+          </ul>
+          This gets us values for precision P and recall R per mutator per descriptor: 
+          <math><mrow>
+              <mi>P</mi><mo>=</mo>
+              <mfrac>
+                <msub><mi>T</mi><mi>p</mi></msub>
+                <mrow>
+                  <msub><mi>T</mi><mi>p</mi></msub>
+                  <mo>+</mo>
+                  <msub><mi>F</mi><mi>p</mi></msub>
+                </mrow>
+              </mfrac>
+          </mrow></math>
+          , 
+          <math><mrow>
+              <mi>R</mi><mo>=</mo>
+              <mfrac>
+                <msub><mi>T</mi><mi>p</mi></msub>
+                <mrow>
+                  <msub><mi>T</mi><mi>p</mi></msub>
+                  <mo>+</mo>
+                  <msub><mi>F</mi><mi>n</mi></msub>
+                </mrow>
+              </mfrac>
+          </mrow></math>
+        </p>
+        <p>
+          High precision corresponds to a low false positive rate, and high recall corresponds to a low false negative rate.
+          Ideally, both of these measures should be high for an accurate image similarity detector.
+        </p>
+
+      </article>
       <article id="demo">
         <h1>Demo</h1>
         <p>Select an image to run this demo with. Don't worry, nothing will be sent to any server! All calculations are done in the browser.</p>
@@ -173,22 +287,29 @@ Not same file
           <div class="image-original"></div>
         </div>
       </article>
-      <article id="demo-cont">
+      <article id="demo-resize">
         <h2>Resize</h2>
-        <p>The first step is to size the image down. Since we are aiming for a 64-bit descriptor, the logical target size is 8 by 8 since that will give us 64 pixels to work with.</p>
+        <p>
-        <div id="resize">
+          The first step is to size the image down, and remove all color information.
-          <div class="image-original" id="resize-original"></div>
+          Converting the image to grayscale is done by simply averaging the pixels.
-          <div id="image-resize"></div>
+          Since we are aiming for a 64-bit descriptor, the logical target size is 8 by 8 since that will give us 64 pixels to work with.
-        </div>
+        </p>
         <p>
           It is obvious, but worth noting, that this action is <em>destructive</em>.
           We are throwing away a lot of information here, especially regarding the finer details.
           For now, this is a good thing.
         </p>
         <p>
-          After all, we have to <em>describe</em> our image in only 64-bits, so throwing away information was inevitable.
+          Here we use a nearest neighbour downscaling algorithm that does not preserve aspect ratio.
-          If you squint your eyes the original image can be recognized. Sort of.
+          Press the button to resize your image: <br />
+          <button id="resize-button">Resize</button>
         </p>
+        <div id="resize">
+          <div class="image-original" id="resize-original"></div>
+          <div id="image-resize"></div>
+        </div>
+      </article>
+      <article id="demo-dct">
         <h2>DCT</h2>
         <p>What is DCT?</p>
         <p>Frequency domain. Plaatje. Bla bla.</p>

web/script.js

@@ -11,9 +11,20 @@ let resizedImageContainer = document.getElementById("image-resize");
 let dctCoefficientContainer = document.getElementById("image-dct");
 let formImage = document.getElementById("dctimage");
 
-let originalImage = document.createElement("img");
 let resizedBuffer = null;
 
+// Resets all containers, deletes images, etc
+// Redefines the event handlers for the base image
+function reset() {
+	//imageContainers.forEach((imageContainer) => {
+	for (let imageContainer of imageContainers) {
+		imageContainer.replaceChildren();
+	}
+	resizedImageContainer.replaceChildren();
+	dctCoefficientContainer.replaceChildren();
+	document.getElementById("resize").classList.remove("resize");
+}
+
 // Uses the resized buffer to get DCT coefficients
 function getDCT() {
 	let buf = new Uint8Array(resizedBuffer)
@@ -26,17 +37,6 @@ function getDCT() {
 	}
 }
 
-// Resets all containers, deletes images, etc
-// Redefines the event handlers for the base image
-function reset() {
-	//imageContainers.forEach((imageContainer) => {
-	for (let imageContainer of imageContainers) {
-		imageContainer.replaceChildren();
-	}
-	resizedImageContainer.replaceChildren();
-	dctCoefficientContainer.replaceChildren();
-}
-
 // Result from the resize worker, means we get resized buffer
 resizeWorker.onmessage = function (e) {
 	console.log(e.data);
@@ -46,8 +46,17 @@ resizeWorker.onmessage = function (e) {
 		new Blob([resizedBuffer], { type: 'image/png' })
 	);
 	resizedImageContainer.appendChild(resizedImage);
+
+	// Set the width of the image explictly to help with the animation
+	let image = document.getElementById("resize-original").children[0];
+	image.width = image.width;
+	image.height = image.height;
 }
 
+document.getElementById("resize-button").addEventListener("click", function() {
+	document.getElementById("resize").classList.add("resize");
+});
+
 // User selected an image from disk. Start the demo.
 formImage.addEventListener("change", function() {
 	reset();
@@ -65,5 +74,7 @@ formImage.addEventListener("change", function() {
 			resizeWorker.postMessage(buf);
 		});
 	}
-	document.getElementById("demo-cont").classList.add("visible");
+
+	// Show the rest of the demo.
+	document.getElementById("demo-resize").classList.add("visible");
 });

web/style.css

@@ -71,14 +71,15 @@ code.block {
 
 img {
 	max-width: 100%;
-	image-rendering: pixelated;
 }
 
-#demo-cont {
+/* Demo */
+
+#demo-resize, #demo-dct {
 	visibility: hidden;
 }
 
-#demo-cont.visible {
+#demo-resize.visible, #demo-dct.visible {
 	visibility: visible;
 }
 
@@ -87,37 +88,64 @@ img {
 	min-width: 256px;
 }
 
+/* Resize demo */
+
+#resize {
+	position: relative;
+	height: 512px;
+	width: 100%;
+	transition: height 2s linear;
+}
+
+#resize.resize {
+	height: 256px;
+}
+
+#resize .image-original {
+	position: absolute;
+	width: 100%;
+	height: 512px;
+}
+
 #resize .image-original img {
 	max-height: 50vh;
 	min-height: 257px;
-	max-width: 50vw;
+	max-width: 100%;
 	min-width: 257px;
+	/* height: 512px; */
 }
 
-#image-original img {
+#resize-original {
+	position: relative;
+	height: 512px;
+}
+
+#resize-original img {
 	position: absolute;
 	max-width: 90vw;
-	max-height: 90vh;
+	max-height: 512px;
 	min-width: 257px;
 	min-height: 257px;
 	opacity: 1;
 	transition:
 		width 2s linear,
 		height 2s linear,
-		filter 3s linear,
+		filter 3s linear 2s,
-		opacity 3s ease-in;
+		opacity 3s ease-in 5s;
 }
 
-#image-original img.resize {
+#resize.resize #resize-original img {
 	width: 256px !important;
 	height: 256px !important;
+	filter: grayscale(1) blur(10px);
+	opacity: 0;
 }
 
-#image-original img.grayscale {
+#resize-original img.grayscale {
 	filter: grayscale(1) blur(10px);
 }
 
-#image-original img.fade {
+#resize-original img.fade {
 	opacity: 0;
 }