REVIEW 3 major objections 4 minor 35 references
Douglas-Quaid -- Open Source Image Matching Library
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read An open-source library can automatically match and cluster screenshots with 80% accuracy, F1, true-positive and true-negative rates at a merged distance threshold of about 0.075 on real CERT datasets.
desk verdict The library is real and the design is sensible, but the headline 80% accuracy is an in-sample threshold optimum, not a validated production figure. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The core mechanism is a database of clusters implemented as sorted sets, where each cluster's first element is the most central (representative) picture; a query or new picture is compared first with each cluster's representative pictures and only then with full members of the closest clusters, aiming at sub-linear search. Each picture is stored as a feature vector combining fuzzy perceptual hashes (A-HASH, P-HASH, D-HASH), TLSH, and ORB keypoint descriptors. Distances from these heterogeneous algorithms are not combined as raw numbers; instead each algorithm outputs a distance and a YES/MAYBE/NO decision, and a merging rule combines decisions. A self-calibration step searches thresholds against a small ground-truth sample and target true/false positive and negative rates, producing the configuration file used in production.
What would settle it
Hold out a separately labeled subset of the phishing and onion-domain screenshot pools, run calibration on the standard small sample, then measure accuracy, F1, true-positive rate, and true-negative rate at the calibrated threshold on the held-out subset; if those metrics fall well below 80% (or if the threshold that reaches 80% on calibration gives near-random results on held-out data), the reported performance does not transfer.
Extended reading notes
Core claim
The central claim is that a modular, high-level API library can match and cluster screenshots automatically at a practically useful accuracy on real security datasets. On the paper's evaluation, at a merged distance threshold near 0.075, the system reaches 80% accuracy, an 80% F1 score, 80% true positive rate and 80% true negative rate (with 20% false positives and 20% false negatives). The paper also introduces a calibration procedure that turns user-specified target rates into per-algorithm YES/MAYBE/NO thresholds, so operators do not need to understand the internal distance scales. The intended consequence is a drop in the manual burden of correlating screenshots in threat-intelligence workflows.
Load-bearing premise
The 80% figures assume that the small labeled calibration sample (20-40 images) and the manually constructed ground truth accurately represent the distribution of the full production dataset.
Editorial extensions
If this is right
- Security teams can deploy the library without tuning obscure thresholds; the calibration step sets per-algorithm thresholds from target rates.
- Queries run against cluster representatives rather than all stored images, so response time grows with cluster count rather than image count.
- The YES/MAYBE/NO decision layer can separate visually similar but distinct pages (e.g., PayPal vs non-PayPal), which a single distance threshold would not reliably do.
- Because the library is open source and modular, new feature algorithms can be added without rewriting the matching core; the paper outlines OCR, geometric verification, and human-in-the-loop matching as candidates.
- If deployed in threat-intelligence pipelines, even partial automation of screenshot classification should reduce the number of images analysts must examine by hand.
Reading between the lines
- A natural testable extension is to vary the calibration sample size (20, 40, 100 images) and measure how much the 80% figures degrade on held-out data; the paper does not report this sensitivity.
- The same clustering-plus-decisions design may transfer to other visually repetitive domains, such as duplicate product images or UI regression testing, but the calibration step would need to be rerun per domain.
- The claimed sub-linear query complexity depends on cluster granularity; a threshold set too high would merge all images into one cluster and erase the speed advantage, so speed and accuracy are coupled through calibration.
- If ground-truth errors are systematic (e.g., mislabeled phishing variants), the calibration will absorb them into thresholds, so independent evaluation on fresh labeled data is the real test.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents Douglas-Quaid, an open-source library for image matching and clustering aimed at security analysts who need to classify and correlate screenshots (e.g., phishing pages, sandbox screenshots, onion-domain websites). The library is designed as a modular client-server system using Redis storage, with pluggable matching algorithms (A-HASH, P-HASH, D-HASH, TLSH, ORB) whose distances are converted into human-readable YES/MAYBE/NO decisions and then merged. A self-calibration procedure (Section 4.5) is proposed to set algorithm thresholds from a small ground-truth sample. The paper reports an 80% accuracy, F1 score, true-positive rate, and true-negative rate on a real CERT dataset at a merged distance threshold of about 0.075 (Section 5.1, Figure 12b), discusses speed-related design choices, and outlines future extensions.
Significance. If the 80% figure were a reliable, generalizable performance estimate, this would be a useful contribution: the library is open source, modular, and addresses a real operational need in threat-intelligence workflows. The decision-based merging approach and the self-calibration idea are pragmatic and potentially valuable. The paper also makes its datasets and related tools publicly available, which supports reproducibility. However, the empirical validation as presented does not substantiate the headline performance claim, and the speed evaluation promised in the abstract is not actually reported. The core engineering contribution is credible, but the quantitative claims need substantial revision.
major comments (3)
- [Section 5.1, Figure 12b] The reported 80% accuracy, 80% F1, 80% true-positive rate, and 80% true-negative rate are presented as the best values achievable by sweeping the merged distance threshold on the evaluation set (the text says 'we can at best reach a 80% accuracy... threshold at 0.075'). Because the threshold is the central operating parameter of the matching/clustering pipeline and was selected after inspecting the same data used for scoring, these numbers are in-sample optima, not predictive performance estimates. The paper needs a clear separation between a calibration set (used to set thresholds, e.g., via the Section 4.5 procedure) and a disjoint evaluation set, and the reported metrics should be those obtained at the threshold chosen by calibration, not at the threshold that maximizes the metric on the test set.
- [Section 5.2] The abstract and introduction promise 'quality and speed results', but Section 5.2 describes only the evaluation protocol and a profiling snapshot (Figure 14) of where time is consumed. No quantitative speed results are provided, such as add/request latency versus dataset size, throughput, or scaling curves. Without measured numbers, the claims that the library is 'performance-oriented' and 'much faster than Carl-Hauser' are unsupported.
- [Section 5.1 / Section 2.2] The quality evaluation does not state which dataset (circl-ail-dataset-01 or circl-phishing-dataset-01) was used, how many images were included, or how the ground-truth clusters were constructed. This omission prevents the reader from reproducing the experiment or judging whether the 80% result is specific to a small or easy subset. The paper should specify the dataset, the number of evaluation images, the composition of the ground truth, and the exact evaluation protocol, including whether the evaluation set overlaps with the calibration set used in Section 4.5.
minor comments (4)
- [Section 3.2] The sentence beginning 'the storage uses Redis...' starts with a lowercase letter after a period, and the phrase 'O(√nbpicturesindatabase)' is a typo that should be 'O(√(number of pictures in database))'.
- [Section 3.2 / Section 4.5] The stated calibration subset size is inconsistent: Section 3.2 says '20 to 40 images sampled from a complete dataset', while Section 4.5 says the number 'should not exceed 100'. Please align these numbers.
- [Section 5.1] The text reports '80% true negative, 80% true positive, 20% false positive and 20% false negative'; this is internally consistent (TPR+FN=100%, TNR+FP=100%) but should be accompanied by the actual confusion matrix so readers can verify the rates.
- [References] The citation in the Introduction, '[1][?][2]', contains an unresolved placeholder '?' that should be corrected.
Circularity Check
Headline 80% accuracy is a threshold chosen on the evaluation data itself, not a held-out prediction.
-
fitted input called prediction
[Section 5.1 (Quality), Figure 12b; cf. Section 4.5 (Self-Calibration)]
"Figure 12b shows that if we rely only on distances and put a threshold to evaluate the 'goodness' of matches, we can at best reach a 80% accuracy, 80% F1 score with 80% true negative, 80% true positive, 20% false positive and 20% false negative (threshold at 0.075, roughly)."
Section 4.5 defines the calibration algorithm as fitting thresholds to a ground-truth similarity graph ('the calibration algorithm will seek to optimize the internal parameters of Douglas-Quaid with these inputs'). Section 5.1 then reports the operating point as the best value obtained by 'put a threshold' on the same confusion-matrix data ('we can at best reach'). The 80% numbers at threshold 0.075 are therefore the result of the threshold-selection optimization itself, not an independent evaluation on a held-out set. No train/calibration/evaluation split is described for the reported quality figures, so the accuracy claim is an in-sample fitted maximum rather than a prediction of production performance.
full rationale
No formal derivation or theorem is involved, so there is no self-definitional or uniqueness-import circularity. The main quantitative claim, however, is in-sample: the self-calibration procedure (Section 4.5) optimizes thresholds against a ground-truth clustering, and the Section 5.1 headline is stated as the 'best' result at a distance threshold selected on the evaluation data. That makes the 80% accuracy/F1 figure the value of the fit rather than a validated prediction; a held-out split or externally fixed threshold would be needed to support the production claim. The engineering content of the paper (architecture, modular algorithms, open-source release) is independent and not circular, so the score is moderate rather than maximal.
Assumptions & free parameters
free parameters (4)
- Per-algorithm decision thresholds (yes-to-maybe, maybe-to-no) =
Not given numerically; Figure 10 shows optimized ORB thresholds
- Cluster creation threshold =
Not specified numerically
- Number of representative pictures tested per cluster (PARAMETER) =
Not specified
- Merged distance threshold for reported accuracy =
0.075
assumptions (4)
- domain assumption Normalized Euclidean distances in [0,1] are meaningful across all enabled algorithms
- domain assumption The user-provided ground truth clustering is correct
- ad hoc to paper A 20-40 image sample is representative of the production dataset
- domain assumption Perceptual hash distances and ORB descriptor matching correlate with visual similarity for screenshots
Cite this review
Pith. "Pith review of Douglas-Quaid -- Open Source Image Matching Library." pith.science (2026). https://pith.science/paper/IWCXOJIL
@misc{pith2026190804014,
author = {Pith},
title = {Pith review of: Douglas-Quaid -- Open Source Image Matching Library},
year = {2026},
howpublished = {\url{https://pith.science/paper/IWCXOJIL}},
note = {Machine review of arXiv:1908.04014}
}
read the original abstract
Security analysts need to classify, search and correlate numerous images. Automatic classification tools improve the efficiency of such tasks. However, no open-source and turnkey library was found able to reach this goal. The present paper introduces an Open-Source modular library for the specific cases of visual correlation and Image Matching named Douglas-Quaid. The design of the library, chosen tradeoffs, encountered challenges, envisioned solutions as well as quality and speed results are presented in this paper. We also explore researches directions and future potential developments of the library. Our claim is that even partial automation of screenshots classification would reduce the burden on security teams and that Douglas-Quaid is a step forward in this direction.
Figures
Figures from the paper (13 more)
Reference graph
Works this paper leans on
-
[1]
A. Sampat and A. Haskell, “CNN for task classification using computer screenshots for integration into dynamic calendar/task management systems,” p. 6
-
[2]
Fighting Phishing with Discriminative Keypoint Features,
K. Chen, J. Chen, C. Huang, and C. Chen, “Fighting Phishing with Discriminative Keypoint Features,” vol. 13, no. 3, pp. 56–63
-
[3]
AIL - The design and implementation of an Analysis Information Leak framework,
S. Mokaddem, G. Wagener, and A. Dulaunoy, “AIL - The design and implementation of an Analysis Information Leak framework,” in2018 IEEE International Conference on Big Data (Big Data), pp. 5049–5057
-
[4]
MISP: The Design and Implementation of a Collaborative Threat Intelligence Sharing Platform,
C. Wagner, A. Dulaunoy, G. Wagener, and A. Iklody, “MISP: The Design and Implementation of a Collaborative Threat Intelligence Sharing Platform,” in Proceedings of the 2016 ACM on Workshop on Information Sharing and Collaborative Security - WISCS’16 . ACM Press, pp. 49–56. [Online]. Available: http://dl.acm.org/citation.cfm?doid=2994539.2994542
-
[5]
Large scale reverse image search - A method comparison for almost identical image retrieval,
M. Gaillard and E. Egyed-Zsigmond, “Large scale reverse image search - A method comparison for almost identical image retrieval,” inINFORSID
-
[6]
Implementation and Benchmarking of Perceptual Image Hash Functions
C. Zauner and M. Steinebach, “Implementation and Benchmarking of Perceptual Image Hash Functions.”
-
[7]
Image Matching: An Application-oriented Benchmark
J. Bian, L. Zhang, Y . Liu, W.-Y . Lin, M.-M. Cheng, and I. D. Reid, “Image Matching: An Application-oriented Benchmark.” [Online]. Available: http://arxiv.org/abs/1709.03917
-
[8]
Large-scale image retrieval using transductive support vector machines,
H. Cevikalp, M. Elmas, and S. Ozkan, “Large-scale image retrieval using transductive support vector machines,” vol. 173, pp. 2–12. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S1077314217301364
Show all 35 references
-
[9]
[Online]
[1908.03449] Carl-Hauser – Open Source Image Matching Algorithms Benchmarking Framework. [Online]. Available: https://arxiv.org/abs/1908.03449
1908 arXiv
-
[10]
VisJSClassificator – Manual Visual Collaborative Classification Graph-based Tool
V . Falconieri, “VisJSClassificator – Manual Visual Collaborative Classification Graph-based Tool.” [Online]. Available: http://arxiv.org/abs/1908.02941
1908 arXiv
-
[11]
Open Dataset of Phishing and Tor Hidden Services Screen-captures
——, “Open Dataset of Phishing and Tor Hidden Services Screen-captures.” [Online]. Available: http://arxiv.org/abs/1908.02449
1908 arXiv
-
[12]
[Online]
Repository Structure and Python. [Online]. Available: https://www.kennethreitz.org/essays/ repository-structure-and-python
-
[13]
F.-P. Lin. Good logging practice in Python. [Online]. Available: https://fangpenlin.com/posts/2012/08/26/ good-logging-practice-in-python/
2012
-
[14]
TLSH – A Locality Sensitive Hash,
J. Oliver, C. Cheng, and Y . Chen, “TLSH – A Locality Sensitive Hash,” in2013 Fourth Cybercrime and Trustworthy Computing Workshop. IEEE, pp. 7–13. [Online]. Available: http://ieeexplore.ieee.org/document/6754635/
-
[15]
Distinctive Image Features from Scale-Invariant Keypoints,
D. G. Lowe, “Distinctive Image Features from Scale-Invariant Keypoints,” vol. 60, no. 2, pp. 91–110. [Online]. Available: http://link.springer.com/10.1023/B:VISI.0000029664.99615.94
-
[16]
Anatomy of the SIFT Method,
I. R. Otero and M. Delbracio, “Anatomy of the SIFT Method,” vol. 4, pp. 370–396. [Online]. Available: http://www.ipol.im/pub/art/2014/82/
2014
-
[17]
SURF: Speeded Up Robust Features,
H. Bay, T. Tuytelaars, and L. Van Gool, “SURF: Speeded Up Robust Features,” inComputer Vision – ECCV 2006, A. Leonardis, H. Bischof, and A. Pinz, Eds. Springer Berlin Heidelberg, vol. 3951, pp. 404–417. [Online]. Available: http://link.springer.com/10.1007/11744023_32
2006 doi
-
[18]
ORB: An efficient alternative to SIFT or SURF,
E. Rublee, V . Rabaud, K. Konolige, and G. Bradski, “ORB: An efficient alternative to SIFT or SURF,” in 2011 International Conference on Computer Vision . IEEE, pp. 2564–2571. [Online]. Available: http://ieeexplore.ieee.org/document/6126544/
2011
-
[19]
A comparative analysis of SIFT, SURF, KAZE, AKAZE, ORB, and BRISK,
S. A. K. Tareen and Z. Saleem, “A comparative analysis of SIFT, SURF, KAZE, AKAZE, ORB, and BRISK,” in 2018 International Conference on Computing, Mathematics and Engineering Technologies (iCoMET). IEEE, pp. 1–10. [Online]. Available: https://ieeexplore.ieee.org/document/8346440/
2018
-
[20]
[Online]
CIRCL » AIL - Analysis Information Leak framework - Training Materials. [Online]. Available: https://www.circl.lu/services/ail-training-materials/
-
[21]
Open Source testing framework for image correlation, distance and analysis: CIRCL/carl-hauser,
“Open Source testing framework for image correlation, distance and analysis: CIRCL/carl-hauser,” CIRCL - Computer Incident Response Center Luxembourg. [Online]. Available: https://github.com/CIRCL/carl-hauser
-
[22]
Open source software for image correlation, distance and analysis: CIRCL/douglas-quaid,
“Open source software for image correlation, distance and analysis: CIRCL/douglas-quaid,” CIRCL - Computer Incident Response Center Luxembourg. [Online]. Available: https://github.com/CIRCL/douglas-quaid
-
[23]
Robust image matching based on the information of SIFT,
J. Dou, Q. Qin, and Z. Tu, “Robust image matching based on the information of SIFT,” vol. 171, pp. 850–861. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0030402618309021 19 Douglas-Quaid - Open Source Image Matching Library A PREPRINT
-
[24]
A Survey on Image Retrieval Methods,
A. M. Marshall and D. S. Gunasekaran, “A Survey on Image Retrieval Methods,” p. 15
-
[25]
MISP (core software): Open Source Threat Intelligence and Sharing Platform (formely known as Malware Information Sharing Platform) - MISP/MISP,
“MISP (core software): Open Source Threat Intelligence and Sharing Platform (formely known as Malware Information Sharing Platform) - MISP/MISP,” MISP Project. [Online]. Available: https://github.com/MISP/MISP
-
[26]
[Online]
Redis. [Online]. Available: https://redis.io/
-
[27]
URL Abuse: A Versatile Software for URL review, analysis and black-list reporting - CIRCL/url-abuse,
“URL Abuse: A Versatile Software for URL review, analysis and black-list reporting - CIRCL/url-abuse,” CIRCL - Computer Incident Response Center Luxembourg. [Online]. Available: https://github.com/CIRCL/url-abuse
-
[28]
Classificator for pictures matching and clustering. Fast and visual.: Vincent- CIRCL/visjs_classificator
Vincent-CIRCL, “Classificator for pictures matching and clustering. Fast and visual.: Vincent- CIRCL/visjs_classificator.” [Online]. Available: https://github.com/Vincent-CIRCL/visjs_classificator 20 Douglas-Quaid - Open Source Image Matching Library A PREPRINT Part I Appendices ...
-
[29]
Create Worker Handler
-
[30]
Start Requester Workers
-
[31]
Start Feature Workers
-
[32]
more path independant
Start Cache DB REDIS CACHE INSTANCE Distance Engine Database Accessor Factorize common code DB Common Feature ExtractorREDIS CACHE INSTANCE REDIS CACHE INSTANCE REDIS STORAGE INSTANCE API - Server side add_picture request_picture request_result ping Dump DB graph Wait until ad...
-
[33]
Create DB Handler NOT YET NOT YET Expiration time : 24H (in configuration file) ID Distance Decision ClusterMatch ImageID ClusterID Distance Decision ImageMatch Name Distance Decision AlgoMatch YES/MAYBE/NO DecisionTypes Allow to keep track of top N elements on a specific attribu...
-
[34]
Get distance picture to cluster b
Get top Matching clusters a. Get distance picture to cluster b. Get distance picture to picture c. Get best matching clusters
-
[35]
no result
Get top Matching pictures of these clusters b. Get distance picture to picture c. if best picture distance < matching distance return list of matches else return "no result" Figure 15: Software Components architecture 23 Douglas-Quaid - Open Source Image Matching Library A PRE...
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.