{"id":"11f05462-f0f4-4670-a90b-11d2dcf14fb6","arxiv_id":"1908.04014","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Douglas-Quaid is an open-source image matching library that combines fuzzy hashes and ORB with decision fusion and self-calibration, reporting 80% accuracy on a CERT screenshot dataset.","lead":"A security-focused open-source library, Douglas-Quaid, matches and clusters screenshots using perceptual hashes and ORB features, with tunable thresholds and a self-calibration routine. The paper reports about 80% accuracy on a real CERT dataset and argues that even partial automation eases analysts' work.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 80% accuracy is a threshold tuned on the same evaluation set; without a held-out split the headline performance claim is an in-sample upper bound, not a verified production figure.","rationale":"The paper is an honest engineering report: the library is released, the datasets are public, and the design choices (decision fusion, cluster-based storage, self-calibration) are plausible. I see no reason to doubt the artifact itself. The problem is specifically the evidential weight of the headline quantitative claim. The 80% figure is the strongest concrete claim, and it is derived by threshold scanning on the evaluation data without a held-out check. That is the weakest load-bearing link. A calibration/evaluation split is the standard minimal fix and is feasible here because both the code and the data are public. If the split confirms roughly 80%, the conditional acceptance is vindicated; if not, the contribution remains the library, but the performance claim should be downgraded. This is why I keep the verdict conditional rather than moving to accept or reject. The reader's weakest_assumption overlaps with mine, and their rationale already mentions threshold fitting, but my concern is more narrowly about selection-on-the-test-set, hence partial agreement.","tokens_in":15021,"tokens_out":5382,"duration_ms":53418,"concrete_test":"Re-run the evaluation with a strict split. Use one subset (e.g., the 20-40 image calibration protocol of Section 4.5, or a random half of the dataset) to choose the merged distance threshold and, if applicable, the per-algorithm YES/MAYBE/NO thresholds. Apply the chosen configuration unchanged to the disjoint held-out subset and recompute accuracy, F1, TPR, and TNR at the fixed threshold. Repeat with 5-fold cross-validation over the full labeled dataset and report mean and 95% CI. If the held-out figures are materially below 80% (e.g., more than 5 percentage points), the central claim should be reframed as an in-sample upper bound rather than expected production performance.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 5.1 reports 80% accuracy/F1/TPR/TNR \"at best\" at a merged distance threshold of about 0.075 (Figure 12b). The wording and the absence of any described train/validation split indicate that this operating point was selected after inspecting the confusion matrix of the same data used for scoring. Selecting a threshold on the evaluation set converts the reported numbers from predictive estimates into in-sample maxima. The threshold is the central control parameter of the whole matching/clustering pipeline, and Section 4.5 itself acknowledges that thresholds must be calibrated on a small sample before production. No evidence is given that the 0.075 threshold, or any threshold produced by the Section 4.5 calibration, retains 80% performance on a disjoint set of screenshots. Without that, the headline claim overstates what is known about production behavior. The reader's ground-truth and representativeness concerns are valid but secondary: even a perfectly labeled ground truth would not rescue a number obtained by selecting the threshold on the evaluation set.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Douglas-Quaid, an open-source library for image matching and clustering aimed at security analysts who need to classify and correlate screenshots (e.g., phishing pages, sandbox screenshots, onion-domain websites). The library is designed as a modular client-server system using Redis storage, with pluggable matching algorithms (A-HASH, P-HASH, D-HASH, TLSH, ORB) whose distances are converted into human-readable YES/MAYBE/NO decisions and then merged. A self-calibration procedure (Section 4.5) is proposed to set algorithm thresholds from a small ground-truth sample. The paper reports an 80% accuracy, F1 score, true-positive rate, and true-negative rate on a real CERT dataset at a merged distance threshold of about 0.075 (Section 5.1, Figure 12b), discusses speed-related design choices, and outlines future extensions.","tokens_in":15237,"tokens_out":3226,"duration_ms":31638,"significance":"If the 80% figure were a reliable, generalizable performance estimate, this would be a useful contribution: the library is open source, modular, and addresses a real operational need in threat-intelligence workflows. The decision-based merging approach and the self-calibration idea are pragmatic and potentially valuable. The paper also makes its datasets and related tools publicly available, which supports reproducibility. However, the empirical validation as presented does not substantiate the headline performance claim, and the speed evaluation promised in the abstract is not actually reported. The core engineering contribution is credible, but the quantitative claims need substantial revision.","major_comments":[{"comment":"The reported 80% accuracy, 80% F1, 80% true-positive rate, and 80% true-negative rate are presented as the best values achievable by sweeping the merged distance threshold on the evaluation set (the text says 'we can at best reach a 80% accuracy... threshold at 0.075'). Because the threshold is the central operating parameter of the matching/clustering pipeline and was selected after inspecting the same data used for scoring, these numbers are in-sample optima, not predictive performance estimates. The paper needs a clear separation between a calibration set (used to set thresholds, e.g., via the Section 4.5 procedure) and a disjoint evaluation set, and the reported metrics should be those obtained at the threshold chosen by calibration, not at the threshold that maximizes the metric on the test set.","section":"Section 5.1, Figure 12b"},{"comment":"The abstract and introduction promise 'quality and speed results', but Section 5.2 describes only the evaluation protocol and a profiling snapshot (Figure 14) of where time is consumed. No quantitative speed results are provided, such as add/request latency versus dataset size, throughput, or scaling curves. Without measured numbers, the claims that the library is 'performance-oriented' and 'much faster than Carl-Hauser' are unsupported.","section":"Section 5.2"},{"comment":"The quality evaluation does not state which dataset (circl-ail-dataset-01 or circl-phishing-dataset-01) was used, how many images were included, or how the ground-truth clusters were constructed. This omission prevents the reader from reproducing the experiment or judging whether the 80% result is specific to a small or easy subset. The paper should specify the dataset, the number of evaluation images, the composition of the ground truth, and the exact evaluation protocol, including whether the evaluation set overlaps with the calibration set used in Section 4.5.","section":"Section 5.1 / Section 2.2"}],"minor_comments":[{"comment":"The sentence beginning 'the storage uses Redis...' starts with a lowercase letter after a period, and the phrase 'O(√nbpicturesindatabase)' is a typo that should be 'O(√(number of pictures in database))'.","section":"Section 3.2"},{"comment":"The stated calibration subset size is inconsistent: Section 3.2 says '20 to 40 images sampled from a complete dataset', while Section 4.5 says the number 'should not exceed 100'. Please align these numbers.","section":"Section 3.2 / Section 4.5"},{"comment":"The text reports '80% true negative, 80% true positive, 20% false positive and 20% false negative'; this is internally consistent (TPR+FN=100%, TNR+FP=100%) but should be accompanied by the actual confusion matrix so readers can verify the rates.","section":"Section 5.1"},{"comment":"The citation in the Introduction, '[1][?][2]', contains an unresolved placeholder '?' that should be corrected.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads like a preprint-style system description rather than a complete journal paper. The main issue is that the 80% performance figure is selected by threshold optimization on the evaluation set, which makes the headline claim an in-sample upper bound. This is fixable by re-running the evaluation with a proper calibration/validation/test split and reporting the performance at the calibrated threshold. The authors should also decide whether to provide real speed measurements, since the abstract explicitly promises speed results. The open-source artifact and datasets are valuable, and the design discussion is useful, so I believe the paper can be revised to meet journal standards if the empirical claims are brought in line with what was actually measured."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nDouglas-Quaid is a genuinely useful artifact: an open-source, modular image-matching library with a clean API and a clustering storage layer. The paper describes the design honestly, including the decision fusion layer (YES/MAYBE/NO) and a self-calibration routine. The code and datasets are public, which makes this reproducible and gives it real value to the CERT/incident-response community.\n\nThe main problem is the performance claim. The reported 80% accuracy/TPR/TNR/F1 at a merged distance threshold around 0.075 is the best operating point selected from the same evaluation set. The text even says 'we can at best reach,' which is an in-sample optimum, not a held-out result. There are no error bars, no repeated trials, and no validation on a disjoint sample. That matters because the threshold is the central control parameter. Without a separate calibration set or cross-validation, the numbers do not support production claims. This is not a fatal flaw—the paper is transparent about it being an upper bound—but it means the headline should not be taken at face value.\n\nA second soft spot is the lack of baseline comparisons. The paper references Carl-Hauser but does not report how Douglas-Quaid compares to it or to simpler approaches like just clustering with per-algorithm thresholds. Given the library is described as a performance-oriented implementation of Carl-Hauser's best algorithms, some direct comparison would be natural.\n\nThe self-calibration section is a good idea but described incompletely. It says the calibration subset should be 20-40 images sampled from production, and that thresholds are optimized on that subset. If the evaluation in Section 5.1 used the full dataset rather than a held-out calibration subset, the connection to the self-calibration method is unclear. The paper would be stronger if it showed the calibration-set threshold and then the evaluation-set performance at that threshold.\n\nMinor issues: Figure 10's 'sanity check' line is garbled and needs fixing; some references are incomplete; the paper is uneven in prose quality. These are fixable.\n\nWho is this for? Security analysts and tool builders who need a quick, free way to cluster and search screenshots. They will benefit from the library even though the paper's validation is thin. A rigorous referee could help turn this into a solid software paper by requesting a proper train/test split, a baseline comparison, and error bars.\n\nMy call: it deserves a serious referee, not a desk reject, because the artifact is real and the design ideas are sound. But the performance claims need re-validation before the numbers are cited.","headline":"The library is real and the design is sensible, but the headline 80% accuracy is an in-sample threshold optimum, not a validated production figure.","tokens_in":15775,"tokens_out":2777,"would_cite":true,"duration_ms":28362,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An open-source library can automatically match and cluster screenshots with 80% accuracy, F1, true-positive and true-negative rates at a merged distance threshold of about 0.075 on real CERT datasets.","keywords":["Threat Intelligence","Phishing","Dataset","Open Source","Security","CERT","Incident Response","Visual Detection"],"falsifier":"Hold out a separately labeled subset of the phishing and onion-domain screenshot pools, run calibration on the standard small sample, then measure accuracy, F1, true-positive rate, and true-negative rate at the calibrated threshold on the held-out subset; if those metrics fall well below 80% (or if the threshold that reaches 80% on calibration gives near-random results on held-out data), the reported performance does not transfer.","tokens_in":14782,"feed_emoji":"🖼️","tokens_out":4926,"duration_ms":46662,"temperature":0.7,"pith_summary":"Douglas-Quaid is an open-source image-matching library aimed at security analysts who must sort and correlate screenshots of phishing pages, sandboxed sites, and onion services. The paper argues that even partial automation of this sorting would reduce analyst workload, and claims the library can reach 80% accuracy, 80% F1, 80% true positives, and 80% true negatives at a merged distance threshold of about 0.075 on real CERT datasets. Instead of exposing obscure thresholds, the library calibrates its own parameters from a small labeled sample and a ground-truth file. The design stores each image as a set of features and clusters similar images around representative pictures, so queries compare against clusters rather than every stored image.","feed_headline":"Screenshot matcher hits 80% accuracy on real CERT images","feed_subtitle":"Open-source library clusters phishing and hidden-service screenshots automatically, calibrating its own thresholds from a small labeled…","key_machinery":"The core mechanism is a database of clusters implemented as sorted sets, where each cluster's first element is the most central (representative) picture; a query or new picture is compared first with each cluster's representative pictures and only then with full members of the closest clusters, aiming at sub-linear search. Each picture is stored as a feature vector combining fuzzy perceptual hashes (A-HASH, P-HASH, D-HASH), TLSH, and ORB keypoint descriptors. Distances from these heterogeneous algorithms are not combined as raw numbers; instead each algorithm outputs a distance and a YES/MAYBE/NO decision, and a merging rule combines decisions. A self-calibration step searches thresholds against a small ground-truth sample and target true/false positive and negative rates, producing the configuration file used in production.","core_discovery":"The central claim is that a modular, high-level API library can match and cluster screenshots automatically at a practically useful accuracy on real security datasets. On the paper's evaluation, at a merged distance threshold near 0.075, the system reaches 80% accuracy, an 80% F1 score, 80% true positive rate and 80% true negative rate (with 20% false positives and 20% false negatives). The paper also introduces a calibration procedure that turns user-specified target rates into per-algorithm YES/MAYBE/NO thresholds, so operators do not need to understand the internal distance scales. The intended consequence is a drop in the manual burden of correlating screenshots in threat-intelligence workflows.","pith_inferences":["A natural testable extension is to vary the calibration sample size (20, 40, 100 images) and measure how much the 80% figures degrade on held-out data; the paper does not report this sensitivity.","The same clustering-plus-decisions design may transfer to other visually repetitive domains, such as duplicate product images or UI regression testing, but the calibration step would need to be rerun per domain.","The claimed sub-linear query complexity depends on cluster granularity; a threshold set too high would merge all images into one cluster and erase the speed advantage, so speed and accuracy are coupled through calibration.","If ground-truth errors are systematic (e.g., mislabeled phishing variants), the calibration will absorb them into thresholds, so independent evaluation on fresh labeled data is the real test."],"forward_implications":["Security teams can deploy the library without tuning obscure thresholds; the calibration step sets per-algorithm thresholds from target rates.","Queries run against cluster representatives rather than all stored images, so response time grows with cluster count rather than image count.","The YES/MAYBE/NO decision layer can separate visually similar but distinct pages (e.g., PayPal vs non-PayPal), which a single distance threshold would not reliably do.","Because the library is open source and modular, new feature algorithms can be added without rewriting the matching core; the paper outlines OCR, geometric verification, and human-in-the-loop matching as candidates.","If deployed in threat-intelligence pipelines, even partial automation of screenshot classification should reduce the number of images analysts must examine by hand."],"supporting_citations":[{"why":"Supplies the algorithm benchmarking framework from which the library's performance-oriented algorithm set is drawn.","marker":"[9]"},{"why":"Used to build and visualize ground-truth clusters for the evaluation.","marker":"[10]"},{"why":"Provides the real phishing and hidden-service screenshot datasets used for calibration and quality measurement.","marker":"[11]"},{"why":"Comparative benchmark of keypoint detectors that led the authors to focus on ORB.","marker":"[19]"},{"why":"Defines the ORB keypoint descriptor and matching algorithm used as a core feature extractor.","marker":"[18]"},{"why":"Defines the TLSH locality-sensitive hash used as one of the fuzzy-hash features.","marker":"[14]"}],"fun_headline_variants":["Douglas-Quaid: open-source image matcher hits 80% accuracy","Open-source screenshot matcher auto-calibrates for 80% accuracy","Calibrated screenshot clustering: 80% accurate, open source","Douglas-Quaid: auto-threshold screenshot matching at 80%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 80% figures assume that the small labeled calibration sample (20-40 images) and the manually constructed ground truth accurately represent the distribution of the full production dataset.","fun_headline_variants_meta":{"raw":{"variants":["Douglas-Quaid: open-source image matcher hits 80% accuracy","Open-source screenshot matcher auto-calibrates for 80% accuracy","Calibrated screenshot clustering: 80% accurate, open source","Douglas-Quaid: auto-threshold screenshot matching at 80%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001085,"raw_usage":{"total_tokens":4459,"prompt_tokens":793,"completion_tokens":3666,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":409,"completion_tokens_details":{"reasoning_tokens":3589}},"tokens_in":409,"tokens_out":3666,"duration_ms":23719,"temperature":1.0,"reasoning_tokens":3589,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:54:04.010215+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out a separately labeled subset of the phishing and onion-domain screenshot pools, run calibration on the standard small sample, then measure accuracy, F1, true-positive rate, and true-negative rate at the calibrated threshold on the held-out subset; if those metrics fall well below 80% (or if the threshold that reaches 80% on calibration gives near-random results on held-out data), the reported performance does not transfer.","supporting_citations":[{"cited_title":"Carl-Hauser -- Open Source Image Matching Algorithms Benchmarking Framework","cited_arxiv_id":"1908.03449","evidence_quote":"Supplies the algorithm benchmarking framework from which the library's performance-oriented algorithm set is drawn."},{"cited_title":"VisJSClassificator -- Manual Visual Collaborative Classification Graph-based Tool","cited_arxiv_id":"1908.02941","evidence_quote":"Used to build and visualize ground-truth clusters for the evaluation."},{"cited_title":"Open Dataset of Phishing and Tor Hidden Services Screen-captures","cited_arxiv_id":"1908.02449","evidence_quote":"Provides the real phishing and hidden-service screenshot datasets used for calibration and quality measurement."},{"cited_title":"A comparative analysis of SIFT, SURF, KAZE, AKAZE, ORB, and BRISK,","cited_arxiv_id":null,"evidence_quote":"Comparative benchmark of keypoint detectors that led the authors to focus on ORB."}],"review_version":1}