{"id":"2e2049ca-e5da-48cb-b387-043d8ed8f6dc","arxiv_id":"1908.03449","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Carl-Hauser is an open-source framework that benchmarks image matching algorithms on phishing website screenshots, with results showing ORB edges out fuzzy hashes on accuracy but trails on speed.","lead":"This paper presents an open-source tool called Carl-Hauser that benchmarks image matching algorithms on phishing website screenshots, along with open datasets. It compares fuzzy hashing and ORB feature matching, reporting that ORB gets more true positives while hashes are much faster.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ORB-vs-fuzzy-hash conclusion in Section 5 is not supported by the reported experiment: the best ORB configuration is selected from a large parameter sweep on the same single dataset, with no held-out set, error bars, or independent ground-truth validation, and the raw margin is smaller than…","rationale":"The reader correctly identifies the manually built ground-truth graph as a weak point: Appendix 10 defines true positive as intersection with that graph, and Section 3 says the ground truth 'needs to be provided' and can be built with a tool from the same team, without stated labeling criteria or inter-annotator validation. I agree that this threatens the validity of all quality scores. However, I would put the more acute problem earlier in the inference chain: even if the ground truth were perfect, the paper selects the best ORB configuration from a large parameter sweep and then compares it to fixed hash baselines on the same data. This is test-set selection bias, and the Appendix 9 tables make the effect visible: ORB configurations with 'ENABLED' versus 'DISABLED' preprocessing or different matcher settings range from 0.0 to 0.65 true-positive. A held-out split or pre-registered parameter choice is therefore the single check most likely to change the Section 5 conclusion. The framework contribution deserves credit: the code is open-source, the datasets are published, and the pipeline is described in enough detail to reproduce the experiment, which supports a conditional rather than rejecting verdict. Still, the stated 'about 10%' margin is not robustly established, so the empirical claim should be relabeled preliminary until the split-half test is run. This does not change the reader's conditional verdict, so I mark the outcome as unchanged.","tokens_in":16425,"tokens_out":5558,"duration_ms":60156,"concrete_test":"Run a pre-registered, split-half evaluation on the Appendix 9 dataset (207 screenshots). First, split images at the level of ground-truth cliques (or by originating website) into two disjoint halves. On half A, run the same configuration sweep and pick the ORB configuration with the best true-positive rate; freeze all parameters and thresholds. On held-out half B, compute true-positive rates for that ORB configuration and for D-Hash, A-Hash, and TLSH against the ground-truth graph, with 95% bootstrap confidence intervals over images. Independently rebuild the ground-truth edges for half B with two annotators who do not see the original graph; report inter-annotator agreement and score sensitivity. If the held-out ORB margin over the best hash is not positive and at least 10% relative, the Section 5 claim should be downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline empirical claim (Section 5: 'ORB outperforms fuzzy-hash algorithms ... about 10% more true positive matches') is the part that would make the evaluation informative, and it is not established by the reported procedure. Appendix 9 reports single-run point estimates on one dataset of 207 phishing screenshots. The framework's design (Section 3) is to 'explore the parameters space' and generate many configurations; the intersection matrix in Appendix 10 shows dozens of ORB variants. The tables then display best-scoring ORB configurations after this search on the same data used for scoring. That is selection on the test set: the maximum of many noisy estimates is biased upward, so the comparison to fixed hash baselines (AHash, DHash, TLSH) is tilted toward ORB. The same tables show ORB configurations with true-positive scores from 0.0 to 0.65, so 'ORB outperforms' is a property of a chosen configuration, not of ORB as a method. There are no confidence intervals, bootstrap replicates, or repeated runs, and the margin is smaller than stated: best ORB raw TP is 0.65263 versus D-Hash 0.60386 on raw PNG, about 4.9 percentage points rather than 10; normalized values are 0.776 versus 0.718, about 8% relative. All scores also depend on a single manually built ground-truth graph (Appendix 10, Eq. 1) with no documented criteria for an edge, no inter-annotator agreement, and a labeling tool authored by the same team; label noise would shift every algorithm's score and can change the ranking. The framework itself is real and open-source, and the paper frames results partly as a usage example, but the empirical conclusion as stated overreaches.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Carl-Hauser, an open-source framework for benchmarking image-matching algorithms, and applies it to a dataset of phishing website screenshots. The framework automates the evaluation pipeline: given an input folder of images and a ground-truth graph, it runs multiple algorithms across a parameter sweep, records quality and timing metrics, and produces overview tables, intersection matrices, and graph outputs. The paper's stated main contribution is the benchmarking framework itself, and it further reports an illustrative comparison between ORB (OpenCV) feature matching and fuzzy-hash algorithms (ImageHash, TLSH), concluding in Section 5 that 'ORB outperforms fuzzy-hash algorithms in almost all sounds configuration (about 10% more true positive matches).' The dataset and code are offered as open data and open source.","tokens_in":40,"tokens_out":3040,"duration_ms":79033,"significance":"If the framework operates as described, it provides a reusable, open-source evaluation harness for a practical security task (phishing screenshot matching), and the public release of the phishing screenshot dataset is a useful community resource. The paper also gives a clear scoring definition (edge intersection ratio with a ground-truth graph, normalized by the maximum attainable score) and documents practical observations about text noise and parameter sensitivity of ORB. However, the headline empirical claim about ORB versus fuzzy hashes is not established by the reported procedure, for reasons detailed below. The significance of the paper therefore rests mainly on the tool and dataset rather than on the quantitative comparison.","major_comments":[{"comment":"The claim that 'ORB outperforms fuzzy-hash algorithms ... about 10% more true positive matches' is not supported by the reported experiment. Section 3 describes the framework as exploring the parameter space and generating many configurations; Appendix 9 then presents, as the 'best' ORB results, the highest true-positive scores selected from that sweep on the same single 207-image dataset. Because the best of many noisy estimates is biased upward when scored on the same data used for selection, the comparison against fixed hash baselines is tilted toward ORB. No held-out set, cross-validation, repeated runs, or confidence intervals are provided. Moreover, the stated margin is quantitatively inaccurate: the best raw ORB true-positive rate in the PNG table is 0.65263 versus 0.60386 for D-Hash, a difference of about 4.9 percentage points, not 10; the normalized values (0.77641 vs 0.71839) give about 8.1 percent relative improvement. The empirical comparison should be re-run with proper model selection (e.g., validation/tuning separate from test), with uncertainty estimates, and the textual claim corrected to match the actual numbers.","section":"Section 5 and Appendix 9"},{"comment":"The evaluation's validity depends entirely on a single manually built ground-truth graph, but the paper gives no criteria for what constitutes a match, no inter-annotator agreement, and no sensitivity analysis. Section 3 states that the ground-truth file 'can easily be built with VisJS-Classificator[9]', and Appendix 10 defines the true-positive score as the intersection ratio between the algorithm's output graph and this ground-truth graph. Since the labeling tool is authored by the same team and the labeling procedure is undocumented, label noise or systematic bias would shift every algorithm's score and could change the ORB-versus-hash ordering. The authors should document the labeling instructions, report how the ground-truth graph was constructed (including the clique structure and outlier treatment), and ideally provide at least two independent annotators with a measure of agreement.","section":"Section 3 and Appendix 10"},{"comment":"The intersection matrix in Figure 13 is unreadable: the axis labels are an unbroken string of configuration names, and the figure itself contains the note 'CONCATENE AND MAKE IT MORE READABLE : FOR NEXT RELEASE'. As printed, the claimed observations about clusters of similar ORB configurations and the orthogonality between hash-based and ORB-based outputs cannot be verified by a reader. The matrix should be replaced with a legible figure with abbreviated but unique labels, or the underlying data should be provided in a machine-readable table.","section":"Appendix 10, Figure 13"}],"minor_comments":[{"comment":"There are several typos and stylistic issues: 'litterature' in the abstract, 'Is this paper' should be 'In this paper', and 'bench-marking' is an awkward hyphenation in Section 1.1.","section":"Abstract and Section 1"},{"comment":"The text contains an unresolved reference: 'Figure 11 and ?? are examples.' This should be fixed to a proper figure citation or rewritten.","section":"Section 4.2.2"},{"comment":"The word 'preformance' appears in the scalability bullet; it should be 'performance'.","section":"Section 5"},{"comment":"The dataset size is described as '470+ pictures' in Section 2 but Appendix 9 states the evaluation was conducted on 207 screenshots. This inconsistency should be clarified (e.g., the full dataset versus the evaluation subset).","section":"Section 2 and Appendix 9"},{"comment":"Several references are incomplete or inconsistently formatted (e.g., entries with missing volume/pages such as [1], and some with only 'pp. 6'). The bibliography should be cleaned up.","section":"References"},{"comment":"Equation (1) defines Iratio but the notation '♯' for set cardinality is not introduced; a brief explanation would improve readability.","section":"Appendix 10"}],"recommendation":"major_revision","confidential_remarks":"The paper is best read as a tool/system paper rather than a rigorous comparative evaluation. The framework and dataset are genuine contributions, but the empirical comparison needs substantial strengthening (held-out evaluation, uncertainty quantification, and a documented ground-truth process) before the quantitative claims can be considered reliable. The informal presentation style and the self-referential ground-truth tool suggest that the paper may be better suited to a workshop or a systems-oriented venue, but within the current venue it needs the revisions described above."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things. First, this is a genuine artifact paper: Carl-Hauser is open source, the datasets are released, and the framework is described well enough to be usable. Second, the empirical conclusion in Section 5 \\u2014 that ORB outperforms fuzzy-hash algorithms by about 10% true positives \\u2014 is not supported by the data as presented. The best ORB configuration was picked from a large parameter sweep on the same single dataset of 207 screenshots, with no held-out set, no error bars, and no repeated runs. The raw margin over D-Hash is closer to 4.9 percentage points (0.65263 vs 0.60386), and normalized it's about 8% relative. That's a real difference but the \\u201cabout 10%\\u201d phrasing overstates it, and selection-on-test-set bias is a live concern.\n\nWhat is genuinely new: an open benchmarking harness tailored to screenshot/website images, an open phishing-screenshot dataset, and a first pass at comparing ORB variants against perceptual hashes in that domain. Existing benchmarks (Bian et al., Zauner et al.) are natural-image or general-purpose; this is a different target. The framework's design is sensible \\u2014 configuration generation, execution handler, graph output, inclusion matrix \\u2014 and the visualizations are useful for spotting failure modes. The paper is also honest: Section 5 lists real challenges (text noise, parameter sensitivity, cost), and the conclusion explicitly frames the results as a usage example of the dataset.\n\nThe soft spots are real but localized. The ground truth is a manually built graph, made with the author's own tool, with no stated edge criteria and no inter-annotator agreement. That would be fine for an illustrative demo but not for a headline comparative claim. The manuscript itself is unfinished: broken figure references, at least one \\u201c??\\u201d citation, and the inclusion matrix appendix literally says \\u201cCONCATENE AND MAKE IT MORE READABLE : FOR NEXT RELEASE.\\u201d That will not survive a serious review without cleanup.\n\nWho is this for: security analysts and tool builders who want a turnkey way to evaluate image-matching algorithms on screenshot collections. For them, the framework and dataset are useful even if the specific comparison is weakened. A serious referee should engage with this \\u2014 the artifact deserves attention \\u2014 but the empirical section needs either major revision (held-out evaluation, error bars, or independent ground truth) or an explicit reframing as illustrative rather than conclusive.\n\nRecommendation: send to peer review, but expect heavy revision on the evaluation. The framework and data are the contribution; the ORB-vs-hash ranking should not be published as a finding in its current form.","headline":"A real, useful open-source benchmarking framework and dataset for phishing-screenshot matching, but the paper's headline ORB-beats-hash claim is selected on the test set and not supported by the reported experiment.","tokens_in":17293,"tokens_out":2410,"would_cite":true,"duration_ms":23993,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ORB beats fuzzy hashes on phishing screenshots","keywords":["image matching","benchmarking framework","phishing detection","screenshot similarity","ORB","fuzzy hashing","security automation","open data"],"falsifier":"Run the framework on the same phishing dataset but with a ground-truth graph built independently by two or more annotators who label matches without seeing each other's work, then compare ORB's normalized true-positive rate to D-Hash's; if the roughly 10 percentage-point gap shrinks or disappears, the headline comparison is an artifact of the original labels.","tokens_in":16248,"feed_emoji":"🖼️","tokens_out":8155,"duration_ms":77310,"temperature":0.7,"pith_summary":"This paper introduces a free and open-source benchmarking framework that evaluates image-matching algorithms on folders of screenshots and scores them against a manually built ground-truth graph. The main claim is that on phishing and onion-site screenshot datasets, ORB keypoint matching outperforms fuzzy perceptual hashes in nearly all tested configurations, with roughly 10% more true positive matches. The framework produces quality metrics, timing and memory statistics, graph visualizations, and configurable preprocessing such as OCR-based text hiding. If correct, it gives security teams a turnkey way to compare matching algorithms on their own data and suggests feature-based matching is the more accurate family for screenshot correlation.","feed_headline":"ORB beats fuzzy hashes on phishing screenshots","feed_subtitle":"Open-source benchmark framework shows keypoint matching finds about 10% more true matches than perceptual hashes.","key_machinery":"The load-bearing mechanism is the framework's ground-truth graph and its intersection-ratio scoring. Each screenshot is a node; an edge between two nodes means these pictures should match. An algorithm's output graph is compared with the ground-truth graph by the ratio of the number of output edges that also appear in the ground truth to the total number of output edges, normalized by the maximum score any forced-guess algorithm can reach. The framework auto-generates configuration files that sweep parameter spaces, runs preprocessing including text hiding, core matching, and postprocessing, and outputs overview tables, an inclusion matrix, and per-pair quality matrices. RANSAC homography verification—checking how much a picture must be deformed to fit its match—is the postprocessing step that most improves true-positive rate but costs the most.","core_discovery":"The central claim is that an open-source, configurable harness can make image-matching benchmarks for security screenshots practical, and that on a 207-image phishing dataset the keypoint-descriptor algorithm ORB, especially with RANSAC homography filtering, reaches the highest true-positive rates—about 10 percentage points above the best fuzzy perceptual hashes. Text is the main failure mode for ORB: corner detectors fire on letters, and OCR-based text hiding only partially recovers the lost performance. Fuzzy hashes tolerate text but plateau at lower accuracy. The paper also finds that ORB is orders of magnitude slower than hashes, that a bag-of-words compression of ORB descriptors restores near-constant-time comparison without losing accuracy, and that combining algorithms is complicated by incomparable score ranges.","pith_inferences":["A natural next benchmark is to add a learned deep-learning feature matcher to the same harness; the framework's plug-in design makes this a direct test of whether ORB's lead survives the newest descriptor families.","An oracle-text-removal experiment could isolate how much of ORB's remaining error comes from OCR imperfection versus from keypoint detectors firing on non-text visual structure such as logos.","If the roughly 10% gap replicates on larger, noisier screenshot sets, security tools that currently index screenshots with hashes for near-duplicate detection could afford to switch to feature-based matching when compute budgets allow.","The framework could be extended to output a calibrated per-pair confidence score usable for thresholding, which the paper identifies as needed for algorithm combination."],"forward_implications":["Security teams can reuse the open-source framework to benchmark any matching library on their own screenshot corpora without building a harness from scratch.","ORB with RANSAC homography filtering is the best tested option for accuracy in phishing screenshot correlation, while fuzzy hashes remain the best speed-first option.","Because the inclusion matrix shows hash and ORB outputs are largely disjoint, a score-combining ensemble could beat either family; the paper notes this combination is still an open problem.","The reported failure of the funnel approach means cheap pre-filtering with hashes cannot safely discard candidates before expensive ORB matching, because the two families catch different true matches.","Bag-of-words compression of ORB descriptors offers a path to near-constant-time feature matching, closing the speed gap without the reported accuracy loss."],"supporting_citations":[{"why":"Supply the companion manual labeling tool used to build the ground-truth graph, without which no true-positive score can be computed.","marker":"[9]"},{"why":"Defines the locality-sensitive hash baseline that the framework evaluates alongside perceptual hashes.","marker":"[10]"},{"why":"Defines the ORB descriptor and matching method whose tuned configurations achieved the best true-positive rates.","marker":"[14]"},{"why":"The comparative performance study the paper cites as the reason for focusing on ORB among keypoint detectors.","marker":"[15]"},{"why":"Provides the clique-detection procedure used to identify clusters of similar nodes in the ground-truth graph.","marker":"[18]"}],"fun_headline_variants":["ORB beats fuzzy hashes by 10 points on phishing shots","Open-source matcher benchmark crowns ORB on phishing images","Text hurts ORB, but it still tops hash algorithms on phishing","Phishing screenshot match: ORB keypoints outperform hashes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking of algorithms depends entirely on the manually built ground-truth graph being correct, yet the paper provides no explicit criteria for what counts as a match and no independent validation of the labels.","fun_headline_variants_meta":{"raw":{"variants":["ORB beats fuzzy hashes by 10 points on phishing shots","Open-source matcher benchmark crowns ORB on phishing images","Text hurts ORB, but it still tops hash algorithms on phishing","Phishing screenshot match: ORB keypoints outperform hashes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000553,"raw_usage":{"total_tokens":2539,"prompt_tokens":751,"completion_tokens":1788,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":367,"completion_tokens_details":{"reasoning_tokens":1716}},"tokens_in":367,"tokens_out":1788,"duration_ms":16167,"temperature":1.0,"reasoning_tokens":1716,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:12:04.290048+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the framework on the same phishing dataset but with a ground-truth graph built independently by two or more annotators who label matches without seeing each other's work, then compare ORB's normalized true-positive rate to D-Hash's; if the roughly 10 percentage-point gap shrinks or disappears, the headline comparison is an artifact of the original labels.","supporting_citations":[{"cited_title":"Classificator for pictures matching and clustering. Fast and visual.: Vincent- CIRCL/visjs_classificator","cited_arxiv_id":null,"evidence_quote":"Supply the companion manual labeling tool used to build the ground-truth graph, without which no true-positive score can be computed."},{"cited_title":"A Procedure for Clique Detection Using the Group Matrix,","cited_arxiv_id":null,"evidence_quote":"Provides the clique-detection procedure used to identify clusters of similar nodes in the ground-truth graph."}],"review_version":1}