Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Carl-Hauser -- Open Source Image Matching Algorithms Benchmarking Framework

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ORB beats fuzzy hashes on phishing screenshots

desk verdict A real, useful open-source benchmarking framework and dataset for phishing-screenshot matching, but the paper's headline ORB-beats-hash claim is selected on the test set and not supported by the reported experiment. read the letter →

arxiv 1908.03449 v1 pith:T52EA6XV submitted 2019-08-09 cs.CR

Vincent Falconieri This is my paper · ORCID
classification cs.CR
keywords imagematchingbenchmarkingframeworkphishingdetectionscreenshotsimilarityORBfuzzyhashingsecurityautomationopendata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces a free and open-source benchmarking framework that evaluates image-matching algorithms on folders of screenshots and scores them against a manually built ground-truth graph. The main claim is that on phishing and onion-site screenshot datasets, ORB keypoint matching outperforms fuzzy perceptual hashes in nearly all tested configurations, with roughly 10% more true positive matches. The framework produces quality metrics, timing and memory statistics, graph visualizations, and configurable preprocessing such as OCR-based text hiding. If correct, it gives security teams a turnkey way to compare matching algorithms on their own data and suggests feature-based matching is the more accurate family for screenshot correlation.

What carries the argument

The load-bearing mechanism is the framework's ground-truth graph and its intersection-ratio scoring. Each screenshot is a node; an edge between two nodes means these pictures should match. An algorithm's output graph is compared with the ground-truth graph by the ratio of the number of output edges that also appear in the ground truth to the total number of output edges, normalized by the maximum score any forced-guess algorithm can reach. The framework auto-generates configuration files that sweep parameter spaces, runs preprocessing including text hiding, core matching, and postprocessing, and outputs overview tables, an inclusion matrix, and per-pair quality matrices. RANSAC homography verification—checking how much a picture must be deformed to fit its match—is the postprocessing step that most improves true-positive rate but costs the most.

What would settle it

Run the framework on the same phishing dataset but with a ground-truth graph built independently by two or more annotators who label matches without seeing each other's work, then compare ORB's normalized true-positive rate to D-Hash's; if the roughly 10 percentage-point gap shrinks or disappears, the headline comparison is an artifact of the original labels.

Watch

Extended reading notes

Core claim

The central claim is that an open-source, configurable harness can make image-matching benchmarks for security screenshots practical, and that on a 207-image phishing dataset the keypoint-descriptor algorithm ORB, especially with RANSAC homography filtering, reaches the highest true-positive rates—about 10 percentage points above the best fuzzy perceptual hashes. Text is the main failure mode for ORB: corner detectors fire on letters, and OCR-based text hiding only partially recovers the lost performance. Fuzzy hashes tolerate text but plateau at lower accuracy. The paper also finds that ORB is orders of magnitude slower than hashes, that a bag-of-words compression of ORB descriptors restores near-constant-time comparison without losing accuracy, and that combining algorithms is complicated by incomparable score ranges.

Load-bearing premise

The ranking of algorithms depends entirely on the manually built ground-truth graph being correct, yet the paper provides no explicit criteria for what counts as a match and no independent validation of the labels.

Editorial extensions

If this is right

  • Security teams can reuse the open-source framework to benchmark any matching library on their own screenshot corpora without building a harness from scratch.
  • ORB with RANSAC homography filtering is the best tested option for accuracy in phishing screenshot correlation, while fuzzy hashes remain the best speed-first option.
  • Because the inclusion matrix shows hash and ORB outputs are largely disjoint, a score-combining ensemble could beat either family; the paper notes this combination is still an open problem.
  • The reported failure of the funnel approach means cheap pre-filtering with hashes cannot safely discard candidates before expensive ORB matching, because the two families catch different true matches.
  • Bag-of-words compression of ORB descriptors offers a path to near-constant-time feature matching, closing the speed gap without the reported accuracy loss.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next benchmark is to add a learned deep-learning feature matcher to the same harness; the framework's plug-in design makes this a direct test of whether ORB's lead survives the newest descriptor families.
  • An oracle-text-removal experiment could isolate how much of ORB's remaining error comes from OCR imperfection versus from keypoint detectors firing on non-text visual structure such as logos.
  • If the roughly 10% gap replicates on larger, noisier screenshot sets, security tools that currently index screenshots with hashes for near-duplicate detection could afford to switch to feature-based matching when compute budgets allow.
  • The framework could be extended to output a calibrated per-pair confidence score usable for thresholding, which the paper identifies as needed for algorithm combination.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces Carl-Hauser, an open-source framework for benchmarking image-matching algorithms, and applies it to a dataset of phishing website screenshots. The framework automates the evaluation pipeline: given an input folder of images and a ground-truth graph, it runs multiple algorithms across a parameter sweep, records quality and timing metrics, and produces overview tables, intersection matrices, and graph outputs. The paper's stated main contribution is the benchmarking framework itself, and it further reports an illustrative comparison between ORB (OpenCV) feature matching and fuzzy-hash algorithms (ImageHash, TLSH), concluding in Section 5 that 'ORB outperforms fuzzy-hash algorithms in almost all sounds configuration (about 10% more true positive matches).' The dataset and code are offered as open data and open source.

Significance. If the framework operates as described, it provides a reusable, open-source evaluation harness for a practical security task (phishing screenshot matching), and the public release of the phishing screenshot dataset is a useful community resource. The paper also gives a clear scoring definition (edge intersection ratio with a ground-truth graph, normalized by the maximum attainable score) and documents practical observations about text noise and parameter sensitivity of ORB. However, the headline empirical claim about ORB versus fuzzy hashes is not established by the reported procedure, for reasons detailed below. The significance of the paper therefore rests mainly on the tool and dataset rather than on the quantitative comparison.

major comments (3)
  1. [Section 5 and Appendix 9] The claim that 'ORB outperforms fuzzy-hash algorithms ... about 10% more true positive matches' is not supported by the reported experiment. Section 3 describes the framework as exploring the parameter space and generating many configurations; Appendix 9 then presents, as the 'best' ORB results, the highest true-positive scores selected from that sweep on the same single 207-image dataset. Because the best of many noisy estimates is biased upward when scored on the same data used for selection, the comparison against fixed hash baselines is tilted toward ORB. No held-out set, cross-validation, repeated runs, or confidence intervals are provided. Moreover, the stated margin is quantitatively inaccurate: the best raw ORB true-positive rate in the PNG table is 0.65263 versus 0.60386 for D-Hash, a difference of about 4.9 percentage points, not 10; the normalized values (0.77641 vs 0.71839) give about 8.1 percent relative improvement. The empirical comparison should be re-run with proper model selection (e.g., validation/tuning separate from test), with uncertainty estimates, and the textual claim corrected to match the actual numbers.
  2. [Section 3 and Appendix 10] The evaluation's validity depends entirely on a single manually built ground-truth graph, but the paper gives no criteria for what constitutes a match, no inter-annotator agreement, and no sensitivity analysis. Section 3 states that the ground-truth file 'can easily be built with VisJS-Classificator[9]', and Appendix 10 defines the true-positive score as the intersection ratio between the algorithm's output graph and this ground-truth graph. Since the labeling tool is authored by the same team and the labeling procedure is undocumented, label noise or systematic bias would shift every algorithm's score and could change the ORB-versus-hash ordering. The authors should document the labeling instructions, report how the ground-truth graph was constructed (including the clique structure and outlier treatment), and ideally provide at least two independent annotators with a measure of agreement.
  3. [Appendix 10, Figure 13] The intersection matrix in Figure 13 is unreadable: the axis labels are an unbroken string of configuration names, and the figure itself contains the note 'CONCATENE AND MAKE IT MORE READABLE : FOR NEXT RELEASE'. As printed, the claimed observations about clusters of similar ORB configurations and the orthogonality between hash-based and ORB-based outputs cannot be verified by a reader. The matrix should be replaced with a legible figure with abbreviated but unique labels, or the underlying data should be provided in a machine-readable table.
minor comments (6)
  1. [Abstract and Section 1] There are several typos and stylistic issues: 'litterature' in the abstract, 'Is this paper' should be 'In this paper', and 'bench-marking' is an awkward hyphenation in Section 1.1.
  2. [Section 4.2.2] The text contains an unresolved reference: 'Figure 11 and ?? are examples.' This should be fixed to a proper figure citation or rewritten.
  3. [Section 5] The word 'preformance' appears in the scalability bullet; it should be 'performance'.
  4. [Section 2 and Appendix 9] The dataset size is described as '470+ pictures' in Section 2 but Appendix 9 states the evaluation was conducted on 207 screenshots. This inconsistency should be clarified (e.g., the full dataset versus the evaluation subset).
  5. [References] Several references are incomplete or inconsistently formatted (e.g., entries with missing volume/pages such as [1], and some with only 'pp. 6'). The bibliography should be cleaned up.
  6. [Appendix 10] Equation (1) defines Iratio but the notation '♯' for set cardinality is not introduced; a brief explanation would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; the paper is a benchmarking/empirical evaluation, and no claimed result reduces to its inputs by construction.

full rationale

The paper's contribution is an open-source benchmarking framework and an illustrative evaluation. Its headline empirical claim, 'ORB outperforms fuzzy-hash algorithms in almost all sounds configuration (about 10% more true positive matches),' is a direct report of the quality metric defined in Appendix 10, Eq. (1), as the intersection ratio between an algorithm output graph and the manually built ground-truth graph. No parameter is fitted to a subset of data and then renamed as a prediction; the framework's parameter-space exploration (Section 3) generates configurations whose scores are tabulated in Appendix 9, and the paper does not claim a held-out forecast from those scores. The ground-truth file is described as needing to be provided and as buildable with the author's VisJS-Classificator tool, but that is data annotation, not a derived quantity; the correctness of the labels is an assumption about benchmark quality, not a circular reduction. Citations to prior work (TLSH, ORB, SIFT/SURF, and the benchmark of Tareen and Saleem) are standard external references, and the only self-citation, [9] for VisJS-Classificator, is not invoked as a proof, theorem, or uniqueness argument. Statistical concerns such as selecting the best ORB configuration on one dataset or relying on unvalidated manual labels are validity risks, not circularity, under the requirement that a circular step must reduce to its own inputs by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central comparison rests on a manually constructed ground truth, a defined quality metric, and a parameter sweep; none of these are externally imposed benchmarks.

free parameters (4)
  • ORB feature count = 500
    All ORB configurations use 500 features; this is an arbitrary choice likely tuned for the dataset.
  • RANSAC distance threshold = 0.96 to 0.97
    In Section 4.2.2 and Figure 11, matches above 0.96/0.97 are called the 'critical threshold'; set by inspection of this dataset.
  • ORB matching filter type
    The parameter sweep includes NO_FILTER, RATIO_CORRECT, RATIO_BAD, and FAR_THRESHOLD; selecting the best on the test set is a free choice.
  • Ground truth match criterion
    The definition of 'similar' in the ground truth graph is subjective and not formalized; it is a hidden parameter affecting all scores.
assumptions (4)
  • domain assumption Ground truth graph correctly represents visual similarity among dataset images.
    All quality scores are computed as intersection with this graph; no validation is provided.
  • standard math Intersection ratio (Equation 1) is a valid measure of matching quality.
    It is a definition, but its use as a quality metric assumes that edge overlap is the right target.
  • domain assumption Rank-1 matching output is the correct unit of evaluation.
    The framework evaluates each algorithm's top guess for every image; this ignores other ranked outputs.
  • domain assumption The phishing dataset is representative of the security screening task.
    Results on 207-470 screenshots are presented as general conclusions about algorithm performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Carl-Hauser -- Open Source Image Matching Algorithms Benchmarking Framework." pith.science (2026). https://pith.science/paper/T52EA6XV

@misc{pith2026190803449,
  author       = {Pith},
  title        = {Pith review of: Carl-Hauser -- Open Source Image Matching Algorithms Benchmarking Framework},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T52EA6XV}},
  note         = {Machine review of arXiv:1908.03449}
}
read the original abstract

Security analysts need to classify, search and correlate numerous images. Automatic classification tools improve the efficiency of such tasks. Many Image-Matching algorithms are presented in the litterature. The present paper introduces and provides a Open-Source benchmarking and evaluation tool for these algorithms. Is this paper, the framework evaluates algorithms on illustrative datasets, which are constituted of phishing and onion websites. Datasets are provided as Open-Data.

Figures

Figures reproduced from arXiv: 1908.03449 by the authors.

Figure 1
Figure 1. Carl-Hauser logo Image correlation for security event correlation purposes is nowadays mainly manual. No open-source tool provides easy correlations, without regard to the technology used. Ideally, the extraction of links or correlation between these images could be fully automated. Even partial automation would reduce the burden of this task on security teams. The main contribution of this paper is a free and open-… view at source ↗
Figure 2
Figure 2. Dataset’s samples 3 Materials and Methods In the quest of an use-case-specific Image-matching library, algorithms and approaches are numerous. Benchmarking each one of them is tough and time-consuming. Therefore, we developed a benchmarking framework. The envisioned goal is to allow a fast implementation and fast evaluation of any new Image-matching library or algorithm, that could come up at a later date. 2 [PITH_… view at source ↗
Figure 3
Figure 3. Global overview of benchmarking framework [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Available pre-computation example - Text hider [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]
Figure 5
Figure 5. Figure 5: Request picture (on the left) and top matches (on the right, top 1 to top 3 from left to right). [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Graph visualization and successes 6 [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Graph visualization and issues 7 [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: ORB and RANSAC algorithm A few other matches are displayed in [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Matrix transformation visualisation - ORB - RANSAC Filtering - Visualisation of transformation matrix [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: The first column is the candidate picture, second column is the request picture, third column is the transformed [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: Results - ORB - RANSAC Filtering - No matrix filter [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: Global overview of test framework 16 [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Intersection matrix - Guess of rank 1 for each image comparison [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Douglas-Quaid -- Open Source Image Matching Library

    cs.CR 2019-08 conditional novelty 5.0 of 10

    Douglas-Quaid is an open-source image matching library that combines fuzzy hashes and ORB with decision fusion and self-calibration, reporting 80% accuracy on a CERT screenshot dataset.

Reference graph

Works this paper leans on

18 extracted references · 5 canonical work pages · cited by 1 Pith paper

  1. [9]

    Classificator for pictures matching and clustering. Fast and visual.: Vincent- CIRCL/visjs_classificator

    Vincent-CIRCL, “Classificator for pictures matching and clustering. Fast and visual.: Vincent- CIRCL/visjs_classificator.” [Online]. Available: https://github.com/Vincent-CIRCL/visjs_classificator

    work page

  2. [1]

    CNN for task classification using computer screenshots for integration into dynamic calendar/task management systems,

    A. Sampat and A. Haskell, “CNN for task classification using computer screenshots for integration into dynamic calendar/task management systems,” p. 6

  3. [2]

    Predicting Phishing Websites Using Classification Mining Techniques with Experimental Case Studies,

    M. Aburrous, M. A. Hossain, K. Dahal, and F. Thabtah, “Predicting Phishing Websites Using Classification Mining Techniques with Experimental Case Studies,” in2010 Seventh International Conference on Information Technology: New Generations, pp. 176–181

    work page

  4. [3]

    Fighting Phishing with Discriminative Keypoint Features,

    K. Chen, J. Chen, C. Huang, and C. Chen, “Fighting Phishing with Discriminative Keypoint Features,” vol. 13, no. 3, pp. 56–63

  5. [4]

    AIL - The design and implementation of an Analysis Information Leak framework,

    S. Mokaddem, G. Wagener, and A. Dulaunoy, “AIL - The design and implementation of an Analysis Information Leak framework,” in2018 IEEE International Conference on Big Data (Big Data), pp. 5049–5057

  6. [5]

    MISP: The Design and Implementation of a Collaborative Threat Intelligence Sharing Platform,

    C. Wagner, A. Dulaunoy, G. Wagener, and A. Iklody, “MISP: The Design and Implementation of a Collaborative Threat Intelligence Sharing Platform,” in Proceedings of the 2016 ACM on Workshop on Information Sharing and Collaborative Security - WISCS’16 . ACM Press, pp. 49–56. [Online]. Available: http://dl.acm.org/citation.cfm?doid=2994539.2994542

    arXiv 2016

  7. [6]

    Large scale reverse image search - A method comparison for almost identical image retrieval,

    M. Gaillard and E. Egyed-Zsigmond, “Large scale reverse image search - A method comparison for almost identical image retrieval,” inINFORSID

  8. [7]

    Implementation and Benchmarking of Perceptual Image Hash Functions

    C. Zauner and M. Steinebach, “Implementation and Benchmarking of Perceptual Image Hash Functions.”

Show all 18 references
  1. [8]

    Image Matching: An Application-oriented Benchmark

    J. Bian, L. Zhang, Y . Liu, W.-Y . Lin, M.-M. Cheng, and I. D. Reid, “Image Matching: An Application-oriented Benchmark.” [Online]. Available: http://arxiv.org/abs/1709.03917

    arXiv

  2. [10]

    TLSH – A Locality Sensitive Hash,

    J. Oliver, C. Cheng, and Y . Chen, “TLSH – A Locality Sensitive Hash,” in2013 Fourth Cybercrime and Trustworthy Computing Workshop. IEEE, pp. 7–13. [Online]. Available: http://ieeexplore.ieee.org/document/6754635/

  3. [11]

    Distinctive Image Features from Scale-Invariant Keypoints,

    D. G. Lowe, “Distinctive Image Features from Scale-Invariant Keypoints,” vol. 60, no. 2, pp. 91–110. [Online]. Available: http://link.springer.com/10.1023/B:VISI.0000029664.99615.94

  4. [12]

    Anatomy of the SIFT Method,

    I. R. Otero and M. Delbracio, “Anatomy of the SIFT Method,” vol. 4, pp. 370–396. [Online]. Available: http://www.ipol.im/pub/art/2014/82/

    2014

  5. [13]

    SURF: Speeded Up Robust Features,

    H. Bay, T. Tuytelaars, and L. Van Gool, “SURF: Speeded Up Robust Features,” inComputer Vision – ECCV 2006, A. Leonardis, H. Bischof, and A. Pinz, Eds. Springer Berlin Heidelberg, vol. 3951, pp. 404–417. [Online]. Available: http://link.springer.com/10.1007/11744023_32

    2006 doi

  6. [14]

    ORB: An efficient alternative to SIFT or SURF,

    E. Rublee, V . Rabaud, K. Konolige, and G. Bradski, “ORB: An efficient alternative to SIFT or SURF,” in 2011 International Conference on Computer Vision . IEEE, pp. 2564–2571. [Online]. Available: http://ieeexplore.ieee.org/document/6126544/

    2011

  7. [15]

    A comparative analysis of SIFT, SURF, KAZE, AKAZE, ORB, and BRISK,

    S. A. K. Tareen and Z. Saleem, “A comparative analysis of SIFT, SURF, KAZE, AKAZE, ORB, and BRISK,” in 2018 International Conference on Computing, Mathematics and Engineering Technologies (iCoMET). IEEE, pp. 1–10. [Online]. Available: https://ieeexplore.ieee.org/document/8346440/

    2018

  8. [16]

    Cognitive Fit: An Empirical Study of Information Acquisition,

    I. Vessey and D. Galletta, “Cognitive Fit: An Empirical Study of Information Acquisition,” vol. 2, no. 1, pp. 63–84. [Online]. Available: http://pubsonline.informs.org/doi/abs/10.1287/isre.2.1.63

    doi

  9. [17]

    [Online]

    ORB (Oriented FAST and Rotated BRIEF) — OpenCV 3.0.0-dev documentation. [Online]. Available: https://docs.opencv.org/3.0-beta/doc/py_tutorials/py_feature2d/py_orb/py_orb.html#orb

  10. [18]

    A Procedure for Clique Detection Using the Group Matrix,

    F. Harary and I. C. Ross, “A Procedure for Clique Detection Using the Group Matrix,” vol. 20, no. 3, pp. 205–215. 14 Carl-Hauser - Open Source Image Matching Algorithms Benchmarking Framework A PREPRINT Part I Appendices 8 Detailed view of the framework The exact implementatio...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.