Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Open Dataset of Phishing and Tor Hidden Services Screen-captures

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper's main contribution is not an algorithm but the release of two human-verified, human-classified datasets of phishing and Tor hidden-service screenshots, each image keyed by cryptographic hashes and supplied with labels for…

desk verdict The datasets are a genuine and useful public resource, but the abstract overstates the completeness and reliability of the AIL labels. read the letter →

arxiv 1908.02449 v1 pith:G4BHVVN3 submitted 2019-08-07 cs.CR

Vincent Falconieri This is my paper · ORCID
classification cs.CR
keywords phishingTorhiddenservicesoniondomainsscreenshotdatasetsvisualimagematchingclassificationopendatathreatintelligence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that it provides ready-made, human-verified datasets for visual correlation and automatic classification of phishing and Tor hidden-service screenshots. One set holds roughly 460 screenshots of phishing pages; the other holds roughly 37,500 screenshots of onion domains scraped from the Tor network. Each image has a stable hash triplet and a human-readable name, and the phishing set comes with two independent label schemes plus a graph of clusters. The author argues that these labeled images are the missing raw material for training automatic tools that would let security analysts correlate screenshots instead of doing it by hand.

What carries the argument

The load-bearing object is the dataset archive itself: screenshots named by hashing each file's bytes into a human-readable adjective-triple, shipped with a JSON reference mapping names to MD5, SHA1, and SHA256, plus label files that map filenames to classes. That naming scheme makes picture mentions stable across tools, while the label files provide the ground truth an automatic classifier would train on; the hash reference makes the corpus auditable and de-duplicable.

What would settle it

Re-annotate a random sample of a few hundred images from each dataset with independent labelers and compute agreement (for example, Cohen's kappa) against the published labels; if agreement is low on common classes such as 'other' and 'error page', the datasets' usefulness as ground truth collapses.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is the availability, not a classification algorithm: two free and open datasets that have been reviewed picture by picture, stripped of personal or harmful content, labelled by hand, and shipped with per-file MD5, SHA1, and SHA256 hashes. The phishing dataset (about 460 images) is labelled twice, once with a collaborative tagging tool and once with a graph-based classifier that also yields cluster membership; the dark-web dataset (about 37,500 images) carries partial labels drawn from a public dark-web taxonomy. The paper demonstrates on a small sample that fuzzy-hash matching against these screenshots can group pages by visual structure or theme, but explicitly frames that as a glimpse, not the contribution.

Load-bearing premise

The load-bearing premise is that the hand-made labels, made by one reviewer without documented agreement checks, are accurate enough to serve as training ground truth; if the labels are noisy or inconsistent, classifiers trained on them inherit that noise.

Editorial extensions

If this is right

  • Analysts can download both archives and use the label files as training ground truth for screenshot classifiers without collecting or labeling their own corpus.
  • The phishing set's brand and login-form labels support clustering that can track a phishing brand across domain changes.
  • The dark-web taxonomy labels allow automatic triage of onion sites by topic, motivation, and structure, including scam, marketplace, and login-form categories.
  • The per-file hashes let a future tool de-duplicate or verify screenshots and let different teams compare results on identical images.
  • Even partial automation of screenshot classification would reduce the manual correlation burden on security teams.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension: because each screenshot ships with a hash triplet, independent researchers could relabel a random subset and quantify agreement, giving the corpus an inter-annotator reliability measure the paper leaves out.
  • Extension: the partially labeled dark-web set could be exploited with label propagation from the labeled images, letting researchers train coarse topic classifiers without full manual annotation.
  • Extension: the same hash-readable naming and labeling conventions transfer naturally to screenshots from malware sandboxes or email attachments, broadening visual correlation beyond phishing and onion pages.
  • Extension: if the label noise is low, simple visual features alone should separate broad classes such as login forms, marketplaces, and error pages; this is a direct test readers can run against the published labels.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper announces two open datasets of website screenshots for security research: circl-phishing-dataset-01 (about 460 phishing screenshots) and circl-ail-dataset-01 (about 37,500 Tor hidden-service screenshots). It describes the data sources (MISP, URLAbuse, AIL), the file formats (JSON hash mappings, DataTurks label exports, VisJS cluster and graph exports), the naming convention based on content hashes, and the manual labeling process using DataTurks and VisJS-Classiificator. A short usage example illustrates fuzzy-hash based image matching on the phishing dataset, and future work mentions extending the datasets and improving the AIL classification. The authors state explicitly that the main contribution is the availability of the two human-verified and human-classified datasets, not the research results shown in the usage examples.

Significance. If the two datasets are actually downloadable, correctly described by the provided hash files, and accompanied by usable label files, the paper serves a useful community function: it addresses a genuine gap, since few open datasets of website screenshots exist specifically for phishing and Tor hidden services. The hash-to-filename reference JSON is a concrete reproducibility aid, and the MISP dark-web taxonomy-based labels give the AIL dataset a standard vocabulary. The paper is also honest in scope: it does not claim algorithmic novelty and explicitly labels the matching experiments as preliminary. The central weakness is that the papers stated contribution is the availability of human-classified data, yet the completeness and quality of that classification are never measured, especially for the AIL dataset.

major comments (3)
  1. [Section 2.2, Figures 2b and 3] The central claim is the availability of two human-classified datasets, but for circl-ail-dataset-01 the paper reports only that the classification 'is partial to date' and provides no fraction of labeled versus unlabeled images. The label-frequency figures are based on 800 and 9,500 sampled pictures out of roughly 37,500, which suggests the label file may cover only a minority of the images. The authors should state explicitly how many of the 37,500 pictures have at least one label, how many are unlabeled, and whether the label file contains entries for all pictures or only for the labeled subset.
  2. [Section 2.3] The manual classification is described as done by 'we' with no indication of the number of annotators, no inter-annotator agreement, and no quality metric. Since the paper claims human classification as a contribution, at minimum the authors should report the number of reviewers, the level of consistency (e.g., a random re-labeling sample or a second pass), and the criteria used to resolve ambiguous cases. Without such information, the reliability of the labels as ground truth for training classifiers is unquantified.
  3. [Section 4 and Figures 5a/5b] The usage example is presented as evidence that the dataset supports image-matching research, but it contains no quantitative evaluation: there are no numbers of true/false matches, no comparison of algorithms, and no measure of retrieval accuracy. This would be acceptable if the section were explicitly framed as an illustration only, but the current text claims that structures 'can be easily detected' and content 'can be matched with some confidence.' Please either add quantitative results (e.g., precision/recall on a labeled subset) or rephrase the claims as purely anecdotal.
minor comments (5)
  1. [Abstract and Section 1.1] The phrase 'human-verified and human-classified' is used as the main characterization of both datasets, but the AIL dataset is only partially classified. Please qualify the claim in the abstract, e.g., 'human-verified, and human-classified for the phishing dataset and partially human-classified for the AIL dataset.'
  2. [Section 2.2] The collision-handling description is unclear: it says bytes are 'temporary adding bytes to each colliding file,' which suggests modifying the file content to resolve a hash-name collision. If the file bytes are changed, the hash-to-file mapping would change as well. Please clarify whether this operation modifies the picture bytes or only the filename input to the codename generator.
  3. [Section 3.2.1 and Annex 7.1.5] The two added labels are written differently in the text ('error page' and 'other') and in the annex ('error_page' and 'other'). Please unify the notation so users can parse the label file unambiguously.
  4. [References] The citation for the MISP dark-web taxonomy is missing a key or full reference; the text shows 'taxonomy9[?]'. Please add the complete reference for the MISP taxonomies repository.
  5. [Throughout] Several minor typos and formatting issues occur, including 'screen-captures' inconsistency in the title, '10000' without thousands separator, 'classification' with unusual ligatures, and the mention of 'Douglas-Quaid' and 'Carl-Hauser' as footnotes rather than as proper references. These do not affect the technical content.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's contribution is the availability of two externally hosted datasets, not a derived prediction or fitted result.

full rationale

The paper's central claim is explicitly the availability of two human-verified, human-classified datasets, and it states that the usage examples are not the main contribution. No derivation chain exists: the labels were produced by manual review, not by fitting a parameter to a subset and predicting a related quantity. Self-citations to CIRCL tools (AIL, MISP, Carl-Hauser, Douglas-Quaid) are contextual provenance statements, not load-bearing evidence for the datasets' existence; even if those tool citations were unsupported, the dataset links, hashes, and label files would remain independently checkable. The acknowledged partial classification of the AIL dataset in Section 2.2 is a candid data-quality limitation, not circularity, because the paper never claims that the labels are derived from anything other than human review. Concerns about label completeness and inter-annotator reliability are correctness or quality risks, not circular-reasoning defects. Accordingly, none of the enumerated circularity patterns is present, and the honest finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no fitted parameters or invented entities. It relies on domain assumptions about the quality and representativeness of manually assigned labels and about the fidelity of screenshots produced by CIRCL tools.

assumptions (3)
  • domain assumption Manual review of screenshots produces accurate and consistent ground-truth labels.
    Invoked in Section 2.3 'Classification Methodology'; no inter-annotator agreement or validation is provided.
  • domain assumption Screenshots captured by MISP, URLAbuse, and AIL faithfully represent the target websites.
    The datasets are collected via these tools (Section 2.1); rendering artifacts or capture errors are not discussed.
  • domain assumption The dataset samples are representative enough to train useful classifiers.
    The AIL dataset is a scrape sample and the phishing dataset is a subset of reported incidents, so representativeness is assumed rather than demonstrated (Sections 2.1 and 2.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Open Dataset of Phishing and Tor Hidden Services Screen-captures." pith.science (2026). https://pith.science/paper/G4BHVVN3

@misc{pith2026190802449,
  author       = {Pith},
  title        = {Pith review of: Open Dataset of Phishing and Tor Hidden Services Screen-captures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G4BHVVN3}},
  note         = {Machine review of arXiv:1908.02449}
}
read the original abstract

Security analysts need to classify, search and correlate numerous images. Automatic classification tools improve the efficiency of such tasks. However, the main resources to develop these tools are datasets, which are introduced and provided by the present paper, for the specific cases of visual correlation of phishing and onion websites. CIRCL's Open-Source tools are the sources of these screenshots, which had been manually verified against personal information leaks. Usage examples of these datasets are proposed in the current paper. These researches directions are, however, not the main contribution of the paper. The main contribution is the availability of the two datasets.

Figures

Figures reproduced from arXiv: 1908.02449 by the authors.

Figure 1
Figure 1. Dataset’s samples (a) Phishing dataset label frequency (b) AIL dataset label frequency (on 800 sampled pictures) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Label frequency per dataset 3 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Label frequency for AIL dataset 4 [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Few examples of clusters found with VisJS Classificator interface. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Usage examples of Phishing dataset to train an Image-Matching tool. Most left-hand picture is requesting [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Douglas-Quaid -- Open Source Image Matching Library

    cs.CR 2019-08 conditional novelty 5.0 of 10

    Douglas-Quaid is an open-source image matching library that combines fuzzy hashes and ORB with decision fusion and self-calibration, reporting 80% accuracy on a CERT screenshot dataset.

Reference graph

Works this paper leans on

5 extracted references · cited by 1 Pith paper

  1. [2]

    CNN for task classification using computer screenshots for integration into dynamic calendar/task management systems,

    A. Sampat and A. Haskell, “CNN for task classification using computer screenshots for integration into dynamic calendar/task management systems,” p. 6

  2. [4]

    Fighting Phishing with Discriminative Keypoint Features,

    K. Chen, J. Chen, C. Huang, and C. Chen, “Fighting Phishing with Discriminative Keypoint Features,” vol. 13, no. 3, pp. 56–63

  3. [1]

    AIL - The design and implementation of an Analysis Information Leak framework,

    S. Mokaddem, G. Wagener, and A. Dulaunoy, “AIL - The design and implementation of an Analysis Information Leak framework,” in2018 IEEE International Conference on Big Data (Big Data) , pp. 5049–5057

  4. [3]

    Predicting Phishing Websites Using Classification Mining Techniques with Experimental Case Studies,

    M. Aburrous, M. A. Hossain, K. Dahal, and F. Thabtah, “Predicting Phishing Websites Using Classification Mining Techniques with Experimental Case Studies,” in2010 Seventh International Conference on Information Technology: New Generations, pp. 176–181

  5. [5]

    MISP: The Design and Implementation of a Collaborative Threat Intelligence Sharing Platform,

    C. Wagner, A. Dulaunoy, G. Wagener, and A. Iklody, “MISP: The Design and Implementation of a Collaborative Threat Intelligence Sharing Platform,” in Proceedings of the 2016 ACM on Workshop on Information Sharing and Collaborative Security - WISCS’16 . ACM Press, pp. 49–56. [Online]. Available: http://dl.acm.org/citation.cfm?doid=2994539.2994542 8 Open Dat...

    arXiv 2016

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.