{"id":"a6de02f0-b4a8-4819-a75e-4991a75147d3","arxiv_id":"1908.02941","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A collaborative graph-based labeling tool for open-set image classification, shown in a small benchmark to be slower but rated more consistent than a standard tool.","lead":"VisJSClassificator is an open-source tool that lets people label images by arranging them into clusters on a shared, collaborative graph, without needing a pre-defined list of labels. It targets security analysts facing large scraped image sets, but its own small tests show it is slower than a standard labeling tool while users rated its labeling quality higher.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant objection identified; the main claim is a modest software contribution, not a statistically validated performance claim.","rationale":"The reader's weakest assumption identifies the claim that continuously seeing cluster contents improves labeling consistency, based on Section 3.3.2 and Table 3. I agree that this evaluative claim is unsupported: the benchmarks use three participants in the first test and one in the second, conditions vary across participants, and the authors themselves disclaim statistical representativeness. However, I do not consider this assumption load-bearing for the paper's stated central contribution, which is the introduction of the tool itself. The conclusion claims no superiority in speed or consistency; it claims a new open-source tool with certain features. The main claim is therefore best tested by running the artifact, not by reanalyzing the small user study. This makes the reader's concern a real limitation but not the central weak point. Since the repository is linked and the paper's descriptions are coherent, I find no load-bearing objection and recommend keeping the CONDITIONAL verdict unchanged, with the condition being reproducible artifact verification.","tokens_in":4571,"tokens_out":7461,"duration_ms":83645,"concrete_test":"Clone https://github.com/Vincent-CIRCL/visjs_classificator, run the NodeJS server, load a folder of test images in two browser clients, and perform the full workflow: select images with a right-drag, press 'C' to create an anchor, rename the cluster, export and re-import the JSON graph, and confirm that the second client sees the new cluster in real time. If the repository is missing, not installable, or lacks the claimed collaborative state sharing, the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim, as stated in the Conclusion, is that VisJSClassificator is a new open-source tool for manual, collaborative, graph-based classification requiring no predefined label set. For this claim to hold, the linked repository must exist and implement the described workflow. The paper supplies an architecture sketch (Section 2.1.1), interaction description (Section 2.1.2), JSON export format (Section 2.1.4), and screenshots, and these are internally consistent. The evaluative material in Section 3 is explicitly non-representative: the paper itself says 'These results are not statistically representative'. Thus the consistency and speed observations are limitations of the evaluation, not load-bearing support for the central claim. The one condition not yet confirmed is artifact verification: the repository was not executed or inspected in this review. That is a routine but real check for a software-contribution paper, not an identified flaw in the argument.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents VisJSClassificator, an open-source, graph-based, collaborative manual image classification tool designed for open-set classification problems where the set of labels is not known in advance. The tool is built on VisJS with a NodeJS and Socket.io backend for real-time collaboration, supports clustering pictures by creating 'anchor' nodes, renaming clusters, and exporting/importing a JSON graph structure. The authors evaluate the tool with two small experiments comparing it against DataTurks, one with 85 images and three participants and one with 475 images and a single participant, plus a participant feedback questionnaire. The paper concludes that its main contribution is the introduction of the tool, which is available at github.com/Vincent-CIRCL/visjs_classificator.","tokens_in":4695,"tokens_out":2628,"duration_ms":31074,"significance":"If the tool is functional and maintained, it addresses a genuine niche: analysts who need to label datasets where both the labels and their number are unknown, while also needing collaborative work and continuous visualization of cluster contents. The manuscript's strengths include its clear architecture description (Section 2.1.1), the documented JSON data format (Listing 1), and an unusually honest treatment of the evaluation's limitations, with the authors explicitly stating that the results are not statistically representative. The open-source availability is explicitly named and is central to the contribution. The empirical comparison is underpowered and confounded, but the paper's central claim is the existence and design of the tool rather than a statistically validated performance superiority, so the evaluation weaknesses do not undermine the main contribution if properly framed as anecdotal.","major_comments":[{"comment":"The statement 'We have to highlight that the relative quality of clustering/labelling was lower with dataturks than with visjs' is presented as a finding, but it is based on a single participant and uncontrolled conditions. The paper itself acknowledges in Section 3 that the results are not statistically representative, so this claim should be reframed as an exploratory observation, not as a highlighted result. Please remove the 'We have to highlight' phrasing and explicitly state that this is one participant's experience.","section":"Section 3.3.2 and Table 3"},{"comment":"The 85-picture comparison is confounded: participant 1 was given labels and already knew the dataset, while participants 2 and 3 had to define labels without prior dataset knowledge. Reporting the times side by side in a single table without per-row condition annotations makes the rows look directly comparable, which they are not. Add explicit footnotes or a separate design column so that the reader is not misled about the basis for the comparison.","section":"Section 3.3.1 and Table 1"}],"minor_comments":[{"comment":"The phrase 'seemed limited' should read 'seem limited' to match the present tense of the rest of the abstract.","section":"Abstract"},{"comment":"The sentence 'In datasets become larger' is missing a word; it should be 'As datasets become larger'.","section":"Section 1"},{"comment":"The caption of Figure 4a says 'On the right-hand side is the picture to be labelled, on the right-hand side are labels to choose from'; one of these should presumably be 'left-hand side'.","section":"Section 3.2"},{"comment":"The text refers to 'Results are presented in Figure 3', but the feedback results are presented in Table 3; the cross-reference should be corrected.","section":"Section 3.3.3"},{"comment":"The JSON listing is hard to read in the preprint because many spaces and line breaks appear to have been collapsed; please format the listing with proper indentation.","section":"Listing 1"},{"comment":"The bullet 'Improvement in speed is clear' is ambiguous: the table shows VisJS is slower than DataTurks, so the improvement is not in tool speed but in the participant's speed over the course of a single dataset. Please reword to avoid confusion.","section":"Section 3.3.1"}],"recommendation":"minor_revision","confidential_remarks":"The manuscript is a modest software-presentation paper; the evaluation section is weak but the authors are explicit about its limitations. The central claim is the existence and availability of the tool, which I did not execute or inspect during review. If the repository is functional, the paper is acceptable for publication after the evaluation claims are properly downgraded to anecdotal status. I would recommend that the editor verify the GitHub link and the artifact's basic functionality as part of the acceptance process."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a modest tool paper, honest about its limits, and worth a look if you care about open-set labeling workflows. The central contribution is the open-source tool itself, not the evaluation.\n\nWhat's new: VisJSClassificator applies a graph visualization library to manual collaborative labeling, letting analysts see every picture already in each cluster at all times. The design is concretely described: anchor nodes for clusters, Socket.io for state sharing, client-side-only node positions, and a JSON export/import format. That combination isn't in the cited literature, so the novelty claim is reasonable as a new application. The paper also does something many tool papers don't: it explicitly says the results are not statistically representative and shows the raw numbers instead of hiding them.\n\nThe soft spots are all in the evaluation section. Three participants for the 85-image test, with inconsistent conditions (participant 1 got predefined labels in both conditions); only one person for the 470-image test; no statistical tests; user feedback on a tiny sample with high volatility. The claim that the graph view improves labeling consistency is plausible but supported only by anecdote. I would have liked to see at least a description of how clusters were compared, or a second rater, or any attempt to measure label consistency quantitatively. The related work is also thin—no discussion of other open-set labeling tools, and the Open Set Classification citation is literally a placeholder '[?]'.\n\nNone of this sinks the central claim. The tool is what it claims to be, and the empirical results are presented as exploratory, not confirmatory. The stress-tester's point about artifact verification is real: someone should actually clone the repo and check that the workflow works before publishing a strong claim, but that's a reviewer task, not a flaw in the paper's logic.\n\nWho is this for? People building labeling tooling for security analysts, or anyone who wants a data point on graph-based labeling UI. It's not going to change your research agenda, but it's a legitimate software contribution. I'd send it to peer review—probably a workshop or a software/demo track—and ask the authors to add a proper artifact appendix and a modest evaluation with matched conditions.","headline":"A small, honest tool paper: the graph-based labeling tool is real and open-source, the evaluation is explicitly anecdotal, and the central claim holds up enough for serious peer review.","tokens_in":5188,"tokens_out":2419,"would_cite":false,"duration_ms":25709,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces a manual, collaborative, graph-based image-labeling tool for open-set classification, where labels and their number are discovered during labeling rather than fixed in advance.","keywords":["Open Set Classification","image labeling","graph-based classification","collaborative labeling","ground truth","clustering","visual detection","phishing screenshots"],"falsifier":"Run a controlled study where at least twenty annotators label the same 475-image dataset with both interfaces, measuring average pairwise label agreement and time per picture. If the graph interface does not produce measurably higher agreement per hour of effort, the paper's central quality claim fails.","tokens_in":4365,"feed_emoji":"🏷️","tokens_out":5661,"duration_ms":59749,"temperature":0.7,"pith_summary":"Image-classification systems need labeled ground truth, but standard labeling tools assume the label list is already known. This paper introduces a manual, collaborative, graph-based labeling tool for the open-set case, where classes are unknown until encountered. In the tool each picture is a node and every cluster is an \"anchor\" node, so the content of each cluster stays visible on screen throughout labeling. On a phishing-screenshot dataset, users of the graph tool were slower than with a standard labeling interface but reported higher labeling-quality scores and appeared to label more consistently. The claim is that showing every picture already assigned to a label sustains consistency without requiring a first full pass to invent labels.","feed_headline":"Open-source graph tool labels images with no preset classes","feed_subtitle":"A collaborative graph interface keeps every cluster visible, trading speed for labeling consistency.","key_machinery":"The load-bearing object is a labeled graph whose nodes are the images plus one \"anchor\" node per cluster; an edge from an image node to an anchor means the image belongs to that cluster, and the anchor's text label names the cluster. Making this graph the live interface is what lets a user compare an unlabeled picture against every picture already placed, so label consistency does not depend on memory. A real-time server shares the graph structure among collaborators, while picture positions stay client-side for performance. The exported graph—nodes, anchors, edges, and labels—doubles as the ground-truth file for training or evaluating automatic classifiers.","core_discovery":"The central discovery is that a graph representation can serve as both the labeling interface and the ground-truth format for open-set image classification. A user drags pictures, selects a set, presses one key to create a cluster anchor, and names the anchor; because the graph is always visible, the user can see exactly what is already inside every cluster before assigning a new picture. The tool shares graph state across collaborators in real time, exports the resulting clusters and labels as structured data, and can load graphs up to thousands of pictures at usable frame rates. In the reported comparison, the graph-based tool took longer per picture than a conventional list-and-label tool, but the clusters it produced were judged more consistent and its labeling quality was rated 4.5 against 3 on a 5-point scale.","pith_inferences":["The privacy of node positions suggests the shared state could be summarized and merged across sessions, so parallel labeling of disjoint image subsets may be combined without sending every layout detail.","The reported \"aversion to label creation\" points to a concrete usability cost: if cluster naming is the bottleneck, adding cheap split, merge, and rename operations is the natural next experiment to test.","A larger multi-annotator study comparing inter-rater agreement, not just speed, would turn the consistency hypothesis into a measurable property; the paper's current participant counts are too small for that.","The same graph-based labeling loop could apply to any collection of visual or textual items where an item can be shown as a thumbnail, not only to screenshots."],"forward_implications":["Datasets with unknown content can be labeled in one pass: no preliminary scanning of the whole dataset to fix a label list, and no second pass to relabel.","Collaborators always see the same cluster structure in real time, so teams can split labeling work without losing a shared notion of each label.","The exported graph provides a ground-truth representation, with clusters and labels, ready to feed automatic classification and image-matching evaluation.","Because new labels can be created mid-task and are visible immediately, the tool is suited to open-set data such as scraped screenshots where the class inventory is not known ahead."],"supporting_citations":[{"why":"Provides the motivating data-generation context: a daily scraper of thousands of screenshots that require classification.","marker":"[1]"},{"why":"Makes the tool itself available as open-source software, which is the paper's main contribution.","marker":"[2]"}],"fun_headline_variants":["Graph tool makes open-set image labeling collaborative","Visual graph labels images without preset classes","Collaborative graph labeling for open-set classification","Graph-based labeling keeps clusters visible for consistent tags","Manual graph labeling tool for flexible image classification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach assumes that continuously seeing every picture already placed in a cluster improves labeling consistency enough to justify the slower interaction, an effect supported only by a handful of participants in the reported tests.","fun_headline_variants_meta":{"raw":{"variants":["Graph tool makes open-set image labeling collaborative","Visual graph labels images without preset classes","Collaborative graph labeling for open-set classification","Graph-based labeling keeps clusters visible for consistent tags","Manual graph labeling tool for flexible image classification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000134,"raw_usage":{"total_tokens":1032,"prompt_tokens":733,"completion_tokens":299,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":349,"completion_tokens_details":{"reasoning_tokens":233}},"tokens_in":349,"tokens_out":299,"duration_ms":3591,"temperature":1.0,"reasoning_tokens":233,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:28:21.879449+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled study where at least twenty annotators label the same 475-image dataset with both interfaces, measuring average pairwise label agreement and time per picture. If the graph interface does not produce measurably higher agreement per hour of effort, the paper's central quality claim fails.","supporting_citations":[{"cited_title":"AIL - The design and implementation of an Analysis Information Leak framework,","cited_arxiv_id":null,"evidence_quote":"Provides the motivating data-generation context: a daily scraper of thousands of screenshots that require classification."},{"cited_title":"Classificator for pictures matching and clustering. Fast and visual.: Vincent- CIRCL/visjs_classificator","cited_arxiv_id":null,"evidence_quote":"Makes the tool itself available as open-source software, which is the paper's main contribution."}],"review_version":1}