{"id":"ffe81a52-ba8a-4621-b159-b9d26de3bf56","arxiv_id":"2502.02759","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"ClarAVy, using sparse variational Bayesian aggregation of antivirus detections, labels malware families with 64.16% accuracy on MOTIF and 56.88% on MalPedia, outperforming prior tools by 8 to 12 percentage points.","lead":"ClarAVy is a malware labeling tool that automatically assigns family names to malicious files by combining antivirus detections with a Bayesian voting method. It reports 8 to 12 percentage points higher labeling accuracy than previous tools and handles tens of millions of files.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The t_i∈c_i restriction caps achievable accuracy whenever AV coverage omits the true family; the paper's own 18.64% MOTIF figure shows this is not a corner case, so the reported accuracy should be read as a coverage-dependent ceiling rather than a general property of the aggregator.","rationale":"The reader's weakest_assumption correctly identifies Section 5.5's t_i∈c_i restriction as the main structural limitation. The paper itself confirms the practical impact in Section 6.4: 18.64% of MOTIF files have no AV detection containing the true family, so no AV-based method can label them correctly, and 11.53% have only a single supporting detection. This means the reported accuracies are ceilings conditional on AV family coverage, not unconditional properties of SparseIBCC. I agree this is the most load-bearing concern because it directly bounds the central accuracy claim and its generalizability to other corpora. However, the paper discloses the limitation explicitly and the assumption is a natural consequence of using AV detections as the only signal, so it does not invalidate the comparison with prior AV-based tools; those tools face the same coverage ceiling. The proposed split-subset analysis would quantify how much of the remaining error is due to the assumption versus the aggregation itself. The reader's CONDITIONAL verdict remains appropriate because the accuracy claim is plausible and well-supported on the evaluated datasets, but the code is unreleased and the runtime analysis in Section 6.3 contains a clear interpretive error (a quadratic fit cannot establish sub-linear scaling). Those secondary issues, plus the disclosed coverage limitation, justify keeping the verdict conditional rather than full acceptance.","tokens_in":16332,"tokens_out":18362,"duration_ms":188932,"concrete_test":"Recompute ClarAVy's MOTIF accuracy on only the 81.36% of files for which t_i∈c_i holds, and compare it with the theoretical ceiling of 81.36% on that subset; also report accuracy on the 18.64% subset separately. If the feasible-subset accuracy is close to the ceiling and the impossible-subset accuracy is 0%, the t_i∈c_i assumption is the binding constraint. If the feasible-subset accuracy is far below the ceiling, the aggregation method itself is the main source of error and the limitation is less central.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.5 assumes t_i∈c_i, meaning the true family must appear among the family tokens in the file's AV scan report; the posterior is therefore restricted to families present in that report. Section 6.4 discloses that 18.64% of MOTIF scans are impossible to label because no AV detection contains the true family, and an additional 11.53% have the true family in only a single detection. Thus on roughly 30% of MOTIF the method is either wrong by construction or dependent on one weak signal. The reported 64.16% accuracy is an upper bound under this coverage constraint, and the method cannot recover families absent from AV detections. This makes the central accuracy claim sensitive to AV family coverage in the target corpus: on corpora with sparser coverage the accuracy ceiling will be lower, and the comparison with prior tools is partly determined by how often the assumption holds rather than solely by the quality of the aggregation. The paper is transparent about the limitation, so this is not a hidden flaw, but it is the most load-bearing constraint on the central claim: it defines the conditions under which the reported accuracies are achievable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ClarAVy is an antivirus-detection-based malware family labeling tool. The paper describes three components: improved AV detection parsing, a two-tier alias resolution scheme (trivial, sibling, and parent-child aliases), and SparseIBCC, a sparse variational Bayesian implementation of the IBCC/Dawid-Skene label aggregation model, plus a confidence-score model for practical use. The evaluation compares ClarAVy with five prior tools on MOTIF and MalPedia, using externally derived ground-truth labels, and reports 64.16% and 56.88% family-labeling accuracy, respectively, which is 8 and 12 percentage points above the best prior tool. The paper also reports scaling to 39,747,485 VirusTotal reports and analyzes runtime as a function of dataset size.","tokens_in":16586,"tokens_out":5900,"duration_ms":56402,"significance":"The central comparison is meaningful because the MOTIF and MalPedia labels are not AV-derived, so the reported gains are not circular. The paper is also transparent about the main limitation: Section 6.4 quantifies how often AV detections lack or nearly lack the true family. The SparseIBCC memory analysis is a concrete contribution that makes large-scale aggregation feasible, and the authors state that the tool will be released. If the reported numbers hold, ClarAVy would be a new practical baseline for AV-based family labeling. However, the headline accuracy is an upper bound conditioned on AV family coverage, the runtime-scaling evidence in Section 6.3 is internally inconsistent, and the confidence-score evaluation needs clarification before the practical claims are fully supported.","major_comments":[{"comment":"The assumption t_i ∈ c_i in Section 5.5 restricts the posterior to families appearing in a file's AV scan report, and Section 6.4 reports that 18.64% of MOTIF scans contain no detection with the true family and another 11.53% contain it only once. This means the reported 64.16% and 56.88% accuracies are coverage-dependent ceilings rather than properties of the aggregation method alone: on corpora with sparser AV family coverage the achievable accuracy is lower by construction, and the comparison with prior tools is partially determined by how often the coverage assumption holds. Please report accuracy both on the full test sets and on the subset of scans for which the true family appears at least once, and discuss how the ceiling would shift under lower coverage.","section":"§5.5 and §6.4"},{"comment":"The text states that Figure 7 includes a quadratic curve of best fit and then concludes that SparseIBCC's runtime complexity is 'sub-linear in practice.' A quadratic fit is not evidence of sub-linear scaling; at best it is a local fit whose shape depends on the range, and at worst it contradicts the claim. Please report the fitted functional form with its parameters and goodness of fit, or measure scaling explicitly (e.g., wall-clock time per iteration or per report as N grows), and reconcile the statement with the empirical curve.","section":"§6.3 and Figure 7"},{"comment":"The confidence-score model in Section 5.6 is trained on the combined MOTIF and MalPedia sets with five-fold cross-validation, but Section 6.2 does not state whether Figures 5 and 6 evaluate accuracy on held-out cross-validation folds or on the training data. If the latter, the high accuracy at confidence thresholds is optimistic. Please state explicitly which predictions were used and, if necessary, report the held-out curves.","section":"§5.6 and §6.2"},{"comment":"The evaluation uses default values for a large number of manually selected parameters (S, T, E, C, M, plus the 934 hand-written parsing rules and curated alias/placeholder lists), but no sensitivity analysis is reported. The 8–12 point improvement over AVClass is the central claim, and it is not clear how stable this margin is to reasonable changes in these thresholds (for example, S ∈ [0.9, 0.99] or T ∈ [500, 2000]). Please add a sensitivity analysis for the new alias-resolution thresholds and, at minimum, report the accuracy range across plausible settings.","section":"§6.1 and Sections 4.2–4.3, 5.2"},{"comment":"The MalPedia benchmark is cleaned by the authors before evaluation: generic family names are removed and unresolved aliases are fixed, but the procedure is not quantified and the cleaned labels are not released. Because the comparison in Table 2 depends on this cleaning—and on how each tool's alias mapping relates to it—the 12-point advantage on MalPedia may be sensitive to the cleaning decisions. Please publish the cleaned label mapping, quantify how many files and labels were changed, and verify that the results are stable under reasonable alternative cleaning choices.","section":"§6.1, MalPedia cleaning"}],"minor_comments":[{"comment":"The sentence 'Automated tools using that label malware using antivirus detections lack accuracy and/or scalability' appears to contain a typo; please rephrase.","section":"Abstract"},{"comment":"The word 'aggretation' should be 'aggregation'.","section":"Section 2, item 3"},{"comment":"The phrase 'Shannon’s entropy the detected families' is missing 'of'; it should read 'Shannon’s entropy of the detected families.'","section":"§5.6"},{"comment":"The word 'recieve' should be 'receive.'","section":"§6.2"},{"comment":"The spelling 'ClarA Vy' is used inconsistently in the running text; please standardize to 'ClarAVy.'","section":"§6.1.6"},{"comment":"Reference [29], cited as AVClass by Sebastián and Caballero, points to a GitHub URL for 'avclassplusplus'; please verify that the reference and URL match the intended tool.","section":"References"},{"comment":"The main accuracy numbers are reported as point estimates; adding standard errors or confidence intervals would help readers judge the stability of the differences, even though the observed gaps are large.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a tool paper for a companion venue, and the scope is appropriate. I did not find evidence of circularity in the central accuracy comparison. The main risks are overstatement of the scalability evidence and the coverage-dependent interpretation of the headline accuracy; both are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a serious, useful paper, and the main accuracy claim probably holds. SparseIBCC is a genuine adaptation: sparse confusion matrices plus restricting the posterior to families observed in each file's AV report makes VB-IBCC tractable at roughly 50k classes and 40M reports. The ablation (ClarAVy with plurality voting vs SparseIBCC) isolates the aggregator's contribution at about 4-5 points, and the comparisons against AVClass, EUPHONY, AVClass++, Sumav, and TagClass are the right ones. External ground truth from MOTIF and MalPedia is a real strength; unlike some labeling papers, this one isn't evaluating against its own output.\n\nWhat deserves credit: the paper is honest about its load-bearing assumption. Section 5.5 says outright that t_i must be in c_i, and Section 6.4 discloses that 18.64% of MOTIF scans can't be correctly labeled by any AV aggregation because no detection contains the true family, with another 11.53% depending on a single detection. That transparency lets a reader calibrate the 64.16% and 56.88% numbers. They should be read as an upper bound for corpora with AV coverage like MOTIF/MalPedia, not as a property of the aggregator alone.\n\nSoft spots, in order. First, no confidence intervals anywhere; with roughly 3k and 10k evaluation files, a few percent swing is plausible. The 8-12 point lead is larger than that, so this does not kill the claim, but the numbers need error bars. Second, the confidence-score evaluation is underspecified: Figures 5 and 6 don't say explicitly whether accuracy is computed on held-out CV folds, and the XGBoost was trained on the same two datasets. As written, you can't rule out leakage. Third, Section 6.3 says a quadratic curve of best fit \"indicates SparseIBCC's runtime complexity is sub-linear.\" That is backwards; a quadratic fit indicates super-linear growth in the sampled range, even if the coefficient is small. The text needs correction. Fourth, MalPedia was cleaned before evaluation; not fatal, but it means the MalPedia number is on a filtered subset. Fifth, code isn't released yet, only promised. For an empirical tool paper, that matters.\n\nThe t_i in c_i restriction is a real limitation, but the authors flag it, and the restriction is exactly what makes the method scale. I don't think it invalidates the central contribution; it sets the conditions under which the tool is the right choice.\n\nVerdict: send to review. The paper deserves referee time and would benefit from a request for released code, corrected runtime text, and CV-clarified confidence curves. I'd cite it if I worked on malware labeling, and it's a reasonable reading-group paper for a security/ML audience.","headline":"Solid applied ML paper: SparseIBCC gives a real accuracy gain for AV-based malware family labeling, but the headline numbers are coverage-dependent and the runtime section needs correction.","tokens_in":17117,"tokens_out":2330,"would_cite":true,"duration_ms":21930,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ClarAVy beats the prior best malware family labeler by 8-12 points using a sparse Bayesian aggregator, and scales to 40 million scan reports.","keywords":["malware family labeling","antivirus detection aggregation","Dawid-Skene algorithm","variational Bayes","sparse IBCC","alias resolution","malware classification","scalable labeling"],"falsifier":"Take the subset of MOTIF files whose true family is absent from every antivirus detection — the 18.64% the paper identifies — and measure ClarAVy's accuracy on that subset against ground truth: it would necessarily be zero, because the tool can only output a family that appeared in the scan. A simpler test is to construct a synthetic scan report with the true family deliberately withheld from all vendors and confirm that ClarAVy never outputs it, showing that the accuracy ceiling is set by antivirus coverage rather than by the aggregation method.","tokens_in":16110,"feed_emoji":"🦠","tokens_out":11649,"duration_ms":91448,"temperature":0.7,"pith_summary":"The paper argues that automated malware family labeling from antivirus scan reports has been held back by three fixable defects: careless parsing of detection strings, failure to resolve family-name aliases, and aggregation strategies that are either too naive or too memory-hungry for real-world corpora. ClarAVy addresses all three, and its new sparse variational Bayesian aggregator, SparseIBCC, lifts family-label accuracy to 64.16% on the MOTIF benchmark and 56.88% on MalPedia, roughly 8 and 12 percentage points above the best previous tool. If correct, this gives security teams a practical way to label tens of millions of files automatically, providing better training labels for downstream machine-learning classifiers and faster triage for incident responders.","feed_headline":"Malware labeling tool beats prior best by 8-12 points","feed_subtitle":"ClarAVy fuses antivirus reports via sparse Bayesian aggregation, beating the prior tool while labeling 40M files.","key_machinery":"SparseIBCC — the paper's sparse implementation of the Independent Bayesian Classifier Combination (VB-IBCC) algorithm — is the load-bearing component. VB-IBCC approximates the Dawid-Skene EM iteration with variational Bayes; SparseIBCC keeps the same statistical model but represents each antivirus product's confusion matrix only over families that co-occur, and it computes each file's class posterior only over families present in that file's scan report. This is what turns a 50,000-class, 40-million-file inference problem into something that runs in about an hour on a 128-thread machine, making the reported accuracy and scale possible.","core_discovery":"The central discovery, on the paper's own terms, is that extreme-multiclass crowdsourcing models can be made to work for malware at web scale by exploiting two sparsity patterns. First, nearly all pairs of malware families never co-occur in scan reports, so the per-antivirus confusion matrices of a variational Bayesian Dawid-Skene model can be stored sparsely at about 1/200 of the dense memory. Second, the posterior for a file can be restricted to the families that actually appear in that file's scan report, shrinking an O(NL) problem to O(NK) and cutting memory by a further factor of 500-5000. Together with manually crafted rules for parsing 103 antivirus products and for resolving trivial, sibling, and parent-child aliases, these changes let ClarAVy label 39,747,485 scan reports in about 37 hours while beating the prior leading tool by 8 points on MOTIF and 12 on MalPedia.","pith_inferences":["A natural extension the paper does not test: the sparse-confusion representation should transfer to other extreme-multiclass crowdsourcing tasks, such as fine-grained image annotation or labeling rare scientific entities, wherever class co-occurrence is rare.","One testable prediction from the paper's own case studies is that richer family vocabularies from antivirus vendors would raise accuracy more than any further refinement of the aggregation algorithm.","A hybrid that adds static or dynamic malware analysis for files whose true family never appears in the scan report could push past the AV-coverage ceiling; the paper stops at AV-only labeling.","The 52,371-family taxonomy and 4,472 alias pairs generated from 40 million reports are a reusable asset for the malware research community, but the paper does not analyze the structure of that alias graph."],"forward_implications":["Security analysts can get family labels for tens of millions of files in about a day of compute, making whole-corpus labeling practical rather than a research luxury.","Downstream malware classifiers trained on ClarAVy labels should see better label quality than those trained on prior aggregation tools, since ClarAVy beats the leading baseline by 8-12 percentage points on two benchmarks.","The confidence score gives a precision lever: on the MOTIF set, keeping only scans with confidence at or above 70% yields roughly 90% label accuracy.","Antivirus coverage, not aggregation method, sets the ceiling: 18.64% of MOTIF scans cannot be correctly labeled from AV reports alone no matter how the votes are combined.","Most remaining mislabels are near misses (variants of the true family or catch-all names used by vendors), so improving vendors' family vocabularies would do more than further algorithmic tuning."],"supporting_citations":[{"why":"Provides the VB-IBCC variational Bayesian model that SparseIBCC adapts to extreme multiclass malware labeling.","marker":"[33]"},{"why":"Contributes the sparse implementation of VB-IBCC that ClarAVy extends to the large-L, small-K regime.","marker":"[32]"},{"why":"The original Dawid-Skene EM algorithm for aggregating noisy annotator labels, the statistical foundation of the aggregation strategy.","marker":"[7]"},{"why":"Defines the prior leading tool and the sibling-alias co-occurrence method that ClarAVy adapts for family alias resolution.","marker":"[30]"},{"why":"Supplies the MOTIF ground-truth benchmark with labels derived from open-source reporting, one of the two evaluation datasets.","marker":"[14]"},{"why":"Supplies the MalPedia ground-truth benchmark with labels from open-source reporting and YARA rules, the other evaluation dataset.","marker":"[27]"},{"why":"Provides the corpus of roughly 40 million antivirus scan reports used for the scale test and taxonomy generation.","marker":"[1]"},{"why":"Earlier ClarAVy work that supplies the parsing rules, tag taxonomy, and known antivirus product relationships reused here.","marker":"[16]"}],"fun_headline_variants":["ClarAVy labels 40M malware files with 8-12 pt accuracy gain","Sparse Bayesian aggregation powers malware labeling at 40M scale","Malware family labeling gets 8-12 point boost and 40M-file scale","ClarAVy: accurate malware labels for 40 million files, 8-12 pts better","Beats prior tool by 8-12 points while labeling 40M files"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a file's true family is always one of the family names that appears in that file's antivirus scan report, and the paper reports that 18.64% of MOTIF files violate this assumption.","fun_headline_variants_meta":{"raw":{"variants":["ClarAVy labels 40M malware files with 8-12 pt accuracy gain","Sparse Bayesian aggregation powers malware labeling at 40M scale","Malware family labeling gets 8-12 point boost and 40M-file scale","ClarAVy: accurate malware labels for 40 million files, 8-12 pts better","Beats prior tool by 8-12 points while labeling 40M files"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000817,"raw_usage":{"total_tokens":3565,"prompt_tokens":919,"completion_tokens":2646,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":2551}},"tokens_in":535,"tokens_out":2646,"duration_ms":17895,"temperature":1.0,"reasoning_tokens":2551,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T11:14:25.587130+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the subset of MOTIF files whose true family is absent from every antivirus detection — the 18.64% the paper identifies — and measure ClarAVy's accuracy on that subset against ground truth: it would necessarily be zero, because the tool can only output a family that appeared in the scan. A simpler test is to construct a synthetic scan report with the true family deliberately withheld from all vendors and confirm that ClarAVy never outputs it, showing that the accuracy ceiling is set by antivirus coverage rather than by the aggregation method.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the VB-IBCC variational Bayesian model that SparseIBCC adapts to extreme multiclass malware labeling."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the sparse implementation of VB-IBCC that ClarAVy extends to the large-L, small-K regime."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the prior leading tool and the sibling-alias co-occurrence method that ClarAVy adapts for family alias resolution."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MalPedia ground-truth benchmark with labels from open-source reporting and YARA rules, the other evaluation dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the corpus of roughly 40 million antivirus scan reports used for the scale test and taxonomy generation."},{"cited_title":"Joyce, Edward Raff, Charles Nicholas, and James Holt","cited_arxiv_id":null,"evidence_quote":"Earlier ClarAVy work that supplies the parsing rules, tag taxonomy, and known antivirus product relationships reused here."}],"review_version":1}