{"id":"0ab42326-14e1-4098-b806-14aa9a7e032a","arxiv_id":"2412.06166","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"MVD trains a multi-lingual CodeBERT classifier with a focal-plus-logit-adjustment loss and distillation-based incremental learning, claiming large gains over single-language baselines that mostly vanish under fair comparison.","lead":"MVD is a proposed framework that trains one neural network to spot security bugs in Python, Java, C/C++, C#, JavaScript, and TypeScript source code at once. Its headline claim of beating single-language models by up to 193% is backed by an unfair comparison, so the real gain is much smaller.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 83.7%-193.6% gains are computed against off-language LineVul baselines; against the per-language LineVul model, MVD is worse on C# and JavaScript and only 0.6% better on Python.","rationale":"The reader's prose rationale correctly identifies that the percentage wins are calculated against off-language baselines, but the reader's formal weakest_assumption is the project-leakage risk from the random 8:1:1 function-level split. I agree that project leakage is a serious threat to validity and that a project-grouped split should be run before accepting the absolute PR-AUC values. However, the single most load-bearing problem for the paper's central claim is the construction of the 83.7%-193.6% comparison itself. A reader naturally understands this as MVD beating the LineVul model trained for each language on that language's test set. That is not what the numbers in Section V-A do: they pool MVD's average across six languages against each single-language baseline's average across all six languages, including languages outside the baseline's training distribution. The correctly paired comparisons in Table II show MVD loses on C# and JavaScript, is nearly tied on Python, and only clearly wins on C/C++, Java, and TypeScript. The aggregate improvement over the six same-language baselines is about 11.9% in PR-AUC, a much weaker and more nuanced claim than the abstract's 83.7%-193.6%. Because this inflated comparison is the paper's headline evidence and is repeated in the abstract and conclusion, the central claim is not supported as stated. The FOLA loss definition in Equation (1) also appears inconsistent with the cited logit-adjustment method, and the distillation loss in Equation (2) mixes logits with probabilities, but I do not need those issues to reject the paper: the baseline comparison alone is decisive. I would keep the reader's REJECT verdict, hence UNCHANGED.","tokens_in":19342,"tokens_out":7188,"duration_ms":66129,"concrete_test":"Recompute the six headline relative gains from Table II as (PR-AUC_MVD_lang - PR-AUC_LineVul_same_lang) / PR-AUC_LineVul_same_lang for each language, using only the row where the LineVul model was trained on the same language as the test set. Verify whether the resulting range is approximately -4.5% to +148.4% with an average PR-AUC gain of about 11.9%, rather than the claimed 83.7%-193.6%. No reruns are needed because all required numbers are already in Table II.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, repeated in the abstract, Section V-A, and the conclusion, is that MVD outperforms state-of-the-art single-language LineVul models by 83.7%-193.6% in PR-AUC. Table II shows that these percentages are not per-language comparisons. For each test language, MVD versus LineVul trained on that same language yields: Python 0.8875 vs 0.8824 (+0.6%), C/C++ 0.3418 vs 0.2534 (+34.9%), Java 0.4204 vs 0.3216 (+30.7%), C# 0.7352 vs 0.7427 (-1.0%), JavaScript 0.5345 vs 0.5594 (-4.5%), and TypeScript 0.3065 vs 0.1234 (+148.4%). The 83.7%-193.6% figures instead compare MVD's average PR-AUC (0.5376) to each language-specific LineVul model's average PR-AUC over all six test languages, for example 0.2926 for LineVul-Python. That compares MVD, which was trained on all six languages, against a model evaluated mostly on languages on which it was never trained. This is not the state-of-the-art comparison claimed. With correctly paired baselines, MVD is worse on two of the six languages, essentially tied on Python, and the average PR-AUC gain is roughly 11.9%, not 83.7%-193.6%. The random-split leakage identified by the reader may further inflate absolute numbers, but this baseline-construction flaw alone invalidates the paper's headline quantitative claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MVD, a multi-lingual vulnerability detection framework built on CodeBERT. MVD is trained as a multi-class classifier over vulnerable classes per language plus a clean class, uses a proposed FOLA loss that combines focal loss with a logit-adjustment term to handle class imbalance, and includes an incremental learning module based on knowledge distillation for adding new languages. The authors curate a dataset of over 11K vulnerable functions across Python, C/C++, Java, C#, JavaScript, and TypeScript, and report experiments comparing MVD to per-language LineVul models, an ablation study of the proposed components, and incremental learning results. The central claim is that MVD outperforms state-of-the-art single-language models by 83.7% to 193.6% in PR-AUC.","tokens_in":19633,"tokens_out":4618,"duration_ms":41707,"significance":"If the headline claim were valid, this would be a practically useful result: a single model that matches or exceeds language-specific state-of-the-art detectors across six languages would substantially reduce the cost of multi-language vulnerability detection. The paper also contributes a curated multi-lingual dataset, public artifact links, and an incremental learning setting that is relatively unexplored for vulnerability detection. The ablation study is a reasonable attempt to attribute gains to the multi-class formulation, the FOLA loss, and fine-tuning. However, the quantitative central claim is not supported by the presented analysis: the 83.7%–193.6% figures are computed against off-language LineVul models, and the same-language comparisons in Table II show MVD losing on two of six languages and nearly tying on a third. The additional risk of project-level data leakage from the random function-level split further undermines the reliability of the reported absolute numbers. The contribution is therefore not established as stated.","major_comments":[{"comment":"The headline claim that MVD outperforms LineVul by 83.7%–193.6% in PR-AUC is not supported by a same-language comparison. In Table II, the 'Average' row compares MVD's average PR-AUC (0.5376) against each LineVul model's average PR-AUC over all six test languages (e.g., LineVul-Python = 0.2926), which includes languages that the LineVul model was never trained on and for which it performs near chance. Against the correctly paired same-language baseline, MVD improves by 0.6% on Python, 34.9% on C/C++, 30.7% on Java, and 148.4% on TypeScript, but is worse by 1.0% on C# and 4.5% on JavaScript. The average improvement over same-language baselines is approximately 11.9%, not the 83.7%–193.6% repeated in the abstract, Section V-A, and conclusion. This error is load-bearing for the paper's central quantitative claim.","section":"Section V-A, Table II"},{"comment":"The evaluation uses a random 8:1:1 split of individual functions into training, validation, and testing sets without grouping by project. Because functions from the same repository can appear in both training and testing, the model can memorize project-specific identifiers and patterns, inflating PR-AUC for both MVD and the LineVul baselines. The threats-to-validity section does not mention this issue. The authors should either repeat the main experiments with a project-level split or quantitatively show that the overlap does not affect the conclusions; without this, the reported absolute PR-AUC values and the claimed gains in RQ1, RQ2, and RQ3 do not establish generalization to unseen projects.","section":"Section IV-D, Section VI"},{"comment":"The proposed FOLA loss relies on hyperparameters α_t, γ, and τ, but none of these values are reported in the experimental setup or anywhere else in the manuscript. This omission prevents reproduction of the method and makes it impossible to assess whether the class-imbalance handling is actually responsible for the observed results. In addition, the term τ log(q_t) as written in Eq. (1) adds a constant per-class offset to the loss, which is not the standard implementation of logit adjustment from the cited work [38]; logit adjustment is normally applied to the logits before softmax. The relation between Eq. (1) and the cited method should be clarified, and all loss hyperparameters should be reported.","section":"Equation (1), Section IV-D"}],"minor_comments":[{"comment":"The sentence 'MVD was approximately 7% better PR-AUC than that (0.5008) of the LineVul models trained for each language individually and requiring nearly five times more resources' is confusing: the number 0.5008 does not appear in Table II, and the computation of the five-fold resource comparison is not described.","section":"Section V-A"},{"comment":"The rows labeled 'w/o-X' and 'inc-X' are not defined in the table caption; the caption should explain that w/o-X denotes a model trained on all languages except X, and inc-X denotes that model after incremental learning on X.","section":"Table IV"},{"comment":"The statement that 'the performance on the original languages did not degrade significantly' is not fully supported by Table IV: for example, inc-Java drops Python PR-AUC from 0.8898 to 0.8059, a relative decrease of about 9.4%, which is larger than 'modest' for that language. The claim should be qualified with the observed variance.","section":"Section V-C"},{"comment":"The related work on incremental learning is brief and does not discuss knowledge-distillation-based continual learning methods, which are the direct technical basis for the proposed extension module; adding a short discussion would improve context.","section":"Section II-C"},{"comment":"Figure 1 is difficult to read at the current resolution; the three panels would benefit from higher-resolution rendering and larger labels.","section":"Figure 1"}],"recommendation":"reject","confidential_remarks":"The reader's stress-test analysis is correct: the 83.7%–193.6% improvements in Section V-A are an artifact of comparing MVD to each LineVul model averaged over all six test languages, rather than to the same-language baseline. With the same-language baselines, MVD loses on C# and JavaScript, and the average gain is about 11.9%. The function-level random split in Section IV-D introduces a likely project-level leakage that is not addressed in the threats to validity. Both issues are load-bearing for the paper's central claims, and the second would require substantial new experiments. I therefore recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The paper's actual contribution is a first attempt at one model for vulnerability detection across six languages, with a new curated dataset and an incremental-learning extension. The 83.7%-193.6% headline, though, is built on comparing the multi-lingual MVD to single-language LineVul models evaluated mostly on languages they never saw. Per-language, MVD beats LineVul on C/C++ (0.342 vs 0.253), Java (0.420 vs 0.322), and TypeScript (0.307 vs 0.123), ties on Python (0.888 vs 0.882), and loses on C# and JavaScript. The real benefit is having one detector and modest gains on half the languages, not the dramatic superiority claimed.\n\nThe dataset and the multi-class design are legitimate. Curating 11K+ vulnerable functions across six languages is real work, and the ablation study gives each component a fair test. The incremental-learning experiment is the most interesting part: adding a new language with distillation mostly preserves old-language performance, and sometimes improves it. That is a useful result even if the full six-language model stays better.\n\nNow the soft spots, in order of importance. First, the baseline construction invalidates the headline claim. The abstract, Section V-A, and conclusion repeat the 83.7%-193.6% numbers, and they are not supported by Table II. Second, the FOLA loss in Eq. (1) adds τ log(q_t) as a loss term, but the cited logit adjustment method adds it to the logits before softmax. Either the equation or the name is wrong, and the α, γ, τ values are never reported. Third, the distillation loss in Eq. (2) compares the new model's logits to the teacher's softmax probabilities; that is dimensionally mismatched and needs clarification or a temperature-scaled form. Fourth, the random 8:1:1 split does not group by project, so functions from one repo can appear in both training and test. This is a known leakage path in vulnerability detection and is not addressed in the threats section. It likely inflates all absolute numbers and could affect the per-language comparisons.\n\nThe paper deserves a serious referee: the task is timely, the dataset is a concrete resource, and the incremental-learning setup is genuinely novel. But in current form the central claim does not hold. The authors need to redo the comparison against per-language baselines, fix or explain the losses, report the loss hyperparameters, and rerun with a project-aware split. I would send it out but expect major revision; the published version of the headline should be something like 'a single multi-lingual model is competitive with per-language models and extends cheaply to new languages.'","headline":"A useful first multi-lingual vulnerability detector, but the headline gains are an artifact of comparing against off-language baselines – per-language it wins three, loses two, and ties one.","tokens_in":20222,"tokens_out":4385,"would_cite":true,"duration_ms":39308,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single model trained on vulnerabilities in six programming languages outperforms language-specific detectors and adds new languages without forgetting.","keywords":["software vulnerability detection","multi-lingual code analysis","function-level vulnerability prediction","CodeBERT","class imbalance","FOLA loss","incremental learning","PR-AUC"],"falsifier":"Retrain and evaluate MVD and the LineVul baselines with a repository-disjoint split, where all functions from any given project appear in only one of training, validation, or testing; then compare PR-AUC. If MVD's large advantage shrinks or reverses, the reported gains depend on within-project leakage rather than cross-language transfer.","tokens_in":19082,"feed_emoji":"🛡️","tokens_out":6984,"duration_ms":60939,"temperature":0.7,"pith_summary":"This paper introduces MVD, a framework that trains one deep-learning model to detect vulnerable functions in Python, C/C++, Java, C#, JavaScript, and TypeScript at the same time. The authors' central claim is that this multi-lingual model beats the single-language state-of-the-art baseline LineVul on every one of the six languages, with relative PR-AUC improvements from 83.7% to 193.6% and an average PR-AUC of 0.5376 across languages. They also claim MVD can learn a new language through incremental learning using only that language's data, while largely preserving performance on the original languages. If these results hold, polyglot software projects would not need one separately trained detector per language, and adding support for an emerging language would become much cheaper. The evaluation uses a curated dataset of more than 11,000 real-world vulnerabilities.","feed_headline":"One model beats single-language vulnerability detectors by up to 194%","feed_subtitle":"MVD trains on six languages at once and still tops specialized baselines, then adds new languages without forgetting.","key_machinery":"The load-bearing mechanism is a multi-class classifier built on CodeBERT, a transformer-based code model pre-trained on multi-language source code. The classifier has a 'clean' class plus one vulnerable class per language, which lets the model learn both what a vulnerability looks like and how its expression varies across languages; the FOLA loss, a hybrid of Focal loss and logit adjustment, reweights hard examples and adjusts logits by class frequency to keep minority languages from being ignored; and the incremental stage adds a distillation loss that pins the new model's old-language logits to the frozen teacher's outputs. This combination is what lets one model carry cross-language vulnerability knowledge and acquire new languages without retraining on old data.","core_discovery":"MVD is a CodeBERT model fine-tuned as a multi-class classifier whose output has one 'clean' class plus one vulnerable class per supported language. At test time the per-language vulnerable probabilities are summed into a single binary vulnerability score, so the model makes one detection decision regardless of language. During training the composite FOLA loss combines Focal loss with logit adjustment to counteract the heavy class imbalance in the curated data, and a distillation loss is added when extending to a new language: the new model is trained to keep its logits for old languages close to those of the frozen teacher, while learning the new language from new data. On the six-language test sets, MVD reports average PR-AUC of 0.5376, compared with averages between 0.1831 and 0.2926 for the six language-specific LineVul baselines, and the per-language relative gains are the 83.7%-193.6% figures. In the incremental-extension experiments, the model outperforms a language-only baseline for four of six new languages and mostly retains its previous-language performance.","pith_inferences":["Because the evaluation splits functions randomly rather than by project, functions from the same repository can appear in both training and testing; a repository-disjoint split would test whether the reported gains reflect cross-language transfer or project-specific memorization.","The language-classification head may function as an implicit regularizer that forces the representation to separate language-specific vulnerability patterns; removing language supervision while keeping multi-lingual data would isolate this effect.","The language-order sensitivity observed in the incremental experiments suggests that choosing which new language to add and when could be scheduled by similarity to the already learned languages to improve transfer.","The multi-class encoder could be extended with a token-level head to predict line-level vulnerability locations across languages, combining the multi-lingual representation with the localization ability of line-level detectors."],"forward_implications":["A single multi-lingual model can replace six separately trained detectors, cutting training cost by roughly a factor of five while matching or exceeding each per-language detector's PR-AUC.","Vulnerability knowledge transfers across languages: MVD lifts performance on low-resource languages such as TypeScript above what a model trained on that language alone achieves.","New languages can be added incrementally with only the new language's data, and the model outperforms a from-scratch single-language model in four of six tested languages.","Freezing the code model and training only the classifier loses about 30 points of average PR-AUC, so fine-tuning the full model is load-bearing for the result."],"supporting_citations":[{"why":"Supplies CodeBERT, the pre-trained code model whose weights MVD fine-tunes.","marker":"[17]"},{"why":"Provides LineVul, the state-of-the-art single-language baseline that MVD is compared against in RQ1.","marker":"[11]"},{"why":"Supplies the CVEfixes curation methods and tools used to build the six-language vulnerability dataset.","marker":"[41]"},{"why":"Defines the common practice of labeling functions touched by vulnerability-fixing commits as vulnerable, which the data curation follows.","marker":"[43]"},{"why":"Provides CodeSearchNet, the multi-language code corpus that motivates CodeBERT's capacity for cross-lingual representation.","marker":"[34]"},{"why":"Contributes the hard-example weighting term used inside the FOLA loss.","marker":"[37]"},{"why":"Contributes the logit-adjustment term used inside the FOLA loss to counteract class-frequency imbalance.","marker":"[38]"},{"why":"Supplies the learning-without-forgetting distillation idea used for incremental language extension.","marker":"[40]"},{"why":"Supports the claim that LineVul-style CodeBERT models are the current state of the art for function-level vulnerability prediction.","marker":"[24]"}],"fun_headline_variants":["One model, six languages, up to 194% better vulnerability detection","MVD learns new programming languages without forgetting old ones","Multi-lingual bug detector beats single-language tools by 194%","From C++ to Python: MVD finds flaws across six languages at once","Polyglot AI detects vulnerabilities better than language-specific models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 8:1:1 split of functions assumes that functions from the same repository in the test set are not made artificially easy by project-specific patterns learned from training; if the same project's code appears in both sets, the measured PR-AUC overstates detection on code the model has not seen.","fun_headline_variants_meta":{"raw":{"variants":["One model, six languages, up to 194% better vulnerability detection","MVD learns new programming languages without forgetting old ones","Multi-lingual bug detector beats single-language tools by 194%","From C++ to Python: MVD finds flaws across six languages at once","Polyglot AI detects vulnerabilities better than language-specific models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000394,"raw_usage":{"total_tokens":2115,"prompt_tokens":1041,"completion_tokens":1074,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":657,"completion_tokens_details":{"reasoning_tokens":984}},"tokens_in":657,"tokens_out":1074,"duration_ms":10289,"temperature":1.0,"reasoning_tokens":984,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:56:18.727422+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain and evaluate MVD and the LineVul baselines with a repository-disjoint split, where all functions from any given project appear in only one of training, validation, or testing; then compare PR-AUC. If MVD's large advantage shrinks or reverses, the reported gains depend on within-project leakage rather than cross-language transfer.","supporting_citations":[{"cited_title":"A c/c++ code vulnerability dataset with code changes and cve summaries,","cited_arxiv_id":null,"evidence_quote":"Defines the common practice of labeling functions touched by vulnerability-fixing commits as vulnerable, which the data curation follows."},{"cited_title":"Long-tail learning via logit adjustment,","cited_arxiv_id":null,"evidence_quote":"Contributes the logit-adjustment term used inside the FOLA loss to counteract class-frequency imbalance."}],"review_version":1}