{"id":"e2a6ccaf-a9a4-446e-9db6-3afb0fb3f743","arxiv_id":"2504.15632","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"On BigVul with LineVul, representation-level augmentations improve F1 by up to about 9.7%, but random oversampling still wins by a larger margin.","lead":"This paper tests whether five mixup-style augmentation methods, applied directly to the vector representations of vulnerable code, improve deep learning vulnerability detection on the BigVul dataset. It finds they help somewhat, but simple random oversampling of the minority class remains the best strategy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run evaluation without variance reporting leaves the 0.40 F1 gap between Stochastic Perturbation and ROS indistinguishable from noise; the core comparative claims may not reproduce.","rationale":"After reading the paper, the broad claim that simple ROS remains hard to beat is plausible and consistent with prior work by Yang et al. and Daneshvar et al., and the code release is a plus. However, the paper's own quantitative recommendation hinges on comparisons whose differences (0.4 F1) are smaller than typical seed variance for transformer-based vulnerability detectors, and no repeated runs are reported. The reader's weakest assumption correctly identifies this. The augmentation-multiplicity description is ambiguous (rate 23, 22x, 129,690 items), but the more decisive issue is the absence of variance estimation: even a perfectly matched sample count would not make a single-run 0.40-point gap trustworthy. The conditioned-versus-blind difference in Section 5.1 is interpreted mechanistically on the basis of a 0.26-point gap, which strengthens the need for seed variance. The internal 9.67/9.96 inconsistency and the use of a single model and dataset are secondary. I would keep the verdict CONDITIONAL, since the central conclusion may survive re-runs but currently lacks statistical support.","tokens_in":10005,"tokens_out":7797,"duration_ms":68630,"concrete_test":"Rerun the main comparison (No augmentation, ROS, blind and conditioned Stochastic Perturbation, and Linear Interpolation as the runner-up) with at least 10 random seeds, keeping the same fixed split and checkpoint selection, and report the mean F1 with 95% confidence intervals plus a paired test (e.g., Wilcoxon signed-rank) between ROS and blind SP. If the ROS and SP intervals overlap to within the observed 0.40 gap, the ordering and the headline recommendation are inconclusive; if the gap persists with p<0.05, the central claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that embedding-level augmentation cannot beat ROS when balancing rests entirely on single runs in Tables 3 and 4, with no variance reporting. The decisive gaps are small: ROS F1 38.50 versus blind Stochastic Perturbation 38.10 (0.40 points), and blind SP versus conditioned SP 38.10 versus 37.84 (0.26 points). The validation set is used for checkpoint selection, which can amplify noise. Prior LineVul/BigVul results show nontrivial seed variance; without confidence intervals or significance tests, the method ordering and the 'cannot beat ROS' conclusion are not established. Section 5.1 even interprets a 0.26-point SP difference mechanistically, though it may be within run-to-run noise. A secondary internal inconsistency (Abstract and Findings say 9.67% gain, Introduction says 9.96%) does not invalidate the result but signals the need for a careful re-check.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical study of five representation-level (embedding) augmentation methods—Linear Interpolation, Stochastic Perturbation, Linear Extrapolation, Binary Interpolation, and Gaussian Scaling—for deep learning-based vulnerability detection using LineVul and the BigVul dataset. Each method is applied in a blind form and in a novel conditioned form that keeps the embeddings of vulnerable lines fixed. The authors compare against no augmentation, Random Oversampling (ROS), and the VGX generation method. The central findings are that Stochastic Perturbation is the best representation-level method (blind F1 38.10 versus baseline 34.74, relative improvement 9.67%), that all representation-level methods beat VGX, and that none of them surpasses ROS (F1 38.50, improvement 10.82%). The paper recommends ROS for class balancing and identifies Stochastic Perturbation as the best embedding-level alternative.","tokens_in":10130,"tokens_out":6765,"duration_ms":56388,"significance":"If the findings are reproducible, the paper provides useful practical guidance for vulnerability detection research: ROS remains a strong, simple baseline, and embedding-level augmentation is a weak secondary option. The first systematic study of representation-level augmentation for DLVD and the conditioned augmentation variant are novel contributions. The artifact is publicly available on Zenodo, supporting replication. However, the empirical foundation is currently too weak to support the specific ordering of methods because all results come from single runs with no variance reporting or significance testing, and several differences between key methods are smaller than typical run-to-run variation.","major_comments":[{"comment":"All results in Tables 3 and 4 come from a single training run per configuration, and no standard deviations, confidence intervals, or significance tests are reported. The paper's central comparative claims rest on small differences: ROS (F1 38.50) versus blind Stochastic Perturbation (38.10) is a 0.40-point gap, and blind Stochastic Perturbation (38.10) versus the conditioned variant (37.84) is a 0.26-point gap. Because LineVul and BigVul are known to exhibit nontrivial seed-to-seed variance, these gaps may be within run-to-run noise. Without repeated runs (e.g., 5–10 seeds) and accompanying statistical tests, Findings 1 and 2 (SP is the best augmentation; none can beat ROS) are not established. Please provide multi-seed results with means and standard deviations and a paired significance test (e.g., Wilcoxon signed-rank or bootstrap) across seeds, or otherwise justify why a single run is sufficient for the claimed ordering.","section":"Section 4 (Tables 3 and 4)"},{"comment":"The discussion attributes the 0.26-point F1 drop of Stochastic Perturbation under conditioning (38.10 vs 37.84) to a mechanistic difference in dropout expectation. Given that no variance information is reported, this difference is plausibly noise, and the mechanistic explanation is therefore speculative. Please either support the claim with significance testing and, ideally, an ablation that isolates the conditioning effect, or reframe the passage as a hypothesis rather than a finding.","section":"Section 5.1"},{"comment":"All augmentation methods are evaluated at a single augmentation ratio (23 copies per line-annotated vulnerable sample), and no sensitivity analysis is provided. The relative ordering of methods, and in particular the claim that no representation-level method can beat ROS, may depend on this ratio. Please either vary the augmentation ratio (e.g., 10, 23, 50) or explicitly justify why the chosen ratio is sufficient to support the general recommendation in Section 5.4.","section":"Section 3.3"}],"minor_comments":[{"comment":"The Abstract reports the best augmentation gain as 9.67%, whereas the Introduction (Section 1) reports 9.96% for the same Stochastic Perturbation result; please correct the inconsistency.","section":"Abstract and Section 1"},{"comment":"In the Gaussian Scaling bullet, \"with small values of α\" appears to mean \"small values of σ\", since α is set to 1; please fix the notation. Also, in Binary Interpolation, α∈{0,1}^e uses the symbol e without defining it; please define the embedding dimension.","section":"Section 2.3"},{"comment":"The sentence \"Sections 5, 6 and 6\" should be \"Sections 5 and 6\".","section":"Section 1"},{"comment":"The phrase \"augmented the data at a rate of 23\" is ambiguous; clarify that this means each of the 5,895 line-annotated vulnerable samples is used to generate 22 additional copies, so the total number of generated items is 129,690.","section":"Section 3.3"},{"comment":"The statement that results are \"marked in blue/red\" may be inaccessible in grayscale; consider adding symbols (e.g., ▲/▼) to indicate improvement/degradation.","section":"Tables 3 and 4"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's scope and the core question is relevant. The main technical concern is the single-run evaluation, which is fixable but will require additional compute. Please also ask the authors to correct the 9.67% vs 9.96% inconsistency during revision, as it suggests a need for careful proofreading of the reported numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the paper. Short version: a genuinely useful empirical study with a plausible central conclusion, but the evaluation is too thin to support the finer-grained ordering, and the 'cannot beat ROS' headline rests on a 0.4 F1 gap from a single run. Worth engaging, but only with multiple seeds and significance testing before anyone trusts the rankings.\n\nWhat's actually new: this is the first systematic comparison of five embedding-level mixup-style augmentation methods for token-based vulnerability detection on BigVul with LineVul, and it introduces a conditioned variant that keeps vulnerable-line tokens fixed during augmentation. That conditioned variant is a reasonable idea, clearly described. The code is on Zenodo, which is a real plus for replication. The writing is clear, and the related work is honest: the five methods come from Li et al. and Dong et al., and the authors credit them properly.\n\nWhere it's soft: the central comparative claim rests on single runs. Tables 3 and 4 report one F1 per configuration with no confidence intervals and no significance tests. The decisive gap between ROS (38.50) and the best augmentation method, Stochastic Perturbation (38.10), is 0.40 points. The gap between blind and conditioned SP is 0.26 points. Given that prior LineVul/BigVul experiments show seed variance of that magnitude, the ordering of methods—and the claim that no representation-level method beats ROS—is not actually established by this data. The validation set is used for checkpoint selection, which adds another source of noise. Section 5.1 gives a mechanistic explanation for the 0.26-point conditioned drop, which overreads numbers that are likely within run-to-run noise. There's also an internal inconsistency: the abstract and Section 4 say the best augmentation gives a 9.67% gain, but the introduction says 9.96%. It looks like a typo, but it signals the numbers were not carefully rechecked. The augmentation 'rate of 23' is ambiguous—the text says the augmented dataset amounts to 22x the vulnerable samples, which needs a clearer definition.\n\nThe citation pattern is solid; the baselines (VGX, ROS) and prior findings are credited. The conditioned variant is new, and the result that VGX underperforms is consistent with earlier work. The broad conclusion that simple ROS is a strong baseline may well be true, but the paper doesn't prove it from these runs.\n\nWho this is for: practitioners deciding how to balance vulnerability datasets, and researchers working on augmentation for code. It deserves a serious referee—send it to peer review—but the revision should require multiple seeds, variance reporting, and significance testing before the rankings are taken as conclusions.","headline":"Useful first systematic study of embedding-level augmentation for vulnerability detection, but the single-run evaluation can't support the claimed ordering or the 'can't beat ROS' conclusion.","tokens_in":10707,"tokens_out":3086,"would_cite":false,"duration_ms":27004,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Simple oversampling beats embedding augmentation for vulnerabilities","keywords":["vulnerability detection","data augmentation","representation-level augmentation","mixup","stochastic perturbation","random oversampling","deep learning","software security"],"falsifier":"Rerun each configuration (no augmentation, ROS, VGX, and the five methods in both blind and conditioned form) at least ten times with different random seeds on the same dataset and model, and plot the F1 distributions with confidence intervals; if the intervals for Stochastic Perturbation and Random Oversampling overlap substantially, the paper's headline ordering and its recommendation to prefer ROS would not be reproduced.","tokens_in":9748,"feed_emoji":"🛡️","tokens_out":6333,"duration_ms":50280,"temperature":0.7,"pith_summary":"The paper asks whether Mixup-inspired data augmentation at the embedding level can improve deep learning-based vulnerability detection, where vulnerable code samples are rare and datasets are heavily imbalanced. It evaluates five representation-level augmentation methods — linear interpolation, stochastic perturbation, linear extrapolation, binary interpolation, and Gaussian scaling — plus conditioned variants that keep the vulnerable lines' embeddings unchanged, on a transformer-based vulnerability detector trained on a large C/C++ vulnerability dataset. The central finding is that these methods help, raising F1 by up to 9.67%, but none beats Random Oversampling, which raises F1 by 10.82%. The paper therefore recommends Random Oversampling when the goal is to balance the dataset, and identifies Stochastic Perturbation as the strongest representation-level method, especially if one keeps the original class ratio. If correct, the result redirects effort away from embedding-space interpolation toward simpler resampling and toward more intrusive code-level augmentation.","feed_headline":"Simple oversampling beats embedding augmentation for vulnerabilities","feed_subtitle":"Mixup-style embedding edits add up to 9.67% F1, yet Random Oversampling still wins by 10.82%.","key_machinery":"The central object is the generic augmentation formula $h^+ = \\alpha \\odot h + \\beta \\odot h'$, where $h$ and $h'$ are embedding vectors of vulnerable samples and $\\alpha, \\beta$ are method-specific coefficient vectors: uniform interpolation for Linear Interpolation, Bernoulli dropout scaling for Stochastic Perturbation, extrapolation outside the data hull, binary feature exchange, and Gaussian multiplicative noise. The paper's novel adaptation is conditioned augmentation: using the dataset's line-level vulnerability labels and byte-pair tokenization, it locates the tokens belonging to vulnerable lines, excludes them from the augmentation operation, and re-inserts the original vulnerable-token embeddings afterward. This mechanism is what lets the paper test whether preserving the vulnerable part of the representation, rather than allowing it to be mixed or dropped, helps the detector. The same machinery yields the explanation for Stochastic Perturbation's peculiar behaviour: in the conditioned setting the re-insertion breaks the dropout's expectation-preserving regularization, and its F1 drops from 38.10 to 37.84 while most other conditioned methods improve.","core_discovery":"On the paper's own terms, the discovery is that representation-level augmentation, previously effective for code search and code classification, transfers only weakly to vulnerability detection. The best blind method, Stochastic Perturbation, improves F1 from 34.74 to 38.10 (a 9.67% relative gain), while the conditioned variants generally outperform their blind counterparts, with Stochastic Perturbation again best at 37.84 F1. Random Oversampling — simply repeating vulnerable training samples until the classes balance — improves F1 to 38.50 (10.82% relative gain) and beats every augmentation method in both settings. The paper also finds that the vulnerability-generation baseline VGX fails to improve over no augmentation, and concludes that all five representation-level methods are preferable to VGX for large-scale balancing.","pith_inferences":["The noise explanation suggests a testable extension: augmentation methods that constrain the generated embedding to stay near the vulnerable-sample manifold, for example by projecting the mixed vector back toward the original embedding, should outperform naive stochastic perturbation if the noise story is right.","If the result generalizes across models and languages, then the practical value of embedding-level Mixup-style augmentation for vulnerability detection is limited to settings where sampling is impossible, such as privacy-preserving or continuously changing code representations.","The small F1 gaps between Stochastic Perturbation and Random Oversampling, together with the single-run protocol, imply that the ordering should be re-examined with repeated runs before being treated as a stable ranking.","The finding that conditioned augmentation helps most methods but hurts Stochastic Perturbation suggests that the interaction between augmentation and the model's own regularization, such as dropout in the classifier, deserves direct study."],"forward_implications":["Random Oversampling should be the default baseline for class-imbalance handling in deep learning-based vulnerability detection, since it outperforms all five representation-level methods despite being the simplest approach.","Stochastic Perturbation is the best representation-level choice when the class ratio is kept and only the minority class is augmented, giving the largest F1 gain among the augmentation methods.","Conditioning augmentation on vulnerable lines helps most methods, so future representation-level augmentation should exploit line-level vulnerability labels rather than augmenting blindly.","The vulnerability-generation approach VGX is not useful for large-scale balancing of real-world datasets without manual checking, since it failed to beat no augmentation in this setup.","Noise injection at the embedding level is a weak substitute for duplicating real vulnerable samples; the paper attributes this to the noise in synthetically mixed vectors acting as a regularizer that limits learning."],"supporting_citations":[{"why":"Supplies the BigVul C/C++ vulnerability dataset with line-level flaw information used in all experiments.","marker":"[9]"},{"why":"Supplies LineVul, the transformer-based token-level model the augmentation methods are evaluated on.","marker":"[11]"},{"why":"Provides the five representation-level augmentation techniques adapted here and the original code-search application.","marker":"[14]"},{"why":"Establishes Random Oversampling as a strong baseline that outperforms more complex sampling methods, the comparison the paper extends.","marker":"[29]"},{"why":"Supplies VGX, the vulnerability-generation baseline that the paper finds all five augmentation methods beat.","marker":"[20]"},{"why":"Prior work showing VGX and Vulgen fall short of ROS; this paper builds on its recommendation and comparison setup.","marker":"[5]"},{"why":"Defines the Mixup framework from which the augmentation family is derived.","marker":"[30]"}],"fun_headline_variants":["Oversampling wins over Mixup-style augmentation for vuln detection","Random oversampling tops embedding augmentation for vulnerabilities","Vuln detection: plain oversampling beats fancy embedding tricks","Mixup-style augmentation falls short vs random oversampling","Embedding augmentation adds little for vulnerability detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire comparison rests on a single training run per configuration; the gaps between Stochastic Perturbation, its conditioned variant, and Random Oversampling are small enough that unmeasured run-to-run variance could reorder the methods.","fun_headline_variants_meta":{"raw":{"variants":["Oversampling wins over Mixup-style augmentation for vuln detection","Random oversampling tops embedding augmentation for vulnerabilities","Vuln detection: plain oversampling beats fancy embedding tricks","Mixup-style augmentation falls short vs random oversampling","Embedding augmentation adds little for vulnerability detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000732,"raw_usage":{"total_tokens":3262,"prompt_tokens":918,"completion_tokens":2344,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":2267}},"tokens_in":534,"tokens_out":2344,"duration_ms":15747,"temperature":1.0,"reasoning_tokens":2267,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:21:12.122762+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun each configuration (no augmentation, ROS, VGX, and the five methods in both blind and conditioned form) at least ten times with different random seeds on the same dataset and model, and plot the F1 distributions with confidence intervals; if the intervals for Stochastic Perturbation and Random Oversampling overlap substantially, the paper's headline ordering and its recommendation to prefer ROS would not be reproduced.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the five representation-level augmentation techniques adapted here and the original code-search application."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes Random Oversampling as a strong baseline that outperforms more complex sampling methods, the comparison the paper extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior work showing VGX and Vulgen fall short of ROS; this paper builds on its recommendation and comparison setup."}],"review_version":1}