{"id":"f19e0e0c-53f5-43bf-a283-c64cb94d7214","arxiv_id":"2508.02840","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A PSO-guided knowledge distillation framework compresses a CodeBERT vulnerability assessor to 0.6% of its original size while retaining 89.3% of its accuracy.","lead":"This paper combines particle swarm optimization and knowledge distillation to shrink a CodeBERT-based vulnerability assessment model from 476 MB to 3 MB while keeping about 89% of its accuracy. The result could make automated severity scoring cheaper to deploy on low-resource devices.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The PSO fitness function in Section 3.1.2 evaluates candidate student architectures using only GFLOPs and size difference, never training them; if these proxies do not correlate with post-distillation accuracy, the claimed benefit of PSO-based architecture search is unsupported.","rationale":"The reader flagged the same weakest assumption, and I agree: the fitness function is the linchpin of the paper's claimed contribution. The abstract claims a 99.4% size reduction with 89.3% accuracy retention and faster architecture search, but if the search is effectively arbitrary because the proxy fitness is unvalidated, then the size reduction and accuracy are properties of the particular student model, not of the PSO method. The reported MCC drop (from 35.53% to 22.03%) further tempers the headline retention claim, but the primary weakness is the unvalidated fitness proxy. The paper would need to show that proxy fitness correlates with post-distillation performance, or that PSO outperforms random search, to support the central claim. Since this is an empirical question that the authors can address with additional experiments, a conditional verdict is appropriate, pending the requested validation.","tokens_in":15672,"tokens_out":3876,"duration_ms":47435,"concrete_test":"Sample 30 random student architectures from the same 13-dimensional search space defined in Section 3.1.1, distill each with the same KD protocol and hyperparameters used for the PSO-selected student, and evaluate on the same test set. If the PSO-selected student's accuracy and MCC are not above the 75th percentile of the random architectures, then the proxy fitness function does not usefully guide the search, and the 'optimization' contribution is not supported. Alternatively, replace the proxy fitness with a fitness that includes a short distillation validation (e.g., 5 epochs on a 10% subset) and compare the resulting architecture; if it differs substantially, the proxy is unreliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central novelty is PSO-guided student architecture search, but the fitness function (Section 3.1.2) combines GFLOPs and teacher-student size difference without any training or evaluation of candidate architectures. The text asserts higher GFLOPs 'generally suggests a more complex model with stronger computational capability... leading to better model performance,' but this is not established for post-distillation accuracy on this task, and the paper provides no correlation analysis between the proxy fitness and the actual accuracy or MCC of distilled students. Since Algorithm 1 returns the particle with the highest proxy fitness, the reported 3 MB student is simply the proxy-optimal configuration, not necessarily a configuration that is good or better than random search. Without validation of the proxy, the claim that PSO-KDVA 'efficiently identifies optimal model architectures' is unsupported. A related but secondary issue is that the 'state-of-the-art baselines' comparison appears to rely on a single BiLSTM baseline, which is insufficient to support the stated superiority.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PSO-KDVA, a two-stage framework for automatic severity scoring of software vulnerabilities. Stage 1 uses particle swarm optimization to search over a 13-dimensional student architecture space, with a fitness function built from GFLOPs and teacher-student model-size difference rather than from trained performance. Stage 2 distills a CodeBERT teacher into the selected student using task-specific knowledge distillation. On an enhanced MegaVul dataset of 12,071 C/C++ vulnerabilities with four CVSS v3 severity classes, the authors report compressing a 476 MB CodeBERT teacher to a 3 MB student with 89.3% of teacher accuracy, 1.7% higher accuracy than a BiLSTM baseline with 60% fewer parameters, and reductions of 72.1% in training time and 34.88% in search time relative to a genetic algorithm.","tokens_in":15934,"tokens_out":6335,"duration_ms":71353,"significance":"If the empirical claims hold, the paper offers a practically relevant result: a 3 MB model that retains most of CodeBERT's CVSS scoring accuracy would lower deployment barriers for vulnerability triage. The paper has several strengths that should be acknowledged: it targets a real application where model compression has been little studied; it reports MCC and minority-class accuracy in addition to overall accuracy, showing awareness of class imbalance; it includes sensitivity analyses for learning rate and distillation temperature; and it compares PSO search time against a genetic algorithm. The central limitations are that the PSO fitness function is an unvalidated proxy, the reported accuracy differences lack statistical support, and the baseline comparison is very narrow. These issues are fixable: adding proxy validation, error bars, and additional baselines would substantially strengthen the contribution. The paper does not contain internally inconsistent derivations; the contribution is empirical, and the main concerns are evidential rather than logical.","major_comments":[{"comment":"The fitness function that drives the architecture search is never written down. The text says it should account for GFLOPs and model-size difference, and Algorithm 1 repeatedly calls Fitness(...), but no equation, weight, or normalization is given. In addition, the proxy is unvalidated: no analysis shows that higher GFLOPs or larger size difference correlates with post-distillation accuracy or MCC on this task. Since Algorithm 1 returns the proxy-optimal particle, the reported 3 MB student is a proxy-optimal architecture, not necessarily one that is optimal or even good after distillation. Please specify the exact fitness formula and report a correlation analysis between the proxy and actual student performance, or compare the PSO-selected architecture against random search and against training the top-k proxy candidates.","section":"Section 3.1.2, Fitness Function"},{"comment":"The experimental section reports single-run point estimates without error bars, confidence intervals, or significance tests. The headline claims of 89.3% accuracy retention and 1.7% accuracy improvement over BiLSTM are differences of a few percentage points on a four-class, imbalanced problem; with no repeated seeds, it is impossible to tell whether these differences are meaningful. Please report mean and standard deviation over at least 5-10 runs with the same architecture, and state whether differences are statistically significant (e.g., paired bootstrap or a McNemar-style test).","section":"Sections 5.2 and 5.3, Experimental Results"},{"comment":"The accuracy-based headline understates the compression cost. The same table shows that the 3 MB student's MCC drops 37.99% relative to CodeBERT (35.53 to 22.03), while its accuracy drops only 10.73%. Given the highly imbalanced class distribution (e.g., Low has 238 training examples versus High's 4,454), accuracy is dominated by majority classes. The central claim should be rephrased to report MCC retention (about 62.0% of teacher MCC) alongside accuracy retention, and the minority-class results should be included in the abstract-level claims.","section":"Section 5.3, MCC Analysis"},{"comment":"The comparison that supports the claim of outperforming 'state-of-the-art baselines' consists of a single baseline, BiLSTM (Tang et al., 2019). No comparisons are made to other knowledge-distillation compression methods for code models (e.g., the 3 MB code-model compression of Shi et al., TinyBERT, or DistilBERT) or to other SVA models (e.g., DeepCVA or graph-based methods). With only one baseline, the superiority claim is not established. Please add at least one additional modern baseline and temper the wording accordingly.","section":"Section 5.2, Baseline Comparison"},{"comment":"The manuscript states that data 'will be made available on request' and does not provide a repository for the PSO-KDVA implementation, the fitness function, or the exact hyperparameters of the final student architecture. This is a reproducibility concern for an empirical systems paper. Please release the dataset and code, or a detailed configuration with all search and training settings, as part of the revision.","section":"Data Availability and Reproducibility"}],"minor_comments":[{"comment":"The manuscript contains encoding artifacts such as 'BiLSTM ����', inconsistent 'PSO-KDV A' spacing, and garbled symbols in equations and tables; these should be fixed in the camera-ready version.","section":"Throughout"},{"comment":"The phrase 'state-of-the-art baselines' overstates the single BiLSTM comparison; consider replacing it with 'a BiLSTM baseline' or adding the missing baselines.","section":"Abstract and Section 5.2"},{"comment":"The text says GFLOPs values range from 1 to 3 and model-size differences from 0 to 1, but no scaling or weighting of the two terms is reported; specify how they are combined numerically.","section":"Section 3.1.2, Fitness Function"},{"comment":"The contribution summary is repeated nearly verbatim in the abstract and the introduction; please consolidate to avoid redundancy.","section":"Introduction and Abstract"},{"comment":"The discussion of recent knowledge-distillation papers in image and multimodal tasks is tangential and not connected to the proposed SVA method; consider trimming it to keep the paper focused.","section":"Section 7, Related Work"}],"recommendation":"major_revision","confidential_remarks":"To the editor: This is an application-oriented empirical study. Its novelty is modest, as PSO for student architecture search plus knowledge distillation is a known recipe, but the empirical setting (CVSS severity scoring with compressed code models) is of interest to the journal's readership. The main risk is that the PSO fitness proxy is never validated; this is addressable in revision and is the primary reason I recommend major revision rather than rejection. I would also encourage the handling editor to require data/code release and to moderate the 'state-of-the-art' language. The related-work section cites a cluster of recent papers from one group without substantive connection to the method; please ask the authors to trim that material."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing you should know: this is a straightforward compression study. The authors distill CodeBERT into a 3 MB student for CVSS severity scoring, using PSO to search the student architecture. The headline numbers (99.4% size cut, 89.3% accuracy retention) are internally consistent, and the final model beats a BiLSTM KD baseline. But the PSO part is the weak link. The fitness function evaluates candidate architectures on GFLOPs and size difference alone, never training any candidate, and the paper provides no evidence that these proxies correlate with post-distillation accuracy. That matters because the claimed contribution is that PSO finds good architectures efficiently. Without validating the proxy, the search could be no better than random.\n\nWhat's actually new is narrow: applying KD-based compression specifically to CVSS scoring, and swapping in PSO for the genetic algorithm used in Shi et al.'s CodeBERT-to-3MB work. Each component is established, but the combination on this task is new. Credit where due: the paper reports the MCC drop (from 35.5% to 22.0% for the 3 MB student) and the minority-class degradation, which is more transparent than most compression papers.\n\nSoft spots, in order of severity. First, the unvalidated fitness proxy. The text asserts higher GFLOPs imply stronger models, but that is not demonstrated for this task, and there is no correlation analysis against actual accuracy. Training a few random or GA-selected architectures and comparing would fix this. Second, the baseline comparison is just BiLSTM from Tang et al. That is too thin to support the phrase \"state-of-the-art baselines\". They should include a fixed-architecture student without PSO, or a DistilBERT/TinyBERT-style student for code. Third, no code or dataset is released; \"available on request\" is weak for reproducibility. Fourth, the abstract frames accuracy only, while the MCC drop appears later; the paper is honest but the framing is a bit generous.\n\nThese are fixable, not fatal. The central compression result is plausible and the final student is genuinely small. With a validated proxy and broader baselines, this could be a solid engineering contribution. I would send it to peer review with the expectation of major revision. Practitioners working on edge deployment of code models will find the recipe useful; researchers will want the missing validation before relying on the PSO claim.\n\nRecommendation: engage with it, but push for the proxy validation and baseline expansion before acceptance.","headline":"A workmanlike engineering paper on compressing CodeBERT for CVSS scoring with KD and PSO, but the PSO fitness proxy is unvalidated and the 'state-of-the-art' claim rests on a single BiLSTM baseline.","tokens_in":16401,"tokens_out":2727,"would_cite":false,"duration_ms":32040,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A particle-swarm-searched student model can score vulnerability severity at 0.6% of the teacher's size while keeping 89.3% of its accuracy.","keywords":["software vulnerability assessment","knowledge distillation","particle swarm optimization","model compression","CVSS severity prediction","architecture search","CodeBERT","class imbalance"],"falsifier":"Take a random sample of architectures from the same 13-dimensional search space, train and evaluate each after distillation, and rank them by real accuracy; if the PSO-selected architecture is not near the top of that ranking, then the fitness proxy does not actually find the optimal student and the reported gains are not a systematic result.","tokens_in":15523,"feed_emoji":"🛡️","tokens_out":7402,"duration_ms":79422,"temperature":0.7,"pith_summary":"Large pre-trained code models can score vulnerability severity accurately, but they are too big and slow for real-time or embedded security tools. This paper tries to show that a much smaller model can do the job almost as well: it uses particle swarm optimization to pick the student architecture, then knowledge distillation to transfer the teacher's scoring behavior. On an enhanced MegaVul dataset of 12,071 CVSS v3 C++ vulnerabilities, the resulting 3 MB student keeps 89.3% of the 476 MB CodeBERT teacher's accuracy while cutting storage by 99.4%, and it beats a BiLSTM-based distillation baseline by 1.7% accuracy with 60% fewer parameters. The practical payoff is that severity scoring could run on resource-constrained devices where the full teacher cannot.","feed_headline":"Vulnerability scorer shrinks 476 MB to 3 MB, keeps 89% accuracy","feed_subtitle":"Particle swarm search picks the student, then distillation transfers the teacher's severity-scoring skill.","key_machinery":"The load-bearing mechanism is a fitness function that lets particle swarm optimization search without training: it combines GFLOPs, a hardware-independent proxy for computational capacity and therefore expected performance, with the size difference between teacher and student, a proxy for the capacity gap, to score each candidate architecture. PSO then updates particle positions with an inertia weight and acceleration coefficients toward individual and global bests; the winning configuration is trained by knowledge distillation with loss $L_{\\mathrm{KD}} = \\alpha L_{\\mathrm{CE}} + (1-\\alpha)T^2 L_{\\mathrm{KL}}$, so the student learns from soft teacher logits at temperature $T$ while also using ground-truth labels. This two-stage design is what turns a $476$ MB teacher into a $3$ MB student without evaluating any intermediate architecture by training.","core_discovery":"The central claim is that a compact student model for automated vulnerability assessment need not be hand-designed: a 13-dimensional hyperparameter space (tokenizer type, vocabulary size, layer count, hidden size, and similar settings) can be searched by particle swarm optimization, and the best configuration can then be distilled from CodeBERT to reach 3 MB. On the authors' version of MegaVul, PSO-KDVA retains 89.3% of the teacher's accuracy, cuts model size to 0.6% of the original, reduces training time to 27.9%, and beats the BiLSTM distillation baseline by 1.7% accuracy at 40% of that baseline's size. The paper also reports that compression hurts Matthews correlation coefficient more than accuracy, especially on the minority \"Low\" severity class, so the retained accuracy is not uniform across classes.","pith_inferences":["The paper leaves implicit that, if the GFLOPs and size-difference proxies are reliable, the same PSO-then-distill pipeline should transfer to other large code encoders beyond CodeBERT, producing similarly compact students without re-deriving a search space.","Because the fitness function never trains a candidate, swapping GFLOPs for a cheap training-free accuracy estimator would turn the search from a heuristic into an empirically verifiable optimization.","The large MCC drop points to a natural extension the paper names but does not implement: a class-weighted or focal distillation loss to protect minority-severity classes.","The enhanced MegaVul subset contains C++ only, so testing PSO-KDVA on other programming languages would reveal whether the 13-dimensional search space and the fitness balance generalize beyond C++."],"forward_implications":["A 3 MB severity-scoring model can replace a 476 MB CodeBERT teacher, so CVSS scoring could run inside IDEs, CI pipelines, or embedded scanners.","PSO-KDVA uses 60% fewer parameters than the BiLSTM distillation baseline while scoring 1.7% higher accuracy, so the particle swarm search buys real efficiency over a fixed hand-chosen student.","Training the compressed model takes 27.9% of the teacher's training time, and architecture search is 34.88% faster with PSO than with a genetic algorithm.","Compression is uneven: MCC falls 37.99% at 3 MB even though accuracy falls only 10.73%, so deployment should account for worse performance on rare severity classes."],"supporting_citations":[{"why":"Defines the teacher model, CodeBERT, with 125 million parameters and a size of about 476 MB, the baseline to be compressed.","marker":"[8]"},{"why":"Supplies the base dataset that the paper expands to 12,071 CVSS v3 C++ vulnerabilities for training and testing.","marker":"[33]"},{"why":"Provides the task-specific knowledge-distillation baseline, a BERT teacher distilled into a lighter BiLSTM student, that PSO-KDVA is compared against.","marker":"[34]"},{"why":"Establishes the 3 MB target and a prior compression result for code models, motivating the size objective.","marker":"[24]"},{"why":"Motivates using GFLOPs as a search metric for compressing code models and frames student architecture selection as combinatorial optimization.","marker":"[31]"},{"why":"Supplies the GFLOPs-based computational-cost metric used inside the PSO fitness function.","marker":"[32]"},{"why":"Provides the knowledge-distillation formulation, including the capacity gap, temperature, and soft targets, that the student training follows.","marker":"[29]"},{"why":"Defines the particle swarm velocity and position update rules that drive the architecture search.","marker":"[35]"}],"fun_headline_variants":["Vuln scanner compressed 99% via PSO and knowledge distillation","Distilled model: 3 MB, 89% accuracy, 72% faster training","Particle swarm picks student for tiny vulnerability assessor","99% smaller AI still scores software vulnerabilities well","Compact vuln model from PSO and KD retains 89.3% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method assumes that a student's GFLOPs and its size difference from the teacher predict which architecture will be most accurate after distillation, even though no candidate is ever trained during the search.","fun_headline_variants_meta":{"raw":{"variants":["Vuln scanner compressed 99% via PSO and knowledge distillation","Distilled model: 3 MB, 89% accuracy, 72% faster training","Particle swarm picks student for tiny vulnerability assessor","99% smaller AI still scores software vulnerabilities well","Compact vuln model from PSO and KD retains 89.3% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000317,"raw_usage":{"total_tokens":1796,"prompt_tokens":952,"completion_tokens":844,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":751}},"tokens_in":568,"tokens_out":844,"duration_ms":9386,"temperature":1.0,"reasoning_tokens":751,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:22:36.013832+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of architectures from the same 13-dimensional search space, train and evaluate each after distillation, and rank them by real accuracy; if the PSO-selected architecture is not near the top of that ranking, then the fitness proxy does not actually find the optimal student and the reported gains are not a systematic result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the base dataset that the paper expands to 12,071 CVSS v3 C++ vulnerabilities for training and testing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the 3 MB target and a prior compression result for code models, motivating the size objective."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates using GFLOPs as a search metric for compressing code models and frames student architecture selection as combinatorial optimization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GFLOPs-based computational-cost metric used inside the PSO fitness function."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the particle swarm velocity and position update rules that drive the architecture search."}],"review_version":1}