{"id":"126351ae-3b40-4ee4-ab1d-7542d7b1280b","arxiv_id":"2505.24451","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Linear probe accuracy on simple code metrics can guide layer pruning and roughly predict post-fine-tuning vulnerability detection performance, but several headline numbers in the abstract do not match the paper's own tables.","lead":"This paper uses small classifiers called linear probes to look inside large language models and decide which layers can be safely removed before the expensive step of fine-tuning them for code vulnerability detection. It says the approach can cut up to 72% of layers in some models with little or no performance loss, and can estimate final accuracy in minutes instead of hours.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central proxy claim — that probe accuracy on complexity features tracks downstream vulnerability detection — is never directly tested; the paper validates endpoints only, so kcut selection and early estimates rest on an unmeasured correlation.","rationale":"The paper has genuine strengths: a concrete pipeline, two model families, three datasets, released materials, and sanity checks against random pruning and over-aggressive pruning. Those checks show the chosen cutoff is not arbitrary and that pruning can preserve or improve results. However, they do not establish the paper's central predictive claim. The method's two outputs, kcut and the early performance estimate, both depend on the assumption that linear-probe accuracy on cyclomatic complexity and Halstead difficulty is a proxy for downstream vulnerability detection. This is asserted in Section 3.1 and used in Sections 4.1 and 4.2, but never validated by measuring the correlation along the layer axis. The leave-one-out beta calibration uses at most two datasets to fit each beta, and Table 7 shows substantial variation, for example Gemma CC beta from 4 to 28, so the claimed 3% to 8.68% error is not robust evidence of transferable estimation. The proposed full-cutoff sweep would directly test the proxy and settle whether the method is a principled early-assessment tool or an intercept fit on a small set of datasets. This is the same load-bearing concern the reader identified, so the conditional verdict stands.","tokens_in":21938,"tokens_out":4109,"duration_ms":52895,"concrete_test":"On one model/dataset pair, e.g., Gemma on DiverseVul, fine-tune and evaluate pruned models for a sweep of cutoffs k in {2,4,6,8,10,12,14,16} rather than only kcut and floor(kcut/2). Measure downstream multi-class F1 for each pruned model and compare it against the per-layer LP loss profile avg(acck) from Figure 3 using a rank correlation. Repeat the sweep on a second dataset, estimating beta from the remaining two datasets, and check whether the rank order of downstream F1 across k is recovered by LP loss. A significant positive monotonic relationship would support the proxy; a flat or non-monotonic relationship would invalidate the central claim that probes set the pruning cutoff and estimate post-fine-tuning performance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is introduced in Section 3.1: code-complexity features F 'can be regarded as proxies for the presence or absence of vulnerabilities.' Sections 4.1 and 4.2 then use probe accuracy accLP(D) to select the pruning cutoff kcut and to estimate downstream effectiveness via E(D) = accLP(D) + beta. This requires a predictive, ideally monotonic, relationship between per-layer probe accuracy and post-fine-tuning vulnerability-detection performance. The paper never measures that relationship. It validates only that pruning at the chosen kcut preserves or improves F1 and that removing kcut/2 layers, or random layers, is worse. Those checks show kcut is not an unreasonable choice, not that LP loss identifies the best cutoff or that accLP tracks downstream quality. Equation 3 is a constant-offset model whose beta is fitted on at most two other datasets in a three-dataset leave-one-out scheme, and Table 7 shows beta is not stable across settings (e.g., Gemma CC beta ranges from 4 to 28). Thus the 'early estimate' is a fitted intercept on the same three datasets, not a demonstrated proxy. If the probe-to-downstream correlation is weak or non-monotonic, both contributions fail: kcut selection becomes an arbitrary layer choice and the performance estimate reduces to a constant with dataset-dependent noise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LPASS, a method to guide layer pruning and to estimate the post-fine-tuning effectiveness of large language models for multi-class CWE vulnerability detection. Linear probes are trained on internal hidden states at each layer to predict the code-complexity features cyclomatic complexity (CC) and Halstead difficulty (HD). Probe accuracy is aggregated across CWEs, and the layer minimizing the accumulated loss over datasets and features (Eq. 2) is chosen as the pruning cut-off. Separately, probe accuracy on a dataset is added to a constant β, fitted on other datasets, to estimate precision, recall, and F1 (Eq. 3). The authors evaluate the approach on BERT-large and Gemma-2B over DiverseVul, Big-Vul, and PrimeVul, covering 12 CWEs, under layer pruning and quantization. The reported findings include removal of 33.3% (BERT) and 72.2% (Gemma) of layers with no precision loss, early estimates with 3% and 8.68% as lowest and average precision errors, and state-of-the-art accuracy of 86.9% in multi-class vulnerability detection.","tokens_in":22297,"tokens_out":6476,"duration_ms":77355,"significance":"If the proxy relation between probe accuracy on CC/HD and downstream vulnerability-detection performance were established, LPASS would be a practically valuable and inexpensive pre-fine-tuning diagnostic: probes are computed in seconds to minutes, compared with hours of fine-tuning, and the paper offers a concrete pipeline and a large multi-dataset evaluation. The connection of probing to model compression in a security setting is novel, and the manuscript is transparent about its experimental protocol and datasets. However, the significance is conditional on an unvalidated assumption. The paper never directly measures the correlation between probe accuracy and post-fine-tuning vulnerability detection, so the two central contributions—kcut selection and early performance estimation—currently rest on an asserted rather than demonstrated proxy relationship. In addition, several headline claims are not supported by the tables as presented.","major_comments":[{"comment":"The proxy relation is load-bearing but never directly tested. Section 3.1 states that the code features F 'can be regarded as proxies for the presence or absence of vulnerabilities,' and Section 4.1 uses probe accuracy to choose the pruning cut-off, but no experiment measures the correlation between accLP(D) and post-fine-tuning vulnerability-detection metrics. The paper only checks that pruning at kcut is not worse than baseline and that pruning at kcut/2 or random layers is worse; those checks show that kcut is a reasonable choice, not that the LP-loss curve identifies the best layer or that accLP tracks downstream quality. Please provide direct evidence, for example per-layer pruning curves against per-layer probe accuracy, or a correlation test across datasets and configurations.","section":"Section 3.1, Section 4.1, Eq. (2)"},{"comment":"The headline claims of 'no precision loss' and '1.6% of F1-score at a maximum' are contradicted by the paper's own sign convention. The text defines negative effectiveness deltas as improvements over baseline; Table 2 shows positive precision deltas for pruned BERT (e.g., Big-Vul +0.57) and pruned Gemma (e.g., DiverseVul +0.43), and the Gemma PrimeVul F1 delta is +1.6. Thus the data do not support 'no precision loss' or 'compressed versions outperform the original.' The abstract's 86.9% accuracy also does not appear as an accuracy value in Table 2 (86.9% is a baseline F1 for BERT on PrimeVul). Please correct either the claims or the interpretation of the table.","section":"Tables 2 and 3, abstract"},{"comment":"The performance estimator E(D) = accLP(D) + β is a constant-offset calibration, not a derived or validated estimator. β is fitted on at most two datasets in a three-dataset leave-one-out scheme, and Table 7 shows considerable instability across configurations (for Gemma CC, β ranges from 4 to 28). With only three datasets, the average errors in Table 8 cannot establish generalization. Please provide a sensitivity analysis of β, a proper hold-out evaluation across additional datasets, or a derivation of β that does not rely on the evaluation data.","section":"Section 4.2, Eq. (3), Table 7"},{"comment":"The cut-off kcut is selected by minimizing the sum of losses over the same datasets used for evaluation, and the corroborating tests in Table 6 and Figure 4 use the same evaluation datasets. This is selection on evaluation data, so the tests demonstrate internal consistency rather than out-of-sample validity. Please evaluate kcut on a held-out model or dataset, or report how the choice of kcut varies when different subsets of datasets are used in Eq. (2).","section":"Section 4.1, Eq. (2), Section 5.6"}],"minor_comments":[{"comment":"The method is called a linear probe, but Section 4.1 states that the probe is implemented with a Multi-Layer Perceptron; please clarify whether 'linear' refers to the classification head or to the probing methodology.","section":"Section 4.1"},{"comment":"The number of classes for CC and HD (5 and 6 classes, respectively) is based on an 85% coverage criterion that is described only informally; please report the full feature distributions and the exact rule used to set the class boundaries.","section":"Section 5.3"},{"comment":"The x-axis tick labels in Figure 3 are crowded and easy to misread, and the paper alternates between 'kcut' and 'Kcut'; please standardize the notation and improve the figure legibility.","section":"Figure 3"},{"comment":"The limitations paragraph acknowledges that the approach may not transfer to other domains and languages, but it does not acknowledge the missing direct test of the proxy assumption; please add that caveat explicitly.","section":"Section 5.7"},{"comment":"The text states 'Our experimental materials are publicly released' but the Declarations say the code 'will be freely available if the paper is accepted for publication'; these two statements should be reconciled.","section":"Section 5.4 and Declarations"},{"comment":"The state-of-the-art comparison is not controlled for token length, sampling, class balance, or evaluation protocol; please qualify the superiority claim accordingly.","section":"Section 6, Table 9"}],"recommendation":"major_revision","confidential_remarks":"The paper presents an interesting idea and a substantial experimental effort, but the central proxy assumption is not directly validated and some abstract claims appear inconsistent with the tables. I would not reject outright if the authors can add direct evidence of the probe-to-downstream correlation, correct the claim/sign discrepancies, and temper the generalization statements. The current version, however, does not yet support the stated contributions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the specific combination: using linear probes before fine-tuning to pick a layer-pruning cutoff and to estimate post-compression effectiveness for multi-class CWE vulnerability detection. I don't see that exact application in the literature, and the experimental work is solid enough that the pruning result probably holds.\n\nThe study is well run: three public C/C++ datasets, BERT and Gemma, 480k samples, 12 CWEs, comparisons against random layer removal, plus quantization. The central pruning claim is credible—removing 8/24 BERT layers and 13/18 Gemma layers at the probe-selected cutoff keeps F1 close to baseline, and random pruning is clearly worse. The time and memory savings are real and clearly reported. LP computation is cheap (142.97 s for Gemma), which makes the practical angle plausible. The related work is handled fairly; Chen et al. is cited and the difference is stated.\n\nThe problems sit in the abstract and the estimator. 'No precision loss' is contradicted by Table 2, which shows several positive deltas (e.g., BERT on DiverseVul accuracy +0.48, Gemma on PrimeVul F1 +1.6). Those are degradations, and the 1.6% is described as a maximum gain when the table shows it as a loss. The 86.9% accuracy headline doesn't clearly appear in the tables either. More substantively, Equation 3 is a constant-offset calibration: beta is fitted on at most two datasets in a three-dataset leave-one-out, and Table 7 shows beta is not stable (Gemma CC beta ranges from 4 to 28). Calling this an 'early estimate' is a stretch—it is a fitted intercept with dataset-dependent noise. The load-bearing premise, that probe accuracy on cyclomatic complexity and Halstead difficulty tracks downstream vulnerability detection, is never directly tested. The paper validates that pruning at the chosen cutoff works on these two models, but not that the probe signal identifies the best cutoff or that accLP predicts final performance. That's the main gap, and it is the load-bearing one.\n\nThe pruning result is worth taking seriously; the estimator needs substantial rework or much more evidence. The paper should not be desk-rejected. It deserves a referee who can push on the proxy question and on the abstract's claims.\n\nRecommendation: send to peer review, with the expectation of heavy revision on the estimation section and a rewritten abstract.","headline":"Useful probe-guided pruning study for vulnerability detection, but the early-estimation claim rests on an unverified proxy and a fitted constant.","tokens_in":22737,"tokens_out":2716,"would_cite":false,"duration_ms":30497,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Linear probes forecast a compressed LLM's vulnerability-detection quality before fine-tuning.","keywords":["linear probes","LLM compression","layer pruning","vulnerability detection","multi-class CWE classification","cyclomatic complexity","Halstead difficulty","early performance estimation"],"falsifier":"Take any new LLM and the same 12-CWE task, compute the probe-loss curve and the LPASS cut-off and early estimate, then actually fine-tune the model pruned at each candidate layer and measure its F1; if the true best pruning point falls far from the probe-loss minimum, or the early estimate is off by more than the reported average of about 8.7%, the central proxy claim is falsified. A lighter check is to compare probe accuracy per layer against final F1 for several pruned models and see whether the two curves track.","tokens_in":21783,"feed_emoji":"🛡️","tokens_out":8978,"duration_ms":98730,"temperature":0.7,"pith_summary":"This paper argues that cheap linear classifiers known as linear probes, trained on the hidden states of a frozen LLM, can stand in for the expensive cycle of fine-tuning and compressing that LLM for vulnerability detection. The claim is that probe accuracy on two code-complexity features, cyclomatic complexity and Halstead difficulty, tells you how many top layers can be pruned away without hurting the final multi-class vulnerability classifier, and roughly what precision, recall, and F1 that classifier will reach after fine-tuning, before any fine-tuning is done. The authors test this on BERT and Gemma over 480k C/C++ samples covering 12 of MITRE's Top 25 vulnerability types (CWEs), reporting that 8 of 24 BERT layers and 13 of 18 Gemma layers can be removed with no precision loss, and that the early estimate misses the real final metrics by between 3% and about 8.7% on average. The paper itself flags in Section 5.7 that these results are a first attempt and are not yet enough to confirm the approach outside C/C++ or in other domains.","feed_headline":"Probes forecast pruned LLM quality before fine-tuning","feed_subtitle":"Linear classifiers on hidden states pick pruning cut-offs and estimate final vulnerability-detection accuracy within about 3-9%.","key_machinery":"The central object is the linear probe plus the loss curve it induces. A linear probe is a small classifier trained on the hidden-state activations $H_k$ of a single layer $k$ to predict a simple labeled feature; here it predicts the cyclomatic-complexity class and Halstead-difficulty band of a code sample. Per-layer probe accuracy defines a loss, $\\operatorname{loss}_k = \\max_i \\operatorname{avg}(\\operatorname{acc}_i) - \\operatorname{avg}(\\operatorname{acc}_k)$, and the pruning cut-off $k_{\\mathrm{cut}}$ is the layer that minimizes the summed absolute loss across datasets and features. The same probes feed the performance estimator $\\hat{E}(D) = \\operatorname{acc}_{\\mathrm{LP}}(D) + \\beta$, with $\\beta$ calibrated by leave-one-out on other datasets. This machinery carries the whole argument: it turns 'how much can we compress?' into a question about where the model's internal representations stop adding information about code structure.","core_discovery":"The central discovery is that a per-layer accuracy curve of probes, trained only to predict code-complexity labels, marks the place where a pretrained LLM's hidden representations stop adding task-relevant information. The paper shows that the layer $k_{\\mathrm{cut}}$ minimizing the summed probe loss across datasets and features is the right place to prune: 33.3% of BERT's layers and 72.2% of Gemma's layers can be removed with no precision loss on the 12-class vulnerability detection task, and the same probe accuracies, shifted by a dataset-calibrated constant $\\beta$, estimate the final fine-tuned and compressed model's precision, recall, and F1 before fine-tuning runs. Concretely the estimator is $\\hat{E}(D) = \\operatorname{acc}_{\\mathrm{LP}}(D) + \\beta$, and the reported error of this estimate is 3% at best and 8.68% on average. Compressed LPASS models reach 86.9% accuracy in multi-class detection, and the pruned Gemma can outperform its uncompressed version by up to 1.6% F1 while saving 29.4% training time, 23.8% inference time, and 42.98% model size.","pith_inferences":["If probe accuracy is a valid proxy, the same probe-loss curve could rank layers for pruning in other code-understanding tasks, since it only requires some cheap label that correlates with task-relevant structure.","The beta-calibrated estimator suggests probe accuracy and final fine-tuned performance differ by a roughly dataset-independent constant; if that pattern generalizes, precomputed beta values per model family would make the approach a plug-in tool.","A direct test is still missing: the paper never computes the correlation between probe accuracy and final F1 across many models and layers, so measuring that correlation on a broader set of architectures would settle whether the proxy is a genuine regularity or a two-model coincidence.","Because the cut-off is chosen at the minimum of probe loss, the method implies that layers after $k_{\\mathrm{cut}}$ are not merely removable but actively unhelpful, which could be tested by ablating only post-cut-off layers and observing whether internal task-relevant representations degrade."],"forward_implications":["Layer pruning guided by probes removes a third to nearly three-quarters of layers with no measured precision loss on the 12-CWE task, so much of the model depth is redundant for this specific vulnerability detection task.","Probe-based early estimates of final precision, recall, and F1 come within 3% at best and 8.68% on average, letting a developer decide before investing in fine-tuning whether a model is worth compressing.","LPASS-based compressed models outperform the state of the art on multi-class vulnerability detection, reaching 86.9% accuracy, and can beat their own uncompressed versions.","The probe computation is cheap: 142.97 seconds for Gemma and under a hundredth of a second per sample for feature extraction, a reduction of about 99% relative to fine-tuning time, so the decision procedure is affordable on consumer GPUs."],"supporting_citations":[{"why":"Introduces linear classifier probes as a way to read what a network's intermediate layers encode, the method LPASS adapts.","marker":"[23]"},{"why":"The closest prior effort, pruning layers by feature representations in vision models; LPASS extends the idea to pretrained LLMs before fine-tuning and replaces the criterion with probe accuracy.","marker":"[22]"},{"why":"Provides the random layer-removal baseline used to corroborate that the probe-chosen cut-off kcut beats arbitrary pruning.","marker":"[50]"},{"why":"Contributes the DiverseVul dataset of vulnerable and non-vulnerable C/C++ functions used for training and evaluation.","marker":"[1]"},{"why":"Contributes the Big-Vul dataset of vulnerability-fixing commits with CVE summaries, one of the three evaluation datasets.","marker":"[30]"},{"why":"Contributes the PrimeVul dataset, merging four vulnerability datasets including the other two, used as the third evaluation set.","marker":"[3]"},{"why":"Defines BERT, one of the two LLMs whose hidden states are probed and pruned.","marker":"[27]"},{"why":"Defines Gemma, the second and larger LLM used in the study.","marker":"[28]"},{"why":"Defines cyclomatic complexity, one of the two code features the probes are trained to predict.","marker":"[46]"}],"fun_headline_variants":["Probes pick pruning cuts; BERT and Gemma lose no precision","Linear probe loss marks the right layer to prune in LLMs","LPASS: estimate compressed LLM accuracy before fine-tuning","Probe-based compression cuts Gemma 72% layers, 43% size","Early probe accuracy forecasts post-fine-tuning vulnerability detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the accuracy of a linear probe trained to guess cyclomatic complexity and Halstead difficulty from internal activations tracks how well the model will later detect specific vulnerability types after fine-tuning; the paper does not measure that correlation directly, and Section 5.7 says the results are a first attempt not yet confirmed beyond C/C++.","fun_headline_variants_meta":{"raw":{"variants":["Probes pick pruning cuts; BERT and Gemma lose no precision","Linear probe loss marks the right layer to prune in LLMs","LPASS: estimate compressed LLM accuracy before fine-tuning","Probe-based compression cuts Gemma 72% layers, 43% size","Early probe accuracy forecasts post-fine-tuning vulnerability detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000615,"raw_usage":{"total_tokens":2933,"prompt_tokens":1098,"completion_tokens":1835,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":714,"completion_tokens_details":{"reasoning_tokens":1746}},"tokens_in":714,"tokens_out":1835,"duration_ms":17818,"temperature":1.0,"reasoning_tokens":1746,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:22:46.460798+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any new LLM and the same 12-CWE task, compute the probe-loss curve and the LPASS cut-off and early estimate, then actually fine-tune the model pruned at each candidate layer and measure its F1; if the true best pruning point falls far from the probe-loss minimum, or the early estimate is off by more than the reported average of about 8.7%, the central proxy claim is falsified. A lighter check is to compare probe accuracy per layer against final F1 for several pruned models and see whether the two curves track.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The closest prior effort, pruning layers by feature representations in vision models; LPASS extends the idea to pretrained LLMs before fine-tuning and replaces the criterion with probe accuracy."},{"cited_title":"Sajjad, F","cited_arxiv_id":null,"evidence_quote":"Provides the random layer-removal baseline used to corroborate that the probe-chosen cut-off kcut beats arbitrary pruning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the DiverseVul dataset of vulnerable and non-vulnerable C/C++ functions used for training and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the Big-Vul dataset of vulnerability-fixing commits with CVE summaries, one of the three evaluation datasets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the PrimeVul dataset, merging four vulnerability datasets including the other two, used as the third evaluation set."},{"cited_title":"Ebert, J","cited_arxiv_id":null,"evidence_quote":"Defines cyclomatic complexity, one of the two code features the probes are trained to predict."}],"review_version":1}