{"id":"e488647b-468a-4e73-83b5-243f35686dc8","arxiv_id":"2509.00096","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"LLM pruning damages the internal features used to detect false statements; the proposed TPLO method reallocates sparsity to protect them, though the measured improvements are modest and potentially confounded by leakage from the evaluation data.","lead":"Pruning large language models weakens the internal activation signals that lie-detection probes rely on, and the paper proposes a layer-wise sparsity scheme, TPLO, to protect those signals. The approach is tested on three small instruction-tuned models, but the reported gains over existing pruning methods are small and may be inflated by test data leakage.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim is at risk because test labels enter both layer selection and TPLO's sparsity allocation; holdout leakage may explain the reported gains over Wanda and OWL.","rationale":"The reader's weakest assumption identifies the same load-bearing weakness: the layer-wise separability used to construct both layer selection and TPLO's sparsity allocation is computed on the same labeled datasets that are later held out for testing. My reading of Sections 3.2, 4.1, and 5.2 confirms that the leave-one-topic-out protocol does not exclude the test topic from the LSD/SWL computation, so the pruning mask is label-informed. The reported improvements over OWL are also extremely small relative to the standard deviations, and the paper itself notes that ETruthQA 'sometimes improves performance, but not consistently.' Because the central claim depends on these comparisons, the rejection is appropriate; however, this is a corrigible evaluation flaw, and a strict leave-one-topic-out or fully unseen-topic evaluation could settle whether TPLO has genuine value.","tokens_in":21563,"tokens_out":3521,"duration_ms":36024,"concrete_test":"Re-run the full TPLO pipeline under strict leave-one-topic-out: for each of the 12 true/false topics, compute the LSD separability and the resulting SWL/TPLO sparsity allocations using only the remaining 11 topics' activations, then evaluate probing accuracy on the held-out topic. Average the 12 fold accuracies for TPLO and compare with the same leave-one-topic-out treatment of Wanda and OWL. If TPLO no longer consistently outperforms Wanda or OWL (or if the layer chosen as 'highest separability' shifts across folds), the reported advantage is attributable to test-label leakage rather than to preserving truthfulness features. Also report the same comparison with the layer fixed at layer 12 as in the original paper, to separate layer-selection leakage from sparsity-allocation leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's main quantitative claim, that TPLO preserves lie-detection features better than uniform pruning (Wanda) or OWL, rests on an evaluation in which the same labeled true/false statements used for testing are also used to construct the pruning allocation. In Section 3.2, the layer-wise separability (LSD) used to select layer 12 is computed from activations of true and false statements 'shown in Figure 1,' with no indication that held-out test topics are excluded. In Section 4.1, TPLO's layer-wise sparsity is initialized as a copy of SWL's distribution, and SWL is directly defined from the LSD separability values in Figure 1. Since Section 5.2 evaluates via leave-one-topic-out on the same 12 datasets, the labels of the held-out topic can influence: (i) which layer is chosen for probing, and (ii) how much sparsity each layer receives. If pruning shifts which layers carry discriminative truth information, then TPLO's allocation is fitted to the test distribution rather than to a general principle about truth features. The comparison to OWL is especially fragile: in Table 1, TPLO-TTPD with C4 (0.8788) differs from OWL-TTPD with C4 (0.8776) by only 0.0012, well within the reported standard deviations, and no significance tests are provided. The ETruthQA calibration data also sometimes hurts performance (e.g., Wanda-LR average drops from 0.7782 to 0.7835? actually 0.7835 is higher; but SWL-LR drops from 0.7751 to 0.7691, and TPLO-LR drops from 0.8083 to 0.8016), so the contribution of the proposed calibration set is inconsistent. The combination of test-label leakage into hyperparameter selection and statistically unsupported margins means the central claim of improved lie detection is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies how unstructured pruning at 50% sparsity affects the internal activation features used by probing classifiers to distinguish true from false statements in LLMs. The authors define a layer-wise separability (LSD) metric, observe that layer 12 of LLaMA3.1-8B-Instruct has the highest separability, and propose a baseline SWL that allocates sparsity inversely to separability, followed by TPLO that combines SWL's allocation with OWL's outlier-based layer-wise densities. They also introduce ETruthQA, a GPT-4o-enriched version of TruthfulQA, as calibration data. Experiments with LR, CCS, MM, and TTPD probes on true/false datasets and on TruthfulQA report that TPLO preserves lie-detection accuracy (about 88% at 50% sparsity) better than Wanda and OWL while keeping perplexity and general-task performance comparable.","tokens_in":22049,"tokens_out":5990,"duration_ms":55213,"significance":"If correct, this would be a useful finding: it would show that pruning damages internal truthfulness features in a layer-dependent way and that a simple layer-wise allocation can mitigate the damage. The paper includes several strengths: experiments on three model families, four probing techniques, integration with DoLa, and released code/data. However, the evaluation design contains a training/test leakage in the choice of probe layer and sparsity allocation, and the main comparison to OWL is within one standard deviation, so the central claim is not currently supported. The contribution is potentially interesting but needs a substantially cleaner evaluation before it can be assessed.","major_comments":[{"comment":"Test labels leak into the choice of probe layer and into the layer-wise sparsity allocation. LSD is computed in Figure 1 from activations of all true/false statements, and the same statements are later used in the leave-one-topic-out evaluation described in Section 5.2. Thus the held-out topic influences which layer is probed (layer 12 is selected by separability) and how TPLO/SWL distribute sparsity across layers (SWL is defined directly from the LSD values, and TPLO initializes from SWL). Please re-run the evaluation with layer selection and sparsity allocation derived only from the training topics, or from an independent truthfulness corpus, and report per-topic results.","section":"Section 3.2, Section 4.1, Section 5.2"},{"comment":"The claimed significant improvement over OWL is not supported by the reported numbers: TPLO-TTPD with C4 (0.8788) differs from OWL-TTPD with C4 (0.8776) by 0.0012, with overlapping standard deviations (0.0044 vs 0.0042); similar overlaps appear in several LR, CCS, and MM rows. No significance tests are provided anywhere in the table or the text. Please add paired significance tests across repeated runs, or clearly state which comparisons are statistically meaningful.","section":"Table 1"},{"comment":"The ETruthQA calibration data is generated by enriching TruthfulQA with GPT-4o, and TruthfulQA is also used as the evaluation benchmark in Tables 6 and 7. This creates a second leakage path for the TruthfulQA results, because the calibration data is derived from the same benchmark on which the pruned models are later tested. Please use a calibration source disjoint from the evaluation benchmark, or evaluate on a held-out split of TruthfulQA that is never used for enrichment.","section":"Section 4.2, Tables 6 and 7"},{"comment":"Several free choices are made with access to the full true/false dataset: the lambda scaling factor for SWL, the number of early layers replaced by OWL values (10, 12, 12 for the three models), the outlier ratio threshold M=5, and the probe layer index. The manuscript provides no sensitivity analysis or validation-based selection for these hyperparameters, so it is unclear how much of TPLO's advantage comes from test-set fitting. Please report sensitivity to these choices and justify them from training data only.","section":"Section 3.2, Section 4.1"}],"minor_comments":[{"comment":"The text says the probe techniques were introduced in Subsection 4.2, but the probes are described in Section 5.2; please fix the cross-reference.","section":"Section 5.2"},{"comment":"There is a typo in the open-ended generation section: 'we employ OpenAI's GPT-4o API to to assess' should read 'to assess'.","section":"Appendix C"},{"comment":"The table header uses 'enrichedTruthQA' in the calibration-data column, while the rest of the paper uses 'ETruthQA'; please make the terminology consistent.","section":"Table 5"},{"comment":"The average column is described as the average over 12 true/false datasets, but the table only shows five dataset columns; please show all 12 datasets or explain why the others are omitted.","section":"Table 1"},{"comment":"The footnote says the number of early layers is chosen 'based on Figure 3, 10 and 11 respectively'; this should read 'based on Figures 3, 10, and 11' to avoid ambiguity.","section":"Section 4.1 footnote"},{"comment":"The caption says 'across several true false datasets'; add a slash or hyphen for readability ('true/false datasets').","section":"Figure 2 caption"}],"recommendation":"major_revision","confidential_remarks":"The leakage in Sections 3.2 and 4.1 is the main barrier to acceptance. If the authors can re-run the experiments with proper holdout separation and establish that TPLO beats OWL beyond noise, the paper could be salvageable, but I would not accept the current version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the interesting bit: the paper asks whether pruning destroys the internal features that let you detect when an LLM is lying, and that question is genuinely under-studied. The compression literature mostly checks bias, toxicity, fairness; nobody had looked at lie-detection probes. The empirical observation that 50% Wanda pruning degrades probing accuracy across several true/false datasets is new and worth knowing. The authors also test three model families and four probe methods, and they don't hide that their ETruthQA calibration sometimes hurts. That is honest.\n\nNow the soft spots. The main evaluation leaks test information into the method. The layer-wise sparsity allocation (SWL) is built directly from LSD separability measured on the same labeled true/false statements that later appear in the leave-one-topic-out test. TPLO inherits SWL's distribution for the middle layers. So the pruning ratios are effectively fit to the test distribution. If pruning reshuffles which layers carry truth information, the reported gains over Wanda and OWL are partly artifacts. Layer 12 is also selected using those same activations, though the authors do cite Bürger et al. for that choice, so the damage there is smaller. This is not a minor detail; it is the load-bearing part of the causal story.\n\nThe second problem is statistical. In Table 1, TPLO-TTPD with C4 scores 0.8788 versus OWL-TTPD's 0.8776. That difference is one tenth of a standard deviation. The paper claims 'significantly outperforms' without any significance test. The gap over Wanda is larger (about 0.023), but that comparison is confounded by the leakage. The ETruthQA improvement is inconsistent, and on TruthfulQA itself the gains over C4-only calibration are small and sometimes negative. So the practical contribution of the proposed calibration set is not established.\n\nNone of this kills the paper's motivation. The underlying phenomenon is likely real. But as written, the central quantitative claim—that TPLO preserves lie detection better than standard pruning—rests on an evaluation design that cannot distinguish a genuine effect from fitting to the test topics. A revision with held-out hyperparameter selection, a sparsity allocation learned only from training topics, and proper significance testing would be necessary.\n\nWho this is for: people working on compression and safety, and anyone deploying pruned models where honesty matters. The paper deserves a serious referee; the problem is important and the observation is novel. I'd send it to review, but I would not accept it in its current form, and I would expect the authors to redo the evaluation.","headline":"A genuinely new observation about pruning and lie detection, but the evaluation leaks test labels into the method and the main gain over OWL is within noise.","tokens_in":22508,"tokens_out":3456,"would_cite":true,"duration_ms":33311,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Unstructured pruning at 50% sparsity can keep LLM lie detection near 88% accuracy.","keywords":["LLM pruning","hallucination detection","lie detection","probing classifiers","layer-wise sparsity","activation outliers","truthfulness","model compression"],"falsifier":"Recompute TPLO's layer-12 choice and layer-wise sparsity using only the training topics, then evaluate on the held-out topics; if the gap over uniform pruning disappears, the 88% figure is fitted to the test distribution rather than a general effect.","tokens_in":21331,"feed_emoji":"✂️","tokens_out":4891,"duration_ms":39177,"temperature":0.7,"pith_summary":"The paper tries to show that pruning a large language model does not have to destroy its ability to tell true statements from false ones, a capability that lives in the model's internal activation vectors. It reports that standard unstructured pruning at 50% sparsity measurably degrades those activation features, and it proposes a pruning scheme, TPLO, that allocates sparsity unevenly across layers based on where true/false information is most separable and where activation outliers live. If the claim holds, compressed models deployed on edge devices could still be monitored for hallucination, and pruning budgets could be set without silently giving up truthfulness.","feed_headline":"At 50% sparsity, pruned LLMs still catch lies at 88%","feed_subtitle":"A layer-wise pruning scheme preserves the inner activations that probing classifiers use to spot hallucinations.","key_machinery":"The load-bearing object is the Layer-wise Separability of True and False Distribution (LSD): for each layer, the ratio of between-class variance to within-class variance of the final-token residual-stream activations for true versus false statements. It identifies layer 12 in LLaMA3.1-8B-Instruct as the most discriminative layer and supplies the SWL sparsity prior; TPLO then blends that prior with OWL's layer-wise outlier ratios so that layers rich in both discriminative features and activation outliers are pruned least.","core_discovery":"The paper's central discovery is that lie-detection features in LLMs are not uniformly distributed across layers and that pruning destroys them at different rates in different layers. It defines a per-layer separability score (LSD) from the ratio of between-class to within-class variance of activations for true and false statements, shows that moderate pruning keeps it while heavy pruning degrades it, and then constructs TPLO by combining the separability-weighted sparsity assignment (SWL) with the outlier-ratio-weighted assignment (OWL). The result is a 50%-sparse LLaMA3.1-8B-Instruct whose probing classifiers still reach roughly 88% accuracy on twelve true/false datasets, close to the unpruned model and clearly above uniform Wanda pruning.","pith_inferences":["Because TPLO's layer selection and sparsity allocation are computed from the same true/false statement distribution used for evaluation, its measured advantage over uniform pruning may shrink when applied to statement types whose discriminative layer is different; a held-out-topic test would clarify the size of the real gain.","The same LSD-plus-outlier alignment idea could be transferred to other safety-relevant internal features, such as bias or refusal directions, where pruning currently risks silent degradation.","TPLO's reliance on probing a single layer suggests that multi-layer or subspace-based probes might recover even more of the unpruned model's lie-detection accuracy at higher sparsities."],"forward_implications":["A 50% unstructured-pruned LLaMA3.1-8B-Instruct can retain roughly 88% probing accuracy for true/false statements, close to the unpruned model's accuracy.","Uniform pruning with Wanda degrades lie-detection features more than layer-wise allocation does; simply re-weighting by separability (SWL) is not enough and can hurt.","TPLO preserves general zero-shot performance and perplexity on par with Wanda while improving truthfulness probing.","Adding the enriched TruthfulQA calibration set (ETruthQA) moderately helps TruthfulQA multiple-choice and open-ended generation scores, and DoLa decoding can be combined with TPLO for further gains."],"supporting_citations":[{"why":"Supplies the true/false statement datasets, the global truth-direction setup, the TTPD probe, and the layer-12 activation choice this paper builds on.","marker":"Bürger et al. (2024)"},{"why":"Provides Wanda, the base unstructured pruning method whose 50% sparsity defines the uniform-pruning baseline.","marker":"Sun et al. (2023)"},{"why":"Provides OWL outlier-ratio distributions that TPLO aligns with the separability prior.","marker":"Yin et al. (2023)"},{"why":"Sets the calibration-data protocol, the 50% sparsity point, and the general-task evaluation suite used in the experiments.","marker":"Bandari et al. (2024)"},{"why":"Supplies TruthfulQA, the benchmark the paper enriches into ETruthQA and uses for truthfulness evaluation.","marker":"Lin et al. (2021)"},{"why":"Shows that internal activations support truthfulness probing, motivating the logistic-regression probe setup.","marker":"Azaria and Mitchell (2023)"},{"why":"Provides CCS, one of the unsupervised probing baselines the paper evaluates.","marker":"Burns et al. (2022)"},{"why":"Supplies DoLa, the decoding intervention shown to further improve pruned models' truthful generations.","marker":"Chuang et al. (2023)"}],"fun_headline_variants":["Prune weights, keep truth: 88% lie detection at 50% sparsity","Layer-wise pruning preserves LLM's lie-spotting ability","Half the model, full honesty: TPLO keeps 88% accuracy","New method prunes LLMs without killing their truth detectors","Even at 50% sparsity, pruned LLMs catch lies at 88%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's sparsity plan is chosen using the separability of true and false statements measured on the original, unpruned model, and if pruning reorders which layers actually carry that information, the reported gains over uniform pruning would shrink or vanish.","fun_headline_variants_meta":{"raw":{"variants":["Prune weights, keep truth: 88% lie detection at 50% sparsity","Layer-wise pruning preserves LLM's lie-spotting ability","Half the model, full honesty: TPLO keeps 88% accuracy","New method prunes LLMs without killing their truth detectors","Even at 50% sparsity, pruned LLMs catch lies at 88%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0007,"raw_usage":{"total_tokens":3151,"prompt_tokens":928,"completion_tokens":2223,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":2123}},"tokens_in":544,"tokens_out":2223,"duration_ms":14621,"temperature":1.0,"reasoning_tokens":2123,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:48:55.577062+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute TPLO's layer-12 choice and layer-wise sparsity using only the training topics, then evaluate on the held-out topics; if the gap over uniform pruning disappears, the 88% figure is fitted to the test distribution rather than a general effect.","supporting_citations":[],"review_version":1}