Pith. sign in

REVIEW 3 minor 1 cited by

Pruned LLMs can still pick the right multiple-choice answer while failing to generate it in open output.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-27 01:08 UTC pith:OYRGHDZ6

load-bearing objection Pruning creates a generation vs recognition gap on identical questions, with answers demoted rather than lost.

arxiv 2606.17609 v1 pith:OYRGHDZ6 submitted 2026-06-16 cs.CL

The Benchmark Illusion: Pruned LLMs Can Pass Multiple Choice but Fail to Answer

Rui Wen , Lu Sun , Jiayang Liu , Zesheng Xu , Tianshuo Cong , Zheng Li This is my paper
classification cs.CL
keywords pruned LLMsmultiple choice evaluationopen generationmodel compressionbenchmark illusionWanda pruningQA evaluationrecognition versus generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tracks identical multilingual QA questions before and after pruning and finds that high-sparsity methods, especially Wanda, produce many cases where greedy generation yields the wrong answer yet multiple-choice scoring still selects the correct one. In these recognition-only failures the correct answer is typically demoted in the output distribution rather than erased, so it often reappears under beam search, sampling, or a single in-context example. The result is a benchmark illusion: standard multiple-choice tests can overstate how usable a compressed model actually is for tasks that require producing the answer. The authors conclude that compressed models must be evaluated on what they generate, not only on what they can recognize.

Core claim

Under high-sparsity pruning, models frequently fail in greedy open generation on the same questions they still answer correctly under multiple-choice scoring; the correct answer is usually not removed from the model but demoted in rank, and can be recovered by changing the decoding strategy or adding one in-context example.

What carries the argument

Tracking performance on the exact same questions before and after pruning, comparing greedy open generation against multiple-choice selection to isolate demotion versus erasure of answers.

Load-bearing premise

That comparing the same questions before and after pruning cleanly isolates the effect of pruning on generation versus recognition.

What would settle it

Run the identical pruned model on the same QA items and measure whether the rank of the correct answer in the full output distribution drops after pruning while the multiple-choice accuracy stays high.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Multiple-choice benchmarks can overstate the practical performance of heavily pruned models.
  • Pruned models should be tested on open generation rather than recognition alone.
  • The correct answer often remains recoverable with beam search or light prompting after pruning.
  • Wanda pruning at high sparsity produces more of these recognition-only errors than other methods examined.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The pattern suggests pruning primarily disrupts output ranking rather than removing stored knowledge.
  • Real-world deployment of compressed models may need generation-specific checks even when MCQ scores look acceptable.
  • Similar demotion effects could appear under other compression techniques that alter token probabilities.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 3 minor

Summary. The paper claims that high-sparsity pruning (especially Wanda) induces a 'benchmark illusion' in LLMs: models fail greedy open-ended generation on multilingual QA questions while still selecting the correct answer under multiple-choice scoring. The correct answer is typically demoted rather than erased, as it can be recovered via beam search, sampling, or one-shot ICL. The design tracks identical questions pre- and post-pruning to isolate the effect, concluding that MC benchmarks overstate usability of compressed models and that open-generation tests are needed.

Significance. If the empirical results hold, the work identifies a practically important evaluation gap for compressed LLMs. The pre/post tracking on identical questions, combined with recovery experiments distinguishing demotion from erasure, provides direct evidence that standard MC benchmarks can mask production failures. This is a clear, falsifiable contribution to the compression and evaluation literature.

minor comments (3)
  1. [Experimental setup] The multilingual QA setup is integral to the design, but the manuscript should explicitly discuss whether language-specific tokenization or script differences could interact with pruning in ways that affect generation vs. recognition (e.g., in the experimental setup section).
  2. [§4] Clarify the exact decoding parameters (temperature, top-p, beam width) used in the recovery experiments and whether they were held constant across all models and sparsity levels.
  3. [Figures 2-4] Figure captions and axis labels should explicitly state the number of questions per language and the exact pruning ratios tested to allow direct replication.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive assessment of the paper's significance and the recommendation for minor revision. No specific major comments were listed in the report.

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper is a direct empirical comparison of MC vs. open-generation performance on identical questions before/after pruning, with recovery experiments via beam search, sampling, and ICL. No equations, derivations, fitted parameters, or load-bearing self-citations appear; the design isolates pruning effects through pre/post tracking without reducing any claim to its own inputs by construction. This is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Empirical observation paper with no mathematical model, free parameters, axioms, or invented entities.

pith-pipeline@v0.9.1-grok · 5718 in / 955 out tokens · 29729 ms · 2026-06-27T01:08:33.336909+00:00 · methodology

0 comments
read the original abstract

Compressing large language models reduces memory use and inference cost, but it can also create failures that standard benchmarks miss. A pruned model may still perform well on multiple-choice evaluations, yet fail to answer the same question in open generation. We ask what pruning changes: does it erase the correct answer, or does it make the answer harder to produce as the top output? We study this question with multilingual question answering, tracking the same questions before and after pruning. We find a benchmark illusion. Under high-sparsity pruning, especially Wanda, models often fail in greedy open generation while still selecting the correct answer under multiple-choice scoring. In these recognition-only errors, the answer is usually not gone, but demoted: it often reappears with beam search, sampling, or one in-context example. Overall, multiple-choice benchmarks can overstate the usability of compressed LLMs, creating an evaluation blind spot. Compressed models should be tested on what they can produce, not only on what they can recognize.

Figures

Figures reproduced from arXiv: 2606.17609 by Jiayang Liu, Lu Sun, Rui Wen, Tianshuo Cong, Zesheng Xu, Zheng Li.

Figure 1
Figure 1. Figure 1: Many producible answers become recognition-only after pruning. The pruned model fails in greedy open generation but still selects the gold answer from candidates. The full four-way fate break￾down is shown in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Pruning demotes gold answers rather than erasing them. On both-correct items, recognition-only answers stay near the top of the next-token distribution after pruning: the gold first token has a median rank of 3.5, which is close enough to remain reachable but no longer selected by greedy decoding. distractors-only, and 32.1% with unrelated candi￾dates. By contrast, when the prompt includes the gold answer … view at source ↗
Figure 4
Figure 4. Figure 4: Missed answers remain reachable from the distribution. Pruning lowers greedy top-1 accuracy but barely changes whether the answer appears under broader decoding. Among items where pruned greedy decoding fails, beam-5 and 10-sample recovery is substantial for several languages but low for low resource languages. result comes from the MC-only items: when prun￾ing breaks open generation but preserves multiple… view at source ↗
Figure 5
Figure 5. Figure 5: The dissociation generalizes across models and datasets. For two additional model families and a second dataset (XQuAD), the recognition-only error rate remains substantial, and the production-recognition gap ∆PR is positive in every setting. Qwen3-8B has a recognition-only rate of 31.9% and ∆PR = +0.16, close to its TyDiQA result of 28.1% and ∆PR = +0.14. Across all four settings, ∆PR is positive, meaning… view at source ↗
Figure 7
Figure 7. Figure 7: Calibration lifts both formats but leaves the gap. Mean open-generation and MC accuracy under English-only vs. balanced multilingual calibration (9 languages). pruning process exclusively on English text. If the pruning mask simply discards the “wrong” lan￾guages, then switching to a balanced, multilingual calibration set should preferentially rescue open￾ended generation and close the gap. We find that ca… view at source ↗
Figure 8
Figure 8. Figure 8: Full four-way fate of uncompressed-both￾correct items after pruning (WANDA sp = 0.5), the breakdown summarized by [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation

    cs.LG 2026-07 conditional novelty 5.0

    A repetition-gated, short-to-long rollout schedule lets on-policy distillation recover pruned LLMs' generation quality with 71% fewer rollout tokens than a fixed 8192-token horizon, matching its score within two points.

Reference graph

Works this paper leans on

300 extracted references · cited by 1 Pith paper

  1. [1]

    Ahmed Salem and Yang Zhang and Mathias Humbert and Pascal Berrang and Mario Fritz and Michael Backes , title =

  2. [2]

    Inken Hagestedt and Yang Zhang and Mathias Humbert and Pascal Berrang and Haixu Tang and XiaoFeng Wang and Michael Backes , title =

  3. [3]

    Michael Backes and Mathias Humbert and Jun Pang and Yang Zhang , title =

  4. [4]

    Yang Zhang and Mathias Humbert and Tahleen Rahman and Cheng-Te Li and Jun Pang and Michael Backes , title =

  5. [5]

    Pascal Berrang and Mathias Humbert and Yang Zhang and Irina Lehmann and Roland Eils and Michael Backes , title =

  6. [6]

    Michael Backes and Pascal Berrang and Mathias Humbert and Praveen Manoharan , title =

  7. [7]

    Brendan McMahan and Sarvar Patel and Daniel Ramage and Aaron Segal and Karn Seth , title =

    Keith Bonawitz and Vladimir Ivanov and Ben Kreuter and Antonio Marcedone and H. Brendan McMahan and Sarvar Patel and Daniel Ramage and Aaron Segal and Karn Seth , title =

  8. [8]

    Raphael Bost and Raluca Ada Popa and Stephen Tu and Shafi Goldwasser , title =

  9. [9]

    Nicholas Carlini and David Wagner , title =

  10. [10]

    David Evans and Yan Huang and Jonathan Katz and Lior Malka , title =

  11. [11]

    Matt Fredrikson and Eric Lantz and Somesh Jha and Simon Lin and David Page and Thomas Ristenpart , title =

  12. [12]

    Jinyuan Jia and Neil Zhenqiang Gong , title =

  13. [13]

    Apostolos Pyrgelis and Carmela Troncoso and Emiliano De Cristofaro , title =

  14. [14]

    Reza Shokri and Marco Stronati and Congzheng Song and Vitaly Shmatikov , title =

  15. [15]

    Yuxiao Dong and Jie Tang and Sen Wu and Jilei Tian and Nittiest V Chawla and Jinghai Rao and Huanhuan Cao , title =

  16. [16]

    Aditya Grover and Jure Leskovec , title =

  17. [17]

    Simon Oya and Carmela Troncoso and Fernando P

  18. [18]

    Yelena Mejova and Sofiane Abbar and Hamed Haddadi , title =

  19. [19]

    Flavio Souza and Diego de Las Casas and Vinicius Flores and SunBum Youn and Meeyoung Cha and Daniele Quercia and Virgilio Almeida , title =

  20. [20]

    Myers and Jure Leskovec , title =

    Eunjoon Cho and Seth A. Myers and Jure Leskovec , title =

  21. [21]

    Beatrice Perez and Mirco Musolesi and Gianluca Stringhini , title =

  22. [22]

    Tomas Mikolov and Kai Chen and Greg Corrado and Jeffrey Dean , title =

  23. [23]

    Corrado and Jeffrey Dean , title =

    Tomas Mikolov and Ilya Sutskever and Kai Chen and Greg S. Corrado and Jeffrey Dean , title =

  24. [24]

    Jun Pang and Yang Zhang , title =

  25. [25]

    Reza Shokri and Georgios Theodorakopoulos and Jean-Yves Le Boudec and Jean-Pierre Hubaux , title =

  26. [26]

    Hamilton and Jure Leskovec and Dan Jurafsky , title =

    William L. Hamilton and Jure Leskovec and Dan Jurafsky , title =

  27. [27]

    Bryan Perozzi and Rami Al-Rfou and Steven Skiena , title =

  28. [28]

    Dmitry David and Oren Tsur and Ari Rappoport , title =

  29. [29]

    Romero and Brendan Meeder and Jon Kleinberg , title =

    Daniel M. Romero and Brendan Meeder and Jon Kleinberg , title =

  30. [30]

    Essa , title =

    Julia Deeb-Swihart and Christopher Polack and Eric Gilbert and Irfan A. Essa , title =

  31. [31]

    Raphael Ottoni and Diego Las Casas and Joao Paulo Pesce and Wagner Meira Jr and Christo Wilson and Alan Mislove and Virgilio Almeida , title =

  32. [32]

    Gummadi , title =

    Parantapa Bhattacharya and Muhammad Bilal Zafar and Niloy Ganguly and Saptarshi Ghosh and Krishna P. Gummadi , title =

  33. [33]

    Kiran Garimella and Gianmarco De Francisci Morales and Aristides Gionis and Michael Mathioudakis , title =

  34. [34]

    Conover and Mark Meiss and Bruno Goncalves and Alessandro Flammini and Filippo Menczer , title =

    Jacob Ratkiewicz and Michael D. Conover and Mark Meiss and Bruno Goncalves and Alessandro Flammini and Filippo Menczer , title =

  35. [35]

    Janette Lehmann and Bruno Goncalves and Jos

  36. [36]

    Salvatore Scellato and Anastasios Noulas and Cecilia Mascolo , title =

  37. [37]

    Ruth Garcia Gavilanes and Neil O'Hare and Luca Maria Aiello and Alejandro Jaimes , title =

  38. [38]

    Daniele Quercia and Luca Maria Aiello and Rossano Schifanella and Adam Davies , title =

  39. [39]

    Kevin Tang and Manohar Paluri and Fei-Fei Li and Rob Fergus and Lubomir Bourdev , title =

  40. [40]

    Shih and Dongwon Lee , title =

    Jin Yea Jang and Kyungsik Han and Patrick C. Shih and Dongwon Lee , title =

  41. [41]

    Jian Tang and Meng Qu and Mingzhe Wang and Ming Zhang and Jun Yan and Qiaozhu Mei , title =

  42. [42]

    Zhao and Haitao Zheng , title =

    Qingyun Liu and Shiliang Tang and Xinyi Zhang and Xiaohan Zhao and Ben Y. Zhao and Haitao Zheng , title =

  43. [43]

    Huy Pham and Cyrus Shahabi and Yan Liu , title =

  44. [44]

    Hongjian Wang and Zhenhui Li and Wang-Chien Lee , title =

  45. [45]

    Gummadi , title =

    Bimal Viswanath and Alan Mislove and Meeyoung Cha and Krishna P. Gummadi , title =

  46. [46]

    Zhao , title =

    Gang Wang and Manish Mohanlal and Christo Wilson and Xiao Wang and Miriam Metzger and Haitao Zheng and Ben Y. Zhao , title =

  47. [47]

    Zhao , title =

    Gang Wang and Bolun Wang and Tianyi Wang and Ana Nika and Haitao Zheng and Ben Y. Zhao , title =

  48. [48]

    Julian McAuley and Jure Leskovec , title =

  49. [49]

    Manuel Egele and Gianluca Stringhini and Christopher Kruegel and Giovanni Vigna , title =

  50. [50]

    Hao Ma and Dengyong Zhou and Chao Liu and Michael R Lyu and Irwin King , title =

  51. [51]

    David Kempe and Jon Kleinberg and

  52. [52]

    Kun Liu and Evimaria Terzi , title =

  53. [53]

    Zhao , title =

    Alessandra Sala and Xiaohan Zhao and Christo Wilson and Haitao Zheng and Ben Y. Zhao , title =

  54. [54]

    Prateek Mittal and Charalampos Papamanthou and Dawn Song , title =

  55. [55]

    Arvind Narayanan and Vitaly Shmatikov , title =

  56. [56]

    Shouling Ji and Weiqing Li and Prateek Mittal and Xin Hu and Raheem Beyah , title =

  57. [57]

    Mudhakar Srivatsa and Mike Hicks , title =

  58. [58]

    Lars Backstrom and Cynthia Dwork and Jon Kleinberg , title =

  59. [59]

    Qian Xiao and Rui Chen and Kian-Lee Tan , title =

  60. [60]

    Bin Zhou and Jian Pei , title =

  61. [61]

    Davide Proserpio and Sharon Goldberg and Frank McSherry , title =

  62. [62]

    Ahmad Bashir and Mark Crovella and Saikat Guha and Krishna P

    Bimal Viswanath and M. Ahmad Bashir and Mark Crovella and Saikat Guha and Krishna P. Gummadi and Balachander Krishnamurthy and Alan Mislove , title =

  63. [63]

    Michael Backes and Pascal Berrang and Anne Hecksteden and Mathias Humbert and Andreas Keller and Tim Meyer , title =

  64. [64]

    Andrew McGregor and Ilya Mironov and Toniann Pitassi and Omer Reingold and Kunal Talwar and Salil Vadhan , title =

  65. [65]

    Jinyuan Jia and Binghui Wang and Le Zhang and Neil Zhenqiang Gong , title =

  66. [66]

    Shirin Nilizadeh and Apu Kapadia and Yong-Yeol Ahn , title =

  67. [67]

    Shouling Ji and Weiqing Li and Mudhakar Srivatsa and Raheem Beyah , title =

  68. [68]

    Kumar Sharad and George Danezis , title =

  69. [69]

    Gummadi and Nishanth Sastry , title =

    Giridhari Venkatadri and Oana Goga and Changtao Zhong and Bimal Viswanath and Krishna P. Gummadi and Nishanth Sastry , title =

  70. [70]

    Carley , title =

    Fred Morstatter and Jurgen Pfeffer and Huan Liu and Kathleen M. Carley , title =

  71. [71]

    Daniele Quercia and Rossano Schifanella and Luca Maria Aiello , title =

  72. [72]

    Vivek Kulkarni and Rami Al-Rfou and Bryan Perozzi and Steven Skiena , title =

  73. [73]

    Yu-Ru Lin and Drew Margolin and Brian Keegan and Andrea Baronchelli and David Lazer , title =

  74. [74]

    Jisun An and Ingmar Weber , title =

  75. [75]

    Ferda Ofli and Yusuf Aytar and Ingmar Weber and Raggi al Hammouri and Antonio Torralba , title =

  76. [76]

    Bichen Shi and Georgiana Ifrim and Neil Hurley , title =

  77. [77]

    Mohsen Jamali and Martin Ester , title =

  78. [78]

    Enrico Mariconti and Jeremiah Onaolapo and Syed Sharique Ahmad and Nicolas Nikiforou and Manuel Egele and Nick Nikiforakis and Gianluca Stringhini , title =

  79. [79]

    Lisa Singh and Grace Hui Yang and Micah Sherr and Andrew Hian-Cheong and Kevin Tian and Janet Zhu and Sicong Zhang , title =

  80. [80]

    Mazurek and Michael K

    Lujo Bauer and Lorrie Faith Cranor and Saranga Komanduri and Michelle L. Mazurek and Michael K. Reiter and Manya Sleeper and Blase Ur , title =

Showing first 80 references.