{"id":"17369bca-f892-4269-a506-b16da92d320d","arxiv_id":"2412.07268","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"PTSBench benchmarks post-training sparsity techniques and model families, finding learning-based allocation and block-wise reconstruction most effective, and attention-based models most sparsity-friendly.","lead":"PTSBench is a new benchmark that tests post-training sparsity methods on over 40 vision models across classification, detection, and image generation. It reports which sparsity allocation and reconstruction techniques work best, and which model architectures tolerate sparsity best, providing practical guidance for deploying compressed neural networks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'learning-based > criterion-based > heuristic-based' allocation ranking rests on a single learning-based method (FCPTS) from the authors' group; without a second independent learning-based method, the category-level claim is not established.","rationale":"The reader's weakest_assumption identified exactly the representativeness of FCPTS as the load-bearing premise for the allocation ranking. My reading agrees: the paper's own evidence (Table 2) shows substantial within-method variance from initialization, and the single-method representation is insufficient to support a category-level takeaway. The benchmark is still a substantial empirical resource, and most of its model-level observations (e.g., attention mechanisms being more sparsity-friendly) are plausible, but the central algorithm ranking needs independent validation before it is treated as definitive. The reader's CONDITIONAL verdict is therefore appropriate; my concern does not move it further, so I recommend UNCHANGED. The concrete test—adding one independent learning-based method and computing the same overall metric—directly settles whether the ranking is a property of the learning-based category or of FCPTS specifically.","tokens_in":18876,"tokens_out":4555,"duration_ms":45683,"concrete_test":"Add a second, independently developed learning-based sparsity allocation method to the benchmark, e.g., Soft Threshold Weight Reparameterization [30] adapted to PTS by learning per-layer soft thresholds on the 1,024-image calibration set, and evaluate it under the same Track 1 protocol (no reconstruction, sparsity rates {0.5, 0.6, 0.7, 0.8}, same models/tasks) to compute OM_alloc via Eq. 2. If this method does not consistently outperform L2Norm and ERK, the 'learning-based > criterion-based > heuristic-based' ranking in Table 7 fails. Also report per-seed variance (at least 3 seeds) for FCPTS and the new method to check whether the 2.67-point spread between FCPTS initializations exceeds seed noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline algorithm takeaway in Table 7—'learning-based > criterion-based > heuristic-based' for sparsity allocation—is supported by exactly one learning-based method, FCPTS. Section 3.1 states 'We also include FCPTS [10] as a learning-based method, which is the only existing method.' FCPTS is developed by the same research group as PTSBench (author overlap with Ref. [10]). More importantly, Table 2 shows the two FCPTS variants have OM_alloc 91.69 (L2Norm init) and 94.36 (ERK init), a spread of 2.67 points, while L2Norm and ERK score 83.94 and 80.61. The category-level claim treats 'learning-based' as a homogeneous class, but the observed gap between two initializations of the same algorithm is non-trivial; a different learning-based algorithm could easily fall below L2Norm. Because this ranking is the paper's principal algorithm-level conclusion and is used to recommend 'choosing appropriate initialization' and that 'the final layer is unsuitable for a high sparsity rate', it is load-bearing. Without an independent learning-based allocation method in the benchmark, the conclusion conflates FCPTS-specific behavior with a property of the entire learning-based category. The post-hoc sparsity range selection (Section 4) and missing error bars further weaken, but do not replace, this concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"PTSBench is a benchmark for post-training sparsity (PTS) covering 10+ fine-grained techniques, 3 vision tasks, and over 40 off-the-shelf model architectures. The paper formalizes the common two-stage PTS pipeline (sparsity allocation and reconstruction), introduces five overall metrics (OM_alloc, OM_recon, OM_arch, OM_robust, OM_task), and reports results on five tracks: sparsity allocation, reconstruction, neural architecture, model size robustness, and application tasks. The main takeaways are a category-level ranking of allocation strategies (learning-based > criterion-based > heuristic-based), reconstruction guidance (error correction helps only classification, sparse input is better, block-wise granularity is best), and model-level claims (attention-based models are more sparsity-friendly, large-scale pre-training improves sparsity potential, image generation needs specialized methods). The paper releases code and includes an explicit limitations section.","tokens_in":19045,"tokens_out":7837,"duration_ms":77128,"significance":"The benchmark addresses a practical and underserved area, and the scale of the experiments (over 8000 A800 GPU hours, 40+ models, three tasks) is a genuine asset. The paper is also commendably transparent about some limitations in Section 6, and the open-source framework is likely to be useful to the community. If the headline conclusions hold, they would provide concrete guidance for PTS method design and model selection. However, several load-bearing conclusions are currently under-supported: the category-level allocation ranking rests on a single learning-based method (FCPTS) from the authors' own group, the reconstruction conclusions rely on small differences with no error bars and with suspicious constant values in Table 4, and the architecture conclusions are drawn from confounded comparisons. These issues are fixable, but they require additional experiments or substantially softened claims before the central assertions can be accepted.","major_comments":[{"comment":"The takeaway 'Priority: learning-based > criterion-based > heuristic-based' is supported by exactly one learning-based algorithm, FCPTS, which is from the same research group as this paper (Ref. [10]). Table 2 shows that FCPTS(L2Norm) and FCPTS(ERK) differ by 2.67 points in OM_alloc (91.69 vs 94.36), a spread comparable to the 3.33-point gap between L2Norm and ERK (83.94 vs 80.61). Because the variance across initializations of the single learning-based method is the same order of magnitude as the difference between the two criterion-based methods, the data do not establish that 'learning-based' as a category is superior. The paper should either include at least one independent learning-based allocation method or rephrase the conclusion as 'FCPTS outperforms the evaluated criterion-based and heuristic methods'.","section":"§3.1, Table 2, Table 7 (Track 1)"},{"comment":"The reconstruction track reports no error bars or significance tests, and several entries are suspicious: the GEN MS column is exactly 88.32 for 'w/ Correction', 'w/o Correction', 'Sparse Input', and 'Block-wise', with other GEN values also nearly constant (88.06, 88.11, 88.18). Since the headline conclusion 'block-wise reconstruction is always the best' depends on OM_recon differences of about 2 points (66.76 vs 64.95 vs 62.54), these exact-looking values and the absence of variance make it impossible to assess whether the differences are meaningful. Please report standard deviations across at least multiple seeds (or explain why they are absent) and clarify how the GEN reconstruction gains are computed at each sparsity rate.","section":"§5.1.2, Table 4"},{"comment":"The benchmark restricts evaluation to sparsity rates {0.5, 0.6, 0.7, 0.8} with the justification that lower rates are 'negligible' and higher rates 'collapse'. This is a post-hoc choice that excludes the high-sparsity regime where PTS guidance is arguably most needed, and all overall metrics are averaged over this range. The paper should either validate at least a representative subset of settings at higher sparsity (e.g., 0.9 and 0.95), or explicitly reposition PTSBench as a mid-sparsity benchmark and avoid general statements about 'the sparsification ability of models' that may not transfer beyond 80%.","section":"§4, Section 4"},{"comment":"The model-level conclusions confound architecture with other factors. The claim that 'attention mechanism is a more sparsity-friendly model structure' is based on comparing MobileNetV3 vs MobileNetV2, ViT vs CNNs, and DeiT vs ViT, but these models differ in multiple ways beyond attention (inverted residuals, SE blocks, activation functions, patch size, input resolution, pre-training data, distillation). Similarly, the 'training strategy' conclusion drawn from DeiT vs ViT is confounded because the two models also differ in architecture and pre-training data. Controlled comparisons are needed, such as the same architecture with and without an SE block, or a single architecture pre-trained at different scales, before these model-level conclusions can be considered established.","section":"§5.2.1, Table 5"},{"comment":"Table 2 reports only a single GEN value per allocation method (e.g., 16.40, 78.47, 91.71) with dashes for 60/70/80, while Table 6 reports GEN relative accuracy at all four sparsity rates (78.06, 5.44, 0.18, 0). This means OM_alloc appears to average over different numbers of sparsity rates per task, making the cross-task and cross-method comparisons in Tables 2 and 7 difficult to interpret. Please specify the exact support of the mean in each of Eqs. (2)-(7) and make the per-sparsity-rate GEN results available for the allocation track.","section":"Table 2, Eqs. (2)-(7)"}],"minor_comments":[{"comment":"'Singe' should be 'Single'.","section":"Table 4"},{"comment":"There are typos: 'sparisy' should be 'sparsity' and 'perpectives' should be 'perspectives'.","section":"§5.3"},{"comment":"Time entries such as '3h34' should be written as '3h34m' for consistency with '3h32m'.","section":"Table 3"},{"comment":"The label 'layer31.2.conv1' appears to be a typo for 'layer3.1.2.conv1'.","section":"Figure 3"},{"comment":"The text says the models are calibrated for '20,000 epochs' with 1,024 images and batch size 64; this likely means iterations or steps, and the intended meaning should be clarified.","section":"§4"},{"comment":"The reference list appears twice in the manuscript; the duplicate should be removed.","section":"References"},{"comment":"The notation E^2(...) is undefined; please clarify whether E denotes the arithmetic mean and E^2 denotes the square of that mean.","section":"Eqs. (2)-(7)"}],"recommendation":"major_revision","confidential_remarks":"The paper delivers a large, useful benchmark and an open-source toolkit, and the authors are explicit about some limitations. The main concern is that the paper's most prominent ranking, 'learning-based > criterion-based > heuristic-based', rests on a single learning-based method from the authors' own group, and the reconstruction and architecture conclusions are weakened by missing error bars and confounded comparisons. I would not reject the paper, but these issues need to be addressed with additional evidence or carefully hedged claims before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"PTSBench is a genuinely useful empirical resource. The authors systematically decompose post-training sparsity into allocation and reconstruction components and benchmark them across 40+ models, three vision tasks, and multiple sparsity rates, with code released. That alone is valuable: the community can now compare pluggable techniques on a common harness. The findings that block-wise reconstruction is both better and faster, that sparse input outperforms dense input, and that model-size robustness doesn't track sparsity potential are useful, and I take them as plausible.\n\nThe soft spots are where the paper moves from observations to categorical conclusions. The headline ranking 'learning-based > criterion-based > heuristic-based' for allocation rests on exactly one learning-based method, FCPTS, which comes from the authors' group. The paper says FCPTS is the only existing method, which is true, but then the conclusion silently equates FCPTS's behavior with a property of the whole category. The spread between FCPTS's two initializations (91.69 vs 94.36 OM) is smaller than the gap to L2Norm (83.94), but a different learning-based allocator could easily land below L2Norm. That makes the category-level claim a hypothesis, not a benchmark result. The authors should either add an independent learning-based baseline or explicitly frame the conclusion as 'FCPTS outperforms'.\n\nThere are also methodological gaps. No error bars or significance tests anywhere. For a benchmark that uses 8,000 GPU-hours, I'd accept a single run, but then the rankings should be called trends, not findings. Table 4 reports identical GEN MS values (88.32) across several reconstruction techniques, which looks like the generation task simply doesn't respond to reconstruction; that deserves a comment, not a uniform number. The architecture conclusions are confounded: ViT vs DeiT differ in pretraining data, and MobileNetV3 vs MobileNetV2 differ beyond the SE block. The paper does acknowledge the training-data confound for ViT/DeiT but still attributes the gap to attention structure. The sparsity range 50-80% is defended as post-hoc, but Fig 3 shows a 90% allocation for ResNet-32, which is inconsistent with the stated collapse regime. And '20,000 epochs' of calibration is unexplained; presumably they mean iterations.\n\nNone of this kills the paper's value. It's a solid empirical contribution that deserves peer review, but it needs another learning-based allocation method or a more modest claim, plus basic error reporting. I'd cite it for the benchmark and the reconstruction results, but I would not build on the allocation ranking until it's confirmed by a second independent method.","headline":"A useful benchmark with real code and extensive experiments, but its headline allocation ranking rests on one in-group method and should be read as a hypothesis, not a conclusion.","tokens_in":19683,"tokens_out":4819,"would_cite":true,"duration_ms":42760,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Learned sparsity allocation beats heuristics across 40-plus models","keywords":["post-training sparsity","model compression","benchmark","sparsity allocation","reconstruction granularity","fine-grained pruning","model sparsification","architecture sparsity potential"],"falsifier":"Run the same PTSBench protocol on a held-out set of architectures, such as EfficientNet, ConvNeXt, or a CNN-transformer hybrid, and also at sparsity rates below 50% and above 80%; if block-wise reconstruction fails to beat layer-wise reconstruction, or if a new learning-based allocation method falls below L2Norm, the paper's headline conclusions are refuted.","tokens_in":18557,"feed_emoji":"📊","tokens_out":4524,"duration_ms":99775,"temperature":0.7,"pith_summary":"This paper introduces PTSBench, a systematic benchmark for post-training sparsity (PTS), the practice of pruning a trained neural network to a target sparsity using only a small calibration set and no retraining. The authors benchmark more than ten pluggable PTS techniques across three vision tasks and over forty off-the-shelf models, and they argue that the results settle several open practice questions: learned sparsity allocation outperforms criterion-based and heuristic allocation, block-wise reconstruction is consistently the best reconstruction granularity, sparse reconstruction input beats dense input, and attention-based models are the most sparsity-friendly. The point of the benchmark is to give researchers and practitioners a reliable ranking of PTS choices and model families so they can design better compression methods and choose sparsifiable backbones.","feed_headline":"Learned sparsity beats heuristics across 40+ models","feed_subtitle":"Block-wise reconstruction and attention-based architectures also come out on top in a new post-training sparsity benchmark.","key_machinery":"The benchmark itself is the central instrument: PTSBench organizes evaluation into five tracks (sparsity allocation, reconstruction, neural architecture, model size robustness, and application tasks) and summarizes each with an overall metric, specifically the quadratic mean of relative task accuracies such as OM_alloc and OM_recon. The two-step PTS pipeline, sparsity allocation then activation reconstruction, is the object under study; the paper isolates each fine-grained technique (allocation strategy, error correction, reconstruction input, reconstruction granularity) by swapping one component while holding the rest fixed. FCPTS serves as the sole representative of learning-based allocation, ERK and L2Norm as criterion-based representatives, and Uniform as the heuristic baseline.","core_discovery":"PTSBench's central claim is that the two-step PTS pipeline, sparsity allocation followed by reconstruction, can be decomposed into fine-grained choices whose effects are consistent and measurable. On allocation, the overall metric ranks FCPTS, a learning-based method, first at 94.36, well ahead of the criterion-based L2Norm at 83.94 and ERK at 80.61, with the heuristic Uniform strategy last at 63.59. On reconstruction, block-wise reconstruction achieves the best overall reconstruction gain (66.76) versus layer-wise (64.95) and single-layer (62.54), and using the sparse model's own output as reconstruction input beats dense input, especially at high sparsity. Across architectures, attention-based models (ViT, MobileNetV3, DeiT) rank at the top of sparsity potential; the authors attribute this to attention concentrating information on critical neurons, and they also report that detection models sparsify better than classification models, while image generation collapses beyond 50% sparsity.","pith_inferences":["The paper's ranking of allocation strategies depends on FCPTS being representative of all learning-based allocation; a natural test is whether a different learned method not developed by the same group preserves the ranking.","The correlation between attention mechanisms and sparsity friendliness could be probed causally by ablating the squeeze-and-excitation block in MobileNetV3 against MobileNetV2, extending the paper's observational comparison.","The claim that large-scale pretraining improves sparsity robustness suggests a testable extension: pretrain the same architecture at several data scales and measure the overall metric at fixed sparsity.","The reported collapse of generation tasks beyond 50% sparsity implies that the two-step PTS pipeline may need a fundamentally different allocation objective for diffusion models, not just tuned hyperparameters."],"forward_implications":["If the rankings hold, practitioners should use learned allocation methods over uniform or criterion-based allocation, with an ERK-style initialization for the learned method.","Block-wise reconstruction should become the default reconstruction granularity in PTS, since it is both higher-performing and faster than layer-wise or single reconstruction.","Attention-based architectures should be the preferred backbones when a model will be sparsified for deployment, and large-scale pretraining appears to further increase sparsity friendliness.","Detection backbones tolerate sparsity better than classification models, suggesting that downstream-head parameters can absorb pruning damage.","PTS methods specifically designed for diffusion and generation models are needed, since existing general methods collapse beyond 50% sparsity."],"supporting_citations":[{"why":"Supplies FCPTS, the only learning-based sparsity allocation method included in the benchmark.","marker":"[10]"},{"why":"Provides ERK, the criterion-based allocation baseline and the initialization used by FCPTS.","marker":"[4]"},{"why":"Provides L2Norm, the other criterion-based allocation baseline.","marker":"[14]"},{"why":"Introduces POT, the layer-wise calibration reconstruction approach that motivates the error correction comparison.","marker":"[31]"},{"why":"Motivates the block-wise reconstruction track, transferring the idea from post-training quantization to sparsity.","marker":"[33]"}],"fun_headline_variants":["PTSBench: 40+ models, 10+ methods, one sparsity benchmark","Learning-based sparsity allocation tops PTSBench rankings","Block-wise reconstruction wins in post-training sparsity","Attention-based models sparsify best across 40+ architectures"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's headline rankings assume that FCPTS is a fair representative of learning-based sparsity allocation and that the tested models and sparsity range (50-80%) cover the regimes that matter, so if either fails, the 'learning-based beats criterion-based' conclusion may not generalize.","fun_headline_variants_meta":{"raw":{"variants":["PTSBench: 40+ models, 10+ methods, one sparsity benchmark","Learning-based sparsity allocation tops PTSBench rankings","Block-wise reconstruction wins in post-training sparsity","Attention-based models sparsify best across 40+ architectures"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000296,"raw_usage":{"total_tokens":1741,"prompt_tokens":990,"completion_tokens":751,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":679}},"tokens_in":606,"tokens_out":751,"duration_ms":9212,"temperature":1.0,"reasoning_tokens":679,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:56:13.834864+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same PTSBench protocol on a held-out set of architectures, such as EfficientNet, ConvNeXt, or a CNN-transformer hybrid, and also at sparsity rates below 50% and above 80%; if block-wise reconstruction fails to beat layer-wise reconstruction, or if a new learning-based allocation method falls below L2Norm, the paper's headline conclusions are refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies FCPTS, the only learning-based sparsity allocation method included in the benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces POT, the layer-wise calibration reconstruction approach that motivates the error correction comparison."}],"review_version":1}