{"id":"6a8178f8-7de1-4a9d-b4dd-46527739a633","arxiv_id":"2505.14592","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Most pruning methods designed for convolutional networks fail to transfer to a fully connected network on an IoT intrusion detection dataset, with ThiNet offering the best practical trade-off.","lead":"The authors tested eight neural network pruning methods on a new cybersecurity dataset, using a simpler fully connected network instead of the convolutional networks the methods were designed for. They found that most pruning methods lose accuracy when transferred to this new setting and recommend ThiNet as the most practical option.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DAIS implementation fidelity is the load-bearing weak point: the paper's own disclosure that only one regularizer was used and the training cycle is missing undercuts the broad 'many methods do not generalize' conclusion.","rationale":"I chose implementation fidelity over statistical power because the paper's headline negative result is a claim about the algorithms themselves. The authors are transparent enough to disclose the DAIS port in Section III-B, which is exactly why the concern can be tested. I do not think this requires changing the conditional verdict: the paper should either soften the abstract to 'our port of DAIS did not transfer' or supply the missing implementation details before claiming many methods fail. The ADMM point raised by the reader is weaker than it appears, since masking per filter after the linear layer gives the same computation as masking the weight rows before it. The statistical issue (using z instead of t with n=3) is real but secondary; it would affect the strength of the ranking more than the transferability conclusion.","tokens_in":12761,"tokens_out":7287,"duration_ms":68297,"concrete_test":"Run the DAIS official code on the paper's fully connected model, adapted as minimally as possible: keep all regularizers that have FC analogues, use a controlled lasso coefficient instead of disabling it, and follow the original alternating DARTS training schedule (recover the reference sheet or ask the authors for the missing documentation). Compare F1 at 25-50% pruning on the smaller ACI model to Table III. If DAIS improves from roughly 0.35-0.66 to above 0.9, the paper's 'does not generalize' conclusion is contradicted for this method; if it remains low, the implementation caveat is not decisive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that many CNN-oriented pruning methods do not transfer to a fully connected ACI model is only as strong as the fidelity of the compared implementations. Section III-B discloses that DAIS was run with only one of its three regularizers (the residual-block regularizer was inapplicable and the lasso regularizer was disabled as 'not controllable'), and that the original DARTS-style training cycle could not be reproduced because the reference sheet 'was unable to be located.' DAIS is then reported as failing badly in this setting, with F1 values as low as 0.013 on the larger model. If that collapse is an artifact of the partial port, the statement 'many of them do not generalize' loses its principal supporting example: of the CNN-oriented methods considered (ADMM, DAIS, ThiNet), only ADMM would clearly fail, and the ADMM modification is mathematically neutral because a per-filter mask commutes with the linear layer. The broad transferability conclusion is therefore not established, even though the specific 'ThiNet is competitive' recommendation may survive.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares eight pruning approaches—ADMM-joint, BERT-Theseus, DAIS, Iterative Theseus, ThiNet, random structured pruning, and complete retraining/recreation—applied to a fully connected network trained on the ACI IoT intrusion detection dataset. It reports F1 scores, parameter counts, and pruning times for two model sizes across a grid of requested pruning percentages, proposes Iterative Theseus as a layer-replacement baseline, and concludes that ThiNet is the best practical method while most CNN-oriented pruning methods do not transfer to this fully connected setting.","tokens_in":13035,"tokens_out":5278,"duration_ms":40873,"significance":"If the central finding were fully supported, the paper would offer useful practical guidance for deploying pruned intrusion detection models on resource-constrained edge hardware. The authors provide code and result data, run two model sizes, and include control baselines (random pruning and retraining from scratch), which are appropriate experimental ingredients. However, the broad transferability claim is substantially weaker than the abstract suggests because the DAIS implementation is disclosed to be incomplete, the BERT-Theseus pruning percentages are approximate, and the statistical analysis rests on three runs with normal-theory error bars. The specific recommendation that ThiNet is competitive is more credible than the negative generalization claim, but even that recommendation needs to be expressed relative to a clearer criterion.","major_comments":[{"comment":"The central negative claim that many CNN-oriented pruning methods do not generalize is materially supported by the DAIS results, which reach F1 values as low as 0.013 at 74% retained parameters on the large model. However, the manuscript itself discloses that the DAIS implementation used only one of its three regularizers (the residual-block regularizer was inapplicable and the lasso regularizer was disabled) and that the DARTS-style alternating training cycle could not be reproduced because the reference sheet \"was unable to be located.\" As described, this is not a faithful port of DAIS from [16], so the DAIS failure cannot be cited as evidence that the original algorithm fails to transfer. Please either implement the full method with all applicable regularizers and the documented training schedule, or revise the conclusion to explicitly restrict the claim to the implemented variants and remove DAIS from the list of algorithms whose transferability is being assessed.","section":"Section III-B and Table III"},{"comment":"The mapping from requested pruning percentage to actual model size is approximate for BERT-Theseus, and the table shows identical parameter counts and identical F1 values across four consecutive requested percentages (small model: 329k parameters and F1 0.106 for percentages 0.99 through 0.62; large model: 1233k parameters and F1 0.041 for the same range). These rows are not independent measurements of distinct pruning degrees, yet Figures 3 and 4 connect them as a single curve. Please report the effective per-layer sizes actually produced by the block-selection heuristic and restrict the \"Helped\" characterization to configurations that genuinely differ in model size.","section":"Section V and Table III (BERT-Theseus rows)"},{"comment":"The error bars are computed as 1.960*sigma/sqrt(3) under a normal assumption, which the authors themselves acknowledge may be incorrect for n=3. The standard deviations are large precisely in rows that drive the conclusions (e.g., DAIS small model at 0.08: F1 0.343 +/- 0.52; random structured small model at 0.25: F1 0.345 +/- 0.41; recreation small model at 0.08: F1 0.574 +/- 0.41). The text also makes ordinal claims, such as the order in which the \"Hindered\" algorithms start to degrade, without any significance test or paired comparison. Please provide per-run data or appropriate small-sample intervals and state which pairwise differences are actually supported by the data.","section":"Section V, error-bar formula and Table III"},{"comment":"The concluding recommendation that ThiNet is \"the best algorithm\" for this dataset is not uniformly supported by the reported F1 values. On the large model, Iterative Theseus achieves higher F1 than ThiNet at percentages 0.99 (0.800 vs 0.789), 0.49 (0.745 vs 0.660), and 0.25 (0.702 vs 0.523), while Recreation is close at several points. Since the relative ranking depends on pruning degree, time budget, and whether F1 or efficiency is prioritized, the recommendation should be stated as a Pareto-style or criterion-dependent conclusion rather than a single global winner.","section":"Section VI and Table III"}],"minor_comments":[{"comment":"The footnote correctly observes that Gou et al. [18] does not actually classify BERT-Theseus, but the main text still attributes the classification to that survey; please reconcile the text with the footnote.","section":"Section III-C, footnote 1"},{"comment":"The architecture description is ambiguous: \"a pair of linear layers surrounding 27 fully connected linear layers\" does not make the total layer count precise, and the statement that filters per hidden layer increase by 1 per layer should be accompanied by an explicit schedule for both the 175-filter and 75-filter models.","section":"Section IV"},{"comment":"The phrase \"at just around 0.2%\" appears to mean a pruning fraction of 0.2 (or 20% of filters retained), not 0.2%; please correct the units to avoid confusion with the x-axis of the figures.","section":"Section V"},{"comment":"The use of single and double asterisks in the grouped and under-sampled columns is explained only implicitly; please add a legend or table note defining each symbol and the grouping scheme.","section":"Table II"},{"comment":"Reference [9] is a non-archival blog post used for a general claim about weight-based pruning; consider replacing it with a peer-reviewed source or clearly labeling it as an online resource.","section":"References"},{"comment":"There are several typographical inconsistencies, including \"regulairizers\" (Section III-A), \"thinet\" vs \"ThiNet\", \"BERT-Thesesus\" (Section VI), and \"loose\" for \"lose\"; a careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is transparent about its own limitations, and the DAIS disclosure in Section III-B is particularly damaging to the paper's central negative claim. I do not see evidence of deliberate misrepresentation, but the abstract and conclusion overstate the transferability finding in a way that the experimental design cannot support. A revised version that narrows the claim to the implemented variants, reports effective pruning degrees for BERT-Theseus, and sharpens the statistical analysis would be a useful empirical contribution. The current form would not be suitable for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, honest empirical comparison that is worth engaging, but the broad conclusion that CNN-oriented pruning methods fail on a fully connected NIDS model is only as strong as the most questionable implementation in the batch. The DAIS port uses one of three regularizers and a training cycle that could not be reproduced because the reference sheet was missing. That alone makes the 'many do not generalize' claim shaky. Still, the ThiNet recommendation and the practical observations are likely to survive.\n\nWhat the paper does well: it runs eight methods (including two baselines) on a real IoT intrusion detection dataset, on two model sizes, across ten pruning levels, with three trials per cell and standard deviations reported. Code and data are on Github. The authors are unusually transparent about their deviations: they disclose the ADMM mask placement, the DAIS regularizer situation, and the BERT-Theseus percentage approximation. That transparency is a real credit. The finding that Iterative-Theseus performs well but is far too slow, and that a from-scratch retrain is competitive, is actionable for practitioners. The observation that their initial model was oversized is a sensible post-hoc interpretation.\n\nSoft spots, in order: the DAIS implementation fidelity is the load-bearing one. With DAIS excluded or labeled as an approximation, the 'many methods' claim reduces to ADMM (whose mask-before-or-after-linear-layer change is at least mathematically plausible to be neutral) plus whatever BERT-Theseus is doing. The statistics are also thin: three runs with a z-score-based confidence interval assumes normality, which the authors themselves flag. That is a minor issue for a benchmark, not a fatal one. The BERT-Theseus pruning percentages are approximate, and the reported parameter counts sometimes increase at the end, which muddies the headline numbers a bit.\n\nRecommendation: send it to peer review. The authors have a reproducible artifact and an honest write-up. The right referee will push them to either re-run DAIS faithfully or soften the generalization claim, and to add a couple more seeds or nonparametric confidence bounds. This is exactly the kind of paper that improves with review rather than being a desk-reject.","headline":"Honest empirical benchmark with a load-bearing DAIS implementation gap; the ThiNet result likely stands, but the 'many methods don't transfer' claim needs caveats.","tokens_in":13478,"tokens_out":2923,"would_cite":false,"duration_ms":33415,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper finds that most CNN-oriented pruning algorithms lose significant accuracy when applied to a fully connected network on the ACI IoT intrusion detection dataset, and that ThiNet is the best practical pruner for this setting.","keywords":["network intrusion detection","deep neural network pruning","IoT","systems on chip","fully connected network","model compression","ThiNet","edge computing"],"falsifier":"Re-run DAIS on this fully connected ACI model with all three regularizers enabled and the original documented training cycle; if its F1 then matches or beats ThiNet across pruning percentages, the paper's conclusion that DAIS does not transfer to fully connected networks would be refuted.","tokens_in":12603,"feed_emoji":"✂️","tokens_out":6438,"duration_ms":47569,"temperature":0.7,"pith_summary":"Pruning reduces a trained network's size so it can run on embedded hardware, but pruning methods are usually designed for convolutional networks. This paper asks whether those methods still work when transplanted to a much simpler fully connected network trained on the ACI-IOT-2023-payload intrusion-detection dataset, testing eight approaches across pruning levels from 99% down to 4% of the original filters. The paper's central finding is that most CNN-era algorithms do not transfer: accuracy drops sharply past a critical pruning point, runtimes become impractical, or results are unstable across random trials. ThiNet, a heuristic that removes filters with the lowest contribution to the next layer, emerges as the best practical method, while the authors' own Iterative-Theseus scores highest on F1 but at a far greater time cost. The authors conclude that the field cannot assume CNN pruning methods will work on other network types, especially for resource-constrained edge intrusion detection.","feed_headline":"ThiNet wins where CNN pruning methods fail on IoT intrusion detection","feed_subtitle":"Only a few pruning methods survive transfer to fully connected layers, and ThiNet is the practical pick.","key_machinery":"The argument rests on an empirical comparison protocol: a fixed fully connected network (two outer linear layers plus 27 hidden layers with leaky ReLU and dropout) trained on the ACI-IOT-2023-payload dataset, then pruned by each method at ten target percentages from 99% down to 4%. Results are scaled relative to the original F1 score and parameter count, with each configuration run three times and averaged. The key contrast is between 'Hindered' algorithms, whose F1 falls past a critical pruning point, and 'Helped' algorithms, which improve as more is removed. Within this setup, ThiNet's mechanism—ranking each filter by the total activation it contributes to the next layer and pruning the lowest-ranked filters—is what best preserves accuracy per unit of pruning time.","core_discovery":"The paper claims that pruning algorithms designed for convolutional neural networks largely fail to generalize to a fully connected network on the ACI IoT intrusion-detection benchmark. Of the eight methods tested, ThiNet offers the best trade-off between F1 score and pruning time, with a retrained-from-scratch model close behind. Two algorithms are 'helped' by aggressive pruning—DAIS and BERT-Theseus—but DAIS is highly variable between runs and BERT-Theseus only becomes competitive when large layer blocks are removed. The authors conclude that most CNN pruning methods cannot be transferred to fully connected networks without significant loss of potency, and interpret the strong performance of retraining as evidence that the original model was oversized for the dataset.","pith_inferences":["If the transfer failure is real, it suggests that the implicit redundancy structure in convolutional networks (spatial correlation across channels) is what many pruning regularizers rely on; fully connected layers, where each filter has no spatial structure, remove that crutch.","A decisive follow-up would be to run ThiNet and a full-fidelity DAIS/ADMM on the same fully connected model to separate algorithmic failure from implementation drift.","For edge deployment, the paper's results imply that a simple combination of random-structured pruning plus retraining (or no pruning at all with a smaller architecture) may be a cheaper baseline than any learned pruning method for this dataset."],"forward_implications":["On a fully connected network trained on ACI IoT data, the practical choice is ThiNet: it keeps high F1 scores at moderate pruning levels and finishes pruning in a fraction of Iterative-Theseus's time.","Simply retraining a model resized to the pruned dimension is nearly as good as most dedicated pruning methods, suggesting the original 29-layer model was oversized for this dataset.","BERT-Theseus is only useful when removing large continuous chunks of layers; at light pruning it underperforms because equal-sized replacement modules must be trained from scratch.","Most CNN-oriented pruning algorithms cannot be moved to fully connected networks without significant loss of accuracy, which matters for deploying pruned detectors on edge hardware.","DAIS in its tested form is unreliable on non-residual, fully connected networks, a consequence of the missing regularizers and an unrecoverable training schedule."],"supporting_citations":[{"why":"Supplies the ACI-IOT-2023-payload dataset used to train and evaluate all pruning methods.","marker":"[3]"},{"why":"Defines ThiNet's filter-pruning heuristic, which the paper finds to be the best practical method for this task.","marker":"[17]"},{"why":"Presents ADMM-joint, one of the compared methods whose transfer behavior to a fully connected network is assessed.","marker":"[12]"},{"why":"Presents DAIS, the compared method that proved unstable under the test conditions.","marker":"[16]"},{"why":"Defines BERT-Theseus, the progressive module-replacing method that the authors adapt into Iterative-Theseus.","marker":"[5]"},{"why":"The structured-pruning survey that motivated the selection of candidate algorithms and the study's framing.","marker":"[6]"},{"why":"Describes the DARTS training scheme that DAIS depends on and that the paper could not fully replicate.","marker":"[19]"}],"fun_headline_variants":["ThiNet beats CNN pruning on IoT detection","Most CNN pruning fails on IoT nets; ThiNet works","CNN pruning won't transfer to IoT detection; ThiNet does","ThiNet best when CNN pruning fails on IoT detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the re-implementations of ADMM and DAIS faithfully capture the original algorithms, even though DAIS was run with only one of its three regularizers and an unrecoverable training schedule, and ADMM's mask was applied after the linear layer rather than during matrix multiplication.","fun_headline_variants_meta":{"raw":{"variants":["ThiNet beats CNN pruning on IoT detection","Most CNN pruning fails on IoT nets; ThiNet works","CNN pruning won't transfer to IoT detection; ThiNet does","ThiNet best when CNN pruning fails on IoT detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000813,"raw_usage":{"total_tokens":3497,"prompt_tokens":809,"completion_tokens":2688,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":425,"completion_tokens_details":{"reasoning_tokens":2622}},"tokens_in":425,"tokens_out":2688,"duration_ms":18373,"temperature":1.0,"reasoning_tokens":2622,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:31:09.893309+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run DAIS on this fully connected ACI model with all three regularizers enabled and the original documented training cycle; if its F1 then matches or beats ThiNet across pruning percentages, the paper's conclusion that DAIS does not transfer to fully connected networks would be refuted.","supporting_citations":[{"cited_title":"Thinet: A filter level pruning method for deep neural network compression,","cited_arxiv_id":null,"evidence_quote":"Defines ThiNet's filter-pruning heuristic, which the paper finds to be the best practical method for this task."},{"cited_title":"Dnn compression by admm-based joint pruning,","cited_arxiv_id":null,"evidence_quote":"Presents ADMM-joint, one of the compared methods whose transfer behavior to a fully connected network is assessed."},{"cited_title":"Dais: Automatic channel pruning via differentiable annealing indicator search,","cited_arxiv_id":null,"evidence_quote":"Presents DAIS, the compared method that proved unstable under the test conditions."},{"cited_title":"Structured pruning for deep convolutional neural networks: A survey,","cited_arxiv_id":null,"evidence_quote":"The structured-pruning survey that motivated the selection of candidate algorithms and the study's framing."}],"review_version":1}