{"id":"6773f901-7649-42c8-b3de-55038d2d7b53","arxiv_id":"2508.05239","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"An ICA-based structured pruning method that preserves functional neuron networks shows mixed performance against FLAP, with overclaimed state-of-the-art results.","lead":"This paper proposes pruning large language models by first finding \"functional networks\" of neurons that co-activate, using the same statistical method (ICA) neuroscientists use on brain scans, then deleting neurons outside those networks. The method matches or slightly beats existing pruning baselines in a few settings but loses to them in others, and key implementation details are missing.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No control shows the ICA-derived masks outperform random or activation-magnitude neuron selection; the undisclosed threshold is the only link from components to pruning.","rationale":"I read the paper's central claim as twofold: (a) ICA can identify functional networks and key neurons in LLMs, and (b) preserving them yields state-of-the-art structured pruning. The load-bearing inferential step is between CanICA output and pruning: thresholded source matrix → OR mask → preserved neurons. The paper never reports the threshold, never states the rule that maps a threshold to a fixed pruning rate, and never tests whether the specific mask matters. That is the weakest point because if a random or activation-magnitude mask performs the same, the core explanatory claim about functional networks is unsupported even if the numbers are reproducible. I therefore agree with the reader's REJECT verdict and with their identification of the threshold as an unvalidated premise, but I would frame the concern more broadly: the missing causal control is what makes the unreported threshold especially damaging. Secondary issues—n_components tuned on the test metric (with 256 outperforming the chosen 128 in the hyperparameter table), no variance estimates, and mixed 30% results against FLAP—reinforce rejection but are not the single load-bearing point. Verdict unchanged.","tokens_in":12724,"tokens_out":6309,"duration_ms":78735,"concrete_test":"Using the released code, reproduce the Vicuna-7B-v1.5 20% pruning run and extract the actual threshold and per-layer mask cardinalities. Then create two control masks with identical per-layer cardinalities: (1) random neuron selection, and (2) selection of the top mean-|z| neurons from the same calibration activations. Evaluate all three on Wikitext-2 perplexity and the six zero-shot tasks. If either control is within roughly 1 perplexity point or 1 accuracy point of the CanICA mask, the ICA functional-network identification is not load-bearing for the reported results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that thresholding the CanICA source matrix identifies key neurons whose preservation maintains LLM function—is never given a causal test. In 'Identify and Preserve Functional Networks,' masks are obtained by thresholding the source signal matrix and OR-aggregating across 80 runs, but the threshold is never reported, and the paper does not say how the 20%/30% pruning ratio is enforced (e.g., per-layer threshold to reach a target cardinality, or a global threshold whose union is then kept). More importantly, no ablation compares this mask with (i) a random mask of the same size, (ii) a mask from raw z-scored activation magnitudes, or (iii) a mask with neuron labels permuted. Without such controls, the results are equally consistent with the hypothesis that almost any 80% neuron subset performs similarly, making 'functional networks' decorative rather than causal. The overclaim is visible in Tables 4–5: at 30% pruning FLAP beats CanICA on most tasks, so the SOTA statement is not supported even before this control. The undisclosed threshold is not merely a reproducibility gap; it hides whether the central mechanism has any effect.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a structured pruning method for LLMs based on identifying 'functional networks' with canonical independent component analysis (CanICA), applied layer-by-layer to MLP neuron activations. The source signal matrix from CanICA is thresholded to obtain binary masks, OR-aggregated across 80 runs of 40 calibration samples each, and used to preserve 'key neurons' before pruning. Experiments on LLaMA2-7B-chat, Vicuna-7B-v1.5, and ChatGLM3-6B-base report perplexity and zero-shot accuracy at 20% and 30% pruning rates, with comparisons to LLM-Pruner, SliceGPT, FLAP, and Shortened LLaMA. The abstract claims state-of-the-art performance and successful identification of functional networks.","tokens_in":13039,"tokens_out":5023,"duration_ms":50100,"significance":"The cross-disciplinary idea—adapting spatial ICA from neuroimaging to discover neuron groups in LLMs and using those groups for pruning—is novel and potentially useful. If properly validated, it would offer a new grouping criterion for structured pruning and contribute to mechanistic interpretability. The paper provides a public code repository, evaluates on three model families, and reports calibration-sample-size sensitivity. However, the current evidence does not establish the central mechanism: the mask-generation threshold is undisclosed, no comparison against random or magnitude-based masks is provided, and the state-of-the-art claim is not supported at 30% pruning. These gaps are addressable but require substantial additional analysis.","major_comments":[{"comment":"The method relies on thresholding the CanICA source signal matrix to obtain binary masks, but the threshold is never reported, nor is the procedure for enforcing the target pruning ratio (e.g., per-layer percentile vs. global threshold). Without this information, the pruning decision is under-specified and the experiments are not reproducible. Since the threshold is the only link between ICA components and the set of preserved neurons, this is a load-bearing omission.","section":"Identify and Preserve Functional Networks"},{"comment":"The abstract and conclusion claim state-of-the-art performance, but the 30% pruning results contradict this. In Table 5 (LLaMA2-7B, 30%), FLAP outperforms CanICA on all seven reported metrics; in Table 4 (Vicuna-7B, 30%), FLAP wins four metrics, including Wikitext-2 perplexity (21.4575 vs. 25.9746 for CanICA without bias). The SOTA claim is therefore not supported by the full set of results.","section":"Results (Tables 4-5)"},{"comment":"The hyperparameter n_components is selected by evaluating post-pruning perplexity on Wikitext-2 (the same dataset used for the main evaluation). The reported table shows n_components=256 gives lower perplexity (18.4713) than the chosen 128 (18.9421), yet 128 is used. This is test-set fitting, and the model selection is not independent. Additionally, the paper states n_components is the only hyperparameter, ignoring the mask threshold and any parameters governing the z-score preprocessing or the group aggregation; these are also hyperparameters that affect the mask.","section":"Impact of Hyperparameters"},{"comment":"No control experiments isolate the contribution of the ICA-derived functional networks. A random mask of the same size, a mask based on raw z-scored activation magnitudes, or a mask with component labels permuted would be necessary to show that the identified functional networks, rather than any broad subset or a simple importance score, are responsible for the observed performance. Without such ablations, the results are consistent with the hypothesis that neuron selection via ICA masking is not causally meaningful.","section":"Method / Evaluation"}],"minor_comments":[{"comment":"The table of n_components results is referenced as 'Table??' in the text; the reference is unresolved.","section":"Impact of Hyperparameters"},{"comment":"'School of Pyhisic' should be 'School of Physics' (typo).","section":"Author affiliation"},{"comment":"Grammar: 'Most of them overlooks' should be 'Most of them overlook'.","section":"Abstract"},{"comment":"Table 3 reports CanICA results on ChatGLM3 with no comparison baselines. The sentence 'the proposed method achieves the best performance in several tasks' is meaningless without competitors; please clarify that these are standalone results.","section":"Comparison Results (ChatGLM3)"},{"comment":"The limitations paragraph acknowledges that ICA is linear, but does not mention the missing threshold or the absence of random-baseline controls, which are the main threats to the central claim. A revised discussion should address these.","section":"Discussion"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising idea but insufficient evidence. The authors should be required to (1) report the mask threshold and the pruning-ratio enforcement mechanism, (2) add random and magnitude-based control masks, (3) select n_components on a validation split or justify the choice, and (4) temper the SOTA claim in view of the 30% results. If the controls show that any 80% mask performs similarly, the paper would need to be reframed as a neuroscience-inspired analysis rather than a pruning method."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the ICA-for-pruning idea is genuinely new, but the paper as written overclaims. The evidence doesn't show the functional-network masks are doing the work, and the state-of-the-art claim falls apart at 30% pruning.\n\nWhat's new: applying spatial ICA (CanICA) to MLP activations to extract functional networks, thresholding the source matrix to get binary masks, and OR-aggregating across runs to form a pruning mask. That's a real departure from per-neuron importance scoring. The brain-network analogy is made concrete, not just decorative. The experimental setup is mostly fair: same calibration data, no recovery training, FLAP with and without bias compensation included. Layer-by-layer ICA is a sensible way to handle large models.\n\nThe soft spots are real. At 30% pruning on LLaMA2-7B, FLAP gives Wikitext2 perplexity 23.93; the proposed CanICA+bias gives 25.49, and FLAP wins most zero-shot tasks. At 20% the results are mixed, so 'state-of-the-art' is too strong.\n\nMore fundamental: there's no causal control. The paper never compares the ICA mask against a random mask of the same size, a mask from activation magnitudes, or a mask with neuron labels permuted. Without those, the results are consistent with the possibility that almost any 80% subset performs similarly. The threshold that converts the CanICA source matrix into a binary mask is never reported, and the paper doesn't state how the 20%/30% target is enforced. That's not just a reproducibility gap—it hides whether the central mechanism has any effect. The hyperparameter choice also looks circular: n_components is set to 128, but the sensitivity table shows 256 gives lower perplexity (18.47 vs 18.94) on Wikitext-2, the same dataset used for evaluation. No error bars or variance estimates are provided.\n\nCredit where due: the paper is clearly written, engages with the relevant pruning literature, and the discussion acknowledges limitations (local vs global analysis, linearity of ICA). It's not a careless submission—it's an interesting idea with weak supporting evidence.\n\nRecommendation: send it to peer review rather than desk-reject, because the method is novel and the missing controls are fixable. A revision reporting the threshold, adding the ablations, and toning down the SOTA claim could turn this into a decent paper. As is, I wouldn't rely on the empirical conclusions.","headline":"Interesting idea, weak evidence: the ICA-based pruning masks aren't shown to beat simple baselines, and the SOTA claim doesn't survive the paper's own numbers.","tokens_in":13518,"tokens_out":3497,"would_cite":false,"duration_ms":35616,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that LLM pruning by preserving ICA-discovered functional networks keeps performance closer to the original than importance-based pruning.","keywords":["structured pruning","large language models","functional networks","independent component analysis","CanICA","MLP pruning","model compression","brain-inspired AI"],"falsifier":"Prune LLaMA2-7B at 20% using the paper's pipeline but sweep the binarization threshold from the 10th to the 90th percentile of the source signal values while holding the retained neuron count fixed; if Wikitext-2 perplexity and zero-shot scores stay flat across thresholds, then the functional-network selection itself is not doing the work, and any mask of the same size would perform the same.","tokens_in":12655,"feed_emoji":"🧠","tokens_out":9647,"duration_ms":100369,"temperature":0.7,"pith_summary":"This paper tries to establish that a large language model can be pruned more faithfully by treating its MLP neurons as though they were brain voxels and identifying 'functional networks'—groups of neurons that activate together—with independent component analysis (CanICA). Existing structured pruners score each neuron or structural unit and cut the low scorers, which the authors argue ignores how neurons collaborate; removing the wrong neurons can tear the model's macro functional architecture even when each removal looks harmless. The proposed pipeline collects gate- and up-projection activations on calibration text, z-scores them, decomposes each layer with CanICA into 128 components, thresholds the component maps into binary masks, and OR-aggregates masks across 3200 samples to decide which MLP neurons survive. On LLaMA2-7B-chat, Vicuna-7B-v1.5 and ChatGLM3-6B-base at 20–30% pruning, this selection matches or beats LLM-Pruner, FLAP, SliceGPT and Shortened LLaMA on most zero-shot tasks and gives lower Wikitext-2 perplexity in most settings. If the claim holds, co-activation groupings rather than individual importance scores are the right unit for structured pruning, which would make LLM compression less destructive and give a data-driven map of where model functions actually live.","feed_headline":"Keeping co-active neuron groups cuts LLM pruning loss","feed_subtitle":"Layer-by-layer CanICA finds MLP neurons that fire together; keeping them limits perplexity loss at 20-30% pruning.","key_machinery":"CanICA (canonical independent component analysis), a spatial ICA algorithm developed for fMRI analysis. The observed signals are z-scored outputs of the gate and up projections; the source signal matrix $S$ has one row per functional network and one column per neuron, and thresholding $S$ yields binary masks. The machinery's job is to convert raw co-activation statistics into a discrete pruning decision, and the OR-aggregation across runs turns unstable single-batch estimates into a global mask.","core_discovery":"The central discovery argued for is that LLMs contain spatially independent functional networks in their MLP hidden states, and that preserving a superset of neurons belonging to these networks—rather than pruning by per-neuron importance—keeps the model's knowledge and reasoning largely intact. Concretely: the outputs of the gate and up projections are z-scored per neuron, stacked into a signal matrix, and decomposed by CanICA into 128 components per layer per batch of 40 calibration samples. Each component row is thresholded to a binary mask; OR-aggregating masks over 3200 samples and multiple runs yields a global mask that selects which MLP neurons survive. The paper reports that at 20% a","pith_inferences":["A global, cross-layer version of the same idea is the natural next step the authors flag: since information flows through Transformer blocks, functional networks probably span layers, and preserving those might cut performance loss further.","The thresholding step is underdetermined in the paper: the threshold on the source signal matrix is never reported or ablated. Until a threshold sweep shows that mask content, not just neuron count, drives the gains, the method's stated mechanism is not fully pinned down.","The same networks could double as an interpretability atlas: labeling each ICA component with a larger model's explanations would test whether these functional networks correspond to coherent behaviors, and would give pruning a semantic rather than purely statistical justification.","A nonlinear decomposition (the authors mention autoencoders) on the same activation matrices would show whether the linear ICA assumption hides important neuron collaborations; if nonlinear networks find different essential sets, the linearity of CanICA is a real limitation."],"forward_implications":["At 20% pruning, the CanICA mask beats the compared importance-based pruners on most of the seven evaluation tasks for both Vicuna-7B and LLaMA2-7B, and it lowers Wikitext-2 perplexity relative to FLAP and LLM-Pruner in nearly all settings.","At 30% pruning the method still holds the best or near-best scores on PIQA and WinoGrande and keeps perplexity far below SliceGPT and Shortened LLaMA, showing that the grouping criterion degrades more gracefully than block- or PCA-based pruning.","The number of ICA components is a real hyperparameter: 128 or 256 components produce the lowest perplexity, while 512 components degrade performance, so the method has a working range rather than a single magic setting.","Calibration data need scales: perplexity falls as samples increase from 40 to roughly 1500 and then plateaus or worsens, meaning the method has an optimal data budget rather than 'more is always better'.","Because the procedure runs layer-by-layer on MLP activations only, it leaves attention and embedding structure untouched, so it can be combined orthogonally with depth pruning methods."],"supporting_citations":[{"why":"Supplies the CanICA spatial-ICA algorithm that the whole method is built on; the paper's pruning masks come from thresholding CanICA source signals.","marker":"Varoquaux et al. 2010b,a"},{"why":"LLM-Pruner, the main group-based width-pruning baseline; the paper's motivation (groups of coupled neurons) and its comparison target.","marker":"Ma, Fang, and Wang 2023"},{"why":"FLAP, the fluctuation-based importance pruning baseline that the paper compares against with and without its bias-compensation term.","marker":"An et al. 2024"},{"why":"SliceGPT, the PCA-based pruning baseline; its variance-only criterion is the contrast for the paper's functional-relevance criterion.","marker":"Ashkboos et al. 2024"},{"why":"Wikitext-2, used both as the calibration dataset for extracting neuron signals and as the perplexity evaluation dataset.","marker":"Merity et al. 2016"},{"why":"Supplies the evaluation harness used to measure zero-shot accuracy on PIQA, HellaSwag, WinoGrande, OpenBookQA, ARC-E and ARC-C.","marker":"Gao et al. 2024"},{"why":"Provides the LLaMA2 architecture and weights used in the main pruning experiments.","marker":"Touvron et al. 2023"},{"why":"Provides the Vicuna-7B-v1.5 model used as the second main experimental subject.","marker":"Chiang et al. 2023"},{"why":"Supplies ChatGLM3-6B-base, the third architecture used to test transferability of the method.","marker":"GLM et al. 2024"},{"why":"Establishes the functional brain network concept that the paper applies by analogy to LLM neurons.","marker":"Bullmore and Sporns 2009"}],"fun_headline_variants":["Prune LLMs by saving neuron teams, not lone stars","Functional networks guide LLM pruning with less loss","CanICA spots neuron clusters to prune LLMs smarter","Preserve neuron groups to slash LLM pruning damage","Pruning LLMs? Keep functional networks intact"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that the numerical weights CanICA assigns to neurons can be turned into a yes/no list of which neurons to keep; the paper never reports the cutoff used to make that turn and never checks whether the weights, rather than just the number of kept neurons, cause the gains.","fun_headline_variants_meta":{"raw":{"variants":["Prune LLMs by saving neuron teams, not lone stars","Functional networks guide LLM pruning with less loss","CanICA spots neuron clusters to prune LLMs smarter","Preserve neuron groups to slash LLM pruning damage","Pruning LLMs? Keep functional networks intact"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000174,"raw_usage":{"total_tokens":1122,"prompt_tokens":753,"completion_tokens":369,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":293}},"tokens_in":497,"tokens_out":369,"duration_ms":4434,"temperature":1.0,"reasoning_tokens":293,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T23:27:54.106186+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Prune LLaMA2-7B at 20% using the paper's pipeline but sweep the binarization threshold from the 10th to the 90th percentile of the source signal values while holding the retained neuron count fixed; if Wikitext-2 perplexity and zero-shot scores stay flat across thresholds, then the functional-network selection itself is not doing the work, and any mask of the same size would perform the same.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the evaluation harness used to measure zero-shot accuracy on PIQA, HellaSwag, WinoGrande, OpenBookQA, ARC-E and ARC-C."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the functional brain network concept that the paper applies by analogy to LLM neurons."}],"review_version":1}