{"id":"b7decc74-c899-4d30-ac30-85313195d066","arxiv_id":"2607.22720","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"A static pruning method ranks attention and MLP modules by how much deleting them shifts the final token distribution, then encodes that ranking into learned scalar gates for zero-overhead inference.","lead":"CausalGate proposes to decide which parts of a language model can be safely switched off by temporarily deleting each module and measuring how much the model's predictions change. The authors report better pruning trade-offs than standard heuristics, but their headline language-modeling figures are undermined by using the same data for training, tuning, and evaluation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CausalGate's static top-K mask assumes one-at-a-time intervention scores form an input-independent, additive importance ordering; no experiment tests joint removals, so the central 'global importance' claim is unsupported.","rationale":"The central assertion is that intervention-derived KL gives a true global importance ordering. That assertion is what would have to be true for the method to be a principled alternative to heuristics. The paper's empirical section cannot establish this because the one-at-a-time scores are never validated against the actual joint masks used at inference. The reader's weakest_assumption captures exactly this gap. I agree with that assessment. I set aside the WikiText-2 leakage as an additional, independent flaw; however, the interaction problem is more load-bearing for the conceptual contribution: if joint effects dominate, the top-K selection is not a distillation of 'true' importance but an arbitrary heuristic, regardless of how carefully the evaluation is de-contaminated. The concrete test—greedy vs. top-K and pairwise additivity—directly settles it. Since no such test is present, the central claim is unsupported and the REJECT verdict stands.","tokens_in":13123,"tokens_out":8890,"duration_ms":89988,"concrete_test":"On TinyLlama-1.1B, at budgets ρ=0.10 and ρ=0.20, compare three masks using C4 PPL (held out): (i) CausalGate's top-K by learned gates; (ii) greedy backward elimination—start from the full model and iteratively zero the module whose removal increases C4 PPL least given the current active set; (iii) 20 random masks. If the greedy mask's PPL is >10% relative lower than CausalGate's, the marginal ranking is not a valid ordering for joint removal. Additionally, for 20 randomly selected module pairs, compare KL(zero both) with KL(zero i)+KL(zero j); large discrepancies directly falsify the additive assumption.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim depends on an untested additivity/order-stability assumption. CausalGate estimates each module's importance by zeroing it alone (Eq. 1, Algorithm 1), then distills these marginal effects into scalar gates and at inference removes the lowest-gate modules jointly (Eq. 12). For the resulting top-K mask to be the 'true global importance hierarchy,' the marginal ordering must be preserved under simultaneous removals: if module A is less important than B when all else is intact, A must remain less important after other modules have already been zeroed. Nothing in the method tests this. The paper's only related admission is input-specific variability (Section 5); it never considers module-module interaction. At 20% budget, 9 of 44 modules are removed together, so interactions are not a small-correction regime. If the ranking is not additive, the reported Table 1 trade-offs reflect the calibration set, not a general property, and the central claim fails even before considering the WikiText-2 training/evaluation overlap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CausalGate, a post-hoc structural pruning method for Transformer LLMs. During a calibration phase, each attention and MLP sub-module is individually zeroed out, and its importance is measured by the KL divergence between the resulting final-token distribution and that of the intact model (Eq. 1). These intervention scores are normalized, floored, and smoothed with an EMA, then distilled into a single learnable scalar gate per module using MSE, pairwise ranking, sparsity, and language-modeling losses (Eqs. 2–10). At inference, a static top-K mask is derived from the learned gate values (Eq. 12), so no per-input routing is needed. Experiments on TinyLlama-1.1B, Qwen2.5-3B, and Llama-3.1-8B report perplexity on WikiText-2/C4/PTB and accuracy on HellaSwag, PIQA, CSQA, and WinoGrande, along with measured latency speedups. The central claim is that one-at-a-time intervention-based KL importance defines a 'true global' module hierarchy that, once distilled into static gates, consistently outperforms dynamic-routing and layer-skipping baselines at 5–40% module removal.","tokens_in":13372,"tokens_out":3511,"duration_ms":34024,"significance":"If the empirical claims were valid, the paper would make a useful contribution to LLM structural pruning: it proposes a conceptually clear intervention-based criterion, operates at the finer attention/MLP sub-layer level, and demonstrates real hardware speedups at 10–20% removal. The idea of distilling costly intervention scores into static scalar gates is attractive and directionally plausible. However, the current evidence is not sufficient to support the central claims. The evaluation conflates training and test data (WikiText-2 is used both for calibration/gate training and for evaluation), hyperparameters are selected on the same test metric, the headline assertion of consistent improvement is contradicted by multiple rows of Table 1, and the key additivity assumption behind the static top-K mask is never tested. These are not cosmetic issues; they directly undermine the paper's main conclusions. The paper does not provide code, error bars, or ablations that would allow a reader to separate genuine gains from calibration leakage.","major_comments":[{"comment":"Training/evaluation leakage is explicit: Table 4 lists WikiText-2 as both 'Training + Evaluation,' and the CausalGate objective (Eq. 10) includes an LM loss λ_LM L_LM that is trained on calibration data, which is evidently WikiText-2. The reported WikiText-2 perplexities in Table 1 are therefore not held-out measurements. In addition, Table 3 selects hyperparameters using WikiText-2 perplexity at the 10% removal budget, so the WikiText-2 column is further optimistically biased by model selection on the test set. The C4 and commonsense results partially mitigate this, but the central demonstration of quality preservation relies heavily on the WikiText-2 column, and the paper does not report a valid held-out language-modeling evaluation.","section":"§4, Table 4 and §3, Eq. (10)"},{"comment":"The central methodological assumption is that the marginal, one-at-a-time intervention scores of Eq. (1) form an additive, order-stable importance hierarchy that remains valid when many modules are zeroed simultaneously to form the static top-K mask of Eq. (12). Nothing in the paper tests this. At 20% removal, 9 of 44 modules are skipped together; module-module interactions can then easily reorder marginal importance. The only limitation acknowledged in §5 is input-specific variability, not interaction effects. Without a joint-removal experiment comparing the marginal ranking against greedy or jointly optimized rankings, the paper's claim that the learned gates encode the 'true global computational importance' is unsupported.","section":"§3, Eq. (1) and Eq. (12)"},{"comment":"The abstract claims CausalGate 'consistently outperforms' all baselines, but Table 1 contains counterexamples even at the lowest budget. At 5% removal, CALM Hidden-State achieves PIQA 0.734 vs CausalGate 0.723; at 30%, CALM (Softmax) and CALM (Hidden-State) both achieve PIQA 0.600 vs CausalGate 0.540; at 40%, CALM Hidden-State achieves WikiText PPL 1472 vs CausalGate 1365 and PIQA 0.570 vs 0.500. These are not matched with error bars, and the single omnibus Wilcoxon test over task-wise profiles does not establish 'consistently' superior behavior. Moreover, the baselines are not obviously matched: CALM and MoD are dynamic, per-input policies, while CausalGate removes the same modules for every input, so the comparison at a fixed 'architectural savings tier' is not apples-to-apples and the paper does not specify how the tiers were enforced for dynamic baselines.","section":"Table 1"},{"comment":"The paper reports no variance or replication information for any number in Table 1 or Table 2. A single run on a single V100 with no seeds, no confidence intervals, and no code release makes it impossible to assess whether the reported differences are meaningful. Given that Table 3 shows small perplexity differences across many hyperparameter settings (e.g., 18.44 vs 18.45), noise may be comparable to the reported improvements. The lack of reproducibility artifacts is a serious limitation for an empirical paper whose core contribution is a specific performance comparison.","section":"§4, Experimental Setup"}],"minor_comments":[{"comment":"The notation P(Y | do(M3 = 0), X) is not a standard Pearl do-operator: the intervention is a concrete computational change (zeroing a residual branch), not a graph surgery with identifiable causal estimands. Using 'do' language is misleading unless the causal assumptions are formalized.","section":"Appendix C, Figure 9"},{"comment":"Equation (1) defines the score for a single input x and a single forward pass, but the text and Figure 10 refer to 'average KL' over a calibration set. The paper should specify how scores are aggregated over sequences, token positions, and batches; this matters because the method is described as 'input-agnostic' without formally demonstrating that the ranking is stable across inputs.","section":"§3, Eq. (1)"},{"comment":"The text says Figure 7 verifies 'multiple compression tiers,' but the figure is described as reporting only approximately 5% and 10% budgets. Either report the full set of tiers or adjust the description.","section":"§5, Figure 7"},{"comment":"Latency and throughput are reported as single numbers with no measurement protocol or repetition. It is unclear whether these are median, mean, or best-of runs, and whether the speedup comes from skipping modules in a memory-bound regime or from other implementation details.","section":"Table 2"},{"comment":"The Spearman (0.78) and Pearson (0.51) correlations in Figure 5 are reported without confidence intervals or statistical tests. The claim that gates 'recover much of the causal hierarchy' is not substantiated by a single correlation point.","section":"§5"}],"recommendation":"reject","confidential_remarks":"The paper has a promising idea, but the evaluation as presented is not usable: WikiText-2 is both calibration/training and test data, hyperparameters are tuned on the same test metric, and the one-at-a-time-to-top-K additivity assumption is untested. Even the 'consistent outperformance' claim is contradicted by Table 1. These are fixable in principle, but would require a substantially different experimental setup and additional experiments; as it stands, the manuscript does not meet the standard for publication. I would not encourage a quick resubmission without the core evaluation being redone."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is reasonable and worth a serious look: instead of using activation magnitudes or similarity heuristics to decide which transformer sublayers to skip, zero out each attention/MLP module's output, measure the KL divergence in the final logits, and distill that ranking into static scalar gates for zero-overhead inference. That is a legitimate extension of existing block-influence and learned-gating work, and the sub-layer granularity plus the EMA+ranking distillation is a genuine, if modest, assembly. I give the authors credit for framing the problem cleanly, reporting real hardware latency numbers, and being explicit about their limitations section.\n\nThe problem is the evidence. The most damaging issue is that WikiText-2 is used both to train the gates (through the language-modeling loss and the OFAT hyperparameter sweep) and to report the headline perplexity in Table 1. Those numbers are not held out, so the central claim of “consistent outperformance” on language modeling is unsupported as stated. On the downstream tasks, the claim also overreaches: CALM Hidden-State beats CausalGate on PIQA at 5% and 20%, and on WinoGrande at 10%, which contradicts “consistently outperforms.” There are no error bars, no multiple seeds, and no code, so I can’t tell how much of the margin is noise or hyperparameter luck. The baselines may also be weakly implemented, but without code or detailed protocols I can’t verify.\n\nThe stress-test note about additivity is a real soft spot. The method assumes that one-at-a-time importance scores remain valid when several modules are removed jointly. At 20% budget, that’s 9 out of 44 modules dropped together, so interactions are not negligible. The paper doesn’t test this, and the limitation section only mentions input variability, not module interactions. That said, this is an empirical question, not a logical contradiction; the method could still work well in practice even if the ranking isn’t perfectly additive.\n\nOn the plus side, the paper is clearly written, the algorithm is concrete, and the idea of distilling intervention scores into static gates is practical. I’d like to see the authors hold out WikiText-2 (or use a separate validation split for OFAT), report seeds and error bars, release code, and compare directly against block-influence methods like ShortGPT at matched sub-layer granularity. If those results hold, the paper would be a solid contribution to the pruning literature.\n\nThis deserves a serious referee rather than a desk reject, because the method is plausible and the empirical setup is fixable. I wouldn’t cite it in its current form, but I’d read a revised version carefully.","headline":"A sensible intervention-based pruning idea with a competent pipeline, but the empirical evidence as presented doesn't support the headline claims due to evaluation leakage and missing controls.","tokens_in":13892,"tokens_out":1320,"would_cite":false,"duration_ms":16250,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"To prune a language model safely, CausalGate does not observe activations—it intervenes: zero out each module's output, measure the shift in the final token distribution, and distill that causal ranking into a static gate.","keywords":["causal intervention","transformer module pruning","layer skipping","KL divergence","static gates","LLM inference acceleration","attention and MLP sub-layers","compute-efficient inference"],"falsifier":"A concrete test: compute CausalGate's individual module KL scores; then, for a given budget, skip the lowest-ranked modules jointly and measure the actual output KL against the unpruned model. If joint removal causes much larger distribution shifts than the individual scores predict—or if a ranking explicitly computed from leave-K-out interventions gives better downstream accuracy at the same budget—the global additive ranking assumption is falsified.","tokens_in":12961,"feed_emoji":"✂️","tokens_out":4871,"duration_ms":40595,"temperature":0.7,"pith_summary":"CausalGate argues that the right way to decide which transformer sub-modules can be safely skipped is to intervene, not observe: zero out each attention or MLP module's output, run the model, and measure how much the predicted next-token distribution shifts, quantified by KL divergence. The paper contends that this causal score captures non-linear structural computations that activation-magnitude or similarity-based heuristics miss, so it should be the basis for structural pruning. It then distills the intervention scores into a fixed set of static per-module gates, trained once with EMA smoothing and a pairwise ranking loss, so inference is a straight-line execution with no routing overhead. Across language modeling and commonsense reasoning benchmarks on three model families, CausalGate is claimed to hold up better than dynamic routing and layer-skipping baselines at 5–40% module removal, with measured 1.08–1.20x latency speedups at 10–20% removal.","feed_headline":"Zero a module, measure the damage: better LLM pruning","feed_subtitle":"CausalGate ranks modules by how much their removal shifts predictions, then skips the redundant ones at zero routing cost.","key_machinery":"The central object is the causal importance score Δ_i = D_KL(p(y|x, m_i→0) ∥ p(y|x)): the KL divergence between the final logit distribution when module i's residual output is forced to zero and the original distribution. This score is computed per module over a calibration batch, normalized and EMA-smoothed to stabilize rankings, and then encoded into learned scalar gates g_i = σ(θ_i); a pairwise ranking loss forces the gates to respect the intervention-derived ordering. At inference, modules above a top-K cut are kept, the rest skipped.","core_discovery":"CausalGate's central claim is that a transformer module's true global importance is its causal influence on the final logit distribution: zero out the module's residual output, run the model, and take the KL divergence between the intervened and original token distributions. The paper argues this intervention-based score is a better pruning criterion than observational heuristics such as activation magnitudes or hidden-state similarity, because it captures non-linear structural computations. It then shows that these scores can be distilled into a fixed ranking of static scalar gates—one per attention or MLP sub-module—trained with EMA-smoothed targets and a pairwise ranking loss, so that inf","pith_inferences":["The paper leaves input-specific importance untested; a natural extension is to compare the global gate ranking against input-conditioned rankings to quantify when the fixed mask sacrifices accuracy for simplicity.","Because interventions are applied one-at-a-time, interaction effects among simultaneously skipped modules are never measured; a leave-multiple-out diagnostic would reveal whether the additive ordering assumption holds at aggressive budgets.","The calibration cost (one forward pass per module, per batch) is not reported; a deployment-focused extension would measure total calibration wall-clock and whether fewer intervention steps could produce the same ranking."],"forward_implications":["If causal importance is the right criterion, static module selection can replace dynamic routers, eliminating branch-prediction and routing overhead while getting comparable or better quality at matched compute budgets.","Treating attention and MLP blocks as independently prunable units gives finer-grained trade-offs than uniform layer dropping, because importance is concentrated in a subset of MLPs while many later-layer attentions are near-redundant.","The learned gate ranking (Spearman 0.78 against intervention scores) means expensive per-module zeroing is needed only once, at calibration; deployment runs the frozen mask.","Measured speedups of 1.08–1.20x at 10–20% removal indicate the theoretical FLOP savings convert to real latency gains on current hardware."],"fun_headline_variants":["Zero out modules, measure KL damage to prune LLMs","Causal importance beats heuristics for transformer pruning","Intervene, measure logit shift, then prune static gates","CausalGate: causal scores for zero-overhead LLM pruning","Zero modules to rank them, then distill into gates"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that one-at-a-time interventions over a calibration batch produce a fixed input-independent ranking that stays valid when many modules are skipped together.","fun_headline_variants_meta":{"raw":{"variants":["Zero out modules, measure KL damage to prune LLMs","Causal importance beats heuristics for transformer pruning","Intervene, measure logit shift, then prune static gates","CausalGate: causal scores for zero-overhead LLM pruning","Zero modules to rank them, then distill into gates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000176,"raw_usage":{"total_tokens":1109,"prompt_tokens":709,"completion_tokens":400,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":453,"completion_tokens_details":{"reasoning_tokens":319}},"tokens_in":453,"tokens_out":400,"duration_ms":3967,"temperature":1.0,"reasoning_tokens":319,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T12:23:46.161877+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: compute CausalGate's individual module KL scores; then, for a given budget, skip the lowest-ranked modules jointly and measure the actual output KL against the unpruned model. If joint removal causes much larger distribution shifts than the individual scores predict—or if a ranking explicitly computed from leave-K-out interventions gives better downstream accuracy at the same budget—the global additive ranking assumption is falsified.","supporting_citations":[],"review_version":1}