{"id":"5903b664-5892-46d1-a1f9-9fae92851b67","arxiv_id":"2607.17568","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A training-free joint attention+FFN pruning method whose Fisher-derived interaction matrix, recovered from M single-unit ablations as a Gram product, beats node-only saliency baselines on 3B–24B models.","lead":"CoCurve picks which parts of a large language model to cut by measuring the joint damage of removing pairs of units, not just each unit's individual importance, using cheap single-unit probes and one quadratic optimization. On 3B–24B models it keeps code and knowledge stronger than nine training-free baselines, especially when a model's feed-forward channels are not redundant.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-ablation additivity (Eq. 5) is the linchpin of the Gram estimator; a direct pairwise ablation test is needed to confirm H is not a phantom at moderate ρ.","rationale":"The reader's weakest_assumption is exactly the single most load-bearing concern: the entire estimator reduces to a Gram product under an additivity approximation that is never directly validated. The paper's surrogate-fidelity experiments (Fig. 6–7) are suggestive but not decisive: Fig. 6 is a cross-ratio check with perfect monotonicity that could hold even if H were wrong at fixed budget; Fig. 7 is one model, 26 sets, and the strong Spearman is inflated by random sets being far worse. Neither test directly checks whether δz_u + δz_v approximates the simultaneous-removal perturbation for the specific pairs the method selects. The Qwen2.5-32B result shows that turning on edges can sharply degrade calibration KL, which the paper attributes to redundancy; but it also demonstrates that the edge term can mislead, and the gate is a post-hoc fix. If additivity fails on low-redundancy models at moderate ρ, the central claim that off-diagonal entries are co-pruning curvature edges is false, and the reported gains would be empirical accidents of the greedy solver rather than consequences of the derived geometry. The proposed pairwise ablation test directly measures the additivity residual and the fidelity of H_uv against actual joint damage, thereby settling whether the Gram is a phantom. I agree with the reader's CONDITIONAL verdict: the method is promising and honestly reported, but this assumption is insufficiently verified to warrant ACCEPT.","tokens_in":40385,"tokens_out":6232,"duration_ms":54359,"concrete_test":"On Llama-3.1-8B-Instruct and Mistral-7B, sample K=200 unit pairs stratified by type (attn-FFN, attn-attn, FFN-FFN) and layer distance. For each pair, compute δz_u (mask u), δz_v (mask v), and Δz_{u,v} (mask both) on the same 128 C4 sequences used for calibration, measuring the whitened Fisher-norm residual ||Δz_{u,v} − (δz_u + δz_v)||_F / ||Δz_{u,v}||_F, and compare H_uv to the measured joint-interaction term δz_u^T F δz_v. Report median and 90th-percentile relative error, separately for pairs actually co-selected by CoCurve at 20% and 30% pruning. If the median error exceeds ~0.1 among co-selected pairs, the Gram surrogate is unreliable at moderate ρ and the central claim fails; if it is below ~0.05, the additivity concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that H's off-diagonal entries are co-pruning curvature edges recovered from M single-unit ablations as a Gram product (Abstract; Prop. 4, Eq. 6) — rests on the additivity approximation Δz_s ≈ Σ_u s_u δz_u (Section 2.2, Eq. 5). This is a surrogate, as Remark 2 concedes, and the error is only bounded by O(‖s‖³₁) with an unchecked constant (Prop. 5). The paper's fidelity evidence (Figs. 6–7) is on a single model (Llama-3.1-8B-Instruct) and Fig. 7's Spearman 0.81 across 26 sets is largely driven by random vs. structured sets; it does not establish that pairwise H_uv accurately predicts the extra damage of removing u and v together, nor that additivity holds for the cross-module, cross-layer pairs that the method claims to model. If the logit perturbation from jointly masking two units deviates significantly from the sum of single-unit perturbations at moderate ρ, H is a phantom: the off-diagonal entries are not co-pruning curvature, and the greedy QP optimizes a wrong objective. The redundancy gate (Section T) empirically damps λ on high-redundancy models, but a gate cannot repair a surrogate that misranks low-redundancy architectures where additivity might still fail. This is the load-bearing assumption of the entire method.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CoCurve, a training-free structured pruning method for LLMs. It defines pruning risk as the token-level KL divergence between the frozen model and its masked copy, and shows that a second-order expansion yields a Fisher matrix H whose diagonal is node saliency and whose off-diagonal entries are co-pruning curvature edges. Under a single-ablation additivity approximation, H is recovered as a Gram product of M single-unit ablation feature vectors, avoiding pairwise sweeps or gradients. Pruning is then solved as a one-shot budgeted quadratic program with a greedy solver. Experiments on four 3B–8B models and a 24B scaling study compare CoCurve against nine training-free baselines, reporting strong results on perplexity and code generation, with ablations attributing the gain to cross-module edges and a calibration-derived redundancy statistic gating the edge strength.","tokens_in":40605,"tokens_out":7548,"duration_ms":67753,"significance":"If the surrogate is valid, CoCurve is a meaningful advance: it models cross-module, cross-layer interactions at O(M) cost without labels, gradients, or recovery, and the empirical program is unusually thorough. The paper includes honest admission that H is a surrogate (Remark 2), a non-trivial same-budget surrogate discrimination test (Fig. 7, Spearman 0.81 across 26 sets), a matched-pair causal test for bridge units (Table 4), and extensive robustness checks on calibration seed, source, and budget. These are real strengths. However, the central additivity assumption is not directly tested, and the predictive redundancy gate is partly circular because its threshold is extracted from the same measurements used to validate it. The manuscript needs revision before these claims are fully supported.","major_comments":[{"comment":"The load-bearing claim is that off-diagonal entries H_uv are co-pruning curvature edges, recovered by a Gram product of single-unit ablation features. This equality holds only under single-ablation additivity Δz_s ≈ Σ_u s_u δz_u. Proposition 5 bounds the error by O(||s||_1^3) with an unchecked constant, and Remark 2 explicitly concedes that H is a surrogate. The fidelity evidence is set-level: Figure 7 validates ranking of 26 pruning sets at fixed budget, not the individual pairwise entries H_uv, and Figures 6–7 are only on Llama-3.1-8B-Instruct. I request a direct pairwise test: for representative cross-module and same-module pairs (including solver-selected pairs), compare the measured joint logit perturbation or joint KL damage with the additive prediction δz_u + δz_v at moderate ρ, and report whether the Gram ranking of pairs is preserved. Without this, the 'co-pruning curvature edge","section":"Section 2.2, Eq. (5), Prop. 5; Section R, Figs. 6–7"},{"comment":"The claim that a single calibration statistic predicts a priori when edges help is weakened by how the gate is set. The threshold |corr|_FFN ≳ 0.12 is introduced after presenting Table 21, and the same table is then used as evidence for the rule's predictive power. This is circular: the boundary is fit to the data used to validate it. The term 'pre-registered' is not supported by any protocol fixed before the measurements. To make the contribution stand, the threshold should be set on a discovery subset (or derived analytically) and then evaluated on held-out families; alternatively, report the continuous edge-benefit-versus-|corr| relationship and avoid a hard threshold. As written, the gate is a post-hoc tuning rule.","section":"Section T, Table 21"},{"comment":"The statement that the edge term 'never' goes below the OBD diagonal floor is too strong. The λ=0 endpoint is exactly the diagonal selector, but for λ>0 the edge term can hurt, as shown by the forced-λ=1 rows in Table 23 and the negative edge benefits in Table 21. The method's actual protection comes from damping λ, not from a guaranteed floor. Relatedly, Proposition 1 assumes as an axiom that the third-order remainder T(s) ≥ 0 on the feasible set ('co-removal is on average super-additively harmful'); this is not derived and may be violated when H has strong negative off-diagonals. The empirical λ sweep is suggestive, but the proposition should be labeled as conditional on that assumption, and Falcon3's λ=0.5 should be acknowledged as a model-specific operating point rather than 'not a free hyperparameter.'","section":"Section 2.4, Prop. 1; Section 3.4"}],"minor_comments":[{"comment":"The Limitations section says 'a complete no-recovery comparison at ≥13B is compute-bound and left to future work,' but Section T and Table 14 report a complete 14-task no-recovery protocol on Mistral-Small-24B at all ratios. Please reconcile this inconsistency.","section":"Section 6 vs. Section T"},{"comment":"Spearman ρ=1.00 over five ratio points is not informative by itself; the meaningful same-budget test is Figure 7. Consider reporting per-ratio Spearman or more sampled sets.","section":"Figure 6"},{"comment":"The sign-handling ablation reports that signed, clipped, and absolute variants coincide on Llama-3.1-8B-Instruct. This does not demonstrate that signed handling matters; please state this explicitly and, if possible, show a model where signs differ.","section":"Table 18"},{"comment":"The statement that 'not one of the 2.36M off-diagonal correlations exceeds 1' is a tautology for correlation values; remove or replace with a conditioning or numerical-range check.","section":"Table 21"}],"recommendation":"major_revision","confidential_remarks":"The paper is substantive and the empirical effort is unusually thorough, but two load-bearing issues need to be fixed before I can support acceptance: (i) direct validation of single-ablation additivity for pairwise off-diagonals, and (ii) de-circularizing the redundancy-gate threshold. The internal inconsistency about the 24B evaluation should also be corrected. If the authors add the pairwise test and re-frame the gate as a discovery/validation exercise, I would be supportive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read CoCurve carefully, and you should know two things up front. First, it is one of the more honest pruning papers I have seen this year: the surrogate is explicitly labeled as a surrogate, the same-budget discrimination test is the right kind of validation, the bridge-unit causal test is well matched, and the appendix reports losing cells, seed variance, under-pruning confounds, and even recovery-enabled bake-offs where CoCurve loses. Second, the central mechanism — cross-module attention-to-FFN off-diagonal curvature recovered from M single-unit ablations — is genuinely new and is supported by the internal evidence for low-FFN-redundancy models in the moderate regime.\n\nWhat is actually new: the Gram-product recovery of the full interaction matrix from single ablations is elegant, the shared-budget greedy solve is a clean way to let the attention/FFN split emerge, and the redundancy gate (edge benefit scales inversely with FFN redundancy) is a useful architectural rule, even if the threshold is drawn from the same measurements used to demonstrate it. The derivations in Propositions 2-4 and 9-10 check out, and the empirical work is unusually thorough for a training-free pruning paper.\n\nWhere are the soft spots? The biggest one is also the one the stress-test note flags: Equation (5)'s single-ablation additivity is the linchpin, and the paper validates it only at the set/ranking level (Spearman 0.81 on one model, largely separating random from structured sets) plus the bridge causal test. That is decent evidence the surrogate ranks well, but it does not establish that pairwise H_uv accurately predicts the extra damage of joint removal, especially for the cross-module pairs the method is built on. The paper's defense that pointwise interactions are ill-conditioned is reasonable, but a direct pairwise ablation study on a few cross-module pairs would settle it. This is a fixable gap, not a fatal flaw.\n\nThe other concerns are smaller but real. The \"never below the diagonal\" claim in Section 2.4 is only true when the gate damps lambda on high-redundancy models; the gate threshold is fit to the same Table 21 data it predicts, so there is circularity. The Falcon3 lambda=0.5 exception is a benchmark-informed operating point in a method advertised as label-free. GSM8K is omitted from the main table where CoCurve loses to 2SSP, though the paper discloses this in an appendix and the near-floor band is genuine. And the 14.5x headline is a ratio of two collapsed perplexity values at 40% — technically correct but easy to over-read.\n\nOverall: the central argument holds up for its intended regime. The paper deserves a serious referee, and I would want the pairwise ablation check and a clearer pre-registration of the gate threshold before acceptance. I would bring it to reading group and would likely cite it if I were working on structured pruning.","headline":"CoCurve is a serious, unusually honest training-free structured pruning paper whose cross-module edge matrix is a real idea, but the load-bearing additivity assumption needs a direct pairwise check and the 'label-free' story is a bit stretched.","tokens_in":41315,"tokens_out":1852,"would_cite":true,"duration_ms":19399,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Structured LLM pruning should score pairs of units, not single units: the paper derives a Fisher matrix whose off-diagonal entries are the extra damage of removing two units together, and recovers the whole matrix from one forward pass per","keywords":["structured pruning","large language models","co-pruning curvature","Fisher information matrix","attention heads","FFN channel groups","training-free","second-order Taylor expansion"],"falsifier":"Measure additivity directly on a 3B–8B model at 20% pruning: pick 20–30 pairs of attention/FFN units, mask each unit alone and both together, and compare the measured joint token-KL to the value predicted from the Gram H (½(H_uu + H_vv) + H_uv). Alternatively, repeat the same-budget ranking test of the paper — predicted ½ s^T H s versus measured KL over a pool of structured and random mask sets — and check whether the Spearman rank correlation is substantially below the reported 0.81 on a held-out model.","tokens_in":40064,"feed_emoji":"✂️","tokens_out":4768,"duration_ms":42157,"temperature":0.7,"pith_summary":"Structured pruning of large language models is usually done by ranking each attention head or FFN channel group independently, implicitly assuming that the damage of removing a set equals the sum of individual damages. This paper argues that this view fails for Transformers, whose sublayers are coupled through a shared residual stream, and that two individually weak units can be jointly indispensable. The central claim is that the missing information is exactly the pairwise co-pruning curvature: the off-diagonal of a Fisher matrix that comes from a second-order expansion of the token-level KL divergence between the frozen model and its masked copy. The paper shows that this full M-by-M interaction matrix is a Gram product of M single-unit ablation features, so modeling the edges costs no more than computing the diagonal, and that pruning then becomes a single budgeted quadratic program solved greedily. If correct, training-free structured pruning can preserve fragile generative capabilities — code and knowledge, where independent scoring collapses — without labels, gradients, or fine-tuning.","feed_headline":"Edges, not nodes, decide which LLM units to prune","feed_subtitle":"Pairwise Fisher curvature, recovered from one ablation per unit, preserves code and knowledge where independent scoring collapses.","key_machinery":"The load-bearing object is the M×M co-pruning curvature (Fisher) matrix H, defined as the Hessian of the pruning risk R(s) = E[ D_KL(p_0 || p_s) ] at s=0. Its diagonal is an Optimal-Brain-Damage style node saliency; its off-diagonal entries are the joint-removal interaction terms. The efficiency trick is the Gram identity: with whitened single-unit logit-ablation features δz̃_u = √p_0 ⊙ (δz_u − E_{p_0}[δz_u]), each entry satisfies H_uv = (1/P) Σ δz̃_u^T δz̃_v, so the entire matrix is a Gram matrix of M single-ablation feature vectors and costs M forward passes instead of O(M²) pairwise ablations. The solver consumes this matrix through the running marginal risk Δ(u|S) = ½ H_uu + Σ_{v∈S} H_uv","core_discovery":"The paper's central discovery is that the second-order Taylor expansion of a token-level self-distillation KL — the divergence between the frozen full model and its masked copy — yields one Fisher matrix H whose diagonal is exactly the classical node saliency and whose off-diagonal entries are co-pruning curvature edges: the extra distortion of removing two units together. Under an additivity approximation, each entry H_uv equals the Fisher-weighted inner product of the two units' single-ablation logit shifts; after whitening by the teacher probabilities, this becomes a plain Gram product, so the full M×M matrix is obtained from M forward-only ablation passes, with no pairwise sweeps, gradie","pith_inferences":["If the additivity approximation degrades as predicted by the paper's O(∥s∥³) bound, the same Gram machinery could be iterated: re-estimating H on the pruned model in rounds would push the trust region further, turning the one-shot method into an incremental one and potentially extending the usable compression ratio.","The locality result (edges concentrated within and between nearby layers) suggests a modular variant that computes only within- or adjacent-layer blocks, cutting the Gram cost further with little accuracy loss; the paper itself notes this as an efficiency opportunity.","The paper's observed split between calibration KL and downstream capability — stronger edges can hurt perplexity while saving code generation — implies that any single-metric evaluation protocol for training-free pruning can misorder selectors; reporting both a distributional metric and a generative stress test is a testable improvement for future benchmark practice."],"forward_implications":["If the central claim holds, training-free structured pruning is no longer limited to independent rankings: interaction-aware selection is as cheap as the diagonal, so edge-aware co-pruning is affordable in any deployment setting at the same calibration budget.","The Gram reconstruction makes the interaction matrix positive semi-definite by construction, giving a free numerical correctness check that edge-aware pruning ships with.","The measured FFN-redundancy statistic |corr|_FFN, computed from calibration H, becomes a pre-hoc architecture-level rule for whether to trust or damp the edge term, making the method predictable across model families without labels.","At aggressive ratios, the same edge blocks used for selection can be reused for a closed-form output-rescale compensation, reducing the super-linear collapse region without fine-tuning or gradient updates.","Because the λ=0 endpoint of CoCurve is exactly a structured second-order saliency, the method contains a fallback: when edges are not trustworthy on high-redundancy architectures, it degrades gracefully to the classical diagonal selector rather than failing."],"fun_headline_variants":["CoCurve: prune LLM units by interaction curvature","Joint pruning via Fisher off-diagonals, no fine-tuning","Single ablation per unit, full co-pruning matrix","Capture cross-module pruning damage with curvature edges","Training-free pruning that respects unit interactions"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire edge estimator rests on single-ablation additivity: the logit shift from removing several units equals the sum of each unit's individual shift, so the pairwise entries of H are inner products of single-unit ablation features — if this fails at moderate pruning ratios, H is a phantom objective that the greedy solver optimizes.","fun_headline_variants_meta":{"raw":{"variants":["CoCurve: prune LLM units by interaction curvature","Joint pruning via Fisher off-diagonals, no fine-tuning","Single ablation per unit, full co-pruning matrix","Capture cross-module pruning damage with curvature edges","Training-free pruning that respects unit interactions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1215,"prompt_tokens":773,"completion_tokens":442,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":381}},"tokens_in":517,"tokens_out":442,"duration_ms":4393,"temperature":1.0,"reasoning_tokens":381,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T17:36:54.968868+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure additivity directly on a 3B–8B model at 20% pruning: pick 20–30 pairs of attention/FFN units, mask each unit alone and both together, and compare the measured joint token-KL to the value predicted from the Gram H (½(H_uu + H_vv) + H_uv). Alternatively, repeat the same-budget ranking test of the paper — predicted ½ s^T H s versus measured KL over a pool of structured and random mask sets — and check whether the Spearman rank correlation is substantially below the reported 0.81 on a held-out model.","supporting_citations":[],"review_version":1}