{"id":"2afdf175-7f6d-4918-879a-88e7856d7790","arxiv_id":"2507.18807","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The squared gradient accumulator from Adam can serve as a free, drop-in approximation of the empirical Fisher diagonal across model merging, pruning, task embedding, and continual learning, with comparable performance.","lead":"This paper tests whether the squared-gradient memory that Adam-style optimizers already keep during training can replace the Fisher information diagonal, a standard way to measure which model parameters matter. Across six applications the authors find the recycled accumulator performs about as well as the true Fisher, but at zero extra computation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported Squisher-Fisher differences exceed 'little impact' in several settings, and the paper provides no error bars or repeated seeds to separate signal from noise.","rationale":"The reader correctly identifies the EMA-over-trajectory assumption as a conceptual weakness, and the paper's own ablations support that concern. However, the most load-bearing problem for the central empirical claim is the absence of uncertainty quantification: the reported point estimates show differences between Squisher and Fisher that are sometimes large and unstable, and Section A.2 explicitly says the comparison is not definitive. This does not invalidate the paper's idea, which is plausible and well-motivated, but it means the headline conclusion is not yet established. The requested test would directly settle whether the observed gaps are real or within noise. Since this is an addressable empirical gap, the existing CONDITIONAL verdict remains appropriate.","tokens_in":18206,"tokens_out":9048,"duration_ms":99134,"concrete_test":"Re-run the Fisher-merging experiment (Section 3.1) with the same random seeds for the Fisher and Squisher pipelines across at least 5 independent fine-tuning runs per dataset, at both the 'final' and 'best' checkpoints, and report the paired difference (Squisher minus Fisher) with a 95% confidence interval. If the interval for the average difference excludes zero in either direction, the 'little impact' claim is falsified for this setting; if it includes zero, the claim gains support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim ('replacing the Fisher with the Squisher had little impact') rests entirely on the comparisons in Figure 2 and Tables 3-8. These comparisons show large, inconsistent gaps: Fisher merging averages 53.6 vs 58.0 for Fisher vs Squisher, with per-dataset differences up to 23.5 points (ROPES: 12.6 vs 36.1); task-embedding all-class MRR is 0.312 (Fisher) vs 0.261 (Squisher) in Table 7; and Section A.2 reports that switching from final to best checkpoints flips the ranking (Fisher best 59.0 vs Squisher best 55.1) and that pruning at 10 vs 15 epochs changes which method is better. No confidence intervals or repeated-seed statistics are reported for most settings. The paper itself concludes in Section A.2 that 'we cannot definitively conclude which method is superior, as the results exhibit considerable variability.' If the true Squisher-minus-Fisher difference is as large as the observed point estimates in some settings, then the 'drop-in' recommendation is unsafe. The reader's concern about the EMA over the training trajectory is the likely mechanism: beta2 sensitivity (Table 2) and checkpoint sensitivity (Tables 9-10) show the Squisher's value is not anchored to final-parameter Fisher statistics. But the immediate load-bearing gap is that the headline claim is not statistically supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using the squared gradient accumulator maintained by adaptive optimizers such as Adam/AdamW as a zero-cost proxy for the diagonal Fisher information matrix, calling this proxy the Squisher. Section 2 derives an algebraic connection between the Squisher and the diagonal of the joint empirical Fisher (Eq. 12), building on the joint Fisher construction of Lin et al. (2024). Section 3 evaluates the Squisher in six settings: Fisher merging, UBGM merging, Fisher pruning, FISH masking, task embeddings, and EWC continual learning, reporting that the Squisher performs comparably to the Fisher and outperforms Fisher-free baselines. Section 4 ablates the effect of rescaling, the EMA coefficient, and the joint-Fisher construction, and Appendix A.2 examines sensitivity to training duration and checkpoint selection. The paper claims that the Squisher is a practical drop-in replacement for the Fisher in parameter-importance applications at no additional computational cost.","tokens_in":18494,"tokens_out":7965,"duration_ms":76018,"significance":"If the central claim holds, the paper offers a genuinely practical contribution: it removes the need for per-sample gradient computations and training-data access when computing Fisher diagonals for importance-based applications. The derivation in Section 2.3 is clean and parameter-free, and the empirical coverage across five distinct application families is a strength. The runtime measurements in Section 3.8 and Appendix A.3 quantify the savings concretely. However, the significance is tempered by the statistical fragility of the headline claim: most comparisons are single runs, and the paper's own Appendix A.2 documents checkpoint and epoch sensitivities that reverse the relative ordering of the two methods in some settings. The EWC application also relies on an empirically chosen scaling whose derivation is not supplied. These issues are addressable, but they currently limit the strength of the conclusions that can be drawn.","major_comments":[{"comment":"The headline claim that 'replacing the Fisher with the Squisher had little impact on performance' is not statistically supported by the reported evidence. Tables 3, 4, 6, 7, and 8 present single-run results without error bars or repeated-seed statistics; Tables 5 and 10 give five-run averages, and Table 10 shows that changing the number of training epochs flips which method is better. The per-dataset gaps are large in places (e.g., ROPES 12.6 vs 36.1 in Table 3; all-class MRR 0.312 vs 0.261 in Table 7), which is inconsistent with a characterization of 'little impact.' Appendix A.2 itself concludes that 'we cannot definitively conclude which method is superior, as the results exhibit considerable variability.' The paper should either provide confidence intervals over multiple seeds for all settings or explicitly downgrade the conclusion to 'comparable within observed variability,' with a discussion of when the differences are practically consequential.","section":"Section 3.7, Tables 3–8, Appendix A.2"},{"comment":"The scaling of the Squisher in the EWC experiment is inconsistent with the derivation in Section 2.3. The text states that when using mini-batches, one should replace N by the batch size B in all expressions. For the Adam accumulator, whose squared-gradient term is (1/B) times the sum of per-example gradients squared, the quantity in Eq. (12) becomes B times the squared mean gradient, i.e., B(1/B sum g)^2 = (1/B)(sum g)^2. Section 3.6, however, reports that scaling the Squisher by the dataset size N 'provided best performance' and admits the lack of a formal theoretical justification. This is not merely a presentation issue, because EWC is the one setting in which the scale of the importance weights changes the learning behavior. Please clarify the notation (N vs B) and either derive the correct mini-batch scaling or explicitly present the N-scaling as an empirical tuning heuristic with a sensitivity analysis.","section":"Section 2.3, Eq. (12); Section 3.6"},{"comment":"The checkpoint and training-duration sensitivity documented in Appendix A.2 is a load-bearing limitation for the generality of the Squisher. Table 9 shows that switching from final to best checkpoints reverses the relative performance of Fisher merging (final: 53.6 vs 58.0 in favor of the Squisher; best: 59.0 vs 55.1 in favor of Fisher). Table 10 similarly shows that pruning at 15 versus 10 epochs changes which method is better. These results indicate that the Squisher's quality is not anchored to final-parameter Fisher statistics, but depends on the training trajectory and checkpointing protocol. The paper's conclusion should incorporate this dependence rather than presenting the Squisher as a universally safe drop-in replacement.","section":"Appendix A.2, Tables 9–10"}],"minor_comments":[{"comment":"The text states that even with the low beta2 value of 0.95, 'the Squisher continued to outperform the baseline,' but for split CIFAR-100 the reported accuracy is 61.53, which is below the baseline of 61.82 in Table 8. Please correct the statement or the table.","section":"Table 2 (ablation)"},{"comment":"The table header says 'a sparsity of 0.50%' but the setup in Section 3.4 describes masking 50% of the model's weights. Please reconcile the percentage.","section":"Table 6"},{"comment":"The claim that 'the Squisher-based task embedding produced a better mean reciprocal rank than the Fisher-based one' is only true within class groups; for all-class MRR, the Fisher is better for both classification/regression (0.312 vs 0.261) and question-answering (0.350 vs 0.301). Please qualify the claim accordingly.","section":"Section 3.5 and Table 7"},{"comment":"The box labeled 'Squisher Nv(t)' does not define N; as the scaling discussion in Section 2.3 and Section 3.6 uses both dataset size and batch size, the figure should specify which quantity N denotes to avoid confusion.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper has a good practical idea and a clean algebraic derivation, but the empirical support for the central 'drop-in' claim is weaker than the abstract suggests. The single-run results in most tables, the checkpoint/epoch sensitivity in Appendix A.2, and the undocumented EWC scaling are concrete issues that can be fixed with additional experiments and clearer writing. I would not reject, but I would not accept without the authors addressing the statistical support for 'comparable performance' and the scaling inconsistency."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the Squisher is a good idea, the derivation via the joint Fisher is clean, and the paper deserves a real referee. But the headline empirical claim—that replacing the Fisher with the Squisher \"had little impact\"—is not actually supported by the numbers as reported.\n\nWhat's new: Kingma and Ba already asserted that Adam's second moment approximates the Fisher diagonal, and FAdam and IVON poked at the connection, but nobody to my knowledge laid out the relationship this cleanly. Section 2.3 is the real contribution: using Lin et al.'s joint Fisher, the squared mini-batch gradient is exactly the diagonal joint empirical Fisher (up to the N scaling), and the Squisher is just that quantity with an EMA over training. They are honest about the three approximations: EMA vs. final-parameter gradient, sum-then-square vs. square-then-sum, and the N scale. The ablation in Table 2 separating \"joint\" from the full Squisher is a nice touch—it shows the EMA is the main source of deviation.\n\nThe breadth of experiments is a plus: Fisher merging, UBGM, pruning, FISH mask, Task2Vec, EWC. And the runtime numbers make the practical case concrete: the Squisher is genuinely free.\n\nNow the soft spots, and they are real. The central claim of \"comparable performance\" rests on point estimates without error bars in most settings. The stress-test numbers are fair: Table 3 shows Fisher merging averages 53.6 vs. 58.0, and on ROPES the Squisher gets 36.1 vs. the Fisher's 12.6—that is not \"little impact\" unless you believe it's noise, and the paper gives no repeated-seed evidence for most tables. Table 7 shows task-embedding MRR differences in both directions depending on whether you look in-class or all-class. And Section A.2 directly contradicts the summary: switching from final to best checkpoints flips the ranking (Fisher 59.0 vs. Squisher 55.1; final: Squisher 58.0 vs. Fisher 53.6), and the paper itself says \"we cannot definitively conclude which method is superior.\" That is a fair statement, but it means the abstract's \"consistently performs similarly\" is doing too much work. The EWC scaling heuristic is openly unjustified, and for a paper whose whole pitch is \"no extra tuning,\" that is a weakness, though a minor one because λ Fisher is also tuned anyway.\n\nAlso, no code or data is released, which for an empirical paper with this many moving pieces should be a condition of acceptance.\n\nWho is this for? Anyone using Fisher diagonals for parameter importance in merging, pruning, continual learning, or task similarity. The derivation and the negative result—that the approximations mostly don't hurt—are useful even if the error bars would change the tone.\n\nRecommendation: send to peer review, but require repeated seeds and error bars, the checkpoint analysis made front-and-center, and code/optimizer-state release. The idea is sound and the derivation is worth publishing; the empirical claim just needs to be calibrated to the actual variability.","headline":"Clean derivation and a genuinely useful free-lunch idea, but the 'comparable performance' claim is shakier than the paper lets on—no error bars, and its own appendix shows the comparison flips with checkpoints.","tokens_in":19032,"tokens_out":4071,"would_cite":false,"duration_ms":38284,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adam's squared-gradient accumulator, recycled as the 'Squisher', performs as well as the expensive Fisher diagonal across merging, pruning, masking, task embedding, and continual learning.","keywords":["Fisher information matrix","squared gradient accumulator","parameter importance","Adam optimizer","model merging","network pruning","elastic weight consolidation","task embeddings"],"falsifier":"Use the paper's own recipe: train VGG-13 on CIFAR-100 with Adam at $\\beta_2 = 0.95$, or stop training early, then prune 75% of parameters using the Squisher ranking and compare accuracy with Fisher pruning and random pruning; the paper's ablation predicts a sharp drop in the $\\beta_2 = 0.95$ regime, and a configuration where the Squisher's pruned accuracy falls to the random-mask level would falsify the drop-in claim. A second, more direct check is to compute the rank correlation between the Squisher's per-parameter scores and the label-sampled Fisher diagonal on the same checkpoint and see whether a low-correlation regime exists in which any application's performance gap to the Fisher-free baseline disappears.","tokens_in":17998,"feed_emoji":"♻️","tokens_out":19499,"duration_ms":171211,"temperature":0.7,"pith_summary":"Adaptive optimizers like Adam already compute a quantity during training — an exponential moving average of squared gradients — that the paper argues can be recycled as a free approximation of the diagonal of the Fisher Information Matrix, the standard measure of parameter sensitivity used in model merging, pruning, sparse fine-tuning, task embedding, and continual learning. Computing the Fisher diagonal after training is expensive and needs the training data, because it requires per-example gradients on sampled labels; the recycled accumulator, which the paper calls the Squisher, costs nothing beyond loading optimizer state. The paper first pins down what the accumulator actually is — a moving-average version of the diagonal of the joint empirical Fisher, off by a known scaling factor — and then tests the substitution in six settings across five applications. Replacing the Fisher with the Squisher changes performance only slightly, in both directions, while both clearly beat the Fisher-free baselines in every setting. If the finding holds, Fisher-based methods become available for any model trained with an adaptive optimizer, with no extra computation and no need for training data or gradient code.","feed_headline":"Adam's cached gradients can replace the costly Fisher for free","feed_subtitle":"In merging, pruning, and continual learning, the recycled accumulator matches the Fisher at zero cost.","key_machinery":"The load-bearing object is the squared gradient accumulator $v^{(t)} = \\alpha v^{(t-1)} + (1-\\alpha)\\big(\\tfrac{1}{N}\\sum_n g_n^{(t)}\\big)^2$, the exponential moving average of squared mean mini-batch gradients that Adam and its derivatives maintain for learning-rate adaptation. The key identity is structural: the accumulator squares a summed (mean) gradient rather than summing squared per-example gradients, so it is exactly a moving-average version of the diagonal of the joint empirical Fisher information matrix, whose entry is $N\\big(\\tfrac{1}{N}\\sum_n g_n\\big)^2$; the joint Fisher and the standard Fisher coincide (Lin et al., 2024), which lets the paper say the accumulator is a Fisher diagonal of a recognizable kind, differing only in that the average runs over the training trajectory and in the scaling factor $N$. This identity does the argumentative work: it converts the question 'can we reuse the accumulator?' from an analogy into the specific empirical question of whether trajectory-averaging and scaling break the usefulness of an otherwise standard Fisher, which the experiments then answer.","core_discovery":"On its own terms, the paper establishes that the squared gradient accumulator $v^{(t)} = \\alpha v^{(t-1)} + (1-\\alpha)\\big(\\tfrac{1}{N}\\sum_n g_n^{(t)}\\big)^2$, the second-moment estimate Adam keeps for adaptive learning rates, is a workable drop-in replacement for the diagonal of the empirical Fisher, $F = \\sum_n g_n^2$, in applications that use the Fisher as a parameter-importance score. The substitution is not a loose metaphor: the accumulator squares the average gradient over a batch (sum-then-square), which makes it a trajectory-averaged version of the diagonal of the joint empirical Fisher, $N\\big(\\tfrac{1}{N}\\sum_n g_n\\big)^2$, identified with the standard Fisher through the joint/standard equivalence; the two differences are the moving average over training instead of an evaluation at the final parameters, and the factor $N$ in scale. In the experiments, the two methods stay close — the Squisher slightly ahead in Fisher merging of eight fine-tuned T5 variants and in the in-class task-embedding rankings, slightly behind in pruning of VGG-13, and nearly identical in uncertainty-based gradient matching, FISH Mask, and EWC — while both clearly beat the Fisher-free baselines in every setting; in EWC, the one application where rescaling matters, multiplying the accumulator by the dataset size restores comparable behaviour. Computing the Fisher in these settings took between 2.5 seconds and roughly 14 hours, while the Squisher costs under a tenth of a second to load.","pith_inferences":["The training-length sensitivity suggests the Squisher is effectively a Fisher diagonal averaged over a neighbourhood of parameter values near the end of training; the natural experiment the paper does not run is to compare the Squisher against an explicit average of per-checkpoint Fisher diagonals over the final stretch of training, which would separate the contribution of the moving average from ","A cheap hybrid is worth testing: because the Squisher costs nothing, one could compute the true Fisher on a small sample and combine the two importance rankings, potentially reducing the merging and pruning instability the paper attributes to noise without paying the full Fisher cost.","The overall equivalence supports a deflationary reading of the Fisher diagonal in these applications — that what carries the value is per-parameter gradient magnitude rather than the likelihood-sampling definition — which would make any optimizer statistic that tracks per-parameter gradient scale, momentum included, a candidate zero-cost importance score, a claim that is testable in the same exper"],"forward_implications":["Any model trained with Adam, AdamW, or a similar adaptive optimizer already carries a parameter-importance score in its optimizer state, so Fisher-based techniques — merging, pruning, sparse fine-tuning, task embeddings, and EWC — can run with zero extra computation and no second pass over the data.","Fisher-based methods become usable precisely where they were previously impractical: when a released model comes without its training data, or when per-example gradient code is unavailable, the accumulator alone suffices.","The practical caveats are bounded and concrete: the Squisher needs a fully trained model and the default smoothing coefficient, since lowering $\\beta_2$ to 0.95 cut EWC accuracy on split CIFAR-100 from 75.30 to 61.53, and merging from under-trained checkpoints ('best model' selection) makes the Squisher lose its edge.","Applications that are invariant to rescaling can use the accumulator as-is, and EWC only needs the accumulator multiplied by the dataset size $N$, so switching from the Fisher adds no tuning burden beyond the $\\lambda$ search EWC requires anyway.","Because the optimizer state now has value on its own, the paper argues that sharing optimizer statistics alongside released models would let anyone apply Fisher-based analysis to any checkpoint for free."],"supporting_citations":[{"why":"introduces the joint Fisher whose sum-then-square diagonal is exactly the structure of the squared gradient accumulator, supplying the theoretical bridge.","marker":"(Lin et al., 2024)"},{"why":"defines Adam and its squared gradient accumulator, the resource being recycled, and is the source of the informal claim that the accumulator approximates the Fisher diagonal.","marker":"(Kingma & Ba, 2014)"},{"why":"defines Fisher merging with the empirical Fisher diagonal as parameter-importance weights, the flagship application where the Squisher is validated on eight T5-Large models.","marker":"(Matena & Raffel, 2022)"},{"why":"defines elastic weight consolidation, the one application where the Fisher's scale changes behaviour and the Squisher needs the dataset-size rescaling.","marker":"(Kirkpatrick et al., 2017)"},{"why":"formulates Fisher pruning, the application where the Squisher's parameter ranking is compared against the Fisher's and slightly underperforms.","marker":"(Theis et al., 2018)"},{"why":"introduces FISH Mask, the sparse fine-tuning method whose mask selection the Squisher replaces in the BERT-Large GLUE experiments.","marker":"(Sung et al., 2021)"},{"why":"introduces Task2Vec task embeddings based on the Fisher diagonal, which the Squisher substitutes in the transferability predictions.","marker":"(Achille et al., 2019)"},{"why":"defines uncertainty-based gradient matching for merging and provides the RoBERTa setup reused for the second merging experiment.","marker":"(Daheim et al., 2024)"},{"why":"provides the intermediate-task transferability protocol and dataset rankings used to evaluate the Squisher-based task embeddings.","marker":"(Vu et al., 2020)"},{"why":"supplies the multitask T5 merging setup and the final-versus-best checkpoint distinction that exposes the Squisher's sensitivity to training length.","marker":"(Tam et al., 2024)"}],"fun_headline_variants":["Recycle Adam's squared gradients as a free Fisher approximation","Squisher: Adam's accumulator as a zero-cost Fisher diagonal","Use Adam's cached gradients for nearly free Fisher estimates","Squeeze Fisher from Adam's moving average at no extra cost","Adam's gradient accumulator doubles as a free Fisher stand-in"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the exponential moving average of squared gradients, accumulated with gradients taken at the moving parameter values throughout training and with the default smoothing coefficient, remains a faithful proxy for the Fisher diagonal evaluated at the final parameters; the paper offers no theoretical argument for this correspondence, only experiments, and its own ablations show the proxy degrades when training is cut short or when $\\beta_2$ is lowered.","fun_headline_variants_meta":{"raw":{"variants":["Recycle Adam's squared gradients as a free Fisher approximation","Squisher: Adam's accumulator as a zero-cost Fisher diagonal","Use Adam's cached gradients for nearly free Fisher estimates","Squeeze Fisher from Adam's moving average at no extra cost","Adam's gradient accumulator doubles as a free Fisher stand-in"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000762,"raw_usage":{"total_tokens":3437,"prompt_tokens":1055,"completion_tokens":2382,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":671,"completion_tokens_details":{"reasoning_tokens":2298}},"tokens_in":671,"tokens_out":2382,"duration_ms":15159,"temperature":1.0,"reasoning_tokens":2298,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:07:32.713267+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use the paper's own recipe: train VGG-13 on CIFAR-100 with Adam at $\\beta_2 = 0.95$, or stop training early, then prune 75% of parameters using the Squisher ranking and compare accuracy with Fisher pruning and random pruning; the paper's ablation predicts a sharp drop in the $\\beta_2 = 0.95$ regime, and a configuration where the Squisher's pruned accuracy falls to the random-mask level would falsify the drop-in claim. A second, more direct check is to compute the rank correlation between the Squisher's per-parameter scores and the label-sampled Fisher diagonal on the same checkpoint and see whether a low-correlation regime exists in which any application's performance gap to the Fisher-free baseline disappears.","supporting_citations":[{"cited_title":"E., and Makhzani, A","cited_arxiv_id":null,"evidence_quote":"introduces the joint Fisher whose sum-then-square diagonal is exactly the structure of the squared gradient accumulator, supplying the theoretical bridge."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines Fisher merging with the empirical Fisher diagonal as parameter-importance weights, the flagship application where the Squisher is validated on eight T5-Large models."},{"cited_title":"Training neural networks with fixed sparse masks","cited_arxiv_id":null,"evidence_quote":"introduces FISH Mask, the sparse fine-tuning method whose mask selection the Squisher replaces in the BERT-Large GLUE experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines uncertainty-based gradient matching for merging and provides the RoBERTa setup reused for the second merging experiment."},{"cited_title":"Merging by matching models in task parameter subspaces","cited_arxiv_id":null,"evidence_quote":"supplies the multitask T5 merging setup and the final-versus-best checkpoint distinction that exposes the Squisher's sensitivity to training length."}],"review_version":2}