{"id":"81d0312a-88a5-4b65-b88a-828e8dec6109","arxiv_id":"2504.20096","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AdaFisher replaces Adam's second-moment estimate with a diagonal Kronecker-factored Fisher preconditioner, reporting higher accuracy and lower perplexity than Adam, K-FAC, Shampoo, and AdaHessian at comparable wall-clock cost.","lead":"This thesis introduces AdaFisher, a deep learning optimizer that preconditions gradients with a cheap diagonal block-Kronecker approximation of the Fisher information matrix. It reports faster convergence and higher accuracy than Adam, K-FAC, Shampoo, and AdaHessian on image classification and language modeling benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The flagship ImageNet-1k comparison uses a weak Adam baseline and a single run; against well-tuned baselines cited in the same table, AdaFisher's margin is small, so the central 'outperforms SOTA' claim is not yet established.","rationale":"The reader identifies the diagonal-dominance assumption as the weakest link; that is a real and well-targeted concern about the method's theoretical justification. However, the single most load-bearing condition for the central claim is the empirical comparison: if the baselines are weak or the protocol is asymmetric, the headline result can appear without AdaFisher being genuinely better. The ImageNet-1k table is the clearest instance because it is a single run, the in-house Adam baseline is far below standard published results, and AdaFisher's margin over the literature baselines is within a range that could easily be seed or tuning variation. The equal-epoch comparison in Section 5.2.3 partially undermines the accuracy claim on at least one benchmark and the wall-clock numbers show AdaFisher is not exactly first-order cost. These observations do not refute the method; they show the evidence is not yet strong enough for the unqualified abstract claim. This keeps the reader's CONDITIONAL verdict unchanged, but for a somewhat different reason than the diagonal-dominance concern alone.","tokens_in":50752,"tokens_out":7245,"duration_ms":77126,"concrete_test":"Rerun the ImageNet-1k ResNet-50 experiment with at least 3 seeds per optimizer under an identical pipeline (same augmentations, batch size 256, 90 epochs, cosine schedule) for AdaFisher, AdamW, SGD-momentum, and Shampoo, using standard well-tuned hyperparameters (e.g., AdamW lr=1e-3 wd=1e-4; SGD lr=0.1 wd=1e-4; AdaFisher as specified in the paper). Report mean±std top-1 accuracy and total wall-clock time. If AdaFisher's mean advantage over the best-tuned AdamW/SGD is less than about 1 point or is within the seed-to-seed spread, the 'outperforms state-of-the-art' claim is unsupported and should be softened to 'competitive on some benchmarks.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that AdaFisher outperforms state-of-the-art optimizers in accuracy and convergence speed at first-order cost. The load-bearing support for that claim is the empirical protocol, and it is weakest in the flagship ImageNet-1k experiment (Section 5.2.1, Table 5.9). There, the in-house Adam baseline reaches only 67.78 top-1, far below the standard AdamW/SGD numbers (~76.3-76.7) cited in the same table. AdaFisher's margins over those literature baselines are 0.55-0.61 points, from a single run with no error bars. If the in-house Adam baseline was not tuned to its standard regime (e.g., Adam rather than AdamW, or a suboptimal learning rate), the reported 9-point gap is an artifact of baseline configuration rather than an AdaFisher advantage. In addition, the WCT protocol in Table 5.1 gives baselines unequal epoch budgets (e.g., Shampoo 36 epochs, K-FAC 107 epochs), so the Table 5.7 accuracies conflate per-step progress with wall-clock speed. Section 5.2.3's equal-epoch comparison even shows Shampoo marginally ahead of AdaFisher on ResNet-18/CIFAR-100 (77.35 vs 77.28), while AdaFisher is 8-14% slower per epoch than Adam (Table 5.11). Thus the headline as stated is not yet supported, even if the diagonal-dominance evidence is accepted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The thesis introduces AdaFisher, an adaptive second-order optimizer that replaces Adam's diagonal second-moment estimate with a diagonal block-Kronecker approximation of the empirical Fisher information matrix (EFIM). The central claim is that this preconditioner captures more curvature than Adam, yields faster convergence and better generalization, and does so at computational cost close to first-order methods. The manuscript supports this claim with a broad empirical study across CIFAR-10/100, Tiny ImageNet, ImageNet-1k, transfer learning, and language modeling, together with ablations on learning-rate schedulers, EMA of Kronecker factors, normalization layers, memory usage, and epoch times. A convergence analysis for convex and non-convex settings is also provided. The paper includes a public code repository and a series of visualizations of optimizer trajectories and Fisher spectra.","tokens_in":51123,"tokens_out":5536,"duration_ms":58764,"significance":"If the central claim held, AdaFisher would be a genuinely useful practical second-order optimizer: the algorithm is simple, the code is public, and the experimental body is broader than a typical conference paper, covering multiple architectures, datasets, and careful hyperparameter tuning. The diagonal-dominance observation for Kronecker factors is a useful empirical contribution, and the ablation/stability analysis is commendably thorough. However, the headline 'outperforms state-of-the-art' is not yet established: the flagship ImageNet comparison rests on an unusually weak in-house Adam baseline and a single run, the wall-clock comparison protocol gives baselines unequal epoch budgets, and the equal-epoch comparison shows Shampoo at least matching AdaFisher on one benchmark while AdaFisher is slower per epoch than Adam. The convergence proof also contains a gap that would need to be closed or explicitly scoped. With careful re-benchmarking and a corrected theory section, this could be a solid contribution.","major_comments":[{"comment":"The ImageNet-1k comparison does not support the claim that AdaFisher outperforms state-of-the-art optimizers. The in-house Adam baseline reaches only 67.78 top-1 accuracy, while the same table lists standard SGD at 76.40 and AdamW at 76.34 from the literature; AdaFisher's margins over those literature baselines are only 0.55 to 0.61 points, and the experiment is a single run with no error bars. This suggests the reported 9-point gap over the in-house Adam is an artifact of baseline configuration rather than an AdaFisher advantage. The authors should tune a proper AdamW baseline, report multiple seeds or at least standard deviations, and clearly state which settings differ from the literature baselines.","section":"Section 5.2.1, Table 5.9"},{"comment":"The wall-clock-time (WCT) protocol gives baselines unequal epoch budgets: in Table 5.1, Shampoo is trained for 36 epochs, K-FAC for 107, and AdaFisher for 200 on CIFAR/Tiny ImageNet. Consequently, the accuracies in Tables 5.7 and 5.8 conflate per-step progress with total wall-clock time. The equal-epoch comparison in Table 5.11 shows Shampoo marginally ahead of AdaFisher on ResNet-18/CIFAR-100 (77.35 vs. 77.28) and AdaFisher being about 14% slower per epoch than Adam (26.58 vs. 23.33 minutes), which undermines the claim of preserving first-order computational efficiency. The authors should present per-epoch learning curves alongside wall-clock results and be explicit about the speed-accuracy trade-off in each comparison.","section":"Section 5.1.1, Tables 5.1 and 5.11"},{"comment":"The diagonal-dominance assumption is load-bearing for the entire preconditioner design, but the evidence is limited to the 37th convolutional layer and 41st linear layer of ResNet-18 on CIFAR-10 at two training steps. Proposition 3.3.2 then applies the diagonal approximation globally to all layers, architectures, and datasets. If off-diagonal correlations in the Kronecker factors are significant in transformers, embeddings, or normalization layers, AdaFisher discards curvature information that K-FAC would retain, and the claimed advantage could disappear. The authors should test the diagonal-concentration hypothesis on representative layers of several architectures (including a transformer and a BatchNorm-only network) across datasets, and report the fraction of energy off the diagonal as a quantitative diagnostic.","section":"Section 3.2 and Proposition 3.3.2"},{"comment":"The stated convergence bound is the standard gradient-descent bound, but the update is preconditioned by (F_D^(t))^{-1}; the proposition does not include a Lipschitz condition or boundedness assumption involving the preconditioner, so the inequality J(theta(k)) - J(theta*) <= ||theta(0)-theta*||^2 / (2 alpha k) does not follow from the statement as written. Proposition 4.3.2 similarly lists bounded preconditioner norm, bounded gradients, unbiased noise, and a monotone ratio condition as assumptions without demonstrating that AdaFisher's actual preconditioner satisfies them. The proofs are deferred to an appendix, but in the submitted text the appendix content is not available for verification. The convergence section should either state the exact assumptions under which the preconditioned update is proven to converge, or clearly label the result as a heuristic bound.","section":"Section 4.3, Proposition 4.3.1"}],"minor_comments":[{"comment":"The text says 'random resized cropping to 224×244' but the standard ImageNet training size is 224×224; please correct this typo.","section":"Section 5.1.1, ImageNet preprocessing"},{"comment":"The text refers to 'Panel (D)' in the discussion of the FIM approximation error, but the figure caption and the surrounding description list only panels (A), (B), and (C); please align the panel references.","section":"Section 6.1.2, Figure 6.2"},{"comment":"The footnote states that Adam and AdaFisher were used for CNNs while AdamW and AdaFisherW were used for ViTs; this conflation of optimizer variant with architecture family should be made more explicit in the main text so that readers do not interpret the table as a single-optimizer comparison.","section":"Table 5.7 footnote"},{"comment":"The update rule is written as theta(t+1) = theta(t) - alpha m(t)/v(t), but the elementwise nature of the division is not defined; please add a sentence clarifying that the operation is applied coordinate-wise or define the vectorized notation.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be a Master's thesis and is closely related to an ICLR 2025 submission cited in the text. The editor may wish to ensure the thesis is self-contained and that the empirical claims are consistent with the conference version. The most important fix is re-benchmarking the ImageNet and WCT experiments with properly tuned, multiple-run baselines; without that, the central practical claim is not defensible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nOne thing you should know before reading: the core idea is real but the headline isn't. AdaFisher's replacement of Adam's second moment with a diagonal block-Kronecker Fisher approximation, plus an EFIM formula for normalization layers, is a sensible new combination of known K-FAC/Adam ingredients. But the claim that it 'outperforms state-of-the-art optimizers' is not currently supported by the evidence. The stress-test note is right: the in-house Adam baseline on ImageNet-1k (Table 5.9) hits 67.78 top-1, far below the ~76.3 AdamW number cited in the same table, and AdaFisher's margin over that literature baseline is ~0.6 points from a single run. That gap is most plausibly an artifact of an undertuned baseline.\n\nWhat the paper does well: a broad experimental sweep—CIFAR-10/100, Tiny ImageNet, ImageNet-1k, transfer learning, GPT-1 language modeling—with code available and ablation studies isolating the EMA, the square-root removal, the normalization-layer EFIM, and the diagonal-FIM approximation. The MAE comparison against the true Fisher diagonal is a nice check. The method is plausible and the empirical core is substantial.\n\nThe soft spots, in order of seriousness: (1) The diagonal-dominance evidence in Section 3.2 is narrow: Gershgorin discs, FFT, and SNR plots for two specific layers (37th conv, 41st linear) of ResNet-18 on CIFAR-10, at two time steps, then Proposition 3.3.2 turns that into a global approximation for every layer, architecture, and dataset. The paper does not test whether this holds in, say, Transformers or large CNNs. (2) The convergence analysis is soft: Proposition 4.3.1 is a standard gradient-descent bound that ignores the inverse-Fisher preconditioner, and Proposition 4.3.2 is an Adam-type bound with assumptions on the Fisher matrix stated rather than proved. Proofs are deferred to an appendix that is not present in this version. That is a rigor gap, not circularity. (3) The WCT protocol gives baselines unequal epoch budgets—Shampoo gets 36 epochs on CIFAR, K-FAC 107—so the wall-clock accuracy comparisons conflate per-step progress with actual speed. In the equal-epoch comparison (Table 5.11), Shampoo is ahead of AdaFisher on ResNet-18/CIFAR-100 (77.35 vs 77.28), and AdaFisher is 8-14% slower per epoch than Adam. That directly contradicts 'superior convergence speed.'\n\nWho should read it: people designing or benchmarking practical second-order optimizers. The method deserves serious referee time, but the manuscript needs a properly tuned AdamW baseline, error bars on all benchmarks, an equal-epoch or equal-time protocol, and a fixed convergence proof. My recommendation: engage with it, but expect heavy revision before the empirical claims can be trusted.","headline":"The new diagonal block-Kronecker Fisher optimizer is plausible, but the central SOTA claim rests on a weak ImageNet baseline and an uneven wall-clock protocol; the empirical core warrants revision, not the headline.","tokens_in":51571,"tokens_out":3635,"would_cite":false,"duration_ms":33834,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","90C26","65K05"],"pacs":[],"model":"deepseek-v4-flash","headline":"A Fisher-based optimizer that approximates curvature with a diagonal block-Kronecker product claims to outperform Adam, K-FAC, and Shampoo at the same wall-clock training cost.","keywords":["Fisher information matrix","second-order optimization","Kronecker-factored approximation","natural gradient descent","Adam","adaptive optimizers","deep learning training","convergence and generalization"],"falsifier":"Measure the spectral norm or energy fraction of the off-diagonal part of $H$ and $S$ across layers for a transformer or a wide CNN; if some layer shows off-diagonal energy comparable to the diagonal and a version of AdaFisher that restores off-diagonal blocks in that layer visibly outperforms the fully diagonal version, the diagonal-dominance premise fails.","tokens_in":50540,"feed_emoji":"⚡","tokens_out":7164,"duration_ms":64203,"temperature":0.7,"pith_summary":"AdaFisher says second-order optimization for deep networks can be made practical by approximating the Fisher information matrix with a diagonal block-Kronecker product. The optimizer replaces the squared-gradient second moment of Adam with this Fisher-based preconditioner, so each update step rescales the gradient by local curvature information at a computational cost close to first-order methods. On image classification (CIFAR-10/100, Tiny ImageNet, ImageNet-1k) and language modeling (WikiText-2, PTB), it reports higher accuracy or lower perplexity than Adam, AdaHessian, K-FAC, and Shampoo under equal wall-clock time. If the claims hold, curvature-aware training no longer requires the expensive per-iteration overhead that has kept second-order methods out of large-scale practice.","feed_headline":"Fisher-based optimizer beats Adam, K-FAC, Shampoo at same cost","feed_subtitle":"AdaFisher swaps Adam's second moment for a diagonal Fisher preconditioner, improving accuracy without slowing training.","key_machinery":"The load-bearing object is the diagonal block-Kronecker approximation of the empirical Fisher information matrix, denoted $\\tilde F_D$ in the paper. For each layer $i$, the Fisher block is factored as $H_{i-1}\\otimes S_i$ in the K-FAC style, then the factors are replaced by their diagonals, min-max normalized, and combined as $\\tilde F^i_D = H'_{D,i-1}\\otimes S'_{D,i}+\\lambda I$. The diagonal dominance of the Kronecker factors is what licenses this truncation; the EMA over batches and the inclusion of BatchNorm and LayerNorm factors in Proposition 3.3.1 are what make the approximation stable enough to use as a preconditioner throughout training.","core_discovery":"On the paper's own terms, the central discovery is that the Fisher information matrix of a deep network does not need to be formed or inverted in full: its layer-wise Kronecker factors are empirically diagonal-dominant, so the essential curvature information can be captured by taking only the diagonal of each factor. AdaFisher builds on K-FAC's factorization $\\hat{F}_i = H_{i-1} \\otimes S_i$, where $H_{i-1}$ is the activation covariance and $S_i$ the sensitivity covariance, then keeps $\\tilde{F}_D^i = H'_{D,i-1} \\otimes S'_{D,i} + \\lambda I$ with min-max normalized diagonal factors and Tikhonov damping. This preconditioner replaces the second moment $v^{(t)}$ in the Adam-style update $\\theta^{(t+1)}=\\theta^{(t)}-\\alpha(\\tilde F_D^{(t)})^{-1} m^{(t)}$, and an exponential moving average over batches keeps the curvature estimate stable. The paper reports that this yields faster convergence and better final accuracy than first-order and other second-order baselines, and that it scales to ImageNet-1k and transformer language models at near-Adam epoch times.","pith_inferences":["One testable extension is to measure diagonal dominance of $H$ and $S$ in transformer blocks and very deep networks; if off-diagonal energy there is large, AdaFisher's approximation could be combined with block sizes larger than 1 to retain the advantage.","The reported tendency to land in flatter minima suggests AdaFisher could be stacked with stochastic weight averaging or sharpness-aware objectives to push generalization further, though the paper does not test this.","The removal of the square root from the adaptive update, which the paper argues is possible because the Fisher preconditioner already carries curvature information, implies that other Adam-family optimizers might adopt a similar normalization and drop the batch-size-dependent rescaling that plain Adam without the square root needs.","If the diagonal Fisher approximation is robust across architectures, the same trick could be applied to Hessian-based natural-gradient variants or to second-order fine-tuning of large language models, cutting memory and compute in regimes where full K-FAC is infeasible."],"forward_implications":["AdaFisher and its weight-decay variant AdaFisherW should reach higher validation accuracy than Adam when both are run for the same wall-clock time, making second-order curvature information available without a per-iteration slowdown.","Distributed AdaFisher averages the Kronecker factors across GPUs before updating, so larger batch sizes (512, 1024) on ImageNet-1k continue to improve validation accuracy rather than degrading it.","The method should reduce hyperparameter sensitivity: the same settings work across multiple learning rates, batch sizes, and scheduler choices, as reported for ResNet-50 on CIFAR-10/100.","In language modeling, AdaFisherW should improve test perplexity on WikiText-2 and PTB compared with AdamW and AdaHessian, while K-FAC and Shampoo fail to scale or converge in the paper's setup.","The diagonal-concentration observation implies that for CNNs similar to ResNet-18, most useful curvature information in the Fisher factors lives on the diagonal, which is why the diagonalized preconditioner does not lose the benefit."],"supporting_citations":[{"why":"introduces K-FAC, whose Kronecker-factored Fisher blocks AdaFisher diagonalizes","marker":"Martens & Grosse, 2015a"},{"why":"extends the factorization to convolutional networks and supplies the patch-expansion operators used for the factors","marker":"Grosse & Martens, 2016"},{"why":"defines Adam, the adaptive update framework whose second moment AdaFisher replaces","marker":"Kingma & Ba, 2015"},{"why":"provides AdaHessian, a diagonal Hessian baseline that AdaFisher must beat","marker":"Yao et al., 2021"},{"why":"provides Shampoo, a Kronecker-preconditioned baseline used for scalability comparison","marker":"V. Gupta et al., 2018"},{"why":"supplies the Gershgorin circle theorem used to justify diagonal dominance of the factors","marker":"Horn & Johnson, 2012"},{"why":"supplies CIFAR-10/100, the main benchmark suite for the empirical claims","marker":"Krizhevsky et al., 2009"},{"why":"supplies decoupled weight decay and the cosine annealing schedule used in training and in AdaFisherW","marker":"Loshchilov & Hutter, 2019"}],"fun_headline_variants":["AdaFisher: diagonal Fisher beats Adam, K-FAC, Shampoo","Cheap Fisher curvature tops Adam, K-FAC, Shampoo","AdaFisher: faster and more accurate than top optimizers","Diagonal Fisher preconditioner outdoes Adam, K-FAC, Shampoo","AdaFisher: cheap Fisher info beats Adam at similar speed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method rests on the assumption that the two per-layer matrices whose Kronecker product approximates the Fisher information are diagonally dominant at every layer and throughout training, so discarding their off-diagonal entries does not remove useful curvature information.","fun_headline_variants_meta":{"raw":{"variants":["AdaFisher: diagonal Fisher beats Adam, K-FAC, Shampoo","Cheap Fisher curvature tops Adam, K-FAC, Shampoo","AdaFisher: faster and more accurate than top optimizers","Diagonal Fisher preconditioner outdoes Adam, K-FAC, Shampoo","AdaFisher: cheap Fisher info beats Adam at similar speed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000934,"raw_usage":{"total_tokens":4022,"prompt_tokens":995,"completion_tokens":3027,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":2935}},"tokens_in":611,"tokens_out":3027,"duration_ms":23355,"temperature":1.0,"reasoning_tokens":2935,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:09:11.584751+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the spectral norm or energy fraction of the off-diagonal part of $H$ and $S$ across layers for a transformer or a wide CNN; if some layer shows off-diagonal energy comparable to the diagonal and a version of AdaFisher that restores off-diagonal blocks in that layer visibly outperforms the fully diagonal version, the diagonal-dominance premise fails.","supporting_citations":[{"cited_title":"A., & Johnson, C","cited_arxiv_id":null,"evidence_quote":"supplies the Gershgorin circle theorem used to justify diagonal dominance of the factors"}],"review_version":1}