{"id":"791d1bdc-b071-47e5-bdb2-167abae8ed51","arxiv_id":"2412.05169","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"SAM and several variants improve average OOD and gradual domain adaptation accuracy over Adam on four small benchmarks, but the new theoretical bounds match rather than improve on prior rates.","lead":"The paper compares eight sharpness-aware minimization (SAM) optimizers on out-of-distribution and gradual domain adaptation tasks and reports average accuracy gains over Adam. It also derives sharpness-based generalization bounds for both settings, but the gradual-domain bound is asymptotically no better than an existing self-training bound.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 3.1's zero-shot comparison confounds optimizer choice with learning rate: SAM uses Adam at lr=1e-2 while the Adam baseline uses lr=1e-3, so the claimed 4.76% and 8.01% gains may be a learning-rate artifact.","rationale":"The paper's most valuable and actionable claim is empirical: SAM and its variants improve zero-shot OOD generalization. That claim rests on an unfair baseline. The reader's weakest-assumption analysis identifies exactly this confound, and I agree. The theory sections are honestly hedged (the GDA bound is explicitly no better than Wang et al. 2022), and the PAC-Bayes condition is a standard assumption, so the theoretical contribution does not create a separate correctness concern at this level. The strongest issue is the uncontrolled learning rate in Table 2; a matched-LR experiment is a small, decisive check. If the gap persists, the paper's empirical conclusion stands; if not, the abstract and conclusion need major revision. Thus the conditional verdict is appropriate; no adjustment is needed.","tokens_in":20515,"tokens_out":6924,"duration_ms":65433,"concrete_test":"Re-run Table 2 with a matched Adam baseline at lr=1e-2 (same weight decay 1e-4, batch size 128, epochs, and three seeds), and also sweep lr in {1e-4,3e-4,1e-3,3e-3,1e-2} for both Adam and SAM. If the best Adam at lr=1e-2 reaches or exceeds SAM on Color MNIST and the average gap drops below 1%, the zero-shot headline is a learning-rate artifact rather than evidence for sharpness-aware minimization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is the zero-shot OOD gain. In Section 3.1 (Model Setup), the paper states that SAM uses Adam with lr=1e-2, while the Adam baseline uses lr=1e-3; no LR sweep is reported for Adam. Table 2 then reports the best accuracy over the variant hyperparameter grid, and the headline numbers are averages of per-dataset relative improvements, dominated by Color MNIST (Adam 84.97 vs SAM 93.88 and FisherSAM 96.53). Because the sharpness perturbation and the learning rate are never varied independently, the observed gap could be caused entirely by the 10x larger LR for SAM rather than by minimization of rho-robust loss. This threatens the main abstract claim and the conclusion that SAM 'significantly improves' zero-shot OOD accuracy. The GDA experiments are less affected because Figure 1 uses rho=0 as the Adam baseline at the SAM LR, but the abstract/body inconsistency (1.52% vs 1.42% strongest-variant improvement) should also be fixed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies sharpness-aware minimization (SAM) and eight of its variants in two out-of-distribution settings: zero-shot OOD generalization and gradual domain adaptation (GDA). On four datasets (Rotated MNIST, Color MNIST, Covertype, Portraits), it reports that SAM improves over an Adam baseline by 4.76% on average in the zero-shot setting and by 0.82% in GDA, with the strongest variants improving by 8.01% and 1.52% (abstract) or 1.42% (body). The theoretical part provides a sharpness-aware error-difference lemma, an OOD generalization bound (Theorem 1), and a GDA bound (Theorem 2); the authors state that the GDA bound is asymptotically no better than the existing bound of Wang et al. (2022). The paper is candid about this limitation and points to several directions for tighter analysis.","tokens_in":20805,"tokens_out":9668,"duration_ms":97770,"significance":"If the empirical claims were established, the paper would offer a useful practical comparison of SAM variants under distribution shift and a reasonable starting point for sharpness-based OOD theory. The strengths are the breadth of the variant survey, the computational-cost comparison in Table 1, the explicit proof appendix, and the honest discussion of the gap between theory and experiments. However, the zero-shot empirical headline is undermined by two confounds: SAM and the Adam baseline use different learning rates, and the reported accuracies are selected as the best over a hyperparameter grid on the test set. The theoretical contribution is modest because it reuses Foret et al.'s PAC-Bayes bound and Wang et al.'s discrepancy machinery, and the paper itself concedes that the GDA bound has no asymptotic advantage over prior work. Still, the derivations are mostly sound and the paper is clearly written; the empirical claims can in principle be repaired with better-controlled experiments.","major_comments":[{"comment":"The zero-shot comparison is confounded by learning rate: SAM uses Adam with learning rate 1e-2, while the Adam baseline uses learning rate 1e-3, and no learning-rate sweep is reported for Adam. Because the perturbation radius and the learning rate are never varied independently, the reported 4.76% and 8.01% average gains cannot be attributed to sharpness-aware minimization; they may be produced entirely by the tenfold larger learning rate. Please rerun the Adam baseline at the same learning rate as SAM, and ideally sweep the learning rate for both optimizers, before claiming that SAM improves zero-shot OOD generalization.","section":"Section 3.1 (Model Setup), Table 2"},{"comment":"The zero-shot results report the best accuracy over the rho grid and other variant-specific hyperparameters for each dataset, which is equivalent to selecting hyperparameters on the test set. This favors SAM variants relative to an Adam baseline that is not given the same selection procedure, and it makes the reported gains difficult to interpret. Please either fix the hyperparameters a priori or choose them on a validation split, and report the selected values for every optimizer and dataset.","section":"Appendix B, Table 2"},{"comment":"For the GDA variant comparison, the number of intermediate domains T is chosen as the optimal T for SAM from Figure 1 and then used for all optimizers. If T is selected to favor SAM, the claim that SAM and FriendlySAM consistently outperform Adam in GDA is biased. Please compare over the full range of T, or choose T by a criterion that does not depend on the optimizer being evaluated.","section":"Section 4.2, Table 3"},{"comment":"In the proof of Lemma 1, the symbol rho is redefined as the maximum Lipschitz constant, while in the statement of the lemma and in S_rho(theta_mu) it denotes the SAM perturbation radius. As written, the proof establishes the bound with sharpness measured at the Lipschitz-constant radius rather than at the algorithm's perturbation radius, which makes the sharpness term in Theorem 1 ambiguous. Please use a distinct symbol, such as L, for the Lipschitz constant and retain rho for the perturbation radius throughout.","section":"Lemma 1, Appendix A.1"}],"minor_comments":[{"comment":"The abstract and Section 1 state that the strongest SAM variants improve over Adam by 1.52% in GDA, while Section 4.3 and the Conclusion report 1.42%; please reconcile these numbers.","section":"Abstract, Section 4.3, Conclusion"},{"comment":"The sentence 'In Table 1, we report the best accuracy values obtained over all hyperparameter settings' should refer to Table 2, which contains the zero-shot accuracy results.","section":"Appendix B"},{"comment":"There is a typo in the introduction: 'SAM remains remains understudied' should read 'SAM remains understudied.'","section":"Section 1"},{"comment":"Lemma 2 states an extra condition E(theta) <= E_{epsilon~N(0,rho^2 I)}[E(theta+epsilon)] that is not part of the standard Foret et al. PAC-Bayes bound; if this condition is needed, it should be stated as an explicit assumption in Theorem 1 and Theorem 2 and justified for the iterates of Algorithm 2 rather than only in a footnote.","section":"Lemma 2"},{"comment":"Figure 1 does not show error bars or the number of seeds used; please add this information or state explicitly that the curves are single runs.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The learning-rate confound in Section 3.1 and the test-set hyperparameter selection in Appendix B are serious because the zero-shot accuracy gains are the paper's primary empirical headline. The GDA experiment is cleaner because rho=0 provides an Adam baseline at the same learning rate, but the T-selection issue in Section 4.2 still needs attention. The theory is a reasonable but incremental repackaging of existing PAC-Bayes and discrepancy bounds; the notation conflict in Lemma 1 must be fixed. I recommend major revision rather than rejection because the empirical claims can, in principle, be re-established with properly controlled experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version. The paper's headline claim—that SAM gives 4.76–8.01% zero-shot OOD gains—is not supported by the experiments as run, because the SAM runs use a ten times larger learning rate than the Adam baseline. That is a direct confound. The GDA story is cleaner, and the eight-variant comparison itself is a useful resource.\n\nWhat's genuinely new: this is the first systematic benchmark of eight SAM variants in both zero-shot OOD and GDA. The theoretical contribution is honest: they add a sharpness term to Wang et al.'s error-difference lemma and then assemble a PAC-Bayes bound for GDA, explicitly stating it is asymptotically no better than prior work. They also flag the theory-empirics gap themselves in Section 6. Credit where due: the related work is thorough and the writing is clear.\n\nThe soft spots are in the experiments. Section 3.1 states that SAM uses Adam with lr=1e-2 and weight decay 1e-4, while the Adam baseline uses lr=1e-3. No LR sweep for Adam is reported. So the 4.76% and 8.01% averages in Table 2 could be a learning-rate artifact. On top of that, the paper reports the best accuracy over the variant hyperparameter grid per dataset (Appendix B), which is effectively fitting to the test set. Both issues need to be fixed before the zero-shot claim can be believed.\n\nThe GDA results are less affected because Figure 1 uses rho=0 (SAM's base optimizer, no perturbation) as the baseline, so at least the optimizer choice is controlled. But there is an abstract/body inconsistency: the abstract says 1.52% improvement for strongest variants, while the body says 1.42%. Also, the GDA experiments use hyperparameters picked from the zero-shot grid, so there is still some selection going on.\n\nThe theory is a modest extension of existing results, not a breakthrough. That is fine as long as it is presented as such, which it mostly is. The PAC-Bayes condition E(theta) <= E_epsilon[E(theta+epsilon)] is assumed to hold for the iterates; that is standard in the SAM literature but worth noting.\n\nBottom line: this deserves a serious referee, but the zero-shot empirical claim needs major revision. If the LR confound and test-set selection are not fixed, the paper's main contribution shrinks to a catalog of SAM variants on GDA with modest gains. I'd send it to review, but with the expectation of a major revision or rejection if the confound holds.","headline":"The zero-shot empirical headline is compromised by a learning-rate confound, but the GDA experiments and the eight-variant comparison are useful and the theory is honestly presented.","tokens_in":21342,"tokens_out":3395,"would_cite":false,"duration_ms":32523,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68Q32"],"pacs":[],"model":"deepseek-v4-flash","headline":"Sharpness-aware minimizers beat Adam on shifted tasks by up to 8.01%.","keywords":["sharpness-aware minimization","out-of-distribution generalization","gradual domain adaptation","zero-shot generalization","PAC-Bayes bounds","SAM variants","distribution shift"],"falsifier":"Train Adam and SAM on the four benchmarks with identical learning rates, schedules, and weight decays; if the 4.76% and 8.01% average margins shrink to noise, the reported advantage is a hyperparameter artifact rather than an effect of sharpness-aware minimization.","tokens_in":20273,"feed_emoji":"🎯","tokens_out":9456,"duration_ms":86889,"temperature":0.7,"pith_summary":"Sharpness-aware minimization (SAM) trains models to be stable under small adversarial weight perturbations, and this paper asks whether that stability pays off when the test distribution differs from training. Across four benchmarks, it reports that SAM beats the Adam baseline by 4.76% average zero-shot out-of-distribution accuracy, and the strongest SAM variants beat Adam by 8.01%; in gradual domain adaptation, where the model self-trains across intermediate domains, the reported gains are 0.82% and 1.52%. The paper also proves sharpness-aware bounds for both settings, bounding target error by source error plus sharpness, parameter drift, Wasserstein distance, and sample-complexity terms. The GDA bound is asymptotically no better than existing self-training bounds, so the theory does not yet explain why SAM helps; the empirical claim, if correct, would make SAM and its variants a simple, practical way to improve accuracy under distribution shift.","feed_headline":"Sharpness-aware minimizers beat Adam on shifted tasks by up to 8.01%","feed_subtitle":"Across four benchmarks, SAM gains 4.76% on zero-shot and 0.82% on gradual adaptation over Adam.","key_machinery":"The machinery is the sharpness-aware objective: instead of minimizing the loss at the current parameters, SAM seeks parameters whose worst-case loss over an $\\ell^2$ ball of radius $\\rho$ is minimized, $\\theta^\\star = \\arg\\min_{\\theta} \\max_{\\|\\beta\\|\\le\\rho} E(\\theta+\\beta)$. The paper's theoretical engine is a \"sharpness-aware error difference\" lemma: for two distributions $\\mu,\\nu$ and two models $\\theta_\\mu,\\theta_\\nu$, $|E^\\rho_\\mu(\\theta_\\mu)-E_\\nu(\\theta_\\nu)| \\le S_\\rho(\\theta_\\mu) + O(\\|\\theta_\\mu-\\theta_\\nu\\| + W_p(\\mu,\\nu))$, where $S_\\rho$ is the $\\rho$-sharpness. Combined with the PAC-Bayes bound, this lemma converts flatness into a distribution-shift guarantee; the GDA theorem applies it domain-by-domain along the intermediate chain, using the discrepancy measure and sequential Rademacher complexity from the online-learning view of self-training.","core_discovery":"On the paper's own terms, the central discovery is that minimizing flatness at train time transfers to new domains: the original SAM improves over Adam on every dataset considered for both zero-shot OOD generalization and gradual domain adaptation, and several SAM variants improve further. The paper reports exact numbers ($4.76\\%$ zero-shot, $8.01\\%$ for the strongest variants, $0.82\\%$ GDA, $1.52\\%$ for the strongest variants) and proposes that FisherSAM's use of an approximate Fisher/Hessian perturbation under cross-entropy loss may explain its zero-shot gains, while FriendlySAM's adaptive perturbation robustness may explain its GDA gains. The matching theoretical contribution is a pair of generalization bounds that place sharpness alongside distribution distance and parameter distance as the quantities governing OOD error, with the GDA bound matching the asymptotic rate of the standard self-training bound.","pith_inferences":["A testable extension is to run SAM and Adam with identical learning rates, schedules, and weight decays; if the headline margins shrink to noise, the reported gains are a hyperparameter artifact rather than an effect of sharpness.","FisherSAM's use of an approximate Fisher/Hessian perturbation is the paper's own explanation for its strong zero-shot results, so probing whether this variant scales to larger, more realistic distribution shifts would directly test that mechanism.","The sharpness-aware bound suggests a common yardstick for comparing flat-minima optimizers on OOD benchmarks: report sharpness, parameter drift, and Wasserstein shift alongside accuracy, and test whether lowering each component lowers target error.","The authors' conjecture of a localized analysis implies a concrete check: measure feature rank, denoising, and feature balance on the four benchmarks to see whether SAM's OOD benefit tracks those quantities rather than final flatness."],"forward_implications":["Using SAM in place of Adam is a drop-in optimizer change that improved zero-shot OOD accuracy on all four benchmarks.","Stronger variants such as FisherSAM and FriendlySAM can add further gains, while cheaper variants like LookSAM and NoSAM still often match or beat original SAM.","The sharpness-aware bounds give a quantitative decomposition of OOD error into sharpness, parameter drift, and distribution distance, suggesting each is a separate lever for improving shifted-domain accuracy.","In gradual domain adaptation, SAM's consistent gain means the self-training loop can be improved during optimization without changing pseudo-labeling or architecture.","Because the GDA bound has the same asymptotic rate as standard self-training, any satisfactory theory of SAM's OOD advantage must go beyond flatness alone."],"supporting_citations":[{"why":"Supplies the SAM algorithm, the $\\rho$-robust objective, and the PAC-Bayes bound used as Lemma 2.","marker":"Foret et al. (2021)"},{"why":"Defines the GDA setting and datasets, and supplies the standard self-training bound that Theorem 2 extends.","marker":"Wang et al. (2022)"},{"why":"Introduces gradual self-training and the Rademacher generalization bound used in Theorem 1.","marker":"Kumar et al. (2020)"},{"why":"Supplies the discrepancy measure and the discrepancy bound used in the GDA proof.","marker":"Kuznetsov & Mohri (2020a)"},{"why":"Defines sequential Rademacher complexity, the online-learning complexity measure in Theorem 2.","marker":"Rakhlin et al. (2015)"},{"why":"Shows that low sharpness alone does not account for SAM's gains, framing the theory-versus-empirics gap.","marker":"Wen et al. (2023)"},{"why":"Proposes FisherSAM, one of the strongest zero-shot variants, whose second-order perturbation the paper credits for gains.","marker":"Kim et al. (2022)"},{"why":"Proposes FriendlySAM, the strongest GDA variant, whose adaptive perturbation the paper credits for robustness to $\\rho$.","marker":"Li et al. (2024)"}],"fun_headline_variants":["SAM beats Adam by up to 8.01% on OOD tasks","SAM outgeneralizes Adam: 4.76% avg zero-shot gain","SAM gains 8.01% over Adam on shifted data","SAM outshines Adam in zero-shot and gradual OOD","SAM: 8.01% OOD gain over Adam"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The empirical comparison treats Adam as a fair control, but SAM was trained with a learning rate of $10^{-2}$ while Adam used $10^{-3}$, so the accuracy gap could come from the larger step size rather than from the sharpness-aware perturbation.","fun_headline_variants_meta":{"raw":{"variants":["SAM beats Adam by up to 8.01% on OOD tasks","SAM outgeneralizes Adam: 4.76% avg zero-shot gain","SAM gains 8.01% over Adam on shifted data","SAM outshines Adam in zero-shot and gradual OOD","SAM: 8.01% OOD gain over Adam"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001416,"raw_usage":{"total_tokens":5779,"prompt_tokens":1066,"completion_tokens":4713,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":682,"completion_tokens_details":{"reasoning_tokens":4621}},"tokens_in":682,"tokens_out":4713,"duration_ms":34871,"temperature":1.0,"reasoning_tokens":4621,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:50:55.635156+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train Adam and SAM on the four benchmarks with identical learning rates, schedules, and weight decays; if the 4.76% and 8.01% average margins shrink to noise, the reported advantage is a hyperparameter artifact rather than an effect of sharpness-aware minimization.","supporting_citations":[{"cited_title":"Sharpness-aware minimization for efficiently improving generalization","cited_arxiv_id":null,"evidence_quote":"Supplies the SAM algorithm, the $\\rho$-robust objective, and the PAC-Bayes bound used as Lemma 2."},{"cited_title":"Understanding gradual domain adaptation: Improved analysis, optimal path and beyond","cited_arxiv_id":null,"evidence_quote":"Defines the GDA setting and datasets, and supplies the standard self-training bound that Theorem 2 extends."},{"cited_title":"Understanding self-training for gradual domain adaptation","cited_arxiv_id":null,"evidence_quote":"Introduces gradual self-training and the Rademacher generalization bound used in Theorem 1."},{"cited_title":"Online learning via sequential complexities","cited_arxiv_id":null,"evidence_quote":"Defines sequential Rademacher complexity, the online-learning complexity measure in Theorem 2."},{"cited_title":"Sharpness minimization algorithms do not only minimize sharpness to achieve better generalization","cited_arxiv_id":null,"evidence_quote":"Shows that low sharpness alone does not account for SAM's gains, framing the theory-versus-empirics gap."},{"cited_title":"F isher SAM : Information geometry and sharpness aware minimisation","cited_arxiv_id":null,"evidence_quote":"Proposes FisherSAM, one of the strongest zero-shot variants, whose second-order perturbation the paper credits for gains."}],"review_version":1}