Pith. sign in

REVIEW 5 major objections 5 minor 39 references

SeWA: Selective Weight Average via Probabilistic Masking

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SeWA turns checkpoint selection for weight averaging into a learned Bernoulli mask, and proves stability-based generalization bounds sharper than SGD's in both convex and non-convex settings.

desk verdict SeWA is a genuinely new and empirically promising way to pick checkpoints for averaging, but the theorem that promises an advantage over SGD does not cover the algorithm as written, because the mask probabilities are learned on the training set. read the letter →

arxiv 2502.10119 v1 pith:L4LPLZZC submitted 2025-02-14 cs.LG

Peng Wang , Shengchao Hu , Zerui Tao , Guoxia Wang , Dianhai Yu , Li Shen , Quan Zheng , Dacheng Tao This is my paper · ORCID
classification cs.LG
keywords weightaveragingcheckpointselectionprobabilisticmaskingGumbel-Softmaxstability-basedgeneralizationmodelSGDstabilitydeeplearningtheory
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Weight averaging helps models generalize and converge, but standard recipes such as SWA and LAWA require manually chosen schedules and hyperparameters. This paper proposes SeWA, which turns the selection of checkpoints to average into a learnable Bernoulli mask over the final k steps of SGD, optimized by Gumbel-Softmax. The central claim is that only a few adaptively selected checkpoints are needed to match or beat methods that average many, and that the stability-based generalization bounds for this procedure are provably sharper than SGD's in both convex and non-convex settings. If this holds, checkpoint averaging becomes an automatic post-training step rather than a hand-tuned one.

What carries the argument

The central object is the probabilistic mask $m_i \in \{0,1\}$ with $m_i \sim \mathrm{Bernoulli}(s_i)$, applied over the final $k$ checkpoints of an SGD run: the averaged model is $\bar{w}_T^K = \frac{1}{K} \sum_{i=T-k+1}^T m_i w_i$. Taking the expectation over masks replaces the discrete subset-selection problem by a continuous optimization over $s$ with $\|s\|_1 \le K$, made differentiable by a Gumbel-Softmax relaxation of each Bernoulli variable. In the stability proof, the mask expectation is what matters: because $\mathbb{E}[m_i] = s_i$, a Lipschitz difference between two runs on neighboring datasets contributes $s L \mathbb{E}[\bar{\delta}_T]$ rather than the full $L \mathbb{E}[\bar{\delta}_T]$, which is exactly how the sparsity factor $s$ enters the convex bound and why the window $k$ moves the exponent in the non-convex bound.

What would settle it

Train SeWA on a dataset and on the same dataset with one example replaced, and record the learned mask probabilities $s$ on both runs. If the largest per-checkpoint change $\sup_i |s_i - s_i'|$ is not small relative to $1/n$, then the assumption that masks are dataset-independent fails and the claimed stability bound does not cover the actual algorithm; the theory would then need a data-dependent stability argument to survive.

Watch

Extended reading notes

Core claim

SeWA's core claim is that the discrete choice of which of the last k SGD checkpoints to average can be solved in a continuous probabilistic space: each checkpoint gets an independent Bernoulli mask $m_i$ with probability $s_i$, and the averaged output is $\bar{w}_T^K = \frac{1}{K} \sum_{i=T-k+1}^T m_i w_i$. The paper derives stability bounds for this object: in the convex setting $\varepsilon_{\mathrm{gen}} \le \frac{2\alpha L^2 s}{n} \left(T - \frac{k}{2}\right)$ with $s = \max_i s_i$, and in the non-convex setting $O_s\left(T^{\frac{c\beta}{c\beta+k}}/n\right)$, both compared favorably against the corresponding SGD bounds, which are recovered as special cases ($k=1$, $s=1$). It also shows the mask probabilities are learnable by minimizing the empirical loss under a sparsity constraint using the Gumbel-Softmax reparameterization. Experiments in behavior cloning, image classification, and text classification show SeWA matching or exceeding the performance of baselines while averaging as few as 10 checkpoints against other methods' 100.

Load-bearing premise

The generalization bounds treat the mask probabilities $s$ as fixed and independent of the training data, but the algorithm actually learns $s$ from the training set; if replacing one example changes the learned masks appreciably, the stability comparison between the two runs is not valid.

Editorial extensions

If this is right

  • Checkpoint averaging can be fully automated: the mask probabilities are learned from the training objective, eliminating the manual schedules used by SWA and LAWA.
  • Fewer checkpoints is not a concession: the sparsity factor $s \le 1$ in the convex bound and $k$ in the non-convex exponent imply that selecting a small subset gives provably no worse a generalization bound than averaging the whole window.
  • Larger candidate windows $k$ strengthen the bound (through the $T - k/2$ term or the exponent), so using many candidate checkpoints while selecting few costs nothing theoretically.
  • The bounds inherit SGD's stability framework, so SeWA applies immediately wherever SGD's $L$-Lipschitz and $\beta$-smooth assumptions hold.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The stability analysis leaves the mask probabilities fixed; a natural experiment is to replace one training example and measure how much the learned $s$ moves, and if it moves much more than $1/n$, a data-dependent stability argument would be needed to sustain Lemma 4.1.
  • Because $s$ is defined as the maximum mask probability, the theory does not distinguish between a few high-probability selections and many low-probability ones; holding $K$ fixed while varying the spread of $s$ would isolate what the bound actually predicts.
  • The same masked-averaging optimization could be applied to other ensemble choices, such as EMA decays, model soups, or intermediate mixes, where the discrete inclusion decision is currently made by hand.
  • The non-convex bound keeps $s$ inside the $O_s$ notation; deriving the exact dependence could suggest an optimally sparse mask that minimizes the bound rather than just the empirical loss.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes SeWA, a checkpoint-averaging method that learns Bernoulli mask probabilities over the last k SGD iterates via Gumbel-Softmax optimization of the training loss, and claims stability-based generalization bounds that are sharper than SGD under both convex and non-convex assumptions. The empirical sections report improvements over SGD, SWA, EMA, LAWA, and random averaging on D4RL behavior cloning, CIFAR-100 image classification, and AG News text classification. The theoretical claims are the paper's main contribution, but the proof as written does not establish the stated bounds for the actual algorithm.

Significance. If the bounds were established, the result would be a useful practical contribution: adaptive sparse checkpoint averaging with a stability guarantee would combine the benefits of LAWA/SWA with reduced hyperparameter tuning. The experimental comparison is reasonably broad and includes multi-seed D4RL results, which supports the empirical claims. However, the theoretical core is not proven for the algorithm that is run: the mask probabilities are data-dependent in Section 5 while the stability analysis treats them as fixed, and the proof of the mask expectation step is algebraically incorrect. Because the claimed sharper-than-SGD bounds are the central advertised contribution, the paper is not yet ready for publication.

major comments (5)
  1. [Section 5, Eq. (13) and Algorithm 1 vs. Lemma 4.1 and Theorem 4.2] The stability proof treats the mask probabilities s as fixed constants, but Section 5 defines s as the minimizer of the training-set empirical loss in Eq. (13) and Algorithm 1 optimizes s on the same dataset used to train the checkpoints. Under Definition 3.5, one must compare A(S) and A(S') for neighboring S,S', and both the SGD iterates and the learned probabilities s(S) and s(S') will differ. Lemma 4.1 factors out a single s and never bounds the total-variation distance between p(m|s(S)) and p(m|s(S')). Consequently, the claimed bound ϵgen ≤ 2αL^2s/n(T−k/2) is not established for the algorithm that is actually proposed. Fixing s before training, or learning it on an independent split, would restore the argument; as written, the central theoretical claim does not cover the proposed method.
  2. [Eq. (4) and Lemma 4.1] In Eq. (4), the averaged output is normalized by K = Σ m_i, which is a random variable, but the proof of Lemma 4.1 replaces the expectation of the normalized difference by (1/k)Σ_i s_i E‖w_i − w'_i‖. These are not equal for k > 1: E[Σ m_i δ_i / Σ m_i] ≠ (1/k)Σ s_i δ_i in general, and the case Σ m_i = 0 is not handled. Since Lemma 4.1 (and Lemma 4.5) is the bridge between the mask model and the stability recurrence, this invalidates the derivations of both Theorem 4.2 and Theorem 4.6 even under the assumption that s is fixed.
  3. [Theorem 4.6 and Appendix C.5] Theorem 4.6 states that SeWA is run with constant step size α ≤ c/T, but the proof in Appendix C.5 sets α = c/t and uses time-dependent estimates such as (1 + cβ/t)^t ≤ e^{cβ}. This is a different algorithm/step-size schedule, and the resulting exponent cβ/(cβ+k) is not derived for the constant-step-size setting stated in the theorem. The proof needs to be reworked to be internally consistent with the theorem statement.
  4. [Appendix C.2, Eqs. (23)–(25)] Unrolling the recurrence in Eq. (23) with constant α gives a weighted sum of k(T − k/2 + 1/2), not k(T − k/2), so Eq. (25) should be 2αL/n (T − k/2 + 1/2) under the stated recurrence. The claimed constant in Theorem 4.2 is therefore not actually derived. This may be fixable by shifting the averaging window, but as written the exact 'T − k/2' improvement over SGD is not supported by the algebra.
  5. [Appendix C, proof attribution] The proof appendix states 'And you can also find it in (Wang et al., 2024b)' immediately before giving the core recurrence. This suggests that the stability analysis is largely a restatement of prior FWA results, while the only new element, the s multiplier, is precisely the step that is unproved for the actual algorithm. The authors should make the proof self-contained or clearly delineate which parts are new contributions, and they should justify the s factor independently rather than treating it as a trivial consequence of Lemma 4.1.
minor comments (5)
  1. [Assumption 3.1] The L-Lipschitz condition is written with norms of function values, but F is real-valued; it should use absolute values, and the claimed gradient bound ‖∇F(u)‖ ≤ L does not follow from the stated inequality without additional assumptions.
  2. [Section 5, Eq. (14)] The Gumbel-Softmax formula uses log(s_i) and log(1−s_i), which are undefined at the boundary s_i ∈ {0,1}; the paper does not specify how the constraint 0 ≤ s ≤ 1 is enforced or how boundary values are handled during optimization.
  3. [Eq. (13) and Algorithm 1] There is a typo 'excepted loss' that should read 'expected loss', and the symbol K is used both as the sparsity target in Eq. (12) and as the random selected count in Eq. (4); this ambiguity should be clarified.
  4. [Remark 4.8 and Eq. (39)] The claim that removing F ∈ [0,1] does not affect the final result is not justified: the term t0/n in Eq. (39) relies on the [0,1] boundedness of F when the first differing example is selected before t0.
  5. [Table 1 and Remark 4.3] The improvement of SeWA over FWA is expressed as the factor s ≤ 1, but after thresholding to the K largest probabilities in Algorithm 1, the effective selection probabilities are not the optimized s; the relation between the s in the theorem and the s produced by the practical algorithm should be stated explicitly.

Circularity Check

2 steps flagged · score 6.0 of 10

SeWA's sharper-than-SGD bound is obtained by treating the training-set-fitted mask probability s as a fixed constant; the proof does not cover the actual mask-learning algorithm.

  1. fitted input called prediction [Section 4 (Lemma 4.1, Theorem 4.2) vs. Section 5 (Eq. 13) and Algorithm 1]
    "min_{s∈Ĉ} F(s) = E_{p(m|s)}L(w(m)) ... where s = sup_{T−k+1≤i≤T} s_i, where s_i denotes the probability of m_i = 1. ... the second inequality is based on taking the expectation for mask mi, and the last inequality because of s = sup_{T−k+1≤i≤T} s_i."

    Equation (13) defines s as the minimizer of the training-set empirical loss, so s = s(S). Lemma 4.1's proof then bounds stability between S and S' by pulling out one s = sup_i s_i as if p(m|s) were a fixed, data-independent distribution. Under Definition 3.5, replacing one example changes both the SGD iterates and the learned mask distribution; the proof never bounds the induced total-variation term |s(S)−s(S')|. Consequently the claimed sharper bound 2αL²s(T−k/2)/n uses the fitted quantity as a free ≤1 multiplier: the theoretical advantage over SGD is not derived from the selection mechanism but is the training-set-fitted parameter inserted into the bound. Remark 4.3 makes this explicit by attributing the improvement to the factor s with 0≤s≤1.

  2. self citation load bearing [Appendix C, first paragraph]
    "By the Lemma 4.1 and 4.5, the proof of Theorem 4.2 and 4.6 can be further decomposed into bounding the difference of the parameters for the last k points of the average algorithm. We provide the proof as follows. And you can also find it in (Wang et al., 2024b)."

    The proof of the central generalization bounds is explicitly attributed to the same authors' prior FWA paper (Wang et al., 2024b). The portion not inherited from that FWA analysis is the s multiplier, which is precisely the training-set-fitted quantity flagged above. Thus the claimed sharper-than-SGD result is not independently derived in this paper: the bound structure comes from a same-author citation chain, and the only new ingredient is a fitted parameter that is trivially ≤1. This makes the self-citation load-bearing for the paper's main theoretical claim.

full rationale

The central issue is that SeWA's theoretical improvement over SGD reduces to the multiplier s, but s is not a fixed hyperparameter in the proposed algorithm: it is learned by minimizing the empirical loss on the training set (Eq. 13, Algorithm 1). Lemma 4.1 and Theorems 4.2/4.6 treat s = sup_i s_i as a constant when taking the expectation over the mask, which is valid only for a pre-fixed Bernoulli distribution independent of the dataset. Under the uniform-stability definition used by the paper, one must compare A(S) and A(S') when a single training example is replaced; with learned mask probabilities, the mask distributions p(m|s(S)) and p(m|s(S')) can differ, and no term in the proof controls that difference. Therefore the bound ϵgen ≤ 2αL²s(T−k/2)/n is not established for the actual SeWA algorithm as implemented. Because Remark 4.3 identifies the improvement over FWA/SGD as exactly the factor s∈[0,1], the sharper bound is partly an artifact of the fitting step: any selection probabilities in [0,1] would yield a formally smaller bound. The FWA portion of the analysis is also sourced to the same authors' prior work, so the theoretical novelty is largely a self-citation chain plus the fitted s factor. The empirical comparisons are separate and could still support the method, but the paper's headline theoretical claim of sharper generalization bounds for SeWA is not supported for the data-dependent mask-learning procedure described in Section 5.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central theoretical claim rests on a fixed mask distribution, but the algorithm learns the mask from the same training data. This mismatch is the main unaccounted assumption. The remaining free parameters are user-set hyperparameters that affect the bound and the experimental results.

free parameters (5)
  • s (mask probabilities) = learned on the training set via Gumbel-Softmax
    s = sup_i s_i is the factor that makes the bound 'sharper'; it is optimized on the same data used for the generalization claim, and the proof assumes it is fixed rather than data-dependent.
  • k (window size) = user-chosen, e.g., final 1000 checkpoints
    k is the length of the averaging window and appears in the bound exponent; it is selected by the user and not tuned by the algorithm.
  • K (sparsity target) = user-chosen (10, 20, 50, 100 in experiments)
    K constrains ||s||_1 <= K and is the number of selected checkpoints; the paper's efficiency claim is based on small K.
  • temperature t (Gumbel-Softmax) = not reported
    t controls the sharpness of the continuous relaxation; no values are given, yet it directly affects the learned mask.
  • Monte Carlo sample count M = not reported
    M is the number of Gumbel samples used to estimate the gradient; this affects optimization stability but is not specified.
assumptions (5)
  • domain assumption F is L-Lipschitz and β-smooth (Assumptions 3.1 and 3.2)
    Used throughout Lemma 3.4 and both generalization theorems to bound gradient differences and update expansiveness.
  • domain assumption F is convex in Theorem 4.2 (Assumption 3.3)
    Convexity provides the non-expansive update rule (Lemma 3.4) that yields the T - k/2 bound.
  • domain assumption F(w;z) in [0,1] in Theorem 4.6
    The boundedness is used to bound P(I ≤ t0) ≤ t0/n; Remark 4.8 claims it is removable but no proof is given.
  • ad hoc to paper Mask probabilities s are independent of the dataset (fixed before seeing S)
    Lemma 4.1 and the proofs of Theorems 4.2 and 4.6 couple the masks on S and S' as the same distribution. The SeWA algorithm in Section 5 learns s from the training set, so this assumption is unstated and contradicts the method.
  • domain assumption Gumbel-Softmax relaxation is a faithful surrogate for the true binary mask objective
    The discrete optimization in Eq. (12) is replaced by a continuous relaxation; no guarantee is provided that the stationary point of the relaxed problem corresponds to a good binary mask.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SeWA: Selective Weight Average via Probabilistic Masking." pith.science (2026). https://pith.science/paper/L4LPLZZC

@misc{pith2026250210119,
  author       = {Pith},
  title        = {Pith review of: SeWA: Selective Weight Average via Probabilistic Masking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L4LPLZZC}},
  note         = {Machine review of arXiv:2502.10119}
}
read the original abstract

Weight averaging has become a standard technique for enhancing model performance. However, methods such as Stochastic Weight Averaging (SWA) and Latest Weight Averaging (LAWA) often require manually designed procedures to sample from the training trajectory, and the results depend heavily on hyperparameter tuning. To minimize human effort, this paper proposes a simple yet efficient algorithm called Selective Weight Averaging (SeWA), which adaptively selects checkpoints during the final stages of training for averaging. Based on SeWA, we show that only a few points are needed to achieve better generalization and faster convergence. Theoretically, solving the discrete subset selection problem is inherently challenging. To address this, we transform it into a continuous probabilistic optimization framework and employ the Gumbel-Softmax estimator to learn the non-differentiable mask for each checkpoint. Further, we theoretically derive the SeWA's stability-based generalization bounds, which are sharper than that of SGD under both convex and non-convex assumptions. Finally, solid extended experiments in various domains, including behavior cloning, image classification, and text classification, further validate the effectiveness of our approach.

Figures

Figures reproduced from arXiv: 2502.10119 by the authors.

Figure 1
Figure 1. Comparison of SeWA with different models on conver￾gence performance. 3.2.1. THE EXPANSIVE PROPERTIES Lemma 3.4. Assume that the function F is β-smooth. Then, (1). (non-expansive) If F is convex, for any α ≤ 2 β , we have ∥wT +1 − w ′ T +1∥ ≤ ∥wT − w ′ T ∥; (2). ((1+αβ)-expansive) If F is non-convex, for any α, we have ∥wT +1−w ′ T +1∥≤(1+αβ)∥wT −w ′ T ∥. Lemma 3.4 tells us that the gradient update becomes non￾expan… view at source ↗
Figure 2
Figure 2. Comparison of different methods on the D4RL bench￾mark. Each data point represents the average cumulative reward across multiple tasks, averaged over 3 random seeds and 20 trajec￾tories per seed. Detailed results are provided in Appendix A. is important to note that LAWA, Random, and our proposed method all utilize the final 1000 checkpoints from the pre￾training process to compute performance metrics without furthe… view at source ↗
Figure 3
Figure 3. From left to right, the figures illustrate the impact of the hyperparameter K on the CIFAR-100 task. Each point corresponds to intervals of 100 checkpoints, with K checkpoints selected and averaged from these intervals using different strategies. 0 5 10 15 20 25 30 35 K=10 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 Accuracy SGD Random LAWA SeWA 0 5 10 15 20 25 30 35 K=20 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 Accuracy… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: From left to right, the figures illustrate the impact of the hyperparameter K on the AG News corpus. Each point corresponds to intervals of 100 checkpoints, with K checkpoints selected and averaged from these intervals using different strategies. 100 classes, the CIFAR…
Figure 5
Figure 5. Figure 5: From left to right, the figures illustrate the impact of the hyperparameter K on the CIFAR-100 task. Each data point represents performance based on intervals of 100 checkpoints, with K checkpoints selected from these intervals using various strategies. The first row c…
Figure 6
Figure 6. Figure 6: From left to right, the figures illustrate the impact of the hyperparameter K on the AG News corpus. Each point corresponds to intervals of 100 checkpoints, with K checkpoints selected from these intervals using different strategies. The first row corresponds to a netw…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 27 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    and Elisseeff, A

    Bousquet, O. and Elisseeff, A. Stability and generalization. The Journal of Machine Learning Research, 2: 0 499--526, 2002

    work page 2002

  3. [3]

    Swad: Domain generalization by seeking flat minima

    Cha, J., Chun, S., Lee, K., Cho, H.-C., Park, S., Lee, Y., and Park, S. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34: 0 22405--22418, 2021

    2021

  4. [4]

    and Papailiopoulos, D

    Charles, Z. and Papailiopoulos, D. Stability and generalization of learning algorithms that converge to global optima. In International conference on machine learning, pp.\ 745--754. PMLR, 2018

    work page 2018

  5. [5]

    and Wagner, T

    Devroye, L. and Wagner, T. Distribution-free performance bounds for potential function rules. IEEE Transactions on Information Theory, 25 0 (5): 0 601--604, 1979

    work page 1979

  6. [6]

    D4rl: Datasets for deep data-driven reinforcement learning

    Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020

    arXiv 2004

  7. [7]

    Train faster, generalize better: Stability of stochastic gradient descent

    Hardt, M., Recht, B., and Singer, Y. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning, pp.\ 1225--1234. PMLR, 2016

    work page 2016

  8. [8]

    Meta-learning pac-bayes priors in model averaging

    Huang, Y., Huang, W., Li, L., and Li, Z. Meta-learning pac-bayes priors in model averaging. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.\ 4198--4205, 2020

    work page 2020

Show all 39 references
  1. [9]

    Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018

    2018 arXiv

  2. [10]

    Categorical reparameterization with gumbel-softmax

    Jang, E., Gu, S., and Poole, B. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=rkE3y85ee

    2017

  3. [11]

    Stop wasting my time! saving days of imagenet and bert training with latest weight averaging

    Kaddour, J. Stop wasting my time! saving days of imagenet and bert training with latest weight averaging. arXiv preprint arXiv:2209.14981, 2022

    2022 arXiv

  4. [12]

    Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013

    2013 arXiv

  5. [13]

    and Lampert, C

    Kuzborskij, I. and Lampert, C. Data-dependent stability of stochastic gradient descent. In International Conference on Machine Learning, pp.\ 2815--2824. PMLR, 2018

    2018

  6. [14]

    and Ying, Y

    Lei, Y. and Ying, Y. Sharper generalization bounds for learning with gradient-dominated objective functions. In International Conference on Learning Representations, 2020

    2020

  7. [15]

    Trainable weight averaging: Efficient training by optimizing historical solutions

    Li, T., Huang, Z., Tao, Q., Wu, Y., and Huang, X. Trainable weight averaging: Efficient training by optimizing historical solutions. In The Eleventh International Conference on Learning Representations, 2022

    2022

  8. [16]

    Deep model fusion: A survey

    Li, W., Peng, Y., Zhang, M., Ding, L., Hu, H., and Shen, L. Deep model fusion: A survey. arXiv preprint arXiv:2309.15698, 2023

    2023 arXiv

  9. [17]

    Improving generalization of pre-trained language models via stochastic weight averaging

    Lu, P., Kobyzev, I., Rezagholizadeh, M., Rashid, A., Ghodsi, A., and Langlais, P. Improving generalization of pre-trained language models via stochastic weight averaging. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp.\ 4948--4954, 2022

    2022

  10. [18]

    J., Mnih, A., and Teh, Y

    Maddison, C. J., Mnih, A., and Teh, Y. W. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=S1jE5L5gl

    2017

  11. [19]

    Learning theory: stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimization

    Mukherjee, S., Niyogi, P., Poggio, T., and Rifkin, R. Learning theory: stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimization. Advances in Computational Mathematics, 25: 0 161--193, 2006

    2006

  12. [20]

    Polyak, B. T. and Juditsky, A. B. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30 0 (4): 0 838--855, 1992

    1992

  13. [21]

    J., Mohamed, S., and Wierstra, D

    Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic backpropagation and variational inference in deep latent gaussian models. In International conference on machine learning, volume 2, pp.\ 2, 2014

    2014

  14. [22]

    Efficient estimations from a slowly convergent robbins-monro process

    Ruppert, D. Efficient estimations from a slowly convergent robbins-monro process. Technical report, Cornell University Operations Research and Industrial Engineering, 1988

    1988

  15. [23]

    T., Kaddour, J., Kumar, A., et al

    Sanyal, S., Neerkaje, A. T., Kaddour, J., Kumar, A., et al. Early weight averaging meets high learning rates for llm pre-training. In Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ NeurIPS 2023), 2023

    2023

  16. [24]

    Learnability, stability and uniform convergence

    Shalev-Shwartz, S., Shamir, O., Srebro, N., and Sridharan, K. Learnability, stability and uniform convergence. The Journal of Machine Learning Research, 11: 0 2635--2670, 2010

    2010

  17. [25]

    Without-replacement sampling for stochastic gradient methods

    Shamir, O. Without-replacement sampling for stochastic gradient methods. Advances in neural information processing systems, 29, 2016

    2016

  18. [26]

    Which mode is better for federated learning? centralized or decentralized

    Sun, Y., Shen, L., and Tao, D. Which mode is better for federated learning? centralized or decentralized. arXiv preprint arXiv:2310.03461, 2023 a

    2023 arXiv

  19. [27]

    Understanding how consistency works in federated learning via stage-wise relaxed initialization

    Sun, Y., Shen, L., and Tao, D. Understanding how consistency works in federated learning via stage-wise relaxed initialization. arXiv preprint arXiv:2306.05706, 2023 b

    2023 arXiv

  20. [28]

    S., McAllester, D., Singh, S., and Mansour, Y

    Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999

    1999

  21. [29]

    Rethinking the inception architecture for computer vision

    Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 2818--2826, 2016

    2016

  22. [30]

    Generalization analysis of stochastic weight averaging with general sampling

    Wang, P., Shen, L., Tao, Z., He, S., and Tao, D. Generalization analysis of stochastic weight averaging with general sampling. In Forty-first International Conference on Machine Learning, 2024 a

    2024

  23. [31]

    A unified analysis for finite weight averaging

    Wang, P., Shen, L., Tao, Z., Sun, Yan Zheng, G., and Tao, D. A unified analysis for finite weight averaging. arXiv preprint arXiv:2411.13169, 2024 b

    2024 arXiv

  24. [32]

    Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8: 0 229--256, 1992

    1992

  25. [33]

    Stability analysis and generalization bounds of adversarial training

    Xiao, J., Fan, Y., Sun, R., Wang, J., and Luo, Z.-Q. Stability analysis and generalization bounds of adversarial training. Advances in Neural Information Processing Systems, 35: 0 15446--15459, 2022

    2022

  26. [34]

    Simple stochastic and online gradient descent algorithms for pairwise learning

    Yang, Z., Lei, Y., Wang, P., Yang, T., and Ying, Y. Simple stochastic and online gradient descent algorithms for pairwise learning. Advances in Neural Information Processing Systems, 34: 0 20160--20171, 2021

    2021

  27. [35]

    Stagewise training accelerates convergence of testing error over sgd

    Yuan, Z., Yan, Y., Jin, R., and Yang, T. Stagewise training accelerates convergence of testing error over sgd. Advances in Neural Information Processing Systems, 32, 2019

    2019

  28. [36]

    Efficient denoising diffusion via probabilistic masking

    Zhang, W., Zhang, Z., Pi, R., Jin, Z., Gao, Y., Ye, J., and Chen, K. Efficient denoising diffusion via probabilistic masking. In Forty-first International Conference on Machine Learning, 2024

    2024

  29. [37]

    Probabilistic bilevel coreset selection

    Zhou, X., Pi, R., Zhang, W., Lin, Y., and Zhang, T. Probabilistic bilevel coreset selection. In International Conference on Machine Learning. PMLR, 2022

    2022

  30. [38]

    Generalization error bounds with probabilistic guarantee for sgd in nonconvex optimization

    Zhou, Y., Liang, Y., and Zhang, H. Generalization error bounds with probabilistic guarantee for sgd in nonconvex optimization. arXiv preprint arXiv:1802.06903, 2018

    2018 arXiv

  31. [39]

    Stability and generalization of the decentralized stochastic gradient descent ascent algorithm

    Zhu, M., Shen, L., Du, B., and Tao, D. Stability and generalization of the decentralized stochastic gradient descent ascent algorithm. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.