ADADELTA: An Adaptive Learning Rate Method

1. On the Provable Suboptimality of Momentum SGD in Nonstationary Stochastic Optimization

   stat.ML 2026-01 conditional novelty 7.0 of 10

   Momentum SGD pays a provable tracking penalty under distribution shift — lag growing as (1−β)⁻¹ and tracking floors worse than vanilla SGD in drift-dominated regimes.

2. Interior-Point Vanishing Problem in Semidefinite Relaxations for Neural Network Verification

   cs.LG 2025-06 conditional novelty 7.0 of 10

   Semidefinite relaxation for deep ReLU verification suffers from 'interior-point vanishing' as depth increases, and removing layer-wise bound constraints mitigates it.

3. Rethinking Attribute Representation and Injection for Sentiment Classification

   cs.CL 2019-08 conditional novelty 7.0 of 10

   The paper proposes CHIM, a chunk-wise importance matrix for representing user and product attributes, and reports that injecting attributes into attention is generally the worst of four locations in a BiLSTM sentiment...

4. Decoding the Micromagnetic Hamiltonian from Magnetic Fingerprints

   cond-mat.mtrl-sci 2026-07 conditional novelty 6.0 of 10

   Independent CNNs plus an Alice–Bob uncertainty pair extract Aex, Ku, D, and Npin from FORC images and recreate the input magnetometry in closed loop.

5. OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

   cs.LG 2026-07 conditional novelty 6.0 of 10

   A meta-pipeline plus LMO four-axis view yields a dual taxonomy of 108 optimizers, and a multi-objective LLM/vision benchmark shows no single family dominates the quality–cost–memory frontier.

6. bAdag: an adaptive block coordinate gradient method for smooth nonconvex functions

   math.OC 2026-06 unverdicted novelty 6.0 of 10

   Introduces bAdag, an AdaGrad-based block coordinate gradient method with ergodic sublinear convergence proofs for smooth nonconvex objectives under block Lipschitz gradient assumptions, covering cyclic, uniform random...

7. A stochastic gradient algorithm for non-separable optimization with convergence guarantee

   math.OC 2026-06 unverdicted novelty 6.0 of 10

   Presents a stochastic gradient algorithm for non-separable optimization with local convergence guarantees under smoothness assumptions.

8. Voronoi-Elitism Genetic Algorithm: A Generic Derivative-Free Routine With Theory and Implementation for Statistical Optimization

   stat.CO 2026-05 unverdicted novelty 6.0 of 10

   VEGA is a derivative-free genetic algorithm that constructs Voronoi neighborhoods around retained elite candidates to balance exploitation and exploration in high-dimensional statistical optimization.

9. De-risking solutions to optimization problems

   math.OC 2026-05 unverdicted novelty 6.0 of 10

   A softmax-based cutting-plane method de-risks solutions of generic optimization problems by reducing an impact metric with limited cost increase, or certifies impossibility.

10. Global Convergence of Sampling-Based Nonconvex Optimization through Diffusion-Style Smoothing

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Recasts sampling-based nonconvex optimization as smoothed gradient descent to obtain non-asymptotic convergence guarantees and introduces the DIDA annealed algorithm that converges to the global optimum.

11. Auto-Conditioned Frank-Wolfe Algorithms

    math.OC 2026-05 unverdicted novelty 6.0 of 10

    The paper proposes an auto-conditioned framework for Frank-Wolfe algorithms that replaces global smoothness constants with local estimators computed from first-order information, achieving convergence to stationary po...

12. Sketched Gaussian Mechanism for Private Federated Learning

    cs.LG 2025-09 conditional novelty 6.0 of 10

    A sketched Gaussian mechanism is shown to have privacy loss that shrinks as the sketch dimension grows, giving communication-efficient federated learning with stronger privacy per noise budget.

13. Dynamic Low-rank Approximation of Full-Matrix Preconditioner for Training Generalized Linear Models

    cs.LG 2025-08 conditional novelty 6.0 of 10

    AdaGram applies a dynamical low-rank integrator to the preconditioner of adaptive gradient methods, achieving full-matrix-like updates with rank 1-5 approximations at low cost on small GLM benchmarks.

14. Dynamic Domain Information Modulation Algorithm for Multi-domain Sentiment Analysis

    cs.CL 2025-05 reject novelty 6.0 of 10

    A new algorithm (DAMA) learns a per-domain scalar step size that modulates the input's domain information via gradients, yielding a modest 0.3% average accuracy improvement over a multi-task baseline.

15. Structural Alignment in Link Prediction

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Knowledge graph link prediction and KGEM hyperparameter preference can be modelled from graph structural features alone; TWIG and TWIG-I demonstrate this and enable cross-KG transfer.

16. Revisiting Transformers through the Lens of Low Entropy and Dynamic Sparsity

    cs.LG 2025-04 conditional novelty 6.0 of 10

    In controlled toy experiments, Transformers converge to lower-entropy distributions than the target, an effect driven by the FFN and linked to dynamic sparsity patterns.

17. prunAdag: an adaptive pruning-aware gradient method

    math.OC 2025-02 conditional novelty 6.0 of 10

    prunAdag separates parameters into optimisable and decreasable sets, updates them with Adagrad-like rules, and provably drives the average gradient norm to zero at rate O(log(k)/sqrt(k+1)).

18. First-Passage Approach to Optimizing Perturbations for Improved Training of Machine Learning Models

    cs.LG 2025-02 conditional novelty 6.0 of 10

    The authors show that when unperturbed neural network training reaches a quasi-steady state, the mean time to a target test accuracy under periodic perturbations can be predicted from a single perturbation experiment.

19. Expert-Free Online Transfer Learning in Multi-Agent Reinforcement Learning

    cs.AI 2025-01 conditional novelty 6.0 of 10

    EF-OnTL lets teams of imperfect RL agents share selected experience online, guided by uncertainty and surprise, and is reported to match or beat expert-based advice baselines in four testbeds.

20. Skeleton and Font Generation Network for Zero-shot Chinese Character Generation

    cs.CV 2025-01 conditional novelty 6.0 of 10

    SFGN generates Chinese characters from radical and stroke captions and transfers style via a transitive-attention mechanism, improving zero-shot generation and error-correction augmentation.

21. Dynamic Learning Rate Decay for Stochastic Variational Inference

    cs.CE 2024-12 conditional novelty 6.0 of 10

    The paper introduces DLRD, an adaptive learning-rate decay rule for SVI based on the signal-to-noise ratio of the variational parameter trajectory.

22. Complexities of Armijo-like algorithms in Deep Learning context

    math.OC 2024-12 reject novelty 6.0 of 10

    Armijo backtracking variants are shown to achieve optimal first-order complexity under (L0,L1) smoothness and on analytic functions, with no hyperparameter tuning required.

23. COAP: Memory-Efficient Training with Correlation-Aware Gradient Projection

    cs.LG 2024-11 conditional novelty 6.0 of 10

    COAP compresses optimizer states via a correlation-aware, occasionally recalibrated low-rank projection, matching AdamW performance while cutting optimizer memory by up to 81%.

24. ExpTest: Automating Learning Rate Searching and Tuning with Insights from Linearized Neural Networks

    cs.LG 2024-11 conditional novelty 6.0 of 10

    ExpTest auto-selects and tunes the learning rate by testing whether the training loss decays exponentially, without needing an initial learning rate choice.

25. Probe-Me-Not: Protecting Pre-trained Encoders from Malicious Probing

    cs.CR 2024-11 conditional novelty 6.0 of 10

    EncoderLock modifies a small set of pre-trained encoder weights so that linear probing succeeds on authorized domains while failing on prohibited domains, in supervised, unsupervised, and zero-shot data scenarios.

26. MARS: Unleashing the Power of Variance Reduction for Training Large Models

    cs.LG 2024-11 conditional novelty 6.0 of 10

    MARS combines STORM-style variance reduction with preconditioned updates and reports improved token efficiency over AdamW on GPT-2 pretraining, though the analyzed and evaluated variants differ.

27. Few-Shot Generalization for Single-Image 3D Reconstruction via Priors

    cs.CV 2019-09 conditional novelty 6.0 of 10

    A single-view 3D reconstruction network that refines a category-averaged prior shape achieves few-shot generalization to novel object classes without retraining or novel-class images.

28. One Model to Learn Both: Zero Pronoun Prediction and Translation

    cs.CL 2019-09 conditional novelty 6.0 of 10

    By jointly predicting and translating zero pronouns in one model, the authors improve Chinese-English BLEU by 5.3 points and Japanese-English BLEU by 2.1 points over a strong baseline.

29. Rethinking Irregular Scene Text Recognition

    cs.CV 2019-08 conditional novelty 6.0 of 10

    Curved synthetic training data, aspect-ratio-preserving resizing, and rotation augmentation push rectification-based scene text recognizers to state-of-the-art accuracy on curved text benchmarks.

30. Defeating Misclassification Attacks Against Transfer Learning

    cs.LG 2019-08 conditional novelty 6.0 of 10

    An ensemble of five pruned two-class differentiators rejects over 90% of targeted and non-targeted misclassification attacks in transfer learning with less than 10% accuracy loss, under a black-box attack model.

31. Controllable Video Captioning with POS Sequence Guidance Based on Gated Fusion Network

    cs.CV 2019-08 conditional novelty 6.0 of 10

    A video captioning model with cross-gated feature fusion and predicted part-of-speech sequence guidance improves caption scores and enables syntactic control.

32. Adaptive Embedding Gate for Attention-Based Scene Text Recognition

    cs.CV 2019-08 conditional novelty 6.0 of 10

    A gating module that modulates the previous-prediction embedding by estimated character-pair correlations improves attention-based scene text recognition accuracy and noise robustness.

33. Spatio-temporal Manifold Learning for Human Motions via Long-horizon Modeling

    cs.GR 2019-08 conditional novelty 6.0 of 10

    STRNN, a spatio-temporal recurrent network with part-based skeleton encoding and long-horizon prediction, generates stable human motions for up to 20,000 frames in open-loop mode.

34. Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networks

    cs.LG 2019-08 conditional novelty 6.0 of 10

    A systematic empirical study with a new tool, LRBench, evaluates 13 learning rate policies and finds that tuned cyclic schedules can beat framework defaults in accuracy and cost.

35. End-to-End Multi-Speaker Speech Recognition using Speaker Embeddings and Transfer Learning

    eess.AS 2019-08 conditional novelty 6.0 of 10

    Conditioning a hybrid CTC/attention ASR on speaker embeddings and adding transfer learning from clean speech reduces word error rate on overlapped two-speaker speech to 14.6%, from a prior best of 25.4%.

36. On the Variance of the Adaptive Learning Rate and Beyond

    cs.LG 2019-08 conditional novelty 6.0 of 10

    The paper identifies high early variance in Adam's adaptive learning rate as the reason warmup helps, and introduces RAdam, which rectifies that variance without needing a warmup schedule.

37. AttentionBoost: Learning What to Attend by Boosting Fully Convolutional Networks

    cs.CV 2019-08 conditional novelty 6.0 of 10

    Per-pixel loss reweighting across stages, inspired by AdaBoost, improves gland segmentation accuracy over fixed-attention and non-boosted multi-stage baselines on a private dataset.

38. Symmetry-constrained Rectification Network for Scene Text Recognition

    cs.CV 2019-08 conditional novelty 6.0 of 10

    A symmetry-constrained rectification network that predicts per-character geometry improves scene text recognition on regular and irregular text, achieving state-of-the-art on CUTE80, SVTP, and ICDAR 2015.

39. How Does Learning Rate Decay Help Modern Neural Networks?

    cs.LG 2019-08 conditional novelty 6.0 of 10

    Learning rate decay works because large learning rates suppress memorization of noisy data, while smaller learning rates enable learning of more complex patterns.

40. Cross-Attention End-to-End ASR for Two-Party Conversations

    eess.AS 2019-07 unverdicted novelty 6.0 of 10

    End-to-end ASR model with speaker-specific cross-attention for two-party conversations outperforms standard models on the Switchboard corpus.

41. Gated Embeddings in End-to-End Speech Recognition for Conversational-Context Fusion

    cs.CL 2019-06 unverdicted novelty 6.0 of 10

    Gated fusion of fastText and BERT embeddings into an end-to-end ASR model captures multi-sentence conversational context and lowers word error rate on the Switchboard corpus.

42. Adaptive Hybrid Particle Swarm Optimization with Gradient Descent

    cs.AI 2026-08 conditional novelty 5.0 of 10

    A diversity-gated sigmoid that blends PSO with gradient descent helps only on smooth-basin problems; with equal evaluation budgets, standard PSO wins most comparisons.

43. Adaptivity via a Parallel Architecture for Stochastic Gradient Methods

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Parallel geometric search over the iteration budget converts static SGD into an adaptive method with approximation factor α_p = (1+1/p)(p+1)^{1/p}.

44. NeuralChaos: Optimal Adapted Approximation of Square Integrable Predictable Processes

    math.PR 2026-07 conditional novelty 5.0 of 10

    A finite-sampling neural architecture is dense in the Hilbert space of square-integrable predictable processes and attains best-N-term chaoslet rates for compressible or Malliavin-regular processes.

45. PAPA: Online Personalized Active Preference Alignment

    cs.LG 2026-07 unverdicted novelty 5.0 of 10

    PAPA directly optimizes diffusion models via real-time user feedback for personalized preference alignment, drawing from variational inference, with an efficiency-enhanced variant EPAPA.

46. Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    AdaNAGED combines zeroth-order gradient-free training, automatic parameter adaptation, and LMO-based non-Euclidean geometry with claimed convergence guarantees, demonstrated on OPT-1.3B fine-tuning.

47. Pareto-Guided Teacher Alignment for Fair Personalized Text Generation

    cs.CL 2026-06 unverdicted novelty 5.0 of 10

    Fairness mitigation in personalized text generation is objective-dependent with methods occupying different regions of the fairness-personalization Pareto frontier rather than any single strategy dominating all objectives.

48. Objective-Function Free Multi-Objective Optimization: Rate of Convergence and Performance of an Adagrad-like algorithm

    math.OC 2026-02 conditional novelty 5.0 of 10

    MO-Adagrad finds Pareto critical points at rate O(1/√k) in the squared norm of a common descent direction while evaluating no objective function.

49. Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Training

    cs.LG 2025-09 unverdicted novelty 5.0 of 10

    Proposes low-rank orthogonalization and derives low-rank Muon and MSGD variants that outperform standard Muon on GPT-2 and LLaMA pretraining while providing iteration complexity bounds.

50. Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulator

    cs.LG 2025-07 conditional novelty 5.0 of 10

    The squared gradient accumulator from Adam can serve as a free, drop-in approximation of the empirical Fisher diagonal across model merging, pruning, task embedding, and continual learning, with comparable performance.

51. Fast Variational Bayes for Large Spatial Data

    stat.CO 2025-07 conditional novelty 5.0 of 10

    New variational Bayes algorithms for NNGP spatial models achieve accuracy close to MCMC while being faster, and improve on existing variational approaches.

52. SoftSignSGD(S3): An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adam

    cs.LG 2025-07 reject novelty 5.0 of 10

    S3, an optimizer with a p-th order momentum denominator, equal EMA coefficients, and Nesterov acceleration, is claimed to match AdamW's 100k-step perplexity at 50k steps while avoiding loss spikes.

53. Subspace-based Approximate Hessian Method for Zeroth-Order Optimization

    cs.LG 2025-07 conditional novelty 5.0 of 10

    ZO-SAH accelerates zeroth-order optimization by estimating and using subspace Hessians via quadratic fitting with evaluation reuse, achieving faster convergence on logistic regression and neural network benchmarks.

54. Fractional-order Jacobian Matrix Differentiation and Its Application in Artificial Neural Networks

    cs.LG 2025-06 reject novelty 5.0 of 10

    The authors introduce a fractional-order Jacobian matrix differentiation method for Autograd, implement it as FLinear in PyTorch, and claim improved performance on DJI and ETTh1 for fractional orders near 0.9.

55. Learning by solving differential equations

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Runge-Kutta optimizers adapted with momentum, preconditioning, or adaptive learning rates can close the large-batch generalization gap and match Adam on small MLP workloads.

56. HessFormer: Hessians at Foundation Scale

    cs.LG 2025-05 conditional novelty 5.0 of 10

    HessFormer delivers distributed Hessian-vector products for 70B-parameter models and reports the first spectral-density plots, including large negative eigenvalues, for DeepSeek-LLM-70B.

57. FAD: Frequency Adaptation and Diversion for Cross-domain Few-shot Learning

    cs.CV 2025-05 reject novelty 5.0 of 10

    A frequency-splitting adapter (FAD) improves Meta-Dataset few-shot accuracy by about half a point over strong baselines, though the evaluation protocol is compromised by test-set hyperparameter selection.

58. Block Circulant Adapter for Large Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Block Circulant Adapter (BCA) fine-tunes LLMs with block circulant weight matrices, matching LoRA/FourierFT task performance while using fewer parameters than LoRA and fewer FLOPs than FourierFT.

59. SemiHMER: Semi-supervised Handwritten Mathematical Expression Recognition using pseudo-labels

    cs.CV 2025-02 conditional novelty 5.0 of 10

    SemiHMER combines dual-branch pseudo-supervision, weak-to-strong augmentation, and a dynamic counting module to improve handwritten math formula recognition on CROHME benchmarks.

60. A Parameter-Free and Near-Optimal Zeroth-Order Algorithm for Stochastic Convex Optimization

    math.OC 2025-02 conditional novelty 5.0 of 10

    POEM is a parameter-free stochastic zeroth-order method that adapts both step size and smoothing automatically and reaches near-optimal oracle complexity.

See all 96 Pith citations