Floating-point neural networks with automatic differentiation can represent arbitrary floating-point functions and their gradients under mild conditions.
hub
Mish: A self regularized non-monotonic activation function
21 Pith papers cite this work. Polarity classification is still indexing.
21
Pith papers citing it
abstract
We propose $\textit{Mish}$, a novel self-regularized non-monotonic activation function which can be mathematically defined as: $f(x)=x\tanh(softplus(x))$. As activation functions play a crucial role in the performance and training dynamics in neural networks, we validated experimentally on several well-known benchmarks against the best combinations of architectures and activation functions. We also observe that data augmentation techniques have a favorable effect on benchmarks like ImageNet-1k and MS-COCO across multiple architectures. For example, Mish outperformed Leaky ReLU on YOLOv4 with a CSP-DarkNet-53 backbone on average precision ($AP_{50}^{val}$) by 2.1$\%$ in MS-COCO object detection and ReLU on ResNet-50 on ImageNet-1k in Top-1 accuracy by $\approx$1$\%$ while keeping all other network parameters and hyperparameters constant. Furthermore, we explore the mathematical formulation of Mish in relation with the Swish family of functions and propose an intuitive understanding on how the first derivative behavior may be acting as a regularizer helping the optimization of deep neural networks. Code is publicly available at https://github.com/digantamisra98/Mish.
hub tools
citation-role summary
background 1
method 1
citation-polarity summary
representative citing papers
Conditional diffusion models trained with BO-aware strategies approximate the optimum distribution, enabling a Diffusion-based Mode Seeking acquisition function with a sub-optimality guarantee that outperforms baselines in experiments.
Floating-point neural networks achieve universal representability for practical activations like ReLU, sigmoid, and tanh under arbitrary reduction orders and bounded ulp errors in activations via a new distinguishability condition.
The authors derive a Maximally Scale-Stable Parameterization (MSSP) for MoE models that achieves robust learning-rate transfer and monotonic performance gains with scale across co-scaling regimes of width, experts, and sparsity.
VidTAG achieves fine-grained global video-to-GPS geolocalization via temporal frame alignment and denoising sequence refinement, reporting 20% gains at 1 km over GeoCLIP and 25% on CityGuessr68k.
DiFlowDubber is the first video dubbing system using a discrete flow matching backbone with two-stage training that pre-trains a zero-shot TTS then adapts it via cross-modal alignment to produce content-consistent, lip-synchronized speech.
A meshless Bayesian 3D tomography framework using physics-informed neural networks enables tractable uncertainty-aware velocity estimation from combined active and passive seismic data.
DiffusionVS applies diffusion policies with online training to visual servoing, reporting near-100% simulation and 93% real-world success rates while improving other VS networks.
Mixture of Activations mixes activation functions token-adaptively in FFNs via lightweight gates, strictly more expressive than fixed or learnable activations, and yields lower pretraining loss from 0.12B to 2B models.
GEM is a new family of C^{2N}-smooth rational activation functions with variants that achieve performance on par with or exceeding GELU on ResNet, GPT-2, and BERT benchmarks.
SeqWM introduces sequential autoregressive agent-wise world models for multi-robot MBRL, outperforming baselines in performance and sample efficiency on Bi-DexHands and Multi-Quadruped tasks with physical robot deployment.
Self-consistency training on real data improves amortized Bayesian model comparison accuracy under distribution shifts, especially in open-world misspecification when analytic or locally accurate surrogate likelihoods are available.
TD-MPC2 scales an implicit world-model RL method to a 317M-parameter agent that masters 80 tasks across four domains with a single hyperparameter configuration.
A source-position-dependent implicit neural representation for the directional weighting in a physics-constrained kernel enables sound field estimation that generalizes to unseen source positions without per-snapshot fine-tuning.
BerLU constructs a C1-differentiable activation with Lipschitz constant 1 via Bernstein polynomial approximation, showing better performance and efficiency than baselines on image classification with ViTs and CNNs.
YOLOv4 achieves 43.5% AP (65.7% AP50) on MS COCO at ~65 FPS on Tesla V100 by integrating WRC, CSP, CmBN, SAT, Mish activation, Mosaic augmentation, DropBlock, and CIoU loss.
First calibration results from the Eos four-tonne water Cherenkov detector validate simulations and reconstruction methods using deployed optical and radioactive sources.
Deep learning models extract content-agnostic voice biomarkers for depression and anxiety from a ~65k-utterance proprietary dataset, achieving 71% sensitivity and specificity when combined with lexical features.
The paper surveys deep learning methods such as Deep Equilibrium Nets and Physics-Informed Neural Networks for solving and estimating high-dimensional dynamic stochastic models in economics and finance.
A literature survey of Small Language Models (1-8B parameters) that can perform comparably or better than larger models, covering general-purpose and task-specific approaches plus creation techniques.
citing papers explorer
-
Floating-Point Networks with Automatic Differentiation Can Represent Almost All Floating-Point Functions and Their Gradients
Floating-point neural networks with automatic differentiation can represent arbitrary floating-point functions and their gradients under mild conditions.
-
Improving Bayesian Optimization via Training-Aware Conditional Diffusion Models
Conditional diffusion models trained with BO-aware strategies approximate the optimum distribution, enabling a Diffusion-based Mode Seeking acquisition function with a sub-optimality guarantee that outperforms baselines in experiments.
-
Expressive Power of Floating-Point Neural Networks with Arbitrary Reduction Orders and Inexact Activation Implementations
Floating-point neural networks achieve universal representability for practical activations like ReLU, sigmoid, and tanh under arbitrary reduction orders and bounded ulp errors in activations via a new distinguishability condition.
-
How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization
The authors derive a Maximally Scale-Stable Parameterization (MSSP) for MoE models that achieves robust learning-rate transfer and monotonic performance gains with scale across co-scaling regimes of width, experts, and sparsity.
-
VidTAG: Temporally Aligned Video to GPS Geolocalization with Denoising Sequence Prediction at a Global Scale
VidTAG achieves fine-grained global video-to-GPS geolocalization via temporal frame alignment and denoising sequence refinement, reporting 20% gains at 1 km over GeoCLIP and 25% on CityGuessr68k.
-
DiFlowDubber: Discrete Flow Matching for Automated Video Dubbing via Cross-Modal Alignment and Synchronization
DiFlowDubber is the first video dubbing system using a discrete flow matching backbone with two-stage training that pre-trains a zero-shot TTS then adapts it via cross-modal alignment to produce content-consistent, lip-synchronized speech.
-
Bayesian three-dimensional seismic travel-time tomography for active- and passive-source seismic data using physics-informed neural network
A meshless Bayesian 3D tomography framework using physics-informed neural networks enables tractable uncertainty-aware velocity estimation from combined active and passive seismic data.
-
DiffusionVS: A Generative Framework for Robust Visual Servoing Based on Diffusion Policy
DiffusionVS applies diffusion policies with online training to visual servoing, reporting near-100% simulation and 93% real-world success rates while improving other VS networks.
-
More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations
Mixture of Activations mixes activation functions token-adaptively in FFNs via lightweight gates, strictly more expressive than fixed or learnable activations, and yields lower pretraining loss from 0.12B to 2B models.
-
Geometric Monomial (GEM): a family of rational 2N-differentiable activation functions
GEM is a new family of C^{2N}-smooth rational activation functions with variants that achieve performance on par with or exceeding GELU on ResNet, GPT-2, and BERT benchmarks.
-
Empowering Multi-Robot Cooperation via Sequential World Models
SeqWM introduces sequential autoregressive agent-wise world models for multi-robot MBRL, outperforming baselines in performance and sample efficiency on Bi-DexHands and Multi-Quadruped tasks with physical robot deployment.
-
Improving the Accuracy of Amortized Model Comparison with Self-Consistency
Self-consistency training on real data improves amortized Bayesian model comparison accuracy under distribution shifts, especially in open-world misspecification when analytic or locally accurate surrogate likelihoods are available.
-
TD-MPC2: Scalable, Robust World Models for Continuous Control
TD-MPC2 scales an implicit world-model RL method to a 317M-parameter agent that masters 80 tasks across four domains with a single hyperparameter configuration.
-
Learning-based Physics-Constrained Neural Kernel for Sound Field Estimation With Source-Position-Dependent Directional Weighting
A source-position-dependent implicit neural representation for the directional weighting in a physics-constrained kernel enables sound field estimation that generalizes to unseen source positions without per-snapshot fine-tuning.
-
Universal Smoothness via Bernstein Polynomials: A Constructive Approximation Approach for Activation Functions
BerLU constructs a C1-differentiable activation with Lipschitz constant 1 via Bernstein polynomial approximation, showing better performance and efficiency than baselines on image classification with ViTs and CNNs.
-
YOLOv4: Optimal Speed and Accuracy of Object Detection
YOLOv4 achieves 43.5% AP (65.7% AP50) on MS COCO at ~65 FPS on Tesla V100 by integrating WRC, CSP, CmBN, SAT, Mish activation, Mosaic augmentation, DropBlock, and CIoU loss.
-
Performance of the Eos detector with water
First calibration results from the Eos four-tonne water Cherenkov detector validate simulations and reconstruction methods using deployed optical and radioactive sources.
-
Voice Biomarkers for Depression and Anxiety
Deep learning models extract content-agnostic voice biomarkers for depression and anxiety from a ~65k-utterance proprietary dataset, achieving 71% sensitivity and specificity when combined with lexical features.
-
Deep Learning for Solving and Estimating Dynamic Models in Economics and Finance
The paper surveys deep learning methods such as Deep Equilibrium Nets and Physics-Informed Neural Networks for solving and estimating high-dimensional dynamic stochastic models in economics and finance.
-
Small Language Models (SLMs) Can Still Pack a Punch: A survey (updated 2026)
A literature survey of Small Language Models (1-8B parameters) that can perform comparably or better than larger models, covering general-purpose and task-specific approaches plus creation techniques.
- Functional-prior-based approaches to Bayesian PDE-constrained inversion using physics-informed neural networks