Mish: A Self Regularized Non-Monotonic Activation Function
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:QQZ22ZJ6record.jsonopen to challenge →
read the original abstract
We propose $\textit{Mish}$, a novel self-regularized non-monotonic activation function which can be mathematically defined as: $f(x)=x\tanh(softplus(x))$. As activation functions play a crucial role in the performance and training dynamics in neural networks, we validated experimentally on several well-known benchmarks against the best combinations of architectures and activation functions. We also observe that data augmentation techniques have a favorable effect on benchmarks like ImageNet-1k and MS-COCO across multiple architectures. For example, Mish outperformed Leaky ReLU on YOLOv4 with a CSP-DarkNet-53 backbone on average precision ($AP_{50}^{val}$) by 2.1$\%$ in MS-COCO object detection and ReLU on ResNet-50 on ImageNet-1k in Top-1 accuracy by $\approx$1$\%$ while keeping all other network parameters and hyperparameters constant. Furthermore, we explore the mathematical formulation of Mish in relation with the Swish family of functions and propose an intuitive understanding on how the first derivative behavior may be acting as a regularizer helping the optimization of deep neural networks. Code is publicly available at https://github.com/digantamisra98/Mish.
This paper has not been read by Pith yet.
Forward citations
Cited by 22 Pith papers
-
Floating-Point Networks with Automatic Differentiation Can Represent Almost All Floating-Point Functions and Their Gradients
Floating-point neural networks with automatic differentiation can represent arbitrary floating-point functions and their gradients under mild conditions.
-
Improving Bayesian Optimization via Training-Aware Conditional Diffusion Models
Conditional diffusion models trained with BO-aware strategies approximate the optimum distribution, enabling a Diffusion-based Mode Seeking acquisition function with a sub-optimality guarantee that outperforms baselin...
-
Expressive Power of Floating-Point Neural Networks with Arbitrary Reduction Orders and Inexact Activation Implementations
Floating-point neural networks achieve universal representability for practical activations like ReLU, sigmoid, and tanh under arbitrary reduction orders and bounded ulp errors in activations via a new distinguishabil...
-
How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization
The authors derive a Maximally Scale-Stable Parameterization (MSSP) for MoE models that achieves robust learning-rate transfer and monotonic performance gains with scale across co-scaling regimes of width, experts, an...
-
Functional-prior-based approaches to Bayesian PDE-constrained inversion using physics-informed neural networks
A unified framework with FPI-BPINN and fParVI-PINN approaches enables functional priors in Bayesian PINN inversion, yielding accurate posterior estimates for 1D seismic tomography and 2D Darcy flow permeability inversion.
-
VidTAG: Temporally Aligned Video to GPS Geolocalization with Denoising Sequence Prediction at a Global Scale
VidTAG achieves fine-grained global video-to-GPS geolocalization via temporal frame alignment and denoising sequence refinement, reporting 20% gains at 1 km over GeoCLIP and 25% on CityGuessr68k.
-
DiFlowDubber: Discrete Flow Matching for Automated Video Dubbing via Cross-Modal Alignment and Synchronization
DiFlowDubber is the first video dubbing system using a discrete flow matching backbone with two-stage training that pre-trains a zero-shot TTS then adapts it via cross-modal alignment to produce content-consistent, li...
-
Bayesian three-dimensional seismic travel-time tomography for active- and passive-source seismic data using physics-informed neural network
A meshless Bayesian 3D tomography framework using physics-informed neural networks enables tractable uncertainty-aware velocity estimation from combined active and passive seismic data.
-
DiffusionVS: A Generative Framework for Robust Visual Servoing Based on Diffusion Policy
DiffusionVS applies diffusion policies with online training to visual servoing, reporting near-100% simulation and 93% real-world success rates while improving other VS networks.
-
More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations
Mixture of Activations mixes activation functions token-adaptively in FFNs via lightweight gates, strictly more expressive than fixed or learnable activations, and yields lower pretraining loss from 0.12B to 2B models.
-
Functional-prior-based approaches to Bayesian PDE-constrained inversion using physics-informed neural networks
Presents fpBPINN framework using FPI-BPINN and fParVI-PINN to enable functional priors in Bayesian PINN-based PDE inversion, with random Fourier features aiding Gaussian prior representation.
-
Geometric Monomial (GEM): a family of rational 2N-differentiable activation functions
GEM is a new family of C^{2N}-smooth rational activation functions with variants that achieve performance on par with or exceeding GELU on ResNet, GPT-2, and BERT benchmarks.
-
Empowering Multi-Robot Cooperation via Sequential World Models
SeqWM introduces sequential autoregressive agent-wise world models for multi-robot MBRL, outperforming baselines in performance and sample efficiency on Bi-DexHands and Multi-Quadruped tasks with physical robot deployment.
-
Improving the Accuracy of Amortized Model Comparison with Self-Consistency
Self-consistency training on real data improves amortized Bayesian model comparison accuracy under distribution shifts, especially in open-world misspecification when analytic or locally accurate surrogate likelihoods...
-
TD-MPC2: Scalable, Robust World Models for Continuous Control
TD-MPC2 scales an implicit world-model RL method to a 317M-parameter agent that masters 80 tasks across four domains with a single hyperparameter configuration.
-
Learning-based Physics-Constrained Neural Kernel for Sound Field Estimation With Source-Position-Dependent Directional Weighting
A source-position-dependent implicit neural representation for the directional weighting in a physics-constrained kernel enables sound field estimation that generalizes to unseen source positions without per-snapshot ...
-
Universal Smoothness via Bernstein Polynomials: A Constructive Approximation Approach for Activation Functions
BerLU constructs a C1-differentiable activation with Lipschitz constant 1 via Bernstein polynomial approximation, showing better performance and efficiency than baselines on image classification with ViTs and CNNs.
-
YOLOv4: Optimal Speed and Accuracy of Object Detection
YOLOv4 achieves 43.5% AP (65.7% AP50) on MS COCO at ~65 FPS on Tesla V100 by integrating WRC, CSP, CmBN, SAT, Mish activation, Mosaic augmentation, DropBlock, and CIoU loss.
-
Performance of the Eos detector with water
First calibration results from the Eos four-tonne water Cherenkov detector validate simulations and reconstruction methods using deployed optical and radioactive sources.
-
Voice Biomarkers for Depression and Anxiety
Deep learning models extract content-agnostic voice biomarkers for depression and anxiety from a ~65k-utterance proprietary dataset, achieving 71% sensitivity and specificity when combined with lexical features.
-
Deep Learning for Solving and Estimating Dynamic Models in Economics and Finance
The paper surveys deep learning methods such as Deep Equilibrium Nets and Physics-Informed Neural Networks for solving and estimating high-dimensional dynamic stochastic models in economics and finance.
-
Small Language Models (SLMs) Can Still Pack a Punch: A survey (updated 2026)
A literature survey of Small Language Models (1-8B parameters) that can perform comparably or better than larger models, covering general-purpose and task-specific approaches plus creation techniques.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.