Pith. sign in

REVIEW 3 major objections 5 minor 45 cited by

Mish: A Self Regularized Non-Monotonic Activation Function

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proposes Mish, an activation function defined as f(x)=x tanh(softplus(x)), and reports that it consistently matches or exceeds ReLU, Swish, and Leaky ReLU across computer vision benchmarks.

desk verdict A genuinely new activation function with broad empirical coverage, but the paper's own statistics do not support the claimed consistent edge over Swish. read the letter →

arxiv 1908.08681 v3 pith:QQZ22ZJ6 submitted 2019-08-23 cs.LG cs.CVcs.NEstat.ML

Diganta Misra This is my paper · ORCID
classification cs.LGcs.CVcs.NEstat.ML
keywords activationfunctionMishself-regularizationnon-monotonicdeeplearningimageclassificationobjectdetectionSwish
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a new neural-network activation function, Mish, defined by $f(x)=x\tanh(\mathrm{softplus}(x))$, and claims that swapping it into standard convolutional networks is a reliable way to improve accuracy without touching other settings. On ImageNet-1k classification, Mish is reported to beat ReLU on ResNet-50 by about 1% Top-1 accuracy; on MS-COCO object detection, it beats Leaky ReLU in YOLOv4 by up to 2.1% in $AP_{50}^{val}$. The author argues that Mish's smooth, non-monotonic, bounded-below shape preserves small negative weights and that the $\Delta(x)=\mathrm{sech}^2(\mathrm{softplus}(x))$ factor in its derivative acts like a preconditioner, smoothing the loss landscape and regularizing training. A reader should care because if these results hold, a working model can often be improved by a one-line activation change, with no retuning.

What carries the argument

The central object is the activation function itself, $f(x)=x\tanh(\mathrm{softplus}(x))$, which combines identity, softplus, and tanh into a smooth non-monotonic function that is bounded below and unbounded above. Its load-bearing identity is the derivative decomposition $f'(x)=\Delta(x)\,\mathrm{swish}(x)+f(x)/x$, where $\Delta(x)=\mathrm{sech}^2(\mathrm{softplus}(x))$; the paper treats $\Delta(x)$ as a preconditioner that smooths the optimization landscape and thereby regularizes training. Mish also uses the self-gating property from Swish, multiplying the unmodulated input by the output of a nonlinear function of the input, and its smoothness and small negative-weight retention are credited with better gradient flow and avoidance of Dying ReLU. An optimized CUDA implementation, Mish-CUDA, is used to mitigate the extra computational cost of the formula.

What would settle it

Train the same architectures (e.g., ResNet-50 on ImageNet-1k and YOLOv4 on MS-COCO) with ReLU, Swish, Leaky ReLU, and Mish across many seeds, keeping all augmentation and hyperparameter settings fixed, and check whether the mean gaps persist above the seed variance. If the 1-2% differences shrink below one standard deviation, or change sign across architectures, the central empirical claim is not settled.

Watch

Extended reading notes

Core claim

The central claim is that Mish is a self-regularized non-monotonic activation function that matches or improves on ReLU, Swish, and Leaky ReLU in the vision settings tested. The paper defines Mish as $f(x)=x\tanh(\ln(1+e^x))$, notes its range is approximately $[-0.31,\infty)$, and derives $f'(x)=\Delta(x)\,\mathrm{swish}(x)+f(x)/x$ with $\Delta(x)=\mathrm{sech}^2(\mathrm{softplus}(x))$. This decomposition is used to argue that the $\Delta$ factor acts as a preconditioner that makes gradients and loss contours smoother, which the author identifies as the likely mechanism behind Mish's gains. Evidence includes a 23-run CIFAR-10 study, architecture sweeps on CIFAR-10, ImageNet-1k results on ResNet and CSP-family models, and MS-COCO object detection with YOLOv4, including combinations with CutMix, Mosaic, self-adversarial training, DropBlock, and label smoothing.

Load-bearing premise

The load-bearing premise is that the reported accuracy differences, roughly 0.4% to 2.1%, are caused by the activation function rather than by run-to-run randomness or unlisted training details, since most large-scale results are single runs without error bars.

Editorial extensions

If this is right

  • Practitioners can treat Mish as a drop-in replacement in standard CNN backbones and expect small accuracy gains without retuning hyperparameters.
  • Mish is more dependable than Swish in large or deep models: the paper reports Swish dropping ImageNet Top-1 by 13.4% on CSP-ResNext-50 while Mish improves it by 1%.
  • Mish is compatible with modern augmentation and regularization schemes such as CutMix, Mosaic, self-adversarial training, DropBlock, and label smoothing, with consistent improvements.
  • The optimized Mish-CUDA implementation brings forward and backward pass times close to native ReLU levels, making the accuracy gains available at modest computational cost.
  • Because Mish consistently matched or exceeded baselines across ResNet, Wide ResNet, DenseNet, MobileNet, ShuffleNet, Inception, EfficientNet, and detection backbones, the reported benefit is not tied to one architecture family.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: if the $\Delta(x)$ preconditioner story is right, then activation design could focus on shaping the derivative's smoothing factor rather than hunting for function shapes; one test would be to vary the width or exponent of the sech-squared factor and measure loss-landscape curvature.
  • The paper leaves non-vision tasks untested; a natural extension is to measure Mish in transformers or recurrent models, where smooth bounded-below activations may behave differently.
  • Because the largest gains appear in hard, deep detection models, a testable extension is to check whether Mish's advantage grows with network depth or with smaller objects, using the per-size AP columns in Table 5.
  • The results were obtained with fixed hyperparameters; a hyperparameter search with Mish as the target, which the author explicitly suggests, could either widen the gains or reveal that some of the baseline comparisons depend on tuning.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a new activation function, Mish, defined as f(x)=x tanh(softplus(x)), and argues that it is a smooth, non-monotonic, 'self-regularized' activation that can replace ReLU, Leaky ReLU, or Swish in deep networks. The manuscript derives the first derivative, offers a speculative interpretation of the derivative's Delta(x) term as a preconditioner, and reports benchmark comparisons on CIFAR-10, ImageNet-1k, and MS-COCO across many architectures, plus a runtime comparison with a CUDA implementation. The central empirical claim is that Mish consistently matches or exceeds ReLU, Leaky ReLU, and Swish while keeping all other network parameters and hyperparameters constant.

Significance. If the empirical claim is robust, Mish would be a useful drop-in activation function: it is simple to define, publicly implemented, and shows a broad pattern of small accuracy gains across many architectures. The paper's strengths are its breadth of architectures, the public code, the exact derivative identity in Eq. (1), and the presence of a 23-run statistical study for one small CIFAR-10 setup. However, the significance is conditional: the headline gains rest largely on single runs with no error bars, and the one repeated-run experiment does not separate Mish from Swish or GELU. With additional uncertainty quantification or with claims rescaled to what the evidence supports, this could be a useful empirical reference point; as it stands, the 'consistently outperforms' claim is stronger than the evidence.

major comments (3)
  1. [§4.4, Table 5] The abstract's headline 2.1% AP50 improvement is taken from YOLOv4pacsp-s, the smallest variant. On the main YOLOv4pacsp model, AP50 improves by only 0.9% while AP75 decreases from 51.0% to 50.2% and APL is essentially flat (59.5% to 59.4%); these are single runs with no error bars. The mixed directions across metrics are exactly what one would expect from seed or configuration noise, so this table does not by itself establish a reliable activation effect.
  2. [§4.3, Table 3] The statement that Mish 'consistently outperforms' ReLU, Leaky ReLU, and Swish is contradicted by the PeleeNet row, where Swish achieves 71.5% Top-1 and 90.7% Top-5 against Mish's 71.4% and 90.4%. The same table also reports a 13.4-point Top-1 drop for Swish on CSP-ResNext-50 without augmentation, which is far larger than any reported activation-function effect in the literature and suggests an uncontrolled difference in training or configuration; citing it as evidence that Swish 'cannot be used in every architecture' is not supported by the controlled-comparison protocol described in the paper.
  3. [§4.1, Table 1] The only repeated-run experiment (23 runs, SqueezeNet on CIFAR-10) does not resolve Mish from its closest competitors. Mish's mean accuracy is 0.16 points above Swish and 0.11 points above GELU, while the per-run standard deviations are 0.40-0.47; no significance test or multiple-comparison correction is reported, so the differences are within plausible noise. The comparison with ReLU (0.82 points) is more favorable but still should be accompanied by a significance statement.
minor comments (5)
  1. [§4.4, text] The sentence listing hyperparameters says epochs, initial learning rate, weight decay, and momentum were set at '500500, 0.01, 5e-4, and 0.9'; '500500' appears to be a typo and should be clarified.
  2. [Table 4] The ReLU and Mish results are identical for CSP-DarkNet-53 with and without the PANet+SPP module (64.5% and 64.9% in both rows); this makes the comparison uninformative and needs an explanation or correction.
  3. [§2, §4.5] The claim that the Delta(x) term 'acts like a preconditioner' is explicitly speculative and should be labeled as a hypothesis rather than presented as an explanation of the empirical results.
  4. [§4.2-§4.4] All benchmark tables lack information on the number of runs, seeds, and variance; adding this information would materially improve the reader's ability to judge the reported 1-3% improvements.
  5. [§3] The 'self-regularized' property is never defined formally; the paper would benefit from a precise statement of what quantity is regularized and how it is measured.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the derivation is exact algebra and the empirical claims rest on external benchmarks.

full rationale

The paper's central mathematical content is the closed-form definition f(x) = x tanh(softplus(x)) and its exact first derivative, Eq. (1)-(4), which are algebraic identities rather than fitted or self-referential constructions. No parameter is fitted to a subset of the benchmarks and then renamed as a prediction; all reported comparisons are external evaluations in which the activation function is swapped while other network and training parameters are held constant. The preconditioning discussion in Section 2 is explicitly speculative ('we speculate', 'may be acting as a regularizer') and is not used to derive the empirical results, so it cannot make the conclusions circular. The paper contains no load-bearing self-citations: it cites standard references for architectures, datasets, and prior activation functions, and it does not invoke any uniqueness theorem or prior author result to rule out alternatives. Concerns about single-run large-scale benchmarks and the statistical resolution of small accuracy differences are methodological correctness risks, not circularity, and therefore do not affect this score.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on standard properties of tanh and softplus, on the empirical assumption that the benchmarks were run under controlled conditions, and on a speculative mechanism (Delta as preconditioner). There are no fitted parameters: the activation function has no trainable parameters and no constants are fitted to data for the benchmark comparisons.

assumptions (3)
  • domain assumption Smoother loss landscapes lead to easier optimization and better generalization, based on Li et al. (2018).
    Used to argue that Mish's smooth profile causes the observed improvement in Section 3 and Figure 4.
  • ad hoc to paper The Delta(x) term in the derivative behaves like a preconditioner that makes gradients smoother.
    Speculative, introduced in Section 2 without independent evidence or a formal derivation.
  • standard math Standard analytic properties of tanh and softplus (differentiability, monotonicity, limits) hold as assumed.
    Basic background for the function definition and derivative computations in Section 3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mish: A Self Regularized Non-Monotonic Activation Function." pith.science (2026). https://pith.science/paper/QQZ22ZJ6

@misc{pith2026190808681,
  author       = {Pith},
  title        = {Pith review of: Mish: A Self Regularized Non-Monotonic Activation Function},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QQZ22ZJ6}},
  note         = {Machine review of arXiv:1908.08681}
}
abstract

We propose $\textit{Mish}$, a novel self-regularized non-monotonic activation function which can be mathematically defined as: $f(x)=x\tanh(softplus(x))$. As activation functions play a crucial role in the performance and training dynamics in neural networks, we validated experimentally on several well-known benchmarks against the best combinations of architectures and activation functions. We also observe that data augmentation techniques have a favorable effect on benchmarks like ImageNet-1k and MS-COCO across multiple architectures. For example, Mish outperformed Leaky ReLU on YOLOv4 with a CSP-DarkNet-53 backbone on average precision ($AP_{50}^{val}$) by 2.1$\%$ in MS-COCO object detection and ReLU on ResNet-50 on ImageNet-1k in Top-1 accuracy by $\approx$1$\%$ while keeping all other network parameters and hyperparameters constant. Furthermore, we explore the mathematical formulation of Mish in relation with the Swish family of functions and propose an intuitive understanding on how the first derivative behavior may be acting as a regularizer helping the optimization of deep neural networks. Code is publicly available at https://github.com/digantamisra98/Mish.

Figures

Figures reproduced from arXiv: 1908.08681 by the authors.

Figure 1
Figure 1. (a) Graph of Mish, ReLU, SoftPlus, and Swish activation functions. As illustrated, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a) Graph of Mish, Swish, and similar validated experimental functions. (b) Train [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Comparison between the output landscapes of ReLU and Mish activation function [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison between the loss landscapes of (from left to right): (a) ReLU, (b) Mish [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: (a) Comparison between Mish, Swish, and ReLU activation functions in terms [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 45 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Silent Impact: Tracking Tennis Shots from the Passive Arm

    cs.HC 2025-07 conditional novelty 7.0 of 10

    Tennis shots can be detected and classified from IMU data on the passive arm alone, with accuracy close to dominant-arm sensors and lower perceived burden.

  2. Compress-Align-Detect: onboard change detection from unregistered images

    cs.CV 2025-07 conditional novelty 7.0 of 10

    A single neural network performs compression, co-registration, and change detection onboard a satellite, achieving F1 up to about 70% at low bitrates on simulated unregistered image pairs.

  3. Why Does Action Chunking Improve Behavioral Cloning Performance in Robotic Control?

    cs.RO 2026-08 conditional novelty 6.0 of 10

    Action chunking in robotic behavioral cloning works mainly because it acts as a delayed-prediction policy and an implicit ensemble, not because of temporal consistency or horizon reduction.

  4. UniRank: Benchmarking Ranking Models for Unified Sequential Modeling and Feature Interaction

    cs.IR 2026-07 conditional novelty 6.0 of 10

    UniRank is an open benchmark that standardizes chronological autoregressive supervision, multi-task evaluation, and capacity controls for 15 unified ranking models on five large datasets.

  5. Rethinking Neural Nonlinearity as Gating

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Common activations and softmax are instances of a single input-conditioned Threshold Gating primitive with few branches, enabling lossless conversion and a unified analog implementation path.

  6. LVM-GP: Uncertainty-Aware PDE Solver via coupling latent variable model and Gaussian process

    stat.ML 2025-07 conditional novelty 6.0 of 10

    A hybrid model coupling a Gaussian process latent field with a neural operator provides uncertainty estimates for forward and inverse PDE problems with noisy data.

  7. Joint Inference of Trajectory and Obstacle in Mean-Field Games via Bilevel Optimization

    math.OC 2025-07 conditional novelty 6.0 of 10

    A bilevel optimization framework with normalizing flows recovers the latent obstacle and optimal trajectories in mean-field games from partial trajectory data, achieving about 0.1 relative L2 error even with fewer tha...

  8. Latent-attention Based Transformer for Near ML Polar Decoding in Short-code Regime

    eess.SP 2025-07 conditional novelty 6.0 of 10

    A code-aware latent-attention transformer matches near-ML error rates for short polar codes only after per-configuration fine-tuning.

  9. EAT: QoS-Aware Edge-Collaborative AIGC Task Scheduling via Attention-Guided Diffusion Reinforcement Learning

    cs.DC 2025-07 conditional novelty 6.0 of 10

    EAT, an attention-guided diffusion reinforcement learning scheduler for collaborative edge AIGC, reduces Stable Diffusion inference latency by 56-74% versus baselines while keeping CLIP quality nearly unchanged.

  10. Transformers Learn Faster with Semantic Focus

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Input-dependent top-k sparse attention makes small transformers converge faster and generalize as well as full attention, while input-agnostic sparsity does not, and the effect is tied to reduced dispersion of attenti...

  11. A Composite Predictive-Generative Approach to Monaural Universal Speech Enhancement

    eess.AS 2025-05 conditional novelty 6.0 of 10

    PGUSE combines a predictive speech enhancer with a diffusion model, fusing their outputs and truncating the diffusion start to improve universal speech enhancement with low inference cost.

  12. Revisiting Feature Interactions from the Perspective of Quadratic Neural Networks for Click-through Rate Prediction

    cs.IR 2025-05 conditional novelty 6.0 of 10

    Hadamard product feature interactions work because they make CTR models quadratic networks, and the proposed QNN-alpha with multi-head Khatri-Rao product and self-ensemble loss achieves state-of-the-art results on six...

  13. FlowQ: Energy-Guided Flow Policies for Offline Reinforcement Learning

    cs.LG 2025-05 reject novelty 6.0 of 10

    FlowQ uses energy-guided flow matching to learn an offline RL policy approximating π(a|s) ∝ πβ(a|s) exp(Q(s,a)) with guidance applied during training rather than at inference.

  14. CoDynTrust: Robust Asynchronous Collaborative Perception via Dynamic Feature Trust Modulus

    cs.CV 2025-02 reject novelty 6.0 of 10

    CoDynTrust gates shared vehicle features by a learned trust score derived from aleatoric and epistemic uncertainty, improving 3D detection under time delays, though its evaluation leaks test-set statistics.

  15. A Hands-free Spatial Selection and Interaction Technique using Gaze and Blink Input with Blink Prediction for Extended Reality

    cs.HC 2025-01 conditional novelty 6.0 of 10

    A gaze-and-blink interaction technique for XR matches Gaze+Pinch in speed but has higher error rates, and a neural filter for involuntary blinks did not significantly reduce those errors.

  16. Parseval Regularization for Continual Reinforcement Learning

    cs.LG 2024-12 conditional novelty 6.0 of 10

    Parseval regularization, a cheap orthogonality-preserving penalty, improves continual RL agents' success on new tasks across gridworld, CARL and MetaWorld benchmarks.

  17. Ternary Stochastic Neuron -- Implemented with a Single Strained Magnetostrictive Nanomagnet

    cond-mat.mes-hall 2024-12 conditional novelty 6.0 of 10

    A strain-engineered magnetostrictive nanomagnet can produce the three-state activation function needed for a ternary stochastic neuron, according to Landau-Lifshitz-Gilbert simulations.

  18. Meteosat Third Generation imagery improves CNN-based SSI retrieval

    physics.ao-ph 2026-07 conditional novelty 5.5 of 10

    Higher-resolution MTG/FCI imagery cuts CNN SSI RMSE under overcast and cloudy skies versus MSG-only, with ~20% overall skill over SARAH-3, but not under clear skies.

  19. Relative Value Learning

    cs.LG 2026-07 conditional novelty 5.0 of 10

    A critic that learns antisymmetric value differences ∆(s_i,s_j)=V(s_i)−V(s_j) has a provably contracting Bellman operator and an unbiased advantage estimator, and PPO with this critic matches standard PPO on Atari.

  20. Measurement of the branching ratio of the $K^{+}\rightarrow\pi^{+}\nu\bar{\nu}$ decay

    hep-ex 2026-07 accept novelty 5.0 of 10

    NA62 measures Br(K⁺→π⁺νν̄) = (9.6±1.9)×10⁻¹¹ from 2016–2024 data, a 20% relative-precision measurement consistent with the Standard Model.

  21. Path optimization method for the sign problem: Insights from random matrix models

    hep-lat 2026-07 conditional novelty 5.0 of 10

    Path optimization improves the average phase factor in the Stephanov model at high chemical potential but not at low chemical potential or in the chiral random matrix model, pointing to the global sign problem as the ...

  22. Learning-based Physics-Constrained Neural Kernel for Sound Field Estimation With Source-Position-Dependent Directional Weighting

    cs.SD 2026-07 conditional novelty 5.0 of 10

    A source-position-dependent implicit neural representation for the directional weighting in a physics-constrained kernel enables sound field estimation that generalizes to unseen source positions without per-snapshot ...

  23. The Algorithm Is Not the Behavior: Learned Priors Override Look-Ahead in a Chess-Playing Neural Network

    cs.LG 2025-08 reject novelty 5.0 of 10

    The paper demonstrates non-monotonic move-policy dynamics in a chess transformer, but its abstract claims a causal safety-prior override result that never appears in the body.

  24. FLUX-Makeup: High-Fidelity, Identity-Consistent, and Robust Makeup Transfer via Diffusion Transformer

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    This document pairs a makeup-transfer abstract with a mismatched body about activation functions, leaving the claimed method unverifiable.

  25. Masked Temporal Interpolation Diffusion for Procedure Planning in Instructional Videos

    cs.CV 2025-07 conditional novelty 5.0 of 10

    MTID improves action sequence planning in instructional videos by adding a learnable latent interpolation module, a masked projection, and a task-adaptive proximity loss to a DDIM-based diffusion model.

  26. Distributional Soft Actor-Critic with Diffusion Policy

    cs.LG 2025-07 reject novelty 5.0 of 10

    DSAC-D couples a diffusion-based value distribution network with a diffusion policy, but its headline state-of-the-art claim is contradicted by its own benchmark table.

  27. Preserving Plasticity in Continual Learning with Adaptive Linearity Injection

    cs.LG 2025-05 conditional novelty 5.0 of 10

    AdaLin, a per-neuron learnable linearity injection gated by activation saturation, preserves plasticity in continual learning and off-policy RL without task boundaries or extra method hyperparameters.

  28. Neural Stereo Video Compression with Hybrid Disparity Compensation

    cs.CV 2025-04 conditional novelty 5.0 of 10

    A hybrid disparity compensation module, combining shifted cost volumes with normalized cross-attention, improves neural stereo video compression by up to 55% bitrate over MV-HEVC on driving benchmarks.

  29. Active RIS-Empowered Covert Satellite-Terrestrial Communications

    eess.SP 2025-04 conditional novelty 5.0 of 10

    An aerial active STAR-RIS with a generative diffusion-model DRL controller improves simulated covert satellite-to-ground capacity and fairness, but with limited reproducibility evidence.

  30. PAGNet: Pluggable Adaptive Generative Networks for Information Completion in Multi-Agent Communication

    cs.MA 2025-02 conditional novelty 5.0 of 10

    PAGNet learns per-agent communication weights and generates global states with a U-Net and GAN discriminator, reporting improved cooperative MARL performance on LBF, Hallway, and SMAC.

  31. Gompertz Linear Units: Leveraging Asymmetry for Enhanced Learning Dynamics

    cs.LG 2025-02 conditional novelty 5.0 of 10

    GoLU (x times e^{-e^{-x}}) matches or beats GELU and Swish on several benchmarks, but the gains often appear only after learning-rate tuning.

  32. Dual-Branch Residual Network for Cross-Domain Few-Shot Hyperspectral Image Classification with Refined Prototype

    cs.CV 2025-04 conditional novelty 4.0 of 10

    A dual-branch residual network with query-prototype contrastive refinement and MMD domain alignment attains the highest mean accuracy on four cross-domain few-shot hyperspectral benchmarks.

  33. Hadamard product in deep learning: Introduction, Advances and Challenges

    cs.LG 2025-04 conditional novelty 4.0 of 10

    A survey that taxonomizes deep learning applications of the Hadamard product into four domains and argues it is a core architectural primitive.

  34. Estimating Probabilities of Causation with Machine Learning Models

    cs.AI 2025-02 conditional novelty 4.0 of 10

    On a synthetic causal model, a multilayer perceptron with the Mish activation function predicts the lower and upper bounds of the probability of necessity and sufficiency for subpopulations with limited data, reaching...

  35. Softplus Attention with Re-weighting Boosts Length Extrapolation in Large Language Models

    cs.CL 2025-01 conditional novelty 4.0 of 10

    A two-stage softplus-based attention mechanism with re-weighting (LSSAR) is reported to keep validation loss nearly flat when a 124M-parameter GPT-2 is tested at up to 16x its 1024-token training length.

  36. Reconstructing Time-of-Flight Detector Values of Angular Streaking Using Machine Learning

    physics.data-an 2025-01 conditional novelty 4.0 of 10

    A convolutional autoencoder trained only on simulated data can denoise angular streaking images and reconstruct up to three missing time-of-flight detectors, with fast inference for online experiments.

  37. Regression Guided Strategy to Automated Facial Beauty Optimization through Image Synthesis

    cs.CV 2025-01 conditional novelty 4.0 of 10

    A face can be automatically 'beautified' by mapping it into StyleGAN2's latent space and using CMA-ES to optimize a learned beauty score, with only qualitative evidence of improvement.

  38. Reinforcement Learning: From Algorithms To Foundation Models

    cs.AI 2026-07 conditional novelty 3.0 of 10

    A dissertation uniting the author's published results: non-exploitable Nash-DQN policies and the FightLadder benchmark for games, plus diffusion/consistency-model world models for RL — a compilation rather than new results.

  39. Hybrid activation functions for deep neural networks: S3 and S4 -- a novel approach to gradient flow optimization

    cs.LG 2025-07 reject novelty 3.0 of 10

    A new parameterized activation function, S4, that blends sigmoid and softsign through a smooth sigmoid-weighted transition is claimed to improve accuracy and convergence on small neural network benchmarks.

  40. Tangma: A Tanh-Guided Activation Function with Learnable Parameters

    cs.NE 2025-07 conditional novelty 3.0 of 10

    Tangma, defined as x times tanh(x plus a learned shift) plus a learned linear term, reaches 99.09% validation accuracy on MNIST and 78.15% on CIFAR-10 after 10 epochs, slightly beating ReLU, Swish, and GELU in these s...

  41. SG-Blend: Learning an Interpolation Between Improved Swish and GELU for Robust Neural Representations

    cs.LG 2025-05 conditional novelty 3.0 of 10

    SG-Blend, a learnable convex combination of a shifted Swish and GELU, reports modest accuracy and BLEU gains on CIFAR-10, IMDB, and a WMT14 subset, with no variance reporting.

  42. Neural Networks Learn Distance Metrics

    cs.LG 2025-02 reject novelty 3.0 of 10

    A two-layer MNIST study argues that networks prefer distance-based representations, but the evidence is mostly architecture-specific and the OffsetL2 layer resembles known RBF units.

  43. Local Control Networks (LCNs): Optimizing Flexibility in Neural Network Data Pattern Capture

    cs.LG 2025-01 conditional novelty 3.0 of 10

    Local Control Networks put a separate learnable B-spline activation on every neuron and report small accuracy gains over MLPs and KANs on benchmark tasks.

  44. SoftReMish: A Novel Activation Function for Enhanced Convolutional Neural Networks for Visual Recognition Performance

    cs.CV 2025-07 reject novelty 2.0 of 10

    SoftReMish is presented as new but reduces to Mish with rescaled input and output, and its MNIST evidence is not self-consistent.

  45. YOLOv1 to YOLOv11: A Comprehensive Survey of Real-Time Object Detection Innovations and Challenges

    cs.CV 2025-08 unverdicted

    A survey of YOLO object detectors from version 1 to version 11 that compiles architectures, benchmarks, and applications, with several factual inconsistencies.

Reference graph

Works this paper leans on

52 extracted references · 20 canonical work pages · cited by 45 Pith papers

  1. [1]

    On the rate of convergence of the preconditioned conjugate gradient method

    Owe Axelsson and Gunhild Lindskog. On the rate of convergence of the preconditioned conjugate gradient method. Numerische Mathematik, 48(5):499–523, 1986

    work page 1986

  2. [2]

    Yolov4: Optimal speed and accuracy of object detection

    Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934, 2020

    arXiv 2004

  3. [3]

    Large-scale machine learning with stochastic gradient descent

    Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Pro- ceedings of COMPSTAT’2010, pages 177–186. Springer, 2010

    work page 2010

  4. [4]

    Improving Deep Learning by Inverse Square Root Linear Units (ISRLUs)

    Brad Carlile, Guy Delamarter, Paul Kinney, Akiko Marti, and Brian Whitney. Im- proving deep learning by inverse square root linear units (isrlus). arXiv preprint arXiv:1710.09967, 2017

    work page Pith review arXiv 2017

  5. [5]

    Xception: Deep learning with depthwise separable convolutions

    François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1251–1258, 2017

    2017

  6. [6]

    Fast and accurate deep network learning by exponential linear units (elus)

    Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289, 2015

    arXiv 2015

  7. [7]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  8. [8]

    Spinenet: Learning scale-permuted backbone for recognition and localization

    Xianzhi Du, Tsung-Yi Lin, Pengchong Jin, Golnaz Ghiasi, Mingxing Tan, Yin Cui, Quoc V Le, and Xiaodan Song. Spinenet: Learning scale-permuted backbone for recognition and localization. arXiv preprint arXiv:1912.05027, 2019

    arXiv 1912

Show all 52 references
  1. [9]

    Dropblock: A regularization method for convolutional networks

    Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V Le. Dropblock: A regularization method for convolutional networks. In Advances in Neural Information Processing Systems, pages 10727–10737, 2018

    2018

  2. [10]

    Rich feature hierar- chies for accurate object detection and semantic segmentation

    Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierar- chies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580–587, 2014. 12 MISRA: MISH ACTIV A TI...

    2014

  3. [11]

    Understanding the difficulty of training deep feed- forward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feed- forward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256, 2010

    2010

  4. [12]

    Lets keep it simple, using simple architectures to outperform deeper and more complex architectures

    Seyyed Hossein HasanPour, Mohammad Rouhani, Mohsen Fayyaz, and Mohammad Sabokrou. Lets keep it simple, using simple architectures to outperform deeper and more complex architectures. arXiv preprint arXiv:1608.06037, 2016

    2016 arXiv

  5. [13]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015

    2015

  6. [14]

    Spatial pyramid pooling in deep convolutional networks for visual recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE transactions on pattern analysis and machine intelligence, 37(9):1904–1916, 2015

    1904

  7. [15]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  8. [16]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016

    2016 arXiv

  9. [17]

    Mobilenets: Effi- cient convolutional neural networks for mobile vision applications

    Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Effi- cient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017

    2017 arXiv

  10. [18]

    Squeeze-and-excitation networks

    Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7132–7141, 2018

    2018

  11. [19]

    Densely connected convolutional networks

    Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017

    2017

  12. [20]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015

    2015 arXiv

  13. [21]

    Deep learning with s-shaped rectified linear activation units

    Xiaojie Jin, Chunyan Xu, Jiashi Feng, Yunchao Wei, Junjun Xiong, and Shuicheng Yan. Deep learning with s-shaped rectified linear activation units. In Thirtieth AAAI Conference on Artificial Intelligence, 2016

    2016

  14. [22]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  15. [23]

    Self- normalizing neural networks

    Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self- normalizing neural networks. In Advances in neural information processing systems , pages 971–980, 2017

    2017

  16. [24]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. MISRA: MISH ACTIV A TION FUNCTION 13

    2009

  17. [25]

    Imagenet classification with deep convolutional neural networks

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012

    2012

  18. [26]

    Mnist handwritten digit database

    Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann. lecun. com/exdb/mnist, 2, 2010

    2010

  19. [27]

    Efficient backprop

    Yann A LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. In Neural networks: Tricks of the trade, pages 9–48. Springer, 2012

    2012

  20. [28]

    Visualizing the loss landscape of neural nets

    Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems, pages 6389–6399, 2018

    2018

  21. [29]

    Preconditioned stochastic gradient descent

    Xi-Lin Li. Preconditioned stochastic gradient descent. IEEE transactions on neural networks and learning systems, 29(5):1454–1466, 2017

    2017

  22. [30]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ra- manan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014

    2014

  23. [31]

    Sgdr: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016

    2016 arXiv

  24. [32]

    Rectifier nonlinearities improve neural network acoustic models

    Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml, volume 30, page 3, 2013

    2013

  25. [33]

    When does label smoothing help? In Advances in Neural Information Processing Systems, pages 4696–4705, 2019

    Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems, pages 4696–4705, 2019

    2019

  26. [34]

    Rectified linear units improve restricted boltzmann machines

    Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pages 807–814, 2010

    2010

  27. [35]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, ...

    2019

  28. [36]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1 (8):9, 2019

    2019

  29. [37]

    Searching for activation functions

    Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017

    2017 arXiv

  30. [38]

    Darknet: Open source neural networks in c

    Joseph Redmon. Darknet: Open source neural networks in c. http://pjreddie. com/darknet/, 2013–2016

    2013

  31. [39]

    Yolov3: An incremental improvement

    Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018. 14 MISRA: MISH ACTIV A TION FUNCTION

    2018 arXiv

  32. [40]

    Dynamic routing between cap- sules

    Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between cap- sules. In Advances in neural information processing systems, pages 3856–3866, 2017

    2017

  33. [41]

    Dropout: a simple way to prevent neural networks from overfitting

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958, 2014

    1929

  34. [42]

    Going deeper with convolutions

    Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pat- tern recognition, pages 1–9, 2015

    2015

  35. [43]

    Efficientnet: Rethinking model scaling for convolu- tional neural networks

    Mingxing Tan and Quoc V Le. Efficientnet: Rethinking model scaling for convolu- tional neural networks. arXiv preprint arXiv:1905.11946, 2019

    1905 arXiv

  36. [44]

    Efficientdet: Scalable and efficient object detection

    Mingxing Tan, Ruoming Pang, and Quoc V Le. Efficientdet: Scalable and efficient object detection. arXiv preprint arXiv:1911.09070, 2019

    1911 arXiv

  37. [45]

    Cspnet: A new backbone that can enhance learning capability of cnn

    Chien-Yao Wang, Hong-Yuan Mark Liao, I-Hau Yeh, Yueh-Hua Wu, Ping-Yang Chen, and Jun-Wei Hsieh. Cspnet: A new backbone that can enhance learning capability of cnn. arXiv preprint arXiv:1911.11929, 2019

    1911 arXiv

  38. [46]

    Pelee: A real-time object detection system on mobile devices

    Robert J Wang, Xiang Li, and Charles X Ling. Pelee: A real-time object detection system on mobile devices. In Advances in Neural Information Processing Systems , pages 1963–1972, 2018

    1963

  39. [47]

    Aggregated residual transformations for deep neural networks

    Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE confer- ence on computer vision and pattern recognition, pages 1492–1500, 2017

    2017

  40. [48]

    Empirical evaluation of rectified activations in convolutional network

    Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853, 2015

    2015 arXiv

  41. [49]

    Cutmix: Regularization strategy to train strong classifiers with lo- calizable features

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with lo- calizable features. In Proceedings of the IEEE International Conference on Computer Vision, pages 6023–6032, 2019

    2019

  42. [50]

    Wide residual networks

    Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016

    2016 arXiv

  43. [51]

    Shufflenet: An extremely efficient convolutional neural network for mobile devices

    Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6848–6856, 2018

    2018

  44. [52]

    Neural architecture search with reinforcement learning

    Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578, 2016

    2016 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.