{"id":"f9253cf6-23fd-44d7-8f63-28df6f5c00f5","arxiv_id":"1908.08681","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Mish, a smooth non-monotonic activation function, is proposed and shown to often match or beat ReLU, Swish, and Leaky ReLU on CIFAR-10, ImageNet, and MS-COCO benchmarks.","lead":"This paper proposes Mish, a new activation function defined as x times tanh(softplus(x)), and reports accuracy gains on several image classification and object detection benchmarks when swapped in for ReLU, Swish, or Leaky ReLU. A generalist might read it because activation functions are a single drop-in change that affects many neural network architectures at once.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline gain is not yet a reliable activation effect: the 2.1% YOLOv4 figure is from the tiny variant, the main variant's other metrics move opposite, and the only repeated-run study cannot resolve Mish from Swish or GELU.","rationale":"The paper's central claim is empirical and would hold if swapping in Mish reliably matched or beat ReLU, Swish, and Leaky ReLU across vision benchmarks. The paper provides a broad benchmark sweep and public code, which is real evidence, and the directional advantage over ReLU and Leaky ReLU appears in nearly every table. However, the strongest advertised number, the 2.1% AP50 gain, is selected from the smallest YOLOv4 variant; the main variant gains only 0.9% and shows AP75 and APL decreasing. Because these are single runs, the differences are within plausible run-to-run noise. The only designed repeated-run experiment (Table 1) shows Mish ahead of Swish by 0.16 points and GELU by 0.11, with standard errors on the order of 0.12-0.13, so those closest-baseline comparisons are not statistically supported. This does not make the claim false; the function is plausible and the ResNet-50 ImageNet result is encouraging. It means 'consistent, reliable gain' is overclaimed relative to the controlled evidence, which is why I keep the reader's CONDITIONAL verdict rather than moving to ACCEPT or REJECT.","tokens_in":11435,"tokens_out":11813,"duration_ms":108075,"concrete_test":"Re-run the Table 5 YOLOv4pacsp and YOLOv4pacsp-s comparisons with at least 5 random seeds per activation, using the exact training script and fixed hyperparameters, and report mean plus/minus standard deviation for AP50 and AP75. If the 95% confidence interval for the Mish-minus-LeakyReLU AP50 difference on the main YOLOv4pacsp model includes zero, or if the AP75 deficit reproduces, the headline claim of a reliable 2.1% gain is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The practical claim that swapping in Mish gives a small, reliable accuracy gain is not supported by the paper's own controlled evidence. In Table 5, the abstract's 2.1% AP50 improvement comes from YOLOv4pacsp-s, the smallest variant; for the main YOLOv4pacsp model the AP50 gain is only 0.9%, while AP75 drops from 51.0% to 50.2% and APL drops from 59.5% to 59.4%. These are single runs, so the mixed signs on other metrics are consistent with seed or configuration noise rather than a clean activation effect. The one repeated-run experiment (Table 1, 23 runs, SqueezeNet on CIFAR-10) has Mish above Swish by only 0.16 percentage points and above GELU by 0.11, with per-run standard deviations around 0.40-0.47; the standard error of the Mish-Swish difference is roughly 0.12, so the difference is not convincingly nonzero, and no multiple-comparison correction is reported. Thus the 'consistently outperforms Swish/GELU' wording is stronger than the evidence, even though Mish does beat ReLU by about 0.8 points in that repeated-run study.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new activation function, Mish, defined as f(x)=x tanh(softplus(x)), and argues that it is a smooth, non-monotonic, 'self-regularized' activation that can replace ReLU, Leaky ReLU, or Swish in deep networks. The manuscript derives the first derivative, offers a speculative interpretation of the derivative's Delta(x) term as a preconditioner, and reports benchmark comparisons on CIFAR-10, ImageNet-1k, and MS-COCO across many architectures, plus a runtime comparison with a CUDA implementation. The central empirical claim is that Mish consistently matches or exceeds ReLU, Leaky ReLU, and Swish while keeping all other network parameters and hyperparameters constant.","tokens_in":11722,"tokens_out":5031,"duration_ms":47192,"significance":"If the empirical claim is robust, Mish would be a useful drop-in activation function: it is simple to define, publicly implemented, and shows a broad pattern of small accuracy gains across many architectures. The paper's strengths are its breadth of architectures, the public code, the exact derivative identity in Eq. (1), and the presence of a 23-run statistical study for one small CIFAR-10 setup. However, the significance is conditional: the headline gains rest largely on single runs with no error bars, and the one repeated-run experiment does not separate Mish from Swish or GELU. With additional uncertainty quantification or with claims rescaled to what the evidence supports, this could be a useful empirical reference point; as it stands, the 'consistently outperforms' claim is stronger than the evidence.","major_comments":[{"comment":"The abstract's headline 2.1% AP50 improvement is taken from YOLOv4pacsp-s, the smallest variant. On the main YOLOv4pacsp model, AP50 improves by only 0.9% while AP75 decreases from 51.0% to 50.2% and APL is essentially flat (59.5% to 59.4%); these are single runs with no error bars. The mixed directions across metrics are exactly what one would expect from seed or configuration noise, so this table does not by itself establish a reliable activation effect.","section":"§4.4, Table 5"},{"comment":"The statement that Mish 'consistently outperforms' ReLU, Leaky ReLU, and Swish is contradicted by the PeleeNet row, where Swish achieves 71.5% Top-1 and 90.7% Top-5 against Mish's 71.4% and 90.4%. The same table also reports a 13.4-point Top-1 drop for Swish on CSP-ResNext-50 without augmentation, which is far larger than any reported activation-function effect in the literature and suggests an uncontrolled difference in training or configuration; citing it as evidence that Swish 'cannot be used in every architecture' is not supported by the controlled-comparison protocol described in the paper.","section":"§4.3, Table 3"},{"comment":"The only repeated-run experiment (23 runs, SqueezeNet on CIFAR-10) does not resolve Mish from its closest competitors. Mish's mean accuracy is 0.16 points above Swish and 0.11 points above GELU, while the per-run standard deviations are 0.40-0.47; no significance test or multiple-comparison correction is reported, so the differences are within plausible noise. The comparison with ReLU (0.82 points) is more favorable but still should be accompanied by a significance statement.","section":"§4.1, Table 1"}],"minor_comments":[{"comment":"The sentence listing hyperparameters says epochs, initial learning rate, weight decay, and momentum were set at '500500, 0.01, 5e-4, and 0.9'; '500500' appears to be a typo and should be clarified.","section":"§4.4, text"},{"comment":"The ReLU and Mish results are identical for CSP-DarkNet-53 with and without the PANet+SPP module (64.5% and 64.9% in both rows); this makes the comparison uninformative and needs an explanation or correction.","section":"Table 4"},{"comment":"The claim that the Delta(x) term 'acts like a preconditioner' is explicitly speculative and should be labeled as a hypothesis rather than presented as an explanation of the empirical results.","section":"§2, §4.5"},{"comment":"All benchmark tables lack information on the number of runs, seeds, and variance; adding this information would materially improve the reader's ability to judge the reported 1-3% improvements.","section":"§4.2-§4.4"},{"comment":"The 'self-regularized' property is never defined formally; the paper would benefit from a precise statement of what quantity is regularized and how it is measured.","section":"§3"}],"recommendation":"major_revision","confidential_remarks":"The paper is essentially an empirical evaluation of a new activation function. Its breadth is appealing, but the central 'consistently outperforms' claim currently rests on single-run large-scale results and a small repeated-run study that cannot resolve Mish from Swish or GELU. I would not reject the manuscript because the proposed function is simple, publicly available, and plausibly useful; however, the authors need either to add repeated-run evidence with significance testing on at least the main comparisons or to substantially soften the claims in the abstract and conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Mish is a real contribution: the formula x·tanh(softplus(x)) is not in the cited prior art, the derivative algebra in Eq. (1) is exact, and the empirical scope is broad — a dozen CIFAR-10 architectures, ImageNet backbones, COCO detection, plus a public Mish-CUDA implementation with timing benchmarks. That breadth, plus the code, makes it a useful paper for practitioners who want a drop-in activation to try. It also explains why Mish has been widely adopted in real detectors.\n\nThe soft spots are real and the stress-test note is right. The headline 2.1% AP50 improvement over Leaky ReLU on YOLOv4 comes from the tiny variant; for the main YOLOv4pacsp model the AP50 gain is 0.9% while AP75 drops from 51.0% to 50.2%. These are single runs, so the mixed signs are consistent with seed or configuration noise. The repeated-run CIFAR-10 study (23 runs, SqueezeNet) has Mish only 0.16 points above Swish and 0.11 above GELU, with per-run standard deviations around 0.4 — the standard error of the difference is roughly 0.12, so the Mish–Swish gap is not convincingly nonzero, and no multiple-comparison correction is reported. 'Consistently outperforms' is stronger than the evidence.\n\nThe preconditioner discussion in Section 2 is speculative, but it is labeled as speculation and is not used to derive the empirical results, so I do not treat it as a flaw. The main methodological weakness is the absence of error bars on the large-benchmark numbers, which makes it impossible to know whether the reported gains reflect the activation function or uncontrolled variation.\n\nWho is this for? Researchers and engineers working on CNNs who want a quick, plausible activation candidate, and anyone studying activation function design. It deserves a serious referee: the idea is simple, the evaluation is extensive, and the weaknesses are addressable with multiple seeds and tempered claims. I would accept it for peer review with a request for that revision.\n\nI'd bring it to reading group, mainly to discuss how little statistical evidence an activation paper needs to become widely adopted.","headline":"A genuinely new activation function with broad empirical coverage, but the paper's own statistics do not support the claimed consistent edge over Swish.","tokens_in":12199,"tokens_out":2407,"would_cite":true,"duration_ms":25194,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes Mish, an activation function defined as f(x)=x tanh(softplus(x)), and reports that it consistently matches or exceeds ReLU, Swish, and Leaky ReLU across computer vision benchmarks.","keywords":["activation function","Mish","self-regularization","non-monotonic","deep learning","image classification","object detection","Swish"],"falsifier":"Train the same architectures (e.g., ResNet-50 on ImageNet-1k and YOLOv4 on MS-COCO) with ReLU, Swish, Leaky ReLU, and Mish across many seeds, keeping all augmentation and hyperparameter settings fixed, and check whether the mean gaps persist above the seed variance. If the 1-2% differences shrink below one standard deviation, or change sign across architectures, the central empirical claim is not settled.","tokens_in":11250,"feed_emoji":"🧠","tokens_out":8418,"duration_ms":74905,"temperature":0.7,"pith_summary":"The paper proposes a new neural-network activation function, Mish, defined by $f(x)=x\\tanh(\\mathrm{softplus}(x))$, and claims that swapping it into standard convolutional networks is a reliable way to improve accuracy without touching other settings. On ImageNet-1k classification, Mish is reported to beat ReLU on ResNet-50 by about 1% Top-1 accuracy; on MS-COCO object detection, it beats Leaky ReLU in YOLOv4 by up to 2.1% in $AP_{50}^{val}$. The author argues that Mish's smooth, non-monotonic, bounded-below shape preserves small negative weights and that the $\\Delta(x)=\\mathrm{sech}^2(\\mathrm{softplus}(x))$ factor in its derivative acts like a preconditioner, smoothing the loss landscape and regularizing training. A reader should care because if these results hold, a working model can often be improved by a one-line activation change, with no retuning.","feed_headline":"Mish out-scores ReLU on ImageNet and Leaky ReLU on COCO","feed_subtitle":"The formula f(x)=x tanh(softplus(x)) raises accuracy when swapped into standard networks.","key_machinery":"The central object is the activation function itself, $f(x)=x\\tanh(\\mathrm{softplus}(x))$, which combines identity, softplus, and tanh into a smooth non-monotonic function that is bounded below and unbounded above. Its load-bearing identity is the derivative decomposition $f'(x)=\\Delta(x)\\,\\mathrm{swish}(x)+f(x)/x$, where $\\Delta(x)=\\mathrm{sech}^2(\\mathrm{softplus}(x))$; the paper treats $\\Delta(x)$ as a preconditioner that smooths the optimization landscape and thereby regularizes training. Mish also uses the self-gating property from Swish, multiplying the unmodulated input by the output of a nonlinear function of the input, and its smoothness and small negative-weight retention are credited with better gradient flow and avoidance of Dying ReLU. An optimized CUDA implementation, Mish-CUDA, is used to mitigate the extra computational cost of the formula.","core_discovery":"The central claim is that Mish is a self-regularized non-monotonic activation function that matches or improves on ReLU, Swish, and Leaky ReLU in the vision settings tested. The paper defines Mish as $f(x)=x\\tanh(\\ln(1+e^x))$, notes its range is approximately $[-0.31,\\infty)$, and derives $f'(x)=\\Delta(x)\\,\\mathrm{swish}(x)+f(x)/x$ with $\\Delta(x)=\\mathrm{sech}^2(\\mathrm{softplus}(x))$. This decomposition is used to argue that the $\\Delta$ factor acts as a preconditioner that makes gradients and loss contours smoother, which the author identifies as the likely mechanism behind Mish's gains. Evidence includes a 23-run CIFAR-10 study, architecture sweeps on CIFAR-10, ImageNet-1k results on ResNet and CSP-family models, and MS-COCO object detection with YOLOv4, including combinations with CutMix, Mosaic, self-adversarial training, DropBlock, and label smoothing.","pith_inferences":["Inference: if the $\\Delta(x)$ preconditioner story is right, then activation design could focus on shaping the derivative's smoothing factor rather than hunting for function shapes; one test would be to vary the width or exponent of the sech-squared factor and measure loss-landscape curvature.","The paper leaves non-vision tasks untested; a natural extension is to measure Mish in transformers or recurrent models, where smooth bounded-below activations may behave differently.","Because the largest gains appear in hard, deep detection models, a testable extension is to check whether Mish's advantage grows with network depth or with smaller objects, using the per-size AP columns in Table 5.","The results were obtained with fixed hyperparameters; a hyperparameter search with Mish as the target, which the author explicitly suggests, could either widen the gains or reveal that some of the baseline comparisons depend on tuning."],"forward_implications":["Practitioners can treat Mish as a drop-in replacement in standard CNN backbones and expect small accuracy gains without retuning hyperparameters.","Mish is more dependable than Swish in large or deep models: the paper reports Swish dropping ImageNet Top-1 by 13.4% on CSP-ResNext-50 while Mish improves it by 1%.","Mish is compatible with modern augmentation and regularization schemes such as CutMix, Mosaic, self-adversarial training, DropBlock, and label smoothing, with consistent improvements.","The optimized Mish-CUDA implementation brings forward and backward pass times close to native ReLU levels, making the accuracy gains available at modest computational cost.","Because Mish consistently matched or exceeded baselines across ResNet, Wide ResNet, DenseNet, MobileNet, ShuffleNet, Inception, EfficientNet, and detection backbones, the reported benefit is not tied to one architecture family."],"supporting_citations":[{"why":"Proposes Swish, the self-gating baseline whose derivative decomposition Mish extends and against which Mish is compared.","marker":"[37]"},{"why":"YOLOv4 and CSP-DarkNet-53 object-detection setup where Mish beats Leaky ReLU by up to 2.1% AP50val.","marker":"[2]"},{"why":"Introduces CSPNet backbones (CSP-ResNet, CSP-DarkNet, CSP-ResNext) used for the ImageNet and COCO comparisons.","marker":"[45]"},{"why":"Defines ResNet architectures used for CIFAR-10, depth, and ImageNet-1k benchmark comparisons.","marker":"[15]"},{"why":"Provides the ImageNet-1k classification benchmark where Mish reports about 1% Top-1 gain over ReLU.","marker":"[7]"},{"why":"Provides the MS-COCO object-detection benchmark used for the YOLOv4 results.","marker":"[30]"},{"why":"Supplies the loss-landscape visualization method used to argue Mish smooths optimization and generalizes better.","marker":"[28]"},{"why":"Defines Leaky ReLU, the main baseline for the YOLOv4 object-detection comparisons.","marker":"[32]"},{"why":"Defines ReLU, the baseline Mish is compared against across classification benchmarks.","marker":"[25, 34]"}],"fun_headline_variants":["Mish beats ReLU on ImageNet, Leaky ReLU on COCO","Mish gains ~1% ImageNet top-1, 2.1% COCO AP","Mish: self-regularized, out-scores ReLU on key benchmarks","Swap ReLU for Mish: ImageNet and COCO accuracy gains","Non-monotonic Mish wins over ReLU and Leaky ReLU"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported accuracy differences, roughly 0.4% to 2.1%, are caused by the activation function rather than by run-to-run randomness or unlisted training details, since most large-scale results are single runs without error bars.","fun_headline_variants_meta":{"raw":{"variants":["Mish beats ReLU on ImageNet, Leaky ReLU on COCO","Mish gains ~1% ImageNet top-1, 2.1% COCO AP","Mish: self-regularized, out-scores ReLU on key benchmarks","Swap ReLU for Mish: ImageNet and COCO accuracy gains","Non-monotonic Mish wins over ReLU and Leaky ReLU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001266,"raw_usage":{"total_tokens":5205,"prompt_tokens":993,"completion_tokens":4212,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":4107}},"tokens_in":609,"tokens_out":4212,"duration_ms":31185,"temperature":1.0,"reasoning_tokens":4107,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:31:22.283834+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same architectures (e.g., ResNet-50 on ImageNet-1k and YOLOv4 on MS-COCO) with ReLU, Swish, Leaky ReLU, and Mish across many seeds, keeping all augmentation and hyperparameter settings fixed, and check whether the mean gaps persist above the seed variance. If the 1-2% differences shrink below one standard deviation, or change sign across architectures, the central empirical claim is not settled.","supporting_citations":[{"cited_title":"Microsoft coco: Common objects in context","cited_arxiv_id":null,"evidence_quote":"Provides the MS-COCO object-detection benchmark used for the YOLOv4 results."},{"cited_title":"Rectifier nonlinearities improve neural network acoustic models","cited_arxiv_id":null,"evidence_quote":"Defines Leaky ReLU, the main baseline for the YOLOv4 object-detection comparisons."}],"review_version":1}