{"id":"e060a404-1c86-4956-8a18-30f172a071e2","arxiv_id":"1908.03682","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"NLReLU, f(x) = ln(β max(0,x) + 1), gives 0.16% to 2.04% higher CNN accuracy than ReLU on MNIST and CIFAR-10, with per-network tuned β.","lead":"This paper proposes a new activation function for neural networks, NLReLU, which applies a logarithm to the positive part of ReLU. On MNIST and CIFAR-10, networks using NLReLU beat ReLU by small margins and matched other modern activations.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Deep ResNet comparison is confounded: NLReLU is evaluated only in placement (0,1,0) while the ReLU baseline uses the standard (1,1,1) placement, so the reported 1.35% average gain may reflect removed activation layers rather than the NLReLU function itself.","rationale":"The reader's CONDITIONAL verdict is appropriate. The reader's weakest assumption flagged that the placement choice might overfit ResNet-50 or the CIFAR-10 validation distribution. My analysis sharpens this: even without any selection bias, the Table 4 comparison is architecturally confounded because NLReLU is used in a sparser activation layout than the ReLU baseline. Table 3 demonstrates that this layout change alone moves accuracy by about 1.6% for NLReLU, which is larger than the reported 1.35% deep-network advantage. Thus the headline deep claim is not established as an effect of the activation function. This does not force a verdict change from CONDITIONAL, but it strengthens the condition: the authors must supply matched-placements experiments (ReLU in (0,1,0) and/or NLReLU in (1,1,1)) before the deep claim can be accepted. The shallow results are less affected because no placement search is reported there, though the per-network beta tuning (Tables 1, 2) and the lack of released code remain secondary concerns.","tokens_in":15429,"tokens_out":4744,"duration_ms":46657,"concrete_test":"Reproduce Table 4 with ReLU in the exact (0,1,0) placement (activation only at position B, no activation at A and C) under the same training protocol, and compare against NLReLU-(0,1,0). If ReLU-(0,1,0) attains accuracy statistically indistinguishable from NLReLU-(0,1,0), the deep-network gain is an architecture artifact. As a second check, run the matched standard placement NLReLU-(1,1,1) versus ReLU-(1,1,1); if NLReLU-(1,1,1) does not beat ReLU-(1,1,1), the paper's claim should be revised to a placement effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing issue is not merely that placement (0,1,0) was selected after inspecting ResNet-50 accuracy (Table 3); it is that Table 4 compares NLReLU under this modified architecture against ReLU under the standard full pre-activation architecture. Section IV-B defines three activation positions: A (before each residual unit's input), B (inside each residual unit), and C (before the network output). Table 3 shows that for NLReLU, keeping only position B (0,1,0) outperforms keeping all three (1,1,1) by 82.34% vs 80.74%. Consequently, the NLReLU entries in Table 4 use a network with nonlinearities removed from positions A and C, while the ReLU baseline presumably retains ReLU at all positions. The observed 1.35% average advantage may therefore be an architecture effect (fewer activation layers, altered gradient flow) rather than an effect of the NLReLU formula. The paper never reports a matched ReLU baseline with the same (0,1,0) placement, nor a matched NLReLU (1,1,1) baseline. This confound undermines the deep-network headline claim even if the placement was chosen a priori. The shallow VGG/AlexNet claims do not involve this placement issue, but they still rely on per-network beta tuning.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a novel activation function, NLReLU, defined as f(x)=β·ln(max(0,x)+1), which modifies the positive part of ReLU with a parametric logarithmic transform while preserving the zero response for negative inputs. The authors claim that NLReLU retains sparse activation, reduces bias shift and heteroscedasticity, alleviates dying ReLU and vanishing gradient problems, and improves classification accuracy over ReLU on MNIST and CIFAR-10 across shallow CNNs (SimpleCNN, LeNet-5, AlexNet, VGG-11/16/19) and deep ResNets (50/101/152/200). The empirical claims are supported by means and standard deviations over 10 runs, with per-network tuning of the β parameter and, for ResNets, a placement search over three activation positions.","tokens_in":15727,"tokens_out":5750,"duration_ms":52282,"significance":"If the reported accuracy gains hold under matched conditions, NLReLU would be a simple, sparsity-preserving activation function that offers small but reproducible gains over ReLU and parity with ELU/SELU on standard benchmarks. The paper's use of 10-run mean±std and comparisons across ten network depths is a strength. However, the central claims are weakened by per-network β tuning and, more importantly, by the unmatched activation-placement configuration in the deep ResNet experiments, which prevents attributing the observed gains to the activation function itself. The mechanism claims rest on a small simulation and an existing ELU theorem, with no formal derivation for NLReLU.","major_comments":[{"comment":"The deep-network comparison is not matched on architecture. The NLReLU entries in Table 4 use placement (0,1,0), i.e., NLReLU is retained only at position B within each residual unit and removed from positions A and C, while the ReLU baseline appears to use the standard full pre-activation placement with the activation at all three positions. The paper does not report a ReLU baseline with the (0,1,0) placement, nor an NLReLU baseline with the (1,1,1) placement. Consequently, the reported 1.35% average improvement of NLReLU over ReLU may reflect the architecture change (fewer nonlinearities, altered gradient flow) rather than the NLReLU formula. Please provide matched comparisons for at least one network, e.g., ReLU-(0,1,0) vs NLReLU-(0,1,0) and NLReLU-(1,1,1) vs ReLU-(1,1,1).","section":"Section IV-B, Tables 3 and 4"},{"comment":"The reported average gains (0.16%, 2.04%, 1.35%) are obtained with a different β value for each network and dataset, selected via a sensitivity analysis exemplified for AlexNet in Fig. 4. No automatic or principled rule for choosing β is provided, and the sensitivity analysis itself shows that accuracy varies with β. This means the headline numbers assume a practitioner will replicate the same oracle-like tuning. Please report results for a fixed β (e.g., β=1.0) across all networks, or provide a learning curve for β with a recommendation that does not depend on test-set performance.","section":"Section IV-A, Tables 1, 2, 4 and Fig. 4"},{"comment":"The claims that NLReLU 'reduces heteroscedasticity in the data distribution among layers and the bias shift effect' are supported by a single simulation (Fig. 2) and by reference to Theorem 2 of [12] for the bias-shift relationship. The simulation description is incomplete: it reports mean and standard deviation of the number of activated neurons but does not specify the training procedure, the network input distribution, or whether the statistics are from a forward pass or after training. Without a reproducible setup and a formal argument connecting NLReLU's derivative to the bias-shift theorem, these mechanism claims are not established. Please either provide the simulation code and a more precise setup, or soften the claims to 'we observe in simulation'.","section":"Section III, Fig. 2"}],"minor_comments":[{"comment":"The abstract contains an incomplete sentence: 'This activation function uses the parametric natural logarithmic transform to improve ReLU and is simply defined as. NLReLU' — the equation is missing or not rendered.","section":"Abstract"},{"comment":"In the experiment setup, 'CIRAR-10' should be 'CIFAR-10'.","section":"Section IV-A"},{"comment":"The text says 'ReLU does not restain the gradient'; 'restain' should be 'restrain'.","section":"Section II-B"},{"comment":"The sentence 'In Section III, we find that the aforementioned activation functions' appears to reference the wrong section; it should likely reference Section II, where those functions are reviewed.","section":"Section III"},{"comment":"The paper does not provide the source of the TensorFlow implementations or a link to code; making the code and hyperparameter configuration public would substantially improve reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The deep-network claim is the weakest point; if the authors can provide a matched ReLU-(0,1,0) baseline and show the gain persists, the paper would be much stronger. The β tuning issue also warrants scrutiny. The novelty of NLReLU is modest (a logarithmic transform of ReLU has appeared in the authors' prior work [18]), but the empirical study is reasonably broad. I do not see evidence of circularity; the accuracy numbers are direct measurements. The absence of code and the incomplete simulation description are practical obstacles to verification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Fairly straightforward read. The function f(x)=ln(β max(0,x)+1) is new as far as the citations show—it doesn't appear in the earlier activation surveys—and it has a nice property: it keeps the sparse zero for negative inputs while compressing large positives with a logarithmic scale. The definition, derivative, and plots for β in [0.7,1.1] are clear.\n\nWhat the paper does well is the systematic empirical sweep: ten networks, two datasets, ten runs each, means and standard deviations. On the shallow CNNs, NLReLU beats ReLU by about 0.16% on MNIST and 2.04% on CIFAR-10, and lands roughly at parity with SELU. Those tables are credible; I don't doubt that for these settings NLReLU is a serviceable drop-in replacement.\n\nThe weak spot is the deep ResNet section. Table 3 shows that for ResNet-50, using NLReLU only at position B (0,1,0) gives 82.34% vs 80.74% for (1,1,1). That's fine as an internal check. But Table 4 compares NLReLU with (0,1,0) against ReLU with the standard (1,1,1) placement. That is not apples-to-apples: the NLReLU networks have nonlinearities removed at positions A and C, which changes the architecture and gradient flow independently of the activation function. The paper never reports a ReLU baseline with (0,1,0), nor an NLReLU network with (1,1,1) across all depths. So the headline '1.35% higher than ReLU on deep ResNets' is not supported as an activation-function effect; it could be an architecture effect. That's a real flaw, not a nitpick.\n\nTwo smaller issues. First, β is tuned per network and dataset using 10-fold cross-validation, while comparison activations use default or fixed parameters. That's a mild bias in NLReLU's favor, though the sensitivity analysis at least shows the results don't swing wildly with β. Second, the mechanism story—reducing heteroscedasticity, bias shift, dying ReLU—rests on a self-made simulation and a citation to the ELU theorem, not on a formal analysis of NLReLU dynamics. It's plausible but thin.\n\nBottom line: the shallow-network contribution is a useful empirical addition to the activation zoo, and the paper deserves a serious referee. But the deep-ResNet claim needs a matched baseline before it becomes believable.","headline":"Simple new activation with credible shallow-net gains; the deep ResNet claim is confounded by a placement mismatch.","tokens_in":16275,"tokens_out":4888,"would_cite":false,"duration_ms":48837,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes NLReLU, an activation function that replaces ReLU's positive branch with a scaled natural logarithm and reports higher classification accuracy across ten convolutional networks.","keywords":["activation function","rectified linear unit","natural logarithm","convolutional neural network","dying ReLU","vanishing gradient","sparse activation","bias shift"],"falsifier":"Run the same ResNet and shallow-CNN comparisons with NLReLU and ReLU using identical seeds, the same per-network $\\beta$, and no placement search; if the average accuracy differences fall inside one standard deviation of the runs, the reported gains are tuning artifacts rather than properties of the log transform.","tokens_in":3311,"feed_emoji":"📈","tokens_out":1960,"duration_ms":84351,"temperature":0.7,"pith_summary":"This paper proposes NLReLU, an activation function that keeps ReLU's zero output for negative inputs but replaces the positive branch with the scaled natural logarithm $f(x)=\\beta\\ln(\\max(0,x)+1)$. The claim is that this simple substitution preserves ReLU's sparse activation while easing two failure modes: neurons that die from large gradients and gradients that vanish in deep stacks. The mechanism offered is that the derivative $\\beta/(x+1)$ shrinks large gradients, pushes per-layer mean activations toward zero, and reduces the uneven spread of activations across layers. Across ten convolutional networks, the paper reports average accuracy gains over ReLU of 0.16% on MNIST, 2.04% on CIFAR-10 shallow nets, and 1.35% on deep ResNets, with performance comparable to SELU. If these results transfer, NLReLU is a low-cost, parameterized activation that can replace ReLU in existing CNN code with one added scalar.","feed_headline":"A log twist on ReLU lifts CNN accuracy by up to 2 percent","feed_subtitle":"Replacing ReLU's positive branch with a scaled natural log is reported to beat ReLU on ten CNNs and match SELU.","key_machinery":"The central object is the NLReLU transfer function with its scalar parameter $\\beta$. Its work is done by the positive-branch derivative $\\beta/(x+1)$: unlike ReLU's constant derivative of 1, this derivative falls as the input grows, so it compresses large gradients before they can kill a neuron or destabilize training, while staying far enough from zero (for $\\beta$ near 1) to avoid the saturation that plagues sigmoids and tanh. The logarithmic transform also compresses large activations, which the paper argues reduces heteroscedasticity across layers and, by shifting mean activations closer to zero, lessens the bias shift effect that slows learning. In deep pre-activation residual networks, the paper identifies a placement rule: NLReLU should be kept at the residual unit's inner activation but removed from the block-input and final-output positions, and this placement choice is part of what produces the reported deep-network gains.","core_discovery":"The paper's central discovery is that a parametric logarithmic transform applied only to the positive branch of ReLU improves network accuracy and training stability without giving up sparsity. Concretely, NLReLU is $f(x)=\\beta\\ln(\\max(0,x)+1)$ with derivative $\\beta/(\\max(0,x)+1)$ for $x>0$ and $0$ otherwise. The decreasing derivative makes the function right-soft saturated, so gradients are minified for large activations; the log scale pulls mean activations toward zero; and the zero branch keeps the representation sparse. The authors report that this combination yields higher test accuracy than ReLU on all ten tested CNN configurations and matches SELU, and that NLReLU networks still converge at learning rates where ReLU networks diverge.","pith_inferences":["Because $\\beta$ is tuned per network and dataset in the experiments, the headline averages are conditional on that tuning; a natural test is whether a fixed $\\beta$ (say 1.0) preserves the reported gains, which the paper does not report.","The mean-shift and variance-reduction mechanism suggests NLReLU could help most in networks trained without batch normalization; the deep experiments all keep BN, so a direct test would be training BN-free deep networks with NLReLU.","The logarithmic compression of large activations implies a regularizing effect on noisy inputs, which is not explored in the paper but is directly measurable on corrupted-input benchmarks."],"forward_implications":["If NLReLU's reported gains hold, swapping ReLU for NLReLU in existing shallow CNNs should give about 0.16% higher accuracy on MNIST and about 2.04% higher on CIFAR-10, with no architectural change beyond the activation.","In deep pre-activation ResNets, the same swap is reported to give 1.35% higher CIFAR-10 accuracy, provided NLReLU is placed inside each residual unit and not at the block input or output.","Because NLReLU's derivative minifies large gradients, networks using it can be expected to tolerate larger learning rates than ReLU before diverging, as demonstrated in the paper's MNIST control experiments.","NLReLU is reported to be comparable to SELU on the tested benchmarks, but without SELU's self-normalizing constraints, making it a simpler alternative that needs only the scalar $\\beta$ per network."],"supporting_citations":[{"why":"Supplies the ReLU baseline that NLReLU modifies and is compared against.","marker":"[1]"},{"why":"Provides the analysis of sparse activation that NLReLU claims to retain and the ReLU sparsity rationale.","marker":"[2]"},{"why":"Supplies the theorem that pushing mean activations toward zero reduces bias shift, which is the paper's mechanism for faster learning.","marker":"[12]"},{"why":"Defines the pre-activation ResNet architecture and the activation positions used in the deep-network experiments.","marker":"[5]"},{"why":"Defines SELU, the strongest comparison activation that NLReLU is reported to match.","marker":"[8]"},{"why":"Provides the MNIST dataset used for the shallow accuracy comparisons.","marker":"[23]"},{"why":"Provides the CIFAR-10 dataset used for both shallow and deep comparisons.","marker":"[24]"},{"why":"Supplies the MSRA initialization used when training the deep ResNet variants.","marker":"[11]"}],"fun_headline_variants":["Log-scaled ReLU beats ReLU on all ten CNN tests","Natural log tweak to ReLU lifts accuracy by up to 2%","NLReLU: ReLU's log branch sharpens learning and accuracy","ReLU with a natural log: less dying, more learning"],"cache_read_input_tokens":18432,"weakest_assumption_plain":"The deep-network result depends on the paper's choice to place NLReLU only inside each residual unit, a configuration selected by comparing all eight placements on the same CIFAR-10 validation set; if that selection overfits the dataset, the reported 1.35% advantage is partly a placement artifact.","fun_headline_variants_meta":{"raw":{"variants":["Log-scaled ReLU beats ReLU on all ten CNN tests","Natural log tweak to ReLU lifts accuracy by up to 2%","NLReLU: ReLU's log branch sharpens learning and accuracy","ReLU with a natural log: less dying, more learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000337,"raw_usage":{"total_tokens":1879,"prompt_tokens":973,"completion_tokens":906,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":828}},"tokens_in":589,"tokens_out":906,"duration_ms":9314,"temperature":1.0,"reasoning_tokens":828,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:05:40.953600+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same ResNet and shallow-CNN comparisons with NLReLU and ReLU using identical seeds, the same per-network $\\beta$, and no placement search; if the average accuracy differences fall inside one standard deviation of the runs, the reported gains are tuning artifacts rather than properties of the log transform.","supporting_citations":[{"cited_title":"Rectified linear units improve restricted boltzmann machines ,","cited_arxiv_id":null,"evidence_quote":"Supplies the ReLU baseline that NLReLU modifies and is compared against."},{"cited_title":"Deep sparse re ctifier neural networks ,","cited_arxiv_id":null,"evidence_quote":"Provides the analysis of sparse activation that NLReLU claims to retain and the ReLU sparsity rationale."},{"cited_title":"Identity mappings in deep residua l networks ,","cited_arxiv_id":null,"evidence_quote":"Defines the pre-activation ResNet architecture and the activation positions used in the deep-network experiments."},{"cited_title":"Self - normalizing neural networks ,","cited_arxiv_id":null,"evidence_quote":"Defines SELU, the strongest comparison activation that NLReLU is reported to match."},{"cited_title":"Gradient - based learning applied to document recognition ,","cited_arxiv_id":null,"evidence_quote":"Provides the MNIST dataset used for the shallow accuracy comparisons."},{"cited_title":"Learning multiple layers of fea tures from tiny images ,","cited_arxiv_id":null,"evidence_quote":"Provides the CIFAR-10 dataset used for both shallow and deep comparisons."},{"cited_title":"Delving deep into rectifiers: Surpassing human - level performance on imagenet classification ,","cited_arxiv_id":null,"evidence_quote":"Supplies the MSRA initialization used when training the deep ResNet variants."}],"review_version":1}