Natural-Logarithm-Rectified Activation Function in Convolutional Neural Networks

Chao Gao; Jianpeng Zhang; Jinghua Qu; Lixin Ji; Yang Liu

REVIEW 3 major objections 5 minor 24 references

Natural-Logarithm-Rectified Activation Function in Convolutional Neural Networks

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proposes NLReLU, an activation function that replaces ReLU's positive branch with a scaled natural logarithm and reports higher classification accuracy across ten convolutional networks.

desk verdict Simple new activation with credible shallow-net gains; the deep ResNet claim is confounded by a placement mismatch. read the letter →

arxiv 1908.03682 v2 pith:ZB5CN4D7 submitted 2019-08-10 cs.LG cs.CVstat.ML

Yang Liu , Jianpeng Zhang , Chao Gao , Jinghua Qu , Lixin Ji This is my paper · ORCID

classification cs.LGcs.CVstat.ML

keywords activationfunctionrectifiedlinearunitnaturallogarithmconvolutionalneuralnetworkdyingReLUvanishinggradientsparsebiasshift

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes NLReLU, an activation function that keeps ReLU's zero output for negative inputs but replaces the positive branch with the scaled natural logarithm $f(x)=\beta\ln(\max(0,x)+1)$. The claim is that this simple substitution preserves ReLU's sparse activation while easing two failure modes: neurons that die from large gradients and gradients that vanish in deep stacks. The mechanism offered is that the derivative $\beta/(x+1)$ shrinks large gradients, pushes per-layer mean activations toward zero, and reduces the uneven spread of activations across layers. Across ten convolutional networks, the paper reports average accuracy gains over ReLU of 0.16% on MNIST, 2.04% on CIFAR-10 shallow nets, and 1.35% on deep ResNets, with performance comparable to SELU. If these results transfer, NLReLU is a low-cost, parameterized activation that can replace ReLU in existing CNN code with one added scalar.

What carries the argument

The central object is the NLReLU transfer function with its scalar parameter $\beta$. Its work is done by the positive-branch derivative $\beta/(x+1)$: unlike ReLU's constant derivative of 1, this derivative falls as the input grows, so it compresses large gradients before they can kill a neuron or destabilize training, while staying far enough from zero (for $\beta$ near 1) to avoid the saturation that plagues sigmoids and tanh. The logarithmic transform also compresses large activations, which the paper argues reduces heteroscedasticity across layers and, by shifting mean activations closer to zero, lessens the bias shift effect that slows learning. In deep pre-activation residual networks, the paper identifies a placement rule: NLReLU should be kept at the residual unit's inner activation but removed from the block-input and final-output positions, and this placement choice is part of what produces the reported deep-network gains.

What would settle it

Run the same ResNet and shallow-CNN comparisons with NLReLU and ReLU using identical seeds, the same per-network $\beta$, and no placement search; if the average accuracy differences fall inside one standard deviation of the runs, the reported gains are tuning artifacts rather than properties of the log transform.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a parametric logarithmic transform applied only to the positive branch of ReLU improves network accuracy and training stability without giving up sparsity. Concretely, NLReLU is $f(x)=\beta\ln(\max(0,x)+1)$ with derivative $\beta/(\max(0,x)+1)$ for $x>0$ and $0$ otherwise. The decreasing derivative makes the function right-soft saturated, so gradients are minified for large activations; the log scale pulls mean activations toward zero; and the zero branch keeps the representation sparse. The authors report that this combination yields higher test accuracy than ReLU on all ten tested CNN configurations and matches SELU, and that NLReLU networks still converge at learning rates where ReLU networks diverge.

Load-bearing premise

The deep-network result depends on the paper's choice to place NLReLU only inside each residual unit, a configuration selected by comparing all eight placements on the same CIFAR-10 validation set; if that selection overfits the dataset, the reported 1.35% advantage is partly a placement artifact.

Editorial extensions

If this is right

If NLReLU's reported gains hold, swapping ReLU for NLReLU in existing shallow CNNs should give about 0.16% higher accuracy on MNIST and about 2.04% higher on CIFAR-10, with no architectural change beyond the activation.
In deep pre-activation ResNets, the same swap is reported to give 1.35% higher CIFAR-10 accuracy, provided NLReLU is placed inside each residual unit and not at the block input or output.
Because NLReLU's derivative minifies large gradients, networks using it can be expected to tolerate larger learning rates than ReLU before diverging, as demonstrated in the paper's MNIST control experiments.
NLReLU is reported to be comparable to SELU on the tested benchmarks, but without SELU's self-normalizing constraints, making it a simpler alternative that needs only the scalar $\beta$ per network.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because $\beta$ is tuned per network and dataset in the experiments, the headline averages are conditional on that tuning; a natural test is whether a fixed $\beta$ (say 1.0) preserves the reported gains, which the paper does not report.
The mean-shift and variance-reduction mechanism suggests NLReLU could help most in networks trained without batch normalization; the deep experiments all keep BN, so a direct test would be training BN-free deep networks with NLReLU.
The logarithmic compression of large activations implies a regularizing effect on noisy inputs, which is not explored in the paper but is directly measurable on corrupted-input benchmarks.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Fairly straightforward read. The function f(x)=ln(β max(0,x)+1) is new as far as the citations show—it doesn't appear in the earlier activation surveys—and it has a nice property: it keeps the sparse zero for negative inputs while compressing large positives with a logarithmic scale. The definition, derivative, and plots for β in [0.7,1.1] are clear.

What the paper does well is the systematic empirical sweep: ten networks, two datasets, ten runs each, means and standard deviations. On the shallow CNNs, NLReLU beats ReLU by about 0.16% on MNIST and 2.04% on CIFAR-10, and lands roughly at parity with SELU. Those tables are credible; I don't doubt that for these settings NLReLU is a serviceable drop-in replacement.

The weak spot is the deep ResNet section. Table 3 shows that for ResNet-50, using NLReLU only at position B (0,1,0) gives 82.34% vs 80.74% for (1,1,1). That's fine as an internal check. But Table 4 compares NLReLU with (0,1,0) against ReLU with the standard (1,1,1) placement. That is not apples-to-apples: the NLReLU networks have nonlinearities removed at positions A and C, which changes the architecture and gradient flow independently of the activation function. The paper never reports a ReLU baseline with (0,1,0), nor an NLReLU network with (1,1,1) across all depths. So the headline '1.35% higher than ReLU on deep ResNets' is not supported as an activation-function effect; it could be an architecture effect. That's a real flaw, not a nitpick.

Two smaller issues. First, β is tuned per network and dataset using 10-fold cross-validation, while comparison activations use default or fixed parameters. That's a mild bias in NLReLU's favor, though the sensitivity analysis at least shows the results don't swing wildly with β. Second, the mechanism story—reducing heteroscedasticity, bias shift, dying ReLU—rests on a self-made simulation and a citation to the ELU theorem, not on a formal analysis of NLReLU dynamics. It's plausible but thin.

Bottom line: the shallow-network contribution is a useful empirical addition to the activation zoo, and the paper deserves a serious referee. But the deep-ResNet claim needs a matched baseline before it becomes believable.

Referee Report

3 major / 5 minor

Summary. The paper proposes a novel activation function, NLReLU, defined as f(x)=β·ln(max(0,x)+1), which modifies the positive part of ReLU with a parametric logarithmic transform while preserving the zero response for negative inputs. The authors claim that NLReLU retains sparse activation, reduces bias shift and heteroscedasticity, alleviates dying ReLU and vanishing gradient problems, and improves classification accuracy over ReLU on MNIST and CIFAR-10 across shallow CNNs (SimpleCNN, LeNet-5, AlexNet, VGG-11/16/19) and deep ResNets (50/101/152/200). The empirical claims are supported by means and standard deviations over 10 runs, with per-network tuning of the β parameter and, for ResNets, a placement search over three activation positions.

Significance. If the reported accuracy gains hold under matched conditions, NLReLU would be a simple, sparsity-preserving activation function that offers small but reproducible gains over ReLU and parity with ELU/SELU on standard benchmarks. The paper's use of 10-run mean±std and comparisons across ten network depths is a strength. However, the central claims are weakened by per-network β tuning and, more importantly, by the unmatched activation-placement configuration in the deep ResNet experiments, which prevents attributing the observed gains to the activation function itself. The mechanism claims rest on a small simulation and an existing ELU theorem, with no formal derivation for NLReLU.

major comments (3)

[Section IV-B, Tables 3 and 4] The deep-network comparison is not matched on architecture. The NLReLU entries in Table 4 use placement (0,1,0), i.e., NLReLU is retained only at position B within each residual unit and removed from positions A and C, while the ReLU baseline appears to use the standard full pre-activation placement with the activation at all three positions. The paper does not report a ReLU baseline with the (0,1,0) placement, nor an NLReLU baseline with the (1,1,1) placement. Consequently, the reported 1.35% average improvement of NLReLU over ReLU may reflect the architecture change (fewer nonlinearities, altered gradient flow) rather than the NLReLU formula. Please provide matched comparisons for at least one network, e.g., ReLU-(0,1,0) vs NLReLU-(0,1,0) and NLReLU-(1,1,1) vs ReLU-(1,1,1).
[Section IV-A, Tables 1, 2, 4 and Fig. 4] The reported average gains (0.16%, 2.04%, 1.35%) are obtained with a different β value for each network and dataset, selected via a sensitivity analysis exemplified for AlexNet in Fig. 4. No automatic or principled rule for choosing β is provided, and the sensitivity analysis itself shows that accuracy varies with β. This means the headline numbers assume a practitioner will replicate the same oracle-like tuning. Please report results for a fixed β (e.g., β=1.0) across all networks, or provide a learning curve for β with a recommendation that does not depend on test-set performance.
[Section III, Fig. 2] The claims that NLReLU 'reduces heteroscedasticity in the data distribution among layers and the bias shift effect' are supported by a single simulation (Fig. 2) and by reference to Theorem 2 of [12] for the bias-shift relationship. The simulation description is incomplete: it reports mean and standard deviation of the number of activated neurons but does not specify the training procedure, the network input distribution, or whether the statistics are from a forward pass or after training. Without a reproducible setup and a formal argument connecting NLReLU's derivative to the bias-shift theorem, these mechanism claims are not established. Please either provide the simulation code and a more precise setup, or soften the claims to 'we observe in simulation'.

minor comments (5)

[Abstract] The abstract contains an incomplete sentence: 'This activation function uses the parametric natural logarithmic transform to improve ReLU and is simply defined as. NLReLU' — the equation is missing or not rendered.
[Section IV-A] In the experiment setup, 'CIRAR-10' should be 'CIFAR-10'.
[Section II-B] The text says 'ReLU does not restain the gradient'; 'restain' should be 'restrain'.
[Section III] The sentence 'In Section III, we find that the aforementioned activation functions' appears to reference the wrong section; it should likely reference Section II, where those functions are reviewed.
[General] The paper does not provide the source of the TensorFlow implementations or a link to code; making the code and hyperparameter configuration public would substantially improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the accuracy claims are direct measurements, and the mechanism discussion is supported by the activation's derivative and an external theorem, not by a self-referential fit.

full rationale

The paper is an empirical comparison, not a derivation of predictions from fitted constants. NLReLU is defined in Eq. (1), and its first derivative in Eq. (2) is used to discuss gradient behavior; these are definitions, not circular reductions. The headline accuracy improvements in Tables 1, 2, and 4 are measured test-set accuracies under stated hyperparameters and beta values. Beta is tuned per network via sensitivity analysis and cross-validation, but each reported accuracy is a direct measurement, not a value forced by the tuned beta. The mechanism claims about reducing heteroscedasticity and bias shift are supported by the simulation in Fig. 2 and by the external theorem of Clevert et al. (Ref. [12]); they do not assume the conclusion. The only self-citation, Ref. [18], is explicitly historical: 'The concept of NLReLU was first used in attention-gated convolutional neural networks [18] for sentence classification.' This prior-use citation is not load-bearing for any derivation or accuracy claim. The ResNet placement choice (0,1,0) selected in Table 3 is a possible experimental confound when compared against the standard ReLU baseline, but that is a comparison-validity concern rather than circularity: the NLReLU accuracy is still measured, not obtained by construction from the chosen placement. No fitted parameter, self-citation chain, or definitional identity makes any reported result equivalent to its own input.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper's central empirical claim is measured, not derived, so the main free parameters are the per-network β scale and the ResNet placement choice. The mechanism arguments import the bias-shift theorem from [12] and rely on a toy simulation that is assumed to transfer to CNNs. No new physical entities are introduced.

free parameters (2)

β (per-network scale) = 0.70 to 1.10 (e.g., 1.00 for SimpleCNN/LeNet-5 MNIST, 0.70 for VGG-11, 1.10 for ResNets)
β controls the log compression slope and is tuned per network and dataset via sensitivity analysis (Fig. 4, Tables 1, 2, 4). The reported accuracy depends on this tuned value.
NLReLU placement in ResNet = Positions A=0, B=1, C=0
A binary configuration chosen after evaluating 8 cases on ResNet-50 (Table 3); used for all ResNet depths in Table 4. This architectural choice is not a parameter of the activation function but affects the measured gains.

assumptions (4)

standard math Chain rule for backpropagation (Eqs. 3-4)
Used to derive NLReLU gradients; standard calculus.
domain assumption Reducing mean activations toward zero speeds learning and reduces bias shift (Clevert et al. [12], Theorem 2)
Adopted from ELU paper and used in Section III to argue NLReLU's mean-shifting property helps convergence; not re-derived here.
domain assumption The fully-connected simulation with standard normal inputs represents CNN activation distributions (Fig. 2)
Used to demonstrate reduced heteroscedasticity; whether this transfers to convolutions and real data is assumed, not tested in the paper.
domain assumption No data augmentation, weight decay, or learning-rate decay for clean comparison
Experimental design choice (Section IV-B-1); the conclusions may not transfer to settings using these tricks.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Natural-Logarithm-Rectified Activation Function in Convolutional Neural Networks." pith.science (2026). https://pith.science/paper/ZB5CN4D7

@misc{pith2026190803682,
  author       = {Pith},
  title        = {Pith review of: Natural-Logarithm-Rectified Activation Function in Convolutional Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZB5CN4D7}},
  note         = {Machine review of arXiv:1908.03682}
}

read the original abstract

Activation functions play a key role in providing remarkable performance in deep neural networks, and the rectified linear unit (ReLU) is one of the most widely used activation functions. Various new activation functions and improvements on ReLU have been proposed, but each carry performance drawbacks. In this paper, we propose an improved activation function, which we name the natural-logarithm-rectified linear unit (NLReLU). This activation function uses the parametric natural logarithmic transform to improve ReLU and is simply defined as. NLReLU not only retains the sparse activation characteristic of ReLU, but it also alleviates the "dying ReLU" and vanishing gradient problems to some extent. It also reduces the bias shift effect and heteroscedasticity of neuron data distributions among network layers in order to accelerate the learning process. The proposed method was verified across ten convolutional neural networks with different depths for two essential datasets. Experiments illustrate that convolutional neural networks with NLReLU exhibit higher accuracy than those with ReLU, and that NLReLU is comparable to other well-known activation functions. NLReLU provides 0.16% and 2.04% higher classification accuracy on average compared to ReLU when used in shallow convolutional neural networks with the MNIST and CIFAR-10 datasets, respectively. The average accuracy of deep convolutional neural networks with NLReLU is 1.35% higher on average with the CIFAR-10 dataset.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 20 canonical work pages

[12]

Fast and accurate deep network learning by exponential linear units (elus) ,

D. A. Clevert, T. Unterthiner, and S. Hochreiter . ( 2015 ). “ Fast and accurate deep network learning by exponential linear units (elus) , ” [ On line ] . Available: https://arxiv.org/abs/ 1511.07289

arXiv 2015
[1]

Rectified linear units improve restricted boltzmann machines ,

V. Nair and G. E. Hinton, “ Rectified linear units improve restricted boltzmann machines , ” i n Proceedings of the 27th International Conference on Machine Learning , Haifa, Israel, 2010 , pp. 807 -

work page 2010
[2]

Deep sparse re ctifier neural networks ,

X. Glorot, A. Bordes, and Y. Bengio, “ Deep sparse re ctifier neural networks , ” i n Proceedings of the 14th International Conference on Artificial Intelligence and Statistic s , Fort Lauderdale, FL, USA, 2011 , pp. 315 -

work page 2011
[4]

Deep residual learning for im age recognition ,

K. He, X. Zhang, S. Ren, and J. Sun, “ Deep residual learning for im age recognition , ” i n Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , Las Vegas, Nevada, USA, 2016 , pp. 770 -

work page 2016
[5]

Identity mappings in deep residua l networks ,

K. He, X. Zhang, S. Ren, and J. Sun, “ Identity mappings in deep residua l networks , ” i n European Conference on Computer Vision , Amsterdam, The Netherlands, 2016 , pp. 630 -

work page 2016
[6]

Densely connected convolutional networks ,

G. Huang, Z. Liu, L. V an Der Maaten, and K. Q. Weinberger, “ Densely connected convolutional networks , ” i n Proceedings of the IEEE Conference on Computer Vision and P attern Recognition , Honolulu, Hawaii, USA, 2017 , pp. 4700 -

work page 2017
[7]

Bridging nonlinearities and stochastic regularizers with Gaussian error linear units ,

D. Hendrycks and K. Gimpel . ( 2016 ) . “ Bridging nonlinearities and stochastic regularizers with Gaussian error linear units , ” [ On line ] . Available: https://arxiv.org/abs/ 1606.08415

arXiv 2016
[8]

Self - normalizing neural networks ,

G. Klambauer, T. Unterthiner, A. M ayr, and S. Hochreiter, “ Self - normalizing neural networks , ” i n Advances in Neural Information Processing Systems , Long Beach, California, USA, 2017 , p p. 971 -

work page 2017

Show all 24 references

[9]

Searching for activation functions ,

P. Ramachandran, B. Zoph, and Q. V. Le . ( 2017 ) . “ Searching for activation functions , ” [ On line ] . Available: https://arxiv.org/ abs / 17 10 . 0 5941

2017
[10]

Rectifier nonlinearities improve neural network acoustic models ,

A. L. Maas, A. Y. Hannun, and A. Y. Ng, “ Rectifier nonlinearities improve neural network acoustic models , ” i n Proceedings of the 30th Internati onal Conference on Machine Learnin g , Atlanta, Georgia, USA, 2013 , pp. 3 -

2013
[11]

Delving deep into rectifiers: Surpassing human - level performance on imagenet classification ,

K. He, X. Zhang, S. Ren, and J. Sun, “ Delving deep into rectifiers: Surpassing human - level performance on imagenet classification , ” i n Proceedings of the IEEE Internationa l Conference on Computer Vision , Santiago, C hile, 2015 , pp. 1026 - 1034

2015
[13]

Understanding and improving convolutional neural ne tworks via concatenated rectified linear units ,

W. Shang, K. S ohn, D. Almeida, and H. L ee, “ Understanding and improving convolutional neural ne tworks via concatenated rectified linear units , ” i n International Conference on Machine Learning , New York City, New York, USA, 2016 , pp. 2217 -

2016
[14]

Empiri cal evaluation of rectified activations in convolutio nal network ,

B. Xu, N. Wang, T. Chen, and M. Li . ( 2015 ) . “ Empiri cal evaluation of rectified activations in convolutio nal network , ” [ On line ] . Available: https://arxiv.org/abs/ 1505.00853

2015 arXiv
[15]

Multivariate neural network operators with sigmoidal activation functions ,

D. Costarelli and R. Spigler, “ Multivariate neural network operators with sigmoidal activation functions , ” Neural Networks , vol. 48 , no . 12 , pp. 72 - 77 , Dec. 2013

2013
[16]

Imagenet classificatio n with deep convolutional neural networks ,

A. Krizhevsky, I. Sutskever , and G. E. Hinton, “ Imagenet classificatio n with deep convolutional neural networks , ” i n Advances in Neural Information Processing Systems , Lake Tahoe, Nevada, USA, 2012 , pp. 1097 -

2012
[17]

Noisy activation functions ,

C. Gulcehre, M . Moczulski , M. Denil, and Y. Bengio, “ Noisy activation functions , ” i n International conference on Machine Learning , New York City, New York, USA, 2016 , pp. 3059 -

2016
[19]

Basic Econometrics , 5 th ed ., McGraw - Hill , New York City, N Y , USA, 2004 , pp. 421 -

2004
[20]

Understanding the difficulty of training deep feedforw ard neural networks ,

X. Glorot and Y. Bengio, “ Understanding the difficulty of training deep feedforw ard neural networks , ” i n International Conference on Artifici al Intelligence and Statistics , Chia Laguna Resort, Sardinia, Italy, 2010 , pp. 249 –

2010
[21]

Improving deep neural network with multiple parametric exponenti al linear units ,

Y. Li, C. Fan, Y. Li, Q. Wu, and Y. Ming, “ Improving deep neural network with multiple parametric exponenti al linear units , ” Neurocomputing , vol. 301 , no. 8 , pp. 11 - 2 , Aug. 2018

2018
[22]

Batch Normalization: Accelerating De ep Network Training by Reducing Internal Covariate Shift ,

S. Ioffe and C. Szegedy, “ Batch Normalization: Accelerating De ep Network Training by Reducing Internal Covariate Shift , ” i n International Conference on Machine Learning , Lille, France, 2015 , pp. 448 -

2015
[23]

Gradient - based learning applied to document recognition ,

Y. LeCun, L. Bottou, Y. Bengio, and P. Ha ffner, “ Gradient - based learning applied to document recognition , ” Proceedings of the IEEE , vol. 86 , no . 11, pp. 2278 – 2323 , Nov. 1998

1998
[24]

Learning multiple layers of fea tures from tiny images ,

A. Krizhevsky and G. Hinton , “ Learning multiple layers of fea tures from tiny images , ” Technical report , University of Toronto , vol. 1, no. 4 , p. 7, 2009

2009
[25]

T ensorflow: A system for large - scale machine learning ,

M. Ab adi et al. , “ T ensorflow: A system for large - scale machine learning , ” i n USENIX Symposium on Operating Systems Design and Implementation , Savannah, Georgia, US A, 2016 , pp. 265 –

2016
[26]

Adam: A Method for Stochastic Optimization ,

D. P. Kingma and J. Ba . ( 2014 ) . “ Adam: A Method for Stochastic Optimization , ” [ On line ] . Available: https://arxiv.org/abs/ 1412.6980

2014 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [12]

Fast and accurate deep network learning by exponential linear units (elus) ,

D. A. Clevert, T. Unterthiner, and S. Hochreiter . ( 2015 ). “ Fast and accurate deep network learning by exponential linear units (elus) , ” [ On line ] . Available: https://arxiv.org/abs/ 1511.07289

arXiv 2015

[2] [1]

Rectified linear units improve restricted boltzmann machines ,

V. Nair and G. E. Hinton, “ Rectified linear units improve restricted boltzmann machines , ” i n Proceedings of the 27th International Conference on Machine Learning , Haifa, Israel, 2010 , pp. 807 -

work page 2010

[3] [2]

Deep sparse re ctifier neural networks ,

X. Glorot, A. Bordes, and Y. Bengio, “ Deep sparse re ctifier neural networks , ” i n Proceedings of the 14th International Conference on Artificial Intelligence and Statistic s , Fort Lauderdale, FL, USA, 2011 , pp. 315 -

work page 2011

[4] [4]

Deep residual learning for im age recognition ,

K. He, X. Zhang, S. Ren, and J. Sun, “ Deep residual learning for im age recognition , ” i n Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , Las Vegas, Nevada, USA, 2016 , pp. 770 -

work page 2016

[5] [5]

Identity mappings in deep residua l networks ,

K. He, X. Zhang, S. Ren, and J. Sun, “ Identity mappings in deep residua l networks , ” i n European Conference on Computer Vision , Amsterdam, The Netherlands, 2016 , pp. 630 -

work page 2016

[6] [6]

Densely connected convolutional networks ,

G. Huang, Z. Liu, L. V an Der Maaten, and K. Q. Weinberger, “ Densely connected convolutional networks , ” i n Proceedings of the IEEE Conference on Computer Vision and P attern Recognition , Honolulu, Hawaii, USA, 2017 , pp. 4700 -

work page 2017

[7] [7]

Bridging nonlinearities and stochastic regularizers with Gaussian error linear units ,

D. Hendrycks and K. Gimpel . ( 2016 ) . “ Bridging nonlinearities and stochastic regularizers with Gaussian error linear units , ” [ On line ] . Available: https://arxiv.org/abs/ 1606.08415

arXiv 2016

[8] [8]

Self - normalizing neural networks ,

G. Klambauer, T. Unterthiner, A. M ayr, and S. Hochreiter, “ Self - normalizing neural networks , ” i n Advances in Neural Information Processing Systems , Long Beach, California, USA, 2017 , p p. 971 -

work page 2017