Empirical Evaluation of Rectified Activations in Convolutional Network
read the original abstract
In this paper we investigate the performance of different types of rectified activation functions in convolutional neural network: standard rectified linear unit (ReLU), leaky rectified linear unit (Leaky ReLU), parametric rectified linear unit (PReLU) and a new randomized leaky rectified linear units (RReLU). We evaluate these activation function on standard image classification task. Our experiments suggest that incorporating a non-zero slope for negative part in rectified activation units could consistently improve the results. Thus our findings are negative on the common belief that sparsity is the key of good performance in ReLU. Moreover, on small scale dataset, using deterministic negative slope or learning it are both prone to overfitting. They are not as effective as using their randomized counterpart. By using RReLU, we achieved 75.68\% accuracy on CIFAR-100 test set without multiple test or ensemble.
This paper has not been read by Pith yet.
Forward citations
Cited by 18 Pith papers
-
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
DCGANs with architectural constraints learn a hierarchy of representations from object parts to scenes in both generator and discriminator across image datasets.
-
Locally Near Optimal Piecewise Linear Regression in High Dimensions via Difference of Max-Affine Functions
ABGD parametrizes piecewise linear functions as difference of max-affine functions and converges linearly to an epsilon-accurate solution with O(d max(sigma/epsilon,1)^2) samples under sub-Gaussian noise, which is min...
-
Score-based Greedy Search for Structure Identification of Partially Observed Linear Causal Models
Introduces Generalized N Factor Model and LGES algorithm that identifies true causal structure including latents up to Markov equivalence class via score-based greedy search.
-
Searching for Activation Functions
Automated search discovers Swish activation f(x) = x * sigmoid(βx) that improves top-1 ImageNet accuracy over ReLU by 0.9% on Mobile NASNet-A and 0.6% on Inception-ResNet-v2.
-
Materialistic RIR: Material Conditioned Realistic RIR Generation
A two-module neural model disentangles spatial layout from material properties to generate controllable and more realistic room impulse responses, reporting gains of up to 16% on acoustic metrics and 70% on material m...
-
Functional Similarity Metric for Neural Networks: Overcoming Parametric Ambiguity via Activation Region Analysis
A functional similarity metric for ReLU networks uses normalized activation region signatures and MinHash to overcome parametric symmetries like neuron permutation and scaling.
-
On Divergence Measures for Training GFlowNets
Introduces statistically efficient estimators for Renyi-α, Tsallis-α, reverse and forward KL divergences with REINFORCE and score-matching control variates for faster GFlowNet training.
-
Improving Description-based Person Re-identification by Multi-granularity Image-text Alignments
The MIA model with GC, RGA, and BFM modules achieves state-of-the-art performance on the CUHK-PEDES dataset for description-based person re-identification.
-
Sparsity Hurts: Simple Linear Adapter Can Boost Generalized Category Discovery
LAGCD inserts residual linear adapters into each ViT block plus a distribution alignment loss to improve generalized category discovery by increasing model flexibility while reducing bias between seen and novel classes.
-
Activation Function Design Sustains Plasticity in Continual Learning
Smooth-Leaky and Randomized Smooth-Leaky activations mitigate loss of plasticity in continual learning by targeting negative-branch shape and saturation behavior.
-
Gamma-Ray Burst Light Curve Reconstruction: A Comparative Machine and Deep Learning Analysis
MLP and Attention U-Net outperform other models in reconstructing GRB light curves on 521 events, cutting plateau parameter uncertainties by 37-41% versus the Willingale baseline while achieving low MSE.
-
High-throughput Onboard Hyperspectral Image Compression with Ground-based CNN Reconstruction
Prequantization-based lossless predictive compression onboard hyperspectral images with CNN ground reconstruction recovers the entire SNR drop at 2 bpp.
-
Adaptive Reorganization of Neural Pathways for Continual Learning with Spiking Neural Networks
SOR-SNN employs Self-Organizing Regulation networks to reorganize a single SNN into sparse pathways, achieving better performance, energy efficiency, memory use, backward transfer, and self-repair on continual learnin...
-
Discriminative Embedding Autoencoder with a Regressor Feedback for Zero-Shot Learning
A new autoencoder model with margin-based discriminative embeddings and regressor feedback outperforms prior zero-shot learning methods on SUN, CUB, AWA1 and AWA2, with larger gains in generalized ZSL.
-
Two-stream Spatiotemporal Feature for Video QA Task
A two-stream spatiotemporal feature extractor with squeeze-and-excitation and attention-based context matching improves text-only video QA on TVQA but shows limitations with visual features.
-
On Reducing Negative Jacobian Determinant of the Deformation Predicted by Deep Registration Networks
Two training mechanisms for unsupervised deep registration networks reduce the number of locations with negative Jacobian determinants in predicted deformations.
-
Modern CNNs for IoT Based Farms
A survey of state-of-the-art CNN architectures for agricultural IoT applications that proposes a tailored classification taxonomy and reviews existing research to guide architecture selection.
-
Deep learning in ultrasound imaging
A review outlining deep learning strategies for adaptive beamforming, spectral Doppler, compressive color Doppler encodings, and structured signal recovery in ultrasound.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.