SmartMixed: A Two-Phase Training Strategy for Adaptive Activation Function Learning in Neural Networks

Amin Omidvar

REVIEW 4 major objections 5 minor 13 references

SmartMixed: A Two-Phase Training Strategy for Adaptive Activation Function Learning in Neural Networks

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SmartMixed lets each neuron learn and freeze its own activation function, ranking near the top across 18 MNIST architectures.

desk verdict Plausible two-phase activation-selection idea with a nice layer-wise preference observation, but the headline ranking is confounded by unequal training epochs and missing error bars. read the letter →

arxiv 2510.22450 v4 pith:BEEPY2BX submitted 2025-10-25 cs.LG cs.AI

Amin Omidvar This is my paper · ORCID

classification cs.LGcs.AI

keywords activationfunctionsGumbel-Softmaxtwo-phasetrainingper-neuronadaptationlayer-wisespecializationMNISTstraight-throughestimatorfeedforwardnetworks

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SmartMixed is a two-phase training strategy that lets every neuron in a feedforward network pick its own activation function from a pool of six candidates—ReLU, Sigmoid, Tanh, Leaky ReLU, ELU, SELU—instead of sharing one global activation. In the first phase, a hard Gumbel-Softmax estimator learns a categorical logit per neuron; after the logits stabilize, the argmax choice is frozen and the whole network trains further with ordinary vectorized operations. On 18 MNIST architectures, SmartMixed finishes in the top three of seven activation strategies on the majority of networks, trailing the best fixed choices like Leaky ReLU and ReLU by small margins. The paper's more striking finding is a consistent layer-wise pattern: early layers favor ReLU and Leaky ReLU, while deeper layers shift toward ELU and SELU. A sympathetic reader would take this as evidence that neuron-level activation specialization is learnable in practice and that depth systematically changes which activation a neuron prefers.

What carries the argument

The load-bearing mechanism is the hard Gumbel-Softmax estimator: each neuron keeps a logit vector over the six activation candidates, adds Gumbel noise, softmax-normalizes at a temperature, then in the forward pass takes the one-hot argmax while in the backward pass replaces the discrete gradient with the soft relaxation via a stop-gradient. This straight-through trick is what makes discrete per-neuron selection differentiable. The second phase converts the learned logits into a fixed mixed network by taking argmax and grouping neurons by activation type so one affine transformation is shared. The work it does is to turn 'which activation for which neuron' into an optimizable categorical cho

What would settle it

Retrain the same 18 architectures with Phase-1 selections replaced by random per-neuron activations, or by a deliberately reversed pattern (ELU/SELU in early layers, ReLU in deep layers). If these control versions match or beat SmartMixed, the learned selections are not carrying the performance; if SmartMixed clearly wins, the learned layer-wise choices are doing real work.

Watch

Extended reading notes

Core claim

The central claim is that per-neuron activation functions can be learned without paying an inference-time cost: Phase 1 trains a hard one-hot mixture using a straight-through Gumbel-Softmax estimator; Phase 2 fixes each neuron's argmax activation and continues training, grouping neurons by chosen activation for vectorized efficiency. The paper reports that this two-phase procedure yields validation accuracy of 98.22% and test accuracy of 98.03% on a representative seven-hidden-layer network—about 2–3 points above the Phase-1 mixture—and places SmartMixed among the top three activation strategies on most of 18 architectures. It further claims that the learned selections are not arbitrary: acr

Load-bearing premise

Everything rests on the straight-through Gumbel-Softmax gradient being an accurate guide to each neuron's genuinely best activation; if the bias in that estimator leads logits to arbitrary values, Phase 1's argmax is meaningless and Phase 2 is just training a randomly mixed network.

Editorial extensions

If this is right

If per-neuron choices are genuinely learned, a single network can contain multiple activation types, and continued training after freezing improves accuracy by 2–3 points over the mixed selection phase.
The layer-wise preference is consistent: early layers favor ReLU and Leaky ReLU, deeper layers favor ELU and SELU, across 18 architectures of varying depth and width.
SmartMixed's inference cost equals that of a fixed-activation network because only one activation is evaluated per neuron, and grouping by activation type keeps operations vectorized.
No single fixed activation dominates; SmartMixed, Leaky ReLU, and ReLU occupy the top-3 slots on most architectures, while Sigmoid consistently underperforms.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves implicit that the same learn-then-freeze recipe could apply to other discrete architectural choices, such as per-layer normalization type or per-neuron quantization, suggesting a general 'learn and commit' strategy.
Because the Gumbel-Softmax temperature is never specified, the reported layer-wise preferences are partly a function of that annealing schedule; sweeping temperature and checking whether the ReLU-to-ELU shift persists would be a direct test.
The straight-through estimator's bias means the argmax at the transition epoch may be a noisy reflection of the true best activation; comparing against exhaustive per-neuron selection via leave-one-out retraining would show how much the learned choices matter.
On harder datasets than MNIST, the preference shift to ELU/SELU in deep layers might become more pronounced or reverse; that is an empirical question the paper does not address.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

What you should know: this is an incremental but sensible method — per-neuron Gumbel-Softmax selection, then freeze the argmax and keep training. The new bit is the freeze-and-continue phase plus the claim that early layers want ReLU/Leaky ReLU and deep layers want ELU/SELU. That observation is the most interesting thing in the paper, and it holds up qualitatively for the one architecture they show in Figure 3.

They also do the right thing by shipping code and running 18 architectures. Credit for that.

The soft spots are real, and one is load-bearing. The paper never says how many epochs the six fixed-activation baselines were trained for. SmartMixed gets 50 + 350 = 400 epochs, and Table I shows that the extra 350 epochs alone lifts accuracy from 95.73% to 98.03%. If the baselines were only trained the equivalent of Phase 1, Table II's top-three rankings are a compute artifact. Even if the baselines also got 400 epochs, we have no evidence of matched schedules or batch size. This directly threatens the headline claim that SmartMixed 'competes' with fixed activations. The paper's own MRR figure puts Leaky ReLU and ReLU ahead of SmartMixed, so the adaptive method is not actually winning.

The second issue is the absence of seeds or error bars. On MNIST with small fully connected nets, the differences between top activation functions are often tiny; without variance, ranks are noise. The Gumbel-Softmax temperature is unspecified and the ST gradient bias is not validated, but I'd call that a minor concern compared to the epoch mismatch — the selections could still be meaningful even if the estimator is biased.

The layer-wise preference claim is framed as a discovery, but it's an empirical readout from the same trained models, and they show the distribution for only one architecture. They say 'experiments consistently revealed similar preferences' without showing the numbers. That's an assertion, not a result.

Net: the idea is plausible, the paper is clearly written, and the code release is good. But the central empirical comparison needs to be re-run with matched compute and multiple seeds before I'd trust the ranking or the 'novel insight.' The layer-wise preference pattern is worth a quick replication; if it holds, that's a useful empirical note.

For peer review: I'd send it out. A competent referee can ask for the matching-epochs control and error bars, and the paper would be stronger for it. It's not a desk reject, but it's a long way from 'adaptive activation functions work better than fixed ones.'

Referee Report

4 major / 5 minor

Summary. The paper proposes SmartMixed, a two-phase training scheme in which each hidden neuron learns a categorical choice over six activation functions (ReLU, Sigmoid, Tanh, Leaky ReLU, ELU, SELU) via a hard Gumbel-Softmax estimator. In Phase 1 the logits are trained for 50 epochs; in Phase 2 the argmax selection is frozen and the resulting mixed-activation network is trained for 350 more epochs. The authors evaluate the method on 18 fully connected MNIST architectures and claim that SmartMixed ranks in the top three among seven activation strategies in the majority of cases. They further claim a consistent layer-wise preference: early layers prefer ReLU/Leaky ReLU, deeper layers prefer ELU/SELU.

Significance. If the claims were established, the paper would provide a simple, practical method for per-neuron activation learning and an interesting empirical observation about layer-wise activation preferences. The strengths are that the method is easy to implement, the code is released, and the evaluation spans 18 architectures with a clear ranking protocol. However, the current evidence does not yet support the headline claims: the comparison against fixed-activation baselines is confounded by unequal training budgets, the results are single runs with no error bars, and there is no control to show that the learned selections, rather than the extra training epochs or the mere presence of mixed activations, cause the reported gains.

major comments (4)

[IV-B, IV-D; Tables I and II] The central ranking claim rests on an uncontrolled comparison. Section IV-B states that SmartMixed trains for alpha=50 Phase-1 epochs plus 350 Phase-2 epochs, i.e. 400 epochs total, and Table I shows that the Phase-2 continuation alone raises test accuracy from 95.73% to 98.03%. Section IV-D says six fixed-activation models were trained but never states their epoch count. If those baselines were trained for 50 epochs, the Table II rankings are dominated by training budget rather than activation-function choice; if they were trained for 400 epochs, this must be stated along with matched schedules. The authors should report matched-budget results, including the Phase-1-only SmartMixed result, and provide the actual test accuracies for every architecture rather than ranks alone.
[IV-D; Figures 4 and 5] All results appear to come from a single run per architecture with no random seeds, no error bars, and no significance testing. MNIST accuracies for these architectures are highly saturated, and small differences can flip ranks. The top-3 claim and the MRR ordering in Figure 4 are not meaningful without at least several seeds and a measure of variance. This is a load-bearing issue because the paper's main evidence is the ranking pattern across 18 architectures.
[III-B and IV-B; Table I] The paper never demonstrates that the learned activation selections are the cause of the Phase-2 improvement. The straight-through gradient estimator in Eq. (3) gives a biased approximation, and the temperature parameter is never specified, so the logits may not converge to genuinely good choices. Moreover, Phase 2 could improve merely because the network is trained longer or because a mixture of activations is beneficial. The authors should add a control in which the same 400-epoch protocol is run with (a) randomly assigned per-neuron activations and (b) a uniformly mixed activation assignment. Without such a control, the 'learned preferences' narrative is unsupported.
[IV-C, IV-D; Figure 3 and Table II] The layer-wise preference claim is based on one representative architecture in Figure 3, while the text asserts that the pattern is consistent across all 18 architectures without presenting the per-layer distributions for the remaining architectures. Additionally, the MRR analysis in Figure 4 shows Leaky ReLU as the best method and ReLU close behind SmartMixed, so the paper's language about 'optimal' per-neuron activation learning is too strong. The authors should provide a quantitative summary of activation distributions for all architectures and temper the claims accordingly.

minor comments (5)

[III-B, Eq. (1)] The formula for the Gumbel noise, g = -log(-log(v+epsilon)+epsilon), is unusual; typically one writes g = -log(-log(v)) with a numerical-stability term applied carefully. Please clarify the exact expression and report the value of epsilon.
[IV-B] Hyperparameters for Phase 1 and Phase 2 are incomplete: optimizer, learning rate, batch size, weight initialization, and the Gumbel-Softmax temperature tau are not reported. The code may contain them, but the paper should state them for reproducibility.
[IV-D, Table II] Table II gives only ranks, not accuracies. The statement that 'performance differences between most activation functions are relatively small' cannot be verified. Include the numeric test accuracy for each model.
[IV-C and Figure 6] The claim that ELU-source neurons have more positive outgoing weights is based on a single heatmap without a color scale, error bars, or a statistical test. Please clarify what 'average connection weights' are averaged over and quantify the observation.
[II, Related Work] Reference [10] is a dynamic graph convolutional network paper and does not appear to be about dynamic activation selection. Please cite an appropriate dynamic-selection activation method or rephrase the sentence.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SmartMixed's claims are empirical training outcomes, not derivations from their own definitions.

full rationale

The paper's central claims—SmartMixed ranks in the top three on most of 18 architectures, and layers show consistent activation preferences—are empirical results of a training procedure, not consequences of a definition or a fitted quantity masquerading as a prediction. The learned per-neuron selections are obtained by optimizing logits with a straight-through Gumbel-Softmax estimator (Eq. 3), and the reported layer-wise preferences are descriptive summaries of the argmax choices in Eq. (7); there is no equation in which an output is set equal to an input by construction. The only external method cited for the core mechanism is Gumbel-Softmax [13], which is an independent, non-self citation. The paper's self-citation [1] is unrelated to the activation-selection mechanism and is not load-bearing. The skeptical concern about unmatched training epochs (Phase 2 adds 350 epochs beyond Phase 1) is a genuine experimental-validity threat to the ranking comparison, but it is not circularity: it does not make the result equivalent to its own inputs by definition. Similarly, the paper's admission that the extra 350 epochs were run 'primarily out of curiosity' and the unspecified Gumbel temperature are limitations of evidence, not circular steps. Under the stated rules requiring a quoted reduction to an input, no circular step can be exhibited.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the Gumbel-Softmax selection being a faithful optimizer of per-neuron activations, the hand-set transition epoch 50, and the transferability of MNIST findings.

free parameters (3)

alpha (transition epoch) = 50
Hand-set number of Phase 1 epochs; the paper says it is adjustable and future work would select it dynamically. The claim that preferences stabilize by this point is supported only by loss curves, not by selection stability.
Gumbel-Softmax temperature tau = not reported
Temperature controls the sharpness of the softmax; no value is given, but it directly affects the learned selections and gradient quality.
epsilon (numerical stability) = not reported
Added in Eq. (1); value unspecified; minor but relevant to the Gumbel sampling.

assumptions (4)

standard math Gumbel-Max trick produces unbiased samples from the categorical distribution defined by the logits (Eq. 6).
This is a standard property of Gumbel noise; used to justify the selection mechanism in Section III-B.
domain assumption Straight-through estimator provides useful gradient signal for the hard one-hot selection (Eq. 3-4).
The ST estimator is a known heuristic with biased gradients; the paper assumes it drives logits toward good activations without validation.
ad hoc to paper Preferences have stabilized by epoch 50 in the representative network.
Section IV-B sets alpha=50 and asserts the loss curves confirm 'activation preference stabilization', but no quantitative criterion is given.
domain assumption MNIST is a sufficient benchmark to draw conclusions about layer-wise activation preferences.
All claims are based on MNIST fully connected networks; generalization is assumed, not tested.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SmartMixed: A Two-Phase Training Strategy for Adaptive Activation Function Learning in Neural Networks." pith.science (2026). https://pith.science/paper/BEEPY2BX

@misc{pith2026251022450,
  author       = {Pith},
  title        = {Pith review of: SmartMixed: A Two-Phase Training Strategy for Adaptive Activation Function Learning in Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BEEPY2BX}},
  note         = {Machine review of arXiv:2510.22450}
}

read the original abstract

The choice of activation function plays a critical role in neural networks, yet most architectures still rely on fixed, uniform activation functions across all neurons. We introduce SmartMixed, a novel two-phase training strategy that allows networks to learn optimal per-neuron activation functions while preserving computational efficiency at inference. In the first phase, neurons adaptively select from a pool of candidate activation functions (ReLU, Sigmoid, Tanh, Leaky_ReLU, ELU, SELU) using a differentiable hard mixture mechanism. In the second phase, each neuron's activation function is fixed according to the learned selection, resulting in a computationally efficient network that supports continued training with optimized vectorized operations. We evaluate SmartMixed on the MNIST dataset using feedforward neural networks of different architectures. Our analysis reveals that neurons in different layers exhibit distinct preferences for activation functions, providing insights into the functional diversity within neural architectures. We also demonstrated that SmartMixed effectively trains the network by allowing neurons to select their preferred activation functions, competing against models using a single fixed state-of-the-art activation function.

Figures

Figures reproduced from arXiv: 2510.22450 by the authors.

**Figure 2.** Training and validation loss curves during Phase 2 (mixed network [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Final activation function distribution across layers after Phase 1 training. Early layers favor ReLU and Leaky ReLU, while deeper layers increasingly [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Mean Reciprocal Rank analysis across all architectures. Leaky ReLU [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Ranking distribution showing the frequency of each rank position (1st through 7th) achieved by each activation function across all 18 architectures. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: Heatmap of average connection weights between neurons grouped [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 4 linked inside Pith

[1]

Using neural network for identifying clickbaits in online news media,

A. Omidvar, H. Jiang, and A. An, “Using neural network for identifying clickbaits in online news media,” inAnnual International Symposium on Information Management and Big Data. Springer, 2018, pp. 220–232

2018
[2]

Deep sparse rectifier neural networks,

X. Glorot, A. Bordes, and Y . Bengio, “Deep sparse rectifier neural networks,” inProceedings of the fourteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2011, pp. 315–323

2011
[3]

Rectifier nonlinearities improve neural network acoustic models,

A. L. Maas, A. Y . Hannun, A. Y . Nget al., “Rectifier nonlinearities improve neural network acoustic models,” inProc. icml, vol. 30, no. 1. Atlanta, GA, 2013, p. 3

2013
[4]

Fast and accurate deep network learning by exponential linear units (elus),

D.-A. Clevert, T. Unterthiner, and S. Hochreiter, “Fast and accurate deep network learning by exponential linear units (elus),”arXiv preprint arXiv:1511.07289, vol. 4, no. 5, p. 11, 2015

arXiv 2015
[5]

Self- normalizing neural networks,

G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter, “Self- normalizing neural networks,”Advances in neural information process- ing systems, vol. 30, 2017

2017
[6]

Searching for activation functions,

P. Ramachandran, B. Zoph, and Q. V . Le, “Searching for activation functions,”arXiv preprint arXiv:1710.05941, 2017

arXiv 2017
[7]

A survey on modern trainable activation functions,

A. Apicella, F. Donnarumma, F. Isgr `o, and R. Prevete, “A survey on modern trainable activation functions,”Neural Networks, vol. 138, pp. 14–32, 2021

2021
[8]

Sigmoid-weighted linear units for neural network function approximation in reinforcement learning,

S. Elfwing, E. Uchibe, and K. Doya, “Sigmoid-weighted linear units for neural network function approximation in reinforcement learning,” Neural networks, vol. 107, pp. 3–11, 2018

2018

Show all 13 references

[9]

Incor- porating second-order functional knowledge for better option pricing,

C. Dugas, Y . Bengio, F. B ´elisle, C. Nadeau, and R. Garcia, “Incor- porating second-order functional knowledge for better option pricing,” Advances in neural information processing systems, vol. 13, 2000

2000
[10]

Dynamic graph convolutional networks,

F. Manessi, A. Rozza, and M. Manzo, “Dynamic graph convolutional networks,”Pattern Recognition, vol. 97, p. 107000, 2020

2020
[11]

Darts: Differentiable architecture search,

H. Liu, K. Simonyan, and Y . Yang, “Darts: Differentiable architecture search,”arXiv preprint arXiv:1806.09055, 2018

2018 arXiv
[12]

Representation learning: A review and new perspectives,

Y . Bengio, A. Courville, and P. Vincent, “Representation learning: A review and new perspectives,”IEEE transactions on pattern analysis and machine intelligence, vol. 35, no. 8, pp. 1798–1828, 2013

2013
[13]

Categorical reparameterization with gumbel-softmax,

E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with gumbel-softmax,”arXiv preprint arXiv:1611.01144, 2016. Fig. 5. Ranking distribution showing the frequency of each rank position (1st through 7th) achieved by each activation function across all 18 architectures...

2016 arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.

[1] [1]

Using neural network for identifying clickbaits in online news media,

A. Omidvar, H. Jiang, and A. An, “Using neural network for identifying clickbaits in online news media,” inAnnual International Symposium on Information Management and Big Data. Springer, 2018, pp. 220–232

2018

[2] [2]

Deep sparse rectifier neural networks,

X. Glorot, A. Bordes, and Y . Bengio, “Deep sparse rectifier neural networks,” inProceedings of the fourteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2011, pp. 315–323

2011

[3] [3]

Rectifier nonlinearities improve neural network acoustic models,

A. L. Maas, A. Y . Hannun, A. Y . Nget al., “Rectifier nonlinearities improve neural network acoustic models,” inProc. icml, vol. 30, no. 1. Atlanta, GA, 2013, p. 3

2013

[4] [4]

Fast and accurate deep network learning by exponential linear units (elus),

D.-A. Clevert, T. Unterthiner, and S. Hochreiter, “Fast and accurate deep network learning by exponential linear units (elus),”arXiv preprint arXiv:1511.07289, vol. 4, no. 5, p. 11, 2015

arXiv 2015

[5] [5]

Self- normalizing neural networks,

G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter, “Self- normalizing neural networks,”Advances in neural information process- ing systems, vol. 30, 2017

2017

[6] [6]

Searching for activation functions,

P. Ramachandran, B. Zoph, and Q. V . Le, “Searching for activation functions,”arXiv preprint arXiv:1710.05941, 2017

arXiv 2017

[7] [7]

A survey on modern trainable activation functions,

A. Apicella, F. Donnarumma, F. Isgr `o, and R. Prevete, “A survey on modern trainable activation functions,”Neural Networks, vol. 138, pp. 14–32, 2021

2021

[8] [8]

Sigmoid-weighted linear units for neural network function approximation in reinforcement learning,

S. Elfwing, E. Uchibe, and K. Doya, “Sigmoid-weighted linear units for neural network function approximation in reinforcement learning,” Neural networks, vol. 107, pp. 3–11, 2018

2018