{"id":"671e9e9c-d7be-45b0-8636-733409aed10a","arxiv_id":"2510.22450","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Per-neuron activation functions selected with Gumbel-Softmax and frozen in a second training phase give competitive MNIST accuracy and reveal that deeper layers favor ELU/SELU.","lead":"SmartMixed lets each neuron pick its own activation function during training, then freezes those choices and keeps training a fast mixed network. On MNIST it ranks in the top three among seven activation strategies on most of 18 architectures, but does not beat a simple fixed Leaky ReLU.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SmartMixed's top-3 ranking may be an artifact of unequal training epochs: fixed baselines' epoch count is unspecified, and Phase 2's +350 epochs already boost accuracy by ~2.3%.","rationale":"The reader's weakest assumption focused on the straight-through estimator's gradient bias. That is a legitimate methodological concern, but it is not the single most load-bearing issue for the paper's strongest claim. The strongest claim is the empirical ranking: SmartMixed appears in the top three on 'the majority' of 18 architectures. This claim is invalid if the comparison is not compute-matched. The paper provides no epoch count for fixed baselines, while SmartMixed's 400 total epochs are explicit. The internal Phase1/Phase2 comparison shows that the extra 350 epochs alone account for a 2.3% accuracy gain, so a fixed ReLU or Leaky ReLU baseline trained for 400 epochs could plausibly rank above SmartMixed. This confound is directly checkable from the code, and it would fully explain the observed rankings without needing to invoke any failure of Gumbel-Softmax. The ST gradient bias is important for the 'learned preferences' interpretation, but even if the preferences are arbitrary, the rankings could still be an artifact of compute. Therefore, the epoch-count confound is the load-bearing concern. The reader did list this as one of several 'addressable issues' but not as the weakest assumption, so I mark partial agreement. The appropriate verdict remains CONDITIONAL: the paper should not be accepted as definitive until the compute-matching issue is resolved. My recommendation does not change the reader's conditional verdict, but it reframes the primary risk.","tokens_in":7543,"tokens_out":4276,"duration_ms":43655,"concrete_test":"Inspect the GitHub repository's training code to determine the exact epoch counts for fixed-activation baselines. Then, on at least three representative architectures (e.g., [784,512,256,128,10], [784,768,512,512,10], and [784,768,512,512,256,256,128,10]), retrain all seven models with identical total epochs, batch size, optimizer, learning-rate schedule, weight initialization, and 5 seeds. Compute test accuracies and re-rank. If SmartMixed still ranks in the top three for a majority of architectures under matched compute, the concern is resolved; if it does not, the central claim is unsupported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim—that SmartMixed ranks top-three among seven activation strategies on most of 18 architectures—rests on an uncontrolled comparison. Section IV-D says 'we trained seven separate models' and ranks them by test accuracy, but never states how many epochs the six fixed-activation baselines were trained. By contrast, SmartMixed is described in Section IV-B as training for α=50 epochs in Phase 1 plus 350 additional epochs in Phase 2, for a total of 400 epochs. Table I shows the internal Phase 1→Phase 2 improvement on the representative architecture: test accuracy jumps from 95.73% to 98.03% after the 350-epoch Phase 2. This demonstrates that simply continuing training for 350 epochs substantially improves accuracy for this mixed network; any fixed-activation network would likely also improve with 350 additional epochs. If the fixed baselines were trained for only 50 epochs, the Table II rankings are confounded by compute rather than by activation-function choice. Even if the baselines were trained for 400 epochs, the paper provides no evidence of matched learning-rate schedules, batch sizes, or other training details. This concern is more load-bearing than the Gumbel-Softmax gradient bias because it directly threatens the headline ranking result, independent of whether the learned selections are meaningful.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SmartMixed, a two-phase training scheme in which each hidden neuron learns a categorical choice over six activation functions (ReLU, Sigmoid, Tanh, Leaky ReLU, ELU, SELU) via a hard Gumbel-Softmax estimator. In Phase 1 the logits are trained for 50 epochs; in Phase 2 the argmax selection is frozen and the resulting mixed-activation network is trained for 350 more epochs. The authors evaluate the method on 18 fully connected MNIST architectures and claim that SmartMixed ranks in the top three among seven activation strategies in the majority of cases. They further claim a consistent layer-wise preference: early layers prefer ReLU/Leaky ReLU, deeper layers prefer ELU/SELU.","tokens_in":7887,"tokens_out":4320,"duration_ms":47527,"significance":"If the claims were established, the paper would provide a simple, practical method for per-neuron activation learning and an interesting empirical observation about layer-wise activation preferences. The strengths are that the method is easy to implement, the code is released, and the evaluation spans 18 architectures with a clear ranking protocol. However, the current evidence does not yet support the headline claims: the comparison against fixed-activation baselines is confounded by unequal training budgets, the results are single runs with no error bars, and there is no control to show that the learned selections, rather than the extra training epochs or the mere presence of mixed activations, cause the reported gains.","major_comments":[{"comment":"The central ranking claim rests on an uncontrolled comparison. Section IV-B states that SmartMixed trains for alpha=50 Phase-1 epochs plus 350 Phase-2 epochs, i.e. 400 epochs total, and Table I shows that the Phase-2 continuation alone raises test accuracy from 95.73% to 98.03%. Section IV-D says six fixed-activation models were trained but never states their epoch count. If those baselines were trained for 50 epochs, the Table II rankings are dominated by training budget rather than activation-function choice; if they were trained for 400 epochs, this must be stated along with matched schedules. The authors should report matched-budget results, including the Phase-1-only SmartMixed result, and provide the actual test accuracies for every architecture rather than ranks alone.","section":"IV-B, IV-D; Tables I and II"},{"comment":"All results appear to come from a single run per architecture with no random seeds, no error bars, and no significance testing. MNIST accuracies for these architectures are highly saturated, and small differences can flip ranks. The top-3 claim and the MRR ordering in Figure 4 are not meaningful without at least several seeds and a measure of variance. This is a load-bearing issue because the paper's main evidence is the ranking pattern across 18 architectures.","section":"IV-D; Figures 4 and 5"},{"comment":"The paper never demonstrates that the learned activation selections are the cause of the Phase-2 improvement. The straight-through gradient estimator in Eq. (3) gives a biased approximation, and the temperature parameter is never specified, so the logits may not converge to genuinely good choices. Moreover, Phase 2 could improve merely because the network is trained longer or because a mixture of activations is beneficial. The authors should add a control in which the same 400-epoch protocol is run with (a) randomly assigned per-neuron activations and (b) a uniformly mixed activation assignment. Without such a control, the 'learned preferences' narrative is unsupported.","section":"III-B and IV-B; Table I"},{"comment":"The layer-wise preference claim is based on one representative architecture in Figure 3, while the text asserts that the pattern is consistent across all 18 architectures without presenting the per-layer distributions for the remaining architectures. Additionally, the MRR analysis in Figure 4 shows Leaky ReLU as the best method and ReLU close behind SmartMixed, so the paper's language about 'optimal' per-neuron activation learning is too strong. The authors should provide a quantitative summary of activation distributions for all architectures and temper the claims accordingly.","section":"IV-C, IV-D; Figure 3 and Table II"}],"minor_comments":[{"comment":"The formula for the Gumbel noise, g = -log(-log(v+epsilon)+epsilon), is unusual; typically one writes g = -log(-log(v)) with a numerical-stability term applied carefully. Please clarify the exact expression and report the value of epsilon.","section":"III-B, Eq. (1)"},{"comment":"Hyperparameters for Phase 1 and Phase 2 are incomplete: optimizer, learning rate, batch size, weight initialization, and the Gumbel-Softmax temperature tau are not reported. The code may contain them, but the paper should state them for reproducibility.","section":"IV-B"},{"comment":"Table II gives only ranks, not accuracies. The statement that 'performance differences between most activation functions are relatively small' cannot be verified. Include the numeric test accuracy for each model.","section":"IV-D, Table II"},{"comment":"The claim that ELU-source neurons have more positive outgoing weights is based on a single heatmap without a color scale, error bars, or a statistical test. Please clarify what 'average connection weights' are averaged over and quantify the observation.","section":"IV-C and Figure 6"},{"comment":"Reference [10] is a dynamic graph convolutional network paper and does not appear to be about dynamic activation selection. Please cite an appropriate dynamic-selection activation method or rephrase the sentence.","section":"II, Related Work"}],"recommendation":"major_revision","confidential_remarks":"The paper is at the level of a workshop or short conference paper in its current form. The core idea is simple and potentially useful, but the empirical evidence needs to be substantially strengthened: matched training budgets, multiple seeds, accuracy tables, and a random-mixed-activation control. These are within the scope of a revision, so I do not recommend rejection, but the current manuscript is not ready for publication in a serious venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is an incremental but sensible method — per-neuron Gumbel-Softmax selection, then freeze the argmax and keep training. The new bit is the freeze-and-continue phase plus the claim that early layers want ReLU/Leaky ReLU and deep layers want ELU/SELU. That observation is the most interesting thing in the paper, and it holds up qualitatively for the one architecture they show in Figure 3.\n\nThey also do the right thing by shipping code and running 18 architectures. Credit for that.\n\nThe soft spots are real, and one is load-bearing. The paper never says how many epochs the six fixed-activation baselines were trained for. SmartMixed gets 50 + 350 = 400 epochs, and Table I shows that the extra 350 epochs alone lifts accuracy from 95.73% to 98.03%. If the baselines were only trained the equivalent of Phase 1, Table II's top-three rankings are a compute artifact. Even if the baselines also got 400 epochs, we have no evidence of matched schedules or batch size. This directly threatens the headline claim that SmartMixed 'competes' with fixed activations. The paper's own MRR figure puts Leaky ReLU and ReLU ahead of SmartMixed, so the adaptive method is not actually winning.\n\nThe second issue is the absence of seeds or error bars. On MNIST with small fully connected nets, the differences between top activation functions are often tiny; without variance, ranks are noise. The Gumbel-Softmax temperature is unspecified and the ST gradient bias is not validated, but I'd call that a minor concern compared to the epoch mismatch — the selections could still be meaningful even if the estimator is biased.\n\nThe layer-wise preference claim is framed as a discovery, but it's an empirical readout from the same trained models, and they show the distribution for only one architecture. They say 'experiments consistently revealed similar preferences' without showing the numbers. That's an assertion, not a result.\n\nNet: the idea is plausible, the paper is clearly written, and the code release is good. But the central empirical comparison needs to be re-run with matched compute and multiple seeds before I'd trust the ranking or the 'novel insight.' The layer-wise preference pattern is worth a quick replication; if it holds, that's a useful empirical note.\n\nFor peer review: I'd send it out. A competent referee can ask for the matching-epochs control and error bars, and the paper would be stronger for it. It's not a desk reject, but it's a long way from 'adaptive activation functions work better than fixed ones.'","headline":"Plausible two-phase activation-selection idea with a nice layer-wise preference observation, but the headline ranking is confounded by unequal training epochs and missing error bars.","tokens_in":8303,"tokens_out":2088,"would_cite":false,"duration_ms":22594,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SmartMixed lets each neuron learn and freeze its own activation function, ranking near the top across 18 MNIST architectures.","keywords":["activation functions","Gumbel-Softmax","two-phase training","per-neuron adaptation","layer-wise specialization","MNIST","straight-through estimator","feedforward networks"],"falsifier":"Retrain the same 18 architectures with Phase-1 selections replaced by random per-neuron activations, or by a deliberately reversed pattern (ELU/SELU in early layers, ReLU in deep layers). If these control versions match or beat SmartMixed, the learned selections are not carrying the performance; if SmartMixed clearly wins, the learned layer-wise choices are doing real work.","tokens_in":7442,"feed_emoji":"🧠","tokens_out":5524,"duration_ms":53160,"temperature":0.7,"pith_summary":"SmartMixed is a two-phase training strategy that lets every neuron in a feedforward network pick its own activation function from a pool of six candidates—ReLU, Sigmoid, Tanh, Leaky ReLU, ELU, SELU—instead of sharing one global activation. In the first phase, a hard Gumbel-Softmax estimator learns a categorical logit per neuron; after the logits stabilize, the argmax choice is frozen and the whole network trains further with ordinary vectorized operations. On 18 MNIST architectures, SmartMixed finishes in the top three of seven activation strategies on the majority of networks, trailing the best fixed choices like Leaky ReLU and ReLU by small margins. The paper's more striking finding is a consistent layer-wise pattern: early layers favor ReLU and Leaky ReLU, while deeper layers shift toward ELU and SELU. A sympathetic reader would take this as evidence that neuron-level activation specialization is learnable in practice and that depth systematically changes which activation a neuron prefers.","feed_headline":"Learned per-neuron activations hit top-3 on most of 18 MNIST nets","feed_subtitle":"Two-phase training freezes each neuron's chosen activation, beating fixed activations while staying inference-cheap.","key_machinery":"The load-bearing mechanism is the hard Gumbel-Softmax estimator: each neuron keeps a logit vector over the six activation candidates, adds Gumbel noise, softmax-normalizes at a temperature, then in the forward pass takes the one-hot argmax while in the backward pass replaces the discrete gradient with the soft relaxation via a stop-gradient. This straight-through trick is what makes discrete per-neuron selection differentiable. The second phase converts the learned logits into a fixed mixed network by taking argmax and grouping neurons by activation type so one affine transformation is shared. The work it does is to turn 'which activation for which neuron' into an optimizable categorical cho","core_discovery":"The central claim is that per-neuron activation functions can be learned without paying an inference-time cost: Phase 1 trains a hard one-hot mixture using a straight-through Gumbel-Softmax estimator; Phase 2 fixes each neuron's argmax activation and continues training, grouping neurons by chosen activation for vectorized efficiency. The paper reports that this two-phase procedure yields validation accuracy of 98.22% and test accuracy of 98.03% on a representative seven-hidden-layer network—about 2–3 points above the Phase-1 mixture—and places SmartMixed among the top three activation strategies on most of 18 architectures. It further claims that the learned selections are not arbitrary: acr","pith_inferences":["The paper leaves implicit that the same learn-then-freeze recipe could apply to other discrete architectural choices, such as per-layer normalization type or per-neuron quantization, suggesting a general 'learn and commit' strategy.","Because the Gumbel-Softmax temperature is never specified, the reported layer-wise preferences are partly a function of that annealing schedule; sweeping temperature and checking whether the ReLU-to-ELU shift persists would be a direct test.","The straight-through estimator's bias means the argmax at the transition epoch may be a noisy reflection of the true best activation; comparing against exhaustive per-neuron selection via leave-one-out retraining would show how much the learned choices matter.","On harder datasets than MNIST, the preference shift to ELU/SELU in deep layers might become more pronounced or reverse; that is an empirical question the paper does not address."],"forward_implications":["If per-neuron choices are genuinely learned, a single network can contain multiple activation types, and continued training after freezing improves accuracy by 2–3 points over the mixed selection phase.","The layer-wise preference is consistent: early layers favor ReLU and Leaky ReLU, deeper layers favor ELU and SELU, across 18 architectures of varying depth and width.","SmartMixed's inference cost equals that of a fixed-activation network because only one activation is evaluated per neuron, and grouping by activation type keeps operations vectorized.","No single fixed activation dominates; SmartMixed, Leaky ReLU, and ReLU occupy the top-3 slots on most architectures, while Sigmoid consistently underperforms."],"fun_headline_variants":["Per-neuron activations for free, top-3 on most MNIST nets","Neurons choose activations, freeze, top-3 on most MNIST nets","Two-phase trick: per-neuron activations, no inference cost","Free per-neuron activations, top-3 on most MNIST nets","Per-neuron activations, zero cost, top-3 on most MNIST nets"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Everything rests on the straight-through Gumbel-Softmax gradient being an accurate guide to each neuron's genuinely best activation; if the bias in that estimator leads logits to arbitrary values, Phase 1's argmax is meaningless and Phase 2 is just training a randomly mixed network.","fun_headline_variants_meta":{"raw":{"variants":["Per-neuron activations for free, top-3 on most MNIST nets","Neurons choose activations, freeze, top-3 on most MNIST nets","Two-phase trick: per-neuron activations, no inference cost","Free per-neuron activations, top-3 on most MNIST nets","Per-neuron activations, zero cost, top-3 on most MNIST nets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.004334,"raw_usage":{"total_tokens":15965,"prompt_tokens":723,"completion_tokens":15242,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":15138}},"tokens_in":467,"tokens_out":15242,"duration_ms":101226,"temperature":1.0,"reasoning_tokens":15138,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T08:03:45.265395+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the same 18 architectures with Phase-1 selections replaced by random per-neuron activations, or by a deliberately reversed pattern (ELU/SELU in early layers, ReLU in deep layers). If these control versions match or beat SmartMixed, the learned selections are not carrying the performance; if SmartMixed clearly wins, the learned layer-wise choices are doing real work.","supporting_citations":[],"review_version":1}