Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

(GG) MoE vs. MLP on Tabular Data

T0 review · 2 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Gumbel-gated mixture-of-experts with embeddings achieves the best average rank across 38 tabular datasets while using far fewer parameters than MLP.

desk verdict A transparent tabular-MoE study whose parameter-efficiency result is solid, but the 'highest performance' headline is an artifact of a lenient ranking rule and needs a paired significance test. read the letter →

arxiv 2502.03608 v1 pith:IOHH3AG3 submitted 2025-02-05 cs.LG cs.AI

Andrei Chernov This is my paper · ORCID
classification cs.LGcs.AI
keywords tabulardatamixtureofexpertsGumbel-Softmaxparameterefficiencymultilayerperceptronnumericalembeddingsgatingregularizationmodelensembling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a cheaper neural network can replace a vanilla MLP on tabular data without losing accuracy. It introduces GG MoE, a mixture-of-experts whose gating network uses Gumbel-Softmax instead of softmax, and compares it with standard MoE and MLP across 38 datasets. The central claim is that GG MoE with a piecewise-linear embedding layer reaches the best average rank, while both MoE variants use far fewer parameters than MLP. The paper itself notes that the accuracy gap over an MLP with embeddings is small enough that significance cannot be confidently claimed, so the credible contribution is a parameter-efficient alternative at roughly equal accuracy.

What carries the argument

The central object is GG MoE, a mixture-of-experts whose gating network replaces softmax with Gumbel-Softmax: $g_G(i \mid x, w_g) = \frac{\exp\left((w_{gi}^\top[x,1] + s_i)/\tau\right)}{\sum_{j=1}^K \exp\left((w_{gj}^\top[x,1] + s_j)/\tau\right)}$, where $s_i$ are i.i.d. Gumbel samples and $\tau$ is a temperature hyperparameter. The temperature controls the entropy of expert selection: low $\tau$ drives toward one-hot selection, high $\tau$ toward uniform weighting, so $\tau$ acts as a regularization dial that prevents the gating from collapsing onto a single expert. The empirical comparison uses the adopted benchmark setup with piecewise-linear numerical embeddings and the one-standard-deviation ranking rule, evaluating MLP, standard MoE, and GG MoE with and without embeddings over the same 38 datasets.

What would settle it

On a fixed subset of the 38 datasets, run a paired significance test (for example, corrected repeated k-fold cross-validation or Wilcoxon signed-rank with multiple-testing control) between GG E+MoE and E+MLP using the paper's tuned hyperparameters; if E+MLP wins or ties on average, the paper's accuracy claim fails and only the parameter-efficiency claim stands.

Watch

Extended reading notes

Core claim

The paper claims that a mixture-of-experts with Gumbel-Softmax gating and an embedding layer (GG E+MoE) achieves the best average rank across 38 tabular datasets compared with standard MoE, MLP, and embedded variants, while using substantially fewer parameters than MLP. The paper is careful to state that the gap over E+MLP is small enough that no confident significant difference can be declared, so the robust finding is parameter efficiency at comparable accuracy. A secondary claim is that embeddings help MoE models more than they help MLP, and that ten Monte Carlo samples suffice for GG MoE inference.

Load-bearing premise

The load-bearing premise is that the ranking rule counting any model within one standard deviation of the leader as tied is the right way to compare models, since the paper concedes that a stricter test would shrink the GG E+MoE versus E+MLP gap to a tie.

Editorial extensions

If this is right

  • GG E+MoE can replace a tuned MLP on tabular data when parameter budgets are tight, with little or no accuracy cost on these 38 datasets.
  • MoE and GG MoE use roughly an order of magnitude fewer parameters than MLP (median 0.08–0.16 million versus 0.88–1.19 million), lowering the cost of ensembling and scaling.
  • Ten Monte Carlo samples suffice for GG MoE inference, making the stochastic gating computationally cheap at deployment.
  • Piecewise-linear embeddings help MoE models more than they help MLP, so embeddings should be treated as part of the architecture rather than as preprocessing.
  • The learned temperature lands near 2 rather than near 0 or the upper bound, indicating the gating keeps all experts active instead of sparsifying.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the paper's own significance caveat makes an accuracy win over a tuned MLP unproven, the practical contribution is a parameter-efficient drop-in, not a new accuracy leader.
  • Editorial inference: the tuned temperatures cluster around 2, so the mechanism is likely soft, high-entropy expert mixing rather than sparse selection; a direct scan of $\tau$ beyond [0.5, 3] would test whether the optimal temperature is an artifact of the search bounds.
  • Editorial inference: the parameter savings suggest stacking many small experts is a cheaper route to MLP-style ensembles; testing GG MoE as an ensemble member rather than as a single model is a natural next experiment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes a mixture-of-experts model with a Gumbel-Softmax gating function ('GG MoE') for tabular data, alongside a standard MoE baseline, and compares both against tuned MLPs on 38 datasets, with and without piecewise-linear numerical embeddings. The authors report that GG E+MoE obtains the best average rank under the adopted ranking rule, that MoE and GG MoE use far fewer parameters than MLPs, that 10 Monte Carlo samples suffice for inference, and that training/inference times are comparable or better than E+MLP on the largest datasets.

Significance. If the claims are supported, the paper provides a useful empirical result: a Gumbel-Softmax-gated MoE with small experts can match tuned MLPs on tabular data at a fraction of the parameter count, which is relevant for scaling and ensembling. The evaluation protocol is transparent in several respects: 15 seeds, 100 Optuna iterations, full search-space tables, all MC sample counts, and parameter statistics are reported, and Section 6 honestly concedes the main weakness of the accuracy comparison. The central reservation is that the Abstract's superiority claim is stronger than the statistical evidence presented, and the parameter-efficiency comparison is shaped by an asymmetric search space.

major comments (2)
  1. [§6, Algorithm 1, Abstract] The headline accuracy claim is not supported by the evidence in its current form. Algorithm 1 (Section 5.4) groups models whose mean score is within one standard deviation of the leader, and Figure 1 then averages per-dataset ranks; Section 6 explicitly concedes that 'the performance gap between GG MoE and E+MLP is small that we cannot confidently declare a significant difference between them.' Under these conditions, the Abstract and Conclusion statements that GG E+MoE 'achieves the highest performance across 38 datasets' are stronger than what the evaluation can establish. The authors should add a paired per-dataset significance test (e.g., Wilcoxon signed-rank on per-dataset mean scores, or strict win/tie/loss counts) and either report the corrected p-values or revise the claim to 'comparable accuracy with lower parameter count'.
  2. [§5.3, Tables 1 and 4] The parameter-efficiency claim is confounded by the search-space restrictions in Table 1. MoE-type experts are limited to {32, 64} hidden units, while MLP block dimension is searched up to 1024; Section 5.3 states this restriction was imposed to 'encourage the use of multiple weak learners'. Consequently, the large parameter gap in Table 4 (e.g., median 0.34M for GG E+MoE vs 0.83M for E+MLP) is partly a consequence of a deliberately asymmetric search space, not a demonstrated architectural property. To make this claim load-bearing, the authors need a matched-capacity experiment or a parameter–accuracy trade-off analysis, and they should report whether the difference survives when search spaces are aligned.
minor comments (5)
  1. [§3.4, Eq. (7)] Equation (7) writes f(x;w) identically for all experts, which would make the mixture trivial; it should read f_i(x;w) or f(x;w_{e_i}), and the MC samples should multiply the corresponding expert outputs. Please correct the notation.
  2. [§4 and Appendix A] Section 4 says 28 datasets were taken from Grinsztajn et al. and 10 from Gorishniy et al., but Appendix A (Table 9) states the opposite; one of the two attributions is wrong and should be fixed.
  3. [§6] In the second bullet of Section 6, 'the performance gap ... is small that' should read 'is so small that'.
  4. [Figure 1] The panel labeled 'Categories' is not self-explanatory; please clarify the grouping of the three model families and make the legend consistent with the rank rows.
  5. [Algorithm 1] Algorithm 1's grouping condition is ambiguous unless the sort order is specified; please state explicitly that models are sorted by descending mean and that a model with mean µ_m is tied with the leader if µ_m ≥ µ_lead − σ_lead.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the benchmark claims are empirical and self-contained; evaluation-rule and search-space caveats are correctness concerns, not derivation loops.

full rationale

The paper's load-bearing claims are empirical benchmark outcomes, not derivations from fitted inputs. The GG MoE prediction rule in Eq. (7) is a Monte Carlo estimate of the expectation under the Gumbel-Softmax gating distribution; it is an honest stochastic forward pass, and the number of MC samples is varied (1, 5, 10, 100) and reported, with no parameter fitted to manufacture a target ranking. The temperature tau is tuned per dataset as a regularizer and is not presented as a predicted quantity; tuning a hyperparameter and then reporting the tuned value is not a circular prediction. The ranking procedure in Algorithm 1, which assigns equal rank to models within one standard deviation of the leader, is an evaluation rule adopted from Gorishniy et al. (2024); it affects how strongly the 'highest performance' claim can be stated, but it is not a derivation that reduces an output to an input. Section 6 explicitly concedes that 'the performance gap between GG MoE and E+MLP is small that we cannot confidently declare a significant difference between them,' which is a limitation on the strength of the accuracy claim, not a circular step. The parameter-efficiency comparison is shaped by the search-space restrictions (MoE expert sizes capped at {32, 64} hidden units while MLP widths search up to 1024), but this is an experimental-design and fairness concern, not circularity: the parameter counts are measured from the tuned models rather than defined as the claim. The paper contains no self-citations and imports no uniqueness theorem from the authors' prior work; external references are used for datasets, baselines, and experimental methodology, not as unverified load-bearing support for the paper's own conclusions. No circular step can be exhibited, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim is an empirical comparison, so the ledger is short. There are no invented entities: GG MoE recombines known components (linear gating, Gumbel-Softmax noise, MLP experts). The free parameters are tuned hyperparameters (expert count, temperature, MC sample count) that the protocol itself admits shape the result. The axioms are the standard unbiasedness of Gumbel-Softmax MC estimation, the adopted TabM ranking rule, and the author's search-space bounds, which are the most contestable choices in the paper.

free parameters (3)
  • Gumbel-Softmax temperature tau = tuned per dataset, U[0.5, 3]; median 2.06 to 2.21
    Section 3.4 and Table 5: tau selects the 'optimal level of uncertainty' in the gating; the GG variant's behavior depends entirely on this tuned knob.
  • Number of experts K = tuned per dataset, U{2,40} with step 1; median 16 to 20
    Table 1 and Table 3: K drives both the accuracy ranking and the parameter-efficiency comparison.
  • Monte Carlo sample count = 10
    Section 6: 10 samples chosen empirically as sufficient; the headline ranks for GG E+MoE use this count.
assumptions (4)
  • standard math Gumbel-Softmax samples provide unbiased Monte Carlo estimates of the gating expectation (Eq. 7).
    The inference-time estimate and the '10 samples suffice' claim in Section 6 rely on this standard property.
  • domain assumption Algorithm 1's one-standard-deviation tie rule is a valid way to rank models.
    The headline 'highest rank' for GG E+MoE is computed with this rule (Section 5.4); a stricter significance test would likely tie GG E+MoE with E+MLP, as Section 6 concedes.
  • ad hoc to paper The search-space bounds (expert size in {32,64}, temperature in [0.5,3]) constitute a fair comparison.
    Section 5.3 states these bounds were chosen to 'encourage the use of multiple weak learners' and to prevent argmax or uniform gating, which biases the comparison toward the MoE-style models.
  • domain assumption Observations are i.i.d. and evaluation splits are fixed per dataset.
    Section 3.1 states the probabilistic framework; train/validation/test splits come from the adopted benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of (GG) MoE vs. MLP on Tabular Data." pith.science (2026). https://pith.science/paper/IOHH3AG3

@misc{pith2026250203608,
  author       = {Pith},
  title        = {Pith review of: (GG) MoE vs. MLP on Tabular Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IOHH3AG3}},
  note         = {Machine review of arXiv:2502.03608}
}
abstract

In recent years, significant efforts have been directed toward adapting modern neural network architectures for tabular data. However, despite their larger number of parameters and longer training and inference times, these models often fail to consistently outperform vanilla multilayer perceptron (MLP) neural networks. Moreover, MLP-based ensembles have recently demonstrated superior performance and efficiency compared to advanced deep learning methods. Therefore, rather than focusing on building deeper and more complex deep learning models, we propose investigating whether MLP neural networks can be replaced with more efficient architectures without sacrificing performance. In this paper, we first introduce GG MoE, a mixture-of-experts (MoE) model with a Gumbel-Softmax gating function. We then demonstrate that GG MoE with an embedding layer achieves the highest performance across $38$ datasets compared to standard MoE and MLP models. Finally, we show that both MoE and GG MoE utilize significantly fewer parameters than MLPs, making them a promising alternative for scaling and ensemble methods.

Figures

Figures reproduced from arXiv: 2502.03608 by the authors.

Figure 1
Figure 1. The average rank for each model across 38 datasets. For each dataset, ranks were computed independently using Algo￾rithm 1. per dataset for each model.4 GG MoE and MoE models have approximately the same number of parameters, which is significantly lower than that of MLP models. This holds true for both models with and without embeddings. However, while the difference in the number of parameters between MLP and E+MLP… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Universal Embeddings of Tabular Data

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Rows of a table are embedded by training a graph auto-encoder on a table-derived weighted graph, giving smaller universal embeddings than EmbDI-style random-walk embeddings on two Kaggle datasets.

Reference graph

Works this paper leans on

13 extracted references · 2 canonical work pages · cited by 1 Pith paper

  1. [3]

    Tabm: Advancing tabular deep learning with parameter-efficient ensembling

    Gorishniy, Y ., Kotelnikov, A., and Babenko, A. Tabm: Advancing tabular deep learning with parameter-efficient ensembling. arXiv preprint arXiv:2410.24210,

    arXiv

  2. [9]

    Main Datasets characteristics. ID T ASKTYPE N NUMFEATURES NCATFEATURESTRAINSIZE VALSIZE TESTSIZE N BINFEATURES NCLASSES WINE BINCLASS 11 - 1787 230 537 - - PHONEME BINCLASS 5 - 2220 285 667 - - ANALCATDATASUPREME REGRESSION 2 - 2836 364 852 5 -MERCEDESBENZGREENERMANUFACTURING REGRESSION 0 3 2946 378 885 356 -KDDCUP09UPSELLING BINCLASS 34 14 3589 461 1078 ...

    work page 1979

  3. [10]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538,

    arXiv

  4. [11]

    B., and Goldstein, T

    Somepalli, G., Goldblum, M., Schwarzschild, A., Bruss, C. B., and Goldstein, T. Saint: Improved neural networks for tabular data via row attention and contrastive pre- training. arXiv preprint arXiv:2106.01342,

    arXiv

  5. [12]

    Batchensemble: an alterna- tive approach to efficient ensemble and lifelong learning

    Wen, Y ., Tran, D., and Ba, J. Batchensemble: an alterna- tive approach to efficient ensemble and lifelong learning. arXiv preprint arXiv:2002.06715,

    arXiv 2002

  6. [1991]

    Tab- transformer: Tabular data modeling using contextual em- beddings

    Huang, X., Khetan, A., Cvitkovic, M., and Karnin, Z. Tab- transformer: Tabular data modeling using contextual em- beddings. arXiv preprint arXiv:2012.06678,

    arXiv 2012

  7. [2016]

    TANGOS: Regularizing Tabular Neural Networks through Gradient Orthogonalization and Specialization

    Jeffares, A., Liu, T., Crabb ´e, J., Imrie, F., and van der Schaar, M. Tangos: Regularizing tabular neural net- works through gradient orthogonalization and special- ization. arXiv preprint arXiv:2303.05506,

    work page Pith review arXiv

  8. [2017]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,

    Loshchilov, I. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,

    arXiv

Show all 13 references
  1. [2018]

    From sparse to soft mixtures of experts

    Puigcerver, J., Riquelme, C., Mustafa, B., and Houlsby, N. From sparse to soft mixtures of experts. arXiv preprint arXiv:2308.00951,

    arXiv

  2. [2019]

    Scarf: Self- supervised contrastive learning using random feature cor- ruption

    Bahri, D., Jiang, H., Tay, Y ., and Metzler, D. Scarf: Self- supervised contrastive learning using random feature cor- ruption. arXiv preprint arXiv:2106.15147,

    arXiv

  3. [2020]

    Categorical repa- rameterization with gumbel-softmax

    Jang, E., Gu, S., and Poole, B. Categorical repa- rameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144,

    arXiv

  4. [2021]

    Better by default: Strong pre-tuned mlps and boosted trees on tabular data

    Holzm¨uller, D., Grinsztajn, L., and Steinwart, I. Better by default: Strong pre-tuned mlps and boosted trees on tabular data. arXiv preprint arXiv:2407.04491,

    arXiv

  5. [2022]

    A review of sparse expert models in deep learning

    Fedus, W., Dean, J., and Zoph, B. A review of sparse expert models in deep learning. arXiv preprint arXiv:2209.01667,

    arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.