Pith. sign in

REVIEW 3 major objections 5 minor 81 references

Softmax-gated Gaussian mixture-of-experts models can be estimated at parametric rates and selected consistently from a single over-specified fit by merging redundant experts along a dendrogram.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

For softmax-gated Gaussian mixtures of experts, merging duplicate fitted atoms along a dendrogram and choosing the level by a height-likelihood score consistently recovers the true number of experts at parametric rates without fitting multiple model sizes.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection A genuinely useful adaptation of dendrogram-based selection to SGMoE, but the key inverse bound rests on an unproven extended linear-independence claim; worth refereeing, needs that gap closed. the 3 major comments →

arxiv 2510.12744 v2 pith:IIDWB4LD submitted 2025-10-14 stat.ML cs.LGmath.STstat.COstat.MEstat.TH

Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps

Do Tien Hai , Trung Nguyen Mai , TrungTin Nguyen , Nhat Ho , Binh T. Nguyen , Christopher Drovandi This is my paper
classification stat.ML cs.LGmath.STstat.COstat.MEstat.TH MSC 62F1262G20
keywords softmax-gated mixture of expertsdendrogram of mixing measuresmodel selectionover-specified mixture modelsVoronoi lossconvergence ratesgate identifiabilityhierarchical aggregation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper attempts to solve three persistent obstacles in softmax-gated Gaussian mixture-of-experts (SGMoE) models: gate parameters are identifiable only up to translation, gate and expert terms are coupled through partial differential relations, and over-specified fits converge slowly. It introduces a fast-rate-aware Voronoi distance that detects redundant expert atoms, a softmax-weighted merge operator, and a dendrogram (aggregation path) over mixing measures. Along this path, the loss does not increase, and the maximum-likelihood estimator achieves parametric (log N / N)^{1/2} rates at every level, even when starting from an over-specified fit. The resulting height-likelihood selection rule, DSC, is consistent: it selects the true number of experts with probability tending to one, without sweeping over candidate sizes. If correct, this gives practitioners a single-fit workflow for model selection that is also less prone to contamination-driven overfitting than standard information criteria.

Core claim

The paper's central assertion is that the three structural obstacles of softmax-gated Gaussian mixtures—gate identifiability up to translation, PDE couplings between gate and expert derivatives, and the tight softmax numerator/denominator coupling—can be overcome simultaneously by organizing the fitted mixing measure into a hierarchy. The key discovery is a monotonicity lemma: the fast-rate-aware Voronoi distance D_FRA does not increase when the closest pair of atoms is softmax-merged, so along the aggregation path the MLE's D_FRA error stays at the parametric (log N / N)^{1/2} level even at exact- and under-fit levels. From this, the paper derives height and likelihood bounds along the path

What carries the argument

The load-bearing machinery is the fast-rate-aware Voronoi distance D_FRA (equation 6), which refines the classical over-fit Voronoi loss by adding five first-order 'merged-moment' block-sums inside each multi-covered cell (sums of weighted parameter deviations and cross-products). These block-sums vanish precisely when a cell's atoms behave like a single softmax-weighted aggregate, so D_FRA certifies where merging is beneficial. The companion softmax-weighted merge operator (equation 8) replaces the closest pair of atoms by their softmax-weighted aggregate, and iterating it produces the dendrogram (aggregation path) of mixing measures. The selection rule DSC = -(height + ε_N × average log-li

Load-bearing premise

The proofs of the central rates and of consistency rest on an extended linear-independence assumption: the basis functions x^{ℓ1} exp((ω^0_{1k}+t1)^⊤ x) ∂^{ℓ2} N(...) must be linearly independent for all exponent ranges up to 2 r̄(|A_k|), but the cited prior lemma (Fact 5) only establishes independence up to exponent 2, so the extension is asserted rather than proved; if it fails, the inverse bound Ex[D_TV(p_G,p_G0)] ≳ D_FRA(G,G0) and with it Theorems 1–4 collapse.

What would settle it

A concrete falsifier: exhibit a parameter configuration for some M≥4 where the polynomial system defining r̄(M) has a non-trivial solution with all p5j≠0 and some p3j≠0 for an exponent r smaller than the assumed r̄; or show, numerically, that the extended set W in the proof of Theorem 5 is linearly dependent for a specific G0—either would invalidate the lower bound and hence the consistency claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A practitioner can fit one over-specified SGMoE (say K=10 when K0=3), run the dendrogram, and select the number of experts without a K-grid search.
  • After merging, the estimated parameters converge to the truth at (log N / N)^{1/2} even if the initial over-specified fit was slow, because the merge collapses the multi-covered Voronoi cell directions.
  • The height-likelihood rule DSC is consistent in the frequentist sense: with probability tending to 1 it selects the true number of components, whenever K0 ≥ 2.
  • Under ε-contamination, DSC keeps selecting the true component count as N grows, while AIC/BIC/ICL overselect; the structural penalty on small heights provides misspecification robustness.
  • The dendrogram yields an interpretable hierarchy of fitted experts, which on the maize proteomics illustration selects two experts and exposes structure that BIC/ICL (one expert) and AIC (eighteen) miss.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same merge-dendrogram recipe is portable to other gated mixture families (e.g., Gaussian-gated or multinomial-logit experts), where the gate symmetries and PDE couplings differ; the main obstacle is computing the analogous algebraic exponent r̄.
  • The height penalty gives a generic topology-aware safeguard: information criteria that ignore atom geometry overselect under contamination, and adding a geometric separation term may fix that in mixture models generally.
  • The single-fit property could be exploited as an EM acceleration: run EM once on an over-specified model, merge along the dendrogram, and use the merged atoms as warm starts for the selected model—this would avoid the repeated EM runs that AIC/BIC/ICL require.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper develops a unified framework for softmax-gated Gaussian mixture of experts (SGMoE) covering parameter estimation and model selection. It introduces a fast-rate-aware Voronoi distance D_FRA (eq. (6)) that augments the existing over-specified loss with first-order merged-moment couplings, and a merging operator (eq. (8)) that collapses near-duplicate atoms. The authors prove monotonicity of D_FRA along the aggregation path (Lemma 1), derive finite-sample convergence rates for the MLE and the merged estimators (Theorem 1), control dendrogram heights and likelihood along the path (Theorems 2--3), and propose a height-likelihood criterion DSC that is claimed to consistently select the true number of experts without sweeping over K (Theorem 4). The main novelty is coupling rate-aware merging with consistent, sweep-free model selection for SGMoE. Simulations on synthetic and maize proteomics data corroborate the qualitative claims.

Significance. If the central theorems hold, the paper makes a valuable contribution to the MoE literature: it provides a constructive procedure that restores near-parametric rates for over-specified softmax-gated Gaussian mixtures, and a model-selection criterion that avoids multi-size training. The connection between polynomial solvability exponents and the rates along a dendrogram is conceptually interesting and empirically plausible. The paper also provides reproducible code-like algorithmic descriptions and extensive simulations, which strengthen the presentation. However, the significance is conditional on closing a key technical gap in the proof of the inverse bound (Theorem 5), as detailed below.

major comments (3)
  1. [Appendix D.3, Step 3 (set W and Fact 5)] The proof of the local inverse bound (eq. (17)) requires linear independence of the basis W containing functions x^{ℓ1} exp((ω0_{1k}+t1)^⊤ x) ∂^{ℓ2} N(...) for 0 ≤ |ℓ1|+ℓ2 ≤ 2r̄(|Ak|), plus gate terms up to |γ| ≤ 1+1{|Ak|>1}. The manuscript asserts this 'similarly to the proof of Fact 5 in Nguyen et al., 2023a', but Fact 5 (quoted as W1) only covers exponents ≤ 2. Since r̄(|Ak|) ≥ 4 when |Ak| > 1, the extended independence for exponents up to at least 8 is not established. Without it, the Fatou argument can yield nonzero limiting coefficients τ_{ℓ1,ℓ2}(k) or ξ_γ(k), so the contradiction in Step 3 fails. This gap directly undermines Theorem 5, and consequently Theorems 1--4 that rely on it. The authors should either supply a proof of the extended linear independence or cite a result that covers the full range.
  2. [Appendix D.2, Lemma 1 proof (convexity for (Δa, Δσ) block)] The proof of Lemma 1 explicitly shows the Jensen-based inequality for the (Δω1, Δb) block, but for the (Δa, Δσ) block it states 'Analogously' and writes the desired inequality. The merge update (8) defines a* and σ* as convex combinations of terms involving cross-products (ω1i−ω1*)(bi−b*) and quadratic terms (bi−b*)², not directly as convex combinations of (ai−a0, σi−σ0). The proof does not show that these cross/quadratic contributions are controlled by the D_FRA sums of the original atoms at the relevant scale. Since Lemma 1 underpins the extension of rates along the aggregation path (Theorem 1), this is a load-bearing gap that should be detailed.
  3. [Appendix D.6, Theorem 4 proof] The proof of Theorem 4 is a concise asymptotic-order sketch that compares DSC values using big-O notation and the positivity of KL divergences. It does not rigorously handle the stochastic dependence between the MLE and its dendrogram heights, nor the uniform control of likelihood deviations on the high-probability events needed to conclude convergence in probability. The statement 'We will give a more detailed proof below' is not followed by an actual detailed argument. While the high-level reasoning is plausible, a formal proof with explicit probabilistic events and rates is required for a theorem of this importance.
minor comments (5)
  1. [Abstract/Introduction] The phrase 'without model sweeps' is used repeatedly; it might be clarified that the method still requires fitting one over-specified model of order K, which is a single fit rather than a sweep over K. The distinction is clear from the text but could be stated explicitly in the abstract.
  2. [Appendix D.3, Fact 5 quote] There is a typo in the quoted Fact 5: 'linearly indeqendent' should be 'linearly independent', and 'w.r.tx and y' should be 'w.r.t. x and y' with periods and spaces.
  3. [Equation (6)] In the definition of D_FRA, the block-sum terms use norms without specifying the exponent; this is clear from context but a brief note that all norms in the block-sum terms are Euclidean 2-norms would prevent ambiguity.
  4. [Theorem 3 proof, Case 2] The proof of Case 2 (κ = K0) relies on Lemma 2 verifying Condition K, but the lemma is stated after the theorem proof. This organizational choice is acceptable, but a forward reference would improve readability.
  5. [Simulation section] The simulation figures and captions are informative, but the text could provide more detail on the precise numerical definition of 'loss' used in Figure 2 (it is stated as the Voronoi distance in eq. (6), but the exact normalization or metric details are not specified).

Circularity Check

1 steps flagged

Theorem chain leans on a higher-order linear-independence claim imported from a self-cited lemma that only covers order 2.

specific steps
  1. self citation load bearing [Appendix D.3, Proof of Theorem 5, Step 3 (after eq. (18), around Fact 5)]
    "Similarly to the proof of Fact 5 in Nguyen et al., 2023a: Fact 5 (Nguyen et al., 2023a, Lemma 2). The set W1 is linearly indeqendent w.r.t x and y ... the set W is linearly independent w.r.t x and y, it follows that τℓ1,ℓ2 (k) = ξγ(k) = 0"

    The inverse bound E_x[D_TV(p_G,p_G0)] ≳ D_FRA(G,G0) — which drives Theorems 1–4 — requires linear independence of W with exponents 0 ≤ |ℓ1|+ℓ2 ≤ 2r̄(|A_k|) (≥8 when |A_k|>1) and gate terms up to |γ| ≤ 1+1{|A_k|>1}. The paper's only support is 'similarly to' the self-cited Fact 5, but Fact 5 as stated covers only 0 ≤ |ℓ1|+ℓ2 ≤ 2 and |γ| ≤ 1. The higher-order independence is not proved and is not contained in the citation. The Fatou step then concludes τ=ξ=0 from this unproved independence, so the central lower bound and consistency theorem reduce to an unverified load-bearing self-citation.

full rationale

The paper's D_FRA and merge operator are compatible by design, but that compatibility is not circular: the block sums in eq. (6) are Taylor coefficients of the density discrepancy, and the merge identities in eq. (8) genuinely preserve these block sums, with Jensen's inequality providing the monotonicity in Lemma 1. The consistency of DSC is a corollary of the height and likelihood controls and is not itself a renamed input. The main circularity concern is the linear-independence step in Appendix D.3: the inverse bound requires linear independence of a basis W far beyond what the self-cited Fact 5 establishes, and the paper asserts the extension 'similarly' without proof. If that extended independence fails, the lower bound D_TV ≳ D_FRA fails and with it Theorems 1, 2, 3, and 4. This is a load-bearing self-citation rather than a definitional identity, so the appropriate score is 4 rather than higher.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The central claims rest on standard mixture-model assumptions plus prior algebraic results. The main hand-chosen ingredients are the DSC weight schedule and the dissimilarity used to define merge order. No new physical entities are introduced.

free parameters (2)
  • DSC weight schedule ε_N = ε_N = log N (practical choice; theory permits any 1 ≪ ε_N ≪ (N/log N)^{1/(2r̄)})
    The trade-off between height and likelihood in DSC is a hand-chosen schedule. Consistency holds for the whole range, but finite-sample behavior depends on the choice.
  • Rate-weighted dissimilarity coefficients in eq. (7) = 2 for ‖(ω1,b)‖ and 1 for ‖(a,σ)‖
    The merge order is defined by this hand-picked dissimilarity; no optimality or uniqueness is proven, and different choices yield different dendrograms.
axioms (4)
  • domain assumption SGMoE identifiability up to softmax translations (t0,t1), with continuous covariate distribution and compact Θ.
    Assumed in Section 2 and inherited from Proposition 1 of Nguyen et al. (2023a); needed for MLE consistency and for Voronoi cells to be meaningful.
  • domain assumption Data are generated exactly by an SGMoE of order K0 for all theoretical claims.
    The theorem statements and proofs assume i.i.d. samples from a fixed SGMoE with K0 experts; misspecification is only studied empirically.
  • ad hoc to paper Linear independence of the extended basis W for exponent range up to 2r̄(|A_k|).
    In D.3 Step 3, the inverse bound requires W to be linearly independent; the paper asserts this 'similarly' to Nguyen et al. (2023a) Fact 5, but Fact 5 as quoted only covers exponents up to 2. No proof of the extension is given.
  • standard math Known algebraic exponents r̄(2)=4, r̄(3)=6, r̄(M)≥7.
    These values are taken from Fact 1 (Nguyen et al. 2023a) and used throughout the rate statements; for M≥4 only lower bounds are available, which the paper acknowledges as a limitation.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps." pith.science (2026). https://pith.science/paper/IIDWB4LD

@misc{pith2026251012744,
  author       = {Pith},
  title        = {Pith review of: Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IIDWB4LD}},
  note         = {Machine review of arXiv:2510.12744}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We develop a unified statistical framework for softmax-gated Gaussian mixture of experts (SGMoE) that addresses three long-standing obstacles in parameter estimation and model selection: (i) non-identifiability of gating parameters up to common translations, (ii) intrinsic gate-expert interactions that induce coupled differential relations in the likelihood, and (iii) the tight numerator-denominator coupling in the softmax-induced conditional density. Our approach introduces Voronoi-type loss functions aligned with the gate-partition geometry and establishes finite-sample convergence rates for the maximum likelihood estimator (MLE). In over-specified models, we reveal a link between the MLE's convergence rate and the solvability of an associated system of polynomial equations characterizing near-nonidentifiable directions. For model selection, we adapt dendrograms of mixing measures to SGMoE, yielding a consistent, sweep-free selector of the number of experts that attains pointwise-optimal parameter rates under overfitting while avoiding multi-size training. Simulations on synthetic data corroborate the theory, accurately recovering the expert count and achieving the predicted rates for parameter estimation while closely approximating the regression function. Under model misspecification (e.g., $\epsilon$-contamination), the dendrogram selection criterion is robust, recovering the true number of mixture components, while the Akaike information criterion, the Bayesian information criterion, and the integrated completed likelihood tend to overselect as sample size grows. On a maize proteomics dataset of drought-responsive traits, our dendrogram-guided SGMoE selects two experts, exposes a clear mixing-measure hierarchy, stabilizes the likelihood early, and yields interpretable genotype-phenotype maps, outperforming standard criteria without multi-size training.

Figures

Figures reproduced from arXiv: 2510.12744 by Binh T. Nguyen, Christopher Drovandi, Do Tien Hai, Nhat Ho, Trung Nguyen Mai, TrungTin Nguyen.

Figure 1
Figure 1. Figure 1: Merging procedure from K = 10 to K = 3 of true mixing measure G0(3) with K0 = 3 components, defined in eq. (11). Maximum Likelihood Over At Most K Experts. When the true order K0 is unknown, we estimate within OK(Θ) := n G = X K′ k=1 exp(ω0k) δ(ω1k, ak, bk, σk) : 1 ≤ K′ ≤ K, (ω0k, ω1k, ak, bk, σk) ∈ Θ o . We analyze the exactly specified case K = K0, the over￾specified case K > K0, and the merging scheme u… view at source ↗
Figure 2
Figure 2. Figure 2: Convergence under three settings: (a) exact-fitted, (b) over-fitted, and (c) merged mixing measures. [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: DSC vs. AIC, BIC, and ICL for selecting K0 = 2 of G0. 0.0 0.2 0.4 0.6 0.8 1.0 X 5 0 5 10 15 y Clean Data Contaminated Data Expert 1 Mean Expert 2 Mean (a) Contaminated sample (n = 5000). 2.0 2.5 3.0 3.5 4.0 4.5 5.0 log10 (n) 0 20 40 60 80 100 P erce nta g e of correct tru e K0 DSC AIC BIC ICL (b) Proportion of correct selections. 2.0 2.5 3.0 3.5 4.0 4.5 5.0 log10 (n) 2.00 2.25 2.50 2.75 3.00 3.25 3.50 3.75… view at source ↗
Figure 4
Figure 4. Figure 4: Model selection under ϵ-contamination: DSC vs. AIC, BIC, and ICL with K0 = 2. After AIC, BIC, and ICL fail to recover K0, we evaluate DSC on 8 additional sample sizes between 5.5 × 104 and 105 . The results show that DSC recovers K0 with a high proportion. robust and recovers clustering structure under mild misspecification ( [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Dendrogram of mixing measure inferred from maize drought-responsive traits dataset. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Voronoi geometry and merge step for SGMoE. Multi-covered cells [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

81 extracted references · 7 linked inside Pith

  1. [1]

    Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control , 19(6), 716--723

    1974

  2. [2]

    Anandkumar, A., Hsu, D., & Kakade, S. M. (2012). A method of moments for mixture models and hidden markov models. In COLT

    2012

  3. [3]

    J., & Yu, B

    Balakrishnan, S., Wainwright, M. J., & Yu, B. (2017). Statistical guarantees for the EM algorithm: From population to sample-based analysis. Annals of Statistics , 45, 77--120

    2017

  4. [4]

    Bao, H., Wang, W., Dong, L., Liu, Q., Mohammed, O.-K., Aggarwal, K., Som, S., Piao, S., & Wei, F. (2022). VLMo : Unified vision-language pre-training with mixture-of-modality-experts. In Advances in Neural Information Processing Systems

    2022

  5. [5]

    Berrettini, M., Galimberti, G., Ranciati, S., & Murphy, T. B. (2024). Identifying Brexit voting patterns in the British house of commons: an analysis based on Bayesian mixture models with flexible concomitant covariate effects. Journal of the Royal Statistical Society Series C: Applied Statistics , 73(3), 621--638

    2024

  6. [6]

    Biernacki, C., Celeux, G., & Govaert, G. (2000). Assessing a mixture model for clustering with the integrated completed likelihood. IEEE Transactions on Pattern Analysis and Machine Intelligence , 22(7), 719--725

    2000

  7. [7]

    Blein-Nicolas, M., Devijver, E., Gallopin, M., & Perthame, E. (2024). Nonlinear network-based quantitative trait prediction from biological data. Journal of the Royal Statistical Society Series C: Applied Statistics , 73(3), 796--815

    2024

  8. [8]

    S., Balliau, T., Welcker, C., Cabrera-Bosquet, L., Nicolas, S

    Blein-Nicolas, M., Negro, S. S., Balliau, T., Welcker, C., Cabrera-Bosquet, L., Nicolas, S. D., Charcosset, A., & Zivy, M. (2020). A systems genetics approach reveals environment-dependent associations between snps, protein coexpression, and drought-related traits in maize. Genome Research , 30(11), 1593--1604

    2020

  9. [9]

    Chamroukhi, F., Samé, A., Govaert, G., & Aknin, P. (2009). Time series modeling by a regression approach based on a latent process. Neural Networks , 22(5–6), 593–602

    2009

  10. [10]

    Chen, J. (1995). Optimal Rate of Convergence for Finite Mixture Models . The Annals of Statistics , 23(1), 221 -- 233. Publisher: Institute of Mathematical Statistics

    1995

  11. [11]

    Chen, Z., Deng, Y., Wu, Y., Gu, Q., & Li, Y. (2022). Towards Understanding the Mixture -of- Experts Layer in Deep Learning . In A. H. Oh, A. Agarwal, D. Belgrave, & K. Cho (Eds.), Advances in Neural Information Processing Systems

    2022

  12. [12]

    C., Nguyen, H

    Chong, M. C., Nguyen, H. D., & TrungTin Nguyen (2024). Risk Bounds for Mixture Density Estimation on Compact Domains via the h- Lifted Kullback – Leibler Divergence . Transactions on Machine Learning Research

    2024

  13. [13]

    A., Terhorst, J., & Nguyen, X

    Do, D., Do, L., McKinley, S. A., Terhorst, J., & Nguyen, X. (2024). Dendrogram of mixing measures: Learning latent hierarchy and model selection for finite mixture models. arXiv preprint arXiv:2403.01684

    Pith/arXiv arXiv 2024

  14. [14]

    Do, D., Do, L., & Nguyen, X. (2025). Strong identifiability and parameter learning in regression with heterogeneous response. Electronic Journal of Statistics , 19(1), 131 -- 203. Publisher: Institute of Mathematical Statistics and Bernoulli Society

    2025

  15. [15]

    G., Le, H

    Do, T. G., Le, H. K., Nguyen, T., Pham, Q., Nguyen, B. T., Doan, T.-N., Liu, C., Ramasamy, S., Li, X., & HOI, S. (2023). HyperRouter : Towards Efficient Training and Inference of Sparse Mixture of Experts . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing Singapore: Association for Computational Linguistics

    2023

  16. [16]

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations

    2021

  17. [17]

    Doss, N., Wu, Y., Yang, P., & Zhou, H. H. (2023). Optimal estimation of high-dimensional Gaussian location mixtures. The Annals of Statistics , 51(1), 62 -- 95. Publisher: Institute of Mathematical Statistics

    2023

  18. [18]

    J., Jordan, M

    Dwivedi, R., Ho, N., Khamaru, K., Wainwright, M. J., Jordan, M. I., & Yu, B. (2020a). Sharp analysis of expectation-maximization for weakly identifiable models. AISTATS

  19. [19]

    J., Jordan, M

    Dwivedi, R., Ho, N., Khamaru, K., Wainwright, M. J., Jordan, M. I., & Yu, B. (2020b). Singularity, misspecification, and the convergence rate of EM . Annals of Statistics , 44, 2726--2755

  20. [20]

    Eigen, D., Ranzato, M., & Sutskever, I. (2014). Learning factored representations in a deep mixture of experts. In ICLR Workshops

    2014

  21. [21]

    Fedus, W., Zoph, B., & Shazeer, N. (2022). Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research , 23, 1--39

    2022

  22. [22]

    D., Nguyen, T., & Arbel, J

    Forbes, F., Nguyen, H. D., Nguyen, T., & Arbel, J. (2022a). Mixture of expert posterior surrogates for approximate Bayesian computation. In JDS 2022 - 53èmes Journées de Statistique de la Société Française de Statistique ( SFdS ) Lyon, France

    2022

  23. [23]

    D., Nguyen, T., & Arbel, J

    Forbes, F., Nguyen, H. D., Nguyen, T., & Arbel, J. (2022b). Summary statistics and discrepancy measures for approximate Bayesian computation via surrogate posteriors. Statistics and Computing , 32(5), 85

  24. [24]

    & Drton, M

    Foygel, R. & Drton, M. (2010). Extended Bayesian Information Criteria for Gaussian Graphical Models . In J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, & A. Culotta (Eds.), Advances in Neural Information Processing Systems , volume 23: Curran Associates, Inc

    2010

  25. [25]

    Frühwirth-Schnatter, S. (2019). Keeping the balance— Bridge sampling for marginal likelihood estimation in finite mixture, mixture of experts and Markov mixture models. Brazilian Journal of Probability and Statistics , 33(4), 706 -- 733

    2019

  26. [26]

    Frühwirth-Schnatter, S., Pamminger, C., Weber, A., & Winter-Ebmer, R. (2012). Labor market entry and earnings dynamics: Bayesian inference using mixtures-of-experts Markov chain clustering. Journal of Applied Econometrics , 27(7), 1116--1137

    2012

  27. [27]

    Frühwirth-Schnatter, S., Pittner, S., Weber, A., & Winter-Ebmer, R. (2018). Analysing plant closure effects using time-varying mixture-of-experts Markov chain clustering. The Annals of Applied Statistics , 12(3), 1796 -- 1830. Publisher: Institute of Mathematical Statistics

    2018

  28. [28]

    Genovese, C. R. & Wasserman, L. (2000). Rates of convergence for the Gaussian mixture sieve. The Annals of Statistics , 28(4), 1105 -- 1127

    2000

  29. [29]

    Guha, A., Ho, N., & Nguyen, X. (2021). On posterior contraction of parameters and interpretability in Bayesian mixture modeling. Bernoulli , 27(4), 2159 -- 2188

    2021

  30. [30]

    & Price, E

    Hardt, M. & Price, E. (2015). Tight bounds for learning a mixture of two gaussians. In STOC

    2015

  31. [31]

    & Kahn, J

    Heinrich, P. & Kahn, J. (2018). Strong identifiability and optimal minimax rates for finite mixture estimation. The Annals of Statistics , 46(6), 2844--2870

    2018

  32. [32]

    & Nguyen, X

    Ho, N. & Nguyen, X. (2016a). Convergence rates of parameter estimation for some weakly identifiable finite mixtures. The Annals of Statistics , 44(6), 2726 -- 2755

  33. [33]

    & Nguyen, X

    Ho, N. & Nguyen, X. (2016b). On strong identifiability and convergence rates of parameter estimation in finite mixtures. Electronic Journal of Statistics , 10(1), 271--307

  34. [34]

    Ho, N., Yang, C.-Y., & Jordan, M. I. (2022). Convergence Rates for Gaussian Mixtures of Experts . Journal of Machine Learning Research , 23(323), 1--81

    2022

  35. [35]

    A., Jordan, M

    Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991). Adaptive mixtures of local experts. Neural computation , 3(1), 79--87

    1991

  36. [36]

    & Tanner, M

    Jiang, W. & Tanner, M. A. (1999). On the identifiability of mixtures-of-experts. Neural Networks , 12(9), 1253--1258

    1999

  37. [37]

    Jordan, M. I. & Jacobs, R. A. (1994). Hierarchical mixtures of experts and the EM algorithm. Neural computation , 6(2), 181--214

    1994

  38. [38]

    Y., & Da, X

    Khalili, A., Yang, A. Y., & Da, X. (2024). Estimation and group-feature selection in sparse mixture-of-experts with diverging number of parameters. Journal of Statistical Planning and Inference , (pp.\ 106250)

    2024

  39. [39]

    Liang, H., Fan, Z., Sarkar, R., Jiang, Z., Chen, T., Zou, K., Cheng, Y., Hao, C., & Wang, Z. (2022). M ^3 ViT : Mixture -of- Experts Vision Transformer for Efficient Multi -task Learning with Model - Accelerator Co -design. In NeurIPS

    2022

  40. [40]

    Manole, T. & Ho, N. (2020). Uniform convergence rates for maximum likelihood estimation under two-component gaussian mixture models. arXiv preprint arXiv:2006.00704

    Pith/arXiv arXiv 2020

  41. [41]

    & Le Pennec, E

    Montuelle, L. & Le Pennec, E. (2014). Mixture of Gaussian regressions model with logistic weights, a penalized maximum likelihood approach. Electronic Journal of Statistics , 8(1), 1661--1695

    2014

  42. [42]

    Nguyen, D. N. & Li, Z. (2024). Joint learning of Gaussian graphical models in heterogeneous dependencies of high-dimensional transcriptomic data. In The 16th Asian Conference on Machine Learning ( Conference Track )

    2024

  43. [43]

    Nguyen, H., Akbarian, P., Nguyen, T., & Ho, N. (2024a). A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts . In Proceedings of The 41st International Conference on Machine Learning

  44. [44]

    Nguyen, H., Nguyen, T., & Ho, N. (2023a). Demystifying Softmax Gating Function in Gaussian Mixture of Experts . In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Advances in Neural Information Processing Systems , volume 36 (pp.\ 4624--4652).: Curran Associates, Inc

  45. [45]

    Nguyen, H., Nguyen, T., Nguyen, K., & Ho, N. (2024b). Towards Convergence Rates for Parameter Estimation in Gaussian -gated Mixture of Experts . In S. Dasgupta, S. Mandt, & Y. Li (Eds.), Proceedings of The 27th International Conference on Artificial Intelligence and Statistics , volume 238 of Proceedings of Machine Learning Research (pp.\ 2683--2691).: PMLR

  46. [46]

    Nguyen, H. D. & Chamroukhi, F. (2018). Practical and theoretical aspects of mixture-of-experts modeling: An overview. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery , 8(4), e1246

    2018

  47. [47]

    D., Chamroukhi, F., & Forbes, F

    Nguyen, H. D., Chamroukhi, F., & Forbes, F. (2019). Approximation results regarding the multiple-output Gaussian gated mixture of linear experts model. Neurocomputing , 366, 208--214

    2019

  48. [48]

    D., Lloyd-Jones, L

    Nguyen, H. D., Lloyd-Jones, L. R., & McLachlan, G. J. (2016). A universal approximation theorem for mixture-of-experts models. Neural computation , 28(12), 2585--2593

    2016

  49. [49]

    Nguyen, H. D. & Nguyen, T. (2025). Modifications of the BIC for order selection in finite mixture models. 2506.20124

    arXiv 2025

  50. [50]

    D., Nguyen, T., Chamroukhi, F., & McLachlan, G

    Nguyen, H. D., Nguyen, T., Chamroukhi, F., & McLachlan, G. J. (2021a). Approximations of conditional probability density functions in Lebesgue spaces via mixture of experts models. Journal of Statistical Distributions and Applications , 8(1), 13

  51. [51]

    D., Nguyen, T., Westerhout, J., & Guo, X

    Nguyen, H. D., Nguyen, T., Westerhout, J., & Guo, X. (2025). Approximation rates for finite mixtures of location-scale models. arXiv preprint arXiv:2508.10612

    Pith/arXiv arXiv 2025

  52. [52]

    Nguyen, T. (2021). Model Selection and Approximation in High -dimensional Mixtures of Experts Models : from Theory to Practice . PhD Thesis , Normandie Université

    2021

  53. [53]

    D., & Forbes, F

    Nguyen, T., Chamroukhi, F., Nguyen, H. D., & Forbes, F. (2021b). Non-asymptotic model selection in block-diagonal mixture of polynomial experts models. Preprint. arXiv:2104.08959

    Pith/arXiv arXiv

  54. [54]

    D., & Forbes, F

    Nguyen, T., Chamroukhi, F., Nguyen, H. D., & Forbes, F. (2022a). Model selection by penalization in mixture of experts models with a non-asymptotic approach. In JDS 2022 - 53èmes Journées de Statistique de la Société Française de Statistique ( SFdS ) Lyon, France

    2022

  55. [55]

    D., & McLachlan, G

    Nguyen, T., Chamroukhi, F., Nguyen, H. D., & McLachlan, G. J. (2023b). Approximation of probability density functions via location-scale finite mixtures in Lebesgue spaces. Communications in Statistics - Theory and Methods , 52(14), 5048--5059

  56. [56]

    Nguyen, T., Forbes, F., Arbel, J., & Duy Nguyen, H. (2024c). Bayesian nonparametric mixture of experts for inverse problems. Journal of Nonparametric Statistics , (pp.\ 1--60)

  57. [57]

    N., Nguyen, H

    Nguyen, T., Nguyen, D. N., Nguyen, H. D., & Chamroukhi, F. (2023c). A non-asymptotic theory for model selection in high-dimensional mixture of experts via joint rank and variable selection. In AJCAI Australasian Joint Conference on Artificial Intelligence 2023

    2023

  58. [58]

    D., Chamroukhi, F., & Forbes, F

    Nguyen, T., Nguyen, H. D., Chamroukhi, F., & Forbes, F. (2022b). A non-asymptotic approach for model selection via penalization in high-dimensional mixture of experts models. Electronic Journal of Statistics , 16(2), 4742 -- 4822

  59. [59]

    D., Chamroukhi, F., & McLachlan, G

    Nguyen, T., Nguyen, H. D., Chamroukhi, F., & McLachlan, G. J. (2020). Approximation by finite mixtures of continuous density functions that vanish at infinity. Cogent Mathematics & Statistics , 7(1), 1750861

    2020

  60. [60]

    D., Chamroukhi, F., & McLachlan, G

    Nguyen, T., Nguyen, H. D., Chamroukhi, F., & McLachlan, G. J. (2023d). Non-asymptotic oracle inequalities for the Lasso in high-dimensional mixture of experts. arXiv:2009.10622

    Pith/arXiv arXiv 2009

  61. [61]

    Nguyen, X. (2013). Convergence of latent mixing measures in finite and infinite mixture models. The Annals of Statistics , 41(1), 370--400

    2013

  62. [62]

    Norets, A. (2010). Approximation of conditional densities by smooth mixtures of regressions. The Annals of Statistics , 38(3), 1733 -- 1766

    2010

  63. [63]

    A., & Tanner, M

    Peng, F., Jacobs, R. A., & Tanner, M. A. (1996). Bayesian Inference in Mixtures -of- Experts and Hierarchical Mixtures -of- Experts Models With an Application to Speech Recognition . Journal of the American Statistical Association , 91(435), 953--960

    1996

  64. [64]

    T., Ramasamy, S., Li, X., Hoi, S., & others (2024)

    Pham, Q., Do, G., Nguyen, H., Nguyen, T., Liu, C., Sartipi, M., Nguyen, B. T., Ramasamy, S., Li, X., Hoi, S., & others (2024). CompeteSMoE – Effective Training of Sparse Mixture of Experts via Competition . arXiv preprint arXiv:2402.02526

    Pith/arXiv arXiv 2024

  65. [65]

    A., Cabrera-Bosquet, L., Grau, A., Coupel-Ledru, A., Millet, E

    Prado, S. A., Cabrera-Bosquet, L., Grau, A., Coupel-Ledru, A., Millet, E. J., Welcker, C., & Tardieu, F. (2018). Phenomics allows identification of genomic regions affecting maize stomatal conductance with conditional effects of water deficit and evaporative demand. Plant, Cell & Environment , 41(2), 314--326

    2018

  66. [66]

    Rakhlin, A., Panchenko, D., & Mukherjee, S. (2005). Risk bounds for mixture density estimation. ESAIM: PS , 9, 220--229

    2005

  67. [67]

    Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics , 6(2), 461--464

    1978

  68. [68]

    T., & Ghosal, S

    Shen, W., Tokdar, S. T., & Ghosal, S. (2013). Adaptive Bayesian multivariate density estimation with Dirichlet mixtures. Biometrika , 100(3), 623--640

    2013

  69. [69]

    & White, H

    Sin, C.-Y. & White, H. (1996). Information criteria for selecting possibly misspecified parametric models. Journal of Econometrics , 71(1), 207--225

    1996

  70. [70]

    Sturmfels, B. (2002). Solving systems of polynomial equations . Number 97 in CBMS Regional Conference Series in Mathematics . American Mathematical Soc

    2002

  71. [71]

    Thai, T., Nguyen, T., Do, D., Ho, N., & Drovandi, C. (2025). Model Selection for Gaussian -gated Gaussian Mixture of Experts Using Dendrograms of Mixing Measures . arXiv preprint arXiv:2505.13052

    Pith/arXiv arXiv 2025

  72. [72]

    van de Geer, S. (2000). Empirical Processes in M -estimation , volume 6. Cambridge university press

    2000

  73. [73]

    Villani, C. (2003). Topics in optimal transportation , volume 58 of Graduate Studies in Mathematics . American Mathematical Society

    2003

  74. [74]

    Villani, C. (2009). Optimal transport: old and new , volume 338. Springer

    2009

  75. [75]

    Westerhout, J., Nguyen, T., Guo, X., & Nguyen, H. D. (2024). On the Asymptotic Distribution of the Minimum Empirical Risk . In Forty-first International Conference on Machine Learning

    2024

  76. [76]

    & Yang, P

    Wu, Y. & Yang, P. (2020). Optimal estimation of Gaussian mixtures via denoised method of moments. The Annals of Statistics , 48, 1987--2007

    2020

  77. [77]

    & Zhou, H

    Wu, Y. & Zhou, H. H. (2021). Randomly initialized EM algorithm for two-component G aussian mixture achieves near optimality in o( n ) iterations. Mathematical Statistics and Learning , 4, 143–220

    2021

  78. [78]

    You, Z., Feng, S., Su, D., & Yu, D. (2021). Speechmoe: Scaling to large acoustic models with dynamic routing mixture of experts. In Interspeech

    2021

  79. [79]

    You, Z., Feng, S., Su, D., & Yu, D. (2022). Speechmoe2: Mixture-of-experts model with improved routing. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics , Speech and Signal Processing ( ICASSP ) (pp.\ 7217--7221)

    2022

  80. [80]

    E., Wilson, J

    Yuksel, S. E., Wilson, J. N., & Gader, P. D. (2012). Twenty Years of Mixture of Experts . IEEE Transactions on Neural Networks and Learning Systems , 23(8), 1177--1193

    2012

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.