CoMoE: Contrastive Representation for Mixture-of-Experts in Parameter-Efficient Fine-tuning

Chaopeng Wei; Jinyuan Feng; Tenghai Qiu; Tianyi Hu; Zhiqiang Pu

REVIEW 4 major objections 4 minor 1 cited by

CoMoE: Contrastive Representation for Mixture-of-Experts in Parameter-Efficient Fine-tuning

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CoMoE adds an auxiliary contrastive objective to mixture-of-experts fine-tuning and claims it recovers the mutual-information gap between inputs and activated versus inactivated experts, improving expert specialization and multi-task…

desk verdict A simple contrastive loss for expert specialization in MoE-LoRA, but the InfoNCE-based theory is invalid and the method is too close to prior work to stand on the current empirical claims. read the letter →

arxiv 2505.17553 v2 pith:Q6PZTPQB submitted 2025-05-23 cs.LG cs.CL

Jinyuan Feng , Chaopeng Wei , Tenghai Qiu , Tianyi Hu , Zhiqiang Pu This is my paper

classification cs.LGcs.CL

keywords mixture-of-expertsparameter-efficientfine-tuningcontrastivelearningmutualinformationexpertspecializationtop-kroutinglow-rankadaptationmulti-task

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes CoMoE, an auxiliary training objective for mixture-of-experts (MoE) parameter-efficient fine-tuning. Its central claim is that expert under-specialization, where experts learn overlapping knowledge, can be fixed by a contrastive loss that pulls each input token's representation toward the experts the router activated and pushes it away from the experts it did not. The paper defines a "mutual-information gap" between inputs and the two expert groups and proves a lower bound relating that gap to the InfoNCE loss. If the bound holds, the method recovers a principled estimate of expert specialization, and the reported experiments show consistent gains on heterogeneous multi-task benchmarks and visible separation of expert workloads without any explicit load-balance loss. The practical payoff would be more modular, higher-capacity MoE fine-tuning that uses roughly half the trainable parameters of standard low-rank-adaptation baselines.

What carries the argument

The central object is the MI gap $\Delta I = I_{\mathrm{top}\text{-}k}(x, M^{+}) - I_{\mathrm{top}\text{-}k}(x, M^{-})$: the difference in mutual information between an input token and the experts the top-$k$ router activates versus those it leaves inactive. The paper approximates this gap with an InfoNCE contrastive loss in which a random activated expert's representation is the query, the remaining activated experts are positive keys, and inactivated experts are negative keys, with an exponential cosine similarity as the score function. The auxiliary contrastive term enters the fine-tuning loss as $L_{\mathrm{total}} = L_{\mathrm{CE}} + \lambda L_{\mathrm{contrast}}$. The mechanism works by pushing each expert's representation to be distinctive for the tokens that route to it while suppressing its response to tokens that do not, which the paper argues recovers the MI gap and yields modular specialists.

What would settle it

Run CoMoE on a synthetic MoE with known expert distributions and estimate the true mutual-information gap by Monte Carlo; if $\Delta I$ does not track $\log(N) - L_{\mathrm{NCE}}$ as $N$ grows under same-token negatives, Theorem 1 as stated fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that a top-$k$ routed MoE can be trained so that the activated experts for a token carry high mutual information with that token while the inactivated experts carry low mutual information. This MI gap is captured by a single InfoNCE-style contrastive loss that uses the activated experts' representations as positives and the inactivated experts' representations as negatives. Theorem 1 states that $\Delta I \ge \log(N) - L_{\mathrm{NCE}}$, where $N$ is the number of negative samples from inactivated experts, so minimizing the contrastive loss maximizes a lower bound on the MI gap. CoMoE adds this objective, weighted by a hyperparameter $\lambda$, to the standard cross-entropy fine-tuning loss; no pretraining or architecture change is needed. Experiments on commonsense reasoning, math reasoning, and multi-task settings with 7B and 2B backbones report average accuracy gains, roughly 50% parameter reduction versus low-rank-adaptation baselines, and task-differentiated expert workloads that emerge without a load-balance loss.

Load-bearing premise

The proof of Theorem 1 assumes the standard InfoNCE bound still applies when the negative samples are the inactivated experts of the same input token, i.e., drawn from the conditional distribution $p(e \mid x)$, rather than independently from the marginal distribution of expert representations; the paper does not justify that substitution.

Editorial extensions

If this is right

CoMoE can be dropped into any top-$k$ routed MoE parameter-efficient fine-tuning method as an auxiliary loss, with no pretraining stage and no architectural change.
The method is claimed to reduce trainable parameters by about half relative to standard low-rank-adaptation baselines while keeping or improving accuracy.
Reported expert workloads become task-specific without any load-balance loss, and expert representations become visibly more separated in the paper's visualizations.
Placing CoMoE in lower transformer layers helps more than placing it in middle or upper layers, suggesting early-layer diversity drives the gain.
With fixed-size negative sampling, training cost stays $O(1)$ in the number of experts, with no reported loss in accuracy in the appendix.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's own experiments do not separate the mutual-information mechanism from a simpler explanation: the loss explicitly repels representations of experts that are alternatives for the same token, so a non-information-theoretic distance loss might replicate the gains; this is testable.
If the InfoNCE bound fails under same-token negatives, the reported improvements could still hold as an empirical regularization effect rather than as evidence for the MI-gap story.
The layer-wise finding suggests that applying CoMoE only to early layers, or combining it with depth-wise routing, could recover most of the benefit at lower cost; the paper does not run that configuration.
The same contrastive principle could extend to non-MoE multi-adapter fine-tuning, where overlapping adapter knowledge is a known failure mode, but the paper does not explore that setting.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the core theory is broken and the novelty is partially anticipated, but there is a usable empirical recipe here. The paper defines an MI gap between activated and inactivated experts in MoE-LoRA, then adds an InfoNCE-style loss that treats activated experts as positives and inactivated experts as negatives. The method itself is simple, and the authors do a lot of work to characterize it: ablations on the number of experts, layer-wise placement, backbone generalization, efficiency, and load-balance visualizations. The fixed-size negative sampling trick that keeps training cost O(1) regardless of expert count is a genuinely useful practical detail.

The problem is Theorem 1. The InfoNCE bound only holds when negatives are sampled from the marginal p(e), but Algorithm 1 builds the negative set from the same input x, i.e. from p(e|x). The appendix proof silently substitutes the conditional for the marginal, and the Jensen step is applied without checking the direction. So the claimed lower bound ΔI ≥ log N − L_NCE has no valid derivation. This is not a minor gap; it is the paper's central justification for the loss. The authors may still have a working trick, but the information-theoretic argument collapses.

There is also a novelty problem. MoELoRA (Luo et al., 2024) already uses contrastive learning in MoE-based PEFT, and SCMoE (Shi et al., 2024) uses unchosen experts as negative contrasts. Both are cited, but neither is in the experimental comparisons. The reader of this paper cannot tell what CoMoE adds beyond those.

The empirics are broad but not airtight. The math gains in Table 3 are spectacular, especially Arithmetic from 2.69 to 80.9, with no explanation. Hyperparameter λ is tuned on the same tasks that later report results, and the main tables have no error bars (the appendix says median over five seeds, but we don't see the spread). The workload visualizations are illustrative but not a quantitative metric.

The paper is worth a skim for anyone working on MoE PEFT, and the fixed-size sampling idea might be worth reusing. But as a submission, the load-bearing theory is invalid and the comparison set omits the most relevant baselines. I would not send this to review in its current form. A shortened empirical paper without the MI claim, with MoELoRA and SCMoE as baselines, would be a more honest submission.

Referee Report

4 major / 4 minor

Summary. The paper proposes CoMoE, an auxiliary contrastive loss for Mixture-of-Experts in parameter-efficient fine-tuning. Under top-k routing, activated experts are treated as positive samples and inactivated experts as negative samples. The authors define a mutual-information gap between the input and the activated versus inactivated experts, claim that the InfoNCE objective lower-bounds this gap (Theorem 1, Eq. 9), and report improvements on commonsense reasoning and math benchmarks with LLaMA-2 7B and Gemma 2B.

Significance. If the theoretical claim were established, the paper would offer a simple, portable auxiliary objective for encouraging expert specialization in LoRA-based MoE, and the experimental scope is reasonably broad: multiple reasoning benchmarks, ablations on the loss weight λ, the number of experts, layer placement, and visualizations of workload balance. However, the theoretical result is the central justification of the method, and the proof rests on an invalid application of the standard InfoNCE bound. The empirical evidence is also weakened by the absence of variance reporting and by unexplained, very large gains on math tasks. Under the stated theory the contribution is not established; at best CoMoE is an ungrounded heuristic auxiliary loss.

major comments (4)

[Section 4.3, Theorem 1 (Eq. 9), Appendix A, Algorithm 1] The InfoNCE lower bound is applied to negative samples that are not drawn from the marginal distribution p(e^-). Standard InfoNCE requires the negatives to be i.i.d. draws from the marginal p(e); here Algorithm 1 constructs the negative set as N = {Normalize(E_i(x)) | i ∉ T}, i.e., inactivated experts evaluated on the same token that produced the query. These negatives are distributed according to p(e^-|x), not p(e^-). Under conditional sampling, E_{e^-~p(e^-|x)}[p(e^-|x)/p(e^-)] = 1 + χ²(p(e^-|x)||p(e^-)), which is not 1, so the density-ratio expectation used in the standard derivation, E[f(x,e^-)/f(x,e^+)] = p(e^+)/p(e^+|x), is no longer valid. The paper neither states nor proves a modified bound for conditional negatives, and the algebra in Appendix A silently substitutes conditional densities for marginal densities. Consequently, the claimed inequality ΔI ≥ log(N) − L_NCE in Eq. (9) is unsupported.
[Appendix A, Eqs. (15) and (16)] Independently of the sampling distribution, the derivation does not establish the equality L_NCE = log N − ΔI. Starting from L_NCE = E log[1 + Σ_i f(x,e_i^-)/f(x,e^+)], Jensen's inequality gives an upper bound on L_NCE, not the equality used in the proof, and the step that converts the logarithm of a sum into a sum of logarithms is not an identity. The standard InfoNCE result lower-bounds the mutual information I(x;e^+); it does not produce a mutual-information gap I(x;e^+) − I(x;e^-) of the form defined in Definition 1. Even after fixing the negative-sampling issue, the stated theorem would require a separate argument that the proof does not provide.
[Section 5.3, Fig. 3; Appendix D] The hyperparameter λ is selected using the evaluation accuracy shown in Fig. 3, and the paper reports only the median of five seeds with no variance or confidence intervals. Because λ is tuned on the test tasks and several reported gains are modest (e.g., the +1.3 average in Table 1), the empirical claim that CoMoE 'consistently' improves performance is not supported by the reported evidence. The authors should report all-seed results, use a held-out validation split for hyperparameter selection, and include error bars in the main tables.
[Table 3] The math-reasoning results report a change from 2.69 to 80.9 on Arithmetic and from 17.6 to 31.4 on AQuA with no explanation. The baseline in Table 3 is LoRA rather than an MoE method, and the LoRA numbers are reproduced from (Luo et al., 2024) without confirming that the training/evaluation protocols are identical to the CoMoE runs. These magnitudes are implausible for a small auxiliary loss and undermine the generalization claim; controlled comparisons with identical settings and variance estimates are needed.

minor comments (4)

[Abstract] The abstract contains the typo 'CoMogB'; this should read 'CoMoE'.
[Appendix D] The statement that final results are the median over five random seeds should be accompanied by the full distribution or at least standard deviations; median-only reporting is insufficient for the claimed gains.
[References] Several bibliographic entries contain formatting errors, for example 'arXiv:2404, 15159' and 'arXiv:2310.185339'; these should be corrected.
[Section 5.3] Figure 3 does not state whether the accuracy values are on a validation split or the test split; this distinction is important because λ is chosen from these curves.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the MI-gap theorem is an application of the external InfoNCE bound, and the only self-citation is motivational rather than load-bearing.

full rationale

CoMoE's derivation chain is not circular. The paper defines an MI gap (Definition 1) from mutual-information terms and then invokes the standard InfoNCE bound (Oord et al., 2018) to relate a contrastive objective to that gap (Theorem 1, Eq. 9; proof in Appendix A). The target quantity ΔI is defined independently of the loss, so the theorem is not true by construction. The one self-citation (Feng et al., 2025, OMoE) motivates the claim that vanilla MoE lacks specialization, but that premise is also supported by in-paper workload and representation analyses (Figs. 4 and 5), so the citation is not load-bearing. The main potential problem—that negative samples are drawn from inactivated experts evaluated on the same input, i.e., from a conditional rather than marginal distribution—would be a correctness/validity defect in the InfoNCE application, not a circularity: the conclusion does not reduce to an input or fitted parameter. Experimental comparisons are against external benchmarks and baselines, and the auxiliary-loss coefficient λ is swept rather than fitted to a subset and then reported as a prediction. Accordingly, no step satisfies the quoted-reduction test for circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or architectural entities. It depends on standard contrastive-learning assumptions that are not justified in the same-input negative setting, plus hyperparameters selected on the evaluation tasks.

free parameters (3)

λ (contrastive loss weight) = 0.01
Tuned on BoolQ, ARC-e, ARC-c, and OBQA in Figure 3, which are the same tasks reported in Table 1; this is selection on the evaluation data.
Temperature τ
Hyperparameter in Eq. 11 for the exponential similarity function; chosen by hand and not justified.
Fixed negative sampling size
Appendix H introduces constant-size negative sampling to reduce O(n) training time; the sample size is a hyperparameter that trades bound tightness for speed.

assumptions (3)

domain assumption The standard InfoNCE bound holds with negative samples drawn from the same input token's inactivated experts
Section 4.3 and Appendix A invoke InfoNCE, but the standard bound requires negatives from the marginal p(e), not from p(e|x) of the same input; no justification is given for this substitution.
domain assumption The score function h(x,e) = exp(E^T(x)·e/τ) approximates the density ratio p(e|x)/p(e)
Critical for the InfoNCE interpretation, standard in contrastive learning, but not verified for expert representations.
domain assumption The router's top-k activated/inactivated split is a meaningful label for contrastive learning
The entire method assumes the router's decision is a good signal for specialization, and that improving MI alignment improves downstream accuracy.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CoMoE: Contrastive Representation for Mixture-of-Experts in Parameter-Efficient Fine-tuning." pith.science (2026). https://pith.science/paper/Q6PZTPQB

@misc{pith2026250517553,
  author       = {Pith},
  title        = {Pith review of: CoMoE: Contrastive Representation for Mixture-of-Experts in Parameter-Efficient Fine-tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q6PZTPQB}},
  note         = {Machine review of arXiv:2505.17553}
}

read the original abstract

In parameter-efficient fine-tuning, mixture-of-experts (MoE), which involves specializing functionalities into different experts and sparsely activating them appropriately, has been widely adopted as a promising approach to trade-off between model capacity and computation overhead. However, current MoE variants fall short on heterogeneous datasets, ignoring the fact that experts may learn similar knowledge, resulting in the underutilization of MoE's capacity. In this paper, we propose Contrastive Representation for MoE (CoMoE), a novel method to promote modularization and specialization in MoE, where the experts are trained along with a contrastive objective by sampling from activated and inactivated experts in top-k routing. We demonstrate that such a contrastive objective recovers the mutual-information gap between inputs and the two types of experts. Experiments on several benchmarks and in multi-task settings demonstrate that CoMoE can consistently enhance MoE's capacity and promote modularization among the experts.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Mixture of Raytraced Experts
cs.LG 2025-07 conditional novelty 7.0 of 10

A stacked Mixture-of-Experts model that dynamically selects ordered sequences of experts via a ray-like stochastic routing process, improving accuracy as more experts are activated.

Reference graph

Works this paper leans on

7 extracted references · 5 canonical work pages · cited by 1 Pith paper

[1]

Obtain the individual expert outputs {/(x)}7_), each with dimen- sionality E;(x) € R?

Expert representation. Obtain the individual expert outputs {/(x)}7_), each with dimen- sionality E;(x) € R?. Algorithm 1: Contrastive Loss Computation among Experts (single-sample) = Input: Top-k expert indices T € N* Expert representations {£;(x)}?_,, where E(x) € RP Temperature T Output: Contrastive loss Leontrast r~U{i,...,k} a< T(r] q «+ Normalize(E,...

work page
[2]

Uniformly sample anindex C Datasets r from {1,...,&}

Anchor selection. Uniformly sample anindex C Datasets r from {1,...,&}. Define the anchor (query) vector as g = Normalize(E7;,;(2)). . Positive set. Aggregate the remaining (k —

work page
[6]

Negative set

expert representations, excluding the one indexed by T|r] from the index set T, into a set P, applying normalization: P = {Normalize(E7 iy (x)) | j Ar}. . Negative set. Collect and normalize represen- tations from experts not included in the top-k indices: N = {Normalize(E;(x)) | 7 € T}. . Similarity computation. Compute cosine similarities between anchor...

work page
[40]

That means four times the number of coffee mugs in the teacher’s lounge is 40 — 8 = 32

How many coffee mugs are in the teacher’s lounge?” A model trained under the single-task setting re- sponds: “Assistant: Eight more than four times the number of coffee mugs in the teacher’s lounge is 40. That means four times the number of coffee mugs in the teacher’s lounge is 40 — 8 = 32. So, the number of coffee mugs in the teacher’s lounge is 32. The...

work page
[2015]

Evaluating the effects of high-throughput structural neuroimaging predictors on whole-brain functional connectome outcomes via network-based vector-on-matrix regression

Parsing algebraic word problems into equa- tions. Transactions of the Association for Computa- tional Linguistics, 3:585—-597. Siming Lan, Rui Zhang, Qi Yi, Jiaming Guo, Shaohui Peng, Yunkai Gao, Fan Wu, Ruizhi Chen, Zidong Du, Xing Hu, et al. 2024. Contrastive modules with tem- poral attention for multi-task reinforcement learning. Advances in Neural Inf...

work page Pith review arXiv 2024
[2021]

) ple |z) =E l — weet) a Nie) ple) e —top-k d We derive Eq. 15 using Jensen’s inequality, noting that log is a concave function. 1 ple

Are nlp models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080-2094. Chen Qian, Zihao Xie, Yife1 Wang, Wei Liu, Yu- fan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. 2024. S...

arXiv 2021
[2022]

In 60th Annual Meeting of the Association for Computa- tional Linguistics, ACL 2022, pages 1-9

Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In 60th Annual Meeting of the Association for Computa- tional Linguistics, ACL 2022, pages 1-9. Association for Computational Linguistics (ACL). Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical com- monsense in nat...

arXiv 2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Obtain the individual expert outputs {/(x)}7_), each with dimen- sionality E;(x) € R?

Expert representation. Obtain the individual expert outputs {/(x)}7_), each with dimen- sionality E;(x) € R?. Algorithm 1: Contrastive Loss Computation among Experts (single-sample) = Input: Top-k expert indices T € N* Expert representations {£;(x)}?_,, where E(x) € RP Temperature T Output: Contrastive loss Leontrast r~U{i,...,k} a< T(r] q «+ Normalize(E,...

work page

[2] [2]

Uniformly sample anindex C Datasets r from {1,...,&}

Anchor selection. Uniformly sample anindex C Datasets r from {1,...,&}. Define the anchor (query) vector as g = Normalize(E7;,;(2)). . Positive set. Aggregate the remaining (k —

work page

[3] [6]

Negative set

expert representations, excluding the one indexed by T|r] from the index set T, into a set P, applying normalization: P = {Normalize(E7 iy (x)) | j Ar}. . Negative set. Collect and normalize represen- tations from experts not included in the top-k indices: N = {Normalize(E;(x)) | 7 € T}. . Similarity computation. Compute cosine similarities between anchor...

work page

[4] [40]

That means four times the number of coffee mugs in the teacher’s lounge is 40 — 8 = 32

How many coffee mugs are in the teacher’s lounge?” A model trained under the single-task setting re- sponds: “Assistant: Eight more than four times the number of coffee mugs in the teacher’s lounge is 40. That means four times the number of coffee mugs in the teacher’s lounge is 40 — 8 = 32. So, the number of coffee mugs in the teacher’s lounge is 32. The...

work page

[5] [2015]

Evaluating the effects of high-throughput structural neuroimaging predictors on whole-brain functional connectome outcomes via network-based vector-on-matrix regression

Parsing algebraic word problems into equa- tions. Transactions of the Association for Computa- tional Linguistics, 3:585—-597. Siming Lan, Rui Zhang, Qi Yi, Jiaming Guo, Shaohui Peng, Yunkai Gao, Fan Wu, Ruizhi Chen, Zidong Du, Xing Hu, et al. 2024. Contrastive modules with tem- poral attention for multi-task reinforcement learning. Advances in Neural Inf...

work page Pith review arXiv 2024

[6] [2021]

) ple |z) =E l — weet) a Nie) ple) e —top-k d We derive Eq. 15 using Jensen’s inequality, noting that log is a concave function. 1 ple

Are nlp models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080-2094. Chen Qian, Zihao Xie, Yife1 Wang, Wei Liu, Yu- fan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. 2024. S...

arXiv 2021

[7] [2022]

In 60th Annual Meeting of the Association for Computa- tional Linguistics, ACL 2022, pages 1-9

Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In 60th Annual Meeting of the Association for Computa- tional Linguistics, ACL 2022, pages 1-9. Association for Computational Linguistics (ACL). Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical com- monsense in nat...

arXiv 2022