Pith. sign in

REVIEW 3 major objections 5 minor 13 references

MEPT: Mixture of Expert Prompt Tuning as a Manifold Mapper

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Prompt-tuning mixture MEPT beats full fine-tuning on SuperGLUE with 79.25% fewer activated prompts.

desk verdict Useful incremental prompt-tuning variant, but the 79% parameter reduction counts only activated experts, not stored ones; worth refereeing with a request to fix the accounting. read the letter →

arxiv 2509.00996 v2 pith:4TY4B2OJ submitted 2025-08-31 cs.LG cs.AIcs.CL

Runjia Zeng , Guangyan Sun , Qifan Wang , Tong Geng , Sohail Dianat , Xiaotian Han , Raghuveer Rao , Xueling Zhang
show 3 more authors
Cheng Han Lifu Huang Dongfang Liu
This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords mixtureofexpertsprompttuningparameter-efficientfine-tuningmanifoldlearningSuperGLUEsoftpromptstop-1routinglargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that pretrain-then-finetune is best understood as manifold mapping: pretraining builds a broad knowledge base, and fine-tuning should activate different neural pathways depending on the input's target data distribution. It proposes MEPT, which replaces a single fixed soft prompt with a sparse mixture of prompt experts at every Transformer layer, plus one shared expert that consolidates common knowledge. On the SuperGLUE development set, MEPT reports higher average accuracy than full fine-tuning and than prior prompt-tuning methods across T5-Base, T5-Large, and Llama-3.2 1B, while cutting trainable prompt parameters by 79.25% relative to deep prompt tuning. The authors take this as evidence that input-dependent routing yields task-specific manifold mappings with better final-layer separability.

What carries the argument

The central object is MEPT, a mixture-of-experts layer embedded inside the soft prompt at every Transformer layer. It consists of a bank of router-expert prompt embeddings, one shared-expert prompt embedding, and a linear-softmax router that consumes the preceding hidden state and selects the single highest-probability router expert via top-1 routing. The always-active shared expert is summed with the chosen router expert to form that layer's prompt embedding. This mechanism converts a fixed prompt into a manifold mapper: router experts specialize to input subspaces while the shared expert consolidates common knowledge, and because only one router expert is active per layer, training memory

What would settle it

Retrain MEPT with the router locked to random-but-fixed expert assignments, or shuffle expert indices across layers after training; if the accuracy gaps persist, then expert specialization is not what carries the gains. A cheaper check is to measure per-layer expert utilization entropy on held-out data: if one expert dominates nearly all inputs across tasks, routing is not meaningfully selective.

Watch

Extended reading notes

Core claim

The paper claims that a soft prompt should be an input-dependent mixture of experts rather than a fixed tensor. MEPT places, at each Transformer layer, a bank of router-expert prompt embeddings, one always-on shared expert, and a linear-softmax router that selects the single highest-scoring router expert for each hidden state. On the SuperGLUE development set, MEPT reports mean accuracy of 78.03 on T5-Base, 85.27 on T5-Large, and 63.00 on Llama-3.2 1B, exceeding P-Tuning v2 by 1.09, 1.48, and 1.94 points respectively and surpassing full fine-tuning on T5-Base and T5-Large. The same sparse design cuts trainable prompt parameters by 79.25% relative to P-Tuning v2. The authors also report that

Load-bearing premise

The load-bearing premise is that a plain linear-softmax router, trained jointly with the prompt experts and without a load-balancing loss, will actually learn to divide inputs among experts in a task-meaningful way; if routing collapses onto one expert or follows input noise, MEPT degenerates to a fixed multi-prompt layer and the reported gains disappear.

Editorial extensions

If this is right

  • MEPT's sparse top-1 routing and single shared expert per layer make the active parameter cost independent of the number of experts, so expert count can grow without growing training or inference cost.
  • Because MEPT beats full fine-tuning on SuperGLUE with roughly 0.1% of trainable parameters, the pretrained backbone appears to already contain task-specific structure that prompt routing can elicit rather than rewrite.
  • Under mixture training, where one prompt set must serve all six SuperGLUE tasks, MEPT keeps a performance lead that widens as more tasks are combined, indicating robustness for multi-task adaptation.
  • The pathway visualizations imply that semantically related tasks share expert routes; a direct corollary is that a new task should transfer best if routed through experts used by its closest neighbors.
  • MEPT also shows competitive results on SQuAD and on quantized larger models in the appendix, suggesting the mechanism transfers beyond classification-style SuperGLUE tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if expert pathways reflect semantic task structure, MEPT could serve as a cheap distribution-shift probe — an input that suddenly routes through nonstandard experts may be out-of-distribution.
  • Editorial inference: the mixture-training results suggest a natural continual-learning extension: freeze old experts when a new task arrives and add new router experts, avoiding retraining the backbone while preserving old-task routes.
  • Editorial inference: since the reported gains grow with task diversity, a stress test would push MEPT to many more tasks and replace manual search over prompt length and expert count with a learned allocation.
  • Editorial inference: the paper deliberately omits a load-balancing loss; a testable extension is to add an expert-level balance loss and check whether multi-task specialization improves or, as the perturbation ablation hints, degrades.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes MEPT, a prompt-tuning method that replaces the fixed per-layer prompt of deep prompt tuning with a small set of router experts plus one shared expert. At every layer a linear-softmax router selects the top-1 router expert, and the selected expert together with the shared expert is prepended to the hidden representation. The method is evaluated on six SuperGLUE tasks using T5-Base, T5-Large, and Llama-3.2 1B, reporting accuracy gains over P-Tuning v2 and other prompt-tuning baselines, and claiming a 79.25% reduction in trainable parameters over P-Tuning v2. The paper also provides ablations on routing variants, prompt depth, shared experts, prompt length, number of experts, and a mixture-training setting, plus t-SNE and neural-pathway visualizations to support the manifold-mapping interpretation.

Significance. The method is simple and the empirical hypothesis — that sparse routing over per-layer prompts improves adaptation relative to a single prompt — is plausible and is supported by a fairly broad set of ablations. If the accuracy results survive a controlled, fully reported comparison, MEPT would be a useful and practical addition to prompt tuning. However, the headline efficiency claim is not correct under the paper's own parameter definitions, and the statistical/comparison protocol is incomplete. The paper's contribution is therefore a promising method whose current presentation overstates the efficiency benefit and leaves the empirical comparison only partially verifiable.

major comments (3)
  1. [Table 1 / §4.4 / §3.2] The reported 'Para' for MEPT (0.13% on T5-Base, 0.12% on T5-Large, 0.11% on Llama) corresponds to the number of activated prompt vectors per layer, not the number of stored trainable parameters. In §3.2 each layer stores R^j ∈ R^{N_r×m×d} plus S^j ∈ R^{N_s×m×d}, so the stored count is (N_r+N_s)·m·d·L. With the searched ranges in Tab. 5 (N_r ∈ {4,10,20}, m ∈ {10,15,20}) and N_s=1, T5-Base ranges from about 0.21% (N_r=4,m=10) to about 1.76% (N_r=20,m=20) of backbone parameters; the upper end exceeds P-Tuning v2's 0.53%. Appendix G.3's 'parameter count is mh' omits N_r and N_s entirely. Thus the 79.25% reduction claim is at most a statement about activated prompts, not trainable parameters, and the §5 assertion that memory overhead is 'irrelevant to the number of experts' is false for parameter storage and optimizer state. All parameter counts and efficiency claims must be recomputed under
  2. [Table 1 / §4.3] Several baselines (XPrompt, ResPrompt, etc.) are marked in the table caption as taken from other papers, not rerun under the protocol used for MEPT. Difference in prompt length, learning rate, epochs, and backbone handling can easily produce 1–2% accuracy gaps. At the same time, MEPT's own hyperparameters (L and N_r) are selected on the SuperGLUE development set, and the same development set is used for the final numbers in Table 1. No held-out or test-set results are reported, and no per-task standard deviations are given. Please rerun all baselines under a single controlled protocol, or clearly separate validation-based tuning from final reporting, and provide per-task variance.
  3. [Table 1 caption] The statement 'Results are statistically significant with respect to all baselines on each PLM (all p-value < 0.005)' is not supported by any test description, test statistic, or per-run/per-task data. With only three runs per model–task cell, it is unclear how a p-value is computed across six tasks or across baselines. If the claim is intended to cover every pairwise baseline difference, it is implausible without reporting the test procedure (e.g., paired bootstrap over examples or over tasks). Please either provide the full statistical testing protocol and results, or remove this claim.
minor comments (5)
  1. [Abstract; §1] Typos: 'avaliable' should be 'available'; 'instinct neural activation pathway' should probably be 'intrinsic' or 'inherent'.
  2. [§4.3] The phrase 'linearly search the best prompt length' presumably means 'grid search'; please use the standard term. Also, 'lengthes' is a typo.
  3. [Code availability] The abstract says 'Our code is avaliable at runjia.tech/emnlp_mept' but §4.3 says 'our full implementation shall be publicly released upon paper acceptance.' Please reconcile these statements and provide a functional repository so the reproducibility claim can be checked.
  4. [§6 / Fig. 4] The pathway visualization and the MAE/cosine similarity numbers are reported for a single seed and a small sample (200 examples per task) without confidence intervals. These analyses are useful as illustration, but the wording 'validated through neural activation pathway visualization' should be softened to 'qualitative support' unless statistical uncertainty is quantified.
  5. [Appendix G.3] The sentence 'the parameter count for the prompt-based method is mh' is inconsistent with §3.2; it should be '(N_r+N_s)·m·h' per layer (and multiplied by the number of layers). The current wording propagates the same activated-vs-stored confusion as the main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the accuracy claims are empirical and the efficiency claim is an architectural accounting issue, not a derivation that reduces to its own inputs.

full rationale

The central accuracy claim (Table 1) is not a fitted input: MEPT is trained on SuperGLUE and compared with baselines under fixed training configurations, so the reported scores are empirical outcomes rather than reductions of the method's equations. The MoE/prompt equations in §3.1–3.2 define the architecture and router; they do not encode the accuracy gains. The manifold discussion in §6 and App. G.1 is a post-hoc interpretation of the trained model, not a derivation that assumes the conclusion, and the cited works (DeepSeek, SMoP, prior prompt-tuning papers) are used as baselines or design inspirations, not as a self-citation chain that forces the result. The only notable issue is the parameter-counting claim: Table 1/§4.4 report 0.11–0.13% 'trainable parameters' while §3.2 stores N_r router experts per layer, so the 79.25% reduction reflects an activated-prompt count rather than total stored trainable parameters. That is an internal consistency/correctness problem, not a circularity, because the efficiency number is a description of the configuration rather than a fitted prediction, and the abstract itself carefully says 'activated prompts.' No step in the paper's argument reduces to its own inputs by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

MEPT introduces two new trainable component types (router expert prompts and shared expert prompts) plus three hyperparameters (Nr, prompt length, learning rate) that are selected on the dev set. The central claim rests on the domain assumptions that neural collapse characterizes these transformers, that a simple router can specialize, and that a single shared prompt stores common knowledge. None of these is derived or externally verified.

free parameters (4)
  • number of router experts Nr = 4, 10, or 20 (task-dependent, selected on dev set)
    Searched over {4,10,20} in section 4.3; the best value is chosen per task based on SuperGLUE development set accuracy, so it is a free parameter fitted to the evaluation data.
  • prompt length L = 10, 15, or 20 (task-dependent)
    Searched over {10,15,20} in section 4.3 and selected on dev set; affects parameter count and performance.
  • learning rate = 1e-5, 5e-5, or 1e-4 for MEPT
    Linearly searched per section B.2 and selected on dev set performance.
  • number of shared experts Ns = 1 per layer (hand-chosen default)
    Set to 1 for efficiency; ablated in Table 13 (routed-to-shared ratios), but the default is a single shared expert per layer.
assumptions (4)
  • domain assumption Deeper layers concentrate within-class features and separate between-class features (neural collapse).
    Stated in section G.1 as the motivation for manifold mapping; not proven for the models and tasks tested.
  • domain assumption A linear-softmax router on the hidden state supplies enough signal to select a useful prompt expert.
    Assumed in section 3.2 without analysis; the router shares no auxiliary objective and no load-balancing loss.
  • ad hoc to paper A single shared prompt expert per layer captures the common knowledge across tasks.
    Introduced in section 3.2 and defended post hoc via case studies in Appendix F and ablations; not derived.
  • domain assumption Pre-trained backbones (T5-Base, T5-Large, Llama-3.2 1B) provide a general manifold that prompt-based adaptation can reshape.
    Underlies the pretrain-then-finetune framing in section 1; taken as background rather than tested.
invented entities (2)
  • Router expert prompts
    purpose: Per-layer trainable prompt embeddings; top-1 selected by the router to specialize on input subspaces.
    Introduced in section 3.2; the only evidence is internal visualization of routing patterns in section 6 and Appendix H, with no external falsifiable prediction.
  • Shared expert prompts
    purpose: Per-layer trainable prompt embeddings that are always active, meant to store common knowledge across tasks.
    Introduced in section 3.2, inspired by DeepSeekMoE shared experts; support is via ablations and case studies, all internal to the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MEPT: Mixture of Expert Prompt Tuning as a Manifold Mapper." pith.science (2026). https://pith.science/paper/4TY4B2OJ

@misc{pith2026250900996,
  author       = {Pith},
  title        = {Pith review of: MEPT: Mixture of Expert Prompt Tuning as a Manifold Mapper},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4TY4B2OJ}},
  note         = {Machine review of arXiv:2509.00996}
}
read the original abstract

Considering deep neural networks as manifold mappers, the pretrain-then-fine-tune paradigm can be interpreted as a two-stage process: pretrain establishes a broad knowledge base, and fine-tune adjusts the model parameters to activate specific neural pathways to align with the target manifold. Although prior fine-tuning approaches demonstrate success, their rigid parameter space limits their ability to dynamically activate appropriate neural pathways, rendering them ill-equipped to adapt flexibly to the diverse and evolving data distributions. In light of this view, we propose a novel approach, Mixture of Expert Prompt Tuning (MEPT), as an effective and efficient manifold-mapping framework. MEPT leverages the Mixture of Experts architecture by integrating multiple prompt experts to adaptively learn diverse and non-stationary data distributions. Empirical evaluations demonstrate that MEPT outperforms several state-of-the-art parameter efficient baselines on SuperGLUE, achieving notable improvements in mean accuracy (e.g., 1.94%) while significantly reducing activated prompts by 79.25%. The effectiveness of MEPT is further supported by theoretical insights from manifold learning and validated through neural activation pathway visualization results. Our code is avaliable at https://runjia.tech/emnlp_mept/.

Figures

Figures reproduced from arXiv: 2509.00996 by the authors.

Figure 1
Figure 1. MEPT (ours) vs concurrent arts under the manifold learning perspective. (a) Average accuracy performance comparison with vanilla prompt tuning methods across varying size model structures. (b) Il￾lustration of the manifold hypothesis, where different neural activation paths alter task manifold geometry (blue: ‘Sentence completion,’ green: ‘Natural language inference,’ orange: ‘Question answering’), transforming non-… view at source ↗
Figure 2
Figure 2. Overview of PT vs. MEPT (ours) frameworks. (a) Vanilla Prompt Tuning. (c) The overall architecture of our proposed MEPT (see §3.2), including router expert, shared expert and router in each layer. achieve universal adaptation. At each layer, prompt representations are constructed by two types of experts: router experts and shared experts. Router Experts as Representation Specialists. While conventional prompts are e… view at source ↗
Figure 3
Figure 3. Overview of the two distinct training schemes [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Neural activation pathway for various tasks in the manifold mapping. The attached table reveals the differences in SuperGLUE. (a) t-SNE visualizations of data distribution using BERT (Devlin et al., 2018). (b) The neural activation pathway for each task, where the high…
Figure 5
Figure 5. Figure 5: The expert utilization in each layer under the [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Detailed neural activation pathway for six SuperGLUE tasks. Each row is normalized to sum to 1, with the activated experts in each layer represented by the deepest color, indicating the probability of selecting a given expert [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 11 canonical work pages

  1. [1]

    Within-Class Convergence:Aslincreases, ∥∥h(l)(x1)−h (l)(x2)∥∥ →0 forx 1, x2 in the same class

    work page

  2. [2]

    Between-Class Separation:Simultaneously, ∥∥h(l)(x1)−h (l)(x2)∥∥ →d >0 forx 1, x2 in different classes

    work page

  3. [3]

    These phenomena motivate our method

    Final Collapse:At the top layer, h(L−1)(x)≈v c for allxin classc, wherev c is the prototype vector for classc. These phenomena motivate our method. Fig. 4 shows within-class convergence (e.g., the green cluster tightens), between-class separation (distinct inter-class margins), and final collapse (compare Fig. 4(a) and Fig. 4(c)). G.2 Key Distinctions fro...

    work page

  4. [7]

    Mixture of Expert Prompt Tuning as a Manifold Mapper

    Adaptive budget allocation for parameter- efficient fine-tuning. InICLR. Wei Zhu, Qiang Qiu, Jiaji Huang, Robert Calderbank, Guillermo Sapiro, and Ingrid Daubechies. 2018. Ldmnet: Low dimensional manifold regularized neu- ral networks. InCVPR. SUMMARY OF THE APPENDIX This appendix contains additional details for the 63rd Annual Meeting of the Association ...

    work page 2018

  5. [13]

    as follows. First, MEPT employs a deep multi-layer MoE architecture, incorporating mul- tiple experts at every Transformer layer, while SMoP restricts MoE application to the input layer, thereby limiting its representational capacity. Sec- ond, MEPT introduces a hybrid design of shared and non-shared experts, enabling it to capture both generalizable know...

    work page 2024

  6. [42]

    common knowledge

    We average the sequence-level representations (h∈R L×d where L is the sequence length and d is the hidden dimension) to obtain a fixed-size vector (¯h∈R d) for each input, enabling direct comparison across examples of varying lengths. These embeddings are then visualized in a 2D scat- ter plot where points are colored according to their source dataset, al...

    work page 2010

  7. [2012]

    The winograd schema challenge. InKR. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In ACL. Zongqian Li, Yixuan Su, and Nigel Collier. 2025. Pt- moe: An efficient finetuning framework for integrat- ing mixture-of-experts into prompt tuning.arXiv preprint arXiv:2505.09519. Yuxin Lin, Xiaodong Ma, Yejin Chu...

    work page Pith review arXiv 2021

  8. [2014]

    InICLR (Workshop Poster)

    Learning factored representations in a deep mixture of experts. InICLR (Workshop Poster). William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. 2007. The thir...

    arXiv 2022

Show all 13 references
  1. [2019]

    In NAACL

    Wic: the word-in-context dataset for evaluat- ing context-sensitive meaning representations. In NAACL. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners.OpenAI blog. Colin Raffel, N...

    2019 arXiv

  2. [2020]

    Matthew E Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih

    Prevalence of neural collapse during the terminal phase of deep learning training.Pro- ceedings of the National Academy of Sciences, 117(40):24652–24663. Matthew E Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018. Dissecting contextual word embeddings: Architectur...

    2018

  3. [2022]

    Xprompt: Exploring the extreme of prompt tuning. InACL. Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. 2018. Ex- ploring the limits of weakly supervised pretraining. InECCV. Sourab Mangrulka...

    2018

  4. [2023]

    Preprint, arXiv:2210.03155

    Understanding neural coding on latent man- ifolds by sharing features and dividing ensembles. Preprint, arXiv:2210.03155. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, e...

    2020 arXiv

  5. [2025]

    XPrompt (Ma et al., 2022) enhances efficiency by prun- ing less informative token-level and piece-level prompts

    is a foundational approach that introduces trainable prompts at the first input layer. XPrompt (Ma et al., 2022) enhances efficiency by prun- ing less informative token-level and piece-level prompts. P-Tuning v2 (Liu et al., 2022b) extends Prompt-Tuning by injecting distinct p...

    2022

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.