{"id":"92a7b2e0-edcf-46a2-b8bb-e5c09751070a","arxiv_id":"2509.00996","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MEPT routes each input through one of several per-layer prompt experts plus a shared expert, reporting modest accuracy gains and a 79% cut in activated prompt parameters on SuperGLUE.","lead":"A prompt-tuning method for language models adds a router that picks specialized prompt vectors at every transformer layer, plus a shared prompt for common knowledge. It reports higher SuperGLUE accuracy than several parameter-efficient baselines while activating far fewer prompt parameters.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's 79.25% parameter reduction counts only activated experts; storing all router experts makes MEPT's trainable-parameter claim several times larger than reported.","rationale":"The reader's weakest assumption was that the router learns meaningful per-input specialization without load balancing. That is a reasonable concern about the paper's mechanism and interpretability, but the central empirical claims — higher accuracy and 79.25% parameter reduction — do not logically require the router to be maximally specialized. Even a degenerate router could yield the reported accuracy if the extra prompt capacity helps. By contrast, the parameter-reduction claim is directly contradicted by the architecture as specified: Table 1 labels the 0.11–0.13% as 'trainable parameters', while §3.2 stores N_r+1 prompt tensors per layer. Counting only the activated expert makes the comparison to P-Tuning v2 asymmetric and inflates the efficiency gain. This is an internal accounting inconsistency, not a disagreement with prior work, and it is checkable from the paper's own formulas. The accuracy claims still suffer from the dev-set selection and copied-baseline issues the reader identified, so the verdict remains CONDITIONAL: the paper needs code, a fair parameter recount, and rerun or properly attributed baselines before its headline claims can be accepted. I disagree with the reader's choice of weakest assumption because the parameter-accounting flaw is more load-bearing for the paper's stated contribution.","tokens_in":22069,"tokens_out":10444,"duration_ms":130982,"concrete_test":"Determine the exact configuration behind Table 1 (from the released code; if unavailable, the best config in Table 5) and compute total stored trainable parameters analytically as L_layers × (N_r + N_s) × m × d + router_parameters for MEPT, and compare with P-Tuning v2's m' × d × L_layers. Equivalently, print HuggingFace PEFT's get_nb_trainable_parameters() for both models under identical setup. If the MEPT/P-Tuning v2 ratio is close to the reported 0.13/0.53 ≈ 0.245, the Para column counted only activated experts; if the ratio is substantially higher, re-report all Para entries as total stored parameters and retract the 79.25% reduction claim in §4.4.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.4/Table 1 headline 'reduces trainable parameters by up to 79.25% (0.53% vs 0.11%)' conflates activated prompts with total trainable parameters. In §3.2, each layer stores N_r router experts plus N_s shared experts, so stored prompt parameters are (N_r+N_s)·m·d per layer, not 2·m·d. The 0.11–0.13% figures match only the one activated router expert plus one shared expert (2·m·d·L). With the hyperparameter ranges searched in Table 5 (N_r ∈ {4,10,20}, m ∈ {10,15,20}), T5-Base stored prompt parameters are at least 12·(4+1)·10·768 ≈ 0.23% (vs 0.13% reported) and up to 12·(20+1)·20·768 ≈ 1.84% — the latter exceeds P-Tuning v2's 0.53%. Appendix G.3 even asserts the prompt parameter count is mh, omitting N_r entirely. Section 5's claim that memory overhead is 'irrelevant to the number of experts' is false for storage and optimizer state: every stored expert is a Parameter in the optimizer. Thus the efficiency half of the central claim is an artifact of comparing activated parameters against a baseline's total trainable parameters.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MEPT, a prompt-tuning method that replaces the fixed per-layer prompt of deep prompt tuning with a small set of router experts plus one shared expert. At every layer a linear-softmax router selects the top-1 router expert, and the selected expert together with the shared expert is prepended to the hidden representation. The method is evaluated on six SuperGLUE tasks using T5-Base, T5-Large, and Llama-3.2 1B, reporting accuracy gains over P-Tuning v2 and other prompt-tuning baselines, and claiming a 79.25% reduction in trainable parameters over P-Tuning v2. The paper also provides ablations on routing variants, prompt depth, shared experts, prompt length, number of experts, and a mixture-training setting, plus t-SNE and neural-pathway visualizations to support the manifold-mapping interpretation.","tokens_in":22443,"tokens_out":8445,"duration_ms":98386,"significance":"The method is simple and the empirical hypothesis — that sparse routing over per-layer prompts improves adaptation relative to a single prompt — is plausible and is supported by a fairly broad set of ablations. If the accuracy results survive a controlled, fully reported comparison, MEPT would be a useful and practical addition to prompt tuning. However, the headline efficiency claim is not correct under the paper's own parameter definitions, and the statistical/comparison protocol is incomplete. The paper's contribution is therefore a promising method whose current presentation overstates the efficiency benefit and leaves the empirical comparison only partially verifiable.","major_comments":[{"comment":"The reported 'Para' for MEPT (0.13% on T5-Base, 0.12% on T5-Large, 0.11% on Llama) corresponds to the number of activated prompt vectors per layer, not the number of stored trainable parameters. In §3.2 each layer stores R^j ∈ R^{N_r×m×d} plus S^j ∈ R^{N_s×m×d}, so the stored count is (N_r+N_s)·m·d·L. With the searched ranges in Tab. 5 (N_r ∈ {4,10,20}, m ∈ {10,15,20}) and N_s=1, T5-Base ranges from about 0.21% (N_r=4,m=10) to about 1.76% (N_r=20,m=20) of backbone parameters; the upper end exceeds P-Tuning v2's 0.53%. Appendix G.3's 'parameter count is mh' omits N_r and N_s entirely. Thus the 79.25% reduction claim is at most a statement about activated prompts, not trainable parameters, and the §5 assertion that memory overhead is 'irrelevant to the number of experts' is false for parameter storage and optimizer state. All parameter counts and efficiency claims must be recomputed under","section":"Table 1 / §4.4 / §3.2"},{"comment":"Several baselines (XPrompt, ResPrompt, etc.) are marked in the table caption as taken from other papers, not rerun under the protocol used for MEPT. Difference in prompt length, learning rate, epochs, and backbone handling can easily produce 1–2% accuracy gaps. At the same time, MEPT's own hyperparameters (L and N_r) are selected on the SuperGLUE development set, and the same development set is used for the final numbers in Table 1. No held-out or test-set results are reported, and no per-task standard deviations are given. Please rerun all baselines under a single controlled protocol, or clearly separate validation-based tuning from final reporting, and provide per-task variance.","section":"Table 1 / §4.3"},{"comment":"The statement 'Results are statistically significant with respect to all baselines on each PLM (all p-value < 0.005)' is not supported by any test description, test statistic, or per-run/per-task data. With only three runs per model–task cell, it is unclear how a p-value is computed across six tasks or across baselines. If the claim is intended to cover every pairwise baseline difference, it is implausible without reporting the test procedure (e.g., paired bootstrap over examples or over tasks). Please either provide the full statistical testing protocol and results, or remove this claim.","section":"Table 1 caption"}],"minor_comments":[{"comment":"Typos: 'avaliable' should be 'available'; 'instinct neural activation pathway' should probably be 'intrinsic' or 'inherent'.","section":"Abstract; §1"},{"comment":"The phrase 'linearly search the best prompt length' presumably means 'grid search'; please use the standard term. Also, 'lengthes' is a typo.","section":"§4.3"},{"comment":"The abstract says 'Our code is avaliable at runjia.tech/emnlp_mept' but §4.3 says 'our full implementation shall be publicly released upon paper acceptance.' Please reconcile these statements and provide a functional repository so the reproducibility claim can be checked.","section":"Code availability"},{"comment":"The pathway visualization and the MAE/cosine similarity numbers are reported for a single seed and a small sample (200 examples per task) without confidence intervals. These analyses are useful as illustration, but the wording 'validated through neural activation pathway visualization' should be softened to 'qualitative support' unless statistical uncertainty is quantified.","section":"§6 / Fig. 4"},{"comment":"The sentence 'the parameter count for the prompt-based method is mh' is inconsistent with §3.2; it should be '(N_r+N_s)·m·h' per layer (and multiplied by the number of layers). The current wording propagates the same activated-vs-stored confusion as the main text.","section":"Appendix G.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be an EMNLP-style submission. The code link points to a personal page rather than a functional repository, and several baseline numbers are taken from prior work, including a paper by some of the same authors (Wang et al. 2023). During revision, the editor should require the actual code, raw per-run results, and a full table of hyperparameters for every baseline. The parameter-count issue is fixable in principle, but it changes the main efficiency claim, so it cannot be treated as a minor typo."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: MEPT is a plausible per-layer top-1 MoE prompt-tuning extension with solid ablations, but the headline efficiency number is an \"activated parameters\" claim presented as a trainable-parameter reduction. That needs to be fixed before I'd trust the table.\n\nWhat's actually new: combining deep per-layer routing, top-1 sparsity, and a shared expert prompt is a legitimate extension of SMoP and PT-MoE, not a breakthrough. The paper does this well in its ablations—router variants, shared-expert removal, depth, sensitivity, mixture training, and a SQuAD sanity check give a clear picture of when the method helps. That honest reporting earns real credit.\n\nThe soft spots are in the efficiency claim and the statistics. The \"Para\" column in Table 1 reports only the activated prompts (one router expert plus one shared expert per layer). With the searched ranges (Nr up to 20, prompt length up to 20), the total stored prompt parameters can exceed P-Tuning v2's count. Appendix G.3's \"mh\" parameter count omits Nr entirely, and Section 5's claim that memory overhead is \"irrelevant to the number of experts\" is false for storage and optimizer state—every stored expert is a parameter with momentum and variance slots. The p<0.005 significance statement is unsupported because several baselines are copied from other papers; you can't do a paired test on someone else's numbers. Selecting hyperparameters on the same dev set used for final reporting also inflates the apparent gains. The manifold-learning section is post hoc commentary, not a derivation, and the t-SNE/pathway figures are illustrative rather than evidence.\n\nNone of this kills the accuracy result, which is modest (0.2–1.9 points over strong prompt-tuning baselines). But the abstract and Section 4.4 specifically overstate the efficiency win, and that framing would mislead readers if published as-is. The ablations are the real contribution.\n\nWho is this for? Researchers working on prompt tuning or MoE-based PEFT will find the routing ablations and training-time comparisons useful. It doesn't resolve a big open question, but it's a careful incremental study.\n\nRecommendation: send it to peer review. The accuracy claim deserves scrutiny, and with a re-benchmarked parameter table, a toned-down significance statement, and a clearer separation of activated vs. stored parameters, the paper could be a decent EMNLP/ACL workshop-level or short-paper contribution.","headline":"Useful incremental prompt-tuning variant, but the 79% parameter reduction counts only activated experts, not stored ones; worth refereeing with a request to fix the accounting.","tokens_in":22946,"tokens_out":2511,"would_cite":true,"duration_ms":32587,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prompt-tuning mixture MEPT beats full fine-tuning on SuperGLUE with 79.25% fewer activated prompts.","keywords":["mixture of experts","prompt tuning","parameter-efficient fine-tuning","manifold learning","SuperGLUE","soft prompts","top-1 routing","large language models"],"falsifier":"Retrain MEPT with the router locked to random-but-fixed expert assignments, or shuffle expert indices across layers after training; if the accuracy gaps persist, then expert specialization is not what carries the gains. A cheaper check is to measure per-layer expert utilization entropy on held-out data: if one expert dominates nearly all inputs across tasks, routing is not meaningfully selective.","tokens_in":1713,"feed_emoji":"🧩","tokens_out":2209,"duration_ms":101620,"temperature":0.7,"pith_summary":"The paper argues that pretrain-then-finetune is best understood as manifold mapping: pretraining builds a broad knowledge base, and fine-tuning should activate different neural pathways depending on the input's target data distribution. It proposes MEPT, which replaces a single fixed soft prompt with a sparse mixture of prompt experts at every Transformer layer, plus one shared expert that consolidates common knowledge. On the SuperGLUE development set, MEPT reports higher average accuracy than full fine-tuning and than prior prompt-tuning methods across T5-Base, T5-Large, and Llama-3.2 1B, while cutting trainable prompt parameters by 79.25% relative to deep prompt tuning. The authors take this as evidence that input-dependent routing yields task-specific manifold mappings with better final-layer separability.","feed_headline":"79.25% fewer prompts, 1.94% better SuperGLUE accuracy","feed_subtitle":"Routes each input to specialist prompts, beating full fine-tuning with far fewer active parameters.","key_machinery":"The central object is MEPT, a mixture-of-experts layer embedded inside the soft prompt at every Transformer layer. It consists of a bank of router-expert prompt embeddings, one shared-expert prompt embedding, and a linear-softmax router that consumes the preceding hidden state and selects the single highest-probability router expert via top-1 routing. The always-active shared expert is summed with the chosen router expert to form that layer's prompt embedding. This mechanism converts a fixed prompt into a manifold mapper: router experts specialize to input subspaces while the shared expert consolidates common knowledge, and because only one router expert is active per layer, training memory","core_discovery":"The paper claims that a soft prompt should be an input-dependent mixture of experts rather than a fixed tensor. MEPT places, at each Transformer layer, a bank of router-expert prompt embeddings, one always-on shared expert, and a linear-softmax router that selects the single highest-scoring router expert for each hidden state. On the SuperGLUE development set, MEPT reports mean accuracy of 78.03 on T5-Base, 85.27 on T5-Large, and 63.00 on Llama-3.2 1B, exceeding P-Tuning v2 by 1.09, 1.48, and 1.94 points respectively and surpassing full fine-tuning on T5-Base and T5-Large. The same sparse design cuts trainable prompt parameters by 79.25% relative to P-Tuning v2. The authors also report that","pith_inferences":["Editorial inference: if expert pathways reflect semantic task structure, MEPT could serve as a cheap distribution-shift probe — an input that suddenly routes through nonstandard experts may be out-of-distribution.","Editorial inference: the mixture-training results suggest a natural continual-learning extension: freeze old experts when a new task arrives and add new router experts, avoiding retraining the backbone while preserving old-task routes.","Editorial inference: since the reported gains grow with task diversity, a stress test would push MEPT to many more tasks and replace manual search over prompt length and expert count with a learned allocation.","Editorial inference: the paper deliberately omits a load-balancing loss; a testable extension is to add an expert-level balance loss and check whether multi-task specialization improves or, as the perturbation ablation hints, degrades."],"forward_implications":["MEPT's sparse top-1 routing and single shared expert per layer make the active parameter cost independent of the number of experts, so expert count can grow without growing training or inference cost.","Because MEPT beats full fine-tuning on SuperGLUE with roughly 0.1% of trainable parameters, the pretrained backbone appears to already contain task-specific structure that prompt routing can elicit rather than rewrite.","Under mixture training, where one prompt set must serve all six SuperGLUE tasks, MEPT keeps a performance lead that widens as more tasks are combined, indicating robustness for multi-task adaptation.","The pathway visualizations imply that semantically related tasks share expert routes; a direct corollary is that a new task should transfer best if routed through experts used by its closest neighbors.","MEPT also shows competitive results on SQuAD and on quantized larger models in the appendix, suggesting the mechanism transfers beyond classification-style SuperGLUE tasks."],"supporting_citations":[{"why":"Defines vanilla soft prompt tuning, the baseline MEPT extends and the prompt-embedding formulation MEPT builds on.","marker":"Lester et al., 2021"},{"why":"Introduces P-Tuning v2, the deep prompt tuning baseline whose parameter count and accuracy MEPT is directly compared against.","marker":"Liu et al., 2022b"},{"why":"SMoP is the sparse mixture-of-prompts baseline; it also supplies the text-to-text SuperGLUE formatting MEPT adopts.","marker":"Choi et al., 2023"},{"why":"DeepSeekMoE's shared-and-routed expert split motivates MEPT's router-expert and shared-expert architecture.","marker":"Dai et al., 2024"},{"why":"Defines SuperGLUE, the benchmark on which all main results are reported.","marker":"Wang et al., 2019"},{"why":"Introduces T5, the encoder-decoder backbone used for the majority of MEPT experiments and its text-to-text formulation.","marker":"Raffel et al., 2020"},{"why":"Supplies the manifold separability and geometry view used to frame fine-tuning as manifold mapping.","marker":"Cohen et al., 2020"},{"why":"LoRA is the low-rank adaptation baseline MEPT exceeds in the appendix's parameter-efficient comparison on T5-Large.","marker":"Hu et al., 2022"},{"why":"Provides the full fine-tuning results on T5 that MEPT is compared against in Table 1.","marker":"Aribandi et al., 2022"}],"fun_headline_variants":["Input-dependent prompts: MoE beats fine-tuning, cuts 79% params","Mixture-of-experts prompts: +1.94 accuracy, -79% trainable","Router picks prompt expert per token, boosts SuperGLUE","Sparse MoE for prompt tuning: better than full fine-tune"],"cache_read_input_tokens":24576,"weakest_assumption_plain":"The load-bearing premise is that a plain linear-softmax router, trained jointly with the prompt experts and without a load-balancing loss, will actually learn to divide inputs among experts in a task-meaningful way; if routing collapses onto one expert or follows input noise, MEPT degenerates to a fixed multi-prompt layer and the reported gains disappear.","fun_headline_variants_meta":{"raw":{"variants":["Input-dependent prompts: MoE beats fine-tuning, cuts 79% params","Mixture-of-experts prompts: +1.94 accuracy, -79% trainable","Router picks prompt expert per token, boosts SuperGLUE","Sparse MoE for prompt tuning: better than full fine-tune"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000146,"raw_usage":{"total_tokens":1041,"prompt_tokens":790,"completion_tokens":251,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":170}},"tokens_in":534,"tokens_out":251,"duration_ms":4000,"temperature":1.0,"reasoning_tokens":170,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:58:52.244869+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain MEPT with the router locked to random-but-fixed expert assignments, or shuffle expert indices across layers after training; if the accuracy gaps persist, then expert specialization is not what carries the gains. A cheaper check is to measure per-layer expert utilization entropy on held-out data: if one expert dominates nearly all inputs across tasks, routing is not meaningfully selective.","supporting_citations":[],"review_version":1}