{"id":"6661c466-1d56-4910-b2d9-039f48560554","arxiv_id":"2505.09519","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"PT-MoE combines low-rank prompt decomposition with routed expert prompts and reports gains over prompt tuning and LoRA across 17 QA and math datasets while using about 25% fewer parameters than LoRA.","lead":"PT-MoE is a new way to fine-tune large language models that splits each learned prompt into a small input-specific part and a shared part, then lets a router pick among prompts for each input. It reports better question-answering and math scores than standard prompt tuning and LoRA while using about 25% fewer trainable parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim depends on best-of-grid single-run comparisons with no error bars; the 1.49 F1 QA margin over PT and the 0.44 math margin over LoRA are within the range that seed noise can produce, so the central claim is not yet established.","rationale":"The reader's weakest assumption is exactly the right pressure point: the paper's own appendix reveals a per-method training-step search and single-run reporting, while the central claim is quantitative ('1.49 F1 over PT, 0.44 accuracy over LoRA'). These margins are small enough that seed noise or a lucky grid point could account for them. The 3B check (Table 6) and the large math gain over PT provide some independent support, but they do not fix the core evaluation-protocol problem for the headline numbers. I did not find a more serious internal inconsistency: the parameter counts, decomposition equations, and routing pseudo-code are coherent, and the 80K parameter budget reconciles with N=2, T=40, R=36 on a 2048-dimension model. Full fine-tuning still beats PT-MoE on both average metrics, so the 'state-of-the-art' phrasing needs qualification, but in context it is best read as state-of-the-art among PEFT methods. Verdict remains CONDITIONAL; no change from the reader is needed.","tokens_in":14547,"tokens_out":6115,"duration_ms":63703,"concrete_test":"Rerun PT-MoE, PT, SMoP, and LoRA on the MRQA and math suites with K=5 random seeds at every grid step in Appendix Table 8; report mean plus/minus standard deviation and paired bootstrap 95% confidence intervals for the PT-MoE minus PT QA F1 difference and the PT-MoE minus LoRA math accuracy difference. If either interval includes 0, or if the per-method best-of-grid ordering changes under a jackknife over the 12 QA datasets, the headline SOTA claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline improvements are selected from a per-method training-step grid: Appendix Table 8 gives {500, 1000, 1500} for PT-based methods and {200, 600, 1000} for LoRA-based methods, and Figure 4 reports \"highest performance after training parameter search.\" Every result table lists a single run with no standard deviation, confidence interval, or significance test. The central SOTA claim therefore rests on the assumption that the chosen grid point represents each method fairly and that run-to-run variance is smaller than the reported gaps. The QA gap over PT is 1.49 F1 and the math gap over LoRA is 0.44 accuracy; both are typical magnitudes of seed-level variation in instruction-tuned LLM finetuning, especially with prompt-length and rank hyperparameters on a 1B model. Because PT-MoE's QA advantage is partly concentrated on DROP (+7.63 F1 over PT in Table 2), a single favorable run on that dataset could drive the average. Without repeated seeds or a paired comparison at a fixed budget, the 'state-of-the-art' claim is an observed outcome of a selection procedure, not a demonstrated property of the method.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PT-MoE, a prompt-tuning framework that combines low-rank matrix decomposition of soft prompts with mixture-of-experts routing. Each of N soft prompts is written as P_i = A_i B, with expert-specific A_i and a shared B, and a learned router selects/weights experts per input. The method is evaluated on 12 MRQA QA datasets and 5 math problem-solving datasets with LLaMA-3.2-1B-Instruct (plus a 3B model for one comparison), and it is compared against full fine-tuning, LoRA, HydraLoRA, PT, DPT, SMoP, and ATTEMPT. The paper reports state-of-the-art F1/EM on QA and accuracy on math, with 80k trainable parameters, and includes ablations over prompt length, expert count, parameter count, routing mechanism, and model size. Code is released at a GitHub repository.","tokens_in":14785,"tokens_out":3160,"duration_ms":32366,"significance":"If the reported gains reproduce under careful evaluation, PT-MoE would be a useful contribution: it is a simple, modular combination of decomposition and MoE routing that obtains competitive results with very few trainable parameters, and the paper's broad coverage of 17 datasets plus ablations on architecture choices would be valuable for the PEFT community. The paper also ships a clear algorithm and a public code link, which makes the claims straightforwardly checkable. However, the central 'state-of-the-art' claim rests entirely on empirical comparisons that are currently reported as single runs, and the headline margins are small enough that they may be explained by training-step selection or seed variance. The contribution is interesting and plausible, but the evidence as presented is not yet sufficient to establish the advertised SOTA claim.","major_comments":[{"comment":"All result tables report a single run with no error bars, no confidence intervals, and no significance tests, and training steps are selected per method from a grid (e.g., {500,1000,1500} for PT-based methods and {200,600,1000} for LoRA-based methods in Table 8, with Figure 4 explicitly showing the 'highest performance after training parameter search'). The headline QA gain over PT is 1.49 F1 and the math gain over LoRA is only 0.44 accuracy; both are within the range that run-to-run variance can produce in instruction-tuned LLM fine-tuning. Because the reported margins and the per-method best-of-grid selection are jointly load-bearing for the central SOTA claim, the authors should provide multiple seeds, standard deviations, and paired significance tests, or a fixed-budget comparison with matched training steps, before the claim can be accepted.","section":"§5.1, §5.2, Appendix Table 8, Figure 4"},{"comment":"The average F1 gain of PT-MoE over PT (58.26 vs 56.77) is heavily influenced by DROP, where PT-MoE improves by 7.63 F1 points over PT (48.02 vs 40.39) while most other datasets show much smaller gains. A single favorable run on DROP could drive a large part of the average difference, and the paper does not report any variance information for individual datasets. The authors should report per-dataset error bars or show that the conclusion is robust to excluding DROP, and they should identify whether DROP's discrete-reasoning requirement is the reason for the large gain or whether it is an evaluation artifact.","section":"Table 2, §5.1"},{"comment":"The PT-MoE soft prompts are initialized via SVD of task-relevant text embeddings, which gives the initial prompt task-related information. The comparison against PT and SMoP does not control for this initialization advantage, so it is unclear how much of the reported improvement comes from the decomposition/MoE architecture rather than from the SVD initialization on task-related text. An ablation that matches initialization across PT, SMoP, DPT, and PT-MoE (e.g., random initialization or the same textual initialization for all methods) is needed to separate the architectural contribution from the initialization confound.","section":"§3, Matrix Decomposition; §5.1, Table 2"},{"comment":"The mathematical-reasoning comparison has two additional load-bearing weaknesses. First, the 0.44-point average accuracy gain over LoRA (56.91 vs 56.47) is smaller than plausible seed-level noise, and no significance test is provided. Second, the routing ablation shows that the best configuration uses 'selective and probationary' routing, but the main results do not report whether the router confidence multiplier (probationary routing) is applied at inference in all experiments; if it is not, the main accuracy numbers and the ablation numbers measure different inference procedures. The paper should state explicitly whether the multiplier is used in the reported main results and in the 3B model comparison in Table 6.","section":"§5.2, Table 4, §5.4, Routing mechanisms"}],"minor_comments":[{"comment":"The heading 'Evaluation Metrices' contains a typo; it should be 'Evaluation Metrics'.","section":"§4.3"},{"comment":"The dataset name 'SV AMP' should be written as 'SVAMP', and the reference for 'MATH_PROBLEMS (Nebrelbug, 2024)' is a Hugging Face Hub entry rather than a citable archival source; a stable dataset identifier or a more formal citation would improve reproducibility.","section":"Table 1"},{"comment":"The variable k is used both for the number of tokens per prompt and for the number of top-k router weights (Steps 7-8), which is confusing; rename one of them (e.g., k_tok and k_top).","section":"Algorithm 1"},{"comment":"The polynomial terms in the case study are typeset without superscripts (e.g., y4 instead of y^4), which makes the example harder to read and the claimed error analysis less clear.","section":"Table 5"},{"comment":"The SMoP reference is listed as 'The 2023 Conference on Empirical Methods in Natural Language Processing' without the usual EMNLP venue name and page/article details; the citation should be completed.","section":"§2"},{"comment":"The legend labels '(N)S: (Non-)Selective' and '(N)P: (Non-)Probationary' are hard to parse; the paper should spell out the four routing configurations explicitly (S/P, S/NP, NS/P, NS/NP) in the figure or caption.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The core idea is solid and the paper is clearly written, but the evaluation protocol is the main barrier to acceptance. I would encourage the editor to ask for repeated-seed experiments, significance tests, and a matched-initialization ablation; these are standard requirements for a SOTA claim in this area. The paper has a public code link, which is good, but the code needs to be checked to confirm that the reported numbers correspond to the released configuration, including the training-step search and the inference-time routing multiplier."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the PT-MoE design is genuinely new: a shared low-rank matrix B with per-prompt matrices A_i and a top-k router. That combination of decomposition and routing is absent from DPT and SMoP, the two closest prior works. Second, the reported performance claims are not supported by the evidence as presented. Every main-table number comes from one run, with training steps selected per method from a grid (Appendix Table 8) and the figure caption admitting these are the 'highest performance after training parameter search.' No error bars, no seeds, no significance tests.\n\nWhat the paper does well: the ablations are thorough and honestly reported. Varying prompt length, expert count, parameter count, routing mechanism, and model size gives useful design guidance. The math gain over standard prompt tuning is large (10.75 accuracy points), and the 3B model check shows the effect is not specific to the 1B base. Code and hyperparameters are in the appendix, which counts for something. The case study is illustrative rather than evidence, but it does show the mechanism in action.\n\nWhere it is soft: the QA head-to-head is the weakest part. The 1.49 F1 gain over PT is small enough to be seed noise, and DROP accounts for a disproportionate share of it—PT-MoE gets +7.63 F1 on DROP, which alone contributes about 0.6 points to the twelve-dataset average. The math advantage over LoRA is 0.44 accuracy points, effectively a tie. The baselines also omit several common PEFT families (adapters, BitFit, etc.), so 'state-of-the-art' needs to be read as 'best among this compared set.' The Limitations section talks about data licensing and open-source collaboration but says nothing about the missing error bars or the grid-search selection, which are the real limitations.\n\nThese are not fatal flaws. The architecture is sensible, the method probably does work, and the large math gain over PT is worth following up. But the central claim should be reworded to 'outperforms the baselines in this comparison' until multi-seed runs with a fixed budget confirm the margins.\n\nFor you: if you work on PEFT or prompt tuning, this is a useful paper to engage with. It deserves peer review—not a desk reject—but a serious referee should ask for repeated runs, fixed training budgets, and a comparison that includes contemporary PEFT baselines.","headline":"A genuinely new PEFT architecture, but the headline 'state-of-the-art' claim rests on single best-of-grid runs with no error bars—real concerns, though the design is sound and worth a serious referee.","tokens_in":15362,"tokens_out":2748,"would_cite":true,"duration_ms":29954,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PT-MoE claims to beat prompt tuning and LoRA on QA and math while using 25% fewer parameters than LoRA.","keywords":["prompt tuning","mixture-of-experts","matrix decomposition","parameter-efficient fine-tuning","question answering","mathematical reasoning","low-rank","routing"],"falsifier":"Re-run the full comparison with a fixed training budget per method, at least five random seeds, and report mean and standard deviation; if PT-MoE's margins over prompt tuning (1.49 F1 on QA, 10.75 accuracy on math) fall within the noise, the central SOTA claim is not supported. Alternatively, re-run with the same training step budget for all methods (no per-method grid) and check if the advantage persists.","tokens_in":14269,"feed_emoji":"🧩","tokens_out":8006,"duration_ms":64392,"temperature":0.7,"pith_summary":"The paper proposes PT-MoE, a prompt-tuning framework that decomposes each soft prompt into a shared low-rank matrix and prompt-specific matrices, then uses a learned router to select and combine these decomposed prompts for each input. The claim is that this combination of matrix decomposition and mixture-of-experts routing outperforms both plain prompt tuning and the LoRA adapter method on extractive question answering and mathematical problem solving, using only 80K trainable parameters—25% fewer than LoRA. The reported gains are a 1.49 F1-point improvement over prompt tuning and 2.13 points over LoRA on QA, and a 10.75-point accuracy improvement over prompt tuning on math. If correct, PT-MoE offers a single parameter-efficient method that stays competitive across two task families where the baselines each have separate strengths.","feed_headline":"PT-MoE beats prompt tuning and LoRA on QA and math","feed_subtitle":"Uses 25% fewer trainable parameters than LoRA while improving F1 by 1.49 points on QA.","key_machinery":"The central object is the decomposed-prompt mixture: each soft prompt $P_i \\in \\mathbb{R}^{T \\times H}$ is written as $A_i B$ with $A_i \\in \\mathbb{R}^{T \\times R}$ and a shared $B \\in \\mathbb{R}^{R \\times H}$; a router computes $w = \\mathrm{softmax}(W \\overline{x} + b)$ with top-$k$ hard selection and straight-through estimation, and the final prompt is the weighted combination. SVD of text-initialized embeddings provides the initialization. This carries the argument because it is what lets the method share parameters across experts (the shared $B$) while still routing inputs to specialized prompt components (the $A_i$), which the paper claims yields the complementary benefits.","core_discovery":"PT-MoE is a prompt tuning variant in which each of $N$ soft prompts is factored as $P_i = A_i B$, with a prompt-specific matrix $A_i$ and a shared matrix $B$ used by all prompts. A router maps the mean token embedding of the input to a distribution over prompts, applies top-$k$ hard selection during training with straight-through estimation, and the final soft prompt is the weighted sum $\\sum_i w_i A_i B$, prepended to the frozen language model's input. The authors claim this integration resolves two counter-intuitive observations—that adding a router to prompt tuning (SMoP) only helps in specific domains, and that low-rank decomposition (DPT) can improve performance in specific areas—by combining them. The evidence is a benchmark comparison across 12 QA datasets and 5 math datasets against prompt tuning, DPT, SMoP, ATTEMPT, LoRA, and HydraLoRA, all controlled to a similar parameter budget, with PT-MoE using 80K parameters.","pith_inferences":["A natural extension would be to apply the same decomposition-plus-routing recipe to LoRA itself—sharing the down-projection and routing among multiple up-projections—and check whether the complementary benefit transfers to adapter-based methods; the paper's HydraLoRA comparison is a different architecture and does not test this.","The much larger math gain over prompt tuning (10.75 points) compared to the QA gain (1.49 points) suggests PT-MoE may be addressing a specific failure mode of prompt tuning in multi-step reasoning, possibly through prompt specialization; a targeted analysis of where the errors disappear would test this.","A testable hypothesis implied by the design: the shared matrix $B$ captures task-agnostic prompt structure while the $A_i$ encode task-specific behavior. This could be checked by freezing $B$ and retraining only the $A_i$ on a new task, or by measuring the similarity of learned $A_i$ across tasks."],"forward_implications":["PT-MoE would give practitioners a single parameter-efficient method that works across extractive QA and mathematical reasoning, where plain prompt tuning and LoRA each have domain-specific advantages.","The result would demonstrate that matrix decomposition and MoE routing are complementary rather than redundant: decomposition enables parameter sharing through $B$, while routing gives dynamic input-dependent selection.","Ablation results suggest concrete design rules: a prompt length around 40 tokens, two experts for in-domain tasks, four for out-of-domain, and selective probationary routing (top-1 selection with confidence-scaled outputs).","At 80K parameters, PT-MoE would deliver these gains with 25% fewer parameters than LoRA in the tested setup, altering the parameter-efficiency tradeoff for PEFT."],"supporting_citations":[{"why":"Defines prompt tuning, the base method PT-MoE extends and the primary baseline it reports beating.","marker":"Lester et al., 2021"},{"why":"Introduces SMoP, the MoE-routing prompt tuning baseline that underperforms and motivates the combination.","marker":"Choi et al., 2023"},{"why":"Introduces DPT, the low-rank decomposition baseline that underperforms and motivates the decomposition component.","marker":"Xiao et al., 2023"},{"why":"Presents the MRQA benchmark that supplies the 12 QA datasets used in the evaluation.","marker":"Fisch et al., 2019"},{"why":"Provides GSM8K, the in-domain mathematical problem solving dataset used for training and evaluation.","marker":"Cobbe et al., 2021"},{"why":"Provides SQuAD, a key in-domain QA dataset within MRQA and a standard reading comprehension benchmark.","marker":"Rajpurkar et al., 2016"},{"why":"Releases the LLaMA-3.2 models used as the frozen base models for all methods.","marker":"Grattafiori et al., 2024"}],"fun_headline_variants":["PT-MoE: 25% fewer params, beats LoRA and PT on QA and math","Matrix-decomposed MoE prompt tuning tops QA and math benchmarks","PT-MoE: efficient MoE prompt tuning with SOTA QA and math results","PT-MoE fuses decomposition and MoE for stronger, leaner prompt tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains are not an artifact of the evaluation setup: each method's training steps were chosen from a small grid and results are single runs without error bars or significance tests, so the 1.49-point F1 and 10.75-point accuracy advantages over prompt tuning could in principle reflect best-of-grid selection or run-to-run variance.","fun_headline_variants_meta":{"raw":{"variants":["PT-MoE: 25% fewer params, beats LoRA and PT on QA and math","Matrix-decomposed MoE prompt tuning tops QA and math benchmarks","PT-MoE: efficient MoE prompt tuning with SOTA QA and math results","PT-MoE fuses decomposition and MoE for stronger, leaner prompt tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00092,"raw_usage":{"total_tokens":3983,"prompt_tokens":1017,"completion_tokens":2966,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":633,"completion_tokens_details":{"reasoning_tokens":2876}},"tokens_in":633,"tokens_out":2966,"duration_ms":20848,"temperature":1.0,"reasoning_tokens":2876,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:29:27.134508+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the full comparison with a fixed training budget per method, at least five random seeds, and report mean and standard deviation; if PT-MoE's margins over prompt tuning (1.49 F1 on QA, 10.75 accuracy on math) fall within the noise, the central SOTA claim is not supported. Alternatively, re-run with the same training step budget for all methods (no per-method grid) and check if the advantage persists.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces SMoP, the MoE-routing prompt tuning baseline that underperforms and motivates the combination."}],"review_version":1}