REVIEW 4 major objections 5 minor 2 cited by
Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Grouping tasks among trained experts lets a Decision Transformer keep improving as it scales to 160 tasks.
desk verdict A genuinely useful three-stage MoE-DT recipe whose own scaling evidence shows the router, not the experts, is the binding constraint. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the group-wise mixture-of-experts layer fused into every transformer block: the block output is $f(x)=x+f_{FFN}(x)+f_{MoE}(x)$ with $f_{MoE}(x)=\sum_i \mathrm{softmax}(f_{\theta_r}(x))_i\, f_{\theta_i}(x)$, so the original feed-forward network keeps shared knowledge while each expert specializes in its own task subset. Three-stage training—backbone first on all tasks and stopped early when gradient conflicts peak, then each expert alone on its own task subset, then the router alone on everything—is what makes the experts learn without the interference that end-to-end MoE training suffers. Task grouping is the other half: random splits already work, and grouping by K-means on per-task agreement-gradient vectors improves the score further. The machinery is doing two things at once: adding parameters cheaply through more experts and shrinking the effective task count each expert sees.
What would settle it
Train M3DT-Gradient on the same 160 tasks with 64 and 80 experts, beyond the 48 tested: the paper's central claim predicts overall normalized score keeps climbing, but if the score plateaus or falls while an oracle that selects by task ID keeps rising, the router rather than task load is the ceiling and the mutual-reinforcement claim is falsified. A supporting check is to watch the router's weight entropy: the claim requires the softmax over experts to stay informative, so a collapse toward near-uniform or near-hard one-hot assignments as experts grow would also count against it.
Extended reading notes
Core claim
On its own terms, the paper's central discovery is that scaling a Decision Transformer to massive multi-task learning fails when parameters are shared, but works when task grouping makes each parameter subset responsible for a few tasks: increasing the number of experts both adds parameters and shrinks each expert's task load, and the two effects reinforce each other. M3DT realizes this by keeping the Prompt-DT feed-forward network as a shared-knowledge backbone, attaching one expert module per task group, and adding a softmax router that blends all experts without task IDs. A three-stage schedule—backbone first, early-stopped just as gradient conflicts peak; experts next, each trained alone on its subset; router last, on all tasks—lets each module specialize without interference. The claim is backed by results showing M3DT outperforms all compared baselines at 10, 80, and 160 tasks, and that its 160-task score rises from about 72 to 78 as experts grow from 8 to 40.
Load-bearing premise
The load-bearing assumption is that the small MLP router, trained last with the backbone and experts frozen, can learn to weight the right experts for every task without being told which task it faces, and that this capacity keeps working as the number of experts grows; the paper's own figures show the learned router trailing a task-ID oracle, with the gap widening as experts increase.
Editorial extensions
If this is right
- Adding experts yields continued performance gains on fixed task sets: normalized score improves by about 11 percent on both 80 and 160 tasks before leveling off near 40 experts (Figure 6).
- Gradient-based grouping beats random grouping by 1.7 points at 80 tasks and 1.5 points at 160 tasks, and M3DT-Gradient's margin over the best baseline grows from 0.1 percent at 10 tasks to 7.5 percent at 160 tasks (Table 1).
- Each component of the three-stage recipe is load-bearing: end-to-end MoE training, training without explicit grouping, and unfreezing experts during router training all cut the score by roughly five to ten points (Table 2).
- Backbone training should stop when gradient conflicts peak; training it for too long or too short degrades the final policy (Figure 7).
- Sparse Top-4 routing breaks scaling as experts grow, so the dense weighted combination of all experts is what carries the reported result (Figure 9).
Reading between the lines
- If the router is the true bottleneck, as the paper's own Figure 8 suggests when the gap between oracle expert selection and the learned router widens with expert count, then scaling M3DT past 48 experts is a router-capacity problem rather than a task-grouping problem: a larger router, or one trained with auxiliary load-balancing, might extend the scaling curve without new algorithmic machinery.
- Because each expert is trained alone on frozen backbone features, the architecture is naturally plug-and-play: a new task arriving after deployment could be absorbed by training one fresh expert and then re-training only the router, turning massive multi-task learning into a cheap continual-learning procedure that the paper does not evaluate.
- The finding that gradient conflicts flatten after roughly 40 tasks implies task diversity has diminishing marginal difficulty; a direct check would be whether M3DT's margin over baselines grows monotonically with task count, which the 10/80/160 comparison suggests and would make the method most valuable in exactly the regime benchmarks rarely test.
- Dense routing means inference cost grows with every expert, so the authors' own Top-K failure leaves an open engineering question: a sparse gate with load balancing, inserted after the staged training, is a testable extension that would decide whether the method can serve massive task sets at bounded compute.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper studies scaling offline multi-task reinforcement learning to 160 continuous-control tasks. It first reports two empirical phenomena: task performance and gradient similarity degrade as the number of tasks grows, and naively widening a Decision Transformer quickly reaches a performance ceiling. Based on these observations, the paper proposes M3DT, a MoE-enhanced Prompt-DT in which each expert is trained on a small grouped task subset and a router, trained after freezing the backbone and experts, assigns softmax weights without task IDs. A three-stage training scheme (backbone, experts, router) is introduced, with task grouping either random or based on gradient agreement vectors. Experiments compare M3DT with MTDT, PromptDT, and HarmoDT at small and large sizes on 10, 80, and 160 tasks, and include ablations of the training stages and analyses of expert scaling and router design.
Significance. If the reported results hold, the paper makes a useful contribution to offline MTRL: it provides a systematic study of task-number and model-size scaling, proposes a modular MoE design with a three-stage training procedure, and reports consistent gains over DT-based baselines at 80 and 160 tasks. The paper also ships code, reports 3-seed means in the main table, includes domain-wise analyses, and openly discusses the router bottleneck and the failure of Top-K routing. These are genuine strengths. However, the central claim that increasing the number of experts 'mutually reinforces' performance through both parameter growth and reduced per-expert task load is only partially supported: the paper's own Figure 8 shows that the router becomes a growing bottleneck as experts increase, and several parameter/expert-count inconsistencies make the parameter-scalability numbers hard to interpret.
major comments (4)
- [A.6; Table 1; Figure 6] The parameter accounting is internally inconsistent. A.6 states that the M3DT backbone is the 5.29M PromptDT, while Figure 6's model-size labels imply an increment of about 25.25M per 8 experts (about 3.16M per expert). With that increment, the 8-expert M3DT reported at 47.87M in Table 1 implies a backbone of roughly 22.6M parameters, not 5.29M. Since the paper's parameter-scalability discussion in Section 5.1 is built on these numbers, the reader cannot currently determine which backbone size was actually used. Please reconcile the backbone size, per-expert size, and router size in a single consistent configuration table.
- [Section 5.1; Table 1; Figure 6] The expert count used for the headline results is ambiguous. The text says 'we use 40 experts in subsequent experiments unless stated otherwise,' but the 160-task M3DT rows in Table 1 report 174.12M parameters, which is the 48-expert point in Figure 6 (the 40-expert point is 148.87M). Please state which configuration produced the reported 7.5% gain at 160 tasks and whether the 40-expert results differ materially from the 48-expert results.
- [Section 5.2; Figures 8 and 9] The paper's own routing analysis weakens the abstract's claim that adding experts 'consistently enhances' performance by mutually reducing task load and increasing parameters. Figure 8 shows the gap between Oracle Expert Selection and actual overall M3DT performance widening as the number of experts grows, and Figure 9 shows that a sparse Top-4 router fails to scale. The text acknowledges this, but the conclusion still presents a smooth scaling story. The central assertion in Section 4.1 that increasing experts reduces task load per parameter subset and thereby reinforces performance should either be stated with the router bottleneck made explicit, or be supported by a quantitative routing-quality diagnostic (e.g., the oracle-vs-actual gap or routing entropy) across expert counts. As written, the 'mutually reinforce' mechanism is not established in the regime where routing difficulty grows.
- [Section 3.2; Figure 3] The claim that naively expanding parameters 'rapidly hits the performance ceiling' rests on Figure 3, which is presented without error bars. At 160 tasks the reported values for 20.02M, 77.78M, and 173.30M are 71.61, 71.68, and 71.65, differences of less than 0.1 normalized points; these are within the seed noise seen elsewhere in the paper (e.g., the standard deviations in Table 1). Since the paper states in A.6 that all experiments use 3 seeds, please add error bars or per-seed points to Figure 3. Similar concern applies to Figure 6, where the 40-vs-48-expert comparison is used to justify the 'diminishing gains' statement without error bars.
minor comments (5)
- [A.4] The heading 'Gradient Simalarity' should be 'Gradient Similarity'.
- [A.5, Table 4] Table 4's caption repeats 'DMControl tasks used in this paper'; the table actually lists model structures, so the caption should be corrected.
- [Section 5.1] The percentage improvements (0.1%, 4.3%, 5.4%) are not exactly recoverable from Table 1; for example, at 10 tasks the M3DT-Random advantage over MTDT-Large is under 0.2 normalized points. Please state explicitly which variant is being compared and round consistently.
- [B.5] In the expert-design experiment, the sentence comparing M3DT-Gradient-Big to 'M3DT-Random scored 77.89' appears to intend a comparison with M3DT-Gradient, whose Table 2 score is 77.89. Please correct the label.
- [Figure 6] The legend and axis labels are confusing because both panels repeat 'Parameter scalability on ...' and the horizontal lines are labeled only by task scale. Please clarify which curves correspond to M3DT-Random and M3DT-Gradient in each panel.
Circularity Check
M3DT's scaling claims rest on external baselines and ablations; no step reduces by construction to its own inputs.
full rationale
The paper's derivation chain is empirical rather than definitional: it builds a Dataset of 160 tasks, trains a Decision Transformer backbone, adds grouped experts, trains a router, and evaluates against external baselines (MTDT, PromptDT, HarmoDT). No reported quantity is defined in terms of the conclusion it supports. The central mechanism — increasing the number of experts reduces the number of tasks assigned to each expert — is a structural property of the task-grouping scheme, but the paper does not treat that property as a proof of performance; it tests the effect in Figure 6 and Table 1, and it reports diminishing returns beyond 40 experts and a widening router gap in Figures 8 and 9. These are honest empirical analyses, not circular derivations. The self-citations are not load-bearing: the agreement vector from Hu et al. (2024) is used only for the gradient-based grouping variant, the formula is restated in Appendix C, and the random-grouping variant M3DT-Random, which does not rely on that cited tool, also outperforms the baselines. The QPO citation (Kong et al., 2025) appears only in a related-work remark about text prompting and does not support any conclusion. The paper also does not invoke any uniqueness theorem or imported ansatz to force its choice of architecture; MoE is introduced as a design choice and ablated in Table 2. The parameter-count inconsistency between Appendix A.6 (5.29M backbone) and Table 1/Figure 6 is a reporting and reproducibility concern, not a circularity concern, because it does not affect the logical relation between inputs and claimed conclusions. Overall, the central claims are supported by controlled comparisons against external methods and by internal ablations, so the circularity burden is negligible.
Assumptions & free parameters
free parameters (3)
- Number of experts =
40 in text; 48 in Table 1 (174.12M at 160 tasks)
- Backbone training steps =
400,000
- K-means grouping for gradient-based assignment =
k equals the number of experts
assumptions (3)
- domain assumption Normalized scores from four benchmarks (Meta-World success rate, DMC reward scaling, Cheetah and Ant return mapping) are commensurable and equally weighted.
- domain assumption Gradient similarity (cosine between mean and per-task gradients) is a valid measure of task conflict.
- ad hoc to paper The early-stopped backbone (400k steps) retains enough shared knowledge for all tasks, so frozen experts can build on it.
Cite this review
Pith. "Pith review of Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer." pith.science (2026). https://pith.science/paper/NL47UEEL
@misc{pith2026250524378,
author = {Pith},
title = {Pith review of: Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer},
year = {2026},
howpublished = {\url{https://pith.science/paper/NL47UEEL}},
note = {Machine review of arXiv:2505.24378}
}
read the original abstract
Despite recent advancements in offline multi-task reinforcement learning (MTRL) have harnessed the powerful capabilities of the Transformer architecture, most approaches focus on a limited number of tasks, with scaling to extremely massive tasks remaining a formidable challenge. In this paper, we first revisit the key impact of task numbers on current MTRL method, and further reveal that naively expanding the parameters proves insufficient to counteract the performance degradation as the number of tasks escalates. Building upon these insights, we propose M3DT, a novel mixture-of-experts (MoE) framework that tackles task scalability by further unlocking the model's parameter scalability. Specifically, we enhance both the architecture and the optimization of the agent, where we strengthen the Decision Transformer (DT) backbone with MoE to reduce task load on parameter subsets, and introduce a three-stage training mechanism to facilitate efficient training with optimal performance. Experimental results show that, by increasing the number of experts, M3DT not only consistently enhances its performance as model expansion on the fixed task numbers, but also exhibits remarkable task scalability, successfully extending to 160 tasks with superior performance.
Figures
Figures from the paper (8 more)
Forward citations
Cited by 2 Pith papers
-
ACE-Brain-0.5: A Unified Embodied Foundational Model for Physical Agentic AI
A single 8B backbone unifies spatial perception, decision making, navigation/manipulation, and progress estimation with SSR+ merging, reporting gains on most spatial benchmarks and competitive action/progress results.
-
OPERA: Offline Policy-guided Expert Routing and Adaptation for Universal Biomedical Image Analysis
A validation-tuned, test-time adaptive ensemble of frozen biomedical vision experts improves classification, segmentation, and multimodal diagnosis across nine datasets without updating expert weights.
Reference graph
Works this paper leans on
-
[2]
Chen, Z., Ngiam, J., Huang, Y ., Luong, T., Kretzschmar, H., Chai, Y ., and Anguelov, D. Just pick a sign: Opti- 9 Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer mizing deep multitask models with gradient sign dropout. Advances in Neural Information Processing Systems, 33: 2039–2050,
-
[3]
For evaluation, we linearly scale the original reward range of [0,1000] to [0,100], using it as our normalized score. A.1.3. M UJOCO LOCOMOTION In this paper, we also employ a diverse array of meta-RL control tasks to construct a dataset with a sufficient number of tasks for exploring the challenges of MTRL when confronted with a large number of tasks. We...
work page 2022
-
[6]
Based on the reward ranges of these environments, we linearly map the return values within the interval [-100, -30] to the normalized range of [0, 100] as our normalized scores, while returns outside this range are directly capped at 0 or 100, respectively. • Ant-dir (Rothfuss et al., 2018): We also use 40 tasks in this domain, each with a goal direction ...
work page 2018
-
[7]
HarmoDT: Harmony Multi-Task Decision Transformer for Offline Reinforcement Learning
Hu, S., Fan, Z., Shen, L., Zhang, Y ., Wang, Y ., and Tao, D. Harmodt: Harmony multi-task decision trans- former for offline reinforcement learning. arXiv preprint arXiv:2405.18080,
-
[8]
Huang, S., Zhang, Z., Liang, T., Xu, Y ., Kou, Z., Lu, C., Xu, G., Xue, Z., and Xu, H. Mentor: Mixture-of-experts network with task-oriented perturbation for visual rein- forcement learning. arXiv preprint arXiv:2410.14972,
-
[9]
B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,
arXiv 2001
-
[11]
Offline rein- forcement learning: Tutorial, review, and perspectives on open problems
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643,
arXiv 2005
-
[12]
Obando-Ceron, J., Sokar, G., Willi, T., Lyle, C., Farebrother, J., Foerster, J., Dziugaite, G. K., Precup, D., and Castro, P. S. Mixtures of experts unlock parameter scaling for deep rl. arXiv preprint arXiv:2402.08609,
Show all 24 references
-
[13]
G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y ., Kay, J., Springenberg, J
10 Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y ., Kay, J., Springenberg, J. T., et al. A generalist agent. arXiv prep...
-
[14]
Promp: Proximal meta-policy search
Rothfuss, J., Lee, D., Clavera, I., Asfour, T., and Abbeel, P. Promp: Proximal meta-policy search. arXiv preprint arXiv:1810.06784,
-
[15]
Outrageously large neural networks: The sparsely-gated mixture-of-experts layer
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538,
-
[16]
Tassa, Y ., Doron, Y ., Muldal, A., Erez, T., Li, Y ., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. Deepmind control suite. arXiv preprint arXiv:1801.00690,
-
[17]
Mujoco: A physics engine for model-based control
Todorov, E., Erez, T., and Tassa, Y . Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems, pp. 5026–5033,
2012
-
[19]
Gradient surgery for multi-task learning
Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, C. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33: 5824–5836, 2020a. Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: A b...
2018
-
[20]
and utilize their dataset with the near-optimal trajectories, which consists of the experience from random to expert (convergence) in SAC-Replay (Haarnoja et al., 2018). The primary metric for evaluating performance in this benchmark is the average success rate across all task...
2018
-
[23]
Specifically, MTDT concatenates an embedding z and a state s as the input tokens, where z is the encoding of task ID
to learn from multitask data. Specifically, MTDT concatenates an embedding z and a state s as the input tokens, where z is the encoding of task ID. In evaluation, the reward-to-go and task ID are fed into the Transformer to provide task-specific information. Leveraging the sca...
2022
-
[24]
M3DT shows more superior performance on complex tasks (MW+DMC). B.4. Training Time Overhead and Computational Cost Our experiments are conducted on RTX 4090 GPU. In the first stage, we train PromptDT-5M for 4e5 steps, which takes approximately 5.2 hours. In the second stage, s...
2024
-
[2012]
Vaswani, A
doi: 10.1109/IROS.2012.6386109. Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems,
2012
-
[2018]
Td-mpc2: Scalable, ro- bust world models for continuous control
Hansen, N., Su, H., and Wang, X. Td-mpc2: Scalable, ro- bust world models for continuous control. arXiv preprint arXiv:2310.16828,
-
[2020]
Sharing knowledge in multi-task deep rein- forcement learning
D’Eramo, C., Tateo, D., Bonarini, A., Restelli, M., and Peters, J. Sharing knowledge in multi-task deep rein- forcement learning. arXiv preprint arXiv:2401.09561,
-
[2021]
D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901,
1901
-
[2022]
Gshard: Scaling giant models with conditional computation and automatic sharding
Lepikhin, D., Lee, H., Xu, Y ., Chen, D., Firat, O., Huang, Y ., Krikun, M., Shazeer, N., and Chen, Z. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668,
2006 arXiv
-
[2023]
Multi-task rein- forcement learning with mixture of orthogonal experts
Hendawy, A., Peters, J., and D’Eramo, C. Multi-task rein- forcement learning with mixture of orthogonal experts. arXiv preprint arXiv:2311.11385,
-
[2024]
Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin
Dou, S., Zhou, E., Liu, Y ., Gao, S., Shen, W., Xiong, L., Zhou, Y ., Wang, X., Xi, Z., Fan, X., et al. Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational...
1932
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.