{"id":"115619d1-db85-4c35-95de-05a9723ee736","arxiv_id":"2505.24378","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"M3DT combines a Decision Transformer with grouped, separately trained expert modules and a learned router, achieving better normalized scores than baselines across 10 to 160 multi-task RL tasks.","lead":"This paper trains a single decision-making model on 160 robot and control tasks at once using a mixture-of-experts design that splits tasks into groups, each handled by separate expert modules. The authors show their M3DT method keeps improving when more experts are added, outperforming standard decision transformers that degrade as tasks multiply.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Router capacity, not expert capacity, bounds M3DT's claimed scalability; the paper's own figures show the routing gap widening with expert count, so the 'mutually reinforce' mechanism is not established beyond 48 experts.","rationale":"The reader's weakest assumption correctly identifies the router as the pivot: the paper's claimed 'mutual reinforcement' between expert expansion and task-load reduction holds only if the router can keep up with the growing expert count. The paper's own figures provide direct evidence that the router does not keep up: the Oracle-versus-actual gap widens in Figure 8, and sparse routing fails in Figure 9. This is a stated limitation but it is load-bearing because the central claim is precisely that increasing experts improves performance; the improvement is real in the tested range but already saturating, and the causal mechanism breaks if the routing difficulty grows faster than expert gains. The concern does not overturn the empirical headline results at 160 tasks, because the method still beats DT-family baselines there; rather, it limits the generalizability of the scaling story and sharpens the conditions under which the claim is true. The additional parameter-count inconsistency (5.29M backbone stated versus ~22.6M implied by Figure 6) is a concrete reporting error that further weakens confidence in the quantitative scalability evidence, but it does not by itself refute the main empirical claims. Both issues support keeping the reader's CONDITIONAL verdict: the paper should either demonstrate router scalability beyond 48 experts (or qualify the claim), and it must correct the backbone parameter reporting and the 40-versus-48 expert inconsistency. My recommendation is UNCHANGED because these are exactly the kinds of conditions that the CONDITIONAL verdict already captures.","tokens_in":19424,"tokens_out":22096,"duration_ms":278661,"concrete_test":"Run M3DT-Gradient on the 160-task benchmark with 64 and 96 experts (adjusting task-group sizes accordingly), with at least 3 seeds and full error bars; simultaneously record Oracle Expert Selection and actual router-based scores. If the overall score decreases or plateaus while the Oracle curve keeps rising, the router is the binding constraint and the 'mutually reinforce' claim fails for larger expert counts. As a complementary check, compute the top-1 routing accuracy (fraction of tasks where the highest-weight expert is the correct one) as a function of expert count; if it declines with N, the router capacity is the limiting factor. Also inspect the released code to verify the backbone parameter count, resolving the 5.29M vs ~22.6M discrepancy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that increasing the number of experts reduces per-expert task load and thereby improves M3DT's overall performance requires that the last-stage MLP router, trained with frozen backbone and experts and without task IDs, can assign accurate softmax weights over all experts for every task, and that this routing quality persists as the expert count grows. The paper's own evidence contradicts the persistence of this assumption: Section 5.2 (Figure 8) shows the gap between Oracle Expert Selection and actual overall performance widening with expert count, and Figure 9 shows a sparse Top-4 router failing to scale; Section 5.1 reports diminishing gains after 40 experts on 160 tasks. Thus the benefit of expert expansion is already being offset by routing difficulty at the top of the tested range. Without evidence that dense softmax routing keeps improving (or at least not degrading) as N grows, the abstract's claim of \"consistent enhancement\" with expert count and the paper's scalability story remain unproven beyond 48 experts. A separate reporting inconsistency compounds this: A.6 states the backbone is 5.29M parameters, but the parameter counts in Table 1 and Figure 6 (47.87M at 8 experts, 174.12M at 48 experts, each expert adding about 3.16M) imply a backbone of roughly 22.6M parameters, not 5.29M, making the parameter-scalability numbers difficult to interpret.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies scaling offline multi-task reinforcement learning to 160 continuous-control tasks. It first reports two empirical phenomena: task performance and gradient similarity degrade as the number of tasks grows, and naively widening a Decision Transformer quickly reaches a performance ceiling. Based on these observations, the paper proposes M3DT, a MoE-enhanced Prompt-DT in which each expert is trained on a small grouped task subset and a router, trained after freezing the backbone and experts, assigns softmax weights without task IDs. A three-stage training scheme (backbone, experts, router) is introduced, with task grouping either random or based on gradient agreement vectors. Experiments compare M3DT with MTDT, PromptDT, and HarmoDT at small and large sizes on 10, 80, and 160 tasks, and include ablations of the training stages and analyses of expert scaling and router design.","tokens_in":19738,"tokens_out":9125,"duration_ms":105640,"significance":"If the reported results hold, the paper makes a useful contribution to offline MTRL: it provides a systematic study of task-number and model-size scaling, proposes a modular MoE design with a three-stage training procedure, and reports consistent gains over DT-based baselines at 80 and 160 tasks. The paper also ships code, reports 3-seed means in the main table, includes domain-wise analyses, and openly discusses the router bottleneck and the failure of Top-K routing. These are genuine strengths. However, the central claim that increasing the number of experts 'mutually reinforces' performance through both parameter growth and reduced per-expert task load is only partially supported: the paper's own Figure 8 shows that the router becomes a growing bottleneck as experts increase, and several parameter/expert-count inconsistencies make the parameter-scalability numbers hard to interpret.","major_comments":[{"comment":"The parameter accounting is internally inconsistent. A.6 states that the M3DT backbone is the 5.29M PromptDT, while Figure 6's model-size labels imply an increment of about 25.25M per 8 experts (about 3.16M per expert). With that increment, the 8-expert M3DT reported at 47.87M in Table 1 implies a backbone of roughly 22.6M parameters, not 5.29M. Since the paper's parameter-scalability discussion in Section 5.1 is built on these numbers, the reader cannot currently determine which backbone size was actually used. Please reconcile the backbone size, per-expert size, and router size in a single consistent configuration table.","section":"A.6; Table 1; Figure 6"},{"comment":"The expert count used for the headline results is ambiguous. The text says 'we use 40 experts in subsequent experiments unless stated otherwise,' but the 160-task M3DT rows in Table 1 report 174.12M parameters, which is the 48-expert point in Figure 6 (the 40-expert point is 148.87M). Please state which configuration produced the reported 7.5% gain at 160 tasks and whether the 40-expert results differ materially from the 48-expert results.","section":"Section 5.1; Table 1; Figure 6"},{"comment":"The paper's own routing analysis weakens the abstract's claim that adding experts 'consistently enhances' performance by mutually reducing task load and increasing parameters. Figure 8 shows the gap between Oracle Expert Selection and actual overall M3DT performance widening as the number of experts grows, and Figure 9 shows that a sparse Top-4 router fails to scale. The text acknowledges this, but the conclusion still presents a smooth scaling story. The central assertion in Section 4.1 that increasing experts reduces task load per parameter subset and thereby reinforces performance should either be stated with the router bottleneck made explicit, or be supported by a quantitative routing-quality diagnostic (e.g., the oracle-vs-actual gap or routing entropy) across expert counts. As written, the 'mutually reinforce' mechanism is not established in the regime where routing difficulty grows.","section":"Section 5.2; Figures 8 and 9"},{"comment":"The claim that naively expanding parameters 'rapidly hits the performance ceiling' rests on Figure 3, which is presented without error bars. At 160 tasks the reported values for 20.02M, 77.78M, and 173.30M are 71.61, 71.68, and 71.65, differences of less than 0.1 normalized points; these are within the seed noise seen elsewhere in the paper (e.g., the standard deviations in Table 1). Since the paper states in A.6 that all experiments use 3 seeds, please add error bars or per-seed points to Figure 3. Similar concern applies to Figure 6, where the 40-vs-48-expert comparison is used to justify the 'diminishing gains' statement without error bars.","section":"Section 3.2; Figure 3"}],"minor_comments":[{"comment":"The heading 'Gradient Simalarity' should be 'Gradient Similarity'.","section":"A.4"},{"comment":"Table 4's caption repeats 'DMControl tasks used in this paper'; the table actually lists model structures, so the caption should be corrected.","section":"A.5, Table 4"},{"comment":"The percentage improvements (0.1%, 4.3%, 5.4%) are not exactly recoverable from Table 1; for example, at 10 tasks the M3DT-Random advantage over MTDT-Large is under 0.2 normalized points. Please state explicitly which variant is being compared and round consistently.","section":"Section 5.1"},{"comment":"In the expert-design experiment, the sentence comparing M3DT-Gradient-Big to 'M3DT-Random scored 77.89' appears to intend a comparison with M3DT-Gradient, whose Table 2 score is 77.89. Please correct the label.","section":"B.5"},{"comment":"The legend and axis labels are confusing because both panels repeat 'Parameter scalability on ...' and the horizontal lines are labeled only by task scale. Please clarify which curves correspond to M3DT-Random and M3DT-Gradient in each panel.","section":"Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The empirical core of the paper is plausible and the ablations are informative, but I cannot recommend acceptance before the parameter/expert-count inconsistencies are resolved and the scaling claims are qualified by the paper's own router analysis. These issues are substantive but fixable with a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this paper gives offline multi-task RL a genuinely new training recipe—three-stage sequential training of backbone, task-grouped experts, and a router—and backs it with careful ablations. The scaling claim in the abstract, though, goes beyond what the data show. By the paper's own Figure 8, the routing gap widens as expert count grows, so \"mutually reinforce\" stops being true in the upper range. The method still wins on 160 tasks, but the mechanism is more \"delay the ceiling\" than \"unlock infinite scaling.\"\n\nWhat's new: the grouped-task assignment plus frozen-backbone expert training plus a frozen-everything router stage is not in the cited MoE-RL works. The scaling analysis (tasks from 10 to 160, model width up to 173M) contributes a useful empirical observation: naively widening PromptDT hits a ceiling beyond ~20M parameters. The ablations in Table 2 are clean—each component (grouping, expert freezing, three-stage) buys a clear margin. Code is available, and the domain-wise appendix tables check out. The authors also list honest limitations in Section 7 (no held-out generalization, inference cost, Top-K router fails).\n\nSoft spots, proportionate: (1) The expert count reporting is inconsistent—text says 40 experts for \"subsequent experiments,\" but Table 1 and Figure 6 use parameter counts that match 48. That needs fixing. (2) The parameter counts don't add up with the stated 5.29M backbone in A.6; Table 1's numbers imply a backbone of roughly 22.6M. That muddies the \"efficient parameter scaling\" argument. (3) Figure 3, the model-scaling heatmap, has no error bars; the ceiling conclusion rests on a single run per cell. (4) The comparison set is DT-family only; no empirical head-to-head with Obando-Ceron's MoE-RL or Mentor, which are cited. (5) The abstract's \"consistently enhances\" is too strong given the plateaus in Figure 6; the paper's own Section 5.2 concedes the router is the bottleneck.\n\nNone of these are load-bearing failures. The three-stage recipe is real, and the 160-task result with gradient grouping (78.21 vs 72.80 for the best baseline) is meaningful. The stress-test note's concern about router capacity is exactly right, and the paper does not hide it. I'd send this to peer review, with the expectation of moderate revisions: fix the inconsistencies, add error bars to Figure 3, temper the abstract, and ideally add one non-DT baseline.\n\nWho's it for: anyone working on scaling offline multi-task RL or MoE for control. Worth a serious referee.","headline":"A genuinely useful three-stage MoE-DT recipe whose own scaling evidence shows the router, not the experts, is the binding constraint.","tokens_in":20277,"tokens_out":4516,"would_cite":true,"duration_ms":50370,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Grouping tasks among trained experts lets a Decision Transformer keep improving as it scales to 160 tasks.","keywords":["multi-task reinforcement learning","offline reinforcement learning","Decision Transformer","mixture-of-experts","task grouping","parameter scaling","task scalability"],"falsifier":"Train M3DT-Gradient on the same 160 tasks with 64 and 80 experts, beyond the 48 tested: the paper's central claim predicts overall normalized score keeps climbing, but if the score plateaus or falls while an oracle that selects by task ID keeps rising, the router rather than task load is the ceiling and the mutual-reinforcement claim is falsified. A supporting check is to watch the router's weight entropy: the claim requires the softmax over experts to stay informative, so a collapse toward near-uniform or near-hard one-hot assignments as experts grow would also count against it.","tokens_in":19223,"feed_emoji":"🤖","tokens_out":12783,"duration_ms":129427,"temperature":0.7,"pith_summary":"The paper argues that the obstacle to scaling offline multi-task reinforcement learning to hundreds of tasks is not model capacity but the task load each parameter must carry, and that the cure is to spread the tasks across many expert modules that are trained separately. It supports this with a 160-task control suite spanning three benchmarks, showing that a plain Decision Transformer degrades as tasks accumulate and that simply widening the network hits a performance ceiling around 20 million parameters. Its method, M3DT, adds a mixture-of-experts layer to a Prompt-DT backbone, assigns each expert a small task group by random or gradient-based clustering, and trains the backbone, the experts, and the router in three separate stages. The payoff is that performance keeps climbing as experts are added, and M3DT-Gradient beats the strongest baseline by about 7.5 percent at 160 tasks. If this is right, the practical route to 'many tasks, one agent' is parameter separation plus staged training, not pure scale.","feed_headline":"Grouped experts push a Decision Transformer to 160 tasks","feed_subtitle":"Splitting tasks across expert groups beats widening one network, gaining 7.5 percent over the best baseline at 160 tasks","key_machinery":"The load-bearing object is the group-wise mixture-of-experts layer fused into every transformer block: the block output is $f(x)=x+f_{FFN}(x)+f_{MoE}(x)$ with $f_{MoE}(x)=\\sum_i \\mathrm{softmax}(f_{\\theta_r}(x))_i\\, f_{\\theta_i}(x)$, so the original feed-forward network keeps shared knowledge while each expert specializes in its own task subset. Three-stage training—backbone first on all tasks and stopped early when gradient conflicts peak, then each expert alone on its own task subset, then the router alone on everything—is what makes the experts learn without the interference that end-to-end MoE training suffers. Task grouping is the other half: random splits already work, and grouping by K-means on per-task agreement-gradient vectors improves the score further. The machinery is doing two things at once: adding parameters cheaply through more experts and shrinking the effective task count each expert sees.","core_discovery":"On its own terms, the paper's central discovery is that scaling a Decision Transformer to massive multi-task learning fails when parameters are shared, but works when task grouping makes each parameter subset responsible for a few tasks: increasing the number of experts both adds parameters and shrinks each expert's task load, and the two effects reinforce each other. M3DT realizes this by keeping the Prompt-DT feed-forward network as a shared-knowledge backbone, attaching one expert module per task group, and adding a softmax router that blends all experts without task IDs. A three-stage schedule—backbone first, early-stopped just as gradient conflicts peak; experts next, each trained alone on its subset; router last, on all tasks—lets each module specialize without interference. The claim is backed by results showing M3DT outperforms all compared baselines at 10, 80, and 160 tasks, and that its 160-task score rises from about 72 to 78 as experts grow from 8 to 40.","pith_inferences":["If the router is the true bottleneck, as the paper's own Figure 8 suggests when the gap between oracle expert selection and the learned router widens with expert count, then scaling M3DT past 48 experts is a router-capacity problem rather than a task-grouping problem: a larger router, or one trained with auxiliary load-balancing, might extend the scaling curve without new algorithmic machinery.","Because each expert is trained alone on frozen backbone features, the architecture is naturally plug-and-play: a new task arriving after deployment could be absorbed by training one fresh expert and then re-training only the router, turning massive multi-task learning into a cheap continual-learning procedure that the paper does not evaluate.","The finding that gradient conflicts flatten after roughly 40 tasks implies task diversity has diminishing marginal difficulty; a direct check would be whether M3DT's margin over baselines grows monotonically with task count, which the 10/80/160 comparison suggests and would make the method most valuable in exactly the regime benchmarks rarely test.","Dense routing means inference cost grows with every expert, so the authors' own Top-K failure leaves an open engineering question: a sparse gate with load balancing, inserted after the staged training, is a testable extension that would decide whether the method can serve massive task sets at bounded compute."],"forward_implications":["Adding experts yields continued performance gains on fixed task sets: normalized score improves by about 11 percent on both 80 and 160 tasks before leveling off near 40 experts (Figure 6).","Gradient-based grouping beats random grouping by 1.7 points at 80 tasks and 1.5 points at 160 tasks, and M3DT-Gradient's margin over the best baseline grows from 0.1 percent at 10 tasks to 7.5 percent at 160 tasks (Table 1).","Each component of the three-stage recipe is load-bearing: end-to-end MoE training, training without explicit grouping, and unfreezing experts during router training all cut the score by roughly five to ten points (Table 2).","Backbone training should stop when gradient conflicts peak; training it for too long or too short degrades the final policy (Figure 7).","Sparse Top-4 routing breaks scaling as experts grow, so the dense weighted combination of all experts is what carries the reported result (Figure 9)."],"supporting_citations":[{"why":"Decision Transformer, the sequence-modeling formulation of offline RL that the MTDT baseline and M3DT build on.","marker":"(Chen et al., 2021)"},{"why":"Prompt-DT, the trajectory-prompted backbone M3DT augments with experts, its main baseline, and the source of the Cheetah-vel and Ant-dir datasets.","marker":"(Xu et al., 2022)"},{"why":"HarmoDT, a task-mask baseline, and the source of the agreement-vector gradient measure used for gradient-based task grouping.","marker":"(Hu et al., 2024)"},{"why":"Switch Transformers, cited as evidence that expert count is the most efficient lever for parameter scaling.","marker":"(Fedus et al., 2022)"},{"why":"Sparsely-gated mixture-of-experts, the Top-K routing scheme whose failure to scale M3DT is documented in Figure 9.","marker":"(Shazeer et al., 2017)"},{"why":"Prior demonstration that MoE unlocks parameter scaling in deep RL and that Top-K routers are unstable during training.","marker":"(Obando-Ceron et al., 2024)"},{"why":"Meta-World, one of the three benchmark families composing the 160-task training set.","marker":"(Yu et al., 2020b)"},{"why":"DMControl, the second benchmark family and the source of the reward-normalization protocol.","marker":"(Tassa et al., 2018)"},{"why":"Supplies the DMControl offline dataset and the custom-task list that extends that domain to 30 control tasks.","marker":"(Hansen et al., 2023)"}],"fun_headline_variants":["Mixture-of-experts scales decision transformer to 160 tasks","Grouped experts beat shared parameters at massive multi-task RL","MoE Decision Transformer gains 7.5% at 160 tasks","Task grouping unlocks parameter scaling for multi-task RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the small MLP router, trained last with the backbone and experts frozen, can learn to weight the right experts for every task without being told which task it faces, and that this capacity keeps working as the number of experts grows; the paper's own figures show the learned router trailing a task-ID oracle, with the gap widening as experts increase.","fun_headline_variants_meta":{"raw":{"variants":["Mixture-of-experts scales decision transformer to 160 tasks","Grouped experts beat shared parameters at massive multi-task RL","MoE Decision Transformer gains 7.5% at 160 tasks","Task grouping unlocks parameter scaling for multi-task RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000152,"raw_usage":{"total_tokens":1204,"prompt_tokens":946,"completion_tokens":258,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":189}},"tokens_in":562,"tokens_out":258,"duration_ms":3577,"temperature":1.0,"reasoning_tokens":189,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:24:50.082639+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train M3DT-Gradient on the same 160 tasks with 64 and 80 experts, beyond the 48 tested: the paper's central claim predicts overall normalized score keeps climbing, but if the score plateaus or falls while an oracle that selects by task ID keeps rising, the router rather than task load is the ceiling and the mutual-reinforcement claim is falsified. A supporting check is to watch the router's weight entropy: the claim requires the softmax over experts to stay informative, so a collapse toward near-uniform or near-hard one-hot assignments as experts grow would also count against it.","supporting_citations":[],"review_version":1}