Multi-Task GRPO: Reliable LLM Reasoning Across Tasks

Aurelien Lucchi; Haitham Bou Ammar; Ilija Bogunovic; Matthieu Zimmer; Sangwoong Yoon; Shyam Sundhar Ramesh; Xiaotong Ji; Zhiyong Wang

Multi-Task GRPO: Reliable LLM Reasoning Across Tasks

T0 review · 4 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Multi-task GRPO claims it can lift worst-task accuracy by 16–28% over standard GRPO and 6% over DAPO while keeping average accuracy competitive.

desk verdict Plausible practical recipe for multi-task GRPO, but the 'improvement' signal is a weak reed and the empirical claims rest on single runs. read the letter →

arxiv 2602.05547 v3 pith:JV7MQWMU submitted 2026-02-05 cs.CL cs.AIcs.LG

Shyam Sundhar Ramesh , Xiaotong Ji , Matthieu Zimmer , Sangwoong Yoon , Zhiyong Wang , Haitham Bou Ammar , Aurelien Lucchi , Ilija Bogunovic This is my paper

classification cs.CLcs.AIcs.LG

keywords multi-taskreinforcementlearningGRPOLLMreasoningworst-taskaccuracytaskreweightingzero-gradientpromptsratio-preservingsamplingdistributionallyrobustoptimization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that the standard recipe for applying GRPO to multiple reasoning tasks at once—averaging per-task losses and sampling uniformly—quietly abandons the weakest tasks. The authors propose Multi-Task GRPO, which does two things: it reweights tasks each step using both each task's reward and how much that task improved over the previous update, and it uses a ratio-preserving sampler so the intended task proportions survive the zero-gradient filtering that GRPO performs. In experiments post-training a 3B model on three and nine reasoning tasks (Countdown, Zebra, ARC), the method reports 16–28% absolute gains in worst-task accuracy over standard GRPO, 6% over DAPO, similar average accuracy, and reaches a given worst-task threshold in half the steps. A sympathetic reader would care because it offers a concrete path to LLM post-training that optimizes reliability across skills rather than average benchmark scores.

What carries the argument

The load-bearing objects are (1) the improvement-aware weight update IWU (Subroutine 1), which uses the per-task improvement I_k^{(t)} = J_GRPO,k(θ_{t+1}) − J_GRPO,k(θ_t) added to λ J_k(θ_t) as the signal for softmax logits ξ, and (2) the Ratio-Preserving Sampler (Algorithm 2), which draws desired post-filtered counts from Multinomial(B, z), inflates sampling by estimated filtering rate (capped at M_acc), and resamples by deficiency until counts are met or budget exhausted. The sampler is what translates the adapted weights into realized gradient proportions.

What would settle it

Take two tasks and set one to always produce all-correct rollouts and the other all-wrong rollouts (or filter both out). Both have J_GRPO = 0, so IWU sees no difference and cannot upweight the failing task; a run would show worst-task accuracy stagnating. Or, without such extreme settings, compare I_k^{(t)} to held-out task accuracy changes across 100 steps; if the correlation is near zero, the reweighting is not the cause of the reported gains.

Watch

Extended reading notes

Core claim

MT-GRPO claims that robustness in multi-task RL post-training can be achieved by decoupling task reweighting from the GRPO loss, which the authors show is unusable for comparing tasks because it evaluates to zero both when a task's rollouts are all wrong and all correct. Instead, task weights are updated from true task rewards J_k plus an improvement signal I_k = J_GRPO,k(θ_{t+1}) − J_GRPO,k(θ_t) (Subroutine 1), which upweights tasks that are both weak and improving slowly, while a ratio-preserving sampler oversamples high filtering-rate tasks and resamples until post-filtered batch counts match a Multinomial(B,z) draw. The paper's experiments claim this yields consistent worst-task accuracy

Load-bearing premise

The load-bearing premise is that the per-step improvement signal I_k^{(t)}—the difference of GRPO objectives before and after an update—accurately tells how much each task is learning; if a failing task's GRPO objective sits at zero (or is filtered away), it can look identical to a task that is already solved.

Editorial extensions

If this is right

A single λ controls the trade-off between worst-task robustness and average accuracy, so practitioners can dial how much to protect weak tasks.
Post-filtering zero-gradient prompts no longer silently skew training toward easy, low-filtering tasks.
Worst-task accuracy targets can be reached in roughly half the steps in the 3-task setting, reducing compute for reliable multi-task models.
The approach extends to 9 tasks across difficulty levels, suggesting it scales beyond a few curated tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not test the case where a task is genuinely stuck at all-correct or all-wrong rollouts; in that regime I_k is zero by construction, so the reweighting would treat a completely failing task the same as a perfectly solved one — an easy ablation would reveal whether this matters.
The improvement signal is estimated from a single policy step; a natural extension is to use a moving average of I_k or holdout rewards to reduce noise before deciding to upweight a task.
The ratio-preserving sampler is a general fix for any GRPO-style method that filters zero-advantage prompts, so it could be dropped into single-task training to control which prompts survive, not just multi-task.
If worst-task accuracy is the right reliability metric, the same reweighting loop could be applied to other RL post-training objectives (PPO, RLOO) wherever the per-task loss has similar zero-ambiguity.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The genuinely useful piece here is the ratio-preserving sampler. The paper correctly observes that zero-gradient filtering distorts the effective task mix in multi-task GRPO, and enforcing post-filtered task proportions is a real fix. That alone is a worthwhile practical contribution, and the ablations (with vs. without RPS) suggest it matters. The reweighting objective itself is standard DRO/Famo, but adapting it to GRPO post-training is a reasonable thing to try, and the experiments on 3 and 9 tasks at least show consistent directional gains in worst-task accuracy.

That said, the central narrative is oversold. The paper says MT-GRPO reallocates effort based on improvement-aware signals, but the improvement signal I_k defined in Eq. (8) inherits exactly the zero-ambiguity the authors themselves flag in §3.1. With the KL coefficient set to zero and mean-centered group advantages, J_GRPO(θ_t) is zero at the behavior policy, so I_k collapses to the one-step surrogate J_GRPO(θ_{t+1})—not a measure of learning progress. The clipping to [0.1, 0.2] in the implementation leaves hardly any room for this signal to influence the weight update. In practice, the reweighting is probably being driven by the reward term λJ_k, and the 'improvement' component is largely decoration. The stress-test note is right about this.

The empirical base is also thinner than the abstract sounds. All accuracy numbers appear to be single runs without seeds or error bars, so the claimed 6% absolute gain over DAPO could easily be noise. The task coverage is narrow (three task families from one suite), and the stronger claims about 'consistently outperforming' need replication across more tasks and random seeds.

None of this kills the method—reward-based reweighting plus the sampler may well work—but the paper as written overclaims the mechanism. The fix is straightforward: run multiple seeds, report error bars, and include an ablation that isolates reward reweighting from the improvement signal. If the improvement signal does no causal work, the paper should be re-scoped around the sampler and the reward-based reweighting.

This is worth sending to peer review. The problem is relevant, the sampler is a real idea, and the zero-gradient imbalance issue deserves discussion. But it needs a serious referee who will push for multi-seed results and a cleaner decomposition of what actually drives the gains.

Referee Report

4 major / 4 minor

Summary. The paper proposes Multi-Task GRPO (MT-GRPO), a post-training algorithm for multi-task LLM reasoning. MT-GRPO combines improvement-aware task reweighting (Subroutine 1) with a ratio-preserving sampler (Algorithm 2) to address two issues in multi-task GRPO: average-reward optimization neglecting worst-task performance, and uneven zero-gradient rates causing realized batch proportions to deviate from intended task weights. The authors evaluate on Qwen-2.5-3B in a 3-task setting (Countdown, Zebra, ARC) and a 9-task setting (easy/medium/hard variants), reporting consistent worst-task accuracy improvements over GRPO, SEC-GRPO, DAPO, and SEC-DAPO, with a claimed 16–28% absolute worst-task improvement over GRPO and 6% over DAPO, while maintaining competitive average accuracy. The paper also provides a theoretical derivation of the objective from a constrained formulation and appendices with ablations.

Significance. If the empirical claims hold, MT-GRPO offers a practical, lightweight modification to GRPO for improving robustness across heterogeneous reasoning tasks. The ratio-preserving sampler directly targets a real structural problem in multi-task GRPO (zero-gradient filtering skewing task proportions), and the paper provides code. However, the central 'improvement-aware' mechanism rests on an unreliable signal: Eq. (8) inherits the zero-ambiguity of the GRPO loss that the paper itself flags in §3.1, and all headline results appear to be single runs with no error bars. The method is plausible and potentially useful, but the evidence as presented is not yet convincing enough for publication without substantial revision.

major comments (4)

[§3.1, §3.2, Eq. (8)] The paper's own §3.1 argues that the GRPO loss is unreliable for task reweighting because it evaluates to zero both when all rollouts are incorrect and when all are correct. Yet Eq. (8) defines the improvement signal I_k^{(t)} as the difference of J_GRPO,k between consecutive iterates, and Subroutine 1 uses this signal to reweight tasks. With the KL coefficient set to 0 (Appendix D) and group-mean-centered advantages, J_GRPO,k(θ_t) is zero by construction, so I_k^{(t)} reduces to J_GRPO,k(θ_{t+1}) — a one-step importance-weighted surrogate, not a measurement of task-wise learning progress. For any prompt with constant rewards across rollouts, I_k contributes zero whether the task is perfectly solved or completely stuck. Thus the 'improvement-aware' reweighting does not actually measure improvement as claimed. The λ J_k(θ_t) term can partly compensate, but then the mechanism is reward-bas
[Appendix D, Subroutine 1] The improvement signal I_k^{(t)} is clipped to [0.1, 0.2] for stability, giving a maximum dynamic range of 0.1. For the λ values used (0.2, 0.25, 0.9, 1.2) and J_k in [0,1], the reward term λ J_k(θ_t) dominates s_k^{(t)} in magnitude. The ablation in Appendix E compares Subroutine 1 (improvement-aware) with Subroutine 2 (regularized reward-only), but these subroutines differ in multiple respects: presence of I_k, softmax-centering structure, and the regularization hyperparameter η. This is not a controlled ablation of the improvement term. To support the claim that I_k is doing causal work, the authors should run a controlled comparison (same update structure with and without I_k, and with I_k unclipped or rescaled) across multiple seeds.
[§6, Figures 4–11] All reported accuracies appear to be single runs with no standard errors or confidence intervals. The headline claim of '6% absolute improvement over DAPO' (abstract, §6.2) is a difference between two point estimates. GRPO training on LLMs is seed-sensitive, and the 6% gap may be within run-to-run noise. Because the central claim is a comparative empirical claim ('consistently outperforms baselines'), the paper should report at least 3 seeds with mean and standard deviation (or another appropriate uncertainty measure) for the main results. This is load-bearing; without it, the robustness gains are not established.
[Abstract, §6.2, Figure 7] The abstract and §6.2 state that MT-GRPO maintains 'competitive average accuracy' while improving worst-task accuracy. In the 9-task experiment, MT-GRPO with λ=1.2 has average accuracy 61.7 versus DAPO's 68.7, a drop of 7 points. This may be an acceptable trade-off, but 'competitive' is misleading for that setting. The paper should qualify this claim and discuss how the trade-off scales with λ, perhaps by reporting the best λ for each notion of competitive average accuracy.

minor comments (4)

[Eq. (11)] Eq. (11) contains a redundant repetition: 'A(x, y_i) = A(x, y_i) = ...'. Please clean up the notation.
[Figure 4/7 captions] The multi-panel captions are dense and the bars are not all individually labeled in the text. Adding explicit numeric accuracy values in the caption or body text would improve readability.
[§B.1] In the derivation of Ω(z), the notation around the double sum Σ_{k,j} μ_{kj} is used inconsistently ('Σ_{k=1}^K Σ_{j=1}^K' vs 'Σ_{k,j}'). Please standardize to avoid confusion.
[§6.1, Figure 5] The claim '50% fewer training steps' should specify the baseline and threshold explicitly. It is stated vaguely in the text and the figure mixes methods; please clarify.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MT-GRPO's derivation is self-contained; the Eq. (8) improvement-signal concern is a non-circular correctness issue.

full rationale

MT-GRPO's derivation chain is self-contained. The robust objective (Eqs. 4–5) is obtained by a standard Lagrangian relaxation derived in Sec. B.1; the alternating updates (Eqs. 6–7) follow from softmax-parameterized gradient descent on the inner objective; and the improvement-aware update in Subroutine 1 is derived in Sec. B.3 from a first-order Taylor expansion of a per-step minimax objective, not assumed as a conclusion. The empirical evaluation uses external ReasoningGym datasets and benchmarks against GRPO/DAPO baselines that do not use MT-GRPO's task weights; no parameter is fitted to the reported worst-task accuracies and then relabeled as a prediction. There are no load-bearing self-citations: the only borrowed conceptual ingredient, the task-level improvement signal, is cited to Famo (Liu et al. [11]), whose authors do not overlap with this paper. The one concern worth flagging is non-circular: Sec. 3.1 warns that J_GRPO is zero-ambiguous for cross-task comparison, and Eq. (8) defines I_k as a difference of J_GRPO values; with the KL coefficient set to 0 (Appendix D) and mean-centered group advantages, J_GRPO,k(θ_t) is zero by construction, so I_k collapses to a one-step surrogate inheriting the same ambiguity. This is a signal-quality / correctness risk for the mechanism, not a circular reduction: the central claim is not forced by its inputs, the comparisons are external, and the paper's reported gains stand or fall on reproducibility rather than on a definitional equivalence.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central algorithm depends on several tuned hyperparameters (λ, clipping bounds, IWU optimizer settings, RPSAMPLER factors) and one domain-specific assumption that the GRPO-loss improvement signal is meaningful despite the paper's own critique of J_GRPO as a task-comparison signal. No new physical entities are introduced; the algorithmic components are not independent entities.

free parameters (4)

λ trade-off parameter = 0.2, 0.25 (Exp 1); 0.1, 0.3, 0.9, 1.2 (Exp 2)
Controls the balance between worst-task reward and improvement in Subroutine 1; the reported gains vary with λ and there is no principled selection rule.
Improvement clipping bounds = [0.1, 0.2]
Section D: improvements I_k^{(t)} are “clipped to {0.1,0.2} for stability”—an ad hoc chosen range that directly affects weight updates.
IWU optimizer hyperparameters = lr 0.025, weight decay 1e-5 (Exp 1); lr 1e-4 (Exp 2)
Gradient descent on task-weight logits in Subroutine 1 uses these values; no sensitivity analysis provided.
RPSAMPLER hyperparameters = M_os=3, N_rs=10 (Exp 1) / 2 (Exp 2), M_acc=5
Oversampling factor, maximum resampling rounds, and maximum acceptance inflation factor are hand-set and affect how faithfully task weights are realized.

assumptions (4)

standard math Lagrangian relaxation and strong duality hold for the constrained multi-task objective (Eq 4–5, B.1, B.3).
The max–min reformulation and the closed form of Ω(z) rely on convexity and compactness of the simplex; standard but unverified in this setting.
ad hoc to paper The GRPO loss difference I_k^{(t)} is a meaningful improvement signal despite being zero for all-correct and all-wrong rollouts.
Invoked in Eq (8) and Subroutine 1, even though §3.1 argues J_GRPO is unreliable for task comparison because of this exact ambiguity.
domain assumption All tasks use the same reward protocol (1.0 correct, 0.1 incorrect-format, 0 otherwise), making task rewards comparable without normalization.
Stated in Section D; if reward scales differed, a single λ would bias reweighting toward high-variance tasks.
domain assumption Zero-gradient filtering removes only non-informative prompts and does not bias the per-task advantage estimates used for policy updates.
The RPSAMPLER correctness argument depends on this; the paper does not analyze bias introduced by the filter-resample loop.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Multi-Task GRPO: Reliable LLM Reasoning Across Tasks." pith.science (2026). https://pith.science/paper/JV7MQWMU

@misc{pith2026260205547,
  author       = {Pith},
  title        = {Pith review of: Multi-Task GRPO: Reliable LLM Reasoning Across Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JV7MQWMU}},
  note         = {Machine review of arXiv:2602.05547}
}

read the original abstract

RL-based post-training with GRPO is widely used to improve large language models on individual reasoning tasks. However, real-world deployment requires reliable performance across diverse tasks. A straightforward multi-task adaptation of GRPO often leads to imbalanced outcomes, with some tasks dominating optimization while others stagnate. Moreover, tasks can vary widely in how frequently prompts yield zero advantages (and thus zero gradients), which further distorts their effective contribution to the optimization signal. To address these issues, we propose a novel Multi-Task GRPO (MT-GRPO) algorithm that (i) dynamically adapts task weights to explicitly optimize worst-task performance and promote balanced progress across tasks, and (ii) introduces a ratio-preserving sampler to ensure task-wise policy gradients reflect the adapted weights. Experiments on both 3-task and 9-task settings show that MT-GRPO consistently outperforms baselines in worst-task accuracy. In particular, MT-GRPO achieves 16-28% and 6% absolute improvement on worst-task performance over standard GRPO and DAPO, respectively, while maintaining competitive average accuracy. Moreover, MT-GRPO requires 50% fewer training steps to reach 50% worst-task accuracy in the 3-task setting, demonstrating substantially improved efficiency in achieving reliable performance across tasks.