{"id":"f13abef5-048e-412f-89e5-648e3ef296d6","arxiv_id":"2601.22900","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Feedback-triggered retries converted into GRPO/DPO training signals lift math and science reasoning accuracy above RLVR baselines like GRPO and Critique-GRPO.","lead":"MulFeRL is a training recipe that lets a language model learn from its own failures: when every sampled answer to a math problem is wrong, a critic writes hints, the model retries, and that improvement becomes a training signal. On five math and three science benchmarks it beat standard RL fine-tuning on two open-weight models (Qwen2.5-7B and Qwen3-4B).","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that verbal feedback drives MulFeRL's gains is untested: the design confounds feedback with additional sampling on all-failed groups.","rationale":"The reader's verdict is CONDITIONAL, and the reader explicitly noted the lack of a multi-turn no-feedback control. We agree this is a real gap, but we argue it is the single most load-bearing concern—more so than the leakage worry—because the self-feedback experiment (Table 3) already shows gains without GPT-4o, partially mitigating the distillation-by-leakage hypothesis. What remains untested is whether any feedback at all is doing causal work, versus the extra sampling occasions created by the all-failed regeneration trigger. The no-feedback control directly adjudicates this. We recommend keeping the verdict CONDITIONAL, with this control as a required condition before the central claim is treated as established. The paper's internal consistency is good, but this confound sits at the heart of the claimed contribution.","tokens_in":23980,"tokens_out":6902,"duration_ms":80460,"concrete_test":"Re-run MulFeRL on Qwen2.5-7B-Base (and optionally Qwen3-4B-Inst) with one change: in the regeneration loop, set c(t) = x for all turns (no external feedback), keeping everything else identical—same K=8, same turn budget T=3, same GRPO/DPO gating and losses, same output format (the model still emits its own <feedback> slot per Figure 6), same training steps, and same compute per step (resample K rollouts each turn). Compare AIME24 and MATH-500 pass@1 against the reported MulFeRL numbers (Table 1). If the no-feedback variant lands within ~1–2 points of MulFeRL (46.22 avg on Qwen2.5), the central claim fails; if it is substantially worse (e.g., ≥3–4 points), feedback is causally necessary.","verdict_should_be":"UNCHANGED","load_bearing_attack":"MulFeRL triggers feedback and regeneration only on all-failed groups. Algorithm 1 (lines 15–17) replaces the context c(t+1) = (x, f(t)) with feedback-conditioned sampling, but the same regeneration loop without feedback (c(t+1) = x) would also give the policy more attempts on hard prompts, and GRPO/DPO could then be applied to any newly mixed or all-positive groups. The paper's only regeneration ablation ('w/o Regeneration', Table 2) removes the extra turns entirely, so it cannot separate the contribution of verbal feedback from the contribution of additional sampling. If resampling alone—under the same turn budget and loss gating, but without external feedback—produces comparable pass@1, then the reported gains are explained by a simple 'sample until some rollouts are correct' effect, not by feedback-guided progress. This would undermine the paper's central claim (§1, §3.2) that verbal feedback converts failed samples into informative learning signals. The concern is concrete and testable; if the no-feedback control matches MulFeRL, the method's novelty collapses to 'GRPO with rejection sampling'.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MulFeRL, a multi-turn reinforcement-learning-from-verifiable-rewards (RLVR) method that injects verbal feedback into training on failed samples. When a sampled group is entirely incorrect, a feedback simulator (typically GPT-4o) produces a structured critique, the policy regenerates conditioned on that feedback, and the model is updated either with GRPO on the regenerated group (if mixed) or with a cross-turn DPO objective (if all-positive). The method also uses a fixed output schema with a dedicated <feedback> slot and masks external feedback tokens from the policy-gradient loss. Training on 4k OpenR1-Math instances with Qwen2.5-7B-Base and Qwen3-4B-Inst, the authors report Pass@1 improvements over SFT, RAFT, CITL-FT, GRPO, Dr.GRPO, and Critique-GRPO on five math benchmarks and show OOD gains on MMLU-Pro, GPQA-Diamond, and TheoremQA. They include ablations of regeneration, DPO, feedback injection, a feedback-simulator study including self-feedback, a test-time feedback scaling experiment, and a format-alignment control.","tokens_in":24283,"tokens_out":4396,"duration_ms":47588,"significance":"If the central claim is correct, MulFeRL is a meaningful step toward using verbal feedback as a dense, learnable signal in RLVR, particularly for the all-failed regime where scalar rewards carry no gradient information. The empirical scope is strong for a preprint: two backbones, eight benchmarks, a format-alignment control, supervised and RL baselines, self-feedback variants, and a detailed appendix with prompts and hyperparameters. The code release and the explicit attention to reproducibility (prompts, hyperparameters, evaluation protocol) are commendable. The main weakness is that the reported experiments do not isolate the causal contribution of verbal feedback from the increased sampling budget on all-failed groups. The no-feedback multi-turn control is absent, and the feedback simulator's answer-freeness is enforced only by prompt instruction rather than verified. These gaps leave the central 'feedback-driven progress' interpretation underdetermined, although the paper's claims would become much more credible if the missing control is run and supports the authors' interpretation.","major_comments":[{"comment":"The central claim that verbal feedback, rather than additional sampling, drives MulFeRL's gains is not tested. Feedback and regeneration are triggered only on all-failed groups, and the only regeneration ablation ('w/o Regeneration') removes the extra turns entirely. This does not separate 'feedback-guided regeneration' from 'more attempts on hard prompts'. I request a control that runs the same algorithm with c(t+1)=x (no feedback) on all-failed groups, with the same group size K, turn budget, and loss gating. If this control matches MulFeRL's Pass@1, the reported gains would be explained by rejection-sampling/GRPO-with-extra-attempts rather than by the verbal feedback mechanism emphasized in §1 and §3.3.","section":"Algorithm 1 (lines 15–17), Eq. (2)–(3), Table 2"},{"comment":"The paper repeatedly claims that feedback is 'answer-free' and that MulFeRL is 'fundamentally different from distillation,' but answer-freeness is only enforced through the simulator prompt. There is no audit of whether GPT-4o's feedback text or the regenerated solutions reveal numeric answers. If feedback leaks the final answer, the verifier-confirmed 'progress' could be distillation from the feedback simulator rather than feedback-induced reasoning. Please add a concrete leakage check: sample feedback outputs on a subset of training prompts, run an answer extractor on the feedback text (or use an answer-redacted control), and report the leak rate. Without this, the non-distillation claim in §3.5 is unsupported.","section":"§3.5, Appendix B.2, Appendix F"},{"comment":"The caption of Table 1 states 'significant improvements across all datasets compared to RL baselines (exact McNemar’s test on paired per-problem Pass@1 outcomes),' but no p-values, test statistics, standard errors, or confidence intervals are reported anywhere in the main text or appendix. Results are averages over five independent runs, yet no variance is given. Since the paper makes a statistical significance claim, please provide the corresponding p-values (or at least error bars / per-run numbers) for the main comparisons in Table 1 and the ablations in Table 2.","section":"Table 1, §4.1"},{"comment":"In the ablation table, the 'w/o Regeneration' row reports numbers that are identical to the GRPO baseline in Table 1 for both backbones (e.g., Qwen2.5-7B-Base: AMC23 42.20, AIME24 16.13; Qwen3-4B-Inst: AMC23 78.60, AIME24 57.47). This raises the concern that the ablated variant was not actually run under MulFeRL's structured output format and reward criterion (format compliance + answer correctness), but instead reused the GRPO baseline numbers, which were obtained with a different format and reward. Please either clarify that the ablation was run under exactly the same reward/format conditions as MulFeRL, or rerun it; otherwise the ablation's attribution of the gain to multi-turn regeneration is not clean.","section":"Table 8 vs. Table 1, Appendix C.1.1"}],"minor_comments":[{"comment":"The surrogate objective Jθ is used in Eq. (1) before it is defined later in §3.4; a forward reference or a brief intuitive definition would help readability.","section":"Eq. (1) / §3.1"},{"comment":"The caption of Figure 13(a) says 'Validation Performance' while (b) says 'Test Performance'; please make the evaluation protocol consistent (or explain the difference).","section":"Figure 13"},{"comment":"For the reader's benefit, consider reporting the standard deviation of the 5 runs, since some of the reported differences between methods are a few points and the claim of consistency across datasets would be strengthened by variance information.","section":"§4.2 / Table 1"},{"comment":"The cross-turn DPO objective scores the previous-turn response y(t-1) under the feedback-augmented context c(t) even though it was sampled under c(t-1). This is a counterfactual evaluation that may make the dispreferred response artificially low-probability. A sentence justifying this choice (or a small experiment comparing scoring under original vs. augmented context) would address a potential concern about the DPO signal being inflated by context mismatch.","section":"Algorithm 1, line 21"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-written and the empirical scope is broad, but the missing no-feedback multi-turn control is the key load-bearing issue. If that control matches the feedback version, the method's novelty is essentially 'GRPO with rejection sampling on failed groups.' The leakage audit is also important because the 'not distillation' claim is central to the framing. These concerns are addressable with additional experiments, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MulFeRL is a solid, clearly-written empirical paper: it combines all-failed-only feedback triggering, cross-turn DPO on feedback-conditioned revisions, and structured feedback injection, and the ablations are unusually thorough for this area. The OpenR1-Math training, eight benchmarks, two backbones, self-feedback variants, and format-alignment control show real care. It deserves a serious referee.\n\nThe soft spot, as your stress-test note says, is a confound the paper never closes. Feedback is triggered only on all-failed groups, and the only 'w/o Regeneration' ablation removes the extra turns entirely. So the observed gains could come from extra sampling of hard prompts under the same turn budget and loss gating, with feedback doing little or nothing. Algorithm 1 makes this concrete: replace c(t+1) = (x, f(t)) with c(t+1) = x and you have that control; the paper doesn't run it. If resampling alone — with the same DPO/GRPO gating — matches MulFeRL, the paper's contribution collapses to 'GRPO with rejection sampling' and the entire §1 narrative about feedback converting dead samples into learning signals is unsupported. The concern is testable and cheap, so a referee should make it required.\n\nAlso fair: Table 1 claims exact McNemar significance but reports no error bars, p-values, or per-run spreads despite five runs. The fixed code link is an anonymous 4open.science URL, not a commit-hashed release. And the 'answer-free feedback' rule is enforced only by prompt; the authors never audit whether GPT-4o's feedback leaks the final answer. If it leaks, the mechanism is distillation, not feedback-guided reasoning.\n\nMinor things are proportionate: the w/o Feedback injection ablation mainly shows the structured slot helps over plain prompting, and the simulator table shows self-feedback alone beats GRPO, which suggests feedback has some value — but both are indirect. The paper's own Limitations section is honest about cost and verifier dependence. The reference list is appropriate and not padded.\n\nVerdict: conditional accept at best, not because the method is bad, but because the signature claim hasn't been isolated. Send it to peer review, and make the no-feedback multi-turn control, leakage audit, and error bars mandatory. This is a publishable system if those checks pass.","headline":"A well-built engineering paper with a genuinely new combination — but the central claim that verbal feedback, not extra resampling, drives the gains is untested, and the authors have all the pieces to test it.","tokens_in":24770,"tokens_out":1905,"would_cite":true,"duration_ms":22423,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Verbal feedback on failed attempts can be converted into a stable reinforcement-learning training signal.","keywords":["reinforcement learning","verifiable rewards","verbal feedback","multi-turn regeneration","GRPO","DPO","mathematical reasoning","self-correction"],"falsifier":"Run the training loop with the feedback removed: after an all-failed group, regenerate the same number of candidates from the same policy with no feedback, and apply the same GRPO/DPO objectives to whatever improvement results. If this no-feedback control matches MulFeRL's gains, the effect attributed to verbal feedback is actually an effect of additional sampling; conversely, if answer-freeness auditing shows the feedback simulator's output contains the final answer, the gains would be explainable as distillation rather than feedback-guided reasoning.","tokens_in":23885,"feed_emoji":"🧠","tokens_out":5527,"duration_ms":51483,"temperature":0.7,"pith_summary":"Reinforcement learning with verifiable rewards stalls when every sampled answer to a problem is wrong: the scalar reward says 'incorrect' but not why, so the gradient signal collapses. MulFeRL tries to close that gap by feeding verbal feedback into the training loop. Whenever an entire rollout group fails, a feedback generator explains the root error and repair steps, the policy regenerates conditioned on that feedback, and the observed improvement is turned into either a within-group GRPO update or a cross-turn DPO preference. The paper reports that this multi-turn scheme, trained on a few thousand sampled math problems, outperforms supervised finetuning and prior RLVR baselines in-domain and transfers to out-of-domain science benchmarks. The reason to care is the claim that rich natural-language feedback, not just success/failure labels, can be made into a stable RL learning signal.","feed_headline":"Verbal feedback turns RL failures into learning signal","feed_subtitle":"MulFeRL's loop converts failed rollouts into GRPO/DPO updates, beating RLVR baselines on math and science benchmarks.","key_machinery":"The load-bearing mechanism is a three-part loop: (1) dynamic multi-turn regeneration, triggered only when all K rollouts fail, which queries a feedback simulator for a concise 'Issue + Fix steps' summary and samples a fresh group conditioned on that feedback; (2) two complementary objectives—in-turn GRPO when the regenerated group is mixed, and cross-turn DPO matching each regenerated correct answer against the previous turn's failed answer at the same index when the group is all-positive; (3) structured feedback injection that places external feedback inside a fixed <feedback> slot in the <thinking> block and masks those tokens from the policy-gradient loss. The work this machinery does is","core_discovery":"On its own terms, MulFeRL's central discovery is that an all-failed rollout group, which standard GRPO treats as zero-information, can be turned into a useful training event by adding a feedback–regeneration loop. The claim is that when feedback-conditioned regeneration converts a previous-turn failure into verifiable success, the pair (previous attempt, revised attempt) is a legitimate preference for DPO, and when it converts the group from all-failed to mixed, the within-group contrast is a legitimate GRPO advantage. The paper further claims that this works because feedback is injected at a fixed slot in the reasoning trace and only the model's own solution tokens are optimized, so the mod","pith_inferences":["If feedback quality is the limiting factor, a cheap way to test the approach's ceiling is to train with a deliberately degraded or randomized feedback simulator; the paper's own simulator-quality sweep implies performance will track feedback quality, so a no-feedback control is the missing isolation experiment.","The answer-freeness of feedback is enforced only by prompt instruction and never audited; if the simulator's Issue/Fix text correlates with the final numeric answer, part of the reported gain could be a distillation artifact. Checking for this leakage across a sample of feedback outputs is a direct falsification.","The cross-turn DPO pairing matches regenerated and previous responses only by index within a group; alternatives like pairing best-of-K regenerations or aggregating multiple feedback turns could extract more signal from near-miss improvements, an extension the paper leaves open.","Because regeneration is triggered only on all-failed groups, the method concentrates learning on the hardest samples; if the verifier is unreliable, those groups are mislabeled and the whole signal is noise, so robustness to verifier error is the natural next stress test."],"forward_implications":["Models trained this way can be deployed in standard single-pass decoding; the training-time feedback loop is not needed at inference.","Giving the trained model external feedback at test time improves accuracy further, and MulFeRL-trained models profit more from each additional feedback turn than prior RL baselines.","The gains are not confined to the training distribution: improvements appear on out-of-domain science and general reasoning benchmarks, suggesting feedback conditioning encourages transferable reasoning rather than domain-specific heuristics.","Even with the base model itself as the feedback source, MulFeRL beats plain GRPO, indicating the gain is not solely distillation from a stronger teacher."],"fun_headline_variants":["Feedback loop turns failed RL rollouts into training signal","MulFeRL: verbal feedback converts RL failures into learning","Multi-turn verbal feedback transforms RL failure into progress","Verbal feedback loop rescues failed samples for RL training"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the feedback simulator reliably locates the root error and guides regeneration toward a verifiable success without revealing the answer; if the feedback is noisy, uninformative, or leaks the solution, the converted 'progress' is just extra sampling or distillation.","fun_headline_variants_meta":{"raw":{"variants":["Feedback loop turns failed RL rollouts into training signal","MulFeRL: verbal feedback converts RL failures into learning","Multi-turn verbal feedback transforms RL failure into progress","Verbal feedback loop rescues failed samples for RL training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000329,"raw_usage":{"total_tokens":1648,"prompt_tokens":694,"completion_tokens":954,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":438,"completion_tokens_details":{"reasoning_tokens":890}},"tokens_in":438,"tokens_out":954,"duration_ms":8038,"temperature":1.0,"reasoning_tokens":890,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T06:19:10.058825+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the training loop with the feedback removed: after an all-failed group, regenerate the same number of candidates from the same policy with no feedback, and apply the same GRPO/DPO objectives to whatever improvement results. If this no-feedback control matches MulFeRL's gains, the effect attributed to verbal feedback is actually an effect of additional sampling; conversely, if answer-freeness auditing shows the feedback simulator's output contains the final answer, the gains would be explainable as distillation rather than feedback-guided reasoning.","supporting_citations":[],"review_version":1}