Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

MulFeRL: Enhancing Reinforcement Learning with Verbal Feedback in a Multi-turn Loop

T0 review · 4 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Verbal feedback on failed attempts can be converted into a stable reinforcement-learning training signal.

desk verdict A well-built engineering paper with a genuinely new combination — but the central claim that verbal feedback, not extra resampling, drives the gains is untested, and the authors have all the pieces to test it. read the letter →

arxiv 2601.22900 v2 pith:FOBSE6NP submitted 2026-01-30 cs.AI

Xuancheng Li , Haitao Li , Yujia Zhou , YiqunLiu , Qingyao Ai This is my paper · ORCID
classification cs.AI
keywords reinforcementlearningverifiablerewardsverbalfeedbackmulti-turnregenerationGRPODPOmathematicalreasoningself-correction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reinforcement learning with verifiable rewards stalls when every sampled answer to a problem is wrong: the scalar reward says 'incorrect' but not why, so the gradient signal collapses. MulFeRL tries to close that gap by feeding verbal feedback into the training loop. Whenever an entire rollout group fails, a feedback generator explains the root error and repair steps, the policy regenerates conditioned on that feedback, and the observed improvement is turned into either a within-group GRPO update or a cross-turn DPO preference. The paper reports that this multi-turn scheme, trained on a few thousand sampled math problems, outperforms supervised finetuning and prior RLVR baselines in-domain and transfers to out-of-domain science benchmarks. The reason to care is the claim that rich natural-language feedback, not just success/failure labels, can be made into a stable RL learning signal.

What carries the argument

The load-bearing mechanism is a three-part loop: (1) dynamic multi-turn regeneration, triggered only when all K rollouts fail, which queries a feedback simulator for a concise 'Issue + Fix steps' summary and samples a fresh group conditioned on that feedback; (2) two complementary objectives—in-turn GRPO when the regenerated group is mixed, and cross-turn DPO matching each regenerated correct answer against the previous turn's failed answer at the same index when the group is all-positive; (3) structured feedback injection that places external feedback inside a fixed <feedback> slot in the <thinking> block and masks those tokens from the policy-gradient loss. The work this machinery does is

What would settle it

Run the training loop with the feedback removed: after an all-failed group, regenerate the same number of candidates from the same policy with no feedback, and apply the same GRPO/DPO objectives to whatever improvement results. If this no-feedback control matches MulFeRL's gains, the effect attributed to verbal feedback is actually an effect of additional sampling; conversely, if answer-freeness auditing shows the feedback simulator's output contains the final answer, the gains would be explainable as distillation rather than feedback-guided reasoning.

Watch

Extended reading notes

Core claim

On its own terms, MulFeRL's central discovery is that an all-failed rollout group, which standard GRPO treats as zero-information, can be turned into a useful training event by adding a feedback–regeneration loop. The claim is that when feedback-conditioned regeneration converts a previous-turn failure into verifiable success, the pair (previous attempt, revised attempt) is a legitimate preference for DPO, and when it converts the group from all-failed to mixed, the within-group contrast is a legitimate GRPO advantage. The paper further claims that this works because feedback is injected at a fixed slot in the reasoning trace and only the model's own solution tokens are optimized, so the mod

Load-bearing premise

The load-bearing premise is that the feedback simulator reliably locates the root error and guides regeneration toward a verifiable success without revealing the answer; if the feedback is noisy, uninformative, or leaks the solution, the converted 'progress' is just extra sampling or distillation.

Editorial extensions

If this is right

  • Models trained this way can be deployed in standard single-pass decoding; the training-time feedback loop is not needed at inference.
  • Giving the trained model external feedback at test time improves accuracy further, and MulFeRL-trained models profit more from each additional feedback turn than prior RL baselines.
  • The gains are not confined to the training distribution: improvements appear on out-of-domain science and general reasoning benchmarks, suggesting feedback conditioning encourages transferable reasoning rather than domain-specific heuristics.
  • Even with the base model itself as the feedback source, MulFeRL beats plain GRPO, indicating the gain is not solely distillation from a stronger teacher.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If feedback quality is the limiting factor, a cheap way to test the approach's ceiling is to train with a deliberately degraded or randomized feedback simulator; the paper's own simulator-quality sweep implies performance will track feedback quality, so a no-feedback control is the missing isolation experiment.
  • The answer-freeness of feedback is enforced only by prompt instruction and never audited; if the simulator's Issue/Fix text correlates with the final numeric answer, part of the reported gain could be a distillation artifact. Checking for this leakage across a sample of feedback outputs is a direct falsification.
  • The cross-turn DPO pairing matches regenerated and previous responses only by index within a group; alternatives like pairing best-of-K regenerations or aggregating multiple feedback turns could extract more signal from near-miss improvements, an extension the paper leaves open.
  • Because regeneration is triggered only on all-failed groups, the method concentrates learning on the hardest samples; if the verifier is unreliable, those groups are mislabeled and the whole signal is noise, so robustness to verifier error is the natural next stress test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes MulFeRL, a multi-turn reinforcement-learning-from-verifiable-rewards (RLVR) method that injects verbal feedback into training on failed samples. When a sampled group is entirely incorrect, a feedback simulator (typically GPT-4o) produces a structured critique, the policy regenerates conditioned on that feedback, and the model is updated either with GRPO on the regenerated group (if mixed) or with a cross-turn DPO objective (if all-positive). The method also uses a fixed output schema with a dedicated <feedback> slot and masks external feedback tokens from the policy-gradient loss. Training on 4k OpenR1-Math instances with Qwen2.5-7B-Base and Qwen3-4B-Inst, the authors report Pass@1 improvements over SFT, RAFT, CITL-FT, GRPO, Dr.GRPO, and Critique-GRPO on five math benchmarks and show OOD gains on MMLU-Pro, GPQA-Diamond, and TheoremQA. They include ablations of regeneration, DPO, feedback injection, a feedback-simulator study including self-feedback, a test-time feedback scaling experiment, and a format-alignment control.

Significance. If the central claim is correct, MulFeRL is a meaningful step toward using verbal feedback as a dense, learnable signal in RLVR, particularly for the all-failed regime where scalar rewards carry no gradient information. The empirical scope is strong for a preprint: two backbones, eight benchmarks, a format-alignment control, supervised and RL baselines, self-feedback variants, and a detailed appendix with prompts and hyperparameters. The code release and the explicit attention to reproducibility (prompts, hyperparameters, evaluation protocol) are commendable. The main weakness is that the reported experiments do not isolate the causal contribution of verbal feedback from the increased sampling budget on all-failed groups. The no-feedback multi-turn control is absent, and the feedback simulator's answer-freeness is enforced only by prompt instruction rather than verified. These gaps leave the central 'feedback-driven progress' interpretation underdetermined, although the paper's claims would become much more credible if the missing control is run and supports the authors' interpretation.

major comments (4)
  1. [Algorithm 1 (lines 15–17), Eq. (2)–(3), Table 2] The central claim that verbal feedback, rather than additional sampling, drives MulFeRL's gains is not tested. Feedback and regeneration are triggered only on all-failed groups, and the only regeneration ablation ('w/o Regeneration') removes the extra turns entirely. This does not separate 'feedback-guided regeneration' from 'more attempts on hard prompts'. I request a control that runs the same algorithm with c(t+1)=x (no feedback) on all-failed groups, with the same group size K, turn budget, and loss gating. If this control matches MulFeRL's Pass@1, the reported gains would be explained by rejection-sampling/GRPO-with-extra-attempts rather than by the verbal feedback mechanism emphasized in §1 and §3.3.
  2. [§3.5, Appendix B.2, Appendix F] The paper repeatedly claims that feedback is 'answer-free' and that MulFeRL is 'fundamentally different from distillation,' but answer-freeness is only enforced through the simulator prompt. There is no audit of whether GPT-4o's feedback text or the regenerated solutions reveal numeric answers. If feedback leaks the final answer, the verifier-confirmed 'progress' could be distillation from the feedback simulator rather than feedback-induced reasoning. Please add a concrete leakage check: sample feedback outputs on a subset of training prompts, run an answer extractor on the feedback text (or use an answer-redacted control), and report the leak rate. Without this, the non-distillation claim in §3.5 is unsupported.
  3. [Table 1, §4.1] The caption of Table 1 states 'significant improvements across all datasets compared to RL baselines (exact McNemar’s test on paired per-problem Pass@1 outcomes),' but no p-values, test statistics, standard errors, or confidence intervals are reported anywhere in the main text or appendix. Results are averages over five independent runs, yet no variance is given. Since the paper makes a statistical significance claim, please provide the corresponding p-values (or at least error bars / per-run numbers) for the main comparisons in Table 1 and the ablations in Table 2.
  4. [Table 8 vs. Table 1, Appendix C.1.1] In the ablation table, the 'w/o Regeneration' row reports numbers that are identical to the GRPO baseline in Table 1 for both backbones (e.g., Qwen2.5-7B-Base: AMC23 42.20, AIME24 16.13; Qwen3-4B-Inst: AMC23 78.60, AIME24 57.47). This raises the concern that the ablated variant was not actually run under MulFeRL's structured output format and reward criterion (format compliance + answer correctness), but instead reused the GRPO baseline numbers, which were obtained with a different format and reward. Please either clarify that the ablation was run under exactly the same reward/format conditions as MulFeRL, or rerun it; otherwise the ablation's attribution of the gain to multi-turn regeneration is not clean.
minor comments (4)
  1. [Eq. (1) / §3.1] The surrogate objective Jθ is used in Eq. (1) before it is defined later in §3.4; a forward reference or a brief intuitive definition would help readability.
  2. [Figure 13] The caption of Figure 13(a) says 'Validation Performance' while (b) says 'Test Performance'; please make the evaluation protocol consistent (or explain the difference).
  3. [§4.2 / Table 1] For the reader's benefit, consider reporting the standard deviation of the 5 runs, since some of the reported differences between methods are a few points and the claim of consistency across datasets would be strengthened by variance information.
  4. [Algorithm 1, line 21] The cross-turn DPO objective scores the previous-turn response y(t-1) under the feedback-augmented context c(t) even though it was sampled under c(t-1). This is a counterfactual evaluation that may make the dispreferred response artificially low-probability. A sentence justifying this choice (or a small experiment comparing scoring under original vs. augmented context) would address a potential concern about the DPO signal being inflated by context mismatch.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the learning signal is derived from verifier rewards on feedback-guided regenerated groups, with no fitted constants and no load-bearing self-citation chain.

full rationale

MulFeRL's derivation chain is self-contained relative to the circularity axis. The training signal is constructed directly from verifier rewards on regenerated groups: feedback is triggered only on all-failed groups (Eq. 2), regenerated candidates are sampled under feedback-conditioned contexts (Eq. 3), and the two objectives are computed from verifier-confirmed reward patterns (Eqs. 4-8). No parameter is fitted to the evaluation benchmarks: lambda, beta, beta_KL, group size K, and turn budget T are fixed hyperparameters (Table 4), and the verifier reward (Eq. 9) is a hand-specified format-plus-answer check, not a learned or fitted quantity. The controlled ablation 'w/o Regeneration' reduces the method to single-turn GRPO and reproduces the GRPO row, while the 'w/o DPO' and 'w/o Feedback injection' ablations isolate component contributions. The absence of a no-feedback multi-turn control is a genuine experimental confound for the claim that verbal feedback specifically (rather than additional sampling) drives gains, but it is an experimental-design gap, not a definitional reduction: the paper never defines the feedback-induced progress signal as identical to the evaluation metric, and the reported held-out pass@1 gains are not equal to the training objective by construction. Self-citations (Shi et al. 2025 and Liu et al. 2026) appear only as background support for general claims about RLVR and training plateaus; they are not load-bearing for the method's derivation. No fitted input is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via self-citation. The central derivation is therefore not circular; the main caveat is a missing control, which falls under experimental validity rather than circularity.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The method rests on hand-set hyperparameters, unverified assumptions about verifier and feedback quality, and an index-matched cross-turn pairing whose semantic correspondence is not established. No new physical or architectural entities are introduced.

free parameters (8)
  • DPO weight λ = 0.01
    Hand-set to balance GRPO and DPO loss scales; no sensitivity study reported.
  • DPO temperature β = 0.005
    Hand-set; no sensitivity study reported.
  • KL coefficient β_KL = 0.001
    Standard RLVR value, but hand-set and not ablated.
  • Group size K = 8
    Hand-set; determines within-group contrast and feedback aggregation; not ablated.
  • Max regeneration turns T = 2 (training), up to 5 in turn-budget ablation
    Hand-set; a turn-budget ablation exists, but the main results use T=2.
  • Feedback subgroup size K_fb / subgroups g = K_fb=2, g=4
    Hand-set for two-step feedback aggregation; no sensitivity analysis.
  • Learning rate = 1e-6
    Hand-set; standard for RLVR post-training; not ablated.
  • Training sample size = 4k instances from reorganized 45k OpenR1-Math subset
    Arbitrary sample size; seed/sample indices not published, so exact training set is unreproducible.
assumptions (5)
  • domain assumption The verifier V returns correct binary outcomes for final answers and format compliance, and outcome correctness is a reliable proxy for reasoning quality.
    Reward is defined in Eq. (9); if the verifier is noisy, all GRPO advantages, DPO preferences, and evaluation numbers degrade together.
  • domain assumption The feedback simulator Fψ produces feedback that identifies root errors and gives actionable fixes without revealing the solution or final answer.
    Enforced only by system prompts (Figures 7–8); no audit of leakage; central to the claim that gains come from feedback-guided learning rather than distillation.
  • domain assumption An all-positive regenerated group is genuinely preferred over the previous all-failed group under the same feedback-augmented context; improvements are due to feedback, not sampling noise.
    DPO pairs in Eq. (6) and Algorithm 1 lines 19–22 treat y^(t+1) as preferred over y^(t) solely on verifier outcomes; with K=8 and temperature 1, all-positive can occur by chance.
  • domain assumption The structured output schema with the <feedback> slot is learnable and does not degrade reasoning relative to free-form generation.
    MulFeRL's reward requires format compliance (Eq. 9); Appendix E.1 shows the format hurts baselines, so the policy must internalize the slot for the method to work.
  • standard math GRPO/DPO objectives and their theoretical guarantees transfer to feedback-conditioned multi-turn sampling.
    Method builds directly on Guo et al. 2025 and Rafailov et al. 2023; standard optimization assumptions are inherited without re-derivation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MulFeRL: Enhancing Reinforcement Learning with Verbal Feedback in a Multi-turn Loop." pith.science (2026). https://pith.science/paper/FOBSE6NP

@misc{pith2026260122900,
  author       = {Pith},
  title        = {Pith review of: MulFeRL: Enhancing Reinforcement Learning with Verbal Feedback in a Multi-turn Loop},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FOBSE6NP}},
  note         = {Machine review of arXiv:2601.22900}
}
read the original abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning across domains, but outcome-only scalar rewards are often sparse and uninformative. This limitation is especially severe for failed samples, where scalar rewards indicate only that a solution is incorrect without explaining why the reasoning breaks down. In this paper, we leverage richer verbal feedback to guide RLVR on failed samples and convert feedback-induced progress into trainable learning signals. We propose MulFeRL (Multi-turn Feedback-guided Reinforcement Learning), a multi-turn, event-triggered RLVR framework that combines progress induction for feedback-guided regeneration of failed samples, progress credit assignment for learning from verifier-confirmed progress, and structured feedback injection for integrating feedback into the model's reasoning process. Trained on sampled OpenR1-Math, MulFeRL outperforms supervised, self-distillation-based, and RLVR baselines in-domain, while also showing strong out-of-domain generalization.

Figures

Figures reproduced from arXiv: 2601.22900 by the authors.

Figure 1
Figure 1. Illustration of MulFeRL. errors and guide corrections, we introduce verbal feedback into GRPO training to provide learning signal on failed samples, where the scalar verifier offers little discrimination. This raises a core question: How can we convert verbal feedback into a stable learning signal that continues to improve the policy on failed samples? Formally, our goal is to design a feedback-derived surrogate obj… view at source ↗
Figure 2
Figure 2. The output format used by MulFeRL. location with explicit delimiters, so next-token prediction naturally conditions the reasoning on the injected feedback. During MulFeRL training, the verifier V enforces answer correctness and schema compliance; we assign reward 1 iff both hold, else 0. To mimic effective human feedback, we prompt ψ to produce two components: Issue (pinpointing the root mistake and concrete errors)… view at source ↗
Figure 3
Figure 3. Test-time multi-turn feedback results on TheoremQA (Qwen2.5-7B-Base). We vary the number of feedback turns. feedback injection consistently hurts performance, implying that structured injection conditions the model’s reasoning on feedback more faithfully than plain prompting, improving feedback utilization and stabilizing training. 5 Analysis 5.1 Impact of Feedback Simulators [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: Impact of training turns on both effectiveness and effi￾ciency (Qwen3-4B-Inst). We vary the number of training turns and report the best performance and the average training time. because feedback-guided regeneration yields improved can￾didates and cross-turn comparati…
Figure 4
Figure 4. Figure 4: Training efficiency (Qwen3-4B-Inst). (a) Training cost across methods. (b) MulFeRL efficiency across training steps: average training time per step and the average number of regener￾ated samples per step in Turn-1/Turn-2, reported as averages over 50-step intervals. to…
Figure 6
Figure 6. Figure 6: Training system prompt. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: System prompt for obtaining feedback. Merge feedback system prompt You merge multiple feedback comments on an incorrect solution.\n\n Task:\n - Combine the feedback into ONE concise, actionable feedback.\n - Deduplicate repeated points.\n - Keep ONLY the earliest/root …
Figure 8
Figure 8. Figure 8: System prompt for merging feedback. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: Prompt for feedback injection and regeneration rollout. SFT/RAFT system prompt and CITL-FT initial prompt You are a reasoning assistant.\n Solve the problem step by step.\n\n Output format (must follow exactly):\n 1) Wrap ALL reasoning inside a single <thinking>...</th…
Figure 10
Figure 10. Figure 10: SFT/RAFT prompt and CITL-FT initial prompt CITL-FT critique system prompt You are a strict reviewer of a solution.\n Given a problem and a proposed solution, identify mistakes and provide actionable repair suggestions.\n\n Rules:\n - Do NOT provide a full correct solu…
Figure 11
Figure 11. Figure 11: Critique prompt for CITL-FT 22 [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: Refinement prompt for CITL-FT 0 100 200 300 400 500 Training Step 0 10 20 30 40 50 Validation Performance (%) MulFeRL (Ours) Critique-GRPO GRPO (a) Qwen2.5-7B-Base. 0 100 200 300 400 Training Step 20 30 40 50 60 70 Test Performance (avg pass@1, %) MulFeRL (Ours) Criti…
Figure 13
Figure 13. Figure 13: Average validation performance on five math benchmarks over training steps. MulFeRL is evaluated on both answer correctness and format compliance, while GRPO and Critique-GRPO are evaluated on answer correctness only. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Different Feedback, Different Updates: Selective Self-Learning from User Interactions for Large Language Models

    cs.AI 2026-08 accept novelty 6.0 of 10

    SLIFT decomposes user feedback into Fix, Spec, and Null parts, then trains a Generalist adapter for fixes and a Specialist adapter for optional refinements, improving LLMs on MemoryBench and WildFB.

Reference graph

Works this paper leans on

24 extracted references · cited by 1 Pith paper

  1. [1]

    Wrap ALL your reasoning inside a single <thinking>...</thinking> block.\n

  2. [2]

    At the very start of <thinking>, output exactly ONE <feedback>...</feedback> block.\n - The <feedback> must be concise and actionable, and use this structure:\n <feedback>\n Issue:\n

  3. [3]

    Likely pitfalls: ...\n

  4. [4]

    Step-by-step plan:\n - (1) ...\n - (2) ...\n - (3) ...\n\n Fix steps:\n

  5. [5]

    = <final answer>’ inside <feedback>.)\n - You MAY include tiny snippets (a short identity, a one-line correction),\n but avoid long derivations or long equations in <feedback>.\n

    \n </feedback>\n - Give guidance / a repair plan, but do NOT give a full solution inside <feedback>.\n - Do NOT output any expression inside <feedback> that directly equals the final result.\n (e.g., do NOT write something like ’... = <final answer>’ inside <feedback>.)\n - You MAY include tiny snippets (a short identity, a one-line correction),\n but avo...

  6. [6]

    After </thinking>, on a new line, output the final numeric answer in the format:\n \boxed{answer}\n Do NOT add any extra text after the boxed answer.\n\n Example (format only):\n <thinking>\n <feedback>\n Issue:\n

  7. [7]

    Likely pitfalls: Misreading quantities; forgetting to combine changes.\n

  8. [8]

    Step-by-step plan:\n - (1) Identify the initial quantity and each change.\n - (2) Choose the correct operation (add/subtract/etc.).\n - (3) Compute carefully.\n\n Fix steps:\n

Show all 24 references
  1. [9]

    Extract numbers and what they represent.\n

  2. [10]

    Write the operation clearly.\n

  3. [11]

    Recompute the final arithmetic once.\n </feedback>\n Alice starts with 3 apples.\n She buys 2 more apples.\n Total apples = 3 + 2 = 5.\n </thinking>\n \boxed{5}\n Figure 6.Training system prompt. 20 MulFeRL: Enhancing Reinforcement Learning with Verbal Feedback in a Multi-turn...

  4. [15]

    \n </feedback> Figure 7.System prompt for obtaining feedback. Merge feedback system prompt You merge multiple feedback comments on an incorrect solution.\n\n Task:\n - Combine the feedback into ONE concise, actionable feedback.\n - Deduplicate repeated points.\n - Keep ONLY th...

  5. [17]

    Where it first goes wrong (quote 1--2 lines): "..."\n

  6. [18]

    Step-by-step errors:\n - (1) ...\n - (2) ...\n - (3) ...\n\n Fix steps:\n

  7. [19]

    \n </feedback> Figure 8.System prompt for merging feedback. 21 MulFeRL: Enhancing Reinforcement Learning with Verbal Feedback in a Multi-turn Loop Feedback injection and regeneration rollout prompt You are a problem reasoning assistant.\n Re-solve the problem from scratch. Use...

  8. [20]

    Start with EXACTLY: <thinking><feedback>{feedback}</feedback>\n

  9. [21]

    Continue your reasoning, then close: </thinking>\n

  10. [22]

    SFT/RAFT system prompt and CITL-FT initial prompt You are a reasoning assistant.\n Solve the problem step by step.\n\n Output format (must follow exactly):\n

    The final answer MUST be written as:\boxed{{answer}}\n Do NOT output anything else.\n\n Problem:\n {question}\n Start your new solution by continuing from the following prefix exactly:\n <thinking><feedback>{feedback}</feedback> Figure 9.Prompt for feedback injection and regen...

  11. [23]

    Wrap ALL reasoning inside a single <thinking>...</thinking> block.\n

  12. [24]

    After </thinking>, on a new line, output the final numeric answer in the format:\n \boxed{answer}\n Do NOT output any extra text after the boxed answer.\n Figure 10.SFT/RAFT prompt and CITL-FT initial prompt CITL-FT critique system prompt You are a strict reviewer of a solutio...

  13. [25]

    Earliest/root mistake: ...\n

  14. [26]

    Why it is wrong: ...\n

  15. [27]

    Fix plan (high-level):\n - (1) ...\n - (2) ...\n - (3) ...\n </critique>\n Figure 11.Critique prompt for CITL-FT 22 MulFeRL: Enhancing Reinforcement Learning with Verbal Feedback in a Multi-turn Loop CITL-FT refinement system prompt You are a reasoning assistant.\n Re-solve th...

  16. [28]

    <thinking> ... </thinking>\n 2)\boxed{answer}\n Do NOT output anything else.\n\n Inputs:\n Problem:{question}\n Initial solution:{initial response}\n Critique:{critique}\n Figure 12.Refinement prompt for CITL-FT 0 100 200 300 400 500 Training Step 0 10 20 30 40 50Validation Pe...

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.