Pith. sign in

REVIEW 11 cited by

RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.08844 v2 pith:DBGUGFH3 submitted 2023-05-15 cs.CL

Afra Feyza Akyürek , Ekin Akyürek , Aman Madaan , Ashwin Kalyan , Peter Clark , Derry Wijaya , Niket Tandon This is my paper · ORCID
classification cs.CL
keywords feedbacklanguagemodelscritiqueoutputsrl4fcritiquesgenerators
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Despite their unprecedented success, even the largest language models make mistakes. Similar to how humans learn and improve using feedback, previous work proposed providing language models with natural language feedback to guide them in repairing their outputs. Because human-generated critiques are expensive to obtain, researchers have devised learned critique generators in lieu of human critics while assuming one can train downstream models to utilize generated feedback. However, this approach does not apply to black-box or limited access models such as ChatGPT, as they cannot be fine-tuned. Moreover, in the era of large general-purpose language agents, fine-tuning is neither computationally nor spatially efficient as it results in multiple copies of the network. In this work, we introduce RL4F (Reinforcement Learning for Feedback), a multi-agent collaborative framework where the critique generator is trained to maximize end-task performance of GPT-3, a fixed model more than 200 times its size. RL4F produces critiques that help GPT-3 revise its outputs. We study three datasets for action planning, summarization and alphabetization and show relative improvements up to 10% in multiple text similarity metrics over other learned, retrieval-augmented or prompting-based critique generators.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Formalizing Learning from Language Feedback with Provable Guarantees

    cs.LG 2025-06 conditional novelty 7.0 of 10

    Introduces a formal framework for learning from language feedback, a transfer eluder dimension complexity measure, and HELiX, a no-regret algorithm whose regret scales with this dimension.

  2. SGIC: A Self-Guided Iterative Calibration Framework for RAG

    cs.CL 2025-06 conditional novelty 6.0 of 10

    SGIC feeds a model's own uncertainty scores back into its prompt for several calibration rounds and improves RAG accuracy on HotpotQA, NQ, and GSM8K.

  3. Error-driven Data-efficient Large Multimodal Model Tuning

    cs.CL 2024-12 conditional novelty 6.0 of 10

    An error-driven teacher-student pipeline extracts a student LMM's missing skills from validation mistakes and retrieves targeted samples from a task-agnostic dataset to fine-tune it.

  4. Refining Answer Distributions for Improved Large Language Model Reasoning

    cs.CL 2024-12 conditional novelty 6.0 of 10

    RAD iteratively refines a distribution over answers by marginalizing over refinement samples, improving accuracy on six arithmetic benchmarks over self-consistency and hint-based prompting.

  5. AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinement

    cs.LG 2024-12 reject novelty 6.0 of 10

    AlphaVerus bootstraps a Llama-70B model to generate Rust code that passes the Verus verifier by iteratively translating Dafny programs and refining candidates with tree search.

  6. I2CR: Intra- and Inter-modal Collaborative Reflections for Multimodal Entity Linking

    cs.CV 2025-08 conditional novelty 5.0 of 10

    A text-first, multi-round visual feedback framework reports state-of-the-art top-1 accuracy on WikiMEL, WikiDiverse, and RichMEL.

  7. Boosting LLM Reasoning via Spontaneous Self-Correction

    cs.AI 2025-06 conditional novelty 5.0 of 10

    SPOC trains LLMs to interleave self-verification and solution attempts in a single pass, reporting gains on math benchmarks, though most gains come from stronger first attempts.

  8. Understanding the Dark Side of LLMs' Intrinsic Self-Correction

    cs.CL 2024-12 conditional novelty 5.0 of 10

    Intrinsic self-correction makes state-of-the-art LLMs overturn correct answers across four task types, and simple question repetition or tiny fine-tuning reduces this damage.

  9. A Survey on Progress in LLM Alignment from the Perspective of Reward Design

    cs.CL 2025-05 conditional novelty 4.0 of 10

    This paper organizes the LLM alignment literature into a reward-design-centered taxonomy and claims the field's evolution runs from rule-based to learned rewards and from RL-based to RL-free optimization.

  10. Towards Intrinsic Self-Correction Enhancement in Monte Carlo Tree Search Boosted Reasoning via Iterative Preference Learning

    cs.LG 2024-12 conditional novelty 4.0 of 10

    Adding a self-correction RL stage before MCTS-DPO improves arithmetic reasoning accuracy by 2.00 to 4.94 points on GSM8K and MATH.

  11. When Data Manipulation Meets Attack Goals: An In-depth Survey of Attacks for VLMs

    cs.CV 2025-02 conditional novelty 3.0 of 10

    A survey that classifies VLM attacks by goal and data manipulation strategy, and reviews defenses and metrics.

Pith tools