REVIEW 25 cited by
Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large Language Models (LLMs) are rapidly surpassing human knowledge in many domains. While improving these models traditionally relies on costly human data, recent self-rewarding mechanisms (Yuan et al., 2024) have shown that LLMs can improve by judging their own responses instead of relying on human labelers. However, existing methods have primarily focused on improving model responses rather than judgment capabilities, resulting in rapid saturation during iterative training. To address this issue, we introduce a novel Meta-Rewarding step to the self-improvement process, where the model judges its own judgements and uses that feedback to refine its judgment skills. Surprisingly, this unsupervised approach improves the model's ability to judge {\em and} follow instructions, as demonstrated by a win rate improvement of Llama-3-8B-Instruct from 22.9% to 39.4% on AlpacaEval 2, and 20.6% to 29.1% on Arena-Hard. These results strongly suggest the potential for self-improving models without human supervision.
Forward citations
Cited by 25 Pith papers
-
SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models
SPaR trains LLMs by having them refine their own failed responses through tree search, producing preference pairs with minimal irrelevant variation, and lifts LLaMA3-8B above GPT-4-Turbo on IFEval.
-
Self-Improvement in Language Models: The Sharpening Mechanism
Self-improvement in language models can be understood as amortizing best-of-N inference-time selection, with minimax-optimal guarantees for SFT and provable coverage-free benefits for RL with exploration.
-
Multilingual Self-Taught Faithfulness Evaluators
STEMF trains multilingual faithfulness evaluators from synthetic data alone, and English-only training yields the best average results across languages.
-
SGPO: Self-Generated Preference Optimization based on Self-Improver
SGPO uses one shared model to refine its own responses and then optimize with DPO on those self-generated preference pairs, outperforming DPO and SPIN on AlpacaEval 2.0 and Arena-Hard without external preference labels.
-
Bridging Offline and Online Reinforcement Learning for LLMs
Semi-online DPO, which syncs the generation model every few update steps, performs nearly as well as fully online DPO and GRPO, while strongly beating offline DPO.
-
From General to Targeted Rewards: Surpassing GPT-4 in Open-Ended Long-Context Generation
ProxyReward trains long-form generation models by rewarding how well an AI judge can answer generated yes/no questions about the response, improving open-source models on ProxyQA.
-
PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifier
A new multi-turn reinforcement learning framework trains a single LLM to both solve math problems and verify its own solutions, revising only when its verifier finds a mistake.
-
Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning
CoVo trains LLMs with a self-generated reward based on the consistency and volatility of intermediate reasoning states, matching supervised RL performance without external labels.
-
Unlocking Recursive Thinking of LLMs: Alignment via Refinement
An offline alignment pipeline using reward-filtered self-refinement data and long chain-of-thought SFT raises an 8B model's AlpacaEval 2 win rate from 25.0% to 51.0% with roughly 14k training examples.
-
LLM in the Loop: Creating the ParaDeHate Dataset for Hate Speech Detoxification
An LLM-only pipeline created the ParaDeHate hate speech detoxification dataset, but the claim that it matches human annotation quality is not supported by the experiments.
-
Arbiters of Ambivalence: Challenges of Using LLMs in No-Consensus Tasks
Across five LLMs and ten no-consensus datasets, neutrality drops sharply when models act as pairwise judges, pointwise judges, or debaters compared to when they generate answers with an explicit neutral option.
-
Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Models
Forcing a self-rewarding LLM's generative judge and implicit DPO reward to agree on preference labels substantially improves its AlpacaEval 2.0 alignment score.
-
R.I.P.: Better Models by Survival of the Fittest Prompts
RIP filters preference-optimization training data by keeping prompts whose rejected responses are high quality and whose chosen/rejected reward gap is small, yielding consistent benchmark gains over unfiltered data.
-
M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluation
M-MAD decouples MQM criteria into four dimensions and uses per-dimension multi-agent debate, achieving better WMT23 meta-evaluation scores than prior LLM-as-a-judge methods and rivaling learned metrics.
-
WarriorCoder: Learning from Expert Battles to Augment Code Large Language Models
A code LLM fine-tuned on winner responses from pairwise expert battles, with instructions mined from chat templates, beats same-size baselines without proprietary LLMs.
-
Self-Generated Critiques Boost Reward Modeling for Language Models
Self-generated, label-filtered critiques jointly trained with reward prediction improve preference accuracy and data efficiency without a stronger teacher model.
-
Overconfidence in LLM-as-a-Judge: Diagnosis and Confidence-Driven Solution
LLM-as-a-Judge systems report confidence that overstates their accuracy, and the paper's TH-Score plus LLM-as-a-Fuser improves calibration.
-
Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future
Anchoring rejected responses to the initial model and choosing responses from a future model raises AlpacaEval 2.0 win rate from 19.69 to 29.44 for Llama3.1-8B.
-
Enterprise Large Language Model Evaluation Benchmark
A 14-task enterprise LLM benchmark built mostly from GPT-4o-generated labels and scored by GPT-4o-as-judge shows open-source models closing the reasoning gap, but the dataset is not public and the evaluation is partly...
-
Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers
EXSEARCH trains LLMs for agentic search by treating search trajectories as latent variables and optimizing a weighted likelihood via expectation-maximization, yielding gains on NQ, HotpotQA, MuSiQue, and 2WikiQA.
-
Revisiting Active Learning under (Human) Label Variation
A position paper that surveys and systematizes how active learning should change when human label variation is treated as a signal rather than noise.
-
Beyond the Surface: Measuring Self-Preference in LLM Judgments
The DBG metric measures LLM self-preference bias as the gap between a judge model's own win rate and the win rate assigned by an ensemble of gold judges.
-
Reward Reasoning Model
Reward models that think before judging, trained via reinforcement learning without human-written reasoning traces, outperform standard reward models and improve with more test-time compute.
-
Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineering
A survey and position paper that reframes post-training of foundation models as a search, verify, and feedback pipeline called verifier engineering.
- Hateful Person or Hateful Model? Investigating the Role of Personas in Hate Speech Detection by Large Language Models
Discussion (0). Continue with ORCID to comment.