REVIEW 4 cited by
Self-Boosting Large Language Models with Synthetic Preference Data
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Through alignment with human preferences, Large Language Models (LLMs) have advanced significantly in generating honest, harmless, and helpful responses. However, collecting high-quality preference data is a resource-intensive and creativity-demanding process, especially for the continual improvement of LLMs. We introduce SynPO, a self-boosting paradigm that leverages synthetic preference data for model alignment. SynPO employs an iterative mechanism wherein a self-prompt generator creates diverse prompts, and a response improver refines model responses progressively. This approach trains LLMs to autonomously learn the generative rewards for their own outputs and eliminates the need for large-scale annotation of prompts and human preferences. After four SynPO iterations, Llama3-8B and Mistral-7B show significant enhancements in instruction-following abilities, achieving over 22.1% win rate improvements on AlpacaEval 2.0 and ArenaHard. Simultaneously, SynPO improves the general performance of LLMs on various tasks, validated by a 3.2 to 5.0 average score increase on the well-recognized Open LLM leaderboard.
Forward citations
Cited by 4 Pith papers
-
SGPO: Self-Generated Preference Optimization based on Self-Improver
SGPO uses one shared model to refine its own responses and then optimize with DPO on those self-generated preference pairs, outperforming DPO and SPIN on AlpacaEval 2.0 and Arena-Hard without external preference labels.
-
Data Swarms: Optimizable Generation of Synthetic Evaluation Data
Data Swarms uses particle swarm optimization over data-generator LLM weights to produce synthetic evaluation data that scores higher on five quantitative evaluation objectives than eight baselines.
-
Amulet: Putting Complex Multi-Turn Conversations on the Stand with LLM Juries
Dialog-act and maxim-aware prompting improves LLM judge accuracy on multi-turn preference data by up to 8 points, with further gains from jury-style voting.
-
RACE-Align: Retrieval-Augmented and Chain-of-Thought Enhanced Preference Alignment for Large Language Models
RACE-Align generates preference pairs from RAG-grounded chain-of-thought answers and applies DPO to align a 1.7B model, showing improved reasoning scores in TCM QA but lacking statistical support.
Discussion (0). Sign in to comment.