REVIEW 6 cited by
Self-Refine Instruction-Tuning for Aligning Reasoning in Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The alignments of reasoning abilities between smaller and larger Language Models are largely conducted via Supervised Fine-Tuning (SFT) using demonstrations generated from robust Large Language Models (LLMs). Although these approaches deliver more performant models, they do not show sufficiently strong generalization ability as the training only relies on the provided demonstrations. In this paper, we propose the Self-refine Instruction-tuning method that elicits Smaller Language Models to self-refine their abilities. Our approach is based on a two-stage process, where reasoning abilities are first transferred between LLMs and Small Language Models (SLMs) via Instruction-tuning on demonstrations provided by LLMs, and then the instructed models Self-refine their abilities through preference optimization strategies. In particular, the second phase operates refinement heuristics based on the Direct Preference Optimization algorithm, where the SLMs are elicited to deliver a series of reasoning paths by automatically sampling the generated responses and providing rewards using ground truths from the LLMs. Results obtained on commonsense and math reasoning tasks show that this approach significantly outperforms Instruction-tuning in both in-domain and out-domain scenarios, aligning the reasoning abilities of Smaller and Larger Language Models.
Forward citations
Cited by 6 Pith papers
-
Aligning VLM Assistants with Personalized Situated Cognition
The authors present PCogAlignBench, an 18k-sample benchmark of visual scenes with role-based users, and PCogAlign, a framework using a cognition-aware reward model to produce responses aligned with each user's roles.
-
ARB: A Comprehensive Arabic Multimodal Reasoning Benchmark
ARB provides 1,356 Arabic multimodal questions with 5,119 human-reviewed reasoning steps and shows leading models score much higher on reasoning fluency than on correct answers.
-
Future-KL Regularized GRPO: Process-Level Credit Assignment from $f$-Divergence Regularization
Abstract claims FRPO adds a future-KL correction to GRPO for better math reasoning; the provided manuscript body is a different PRL paper and does not support those claims.
-
Self-Training Large Language Models with Confident Reasoning
CORE-PO self-trains LLMs to prefer reasoning paths with high self-estimated confidence, improving answer and reasoning accuracy on several benchmarks.
-
Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectives
A survey proposes a macro-meso-micro value framework for agentic AI alignment and maps applications, methods, and benchmarks onto it.
-
Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
A broad survey of LLM alignment that catalogs objectives, benchmarks, SFT/RLHF/DPO methods, and safety challenges, without contributing new experimental or theoretical results.
Discussion (0). Sign in to comment.