REVIEW 14 cited by
Infinity Instruct: Scaling Instruction Selection and Synthesis to Enhance Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Infinity Instruct: Scaling Instruction Selection and Synthesis to Enhance Language Models
read the original abstract
Large Language Models (LLMs) demonstrate strong performance in real-world applications, yet existing open-source instruction datasets often concentrate on narrow domains, such as mathematics or coding, limiting generalization and widening the gap with proprietary models. To bridge this gap, we introduce Infinity-Instruct, a high-quality instruction dataset designed to enhance both foundational and chat capabilities of LLMs through a two-phase pipeline. In Phase 1, we curate 7.4M high-quality foundational instructions (InfInstruct-F-7.4M) from over 100M samples using hybrid data selection techniques. In Phase 2, we synthesize 1.5M high-quality chat instructions (InfInstruct-G-1.5M) through a two-stage process involving instruction selection, evolution, and diagnostic filtering. We empirically evaluate Infinity-Instruct by fine-tuning several open-source models, including Mistral, LLaMA, Qwen, and Yi, and observe substantial performance gains across both foundational and instruction following benchmarks, consistently surpassing official instruction-tuned counterparts. Notably, InfInstruct-LLaMA3.1-70B outperforms GPT-4-0314 by 8.6\% on instruction following tasks while achieving comparable foundational performance. These results underscore the synergy between foundational and chat training and offer new insights into holistic LLM development. Our dataset\footnote{https://huggingface.co/datasets/BAAI/Infinity-Instruct} and codes\footnote{https://gitee.com/li-touch/infinity-instruct} have been publicly released.
Forward citations
Cited by 14 Pith papers
-
Language Generation as Optimal Control: Closed-Loop Diffusion in Latent Control Space
The paper introduces Manta-LM, which approximates the Hamilton-Jacobi-Bellman optimal policy via Flow Matching in a rectified latent control space to enable high-fidelity parallel language generation.
-
SlimSpec: Low-Rank Draft LM-Head for Accelerated Speculative Decoding
SlimSpec replaces the standard LM-head in draft models with a low-rank version to deliver 4-5x faster speculative decoding while preserving full vocabulary and competitive acceptance rates.
-
K12-KGraph: A Curriculum-Aligned Knowledge Graph for Benchmarking and Training Educational LLMs
K12-KGraph is a textbook-derived knowledge graph that powers a new benchmark revealing LLMs' poor curriculum cognition and a small training corpus that outperforms general instruction data on educational tasks.
-
VITA-QinYu: Expressive Spoken Language Model for Role-Playing and Singing
VITA-QinYu is the first expressive end-to-end spoken language model supporting role-playing and singing alongside conversation, trained on 15.8K hours of data and outperforming prior models on expressiveness and conve...
-
ShareChat: A Dataset of Chatbot Conversations in the Wild
ShareChat is a large-scale dataset of 142,808 conversations from five major chatbot platforms that retains native affordances for cross-platform analyses of completeness, citations, and latency.
-
CARD: Cross-component Audio Representation Distillation for Encoder-Free Audio Captioning
Routing early CLAP stages to the audio projector and later stages to the LLM yields encoder-free captioning that beats LLM-only distillation by +12.2 CIDEr-D on AudioCaps.
-
Language Generation as Optimal Control: Closed-Loop Diffusion in Latent Control Space
Language generation is recast as optimal control and solved approximately with flow matching in rectified latent control space to enable high-fidelity parallel text generation.
-
Language Generation as Optimal Control: Closed-Loop Diffusion in Latent Control Space
Manta-LM approximates the HJB equation via flow matching in latent control space to realize closed-loop optimal control for language generation.
-
mllm-shap: A Shapley Value Explainability Platform for Text-Audio Multimodal Large Language Models
mllm-shap is an open-source Python platform extending Shapley Value explainability to text-audio Multimodal LLMs via modality-aware masking and phonetic token grouping.
-
From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizons
FLUID adapts AR LLMs into strictly causal diffusion models with entropy-driven elastic horizons, matching strong AR performance on reasoning and code while using only billions of adaptation tokens.
-
EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillation
EdgeRazor delivers 1.58-1.88 bit quantized LLMs that outperform 2-3 bit baselines by up to 11.3 points while using 4-10x less training compute than leading QAT methods.
-
EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillation
EdgeRazor uses structural mixed-precision quantization, layer-adaptive feature distillation, and entropy-aware KL divergence to achieve 1.88-bit LLMs that outperform prior 2-bit and 3-bit baselines with 4-10x lower tr...
-
MAR: Efficient Large Language Models via Module-aware Architecture Refinement
MAR integrates SSMs and sparsification with new ATMN neurons and SBDS distillation to produce efficient LLMs that match dense-model performance at substantially lower inference energy.
-
Enhancing Fitness Intelligence through Domain-Specific LLM Post-Training
FitOne-8B/32B models improve average scores on ACSM-EP and NSCA-CSCS certification exams by up to 12.73% over base Qwen3 while retaining general capabilities.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.