REVIEW 5 cited by
Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Model
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Recent advancements in large language models (LLMs) have highlighted the importance of extending context lengths for handling complex tasks. While traditional methods for training on long contexts often use filtered long documents, these approaches lead to domain imbalances, limiting model performance. To address this, techniques like random document concatenation (Standard) and similarity-based methods (KNN, ICLM) have been developed. However, they either sacrifice semantic coherence or diversity. To balance both aspects, we introduce Quest, a query-centric data synthesis method aggregating semantically relevant yet diverse documents. Quest uses a generative model to predict potential queries for each document, grouping documents with similar queries and keywords. Extensive experiments demonstrate Quest's superior performance on long-context tasks, achieving remarkable results with context lengths of up to 1M tokens and confirming its scalability across various model sizes.
Forward citations
Cited by 5 Pith papers
-
Libra: Large Chinese-based Safeguard for AI Content
Libra-Guard, a Chinese-language AI content safety model, reaches 86.8% accuracy on the new 5,720-sample Libra-Test benchmark, beating open-source baselines and approaching GPT-4o and Claude-3.5-Sonnet.
-
NExtLong: Toward Effective Long-Context Training without Long Documents
Interleaving hard negative distractors between chunks of short documents improves long-context language model performance on HELMET and RULER.
-
Bootstrap Your Own Context Length
The authors synthesize long-context training data from short chunks and fine-tune Llama-3 models to reach 1M-token contexts with top open-source RULER scores.
-
LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions
By prompting an aligned LLM with a document and the special token that precedes a user query, LongMagpie synthesizes long-context instruction data that outperforms prior datasets when used to fine-tune Llama-3-8B.
-
RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?
Long-CoT fine-tuning with QwQ-generated traces improves math and geometry benchmarks, and RedStar-DPO surpasses QwQ on AIME24, but several reported numbers conflict with the paper's own tables.
Discussion (0). Continue with ORCID to comment.