Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

Scaling Instruction-Tuned LLMs to Million-Token Contexts via Hierarchical Synthetic Data Generation

T0 review · 5 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A hierarchical synthetic-data pipeline plus stepwise RoPE scaling extends LLaMA-3.1-8B-Instruct to a 1M-token context window, beating prior baselines on long-context benchmarks with minimal loss on general tasks.

desk verdict Solid, well-ablated synthetic-data recipe for long-context instruction tuning, but the 1M-token understanding claim is not supported by the training data or the model's own undertrained state. read the letter →

arxiv 2504.12637 v1 pith:VOAAFGPV submitted 2025-04-17 cs.CL cs.AI

Linda He , Jue Wang , Maurice Weber , Shang Zhu , Ben Athiwaratkun , Ce Zhang This is my paper · ORCID
classification cs.CLcs.AI
keywords long-contextlanguagemodelssyntheticdatagenerationinstructiontuninghierarchicalquestion-answerRoPEscalingmillion-tokencontextRULERbenchmarkmulti-documentconcatenation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the bottleneck for long-context language models is not architecture but training data: high-quality instruction pairs with truly long contexts are scarce and expensive to annotate. It proposes generating such data synthetically by splitting documents into chunks, summarizing each chunk, and composing question-answer pairs that range from single-chunk details to multi-hop and cross-document synthesis. Combined with stepwise rotation-scaling of position embeddings (180K to 350K to 650K to 1M tokens), the pipeline extends LLaMA-3.1-8B-Instruct to a 1M-token window. On RULER the trained model keeps meaningful accuracy out to 1M tokens, it beats both the untuned base model and a 1M-capable baseline on InfiniteBench, and its LongBench and MMLU scores stay close to the original model's. If correct, the result means million-token instruction tuning no longer requires million-token human annotations.

What carries the argument

The load-bearing mechanism is the two-level hierarchical data generator: a document is split into small (4K-token) and medium (12K-token) chunks, each is summarized by an off-the-shelf short-context LLM, and Algorithm 1 walks question generation through the resulting hierarchy, starting from the whole-document summary, then a random medium chunk, then a small chunk inside it, then alternating between deeper dives into the current chunk and moves to new chunks, so the generated conversation mirrors the document's global-to-local structure. Running alongside this are hierarchical-aware, multi-hop, and specific-detail question prompts, and when several books are concatenated, revisits to earlier documents with a 60 percent probability create cross-document questions. The second mechanism is stepwise RoPE scaling: the model is fine-tuned successively at 180K, 350K, 650K, and 1M tokens, combining the hierarchical data at each length with rotary position embedding rescaling in the spirit of positional interpolation and NTK-aware scaling.

What would settle it

Build a probe set of 1M-token examples where the answer depends on facts in two chunks separated by more than half the context, with no intermediate summary containing either fact. If the trained model's accuracy on such bridge-gap questions collapses at 1M while its accuracy on questions answerable from a single chunk stays high, the RULER-style gains reflect local or summary reliance rather than true long-range integration. The same probe run on the 350K model would show whether the degradation appears as context grows.

Watch

Extended reading notes

Core claim

The central claim is that a locality-guided, hierarchy-aware synthetic data strategy, rather than raw long text, is what teaches an instruction-tuned model to reason across very long contexts. The authors split each document into 4K-token chunks nested inside 12K-token medium sections, summarize every level, and walk a question generator from the global summary down to specific chunks and back, alternating exploitation of the current chunk with exploration of new ones; multi-hop questions stitch 2-4 chunks together, and multiple documents are concatenated, with probabilistic revisits to earlier documents, to reach arbitrarily long contexts. Training proceeds in stages of increasing context length with scaled rotary position embeddings. The 1M-token model scores 62.95 percent on RULER at the full 1M length, against 48.81 percent for zero-shot RoPE scaling of the base model, averages 54.80 on InfiniteBench versus 51.31 for the base model and 41.04 for a 1M-capable comparative model, and lands within about three MMLU points of the base model. The authors themselves note that the 650K and 1M models appear undertrained because of multi-node communication overhead, which they believe depressed those results.

Load-bearing premise

The whole 1M-token claim rests on the assumption that question-answer pairs built from chunk summaries and local excerpts teach the model to genuinely integrate distant context, rather than to answer from whichever nearby summary or local passage holds the answer.

Editorial extensions

If this is right

  • Million-token instruction datasets can be built by concatenating ordinary books with LLM-generated hierarchical QA pairs, removing the need for expensive long-context human annotations.
  • The stepwise RoPE schedule suffices to push an off-the-shelf 8B instruct model to a 1M window using only a few hundred 1M-token samples at the final stage (200 in this paper).
  • Models trained this way keep most of their short-context ability, with LongBench staying within roughly 1.7 points and MMLU within roughly 3 points of the base 8B model at 1M context.
  • Smaller or same-size generator models (7B and 8B) produce data that transfers nearly as well as data from a 72B generator, suggesting the pipeline does not depend on a top-tier generator.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The ablations leave open a sharper test the paper does not run: whether the gains persist when questions require integrating facts from widely separated chunks with no summarizing bridge. A natural experiment is to adversarially construct 1M-token questions whose answers are spread across non-adjacent documents and compare accuracy against same-length questions answerable from a single chunk.
  • The hierarchical summaries act as a learned compression bottleneck; one could test the same data pipeline on models with different position encodings, such as ALiBi, to see whether the data itself, rather than the RoPE rescaling, is doing the work.
  • The paper's own note that the 650K and 1M models are undertrained suggests the reported numbers may be a floor, and retraining with better parallelism or more epochs could raise the headline RULER and InfiniteBench scores.
  • Because the pipeline is data-side, it composes with architectural advances; feeding the same hierarchical data into linear-attention or state-space models would test whether data scarcity or attention cost dominates at very long context.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes a post-training pipeline that uses short-context LLMs to generate long-context instruction-tuning data hierarchically. Documents are split into small and medium chunks, summarized, and used to produce hierarchical, multi-hop, and diverse QA pairs; multiple documents are then concatenated to reach 180K, 350K, 650K, and 1M token training contexts. The authors fine-tune LLaMA-3.1-8B-Instruct with stepwise RoPE scaling on this data and report results on RULER, InfiniteBench, LongBench, and MMLU, claiming strong performance at up to 1M tokens while preserving general abilities. They also include ablations at 100K and 180K context lengths and comparisons against a Gradient AI 1M-context model.

Significance. If the empirical claims hold, the work would be a useful contribution to long-context instruction tuning: it provides a scalable synthetic-data recipe, systematic ablations of data composition choices, and evidence that data quality matters beyond positional-embedding scaling alone. The paper's strengths include explicit ablations at 100K and 180K, validation with multiple generator models, and an attempt to preserve short-context performance. However, the headline 1M-token claim is not yet established: the training data never requires cross-evidence synthesis beyond roughly 16K tokens, the 1M model is acknowledged to be undertrained, RULER comparisons lack error bars, and the main baseline comparison is confounded by different base models. These issues are addressable and do not invalidate the core data-generation idea, but they currently prevent a strong endorsement of the central claim.

major comments (5)
  1. [Section 3.1, Algorithm 1, Appendix C] The QA supervision generated by the pipeline never requires the model to integrate evidence separated by more than roughly 16K tokens: every question is produced from a 4K chunk, a summary, or 2-4 chunks within a 12K medium section (Algorithm 1; Section 3.1), and the multi-document examples concatenate documents without creating cross-document reasoning targets beyond revisiting prior documents (Section 3.2). As a result, the RULER and InfiniteBench improvements at 1M could reflect improved local retrieval and attention over long inputs rather than genuine synthesis across distant evidence. Please add an evaluation that explicitly manipulates the distance between evidence spans (e.g., multi-hop RULER tasks with needles 100K+ tokens apart) or otherwise demonstrate that the model can bind distant evidence.
  2. [Section 4.2, footnote] The footnote in Section 4.2 states that the 650K and 1M models are undertrained and that results dropped due to multi-node training overhead. The flagship 1M RULER numbers in Table 8 and Figure 4 are therefore produced by a model that did not complete training. This is load-bearing: the main contribution is the 1M-context result, and the paper does not show what a fully trained model would achieve. Please retrain to convergence (or address the training instability) and report results, or explicitly reframe the 1M results as preliminary and support them with an analysis of training dynamics.
  3. [Tables 1-8] The paper reports a single point estimate for each model/task, and the RULER numbers in Table 8 have no error bars or repeated-generation variance, although the tasks are sampled from a generative benchmark. Many of the central comparisons are small differences (e.g., Table 1: 180K 59.45 vs 350K 59.26; Table 7: h-h-s-fixed 58.58 vs h-h-h-randomized 58.14), and 'significantly exceeding' in Section 4.2 is not backed by any statistical test. Please provide variance estimates, seeds, and significance tests for the reported differences.
  4. [Tables 1, 2, 3; Section 4.2] The comparison against gradientai/Llama-3-8B-Instruct-Gradient-1048k in Tables 1-6 is confounded with base model: the Gradient model starts from Llama-3-8B, whereas all proposed models start from LLaMA-3.1-8B-Instruct, which is substantially stronger on MMLU (68.21 vs 60.48 in Table 3). The claim that the method 'outperforms' the Gradient model is therefore not an apples-to-apples comparison. A matched evaluation using the same base model (or a Llama-3.1-based 1M baseline) is needed.
  5. [Section 4.2] Section 4.2 describes 'stepwise rope scaling' but does not specify the scaling method, factors, schedule, or the exact training recipe (e.g., whether NTK-aware or YaRN interpolation was used, how the RoPE base frequency was changed at each stage). Since the result depends on this recipe as much as on the synthetic data, the missing details prevent reproduction and make it impossible to attribute the gains to the data pipeline. Please provide the full configuration (or point to released code) in the main text or appendix.
minor comments (6)
  1. [Section 4.4, Table 7] The text 'hs-hs-fixed-no-sum scored 58.03, slightly below hs-hs-hs-fixed (58.58)' does not match Table 7, which shows h-h-s-fixed-no-sum at 58.03 and h-h-s-fixed at 58.58, while hs-hs-hs-fixed is 59.45. Please correct the configuration labels and numbers.
  2. [Section 4.1] The RULER evaluation description says 130 tasks per context length were sampled, but no seed or version of RULER is given; please report the exact configuration for reproducibility.
  3. [Abstract and Introduction] The claim that no open-source instruction-tuning dataset exceeds 100K tokens is not accompanied by a statement of whether the generated 180K/350K/650K/1M dataset will be released; the reproducibility section only provides code. Please clarify the release status.
  4. [Section 4.2] 'Significantly exceeding' is used for differences without statistical tests; please either add tests or use descriptive language.
  5. [Section 4.3, Figure 5] The RULER curves for Llama-3.1-8B and Qwen-2.5-7B generators are not accompanied by the numerical table that exists for the Qwen-2-72B generator (Table 8); please add the analogous numbers for reproducibility.
  6. [General] There are minor typographical errors such as 'Table 2, , our models' in Section 4.2 and '4 hierachies' in Appendix E; please proofread the manuscript.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper's claims are supported by external benchmarks and empirical ablations rather than by construction or self-citation.

full rationale

The paper's load-bearing claim is that a hierarchical synthetic QA pipeline plus stepwise RoPE scaling yields a model that performs well on RULER and InfiniteBench while retaining general-task performance. Nothing in the training-data construction fixes or predicts those benchmark scores. The QA pairs are generated from book chunks and summaries (Section 3.1, Algorithm 1), whereas evaluation uses external benchmarks RULER, InfiniteBench, LongBench, and MMLU with fixed protocols (Section 4.1). The ablations (Section 4.4) choose among data-recipe variants using InfiniteBench at 100K/180K, but the later 350K/650K/1M results are new evaluations, not statistical consequences of the ablation choices. There is no fitted parameter that is later relabeled as a prediction, and no load-bearing self-citation: the authors cite standard related work and external baselines such as the Gradient AI model, and their own Together dataset/API are infrastructure rather than evidence for the central claim. The footnote that 650K and 1M models are undertrained is an explicit limitation about training time, not a circular reduction. The skeptic's concern that generated QAs may not force true cross-100K synthesis is a substantive generalization/correctness risk, not a circularity: the training data and evaluation tasks are distinct, and the measured gains could be real even if the mechanism is partly local retrieval. Overall, the derivation is self-contained and empirically grounded, so the appropriate circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim depends on hand-chosen data mixture parameters (N1, N2, N3, revisit probability, chunk sizes), on the quality of machine-generated labels, on benchmark validity, and on no train/test contamination. These are not fitted to the evaluation results; the method is empirical rather than derived.

free parameters (5)
  • N1 = 5
    Number of hierarchical and diverse QA pairs generated per document. Chosen by hand in Section 3.2; ablations use 6 instead in some configs.
  • N2 = 9
    Number of diverse QA pairs sampled from previously visited documents. Hand-chosen in Section 3.2.
  • N3 = 3
    Number of follow-up hierarchical questions per revisited document. Hand-chosen in Section 3.2.
  • revisit probability = 0.6
    Probability of revisiting a previous document for follow-up questions. Chosen in Algorithm 2 and Section 3.2.
  • chunk sizes = 4K and 12K tokens
    Document chunk sizes for hierarchical splitting in Section 3.1; these determine the granularity of questions and summaries.
assumptions (4)
  • domain assumption Synthetic QA pairs generated by Qwen-2-72B are accurate and not hallucinated.
    The whole instruction dataset is machine-generated; any systematic hallucination would propagate into training labels. Section 4.1 describes generation by API without human verification.
  • domain assumption RULER, InfiniteBench, LongBench, and MMLU are valid measures of the claimed capabilities.
    Benchmark scores are treated as proxies for long-context reasoning and general ability; the paper does not validate transfer to real-world long-context tasks.
  • domain assumption The training book corpus does not overlap with benchmark evaluation documents.
    Both training data (Together long books) and InfiniteBench book tasks draw on public-domain books; contamination is not checked, so reported gains could be inflated. Section 4.1 datasets.
  • domain assumption Stepwise RoPE scaling preserves pretrained knowledge.
    The paper uses stepwise rope scaling without specifying the schedule or scaling factors, assuming the base model's abilities survive the extension. Section 4.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling Instruction-Tuned LLMs to Million-Token Contexts via Hierarchical Synthetic Data Generation." pith.science (2026). https://pith.science/paper/VOAAFGPV

@misc{pith2026250412637,
  author       = {Pith},
  title        = {Pith review of: Scaling Instruction-Tuned LLMs to Million-Token Contexts via Hierarchical Synthetic Data Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VOAAFGPV}},
  note         = {Machine review of arXiv:2504.12637}
}
read the original abstract

Large Language Models (LLMs) struggle with long-context reasoning, not only due to the quadratic scaling of computational complexity with sequence length but also because of the scarcity and expense of annotating long-context data. There has been barely any open-source work that systematically ablates long-context data, nor is there any openly available instruction tuning dataset with contexts surpassing 100K tokens. To bridge this gap, we introduce a novel post-training synthetic data generation strategy designed to efficiently extend the context window of LLMs while preserving their general task performance. Our approach scalably extends to arbitrarily long context lengths, unconstrained by the length of available real-world data, which effectively addresses the scarcity of raw long-context data. Through a step-by-step rotary position embedding (RoPE) scaling training strategy, we demonstrate that our model, with a context length of up to 1M tokens, performs well on the RULER benchmark and InfiniteBench and maintains robust performance on general language tasks.

Figures

Figures reproduced from arXiv: 2504.12637 by the authors.

Figure 1
Figure 1. High-level overview over our approach to automatically generate QA pairs for long context [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. High-level overview over our approach to generate order-following QAs. (1) Input a raw [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. High-level overview over our approach to curate long context data using multiple doc [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Effective context length up 1M tokens using Qwen-2-72B-Instruct as generator on RULER [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Effective context length using Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct as generators on RULER [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL

    cs.AI 2026-05 conditional novelty 7.0 of 10

    Masked diffusion language models, not larger autoregressive LLMs, are the better building block for text-based world models in agentic RL, improving rollout fidelity, diversity, and downstream task success.

Reference graph

Works this paper leans on

34 extracted references · 1 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  3. [3]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  4. [4]

    ִ|z !|؆l- b)<v kڰ2<WנXy<PG OV< /|4; 'K

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    work page 1999

  5. [5]

    Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H

    Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Z...

    arXiv 2023

  6. [6]

    Longbench: A bilingual, multitask benchmark for long context understanding, 2024

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding, 2024. URL https://arxiv.org/abs/2308.14508

    arXiv 2024

  7. [7]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer, 2020. URL https://arxiv.org/abs/2004.05150

    arXiv 2020

  8. [8]

    Extending context window of large language models via positional interpolation, 2023

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation, 2023. URL https://arxiv.org/abs/2306.15595

    arXiv 2023

Show all 34 references
  1. [9]

    Longlora: Efficient fine-tuning of long-context large language models, 2024

    Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models, 2024. URL https://arxiv.org/abs/2309.12307

    2024 arXiv

  2. [10]

    Mamba: Linear-time sequence modeling with selective state spaces, 2024

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024. URL https://arxiv.org/abs/2312.00752

    2024 arXiv

  3. [11]

    Longt5: Efficient text-to-text transformer for long sequences, 2022

    Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. Longt5: Efficient text-to-text transformer for long sequences, 2022. URL https://arxiv.org/abs/2112.07916

    2022 arXiv

  4. [12]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009.03300

    2021 arXiv

  5. [13]

    Block transformer: Global-to-local language modeling for fast inference, 2024

    Namgyu Ho, Sangmin Bae, Taehyeon Kim, Hyunjik Jo, Yireun Kim, Tal Schuster, Adam Fisch, James Thorne, and Se-Young Yun. Block transformer: Global-to-local language modeling for fast inference, 2024. URL https://arxiv.org/abs/2406.02657

    2024 arXiv

  6. [14]

    Ruler: What's the real context size of your long-context language models?, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models?, 2024. URL https://arxiv.org/abs/2404.06654

    2024 arXiv

  7. [15]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  8. [16]

    Transformers are rnns: Fast autoregressive transformers with linear attention, 2020

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention, 2020. URL https://arxiv.org/abs/2006.16236

    2020 arXiv

  9. [17]

    Loogle: Can long-context language models understand long contexts?, 2024

    Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. Loogle: Can long-context language models understand long contexts?, 2024. URL https://arxiv.org/abs/2311.04939

    2024 arXiv

  10. [18]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts, 2023. URL https://arxiv.org/abs/2307.03172

    2023 arXiv

  11. [19]

    Base of rope bounds context length, 2024

    Xin Men, Mingyu Xu, Bingning Wang, Qingyu Zhang, Hongyu Lin, Xianpei Han, and Weipeng Chen. Base of rope bounds context length, 2024. URL https://arxiv.org/abs/2405.14591

    2024 arXiv

  12. [20]

    Locating and editing factual associations in gpt, 2023

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt, 2023. URL https://arxiv.org/abs/2202.05262

    2023 arXiv

  13. [21]

    Leave no context behind: Efficient infinite context transformers with infini-attention, 2024

    Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient infinite context transformers with infini-attention, 2024. URL https://arxiv.org/abs/2404.07143

    2024 arXiv

  14. [22]

    Yarn: Efficient context window extension of large language models, 2023

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models, 2023. URL https://arxiv.org/abs/2309.00071

    2023 arXiv

  15. [23]

    Smith, and Mike Lewis

    Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation, 2022. URL https://arxiv.org/abs/2108.12409

    2022 arXiv

  16. [24]

    Roformer: Enhanced transformer with rotary position embedding, 2023

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. URL https://arxiv.org/abs/2104.09864

    2023 arXiv

  17. [25]

    A length-extrapolatable transformer, 2022

    Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A length-extrapolatable transformer, 2022. URL https://arxiv.org/abs/2212.10554

    2022 arXiv

  18. [26]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language...

    2023 arXiv

  19. [27]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions, 2023. URL https://arxiv.org/abs/2212.10560

    2023 arXiv

  20. [28]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2...

    2022 arXiv

  21. [29]

    Bloomberggpt: A large language model for finance, 2023

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance, 2023. URL https://arxiv.org/abs/2303.17564

    2023 arXiv

  22. [30]

    Effective long-context scaling of foundation models, 2023

    Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, S...

    2023 arXiv

  23. [31]

    Wizardlm: Empowering large language models to follow complex instructions, 2023

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions, 2023. URL https://arxiv.org/abs/2304.12244

    2023 arXiv

  24. [32]

    Automatic instruction evolving for large language models, 2024

    Weihao Zeng, Can Xu, Yingxiu Zhao, Jian-Guang Lou, and Weizhu Chen. Automatic instruction evolving for large language models, 2024. URL https://arxiv.org/abs/2406.00770

    2024 arXiv

  25. [33]

    bench: Extending long context evaluation beyond 100k tokens, 2024

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. bench: Extending long context evaluation beyond 100k tokens, 2024. URL https://arxiv.org/abs/2402.13718

    2024 arXiv

  26. [34]

    Longskywork: A training recipe for efficiently extending context length in large language models, 2024

    Liang Zhao, Tianwen Wei, Liang Zeng, Cheng Cheng, Liu Yang, Peng Cheng, Lijie Wang, Chenxia Li, Xuejie Wu, Bo Zhu, Yimeng Gan, Rui Hu, Shuicheng Yan, Han Fang, and Yahui Zhou. Longskywork: A training recipe for efficiently extending context length in large language models, 202...

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.