Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

Bootstrap Your Own Context Length

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Long-context ability can be bootstrapped from short-context data alone, extending Llama-3 to 1M tokens with near-perfect needle retrieval.

desk verdict A practical, compute-light recipe for extending Llama-3 to 1M context using synthetic data; the main open question is whether the model learns genuine long-context ability or a narrower task profile. read the letter →

arxiv 2412.18860 v2 pith:TB7PEXAM submitted 2024-12-25 cs.CL cs.IR

Liang Wang , Nan Yang , Xingxing Zhang , Xiaolong Huang , Furu Wei This is my paper · ORCID
classification cs.CLcs.IR
keywords long-contextlanguagemodelscontextextensionsyntheticinstructiondatabootstrappingquery-focusedsummarizationback-translationretrieval-augmentedgenerationRULERbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that long-context ability can be bootstrapped from short-context ability alone: a short-context instruction-following model, a text retriever, and a document collection are enough to synthesize instruction-tuning data that trains the same model family to operate on contexts up to 1M tokens. The synthesis workflow generates diverse instructions, retrieves relevant documents, recursively summarizes document chunks with query-focused agents, and produces answers from the summaries. The model is then fine-tuned on the instruction plus the full retrieved documents, with the generated answer as the target. The paper reports that this lightweight post-training extends the Llama-3 family to 1M tokens with near-perfect needle-in-haystack retrieval and a RULER average of 69.6 for the 8B model at 1M, surpassing open long-context baselines. If this holds, it matters because long-context data no longer needs to be scarce or hand-annotated: the model generates its own curriculum from short inputs.

What carries the argument

The load-bearing mechanism is a four-step data-synthesis loop called an agent workflow: (1) an LLM is prompted, with a random text chunk prepended, to generate diverse multi-document instructions and search queries; (2) a text retriever pulls relevant documents from a large corpus; (3) query-focused summarization agents recursively compress each <=4k-token chunk, keeping only information relevant to the instruction, until the summaries fit a short context; and (4) an LLM generates the final answer from the summaries. During fine-tuning, the input is the instruction plus all retrieved documents in full and the target is the generated answer; the intermediate summaries are discarded, so the model must learn end-to-end to do the relevance filtering itself. A second mechanism, instruction back-translation, generates writing instructions from long documents to extend output length. Training uses progressive context-length stages with doubled sequence length and quadrupled RoPE base frequency, plus RingAttention and PoSE-style training where full-length fine-tuning is not feasible.

What would settle it

Train the identical recipe but scramble the document ordering or replace retrieved documents with unrelated documents of equal length while keeping the same query-focused answers; if RULER and needle-in-haystack scores stay high, the model is relying on the answer's style or parametric memory rather than on reading the context. A second check: build a RULER-style QA task where the answer requires combining one fact from each of two distant chunks, with neither chunk individually sufficient, and test SelfLong-8B-1M on it; systematic failure would indicate that end-to-end cross-chunk integration has not actually been learned.

Watch

Extended reading notes

Core claim

The central claim is that you can transfer the capabilities a model already has on short inputs to long inputs by training it on synthetic examples that pair a long concatenated document set with answers that were produced by an agent workflow that only ever read short chunks. On the RULER benchmark, SelfLong-8B-1M scores 89.5 at 32k and 69.6 at 1M, outperforming prior open long-context models such as Llama-3-8B-1M (64.3) and LWM-Text-Chat-1M (60.1) at 1M, while the needle-in-haystack test stays near perfect across depths even at 1M. The same recipe also extends output length: average generated length grows from 1.8k tokens for the base model to 14.5k tokens toward a 17.1k groundtruth. Progressive training through 256k, 512k, and 1M lengths with an adjusted rotary-position-embedding base beats a direct jump to 1M. The authors frame the result as a recipe rather than a full solution, noting that performance still declines as context grows and that long output generation remains imperfect.

Load-bearing premise

The pipeline assumes that when the model is trained on full documents with answers that were produced from short summaries, it learns to perform the same relevance filtering and cross-document integration on its own; if it instead latches onto answer-shaped clues or parametric knowledge, long-context scores will not reflect genuine long-range reasoning.

Editorial extensions

If this is right

  • Any instruction-tuned short-context LLM can in principle be extended to 1M tokens using only readily available open components, with no manual long-context annotation.
  • Long-context RULER scores improve with training length beyond the test length (for example, the 256k test score peaks when trained at 512k), suggesting progressive training is the right schedule.
  • The synthetic-data contribution is necessary: short instruction data alone or an adjusted RoPE base alone gives much lower long-context scores, while removing the synthetic data drops the 1M RULER average from 69.6 to 63.5.
  • Using the model's own short-context variant as the data-synthesis backbone works reasonably (66.1 RULER average at 1M), so full bootstrapping without a stronger proprietary teacher is partially supported.
  • Output length is trainable too: after fine-tuning, the model's generated outputs move from about 1.8k to about 14.5k tokens when the requested length is up to 32k, though repetition remains a failure mode.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The method implicitly treats long-context understanding as decomposable into retrieval plus query-focused summarization; a natural testable extension is whether training the model to emit its own intermediate summaries before the final answer improves performance on RULER-style multi-hop tasks beyond what the current end-to-end training achieves.
  • If the transfer is genuine, the same bootstrapping recipe should work for other formats, such as long code repositories or long multi-document math, as long as a short-context model can generate the intermediate summaries; this is an extension the paper does not test.
  • Because the training answers come from chunk-level summaries, the model may learn to recognize answer-shaped content rather than perform true cross-chunk inference; a probing task that requires combining two facts from distant chunks would separate those mechanisms.
  • The RULER numbers at 256k and beyond use 100 samples per task, so the reported 1M averages may carry sampling noise; re-running with more samples would pin down the degradation curve.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a bootstrapping approach to extend LLM context length by synthesizing long-context instruction-tuning data using a short-context LLM, a retriever, and a document corpus. The synthesis workflow generates instructions, retrieves documents, recursively applies query-focused summarization (QFS) agents to chunks, and generates answers from the summaries. The model is then fine-tuned on the concatenated documents with the generated answers as targets, using progressive training with increased RoPE base frequencies and RingAttention. Additional long-output data is created via instruction back-translation. Experiments with Llama-3 1B/3B/8B show extension to 1M tokens, with SelfLong-8B-1M achieving a RULER average of 69.6 at 1M and near-perfect needle-in-haystack scores. The paper includes ablations demonstrating the contributions of synthetic data, progressive training, and the synthesis model choice, and it reports a drop on IFEval and mixed results on short-context tasks.

Significance. If the main result holds, this is a practically valuable recipe: it turns abundant short-context instruction data and a retrieval corpus into long-context supervision without manual annotation, and it releases open models. The strongest aspects are the controlled ablations (Table 3) showing that synthetic long data, progressive training, and teacher model choice matter, and the use of an external benchmark (RULER) for the main comparison. The central caveat is that the RULER average at long lengths may be dominated by needle-retrieval tasks; the near-zero cwe scores and the agent workflow's own weakness on vt suggest the model may be learning a task profile rather than general long-context reasoning. The paper's honesty about these limitations is a partial mitigation, but the evaluation needs to be strengthened before the broad claim is fully supported.

major comments (3)
  1. [§4.2, Table 8; §5.4, Table 5] The claim that SelfLong-8B-1M 'surpasses other open-source long-context LLMs by a large margin' on RULER rests on the average over 13 tasks, but the per-task table shows that Common Words Extraction (cwe) collapses to near zero at 128k and above (1.9 at 128k, 2.7 at 1M), while needle-retrieval tasks, which closely match the synthetic training distribution, remain near 100. Section 5.4 shows the QFS agent workflow used for synthesis is itself weak on the aggregation-style vt task (25.4 at 128k), and SelfLong-8B-1M is also weak on cwe at long lengths. This pattern is consistent with the model learning the QFS workflow's task profile rather than a general ability to reason over long contexts. The paper should present per-task comparisons against baselines at each length, report results on held-out long-context tasks requiring global aggregation (e.g., ∞Bench or LongBench), and either explain why cwe fails or temper the 'large margin' claim accordingly.
  2. [§A.3] The long-context evaluation at 256k and above uses 100 samples per RULER task, and the needle-in-haystack results come from a single trial per (depth, length) cell. The paper should state the effective sample sizes in the main text and report confidence intervals or at least a variance estimate for the 1M RULER scores. As it stands, a 5-point gap at 1M (69.6 vs 64.3 vs 60.1) could be within noise, and the needle figures give no sense of run-to-run variability.
  3. [§5, Table 2, Figure 3; Appendix A] The long-output extension is evaluated only by matching the target length, not by output quality. The paper itself notes in Appendix A that a preliminary GPT-4o-based quality check found that short outputs frequently receive high scores, undermining the proposed metric. To support the claim of extending the maximum output length, the paper needs a content-aware evaluation (e.g., faithfulness, coherence, or instruction adherence) or should explicitly reframe the result as 'length-controllability' rather than usable long-form generation.
minor comments (5)
  1. [§1, first paragraph] 'Significant advancements has been achieved' should be 'Significant advancements have been achieved'; also, the paper inconsistently uses 'LLaMA' and 'Llama' for the same model family.
  2. [Table 3] The 'w/ adjusted RoPE θ only' row is a valuable baseline, but the paper should note explicitly that this configuration involves no training and that the RoPE adjustment alone yields non-trivial RULER scores at 1M (48.1), which helps contextualize the gains from fine-tuning.
  3. [§5.1, Figure 4] The two panels of Figure 4 use different y-axis ranges, which exaggerates the apparent decline at extended lengths; please use a common scale for direct visual comparison.
  4. [Table 4] The IFEval drop from 77.4 to 65.9 is substantial, and the hypothesis that a better post-training mixture such as Tulu-3 would mitigate it is speculative; this should be stated as a limitation rather than a projection.
  5. [§5.2, Figure 5] The 4M-extension result is supported only by needle-in-haystack tests; the caption and text should state that no RULER or other long-context benchmark is provided for the 2M/4M variants, so this claim is preliminary.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the derivation is self-contained and evaluated on independent external benchmarks.

full rationale

The paper's central claim is that synthetic long-context instruction data, generated by a short-context agent workflow, can be used to fine-tune Llama-3 models to handle contexts up to 1M tokens. The derivation chain is not circular: the training targets are produced by query-focused summarization agents from short chunks, and those intermediate summaries are explicitly discarded during training (Section 3.1); the final training input is the concatenation of retrieved documents and the instruction, with the LLM-generated response as the target. The evaluation uses externally defined benchmarks (RULER, needle-in-a-haystack, Open LLM Leaderboard 2) that are not constructed from the training data and are not fitted by any training hyperparameter. The paper's ablations compare configurations such as 'w/ adjusted RoPE θ only' and 'w/ short data only' against the full method, which further demonstrates that the reported gains are not forced by construction. The use of the authors' own E5-mistral retriever and PoSE-style training is a normal engineering choice, not a load-bearing self-citation: those components are not claimed as the source of the long-context capability and do not define the evaluation metric. The paper also candidly reports limitations, including degraded variable-tracking performance at 1M and imperfect long-output instruction following, which is evidence that the reported results are not manufactured to match a predetermined conclusion. The transfer from short-context summary targets to end-to-end long-context reasoning is an empirical assumption, but that is a correctness risk, not circular reasoning.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on hand-chosen training hyperparameters and unproven domain assumptions about synthetic data transfer and benchmark validity. No new entities are proposed. Standard training details such as learning rate and batch size are included in the paper but omitted from this ledger because they are routine choices rather than method-specific inventions.

free parameters (6)
  • RoPE base frequency schedule = theta: 2M -> 8M -> 32M across stages
    Hand-chosen schedule to support context doubling, Section 3.2 and Table 7.
  • Data mixture sample weights = long-input 0.3, long-output 1.0, Infinity-Instruct 0.3, Tulu-v2 1.0, Prolong 0.1
    Chosen by hand in Section A.1, Table 6; affects the balance of long and short tasks in training.
  • Max training steps per stage = 50 steps per stage, 150 total
    Section A.2, Table 7; the authors note that 300 steps gave no performance improvement.
  • QFS chunk size and summary cap = 4k tokens per chunk; summaries capped at 300 words
    Section 3.1; chosen so each synthesis step fits a short context window.
  • Retrieval sampling range = top-5 retrieved; 1 to 100 documents sampled per instruction
    Section A.1; introduces variable input lengths, but the count is a hand choice.
  • Dedup and downsampling thresholds = cosine similarity 0.85; keep probability 0.05 for documents shorter than 2k tokens
    Appendix A.1; affects diversity of synthetic instructions and corpus composition.
assumptions (4)
  • domain assumption The RULER benchmark is a valid proxy for long-context capability at 1M tokens.
    Section 4.1 selects RULER because it supports automatic evaluation at any context length, but RULER tasks are synthetic and may not reflect real long-document reasoning.
  • domain assumption Query-focused summaries preserve enough information from each document chunk that answers generated from summaries are faithful to the full documents.
    Section 3.1 uses QFS agents to filter chunks and then generates the training answer only from summaries; if summaries discard cross-chunk or cross-document dependencies, the synthetic target may not require true long-range integration.
  • domain assumption The short-context model's instruction-following and reasoning abilities transfer to long-context inputs through fine-tuning on concatenated instruction-plus-documents.
    This is the central premise of the bootstrapping idea, stated in Section 1 and assessed through RULER and NIAH evaluations; it is not independently proven.
  • standard math Quadrupling the RoPE base frequency whenever the context length is doubled keeps position encodings within a useful range.
    Section 3.2 invokes this as a reasonable initialization for progressive extension; it is a standard heuristic from RoPE-based long-context work rather than a result proved in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bootstrap Your Own Context Length." pith.science (2026). https://pith.science/paper/TB7PEXAM

@misc{pith2026241218860,
  author       = {Pith},
  title        = {Pith review of: Bootstrap Your Own Context Length},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TB7PEXAM}},
  note         = {Machine review of arXiv:2412.18860}
}
read the original abstract

We introduce a bootstrapping approach to train long-context language models by exploiting their short-context capabilities only. Our method utilizes a simple agent workflow to synthesize diverse long-context instruction tuning data, thereby eliminating the necessity for manual data collection and annotation. The proposed data synthesis workflow requires only a short-context language model, a text retriever, and a document collection, all of which are readily accessible within the open-source ecosystem. Subsequently, language models are fine-tuned using the synthesized data to extend their context lengths. In this manner, we effectively transfer the short-context capabilities of language models to long-context scenarios through a bootstrapping process. We conduct experiments with the open-source Llama-3 family of models and demonstrate that our method can successfully extend the context length to up to 1M tokens, achieving superior performance across various benchmarks.

Figures

Figures reproduced from arXiv: 2412.18860 by the authors.

Figure 1
Figure 1. The overall workflow for synthesizing long-context instruction tuning data comprises four [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Needle-in-haystack test results. The x-axis represents the context lengths, while the y-axis [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Scatter plot illustrating the relationship between the required generation length and the [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The evolving performance across various test lengths as [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Needle-in-haystack test results when extending the context length up to 4M. For the [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Internalized Reasoning for Long-Context Visual Document Understanding

    cs.CV 2026-03 unverdicted novelty 7.0 of 10

    Synthetic page-ranked reasoning traces plus low-strength model merging give a 32B VLM 58.3 on MMLongBenchDoc, beating a 235B teacher while cutting output tokens ~12× versus explicit reasoning.

  2. UNIBROWSE: A Data-to-Agent Framework for Multimodal BrowseComp

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A unified KG-plus-live-web data pipeline covering all three multimodal BrowseComp information-flow patterns, plus an exploration-degree filter, yields a 35B agent at 54.4 avg accuracy.

  3. LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions

    cs.CL 2025-05 conditional novelty 5.0 of 10

    By prompting an aligned LLM with a document and the special token that precedes a user query, LongMagpie synthesizes long-context instruction data that outperforms prior datasets when used to fine-tune Llama-3-8B.

Reference graph

Works this paper leans on

47 extracted references · 12 canonical work pages · cited by 3 Pith papers

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. ArXiv preprint, abs/2404.14219, 2024. URL https://arxiv.org/abs/2404.14219

    arXiv 2024

  2. [2]

    Make your llm fully utilize the context

    Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, and Jian-Guang Lou. Make your llm fully utilize the context. ArXiv preprint, abs/2404.16811, 2024. URL https://arxiv.org/abs/2404.16811

    arXiv 2024

  3. [3]

    Infinity-instruct

    BAAI. Infinity-instruct. URL https://huggingface.co/datasets/BAAI/Infinity-Instruct

    work page

  4. [4]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. ArXiv preprint, abs/2308.14508, 2023. URL https://arxiv.org/abs/2308.14508

    arXiv 2023

  5. [5]

    Extending context window of large language models via positional interpolation

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. ArXiv preprint, abs/2306.15595, 2023. URL https://arxiv.org/abs/2306.15595

    arXiv 2023

  6. [6]

    Redpajama: an open dataset for training large language models, 2023

    Together Computer. Redpajama: an open dataset for training large language models, 2023. URL https://github.com/togethercomputer/RedPajama-Data

    2023

  7. [7]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=mZn2Xyh9Ec

    work page 2024

  8. [8]

    Longnet: Scaling transformers to 1,000,000,000 tokens

    Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, and Furu Wei. Longnet: Scaling transformers to 1,000,000,000 tokens. ArXiv preprint, abs/2307.02486, 2023. URL https://arxiv.org/abs/2307.02486

    arXiv 2023

Show all 47 references
  1. [9]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. ArXiv preprint, abs/2407.21783, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  2. [10]

    Data engineering for scaling language models to 128k context

    Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net,...

    2024

  3. [11]

    Quest: Query-centric data synthesis approach for long-context scaling of large language model

    Chaochen Gao, Xing Wu, Qi Fu, and Songlin Hu. Quest: Query-centric data synthesis approach for long-context scaling of large language model. ArXiv preprint, abs/2405.19846, 2024 a . URL https://arxiv.org/abs/2405.19846

    2024 arXiv

  4. [12]

    A framework for few-shot language model evaluation, 2024 b

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

  5. [13]

    How to train long-context language models (effectively)

    Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. How to train long-context language models (effectively). ArXiv preprint, abs/2410.02660, 2024 c . URL https://arxiv.org/abs/2410.02660

    2024

  6. [14]

    Scaling synthetic data creation with 1,000,000,000 personas

    Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas. ArXiv preprint, abs/2406.20094, 2024. URL https://arxiv.org/abs/2406.20094

    2024 arXiv

  7. [15]

    Ruler: What's the real context size of your long-context language models? ArXiv preprint, abs/2404.06654, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? ArXiv preprint, abs/2404.06654, 2024. URL https://arxiv.org/abs/2404.06654

    2024 arXiv

  8. [16]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. ArXiv preprint, abs/2410.21276, 2024. URL https://arxiv.org/abs/2410.21276

    2024 arXiv

  9. [17]

    Camels in a changing climate: Enhancing lm adaptation with tulu 2

    Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in a changing climate: Enhancing lm adaptation with tulu 2. ArXiv preprint, abs/2311.10702, 2023. URL https://arxiv....

    2023 arXiv

  10. [18]

    Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, et al. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. ArXiv preprint, abs/2407.02490, 2024 a ....

    2024 arXiv

  11. [19]

    Longrag: Enhancing retrieval-augmented generation with long-context llms

    Ziyan Jiang, Xueguang Ma, and Wenhu Chen. Longrag: Enhancing retrieval-augmented generation with long-context llms. ArXiv preprint, abs/2406.15319, 2024 b . URL https://arxiv.org/abs/2406.15319

    2024 arXiv

  12. [20]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7...

    2024

  13. [21]

    Dense passage retrieval for open-domain question answering

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on E...

    2020 doi

  14. [22]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, p...

    2023

  15. [23]

    T " ulu 3: Pushing frontiers in open language model post-training

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T " ulu 3: Pushing frontiers in open language model post-training. ArXiv preprint, abs/2411.15124, 2024. URL https:...

    2024 arXiv

  16. [24]

    Can long-context language models subsume retrieval, rag, sql, and more? ArXiv preprint, abs/2406.13121, 2024

    Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, S \'e bastien MR Arnold, Vincent Perot, Siddharth Dalmia, et al. Can long-context language models subsume retrieval, rag, sql, and more? ArXiv preprint, abs/2406.13121, 2024. UR...

    2024 arXiv

  17. [25]

    u ttler, Mike Lewis, Wen - tau Yih, Tim Rockt \

    Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \" u ttler, Mike Lewis, Wen - tau Yih, Tim Rockt \" a schel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In ...

    2020

  18. [26]

    Self-alignment with instruction backtranslation

    Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenRevie...

    2024

  19. [27]

    Retrievalattention: Accelerating long-context llm inference via vector retrieval

    Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chengruidong Zhang, Bailu Ding, Kai Zhang, et al. Retrievalattention: Accelerating long-context llm inference via vector retrieval. ArXiv preprint, abs/2409.10516, 2024 a . URL https://arxiv.org...

    2024 arXiv

  20. [28]

    World model on million-length video and language with ringattention

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. ArXiv preprint, abs/2402.08268, 2024 b . URL https://arxiv.org/abs/2402.08268

    2024 arXiv

  21. [29]

    Ringattention with blockwise transformers for near-infinite context

    Hao Liu, Matei Zaharia, and Pieter Abbeel. Ringattention with blockwise transformers for near-infinite context. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024 c . URL https://openreview.net...

    2024

  22. [30]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12: 0 157--173, 2024 d . doi:10.1162/tacl_a_00...

    2024 doi

  23. [31]

    Generative agents: Interactive simulacra of human behavior

    Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1--22, 2023

    2023

  24. [32]

    Llama 3 gradient: A series of long context models, 2024

    Leonid Pekelis, Michael Feil, Forrest Moret, Mark Huang, and Tiffany Peng. Llama 3 gradient: A series of long context models, 2024. URL https://gradient.ai/blog/scaling-rotational-embeddings-for-long-context-language-models

    2024

  25. [33]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydl \' c ek, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. ArXiv preprint, abs/2406.17557, 2024. URL https://arxiv.org/abs/2406.17557

    2024 arXiv

  26. [34]

    Yarn: Efficient context window extension of large language models

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https:...

    2024

  27. [35]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023

    2023

  28. [36]

    Z ero SCROLLS : A zero-shot benchmark for long text understanding

    Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. Z ero SCROLLS : A zero-shot benchmark for long text understanding. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7977--7989,...

    2023 doi

  29. [37]

    Musr: Testing the limits of chain-of-thought with multistep soft reasoning

    Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. Musr: Testing the limits of chain-of-thought with multistep soft reasoning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, ...

    2024

  30. [38]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  31. [39]

    You only cache once: Decoder-decoder architectures for language models

    Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. You only cache once: Decoder-decoder architectures for language models. ArXiv preprint, abs/2405.05254, 2024. URL https://arxiv.org/abs/2405.05254

    2024 arXiv

  32. [40]

    Challenging BIG -bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG -bench tasks and whether chain-of-thought can solve them. In Anna Rogers, Jordan Boyd-Graber, and N...

    2023 doi

  33. [41]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. ArXiv preprint, abs/2302.13971, 2023. URL https:...

    2023 arXiv

  34. [42]

    Improving text embeddings with large language models

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. ArXiv preprint, abs/2401.00368, 2024 a . URL https://arxiv.org/abs/2401.00368

    2024 arXiv

  35. [43]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024 b

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language unders...

    2024 arXiv

  36. [44]

    Effective long-context scaling of foundation models

    Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, S...

    2024

  37. [45]

    \( \) bench: Extending long context evaluation beyond 100k tokens

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. \( \) bench: Extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lingui...

    2024

  38. [46]

    Instruction-following evaluation for large language models

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. ArXiv preprint, abs/2311.07911, 2023. URL https://arxiv.org/abs/2311.07911

    2023 arXiv

  39. [47]

    Pose: Efficient context window extension of llms via positional skip-wise training

    Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. Pose: Efficient context window extension of llms via positional skip-wise training. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 ...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.