Pith. sign in

REVIEW 4 major objections 5 minor 52 references

Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understanding

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Distilled reasoning patterns improve long-context understanding and mitigate the 'lost in the middle' problem in multi-document QA.

desk verdict Useful empirical observation about R1-distilled models on long-context MDQA, but the causal claim is not identified because baselines are not matched on template, token budget, and sampling. read the letter →

arxiv 2507.14849 v1 pith:L2OFAUG3 submitted 2025-07-20 cs.CL

Yifei Wang This is my paper
classification cs.CL
keywords reasoningdistillationlong-contextunderstandinglostinthemiddlemulti-documentquestionansweringchain-of-thoughtpositionbiasin-contextretrievalretrieval-augmentedgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the long chain-of-thought reasoning that small models learn by distillation from a strong reasoning model also improves how they read long contexts. The authors test four distilled open-weight models on multi-document question answering with 10 to 80 documents, varying where the correct answer sits in the context. They find that the distilled models outperform their base counterparts most as context grows, with a 13.29-point exact-match gain for a 14B model at 80 documents, and that performance stays stable across answer positions instead of dipping in the middle. The paper argues that extended reasoning—reflecting, verifying, and cross-checking documents—transfers into better in-context retrieval and mitigates the 'lost in the middle' problem. If true, reasoning ability and long-context understanding are not separate skills but can be bridged by distillation.

What carries the argument

The central object is the extended chain-of-thought reasoning pattern distilled from DeepSeek-R1: a multi-step internal monologue, usually wrapped in a <think> token, that examines documents one by one, cross-references details such as dates and figures, checks contradictions, and synthesizes a final answer. The experimental instrument is the multi-document QA probe (following the 'Lost in the Middle' methodology), which randomizes the gold document across ten equally spaced positions within a context of 10 to 80 documents while inserting semantically similar distractors. The key empirical quantities are exact-match accuracy as a function of document count and target position, and the position-bias curve drawn from those measurements.

What would settle it

Run the same base models on the same multi-document QA with the DeepSeek-R1 chat template, 4096 max new tokens, and temperature 0.6, but without distillation—for instance, by prompting them to think step by step—and check whether the accuracy advantage and middle-position stability at 50 to 80 documents persist; if they do, the distillation-specific explanation fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that distilled reasoning patterns do more than improve math and coding; they also enhance long-context understanding. In single-hop multi-document QA built from NaturalQuestions, DeepSeek-R1 distilled models of sizes 8B, 14B, 32B, and 70B achieve higher exact-match accuracy than their base models under open-book direct QA as the document count grows, with the largest gap at 80 documents, and they show near-constant accuracy regardless of where the gold document appears. The paper attributes this to a learned habit of doc-by-doc analysis with reflection, verification, and cross-document synthesis, which overrides the attention-sink and recency-bias failures of the base models. It also reports that Zero-CoT prompting does not reproduce the benefit, supporting the distinctiveness of the distilled reasoning patterns.

Load-bearing premise

The measured gains come from the reasoning patterns instilled by distillation, not from the different prompt template, larger token budget, or higher sampling temperature that the distilled models receive.

Editorial extensions

If this is right

  • Smaller models distilled from a strong reasoner can handle retrieval-augmented generation over long contexts more reliably than their base versions, narrowing the gap to much larger models.
  • Long-context research gains a new lever: inference-time reasoning deserves attention alongside positional embeddings and attention masks as a cure for 'lost in the middle.'
  • Zero-CoT prompting is not a substitute for the distilled reasoning pattern; in several tested settings it degrades accuracy below direct QA.
  • Because the benefit grows with context length, distillation should matter most precisely in the regime where context windows are largest.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: If the reasoning-pattern account is correct, the same benefit should appear on multi-hop QA and on tasks requiring cross-document synthesis, since the paper names cross-document verification as the mechanism; its own limitation section notes only single-hop tasks were tested.
  • Editorial: The mechanism can be probed directly by masking or truncating the <think> segment of a distilled model at inference time; if accuracy on long-context QA drops, the reasoning steps themselves are doing the work.
  • Editorial: The transfer might be a property of long chain-of-thought in general rather than of distillation specifically; a model that generates extended reasoning traces at inference time without R1 distillation should show a similar, if weaker, effect.
  • Editorial: The results suggest a latency-versus-accuracy design choice for RAG systems: allocate 'thinking' tokens to retrieval-heavy prompts when the context is long, accepting slower responses in exchange for higher hit rates.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper asks whether DeepSeek-R1-style reasoning distillation transfers to long-context understanding. The authors compare DeepSeek-R1-distilled Qwen2.5-14B/32B, Llama-3.1-8B, and Llama-3.3-70B with their base checkpoints on single-hop multi-document QA built from NaturalQuestions, with 10–80 documents and randomized gold-document positions. They report that distilled models achieve higher open-book Exact Match at long context lengths (e.g., +13.29 EM for Qwen2.5-14B at 80 docs), more stable position curves, and longer reasoning traces that verify document details. The paper attributes these gains to 'distilled reasoning patterns' and claims that this mitigates the 'lost in the middle' problem.

Significance. If the causal claim were cleanly identified, the result would be a useful empirical bridge between reasoning distillation and long-context retrieval, with implications for RAG systems and for 'lost in the middle' mitigation. The evaluation uses an external benchmark, varies context length and answer position systematically, and covers four model families; there are no fitted parameters and no hand-built metric that would make the main comparison circular. The main weakness is causal identification: the comparison design changes several inference variables at once, so the current evidence supports only a correlational claim. With controlled ablations and uncertainty quantification, the paper could make a solid contribution; the appended limitations section is honest about the narrow task scope but does not address the identification problem.

major comments (4)
  1. [§2.3, Table 2] The main comparison is confounded. The distilled models are run with the DeepSeek-R1 chat template, max_new_tokens=4096, and temperature=0.6, but no corresponding settings are reported for the direct-QA and Zero-CoT baselines. Consequently, the 'distilled vs. base' contrast in Table 2 varies the checkpoint together with the prompt template, decoding budget, and sampling configuration. Since the abstract claims a mechanistic demonstration that reasoning patterns improve long-context understanding, this is load-bearing: the observed gains (e.g., +13.29 EM for Qwen2.5-14B at 80 documents) could in principle be produced by the template and decoding settings alone. Please add ablations that hold template/decoding fixed and vary only whether a reasoning pattern is elicited (e.g., the base model with the R1 template and a long decoding budget; the distilled model with a plain QA template), or at minimum report and match the baseline decoding settings.
  2. [Table 2] The paper reports single EM numbers without error bars, confidence intervals, or significance tests. Several deltas are small (e.g., Llama-3.3-70B +0.04 at 10 docs; Qwen2.5-32B -0.02 at 10 docs), and the Zero-CoT baseline fluctuates in sign across models. Without repeated runs or bootstrap intervals, the central claim that distilled models 'significantly improve' long-context understanding is not quantitatively supported. Please report uncertainty (e.g., multiple document orderings or random subsets, multiple sampling seeds) and, if possible, a paired significance test.
  3. [§2.2 (closed-book baseline)] The closed-book setting removes the context entirely, so it can only speak to parametric knowledge. It does not control for the alternative explanation that distillation improves general instruction following or output formatting: a distilled model might answer better in the open-book setting simply because it follows the 'answer from the documents' instruction more reliably. Add an open-book control with non-informative but well-formed documents, or an SFT-only model trained on non-reasoning data, to disentangle reasoning-pattern transfer from generic instruction following.
  4. [§2.4 / Figure 2] The positional-invariance claim is based on a visual comparison at 50 documents. To support the claim that distillation mitigates the 'lost in the middle' issue, report a quantitative position-bias measure (e.g., the gap between middle and edge positions, or the variance or slope of EM across positions) and show the position curves for more than one document count. The current figure alone does not allow the reader to assess whether the apparent flatness is within sampling noise.
minor comments (5)
  1. [Table 2] The table formatting is corrupted in places (e.g., the Llama-3.1-8B Zero-CoT row shows '-20.35' immediately before the 20-doc value, and the column header 'Closed-' is incomplete). Please regenerate the table with clear delta notation.
  2. [§2.2] The Zero-CoT prompt is never defined; specify the exact instruction (e.g., 'Let's think step by step') and whether it is appended before or after the documents.
  3. [Throughout] The capitalization 'Deepseek' and 'DeepSeek' is used inconsistently; please unify.
  4. [Figure 2] The figure has no axis labels and no indication of whether the curves are averaged over multiple runs; please add them.
  5. [§2.2] 'close-book' should be 'closed-book', and the phrase 'excluding non-reasoning data from the distillation process' is unclear because distillation data are reasoning traces; please rephrase.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central claim rests on external benchmark comparisons, and no prediction reduces by construction to fitted inputs or self-citations.

full rationale

This paper is an empirical evaluation, not a derivation. It compares publicly released DeepSeek-R1-distilled checkpoints against their base models on the NaturalQuestions multi-document QA benchmark, using Exact Match as an externally defined metric. No parameter is fitted to the reported outcomes, no equation defines a predicted quantity in terms of the data used to produce it, and no result is imported from a self-citation chain. The abstract's causal claim that distilled reasoning patterns improve long-context understanding is an interpretation of measured benchmark differences, not a consequence entailed by the experimental construction. The strongest concern—that the distilled and baseline arms differ not only in distillation but also in prompt template, decoding budget, and sampling temperature—is a threat to internal validity (confounding), not circularity: the observed gains are not logically forced by the setup. The closed-book comparison further provides an independent check against parametric-knowledge confounds. Since the evaluation is benchmark-based and externally falsifiable, the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters were fitted. The central claims rest on domain assumptions about the benchmark and on the causal interpretation of observed model-family differences; no new entities are introduced.

assumptions (4)
  • domain assumption Single-hop multi-document QA on NaturalQuestions is a valid proxy for long-context understanding and in-context retrieval.
    Section 2.1 defines the probing task; the paper generalizes from this task to long-context comprehension without evaluating multi-hop or other long-context benchmarks, as the Limitation section acknowledges.
  • domain assumption The top-k document retrieval selects relevant distractors as in a RAG pipeline; retrieval quality and distractor similarity are sufficient for a fair comparison.
    Section 2.1 states that distractors are selected by semantic similarity sim(d, x_query) > tau, but the retriever and the value of tau are not specified in the paper.
  • ad hoc to paper The DeepSeek-R1 chat template with <think> tokens elicits the same reasoning patterns used during distillation, and the base-model direct QA and Zero-CoT prompts are comparable controls.
    Section 2.2 and Appendix C describe the templates; no matched templates, decoding budgets, or sampling settings are shown to make the comparison fair. This assumption is central to the causal claim.
  • ad hoc to paper Observed performance differences between distilled and base models are caused by reasoning patterns rather than by other training-data or decoding differences.
    No ablation or control model is provided. Section 2.4 attributes the improvement to doc-by-doc analysis and verification, but only case examples are shown.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understanding." pith.science (2026). https://pith.science/paper/L2OFAUG3

@misc{pith2026250714849,
  author       = {Pith},
  title        = {Pith review of: Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L2OFAUG3}},
  note         = {Machine review of arXiv:2507.14849}
}
read the original abstract

Reasoning distillation has emerged as an effective approach to enhance the reasoning capabilities of smaller language models. However, the impact of large-scale reasoning distillation on other critical abilities, particularly in-context retrieval and reasoning, remains unexplored. This gap in understanding is particularly significant given the increasing importance of Retrieval-Augmented Generation (RAG) systems, where efficient acquisition and utilization of contextual information are paramount for generating reliable responses. Motivated by the need to understand how the extended long-CoT process influences long-context comprehension, we conduct a comprehensive investigation using a series of open-source models distilled from Deepseek-R1, renowned for its exceptional reasoning capabilities. Our study focuses on evaluating these models' performance in extracting and integrating relevant information from extended contexts through multi-document question and answering tasks. Through rigorous experimentation, we demonstrate that distilled reasoning patterns significantly improve long-context understanding. Our analysis reveals that distillation fosters greater long-context awareness by promoting more detailed and explicit reasoning processes during context analysis and information parsing. This advancement effectively mitigates the persistent "lost in the middle" issue that has hindered long-context models.

Figures

Figures reproduced from arXiv: 2507.14849 by the authors.

Figure 1
Figure 1. The left panel illustrates the distillation pro [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Positional effects on EM scores in 50-doc QA. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Reasoning pattern analysis of distilled models [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: An instance of implementing long CoT on lengthy inputs including 50 docs from DeepSeek-R1-Distill [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: An instance of implementing long CoT on lengthy inputs including 50 docs from DeepSeek-R1-Distill [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: An instance of implementing long CoT on lengthy inputs from DeepSeek-R1-Distill-Llama-3.1-8B. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: An instance of outputting long CoT on lengthy inputs from DeepSeek-R1-Distill-Llama-3.3-70B-Instruct. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Prompt template design. 11 [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 21 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024. Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint arXiv:2402.00157

    arXiv 2024

  4. [4]

    Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. 2024. https://openreview.net/forum?id=YGTVEmBXtV Make your LLM fully utilize the context . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    work page 2024

  5. [5]

    Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.74 L ong A lign: A recipe for long context alignment of large language models . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1376--1395, Miami, Florida, USA. Association fo...

    doi:10.18653/v1/2024.findings-emnlp.74 2024

  6. [6]

    George Arthur Baker, Ankush Raut, Sagi Shaier, Lawrence E Hunter, and Katharina von der Wense. 2024. http://arxiv.org/abs/2412.10079 Lost in the middle, and in-between: Enhancing language models' ability to reason over long contexts in multi-hop qa

    arXiv 2024

  7. [7]

    Yuhan Chen, Ang Lv, Ting-En Lin, Changyu Chen, Yuchuan Wu, Fei Huang, Yongbin Li, and Rui Yan. 2024. https://doi.org/10.18653/v1/2024.acl-long.601 Fortify the shortest stave in attention: Enhancing context awareness of large language models for effective tool use . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ...

    work page doi:10.18653/v1/2024.acl-long.601 2024

  8. [8]

    Ta-Chung Chi, Ting-Han Fan, Li-Wei Chen, Alexander Rudnicky, and Peter Ramadge. 2023. https://doi.org/10.18653/v1/2023.acl-short.102 Latent positional information is in the self-attention variance of transformer language models without positional embeddings . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volum...

    work page doi:10.18653/v1/2023.acl-short.102 2023

Show all 52 references
  1. [9]

    Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. 2024. https://doi.org/10.1162/tacl_a_00644 Evaluating the ripple effects of knowledge editing in language models . Transactions of the Association for Computational Linguistics, 12:283--298

    2024 doi

  2. [10]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025 arXiv

  3. [11]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. http://arxiv.org/abs/2312.10997 Retrieval-augmented generation for large language models: A survey

    2024 arXiv

  4. [12]

    Albert Gu and Tri Dao. 2024. https://openreview.net/forum?id=AL1fq05o7H Mamba: Linear-time sequence modeling with selective state spaces

    2024

  5. [13]

    Cruxeval: A benchmark for code reasoning, understanding and execution, 2024

    Alex Gu, Baptiste Roziere, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. Cruxeval: A benchmark for code reasoning, understanding and execution, 2024. URL https://arxiv. org/abs/2401.03065

    2024 arXiv

  6. [14]

    Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. 2025. https://openreview.net/forum?id=78Nn4QJTEN When attention sink emerges in language models: An empirical view . In The Thirteenth International Conference on Learning Representations

    2025

  7. [15]

    Two stones hit one bird: Bilevel positional encoding for better length extrapolation

    Zhenyu He, Guhao Feng, Shengjie Luo, Kai Yang, Liwei Wang, Jingjing Xu, Zhi Zhang, Hongxia Yang, and Di He. Two stones hit one bird: Bilevel positional encoding for better length extrapolation. In Forty-first International Conference on Machine Learning

  8. [16]

    Zhenyu He, Guhao Feng, Shengjie Luo, Kai Yang, Liwei Wang, Jingjing Xu, Zhi Zhang, Hongxia Yang, and Di He. 2024. Two stones hit one bird: bilevel positional encoding for better length extrapolation. In Proceedings of the 41st International Conference on Machine Learning, ICML...

    2024

  9. [17]

    Maor Ivgi, Uri Shaham, and Jonathan Berant. 2023. https://doi.org/10.1162/tacl_a_00547 Efficient long-text understanding with short-text models . Transactions of the Association for Computational Linguistics, 11:284--299

    2023 doi

  10. [18]

    Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, and Xin Jin. 2024. http://arxiv.org/abs/2404.12457 Ragcache: Efficient knowledge caching for retrieval-augmented generation

    2024 arXiv

  11. [19]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019 doi

  12. [20]

    Quinn Leng, Jacob Portes, Sam Havens, Matei Zaharia, and Michael Carbin. 2024. https://openreview.net/forum?id=Le9anH3kv1 Long context RAG performance of large language models . In Adaptive Foundation Models: Evolving AI for Personalized and Efficient Learning

    2024

  13. [21]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Proc...

    2020

  14. [22]

    Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. 2024. Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers. arXiv preprint arXiv:2402.19255

    2024 arXiv

  15. [23]

    Hongzhan Lin, Ang Lv, Yuhan Chen, Chen Zhu, Yang Song, Hengshu Zhu, and Rui Yan. 2024. https://openreview.net/forum?id=RcPHbofiCN Mixture of in-context experts enhance LLM s' long context awareness . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  16. [24]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. https://doi.org/10.1162/tacl_a_00638 Lost in the middle: How language models use long contexts . Transactions of the Association for Computational Linguistics, 12...

    2024 doi

  17. [25]

    LINHAO LUO, Yuan-Fang Li, Reza Haf, and Shirui Pan. 2024. https://openreview.net/forum?id=ZGNWW7xZ6Q Reasoning on graphs: Faithful and interpretable large language model reasoning . In The Twelfth International Conference on Learning Representations

    2024

  18. [26]

    Trung Quoc Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. 2024. http://arxiv.org/abs/2401.08967 Reft: Reasoning with reinforced fine-tuning

    2024 arXiv

  19. [27]

    Dongyang Ma, Yan Wang, and Tian Lan. 2025. https://openreview.net/forum?id=7zNYY1E2fq Block-attention for efficient RAG . In The Thirteenth International Conference on Learning Representations

    2025

  20. [28]

    Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. 2022. https://openreview.net/forum?id=-h6WAS6eE4 Locating and editing factual associations in GPT . In Advances in Neural Information Processing Systems

    2022

  21. [29]

    Thomas Merth, Qichen Fu, Mohammad Rastegari, and Mahyar Najibi. 2024. http://arxiv.org/abs/2404.06910 Superposition prompting: Improving and accelerating retrieval-augmented generation

    2024 arXiv

  22. [30]

    Yasumasa Onoe, Michael Zhang, Shankar Padmanabhan, Greg Durrett, and Eunsol Choi. 2023. https://doi.org/10.18653/v1/2023.acl-long.300 Can LM s learn new entities from descriptions? challenges in propagating injected knowledge . In Proceedings of the 61st Annual Meeting of the ...

    2023 doi

  23. [31]

    OpenAI. 2024. Learning to reason with large language models. https://openai.com/index/learning-to-reason-with-llms/

    2024

  24. [32]

    Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. 2023. Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning. arXiv preprint arXiv:2305.12295

    2023 arXiv

  25. [33]

    Bo Pang, Hanze Dong, Jiacheng Xu, Silvio Savarese, Yingbo Zhou, and Caiming Xiong. 2025. http://arxiv.org/abs/2502.03860 Bolt: Bootstrap long chain-of-thought in language models without distillation

    2025 arXiv

  26. [34]

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bart omiej Koptyra, Hay...

    2023

  27. [35]

    Alexander Peysakhovich and Adam Lerer. 2023. http://arxiv.org/abs/2310.01427 Attention sorting combats recency bias in long context language models

    2023 arXiv

  28. [36]

    Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, and Pengfei Liu. 2024. http://arxiv.org/abs/2410.18982 O1 replication journey: A strategic progress report -- part 1

    2024 arXiv

  29. [37]

    Cohen, and Benjamin Han

    Yifu Qiu, Varun Embar, Yizhe Zhang, Navdeep Jaitly, Shay B. Cohen, and Benjamin Han. 2025. http://arxiv.org/abs/2501.08248 Eliciting in-context retrieval and reasoning for long-context large language models

    2025 arXiv

  30. [38]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong...

    2024 arXiv

  31. [39]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. http://arxiv.org/abs/2402.03300 Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    2024 arXiv

  32. [40]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. http://arxiv.org/abs/2408.03314 Scaling llm test-time compute optimally can be more effective than scaling model parameters

    2024 arXiv

  33. [41]

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. https://doi.org/10.1016/j.neucom.2023.127063 Roformer: Enhanced transformer with rotary position embedding . Neurocomput., 568(C)

    2024

  34. [42]

    Yixuan Tang and Yi Yang. 2024. https://openreview.net/forum?id=t4eB3zYWBK Multihop- RAG : Benchmarking retrieval-augmented generation for multi-hop queries . In First Conference on Language Modeling

    2024

  35. [43]

    Jiaan Wang, Fandong Meng, Yunlong Liang, and Jie Zhou. 2025 a . http://arxiv.org/abs/2412.17498 Drt: Deep reasoning translation via long chain-of-thought

    2025 arXiv

  36. [44]

    Kakade, Hao Peng, and Heng Ji

    Ziqi Wang, Hanlin Zhang, Xiner Li, Kuan-Hao Huang, Chi Han, Shuiwang Ji, Sham M. Kakade, Hao Peng, and Heng Ji. 2025 b . https://openreview.net/forum?id=fvkElsJOsN Eliminating position bias of language models: A mechanistic approach . In The Thirteenth International Conference...

    2025

  37. [45]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. http://arxiv.org/abs/2201.11903 Chain-of-thought prompting elicits reasoning in large language models

    2023 arXiv

  38. [46]

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. S mooth Q uant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning

    2023

  39. [47]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. https://openreview.net/forum?id=NG7sS51zVF Efficient streaming language models with attention sinks . In The Twelfth International Conference on Learning Representations

    2024

  40. [48]

    Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. 2025. http://arxiv.org/abs/2502.03373 Demystifying long chain-of-thought reasoning in llms

    2025 arXiv

  41. [49]

    Yijiong Yu, Huiqiang Jiang, Xufang Luo, Qianhui Wu, Chin-Yew Lin, Dongsheng Li, Yuqing Yang, Yongfeng Huang, and Lili Qiu. 2025. https://openreview.net/forum?id=t717joHHSc Mitigate position bias in large language models via scaling a single dimension

    2025

  42. [50]

    Zhenrui Yue, Honglei Zhuang, Aijun Bai, Kai Hui, Rolf Jagerman, Hansi Zeng, Zhen Qin, Dong Wang, Xuanhui Wang, and Michael Bendersky. 2025. https://openreview.net/forum?id=FSjIrOm1vz Inference scaling for long-context retrieval augmented generation . In The Thirteenth Internat...

    2025

  43. [51]

    Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, and Zhangyang Wang. 2024. https://openreview.net/forum?id=fPmScVB1Td Found in the middle: How language models use long contexts better via plug-and-play positional encoding . In The Thi...

    2024

  44. [52]

    Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. 2024. https://openreview.net/forum?id=3Z1gxuAQrA Po SE : Efficient context window extension of LLM s via positional skip-wise training . In The Twelfth International Conference on Learning Representations

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.