Pith. sign in

REVIEW 3 major objections 5 minor 25 references

OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that replacing just 12% of conventional long-context training data with dependency-rich code contexts mined across repository boundaries substantially improves long-range retrieval, state tracking, repository-level code…

desk verdict Genuinely novel cross-repo dependency-mining pipeline with a strong ablation story; two soft spots—missing error bars and unverified LSP resolution accuracy—deserve attention before the mechanism is fully trusted. read the letter →

arxiv 2608.05141 v1 pith:R2TBN5BF submitted 2026-08-05 cs.AI cs.LGcs.SE

Indraneil Paul , Falko Helm , Goran Glavaš , Iryna Gurevych This is my paper
classification cs.AIcs.LGcs.SE
keywords long-contextlanguagemodelscross-repositorycodedatadependencyminingcontextengineeringserverprotocolfine-tuningagentictooluseunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the finite, loosely coupled long texts used to extend language-model context windows—books, papers, and in-repository code—are not the real bottleneck; dependency-dense code that crosses repository boundaries is, and it is nearly unlimited. The authors build a pipeline that recursively follows function and class call sites into installed packages using a language server inside a containerized environment, producing about 6.2B tokens of cross-repository code contexts up to 128K tokens long. Mixing roughly 12% of these contexts into a 50B-token long-context fine-tuning corpus, then instruction-tuning dense base models from 600M to 14B parameters, yields gains on long-range retrieval, state tracking, repository-level code QA, and agentic tool use, with no regression on short-context coding. The central empirical claim is that a modest fraction of dependency-rich cross-repository code data, not an architectural change, moves long-context capability.

What carries the argument

The machinery is the OctoLong acquisition pipeline: a containerized seed repository with dependencies pre-installed, a syntax-tree query layer that enumerates function calls, method calls, and object instantiations, and a language server backend whose go-to-definition lookups resolve each call site to a concrete definition; breadth-first traversal then serializes the resolved definitions into a single context of up to 128K tokens, keeping only contexts with depth at least 5 and hop-success rate above 60%. Diagnostics adapted from attention-based dependency-density scoring and context-sensitive perplexity measure how much of each context's later tokens depend on earlier definitions, and these diagnostics place OctoLong contexts above all traditional long-context sources. The second load-bearing piece is the training recipe: a roughly 50B-token long-context fine-tuning corpus containing about 12% OctoLong data, single-stage frequency-boosted rotary position scaling from 32K to 128K context, a 1:9 model merge with the pre-extension checkpoint to protect short-context performance, and about 10B tokens of instruction tuning that includes agentic and tool-use data.

What would settle it

Compile a random sample of OctoLong contexts and check each call-site-to-definition chain against runtime behavior in the container: if a large fraction of go-to-definition resolutions point to functions or classes that are not the objects actually invoked, then the claimed dependency density is spurious. A complementary experiment trains on the same contexts with dependency chains shuffled so that call sites are paired with unrelated implementations of similar shape; if long-context benchmark scores stay high, the gains are not attributable to genuine dependency learning.

Watch

Extended reading notes

Core claim

The paper's central claim is that cross-repository code contexts are a superior long-context training signal because they carry real, multi-hop dependencies, and that substituting just 12% of a conventional long-context corpus with them improves long-context ability across code and general domains. The OctoLong pipeline implements this by taking installable Python repositories, provisioning each seed repository in a container with its dependencies, indexing call sites with a syntax-tree query layer, resolving each call site to its implementation via the language server's go-to-definition endpoint, and recursively prepending the resolved code breadth-first up to depth 10 or 128K tokens. The resulting contexts score higher than books, papers, retrieval-augmented documents, and in-repo serialized code on adapted attention-based dependency-density diagnostics, and they contain 3 to 23 times more context-sensitive key tokens. Trained as a 128K-context extension with rotary-position scaling followed by supervised fine-tuning, the OctoLong-Instruct models outperform the 18 open-weight baselines and their own control ablation that replaces OctoLong data with a conventional long-context corpus, with the largest ablation gap appearing on agentic tool use.

Load-bearing premise

The load-bearing premise is that the language server's go-to-definition resolves each indexed call site to the function or class the code actually uses; if many resolutions land on wrong implementations because of dynamic imports, shadowed names, or incomplete environments, the measured dependency density is spurious and the benchmark gains would not come from genuine long-range dependency learning.

Editorial extensions

If this is right

  • Corpus composition, not sheer token volume, drives long-context gains: a 12% share of dependency-rich cross-repository contexts moves benchmarks more than an equal number of conventional long tokens.
  • Dependency-dense code data transfers across domains: Python-only OctoLong contexts improve multilingual repository-level recall and general-domain long-context reasoning, so code can serve as a general long-dependency training signal.
  • Agentic tool use is the task most sensitive to dependency-rich contexts: removing OctoLong data from the mix produced the largest drops on function-calling and long-horizon tool-use benchmarks.
  • Context extension need not cost short-context coding: with a post-extension model merge, the 8B model retains short-context code generation and, on API-heavy coding benchmarks at higher sampling budgets, improves it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the 12% share is a single operating point; sweeping it across, say, 3%, 6%, 12%, and 25% would show whether long-context gains saturate or accelerate and whether the optimal share scales with model size.
  • Editorial extension: the paper's dependency diagnostics could be applied as a filter to non-code long-context corpora, predicting which books, papers, or retrieved documents will help long-context fine-tuning before any training is run.
  • Editorial extension: the strong agentic gains suggest a mechanistic link between holding mutually referential definitions in context and tracking multi-turn tool state; a natural test is training on synthetic natural-language dependency chains to see whether the same gains appear outside code.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents OctoLong, a pipeline that uses an AST parser, a language server (jedi), and a package manager to recursively retrieve cross-repository code dependencies, producing long code contexts up to 128K tokens. The authors curate about 6.2B tokens of such data, mix them with roughly 44B tokens of conventional long-context data into a 50B-token LCFT corpus, and train a suite of Qwen3-based models (0.6B to 14B) with context extension to 128K, followed by instruction tuning. They evaluate on long-context coding (LooGLE V2, LongCodeQA, RepoQA), general-domain reasoning (LongBench V2, OpenAI MRCR, AA-LCR), agentic tool use (BFCL V3, τ2-Bench), and short-context coding (LiveCodeBench, BigCodeBench). The central claim is that replacing 12% of traditional LCFT data with OctoLong contexts yields substantial gains. Ablations include replacing OctoLong data with a ProLong control corpus, skipping model merging, and skipping LCFT.

Significance. If the central claim holds, OctoLong provides an effectively unlimited source of dependency-dense long-context data that transfers across code and general domains, which would be a valuable contribution to data engineering for long-context LMs. Strengths include the scale of training across five model sizes, broad benchmark coverage, a detailed data-composition appendix, a decontamination pipeline, and the explicit comparison against a ProLong replication. The paper's main limitations are that the key ablation is confounded and the core semantic-resolution step is unverified; both are addressable with additional experiments.

major comments (3)
  1. [Section 5, RQ1 and Table 2] The -Cross-Repo Code ablation does not isolate the 12% OctoLong contribution. The OctoLong-LCFT and the ProLong control corpora differ in many category proportions, not just the presence of OctoLong data: In-Repo & Code-Adjacent is 21.51% vs 30.00%, Books 9.52% vs 30.00%, Academic Papers 18.55% vs 4.00%, Math 12.22% vs 4.00%, and Educational 3.05% vs 14.00%. The performance differences between OctoLong-8B-Instruct and -Cross-Repo Code could therefore be due to the overall data recipe rather than to the OctoLong data. To support the 'supplanting just 12%' claim, the authors should run an ablation that starts from the ProLong mix and swaps in 12% OctoLong data for an equal token mass of one or more traditional categories while holding all other proportions fixed.
  2. [Section 3, 'OctoLong Data Acquisition' and Limitations] The load-bearing assumption of the pipeline is that jedi's GoToDeclaration returns the semantically correct definition for each indexed call site. The paper only filters on hop success rate (at least 60%) and depth (at least 5), where a hop is successful if jedi returns any definition; it does not verify that the returned target is the function or class actually invoked at runtime. Python's dynamic imports, re-exports, conditional imports, shadowing, and version drift between the declared dependencies and the containerized environment can all produce same-name targets with different semantics. The Limitations section addresses ABI/FFI and polyglot dependencies but not intra-language resolution errors. Without a validation sample or an audit of resolved contexts, Table 1's dependency-density diagnostics and the RQ1/RQ3 gains cannot be attributed unambiguously to genuine cross-repository dependency chains rather than to the length and code-likeness of the data. The authors should provide a manual audit of sampled contexts or compare resolved definitions against runtime-verified call graphs, and report per-repository resolution accuracy.
  3. [Tables 3, 6, and 7] No confidence intervals, significance tests, or variance estimates are reported. Several ablation deltas are small, for example LongBench V2 single-doc -0.68 and multi-doc -0.44 in Table 7, and LongCodeQA 64K -1.48 in Table 6. The abstract's 'substantial gains' claim is stronger than what the point estimates show for these subsets. The authors should report bootstrap confidence intervals over benchmark instances or run multiple training seeds, and indicate which differences are statistically distinguishable from zero.
minor comments (5)
  1. [Table 5] The 'Sequence Parallelism Degree' column is formatted ambiguously; entries like '0.6B: 21.7B: 24B: 28B: 414B: 4' should be presented as a proper table.
  2. [Section 5 and Table 2] The paper says 'replacing OctoLong data with the ProLong control corpus' in Section 5, but the control is a full ProLong replication; the wording should be clarified to avoid implying a single-category swap.
  3. [Section 3, Table 1] The dependency-density diagnostics are computed with a fixed external Phi3 model. The authors should clarify whether these diagnostics were used to select or filter OctoLong contexts, and ideally show a correlation between the diagnostics and downstream ablation gains.
  4. [Abstract and Section 4] The abstract's 'supplanting just 12% of traditional context-extension corpora' phrasing is ambiguous; it should state clearly that OctoLong data constitutes 12.05% of the total LCFT token mix, not 12% of the traditional portion.
  5. [Reproducibility] No link to code or data is provided; releasing the OctoLong pipeline and a sample of generated contexts would materially aid reproducibility and allow reviewers to audit the semantic-resolution step.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is tested by a controlled ablation against external benchmarks, with no fitted parameter or self-citation chain defining the outcome.

full rationale

The paper's central claim is that replacing 12% of a conventional long-context fine-tuning mixture with OctoLong cross-repository code data improves long-context performance. This is evaluated through a direct controlled ablation, 'OctoLong-8B-Instruct - Cross-Repo Code', in which OctoLong data is replaced by a size-matched ProLong control corpus, and the resulting models are compared on external benchmarks (LooGLE V2, LongCodeQA, RepoQA, LongBench V2, MRCR, AA-LCR, BFCL V3, tau2-Bench, LiveCodeBench, BigCodeBench). No parameter of the trained models is fitted to the benchmark outcomes, and the dependency-density diagnostics in Table 1 use a fixed external model (Phi3) rather than the trained OctoLong models, so those diagnostics are not constructed to match the training result. Some cited resources (e.g., The Stack V2, BigCodeBench) have overlapping authors with the present paper, but the central ablation does not depend on those resources, decontamination is described, and self-citation with independent content is not circularity under the review rules. The skeptic's concern about jedi's GoToDeclaration returning semantically correct definitions is a legitimate empirical robustness risk about data quality, not a reduction of the derivation to its own inputs, and the paper's own Limitations section partially acknowledges the pipeline's static-analysis boundaries.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests mainly on the correctness of the LSP-based dependency traversal and on the model-based diagnostics used to validate the data. The 12% mix fraction is a hand-chosen configuration, not a fitted optimum. No new physical or mathematical entities are introduced.

free parameters (4)
  • OctoLong data fraction in LCFT mix = 12.05%
    Single chosen configuration (Table 2). The paper claims gains from 'just 12%' but does not sweep the fraction, so the optimality or robustness of this value is untested.
  • BFS depth limit = 10
    Section 3. Traversal depth capped at 10; alternatives 8 and 15 yield similar dependency diagnostics (Table 1), so the choice is not strongly tuned.
  • Minimum hop success rate = 60%
    Section 3. Contexts below this threshold are discarded; the threshold is hand-chosen and not explicitly justified.
  • Token cap for training contexts = 128K
    Section 3 and Table 5. The paper mines contexts up to millions of tokens but trains only on 128K-capped contexts and skips eval subsets over 128K; the claim is limited to this cap.
assumptions (4)
  • domain assumption The jedi language server resolves GoToDeclaration correctly for the indexed call sites in the containerized environments.
    Section 3 Data Acquisition. The entire data pipeline depends on this; the paper only checks resolution success rates syntactically, not semantic correctness.
  • domain assumption The modified LongAttn and LongPPL diagnostics, computed with Phi3 attention and perplexity, are valid proxies for the training value of a long-context corpus.
    Section 3, Table 1. The claim that OctoLong data is unusually dependency-rich rests on these model-based metrics.
  • domain assumption The 13-gram overlap decontamination removes benchmark contamination from the training corpora.
    Appendix B. For code and agentic trajectories, 13-gram exact overlap is a weak contamination filter; semantic duplicates may remain.
  • domain assumption The replication of the ProLong corpus used as the -Cross-Repo Code control is faithful to the original ProLong recipe.
    Section 4, Table 2. The ablation's validity depends on the control being a proper counterfactual; the paper states it is 'proportionally faithful' but provides no external check.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling." pith.science (2026). https://pith.science/paper/R2TBN5BF

@misc{pith2026260805141,
  author       = {Pith},
  title        = {Pith review of: OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R2TBN5BF}},
  note         = {Machine review of arXiv:2608.05141}
}
read the original abstract

Context lengths of language models (LMs) have dramatically increased, driven by the demands for in-context learning, self-improvement, and long-horizon agentic workflows. Existing long-context corpora, however, are dominated by books, academic articles, and code repositories, which are finite resources and often scarce in long-distance dependencies. In this work, we introduce OctoLong, a context engineering pipeline that instruments an AST parser, a language server backend, and a package manager to facilitate the recursive retrieval of code references, enabling the curation of dependency-rich code contexts of millions of tokens in length. We then train OctoLong-Instruct, a suite of capable long-context open LMs, derived from base models ranging in size from 600M to 14B parameters, via context-extension mid-training on a ~50B-token mixture containing ~6.2B tokens of OctoLong code contexts, followed by ~10B tokens of instruction tuning. Our training ablations and experimental evaluations against 18 state-of-the-art open-weight long-context LMs show that supplanting just 12% of traditional context-extension corpora with OctoLong data yields substantial gains in long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks, while also enhancing API usage in short-context coding scenarios.

Figures

Figures reproduced from arXiv: 2608.05141 by the authors.

Figure 1
Figure 1. Context lengths of open-weight LMs. The trend is exponential, doubling on average every 5 months. attention (Yuan et al., 2025; Han et al., 2024; Belt￾agy et al., 2020) and linear attention (Hatamizadeh et al., 2026; Dao and Gu, 2024). In isolation, these approaches deteriorate the performance on longer contexts (Lu et al., 2024; Huang, 2024), causing LM developers to turn to orthogonal axes of improve￾ment such as … view at source ↗
Figure 2
Figure 2. Overview of the OctoLong pipeline’s core stages. Utilizing AST information, a language server backend leverages package manager metadata to recursively retrieve dependency data and engineer very long code contexts. dependencies and (b) are installable, resulting in 12K repositories. Each Python file from retained repositories becomes a candidate seed: we keep the 92K seeds that pass the OpenCoder’s default code-qual… view at source ↗
Figure 3
Figure 3. Scaling trends of the training loss curves across [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 10 canonical work pages

  1. [2]

    Which Tokens Need Context? A Reference-Based Analysis of Translation Responsibility Using Fertility and Entropy

    Why does the effective context length of llms fall short? InThe Thirteenth International Con- ference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. Ramakrishna Appicharla, Baban Gain, Santanu Pal, and Asif Ekbal. 2026. Which tokens need context? A reference-based analysis of translation responsibility using fertilit...

    work page Pith review arXiv 2025

  2. [3]

    InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025

    Mceval: Massively multilingual code evalu- ation. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. Guoxin Chen, Fanzhe Meng, Jiale Zhao, Minghao Li, Daixuan Cheng, Huatong Song, Jie Chen, Yuzhi Lin, HuiChen,XinZhao,RuihuaSong,ChangLiu,Cheng Chen, Kai Jia, and Ji-Rong Wen. 2026....

    arXiv 2025

  3. [6]

    Mariia Fedorova, Nikolay Arefyev, Maja Buljan, Jin- drich Helcl, Stephan Oepen, Egil Rønningstad, and Yves Scherrer

    OpenReview.net. Mariia Fedorova, Nikolay Arefyev, Maja Buljan, Jin- drich Helcl, Stephan Oepen, Egil Rønningstad, and Yves Scherrer. 2026. Openlid-v3: Improving the precision of closely related language identification - an experience report. InProceedings of the 13th Workshop on NLP for Similar Languages, Varieties andDialects,VarDial@EACL2026,Rabat,Moroc...

    work page 2026

  4. [7]

    Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, HannanehHajishirzi,YoonKim,andHaoPeng.2024

    Ragbench: Explainable benchmark for retrieval-augmented generation systems.CoRR, abs/2407.11005. Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, HannanehHajishirzi,YoonKim,andHaoPeng.2024. Dataengineeringforscalinglanguagemodelsto128k context. InForty-first International Conference on MachineLearning,ICML2024,Vienna,Austria,July 21-27, 2024, volume 235 ofP...

    arXiv 2024

  5. [9]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong,WentaoZhang,GuantingChen,XiaoBi,Y.Wu, Y

    Openthoughts: Data recipes for reasoning models.CoRR, abs/2506.04178. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong,WentaoZhang,GuantingChen,XiaoBi,Y.Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. 2024. Deepseek-coder: When the large lan- guage model meets programming - the rise of code intelligence.CoRR, abs/2401.14196. KshitijGupta,B...

    arXiv 2024

  6. [10]

    In27th International Conference on En- gineering of Complex Computer Systems, ICECCS 2023, Toulouse, France, June 14-16, 2023, pages 116–125

    Polyglot AST: towards enabling polyglot code analysis. In27th International Conference on En- gineering of Complex Computer Systems, ICECCS 2023, Toulouse, France, June 14-16, 2023, pages 116–125. IEEE. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shan- tanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. RULER: what’s the real contex...

    arXiv 2023

  7. [11]

    Junlong Jia, Xing Wu, Chaochen Gao, Ziyang Chen, Zijia Lin, Zhongzhi Li, Weinong Wang, Haotian Xu, Donghui Jin, Debing Zhang, and Binghui Guo

    Entropylong: Effective long-context training via predictive uncertainty.CoRR, abs/2510.02330. Junlong Jia, Xing Wu, Chaochen Gao, Ziyang Chen, Zijia Lin, Zhongzhi Li, Weinong Wang, Haotian Xu, Donghui Jin, Debing Zhang, and Binghui Guo. 2026. Litelong: Resource-efficientlong-contextdatasynthe- sisforllms. InFortiethAAAIConferenceonArtificial Intelligence,...

    arXiv

  8. [12]

    In TheThirteenthInternationalConferenceonLearning Representations, ICLR 2025, Singapore, April 24-28,

    Selective attention improves transformer. In TheThirteenthInternationalConferenceonLearning Representations, ICLR 2025, Singapore, April 24-28,

    work page 2025

Show all 25 references
  1. [13]

    Dacheng Li, Rulin Shao, Anze Xie, Eric P

    OpenReview.net. Dacheng Li, Rulin Shao, Anze Xie, Eric P. Xing, JosephE.Gonzalez,IonStoica,XuezheMa,andHao Zhang. 2023a. Lightseq: Sequence level parallelism for distributed training of long context transformers. CoRR, abs/2310.03294. GuohaoLi,HasanAbedAlKaderHammoud,HaniItani...

    arXiv

  2. [14]

    Yi Lu, Jing Nathan Yan, Songlin Yang, Justin T

    Starcoder 2 and the stack v2: The next genera- tion.CoRR, abs/2402.19173. Yi Lu, Jing Nathan Yan, Songlin Yang, Justin T. Chiu, SiyuRen,FeiYuan,WentingZhao,ZhiyongWu,and AlexanderM.Rush.2024. Acontrolledstudyonlong context extension and generalization in llms.CoRR, abs/2409.12...

    2024 arXiv

  3. [16]

    Maksim Sapronov and Evgeniy Glukhov

    Lost in the middle: An emergent property from information retrieval demands in llms.CoRR, abs/2510.10276. Maksim Sapronov and Evgeniy Glukhov. 2025. On pretraining for project-level code completion.CoRR, abs/2510.13697. 16 Jingyan Shen, Ang Li, Salman Rahman, Yifan Sun, Micah ...

    2025

  4. [17]

    CoRR, abs/2512.23675

    End-to-end test-time training for long context. CoRR, abs/2512.23675. Zecheng Tang, Baibei Ji, Juntao Li, Lijun Wu, Haijia Gui, and Min Zhang. 2025. Revisiting long-context modeling from context denoising perspective.CoRR, abs/2510.05862. ArtificialAnalysis Team. 2025a. Artifi...

    2025

  5. [18]

    Runchu Tian, Yanghao Li, Yuepeng Fu, Siyang Deng, Qinyu Luo, Cheng Qian, Shuo Wang, Xin Cong, ZhongZhang,YesaiWu,YankaiLin,HuadongWang, and Xiaojiang Liu

    Hermes 3 technical report.CoRR, abs/2408.11857. Runchu Tian, Yanghao Li, Yuepeng Fu, Siyang Deng, Qinyu Luo, Cheng Qian, Shuo Wang, Xin Cong, ZhongZhang,YesaiWu,YankaiLin,HuadongWang, and Xiaojiang Liu. 2025. Distance between relevant information pieces causes bias in long-con...

    2025 arXiv

  6. [20]

    InFindingsofthe AssociationforComputationalLinguistics,ACL2026, San Diego, California, United States, July 2-7, 2026, pages 13122–13133

    From 128k to 4m: Efficient training of ultra- longcontextlargelanguagemodels. InFindingsofthe AssociationforComputationalLinguistics,ACL2026, San Diego, California, United States, July 2-7, 2026, pages 13122–13133. Association for Computational Linguistics. Zhangchen Xu, Fengq...

    2026

  7. [22]

    Skyladder: Better and faster pretraining via context window scheduling. InAdvances in Neural Information Processing Systems 38: Annual Confer- enceonNeuralInformationProcessingSystems2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025/MexicoCity,Mexico,November30-Decemb...

    2025

  8. [23]

    OpenReview.net. A Architecture & Training Details Attribute Value Common Attributes Backbone Family Qwen/Qwen3Backbone Type DenseBackbone Sizes 0.6B, 1.7B, 4B, 8B, 14BTuning Strategy Full Fine-TuningTraining Sequence Length 131,072 TokensModel Datatype bfloat16Flash Attention ...

  9. [24]

    corpus, offering full-length narrative and non-fiction prose as naturally long-form text. 2 BookSum 2011 Samples ubaada/booksum-complete-cleaned Book- and chapter-level summarization pairs fromBookSum(Kryscinskietal.,2022),which couple long source documents with abstractive su...

    2011

  10. [25]

    study, contributing curated technical con- tent and scientific prose. Tech Content & Forum Data 1 Marin-StackExchange 2000000 Samples marin-community/stackexchange-markdown StackExchange question-and-answer threads renderedtocleanMarkdownbytheMarincom- munity, covering technic...

    2025

  11. [2021]

    InPro- ceedings of the 6th Workshop on Formal Integrated DevelopmentEnvironment,F-IDE@NFM2021,Held online, 24-25th May 2021, volume 338 ofEPTCS, pages 3–18

    The specification language server protocol: A proposal for standardised LSP extensions. InPro- ceedings of the 6th Workshop on Formal Integrated DevelopmentEnvironment,F-IDE@NFM2021,Held online, 24-25th May 2021, volume 338 ofEPTCS, pages 3–18. Neil Rathi and Alec Radford. 202...

    2021

  12. [2022]

    InProceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics(Volume1: LongPapers), ACL2022, Dublin, Ireland, May 22-27, 2022, pages 8003–8016

    Linkbert: Pretraining language models with document links. InProceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics(Volume1: LongPapers), ACL2022, Dublin, Ireland, May 22-27, 2022, pages 8003–8016. Associ- ation for Computational Linguistics....

    2022 arXiv

  13. [2023]

    InProceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Sin- gapore, December 6-10, 2023, pages 3029–3051

    Enhancing chat language models by scaling high-quality instructional conversations. InProceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Sin- gapore, December 6-10, 2023, pages 3029–3051. Association for Computational Linguist...

    2023

  14. [2024]

    InIEEE International Conference on Software Maintenance and Evolution, ICSME 2024, Flagstaff, AZ, USA, October 6-11, 2024, pages 888–892

    SEART data hub: Streamlining large-scale source code mining and pre-processing. InIEEE International Conference on Software Maintenance and Evolution, ICSME 2024, Flagstaff, AZ, USA, October 6-11, 2024, pages 888–892. IEEE. Tri Dao and Albert Gu. 2024. Transformers are ssms: G...

    2024

  15. [2025]

    InThe Thirteenth International ConferenceonLearningRepresentations,ICLR2025, Singapore, April 24-28, 2025

    MIND: math informed synthetic dialogues for pretraining llms. InThe Thirteenth International ConferenceonLearningRepresentations,ICLR2025, Singapore, April 24-28, 2025. OpenReview.net. Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and...

    2025

  16. [2026]

    A decomposition perspective to long-context reasoning for llms.CoRR, abs/2604.07981. WenhanXiong,JingyuLiu,IgorMolybog,HejiaZhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan N...

    2024 arXiv

  17. [7399]

    Charles Goddard

    Association for Computational Linguistics. Charles Goddard. 2025. Extending afm-4.5b to 64k context length. Technical report, Arcee AI. Shashwat Goel, Rishi Hazra, Dulhan Jayalath, Timon Willi, Parag Jain, William F. Shen, Ilias Leon- tiadis, Francesco Barbieri, Yoram Bachrach...

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.