Pith. sign in

REVIEW 5 major objections 4 minor 94 references

SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimization

T0 review · 5 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Long-context preference training can be replaced by short-context training plus a consistency regularizer.

desk verdict Useful and credible empirical recipe for efficient long-context preference optimization, but the motivating theorem has a fixable gap and the evaluation has some benchmark-selection leakage. read the letter →

arxiv 2505.11166 v3 pith:UF4I7FY7 submitted 2025-05-16 cs.CL cs.AI

Huashan Sun , Shengyi Liao , Yansen Han , Yu Bai , Yang Gao , Cheng Fu , Weizhou Shen , Fanqi Wan
show 3 more authors
Ming Yan Ji Zhang Fei Huang
This is my paper · ORCID
classification cs.CLcs.AI
keywords long-contextalignmentpreferenceoptimizationDPOSimPOORPOrewardcontextcompressioncontextualknowledgelocalization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that aligning an LLM to prefer good answers in long contexts does not require preference pairs drawn from long contexts. It proves a bound (Theorem 1) showing the long-context preference-optimization loss is at most one-third of the corresponding short-context loss plus a term that penalizes reward disagreement between the same response conditioned on the short and long contexts. The paper then trains models with short-context preference optimization and that reward-consistency term, called SoLo-RA, and reports that the resulting models beat vanilla long-context DPO, SimPO, and ORPO on long-context QA benchmarks while using far less training data and compute. A sympathetic reader would care because the result suggests long-context alignment can be bought cheaply if the key assumption about preference order holds.

What carries the argument

The load-bearing identity is Theorem 1's inequality, which splits the long-context loss via Jensen's inequality into three terms: the short-context preference loss and two cross-context reward-difference terms that collapse into the SoLo-RA penalty. SoLo-RA is a chosen-only regularizer of the form $|r_\phi(x_{\text{short}},y)-r_\phi(x_{\text{long}},y)|$, up to each algorithm's reward definition, forcing the policy to assign the same reward to a response whether conditioned on the compressed context or the full context. In preference optimization the policy is the reward model, so this term is claimed to teach the model to locate task-relevant content inside the long context, while the short-context PO term teaches reasoning over that content.

What would settle it

Take a set of long-context QA tasks where the answer depends on information spread across multiple documents, so the task-relevant excerpt extracted from one document is incomplete; measure the preference probabilities $p(y_w \succ y_l|x_{\text{long}})$ and $p(y_w \succ y_l|x_{\text{short}})$ with a held-out reward model. If the long-context preference probability exceeds the short-context one on a substantial fraction of pairs, Assumption 1 and the bound are violated, and the decoupling rationale would not be expected to hold on that task.

Watch

Extended reading notes

Core claim

The central claim is the decoupling inequality $$L_{\eta,\gamma}(x_{\text{long}}) \le \frac{1}{3}L_{3\eta,\gamma/3}(x_{\text{short}}) + \mathbb{E}_y[s(3\eta\,|r_\$\varphi$(x_{\text{short}},y)-r_\$\varphi$(x_{\text{long}},y)|)].$$ This says long-context preference optimization is dominated by a short-context preference loss plus a reward-consistency penalty, provided the preference order is easier to distinguish on the short context than on the full long context, which is the paper's Assumption 1. SoLoPO instantiates the two terms as short-context preference optimization and short-to-long reward alignment (SoLo-RA), and the paper further applies SoLo-RA only to the chosen response for stability and efficiency. Experiments on Qwen2.5-7B, Qwen2.5-14B, and Llama3.1-8B, trained on MuSiQue-derived short/long context pairs, report consistent gains over vanilla PO on LongBenchV1, RULER, LongBenchV2, and NIAH-Plus, along with run-time and memory savings.

Load-bearing premise

The load-bearing premise is that a model can distinguish the chosen from the rejected response at least as well from a short context containing only the task-relevant content as from the full long context; if the long context ever supplies genuinely new information that changes which answer is better, the inequality and the decoupling no longer follow.

Editorial extensions

If this is right

  • Training data for long-context alignment can be built from short contexts: sample responses, pick preference pairs, then concatenate distractors to form $x_{\text{long}}$.
  • SoLoPO cuts compute and memory: only the chosen response is processed on the long context, so run time drops by roughly 42 to 52 percent and the maximum trainable length about doubles in the reported settings.
  • The decoupling applies to any preference-optimization algorithm whose loss function $f$ has an upper bounding function $s$, including DPO, SimPO, ORPO, IPO, and SLiC.
  • A 7B model trained on MuSiQue reaches long-context QA scores close to a 72B instruct model on LongBenchV1, while generalizing across domain and length benchmarks.
  • SoLo-RA improves contextual knowledge localization, as measured on NIAH-Plus, beyond the non-decoupled Expand-Long-PO baseline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the long context adds cross-document evidence that changes the correct answer, Assumption 1 can fail, and the bound would no longer justify dropping long-context preference pairs.
  • Editorial inference: the same two-term objective should transfer to any task where a compressed input preserves decision-relevant content, such as context-faithful generation or complex instruction following.
  • Editorial inference: because SoLo-RA only enforces reward equality on sampled chosen responses, its benefit likely scales with preference-pair quality, so rejection sampling or verifier-based selection could amplify the reported gains.
  • Editorial inference: with compression rate $c$ the theoretical speedup is $2/(2c^2+1)$, so token-level compression is only the start; applying the decoupling to hidden-state or KV-cache compression is a natural extension.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The manuscript proposes SoLoPO, a framework that decouples long-context preference optimization (PO) into two components: a short-context PO term and a short-to-long reward alignment regularizer (SoLo-RA). The authors state a theorem (Theorem 1) bounding the long-context PO loss by a combination of short-context PO loss and a reward-consistency term, and use this bound to justify training on short-context preference pairs plus a chosen-only long-context regularizer. The framework is instantiated for DPO, SimPO, and ORPO and evaluated on LongBenchV1, RULER, LongBenchV2, NIAH-Plus, and short-context benchmarks using Qwen2.5-7B, Llama3.1-8B, and Qwen2.5-14B. The empirical results show consistent improvements over vanilla PO methods and substantial efficiency gains (up to 2.1x trainable length and up to 52% runtime reduction).

Significance. If the decoupling theorem were sound, this would be an important contribution to long-context alignment, both conceptually and practically. The empirical study is broad: it covers three PO algorithms, three model sizes, multiple long- and short-context benchmarks, and a careful efficiency analysis with a FLOP-based speedup model. The paper also provides a detailed data construction pipeline and training configuration, which supports reproducibility. However, because the central theoretical claim has a gap in the proof and the implemented loss deviates from the stated bound, the significance currently rests on the empirical results alone, which, while promising, are not enough to establish the paper's advertised theoretical guarantee.

major comments (5)
  1. [Appendix H.2, Eq. (21)] The step from Eq. (20) to Eq. (21) is not justified. The sum of the first and third terms in Eq. (14) is an expectation over the marginals D_{yw|xshort} and D_{yl|xshort}. Replacing this sum by an expectation over (yw,yl)∼D_y multiplied by P(yw≻yl|x_short) requires either (i) that D_y is the base pair distribution and a normalization constant Z = E_{D_y}[P(yw≻yl|x_short)] is inserted, or (ii) that D_y is already the conditional pair distribution, in which case the factor P(yw≻yl|x_short) is spurious. Neither case is specified, so the subsequent inequality (22) and the final bound (7) are not established. Since Theorem 1 is the basis for the SoLoPO loss in Eqs. (8)-(9), the theoretical decoupling claim is unsupported as written.
  2. [Appendix H.2, Eq. (25)] The substitution of the expectation over (yw,yl)∼D_y of f(3ηΔ_1(yw)) + f(3ηΔ_3(yl)) by the expectation over y∼D_y of f(3ηΔ_1(y)) + f(3ηΔ_3(y)) assumes that yw and yl are interchangeable draws from the same marginal distribution. This holds only for symmetric preference-pair distributions, but it does not hold for the correctness-labeled chosen/rejected pairs used in the experiments (Section 3), where the rejected responses are often degenerate (e.g., "No answer"). The theorem therefore does not cover the actual SoLoPO objective, including the chosen-only variant.
  3. [Section 2.2, Eqs. (7)-(9)] The implemented SoLoPO loss is not the bound derived in Theorem 1. The right-hand side of Eq. (7) has a 1/3 prefactor and the reward-consistency term is s(|3η·(...)|), while Eq. (9) introduces a tunable α and omits the 1/3 prefactor. Moreover, Theorem 1 concerns an abstract loss L_{η,γ}(x_text) defined over the model's preference distribution D_{yw≻yl|xtext}, whereas the implemented loss (Table 1) uses empirical labels where yw≻yl is a fixed correctness judgment, not a probability derived from the reward model. The precise relationship between the theorem and the implemented surrogate needs to be stated explicitly.
  4. [Appendix H.7, Table 12; Assumption 1] Assumption 1 is the load-bearing premise of Theorem 1, but it is validated only on Qwen2.5-7B with synthetic MuSiQue-style contexts, with a satisfaction rate of about 95%. The assumption may fail on real multi-document tasks where the long context contributes information beyond the extracted c_rel, or for models with different attention patterns. Please provide validation on real long-context datasets and at least one additional model, or explicitly restrict the theoretical claim to the settings where the assumption is verified.
  5. [Section 4, Appendix D.1] The final model used for all reported results is the checkpoint with the best LongBenchV1 performance within a single training epoch, and the reward alignment coefficient α is tuned on LongBenchV1 (Appendix D.1). Since LongBenchV1 is also the principal benchmark in Table 3, the reported gains may partly reflect selection on the test set. Please use a held-out validation set for checkpoint and hyperparameter selection, or report sensitivity of the main results to these choices.
minor comments (4)
  1. [Section 2.2] The definition "xshort := x irr := [cirr;I]" appears to mix notations: based on the preceding sentence, it should read x_rel := [c_rel;I] rather than x_irr/c_irr, which denote the irrelevant portion.
  2. [Appendix H.2, Eqs. (15)-(18)] The notation "yw,yl∼D_y, yw≻yl|x_long" is nonstandard and should be defined explicitly; the conditioning on the preference event changes the sampling distribution in a way that appears to ignore a normalization constant.
  3. [Table 14] The listed upper bound s(x) for DPO and SimPO is |x| + 2 log(1+e^{3γ}), which does not align with the constants derived in Proposition 2 (2/3 log(1+e^{ηγ}) after scaling); please reconcile the notation and constants.
  4. [Figure 1(c)] The grouped bar chart is difficult to read because of the large number of bars; consider using a table or a clearer grouping of the benchmarks.

Circularity Check

0 steps flagged · score 2.0 of 10

No material circularity: the SoLoPO objective is the RHS of a derived upper bound, and the paper's self-citations are not load-bearing.

full rationale

Walking the derivation chain from Eqs. (4)-(9) and Theorem 1 (Appendix H.2), the claimed decoupling is an upper-bound argument: L_{eta,gamma}(x_long) is bounded by (1/3)[L_{3eta,gamma/3}(x_short) + E[s(|3eta(r(x_short,y)-r(x_long,y))|)]], and the proposed SoLoPO loss (Eqs. 8-9) is exactly the right-hand side of that bound, with alpha weighting the alignment term. Minimizing a provable upper bound of the long-context loss is a standard surrogate-optimization step, not a case in which the output is equivalent to the input by construction. Assumption 1 (p(y_w≻y_l|x_long) ≤ p(y_w≻y_l|x_short)) is a domain assumption with its own empirical check (Appendix H.7, roughly 95% satisfaction on Qwen2.5-7B), not a restatement of the conclusion. No fitted parameter is renamed as a prediction: alpha and the compression ratio are manually tuned hyperparameters, and the paper explicitly flags this in Appendix B as a limitation. The self-citations (compression rate from [2], CItruS-style compression from [3], and the survey [37] where present authors overlap) are contextual references; none carries the proof or the benchmark comparison, which is evaluated against external LongBench/RULER/LongBenchV2/Open-LLM-Leaderboard. A skeptical concern about Eq. (21) in Appendix H.2 (whether the expectation over D_y needs a normalization constant or the preference-pair distribution) is a proof-validity risk, and the paper's own limitations section admits the evaluation is confined to roughly 32K contexts with synthetic MuSiQue-style data; these are correctness and scope concerns, not circularity. The central claim therefore does not reduce to its inputs by definition, and the only minor self-citations are not load-bearing.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

SoLoPO's central claim rests on an upper bound (Theorem 1) whose derivation assumes (i) the redundancy hypothesis with a task-relevant subset, (ii) Assumption 1 that preference order is easier to discriminate on short than long contexts, and (iii) standard convexity properties of PO losses. The loss itself introduces a balance coefficient alpha that is hand-tuned per method and model, and a compression ratio that is chosen heuristically. No new entities are postulated.

free parameters (2)
  • Reward alignment coefficient alpha = 3 (SoLo-DPO, Qwen2.5-7B), 1 (SoLo-SimPO, Qwen2.5-7B), 1 (SoLo-ORPO, Qwen2.5-7B), 4 (SoLo-ORPO, Llama3.1-8B)
    Balances short-context PO and SoLo-RA in Eq. (9); tuned to best LongBenchV1 performance (Appendix D.1, Figures 3 and 11), i.e., selected on the test benchmark.
  • Context compression ratio / lengths of x_short and x_long = x_short ~1.1K and x_long ~7.5K tokens in training; x_short=1K and x_long=8K in main efficiency analysis
    The ratio defines the SoLo-RA input pair; chosen heuristically per Section 3 and Appendix C, not optimized, but it directly affects the loss behavior and the reported efficiency gains.
assumptions (4)
  • domain assumption Redundancy hypothesis: a long context c_long can be decomposed into task-relevant content c_rel and irrelevant c_irr, and x_short := [c_rel; I] preserves all task-critical information.
    Invoked in Section 2.2 (Problem setting) to justify constructing x_short from c_rel. If real long-context tasks are not reducible to a short relevant excerpt, the entire short-to-long transfer premise is weakened.
  • domain assumption Assumption 1 (Discrimination of preference order): p(yw≻yl|x_long) ≤ p(yw≻yl|x_short).
    Load-bearing for Theorem 1; used to bound the long-context preference expectation by the short-context expectation. Empirically validated only on one model and one synthetic dataset, at about 95% agreement (Appendix H.7).
  • standard math For DPO/SimPO/ORPO, the loss function f is convex and satisfies f(x+γ)+f(−x+γ) ≤ s(|x|) for the chosen s.
    Required by Theorem 1 and used to instantiate SoLoPO for each algorithm (Table 14). The bounds are loose but valid for the specified f.
  • standard math Preferences follow the Bradley-Terry / GPO model p(yw≻yl|x) = σ(r(x,yw) − r(x,yl)).
    Background framework from Section 2.1; the entire analysis of preference losses is built on this representation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimization." pith.science (2026). https://pith.science/paper/UF4I7FY7

@misc{pith2026250511166,
  author       = {Pith},
  title        = {Pith review of: SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UF4I7FY7}},
  note         = {Machine review of arXiv:2505.11166}
}
read the original abstract

Despite advances in pretraining with extended context sizes, large language models (LLMs) still face challenges in effectively utilizing real-world long-context information, primarily due to insufficient long-context alignment caused by data quality issues, training inefficiencies, and the lack of well-designed optimization objectives. To address these limitations, we propose a framework named \textbf{S}h\textbf{o}rt-to-\textbf{Lo}ng \textbf{P}reference \textbf{O}ptimization (\textbf{SoLoPO}), decoupling long-context preference optimization (PO) into two components: short-context PO and short-to-long reward alignment (SoLo-RA), supported by both theoretical and empirical evidence. Specifically, short-context PO leverages preference pairs sampled from short contexts to enhance the model's contextual knowledge utilization ability. Meanwhile, SoLo-RA explicitly encourages reward score consistency for the responses when conditioned on both short and long contexts that contain identical task-relevant information. This facilitates transferring the model's ability to handle short contexts into long-context scenarios. SoLoPO is compatible with mainstream preference optimization algorithms, while substantially improving the efficiency of data construction and training processes. Experimental results show that SoLoPO enhances all these algorithms with respect to stronger length and domain generalization abilities across various long-context benchmarks, while achieving notable improvements in both computational and memory efficiency.

Figures

Figures reproduced from arXiv: 2505.11166 by the authors.

Figure 1
Figure 1. Original PO vs. SoLoPO. (a) SoLoPO decouples long-context PO into two components: short-context PO and short-to-long reward alignment, reducing the complexity of preference data construction and minimizing long-text processing during training. (b) Under identical configurations, SoLoPO exhibits superior training efficiency compared to vanilla methods. (c) SoLoPO outperforms the original PO across various long-contex… view at source ↗
Figure 2
Figure 2. Performance improvements of different short-to-long preference optimization frameworks [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Performance w/ different α in SoLo-ORPO. 4.2 11.9 15.1 15.0 15.5 54.00 66.63 83.62 144.21 179.20 72.52 145.11 235.98 0 5 10 15 20 30 60 90 120 150 180 210 240 270 1K 4K 8K 12K 16K Improvements on QAs-LongbenchV1 Run Time(RT)/min Length of Long-Context (short-context fixed at 1k) Imp.-SoLo-ORPO RT-SoLo-ORPO RT-Vanilla-ORPO 42% 39% [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: Illustration of the construction pipeline for the short-to-long dataset. (1) Irrelevant docu [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Prompt template used for data construction and training, adapted from Li et al. [39] [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Prompt template used for LongBenchV2 evaluation, adapted from Bai et al. [6] [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 8
Figure 8. Figure 8: Overall Performance on LongbenchV2. 1. We report the average score with standard deviation across 5 evaluation runs for each model. 2. All of these metrics are reasonable. 7 https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard 8 https://github.com/El…
Figure 9
Figure 9. Figure 9: Comparison of performance improvements achieved by various ORPO methods relative to [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]
Figure 10
Figure 10. Figure 10: Changes of reward margins and log prob. of rejected response during SoLo-ORPO training [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: Performance with different α in SoLo-DPO and SoLo-SimPO in the Qwen2.5-7B setting. The optimal values of α for SoLo-DPO and SoLo-SimPO are 3 and 1, respectively. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

94 extracted references · 27 canonical work pages

  1. [1]

    A general theoretical paradigm to understand learning from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. InInternational Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR, 2024

    2024

  2. [2]

    Unifying cross-lingual summarization and machine translation with compression rate

    Yu Bai, Heyan Huang, Kai Fan, Yang Gao, Yiming Zhu, Jiaao Zhan, Zewen Chi, and Boxing Chen. Unifying cross-lingual summarization and machine translation with compression rate. InSIGIR 2022 - Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2022 - Proceedings of the 45th International AC...

    2022

  3. [3]

    CItruS: Chunked instruction-aware state eviction for long sequence modeling

    Yu Bai, Xiyuan Zou, Heyan Huang, Sanxing Chen, Marc-Antoine Rondeau, Yang Gao, and Jackie CK Cheung. CItruS: Chunked instruction-aware state eviction for long sequence modeling. In Yaser Al- Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5908–5930, Miami, Flor...

    work page doi:10.18653/v1/2024.emnlp-main.338 2024

  4. [4]

    LongAlign: A recipe for long context alignment of large language models

    Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. LongAlign: A recipe for long context alignment of large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1376–1395, Miami, Florida, USA, November 2024. As...

    2024

  5. [5]

    LongBench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for ...

    2024

  6. [6]

    Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks, 2025

    Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks, 2025. URLhttps://arxiv.org/abs/2412.15204

    arXiv 2025

  7. [7]

    Longwriter: Unleashing 10,000+ word generation from long context LLMs

    Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longwriter: Unleashing 10,000+ word generation from long context LLMs. InThe Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum? id=kQ5s9Yh0WI

    2025

  8. [8]

    Luna: A lightweight evaluation model to catch language model hallucinations with high accuracy and low cost

    Masha Belyi, Robert Friel, Shuai Shao, and Atindriyo Sanyal. Luna: A lightweight evaluation model to catch language model hallucinations with high accuracy and low cost. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, Steven Schockaert, Kareem Darwish, and 10 Apoorv Agarwal, editors,Proceedings of the 31st Internation...

    2025

Show all 94 references
  1. [9]

    Context- dpo: Aligning language models for context-faithfulness.CoRR, abs/2412.15280, 2024

    Baolong Bi, Shaohan Huang, Yiwei Wang, Tianchi Yang, Zihan Zhang, Haizhen Huang, Lingrui Mei, Junfeng Fang, Zehao Li, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, and Shenghua Liu. Context- dpo: Aligning language models for context-faithfulness.CoRR, abs/2412.15280, 2024. URL ht...

    2024 doi arXiv

  2. [10]

    Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons.Biometrika, 39(3/4):324–345, 1952. ISSN 00063444, 14643510. URL http: //www.jstor.org/stable/2334029

    1952

  3. [11]

    LongPO: Long context self-evolution of large language models through short-to-long preference optimization

    Guanzheng Chen, Xin Li, Michael Shieh, and Lidong Bing. LongPO: Long context self-evolution of large language models through short-to-long preference optimization. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview.net/forum?id=q...

    2025

  4. [12]

    LongloRA: Efficient fine-tuning of long-context large language models

    Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. LongloRA: Efficient fine-tuning of long-context large language models. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=6PmJoRfdaK

    2024

  5. [13]

    Flashattention-2: Faster attention with better parallelism and work partitioning, 2023

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. URL https://arxiv.org/abs/2307.08691

    2023 arXiv

  6. [14]

    Smith, and Matt Gardner

    Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. A dataset of information-seeking questions and answers anchored in research papers. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan C...

    2021

  7. [15]

    A survey on long text modeling with transformers

    Zican Dong, Tianyi Tang, Lunyi Li, and Wayne Xin Zhao. A survey on long text modeling with transformers. ArXiv, abs/2302.14502, 2023. URLhttps://api.semanticscholar.org/CorpusID:257232619

    2023 arXiv

  8. [16]

    Longred: Mitigating short-text degradation of long-context large language models via restoration distillation, 2025

    Zican Dong, Junyi Li, Jinhao Jiang, Mingyu Xu, Wayne Xin Zhao, Bingning Wang, and Weipeng Chen. Longred: Mitigating short-text degradation of long-context large language models via restoration distillation, 2025. URLhttps://arxiv.org/abs/2502.07365

    2025 arXiv

  9. [17]

    Ahmad Al-Dahle

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, and et al. Ahmad Al-Dahle. The llama 3 herd of models.ArXiv, abs/2407.21783, 2024. URL https://api.semanticscholar.org/ CorpusID:271571434

    2024 arXiv

  10. [18]

    What is wrong with perplexity for long-context language modeling? InThe Thirteenth International Conference on Learning Representations, 2025

    Lizhe Fang, Yifei Wang, Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, and Yisen Wang. What is wrong with perplexity for long-context language modeling? InThe Thirteenth International Conference on Learning Representations, 2025. URL https://openrevie...

    2025

  11. [19]

    Open llm leader- board v2

    Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open llm leader- board v2. https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard , 2024

    2024

  12. [20]

    Data engineering for scaling language models to 128k context

    Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2...

    2024

  13. [21]

    How to train long-context language models (effectively)

    Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. How to train long-context language models (effectively). In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Proceedings of the 63rd Annual Meeting of the Association for Computationa...

    2025 doi

  14. [22]

    How to train long-context language models (effectively), 2025

    Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. How to train long-context language models (effectively), 2025. URLhttps://openreview.net/forum?id=nwZHFKrYTB

    2025

  15. [23]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. InFirst Conference on Language Modeling, 2024. URLhttps://openreview.net/forum?id=tEYskw1VY2. 11

    2024

  16. [24]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. InInternational Conference on Learning Representations, 2022. URL https://openreview. net/forum?id=uYLFoz1vlAC

    2022

  17. [25]

    Two stones hit one bird: bilevel positional encoding for better length extrapolation

    Zhenyu He, Guhao Feng, Shengjie Luo, Kai Yang, Liwei Wang, Jingjing Xu, Zhi Zhang, Hongxia Yang, and Di He. Two stones hit one bird: bilevel positional encoding for better length extrapolation. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. J...

    2024

  18. [26]

    Measuring mathematical problem solving with the math dataset, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxiv.org/abs/2103.03874

    2021 arXiv

  19. [27]

    Improving long context document-level machine translation

    Christian Herold and Hermann Ney. Improving long context document-level machine translation. In Michael Strube, Chloe Braud, Christian Hardmeier, Junyi Jessy Li, Sharid Loaiciga, and Amir Zeldes, editors,Proceedings of the 4th Workshop on Computational Approaches to Discourse ...

    2023 doi

  20. [28]

    Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Nuria Bel, and Chengqing Zong, editors,Proceedings of the 28th International Conference on Computational Lingui...

    2020 doi

  21. [29]

    ORPO: Monolithic preference optimization without reference model

    Jiwoo Hong, Noah Lee, and James Thorne. ORPO: Monolithic preference optimization without reference model. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11170–11189, Mi...

    2024 doi

  22. [30]

    RULER: What’s the real context size of your long-context language models? InFirst Conference on Language Modeling, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? InFirst Conference on Language Modeling, 2024. URLhttps://openreview.net/forum?id=kIoBbc76Sy

    2024

  23. [31]

    Fewer is more: Boosting math reasoning with reinforced context pruning

    Xijie Huang, Li Lyna Zhang, Kwang-Ting Cheng, Fan Yang, and Mao Yang. Fewer is more: Boosting math reasoning with reinforced context pruning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in Natural Langua...

    2024 doi

  24. [32]

    Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models, 2023

    Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models, 2023. URLhttps://arxiv.org/abs/2309.14509

    2023 arXiv

  25. [33]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  26. [34]

    The NarrativeQA reading comprehension challenge.Transactions of the Association for Computational Linguistics, 6:317–328, 2018

    Tomáš Koˇciský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge.Transactions of the Association for Computational Linguistics, 6:317–328, 2018. doi: 10.1162/tacl_a_00023. URL...

    2018 doi

  27. [35]

    Babilong: Testing the limits of llms with long context reasoning-in-a-haystack

    Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of llms with long context reasoning-in-a-haystack. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, ed...

    2024

  28. [36]

    Bart: Denoising sequence-to-sequence pre-training for natural lan- guage generation, translation, and comprehension

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdel rahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural lan- guage generation, translation, and comprehension. InAnnual Meeting of the Asso...

    2019

  29. [37]

    Fundamental capabilities and applications of large language models: A survey.ACM Comput

    Jiawei Li, Yang Gao, Yizhe Yang, Yu Bai, Xiaofeng Zhou, Yinghao Li, Huashan Sun, Yuhang Liu, Xingpeng Si, Yuhao Ye, Yixiao Wu, Yiguan Lin, Bin Xu, Bowen Ren, Chong Feng, and Heyan Huang. Fundamental capabilities and applications of large language models: A survey.ACM Comput. S...

    2025 doi

  30. [38]

    Pspo*: An effective process-supervised policy optimization for reasoning alignment, 2025

    Jiawei Li, Xinyue Liang, Junlong Zhang, Yizhe Yang, Chong Feng, and Yang Gao. Pspo*: An effective process-supervised policy optimization for reasoning alignment, 2025. URL https://arxiv.org/abs/ 2411.11681

    2025 arXiv

  31. [39]

    Making long-context language models better multi-hop reasoners

    Yanyang Li, Shuo Liang, Michael Lyu, and Liwei Wang. Making long-context language models better multi-hop reasoners. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon...

    2024 doi

  32. [40]

    Compressing context to enhance inference efficiency of large language models

    Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context to enhance inference efficiency of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, page...

    2023 doi

  33. [41]

    SnapKV: LLM knows what you are looking for before generation

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024....

    2024

  34. [42]

    Mdcure: A scalable pipeline for multi-document instruction-following.CoRR, abs/2410.23463, 2024

    Gabrielle Kaili-May Liu, Bowen Shi, Avi Caciularu, Idan Szpektor, and Arman Cohan. Mdcure: A scalable pipeline for multi-document instruction-following.CoRR, abs/2410.23463, 2024. doi: 10.48550/ARXIV . 2410.23463. URLhttps://doi.org/10.48550/arXiv.2410.23463

    2024 doi arXiv

  35. [43]

    A comprehensive survey on long context language modeling

    Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Moore Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, Yuanxing Zhang, Zhuo Chen, Hangyu Guo, Shilong Li, Ziqiang Liu, Yong Shan, Yifan Song, Jiayi Tian, Wenhao Wu, Zhejian Zhou, Ruijie Zhu, Junlan F...

    2025

  36. [44]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternational Conference on Learning Representations, 2019. URLhttps://openreview.net/forum?id=Bkg6RiCqY7

    2019

  37. [45]

    Zhang, Zhilin Yang, Xinyu Zhou, Mingxing Zhang, and Jiezhong Qiu

    Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu, Neo Y . Zhang, Zhilin Yang, Xinyu Zhou, ...

    2025 arXiv

  38. [46]

    When less is more: Investigating data pruning for pretraining llms at scale, 2023

    Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining llms at scale, 2023. URL https://arxiv.org/abs/ 2309.04564

    2023 arXiv

  39. [47]

    SimPO: Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference optimization with a reference-free reward. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=3Tzcot1LKb

    2024

  40. [48]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, an...

    2022

  41. [49]

    Vicky Zhao, Lili Qiu, and Dongmei Zhang

    Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. LLMLingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. In Lun-Wei...

    2024 doi

  42. [50]

    YaRN: Efficient context window extension of large language models

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context window extension of large language models. InThe Twelfth International Conference on Learning Representations,

  43. [51]

    Handling Very Long Contexts in Neural Machine Translation: a Survey

    Ziqian Peng, Rachel Bawden, and François Yvon. Handling Very Long Contexts in Neural Machine Translation: a Survey. Technical Report Livrable D3-2.1, Projet ANR MaTOS, June 2024. URL https: //inria.hal.science/hal-04652584

    2024

  44. [52]

    Infobatch: Lossless training speed up by unbiased dynamic data pruning

    Ziheng Qin, Kai Wang, Zangwei Zheng, Jianyang Gu, Xiangyu Peng, xu Zhao Pan, Daquan Zhou, Lei Shang, Baigui Sun, Xuansong Xie, and Yang You. Infobatch: Lossless training speed up by unbiased dynamic data pruning. InThe Twelfth International Conference on Learning Representatio...

    2024

  45. [53]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.ArXiv, abs/2305.18290,

    arXiv

  46. [54]

    Zero: Memory optimizations toward training trillion parameter models, 2020

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models, 2020. URLhttps://arxiv.org/abs/1910.02054

    2020 arXiv

  47. [55]

    SQuAD: 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras, editors,Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 238...

    2016 doi

  48. [56]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URLhttps://arxiv.org/abs/2311.12022

    2023 arXiv

  49. [57]

    Neil J. A. Sloane and Aaron D. Wyner. Prediction and entropy of printed english. 1951. URL https: //api.semanticscholar.org/CorpusID:9101213

    1951

  50. [58]

    Le, Ed H

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V . Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them, 2022. URLhttps://arxiv.org/abs/2210.09261

    2022 arXiv

  51. [59]

    Generalized preference optimization: A unified approach to offline alignment

    Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Remi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Avila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. InForty-first International C...

    2024

  52. [60]

    LOGO - long context alignment via efficient preference optimization.CoRR, abs/2410.18533, 2024

    Zecheng Tang, Zechen Sun, Juntao Li, Qiaoming Zhu, and Min Zhang. LOGO - long context alignment via efficient preference optimization.CoRR, abs/2410.18533, 2024. doi: 10.48550/ARXIV .2410.18533. URLhttps://doi.org/10.48550/arXiv.2410.18533

    2024 doi arXiv

  53. [61]

    Musique: Multihop questions via single-hop question composition.Transactions of the Association for Computational Linguistics, 10:539–554, 2022

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition.Transactions of the Association for Computational Linguistics, 10:539–554, 2022. doi: 10.1162/tacl_a_00475. URL https://aclanthology.org/...

    2022 doi

  54. [62]

    Leave no document behind: Benchmarking long-context LLMs with extended multi-doc QA

    Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, and Yongbin Li. Leave no document behind: Benchmarking long-context LLMs with extended multi-doc QA. In Yaser Al-Onaizan, Mohi...

    2024

  55. [63]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language unders...

    2024 arXiv

  56. [64]

    Chi, Quoc V Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors,Advances in ...

    2022

  57. [65]

    Kullback–Leibler divergence

    Wikipedia contributors. Kullback–Leibler divergence. https://en.wikipedia.org/wiki/Kullback% E2%80%93Leibler_divergence#cite_note-Csiszar-1, 2024. Accessed: 2024-06-10

    2024

  58. [66]

    Wit and Marie Gillette

    Ernst C. Wit and Marie Gillette. What is linguistic redundancy. 2013. URL https://api. semanticscholar.org/CorpusID:1425655

    2013

  59. [67]

    An efficient recipe for long context extension via middle- focused positional encoding

    Tong Wu, Yanpeng Zhao, and Zilong Zheng. An efficient recipe for long context extension via middle- focused positional encoding. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URLhttps://openreview.net/forum?id=aNHEqFMS0N

    2024

  60. [68]

    Effective long-context scaling of foundation models

    Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, S...

    2024

  61. [69]

    Large language models for generative information extraction: A survey

    Derong Xu, Wei Chen, Wenjun Peng, Chao Zhang, Tong Xu, Xiangyu Zhao, Xian Wu, Yefeng Zheng, Yang Wang, and Enhong Chen. Large language models for generative information extraction: A survey. Frontiers of Computer Science, 18(6):186357, 2024

    2024

  62. [70]

    RECOMP: Improving retrieval-augmented LMs with con- text compression and selective augmentation

    Fangyuan Xu, Weijia Shi, and Eunsol Choi. RECOMP: Improving retrieval-augmented LMs with con- text compression and selective augmentation. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=mlJLVigNHp

    2024

  63. [71]

    Longfaith: Enhancing long-context reasoning in llms with faithful synthetic data.CoRR, abs/2502.12583,

    Cehao Yang, Xueyuan Lin, Chengjin Xu, Xuhui Jiang, Shengjie Ma, Aofan Liu, Hui Xiong, and Jian Guo. Longfaith: Enhancing long-context reasoning in llms with faithful synthetic data.CoRR, abs/2502.12583,

    arXiv

  64. [72]

    Qwen2.5 technical report.ArXiv, abs/2412.15115, 2024

    Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, ...

    2024 arXiv

  65. [73]

    Mindllm: Lightweight large language model pre-training, evaluation and domain application.AI Open, 5: 1–26, 2024

    Yizhe Yang, Huashan Sun, Jiawei Li, Runheng Liu, Yinghao Li, Yuhang Liu, Yang Gao, and Heyan Huang. Mindllm: Lightweight large language model pre-training, evaluation and domain application.AI Open, 5: 1–26, 2024. URLhttps://api.semanticscholar.org/CorpusID:271818589

    2024

  66. [74]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors,Proce...

    2018 doi

  67. [75]

    Longcite: Enabling LLMs to generate fine-grained citations in long-context QA,

    Jiajie Zhang, Yushi Bai, Xin Lv, Wanjun Gu, Danqing Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, and Juanzi Li. Longcite: Enabling LLMs to generate fine-grained citations in long-context QA,

  68. [76]

    Longreward: Improving long-context large language models with AI feedback.CoRR, abs/2410.21252, 2024

    Jiajie Zhang, Zhongni Hou, Xin Lv, Shulin Cao, Zhenyu Hou, Yilin Niu, Lei Hou, Yuxiao Dong, Ling Feng, and Juanzi Li. Longreward: Improving long-context large language models with AI feedback.CoRR, abs/2410.21252, 2024. doi: 10.48550/ARXIV .2410.21252. URL https://doi.org/10.4...

    2024 doi arXiv

  69. [77]

    Extending llama-3’s context ten-fold overnight.CoRR, abs/2404.19553, 2024

    Peitian Zhang, Ninglu Shao, Zheng Liu, Shitao Xiao, Hongjin Qian, Qiwei Ye, and Zhicheng Dou. Extending llama-3’s context ten-fold overnight.CoRR, abs/2404.19553, 2024. doi: 10.48550/ARXIV . 2404.19553. URLhttps://doi.org/10.48550/arXiv.2404.19553

    2024 doi arXiv

  70. [78]

    Iopo: Empowering llms with complex instruction following via input-output preference optimization, 2024

    Xinghua Zhang, Haiyang Yu, Cheng Fu, Fei Huang, and Yongbin Li. Iopo: Empowering llms with complex instruction following via input-output preference optimization, 2024. URL https://arxiv.org/abs/ 2411.06208. 15

    2024 arXiv

  71. [79]

    LONGA- GENT: Achieving question answering for 128k-token-long documents through multi-agent collaboration

    Jun Zhao, Can Zu, Xu Hao, Yi Lu, Wei He, Yiwen Ding, Tao Gui, Qi Zhang, and Xuanjing Huang. LONGA- GENT: Achieving question answering for 128k-token-long documents through multi-agent collaboration. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of t...

    2024 doi

  72. [80]

    URLhttps://openreview.net/forum?id=mMXdHyBcHh

  73. [81]

    LlamaFactory: Unified effi- cient fine-tuning of 100+ language models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. LlamaFactory: Unified effi- cient fine-tuning of 100+ language models. In Yixin Cao, Yang Feng, and Deyi Xiong, editors,Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics...

    2024 doi

  74. [82]

    Instruction-following evaluation for large language models, 2023

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models, 2023. URL https://arxiv.org/ abs/2311.07911

    2023 arXiv

  75. [83]

    PoSE: Efficient con- text window extension of LLMs via positional skip-wise training

    Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. PoSE: Efficient con- text window extension of LLMs via positional skip-wise training. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=3...

    2024

  76. [84]

    Chain-of-thought matters: Improving long-context language models with reasoning path supervision.CoRR, abs/2502.20790, 2025

    Dawei Zhu, Xiyu Wei, Guangxiang Zhao, Wenhao Wu, Haosheng Zou, Junfeng Ran, Xun Wang, Lin Sun, Xiangzheng Zhang, and Sujian Li. Chain-of-thought matters: Improving long-context language models with reasoning path supervision.CoRR, abs/2502.20790, 2025. doi: 10.48550/ARXIV .250...

    2025 doi arXiv

  77. [85]

    Slic-hf: Sequence likelihood calibration with human feedback.arXiv preprint arXiv:2305.10425, 2023

    Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback.arXiv preprint arXiv:2305.10425, 2023

    2023 arXiv

  78. [87]

    Generalizing from short to long: Effective data synthesis for long-context instruction tuning.ArXiv, abs/2502.15592, 2025

    Wenhao Zhu, Pinzhen Chen, Hanxu Hu, Shujian Huang, Fei Yuan, Jiajun Chen, and Alexandra Birch. Generalizing from short to long: Effective data synthesis for long-context instruction tuning.ArXiv, abs/2502.15592, 2025. URLhttps://api.semanticscholar.org/CorpusID:276557686. 16 A...

    2025 arXiv

  79. [90]

    Generalizing from short to long: Effective data synthesis for long-context instruction tuning.CoRR, abs/2502.15592, February 2025

    Wenhao Zhu, Pinzhen Chen, Hanxu Hu, Shujian Huang, Fei Yuan, Jiajun Chen, and Alexandra Birch. Generalizing from short to long: Effective data synthesis for long-context instruction tuning.CoRR, abs/2502.15592, February 2025. URLhttps://doi.org/10.48550/arXiv.2502.15592

    2025 doi arXiv

  80. [93]

    Identify the owner of The Collegian:

  81. [94]

    The Collegian

    Find the founding year of Houston Baptist University: ... The answer is: 1960 rejected To answer the question, we need to identify which institute "The Collegian" is associated with and then find its founding date. However, the provided passages do not explicitly link The Coll...

    1960

  82. [95]

    In addition, we further examine the theoretical validity of the first motivation from adata-sampling perspective

    We aim to reduce the amount of long-text processing during training, thereby improving training efficiency. In addition, we further examine the theoretical validity of the first motivation from adata-sampling perspective. a. Relation π(y|x short)≥π(y|x long) typically holds ow...

  83. [2022]

    doi: 10.1145/3477495.3532071. Publisher Copyright: © 2022 ACM.; 45th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2022 ; Conference date: 11-07-2022 Through 15-07-2022

    2022

  84. [2023]

    URLhttps://api.semanticscholar.org/CorpusID:258959321

  85. [2024]

    URLhttps://openreview.net/forum?id=wHBfxhZu1u

  86. [2025]

    URLhttps://doi.org/10.48550/arXiv.2502.12583

    doi: 10.48550/ARXIV .2502.12583. URLhttps://doi.org/10.48550/arXiv.2502.12583

    doi arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.