From General to Targeted Rewards: Surpassing GPT-4 in Open-Ended Long-Context Generation

Irwin King; Jiele Wu; Minda Hu; Wenqian Cui; Yifei Zhang; Yufei Wang; Zhihan Guo

REVIEW 5 major objections 7 minor 65 references

From General to Targeted Rewards: Surpassing GPT-4 in Open-Ended Long-Context Generation

T0 review · 5 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that open-ended long-text generation can be trained without gold-standard answers by rewarding models on auto-generated proxy question-answer checklists, and that the resulting small open-source models surpass…

desk verdict Useful training-reward idea with consistent gains on ProxyQA, but the 'surpass GPT-4-Turbo' headline is a single unquantified percentage point with shared-evaluator bias and no human check. read the letter →

arxiv 2506.16024 v1 pith:J74QOVOA submitted 2025-06-19 cs.CL cs.AI

Zhihan Guo , Jiele Wu , Wenqian Cui , Yifei Zhang , Minda Hu , Yufei Wang , Irwin King This is my paper

classification cs.CLcs.AI

keywords ProxyRewardopen-endedlong-textgenerationsignaldirectpreferenceoptimizationquestion-answerpairsLLM-as-a-Judgelong-contextQA

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Open-ended long-text generation has no gold-standard answers to train against, and holistic quality judgments are too coarse to serve as rewards. This paper claims that both problems can be solved at once by having an LLM generate, for each question, a checklist of boolean proxy question-answer pairs, then using the fraction of those questions that a reward model can answer from a candidate response as the training reward. It reports that applying DPO with this signal lifts Qwen and Llama instruct models by about 20% relative on the ProxyQA benchmark and lets Qwen2.5-7B-Instruct beat GPT-4-Turbo. The significance, if the claim holds, is a scalable recipe for training informative long-form generation without expensive human annotation.

What carries the argument

The central object is the (meta-question, proxy question-answer pairs) structure, where each meta-question is paired with roughly fifteen boolean questions that function as an objective checklist of key information points. The reward signal is $S(r_i) = \frac{1}{l} \sum_{j=1}^{l} F(a'_{ij}, \hat{a}_{ij})$, the fraction of proxy questions a reward LLM answers correctly after reading the response, with $F$ returning 1 only on an exact match. This converts subjective long-form quality evaluation into a reading-comprehension task that plays to the strengths of long-context LLMs, and it makes Direct Preference Optimization applicable because the reward is defined entirely by the checklist rather than by a separately trained reward model.

What would settle it

Construct two sets of responses to the same meta-questions: one padded with irrelevant text that happens to contain verbatim statements matching the proxy questions, and one concise but genuinely informative. If the padded responses receive systematically higher ProxyReward training scores and higher ProxyQA accuracy, the reward is rewarding surface compliance rather than information quality.

Watch

Extended reading notes

Core claim

The paper's central claim is that targeted reward signals can replace gold-standard references in open-ended long-text generation. It introduces ProxyReward, which contains a dataset of 9,271 meta-questions across 40 domains with 156,506 automatically generated boolean proxy question-answer pairs, and a reward computation that scores a candidate response by whether a reward LLM, here GPT-4o-mini, can correctly answer those proxy questions after reading the response. The score $S(r_i)$ is the fraction of proxy questions answered correctly, and preference pairs are formed by selecting the highest- and lowest-scoring responses for the same meta-question. Training with DPO on these pairs is reported to improve ProxyQA accuracy consistently across model scales and architectures, with Qwen2.5-7B-Instruct reaching 35.07% versus GPT-4-Turbo's 33.94%.

Load-bearing premise

The entire method rests on the assumption that the auto-generated boolean proxy questions, answered by another LLM from the response text, form a valid and unbiased checklist of what a good long-form answer must contain.

Editorial extensions

If this is right

Open-source instruct models trained with ProxyReward improve ProxyQA accuracy by roughly 20% relative to their base versions across both Qwen and Llama families.
Qwen2.5-7B-Instruct reaches 35.07% on ProxyQA, surpassing GPT-4-Turbo's 33.94% despite a smaller parameter count.
The targeted reward beats LLM-as-a-Judge by 9.22 to 11.85 absolute points on Qwen models and by 1.59 to 5.09 points on Llama models.
A second DPO iteration continues to raise accuracy for most models, moving Qwen2.5-7B-Instruct from 33.23% to 35.07%.
Accuracy-based rewards outperform precision-based rewards in the ablation, and the optimal number of iterations depends on model size, with the 1.5B model peaking at the first iteration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural stress test is to regenerate the proxy questions at inference time; if the accuracy gains vanish when the checklist changes, the reward may be teaching models to match a fixed set of questions rather than to be broadly informative.
The comparison with GPT-4-Turbo depends on ProxyQA, which is itself an LLM-evaluated benchmark; pairing ProxyReward training with human preference judgments on a sample of outputs would test whether the 20% gain reflects genuine information quality.
Because the reward model is a strong API model, an open question is whether the framework retains its advantage when the reward model is a smaller open-weight model, which would change the cost and accessibility profile.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Read this one for the idea, not for the headline. ProxyReward turns the ProxyQA evaluation trick—answering reading-comprehension-style questions about a long-form response—into a training reward. That's a natural extension, and to my knowledge it hasn't been done as cleanly before. The dataset generation is fully automatic, and the DPO training with a reward computed from proxy-question accuracy is well specified. The consistent gains across Qwen and Llama models are the real contribution: every trained model improves over its base, and the gap over the LLM-as-a-Judge baseline is large and in the same direction.

Now the soft spots. The central claim that a 7B model "surpasses GPT-4-Turbo" rests on a 1.13-point difference (35.07 vs 33.94) with no error bars or significance testing. For a claim that aggressive, that's a whisper, not a shout. The comparison is also incomplete: GPT-4o (44.98) and DeepSeek-V3 (42.93) are far ahead, so the "surpassing GPT-4" framing depends on which GPT-4 you mean. The title says GPT-4; Table 3 shows GPT-4-Turbo. That's a real overreach.

The bigger methodological concern is shared evaluator bias. Training rewards come from GPT-4o-mini answering proxy questions; evaluation uses GPT-4o on ProxyQA. Both are OpenAI models, and the proxy-QA format is the same. A model that learns to satisfy GPT-4o-mini's boolean checks may also look good to GPT-4o, without genuinely better long-form content. The paper's own Limitations section admits the proxy QA pairs carry model biases, but doesn't test how much of the gain survives a human judge. That's the key missing experiment. Also, the "20% improvement" figure is cherry-picked: it's the relative gain for the 1.5B and 8B models, not the 7B model, and relative gains on an 18-to-22 scale are not the same as on a 32-to-35 scale.

There's also no release of code or data, which hurts reproducibility. The data selection rules (variance-based filtering, preference construction) are post hoc and could overfit the benchmark.

Still, the core idea deserves a serious look, and the paper is written honestly—limitations are acknowledged, not buried. I'd send it to a reviewer who knows RLHF and long-context generation, with a request to focus on (a) error bars, (b) human evaluation, and (c) whether the gain generalizes beyond the proxy-QA metric. If those hold up, this is a useful methodological contribution.

Recommendation: accept for review, expect major revision, and require the authors to release data/code before acceptance.

Referee Report

5 major / 7 minor

Summary. The paper introduces ProxyReward, a reinforcement-learning framework for open-ended long-context generation. It automatically constructs a dataset of meta-questions and associated proxy question-answer pairs, uses an LLM (GPT-4o-mini) to score model responses against these proxy questions, and applies DPO to align open-source models (Qwen and Llama) with the resulting reward signal. The authors report consistent relative improvements over base models on the ProxyQA benchmark (approximately 20% for several configurations) and claim that Qwen2.5-7B-Instruct surpasses GPT-4-Turbo with an accuracy of 35.07% vs 33.94%.

Significance. The idea of converting open-ended long-form quality evaluation into a checklist of boolean proxy questions is an intuitive, cheap, and scalable alternative to gold-standard reference texts, and the proposed pipeline is clear and easy to follow. The consistent improvements across six different Qwen and Llama checkpoints, as well as the comparison to an LLM-as-a-Judge baseline, provide some initial support for the method. However, the headline claim rests on a small margin on a single benchmark whose evaluation mechanism shares a strong structural resemblance with the training reward signal, and no human evaluation or significance testing is reported. These issues make the central claim fragile as it stands.

major comments (5)

[§5.4, Table 3] The headline claim that Qwen2.5-7B-Instruct surpasses GPT-4-Turbo relies on a 1.13 percentage-point difference (35.07 vs 33.94) reported without error bars, multiple evaluation runs, or any significance test. Because the evaluation uses GPT-4o as the judge, which is stochastic in practice, a single evaluation is insufficient to support a ranking. The authors should report variability (e.g., across several evaluation runs or under different temperatures) and ideally add human evaluation on a subset of ProxyQA items to substantiate the 'surpassing' claim.
[§4.3, §5.1, §8] The training reward and the ProxyQA evaluation metric both use the same proxy-question-answer checklist format, with GPT-4o-mini as the training reward model and GPT-4o as the evaluation judge. This shared paradigm creates a real risk of reward overfitting: DPO may optimize the model to satisfy the specific boolean-answer behavior of GPT-4o-mini, which could artificially align with GPT-4o's judgments. The Limitations section (Section 8) explicitly concedes that reliance on LLM-generated proxy question-answer pairs may affect the objectivity and coverage of the reward signals. The paper should provide additional evidence that the ProxyQA gains are not an artifact of this shared evaluator bias—for example, by evaluating trained models with a different judge (including non-LLM metrics or human raters) or on a different long-context generation benchmark.
[§6.2, Table 4; §5.1 Data Selection] The ablation study reports a 'Precision' variant of ProxyReward but never defines it, making the comparison to 'Accuracy' in Table 4 impossible to interpret or reproduce. Similarly, the data selection procedure (preferring high-variance meta-questions and filtering conflicting preference indicators) is described only verbally; Table 5 provides summary statistics but not the actual criteria or thresholds used. The authors should give precise definitions (formulas or exact selection rules) and, if possible, release code or a detailed pseudo-code for dataset construction and selection.
[§5.3, Table 2] The LLM-as-a-Judge baselines are consistently far below the base models (e.g., Qwen2.5-7B-Instruct drops from 32.37 to 25.73; Llama-3.1-8B-Instruct drops from 25.02 to 23.43). The paper attributes this to the generality of judge-based rewards, but an equally plausible explanation is that the DPO hyperparameters (learning rate 5e-7, five epochs) are not appropriate for the judge-reward distribution, making the comparison unfair. The authors should verify that the LLM-as-a-Judge baseline is not being trained with ill-suited hyperparameters or should at least state that a separate hyperparameter search for this baseline was performed.
[§5.4 and §7] The broad claim of 'surpassing GPT-4-Turbo in Open-LTG' is supported by performance on a single benchmark (ProxyQA). The paper's own Related Work lists several other long-context generation benchmarks (e.g., LongBench v2, LongWriter, LongGenBench), but none are used for evaluation. A single benchmark is insufficient to establish a broad capability improvement; at minimum, the authors should add at least one additional long-form generation evaluation and report qualitative characteristics of the improved responses (e.g., length, factual correctness, coherence) to make the claim robust.

minor comments (7)

[Title/Abstract] The title and abstract claim 'surpassing GPT-4', but all experiments compare against GPT-4-Turbo; please align the wording for consistency.
[§3.2] The text says 'As shown in Table 1' when referring to the synthetic preference alignment pipeline, but the actual content is an algorithm block; fix the cross-reference.
[§4.1 and Eq. (4)] Section 4.1 says all proxy answers are boolean values, while the prompt in Appendix A allows 'True', 'False', and 'Not Mentioned'; the reward function in Eq. (4)–(5) actually implements a three-way match. Please align the terminology.
[§5.2] Section 5.2 uses Iteration 1 accuracy (22.10) for Qwen2.5-1.5B when describing a 20.8% relative improvement, while Table 3 later uses Iteration 2 values for all models; please report results consistently.
[§5.1] The maximum completion length is set to 2048 tokens, which seems short for 'long-text generation'; please justify this setting or specify the typical response lengths used in training and evaluation.
[Appendix A] The prompt contains a typo: 'Your are required' should be 'You are required'.
[Algorithm 1] In Algorithm 1, line 2 uses the loop variable w both as an iteration index and as a partition index; rename to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the training reward is an internal proxy-QA score, while the headline result is measured on the external, human-curated ProxyQA benchmark.

full rationale

ProxyReward's reward signal is defined by Eqs. (4)-(5) as the fraction of automatically generated proxy questions that the reward model GPT-4o-mini answers correctly for a generated response, and DPO training optimizes this internal score. The paper's headline claim, however, is assessed on ProxyQA (Tan et al., 2024), a benchmark whose meta-questions, proxy questions, and pre-annotated answers are human-curated and external to the ProxyReward training set. Section 5.1 explicitly distinguishes the reward model ('we employ GPT-4o-mini to compute training rewards') from the evaluation judge ('we utilize GPT-4o to determine the ProxyQA score'). Thus the reported 35.07 vs. 33.94 comparison is not the training reward evaluated on the training data; it is an external measure that is not directly optimized by the reward function. The shared proxy-question-answer paradigm is a plausible generalization or shared-evaluator-bias concern, and the Limitations section honestly concedes that 'reliance on LLM-generated proxy question-answer pairs introduces potential biases and errors inherent to the underlying models,' but this is a validity risk rather than a logical circularity. No equation, fitted parameter, or self-citation makes the evaluation coincide with the training objective by construction. Therefore, under the hard rule requiring an exhibited reduction, no significant circularity is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The method introduces no fitted numerical constants or new physical entities. It relies on the assumption that AI-generated checklists and AI judgments provide a trustworthy reward signal, an assumption the authors themselves flag in the Limitations section. The data selection rules are a paper-specific design choice that affects the training distribution.

assumptions (3)

domain assumption LLM-generated proxy question-answer pairs are valid and unbiased proxies for the information content of long-form responses.
The entire training reward is based on these pairs. The authors acknowledge in Section 8 that LLM-generated proxy pairs 'introduce potential biases and errors inherent to the underlying models, which may affect the objectivity and coverage of the reward signals.'
domain assumption The reward model (GPT-4o-mini) accurately answers proxy questions when given the generated response as the document.
Equation (4) defines the reward as the proportion of correctly answered proxy questions, and Section 4.3 assumes the reward model's reading comprehension is reliable enough to judge informativeness.
ad hoc to paper The post hoc data selection rules (preferring high-variance meta-questions and filtering conflicting preference indicators) produce a training distribution that generalizes to the human-curated ProxyQA benchmark.
Section 5.1 introduces these selection rules without any analysis showing they do not introduce selection bias or that they match the distribution of real open-ended queries.

how reviews work

0 comments

Cite this review

Pith. "Pith review of From General to Targeted Rewards: Surpassing GPT-4 in Open-Ended Long-Context Generation." pith.science (2026). https://pith.science/paper/J74QOVOA

@misc{pith2026250616024,
  author       = {Pith},
  title        = {Pith review of: From General to Targeted Rewards: Surpassing GPT-4 in Open-Ended Long-Context Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J74QOVOA}},
  note         = {Machine review of arXiv:2506.16024}
}

read the original abstract

Current research on long-form context in Large Language Models (LLMs) primarily focuses on the understanding of long-contexts, the Open-ended Long Text Generation (Open-LTG) remains insufficiently explored. Training a long-context generation model requires curation of gold standard reference data, which is typically nonexistent for informative Open-LTG tasks. However, previous methods only utilize general assessments as reward signals, which limits accuracy. To bridge this gap, we introduce ProxyReward, an innovative reinforcement learning (RL) based framework, which includes a dataset and a reward signal computation method. Firstly, ProxyReward Dataset generation is accomplished through simple prompts that enables the model to create automatically, obviating extensive labeled data or significant manual effort. Secondly, ProxyReward Signal offers a targeted evaluation of information comprehensiveness and accuracy for specific questions. The experimental results indicate that our method ProxyReward surpasses even GPT-4-Turbo. It can significantly enhance performance by 20% on the Open-LTG task when training widely used open-source models, while also surpassing the LLM-as-a-Judge approach. Our work presents effective methods to enhance the ability of LLMs to address complex open-ended questions posed by human.

Figures

Figures reproduced from arXiv: 2506.16024 by the authors.

**Figure 2.** ProxyReward overview. Our framework operates in three stages: First, ProxyReward automatically [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** The pipeline of meta-questions and Proxy [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Effects of model size and iteration on Prox [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

65 extracted references · 27 canonical work pages

[1]

Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. 2024 a . Longalign: A recipe for long context alignment of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1376--1395

work page 2024
[2]

Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, and 1 others. 2024 b . Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412.15204

arXiv 2024
[3]

Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, and 1 others. 2023. Benchmarking foundation models with language-model-as-an-examiner. Advances in Neural Information Processing Systems, 36:78142--78167

work page 2023
[4]

Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024 c . Longwriter: Unleashing 10,000+ word generation from long context llms. arXiv preprint arXiv:2408.07055

arXiv 2024
[5]

Bing. 2023. Ai-powered bing with chatgpt’s gpt-4. Language model

work page 2023
[6]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

2020
[7]

Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. 2022. Recurrent memory transformer. Advances in Neural Information Processing Systems, 35:11079--11091

2022
[8]

Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and 1 others. 2024. Alpagasus: Training a better alpaca with fewer data. In The Twelfth International Conference on Learning Representations

2024

Show all 65 references

[9]

Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595

2023 arXiv
[10]

Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. 2023. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing System...

2023
[11]

Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, Andr \'e FT Martins, Graham Neubig, Ankush Garg, Jonathan H Clark, Markus Freitag, and Orhan Firat. 2023. The devil is in the errors: Leveraging large language models for fine-grained machine translation evalua...

2023
[12]

Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. 2024. Data engineering for scaling language models to 128k context. In International Conference on Machine Learning

2024
[13]

Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. 2023. Human-like summarization evaluation with chatgpt. arXiv preprint arXiv:2304.02554

2023 arXiv
[14]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

2024 arXiv
[15]

Albert Gu, Karan Goel, and Christopher R \'e . 2021. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396

2021 arXiv
[16]

Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, and 1 others. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594

2024 arXiv
[17]

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

2025 arXiv
[18]

Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, and 1 others. 2024. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. arXiv preprint arXiv:2407.02490

2024 arXiv
[19]

Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516

2025 arXiv
[20]

o ksal, Timo Schick, Anna Korhonen, and Hinrich Sch \

Abdullatif K \"o ksal, Timo Schick, Anna Korhonen, and Hinrich Sch \"u tze. 2023. Longform: Effective instruction tuning with reverse instructions. arXiv preprint arXiv:2304.08460

2023 arXiv
[21]

Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. 2023. Longeval: Guidelines for human evaluation of faithfulness in long-form summarization. In Proceedings of the 17th Conference of the European Chapter of the Association for ...

2023
[22]

Ishita Kumar, Snigdha Viswanathan, Sushrita Yerra, Alireza Salemi, Ryan A Rossi, Franck Dernoncourt, Hanieh Deilamsalehy, Xiang Chen, Ruiyi Zhang, Shubham Agarwal, and 1 others. 2024. Longlamp: A benchmark for personalized long-form text generation. arXiv preprint arXiv:2407.11016

2024 arXiv
[23]

Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, and Abhinav Rastogi. 2024. https://openreview.net/forum?id=AAxIs3D2ZZ RLAIF : Scaling reinforcement learning from human feedback with AI feedback

2024
[24]

Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale N Fung, Mohammad Shoeybi, and Bryan Catanzaro. 2022. Factuality enhanced language models for open-ended text generation. Advances in Neural Information Processing Systems, 35:34586--34599

2022
[25]

Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. 2024 a . Loogle: Can long-context language models understand long contexts? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16304--16333

2024
[26]

Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. 2024 b . Long-context llms struggle with long in-context learning. arXiv preprint arXiv:2404.02060

2024 arXiv
[27]

Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason E Weston, and Mike Lewis. 2024 c . Self-alignment with instruction backtranslation. In The Twelfth International Conference on Learning Representations

2024
[28]

Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. 2023. Contrastive decoding: Open-ended text generation as optimization. In The 61st Annual Meeting Of The Association For Computational Linguistics

2023
[29]

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

2004
[30]

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024 a . Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

2024 arXiv
[31]

Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, and 1 others. 2025. A comprehensive survey on long context language modeling. arXiv preprint arXiv:2503.17407

2025
[32]

Xiang Liu, Peijie Dong, Xuming Hu, and Xiaowen Chu. 2024 b . Longgenbench: Long-context generation benchmark. arXiv preprint arXiv:2410.04199

2024 arXiv
[33]

Xuezhe Ma, Xiaomeng Yang, Wenhan Xiong, Beidi Chen, Lili Yu, Hao Zhang, Jonathan May, Luke Zettlemoyer, Omer Levy, and Chunting Zhou. 2024. Megalodon: Efficient llm pretraining and inference with unlimited context length. Advances in Neural Information Processing Systems, 37:7...

2024
[34]

Nicolo Micheletti, Samuel Belkadi, Lifeng Han, and Goran Nenadic. 2024. Exploration of masked and causal language modelling for text generation. arXiv preprint arXiv:2405.12630

2024 arXiv
[35]

OpenAI. 2024 a . Gpt-4o mini. https://www.openai.com/. Language model

2024
[36]

OpenAI. 2024 b . Hello gpt-4o. https://openai.com/index/hello-gpt-4o/. Language model

2024
[37]

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing sys...

2022
[38]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

2002
[39]

Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024. Yarn: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations

2024
[40]

Chau Minh Pham, Simeng Sun, and Mohit Iyyer. 2024. Suri: Multi-constraint instruction following for long-form text generation. arXiv preprint arXiv:2406.19371

2024 arXiv
[41]

Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher R \'e . 2023. Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning, pages 28043--28078. PMLR

2023
[42]

Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, and 1 others. 2024. Hellobench: Evaluating long text generation capabilities of large language models. arXiv preprint arXiv:2409.16191

2024 arXiv
[43]

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728--53741

2023
[44]

Natalie Schluter. 2017. The limits of automatic summarisation according to rouge. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 41--45. Association for Computational Linguistics

2017
[45]

Lin Shi, Chiyu Ma, Wenhua Liang, Weicheng Ma, and Soroush Vosoughi. 2024. Judging the judges: A systematic investigation of position bias in pairwise comparative assessments by llms. arXiv preprint arXiv:2406.07791

2024
[46]

Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. 2024. https://arxiv.org/abs/2306.17492 Preference ranking optimization for human alignment . Preprint, arXiv:2306.17492

2024 arXiv
[47]

Shyam Sudhakaran, Miguel Gonz \'a lez-Duque, Matthias Freiberger, Claire Glanois, Elias Najarro, and Sebastian Risi. 2023. Mariogpt: Open-ended text2level generation through large language models. Advances in Neural Information Processing Systems, 36:54213--54227

2023
[48]

Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, Yunlong Feng, Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, and 1 others. 2024. Proxyqa: An alternative framework for evaluating long-form text generation with large language models. arXiv preprint arXiv:2401.15042

2024 arXiv
[49]

Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2020. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006

2020 arXiv
[50]

Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2022. Efficient transformers: A survey. ACM Computing Surveys, 55(6):1--28

2022
[51]

Qwen Team. 2024. Qwen2. 5: A party of foundation models, september 2024. URL https://qwenlm. github. io/blog/qwen2, 5

2024
[52]

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

2023 arXiv
[53]

Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. 2024. Self-taught evaluators. arXiv preprint arXiv:2408.02666

2024 arXiv
[54]

Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. 2024 a . Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge. arXiv preprint arXiv:2407.19594

2024 arXiv
[55]

Tsung-Han Wu, Giscard Biamby, Jerome Quenum, Ritwik Gupta, Joseph E Gonzalez, Trevor Darrell, and David M Chan. 2024 b . Visual haystacks: A vision-centric needle-in-a-haystack benchmark. arXiv preprint arXiv:2407.13766

2024 arXiv
[56]

Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, and 1 others. 2023. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309.16039

2023 arXiv
[57]

Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, and 1 others. 2024. Effective long-context scaling of foundation models. In Proceedings of the 2024 Conference of the North Am...

2024
[58]

Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. 2023. A critical evaluation of evaluations for long-form question answering. In The 61st Annual Meeting Of The Association For Computational Linguistics

2023
[59]

Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. 2024. Rlcd: Reinforcement learning from contrastive distillation for lm alignment. In The Twelfth International Conference on Learning Representations

2024
[60]

Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. Proceedings of the 41th International Conference on Machine Learning

2024
[61]

Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302

2023 arXiv
[62]

Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and 1 others. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283--17297

2020
[63]

Jiajie Zhang, Zhongni Hou, Xin Lv, Shulin Cao, Zhenyu Hou, Yilin Niu, Lei Hou, Yuxiao Dong, Ling Feng, and Juanzi Li. 2024. Longreward: Improving long-context large language models with ai feedback. arXiv preprint arXiv:2410.21252

2024 arXiv
[64]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[65]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. 2024 a . Longalign: A recipe for long context alignment of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1376--1395

work page 2024

[2] [2]

Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, and 1 others. 2024 b . Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412.15204

arXiv 2024

[3] [3]

Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, and 1 others. 2023. Benchmarking foundation models with language-model-as-an-examiner. Advances in Neural Information Processing Systems, 36:78142--78167

work page 2023

[4] [4]

Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024 c . Longwriter: Unleashing 10,000+ word generation from long context llms. arXiv preprint arXiv:2408.07055

arXiv 2024

[5] [5]

Bing. 2023. Ai-powered bing with chatgpt’s gpt-4. Language model

work page 2023

[6] [6]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

2020

[7] [7]

Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. 2022. Recurrent memory transformer. Advances in Neural Information Processing Systems, 35:11079--11091

2022

[8] [8]

Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and 1 others. 2024. Alpagasus: Training a better alpaca with fewer data. In The Twelfth International Conference on Learning Representations

2024