REFACT: Adaptive Fact Restatement for Compact and Faithful Chain-of-Thought Reasoning

Chaojun Xiao; Ge Yu; Huiyuan Xie; Maosong Sun; Xin Dai; Yu Gu; Zhenghao Liu; Zhensheng Jin

REVIEW 2 major objections 6 minor 51 references

Large language models can be trained to restate only the source facts a reasoning step needs, making chain-of-thought reasoning shorter and more faithful to the supplied context.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 09:13 UTC pith:2QM4OGYJ

load-bearing objection A useful and honestly ablated training recipe for citation-grounded reasoning, but the unaddressed train/eval overlap between the training tasks and LongBench/LV-Eval variants blocks acceptance until it is resolved. the 2 major comments →

arxiv 2607.20833 v2 pith:2QM4OGYJ submitted 2026-07-23 cs.CL

REFACT: Adaptive Fact Restatement for Compact and Faithful Chain-of-Thought Reasoning

Zhensheng Jin , Xin Dai , Zhenghao Liu , Chaojun Xiao , Huiyuan Xie , Yu Gu , Ge Yu , Maosong Sun This is my paper

classification cs.CL

keywords chain-of-thought reasoningfact restatementcitation groundinglong-context QAfaithfulnesscounterfactual reasoningreinforcement learningGRPO

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

REFACT claims that large language models can be trained to decide, at each reasoning step, whether contextual evidence is required and to restate only the source facts sufficient to support that step and the final answer. Instead of appending citations after the fact or reproducing evidence at fixed positions, the method makes citations into intermediate reasoning states that can take the form of an entity, phrase, sentence, or paragraph fragment. The training pipeline uses a stronger teacher model to generate citation-aware reasoning traces in contexts of varied lengths, filters them for format, answer correctness, traceability, and answerability, then optimizes a student model with supervised fine-tuning followed by reinforcement learning. Across long-context question answering and counterfactual-conflict benchmarks, the paper reports higher accuracy and faithfulness with substantially fewer reasoning tokens and fewer restated facts, meaning denser and more selective evidence. A sympathetic reader would care because the result suggests that faithfulness and compactness of reasoning can be optimized together rather than traded off.

Core claim

The central claim is that citations should be adaptive: a model should cite only when a reasoning step needs factual support, and it should restate the cited evidence at the granularity the local inference requires. REFACT operationalizes this with tagged fact restatements inside the chain of thought and trains a small student model by imitating teacher-generated trajectories, then optimizing a reward that checks whether the cited content alone is enough for a verifier to answer the question. The paper reports that the resulting traces outperform prior grounding recipes on long-context QA benchmarks, reduce reasoning-token counts by large margins, and on counterfactual questions prefer conte

What carries the argument

The load-bearing object is the tagged fact restatement inside the reasoning trace: an <evidence>…</evidence> span, embedded in the sentence flow rather than appended, whose granularity is chosen by the model. A teacher LLM decides when and at what detail to cite; a verifier LLM checks whether the cited content alone can answer the question; the student is then trained in two stages — SFT on filtered trajectories, followed by GRPO-based reinforcement learning with a weighted reward combining format, answer accuracy, traceability to gold supporting facts, and answerability of the cited content. The answerability reward is what pushes the model toward minimal-yet-sufficient evidence, while the

Load-bearing premise

The reported benchmark gains assume that the questions and supporting facts used to build REFACT's training data never appear in the evaluation benchmarks; the paper does not report checking for that overlap.

What would settle it

Compare the questions and gold supporting facts used to construct REFACT's training data against the evaluation questions in LongBench v1 and LV-Eval, including near-duplicate matching at the answer-bearing-fact level. If a substantial fraction of answer facts or full questions appear in both, the F1 gains and token reductions would be explainable by memorization rather than adaptive citation.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

If REFACT is right, long-context QA accuracy and reasoning-token efficiency improve together: the paper reports consistent F1 gains over strong baselines with substantially shorter reasoning traces on both LongBench v1 and LV-Eval.
Counterfactual faithfulness improves: on ConFiQA, REFACT raises source preference and lowers parametric override, and the effect is accompanied by reduced activation in the mid-to-deep feed-forward layers associated with stored knowledge.
Citation quality as measured against gold supporting facts improves while the number of restated facts falls, so evidence density — answer-relevant facts per restatement — is a viable optimization target rather than citation quantity.
Removing either the evidence-alignment signal or the answerability reward degrades performance, so traceability and answer-sufficiency are both necessary parts of the training objective.
Because the length reduction is not directly rewarded, the citation-utility objective itself appears to discourage redundant restatement, making compactness a byproduct of grounding rather than a separate goal.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If answer-sufficiency is the operative reward, the same recipe could transfer to claim-level grounding in multi-document summarization or synthesis, where the verifier would check whether the restated evidence alone supports each claim.
The paper reports token reduction without an explicit length penalty; an editorial extension would be to test whether adding an explicit verbosity or length term compresses reasoning further, or whether it starts to drop necessary evidence.
The correlation between lower mid-to-deep FFN activation and lower parametric override suggests a cheap diagnostic: activation ratios in those layers could predict whether a model will follow the context under conflict, before any answer is generated.
A direct test of the evaluation's hidden assumption — de-duplicating training and evaluation sources — would tell whether the reported gains reflect generalized citation behavior or memorized answers.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A useful and honestly ablated training recipe for citation-grounded reasoning, but the unaddressed train/eval overlap between the training tasks and LongBench/LV-Eval variants blocks acceptance until it is resolved.

read the letter

Worth reading. The core idea is concrete: train a model to restate source facts at the right granularity, only when needed, and then reward answerability of those citations. That combination of adaptive timing, multi-granularity restatement, and an answerability reward in an SFT-to-RL pipeline is new, and the ablations in Table 4 and Table 7 give real evidence that the components matter. Removing the evidence-alignment index or the answerability reward consistently hurts, which is more than most papers in this area show. The ConFiQA counterfactual results are a nice addition, and the FFN analysis is a reasonable supplementary signal rather than a gimmick. The paper is clearly written and honestly reports that LongBench v2 accuracy is flat; the gain there is just token reduction.

The soft spot is the one from the stress-test note: training on HotpotQA_CARE and MuSiQue and evaluating on LongBench/LV-Eval variants of the same underlying datasets without any deduplication or overlap analysis. This is not an internal inconsistency, and the token-reduction results are less vulnerable to it, as are the gains on 2WikiMQA, MFQA, Qasper, and ConFiQA. But the F1 and citation-F1 gains in Tables 1-3 could be inflated if the test questions or their answer-bearing facts appeared in training. The authors need to report overlap statistics and, ideally, performance split by overlap status. Without that, the central claim that REFACT learns adaptive fact restatement rather than memorization is not fully supported.

Minor concerns: code and data are promised but not released, so results cannot be re-run; no error bars in the main tables; the significance markers are terse. Also, the LongBench v2 section should be more explicit that accuracy is not improved.

This paper is for people working on long-context reasoning, citation grounding, and faithful CoT. It deserves a serious referee but not unconditional acceptance. I would send it to review and ask the reviewers to push on the overlap analysis and the artifact release.

Referee Report

2 major / 6 minor

Summary. The paper proposes REFACT, a two-stage SFT-to-GRPO training framework intended to make LLM chain-of-thought reasoning faithfully grounded in provided contexts through adaptive fact restatement. A teacher model generates citation-grounded reasoning trajectories from HotpotQA_CARE and MuSiQue by restating gold supporting facts at varying granularities, and the trajectories are filtered with format, accuracy, traceability, and answerability criteria. The student is then fine-tuned on length-controlled contexts (32k/64k/128k) and optimized with a weighted citation-utility reward. Evaluations on LongBench v1/v2, LV-Eval, and ConFiQA report improved QA accuracy, substantially shorter reasoning traces, higher citation F1 with fewer restated facts, and lower parametric override. Ablations in Table 4 support the contributions of evidence alignment and the answerability reward.

Significance. If the empirical claims hold, the paper makes a useful contribution: it treats citations not as post-hoc attribution but as intermediate reasoning states that are selectively and adaptively restated, and it demonstrates that this can improve long-context QA and counterfactual faithfulness while reducing token cost. The work has several concrete strengths: the ablations in Table 4 isolate the effect of evidence alignment and the answerability reward; the ConFiQA results are internally consistent with the faithfulness narrative; the compactness result is not directly optimized by the reward, which makes the efficiency claim more interesting; and Appendix A.7 provides explicit prompts that aid reproducibility. The main obstacle to accepting the results at face value is the unexamined overlap between the training and evaluation data, which is a correctness-risk concern rather than an internal inconsistency.

major comments (2)

[Section 4 / Tables 1–3] Training data D_training is constructed from HotpotQA_CARE and MuSiQue (Section 3.2.1, Eqs. 5–6), while the main evaluations in Table 1 use LongBench v1 HotpotQA and MuSiQue, and Table 2 uses LV-Eval HotpotWikiQA. No de-duplication, overlap ratio, or disjoint-subset analysis is reported. Because LongBench HotpotQA/MuSiQue and LV-Eval HotpotWikiQA are derived from the same underlying QA datasets, the exact evaluation questions or their answer-bearing paragraphs may appear in D_training — where they are restated verbatim as gold evidence. Even paragraph-level overlap can inflate F1 and reduce #Facts/#Tok in Tables 1–3. Please report question-level and passage-level overlap statistics (exact match, n-gram, or embedding-based) and rerun the main comparisons on the non-overlapping subset.
[Section 3.2.1 / Table 3] Table 3 measures citation F1 against the same annotated supporting facts F_i that are used to construct the teacher trajectories and that define the R_trace reward. Consequently, the high citation F1 with a low number of restated facts may partly reflect that the student was trained to reproduce exactly these gold spans on these datasets, rather than to adaptively select the minimal sufficient evidence. To separate training-set memorization from adaptive citation behavior, the paper needs a held-out evaluation where the supporting facts/questions do not appear in D_training, or a transfer evaluation on an evidence-annotation benchmark not used for training.

minor comments (6)

[Section 5.2 / Table 4] The text refers to 'SFT (Answer-only)' but Table 4 labels this row 'SFT (label)'; please align the notation. Also, the sentence containing 'removes explicit the aignment index' has a typo ('alignment').
[Table 5] The column header 'Singlg-Hop' should read 'Single-Hop'.
[Appendix A.7] The teacher, generation, and verifier prompts are provided as figures. For reproducibility, it would be helpful to include the exact prompt text in a machine-readable format as well.
[Section 5.3] The LLM-as-judge evaluation uses GPT-5.4 but reports no human agreement or calibration analysis. A short validation or at least a caveat about judge bias would strengthen the trace-quality claims.
[Table 6 / Appendix A.5] On LongBench v2, REFACT's average accuracy is 27.6 (4B) and 35.4 (8B), essentially matching Zero-Shot (27.7 and 35.9) while using far fewer tokens. The text accurately says 'competitive', but the abstract's broad statement about 'improving long-context question answering' should be qualified to emphasize the main gains are on LongBench v1/LV-Eval, with LongBench v2 showing efficiency rather than accuracy gains.
[Section 3.2.2] The prompt text includes the phrase 'Follow it with a command a connecting phrase', which is grammatically unclear; please revise.

Circularity Check

0 steps flagged

No significant circularity: REFACT's empirical claims do not reduce to their training inputs by construction.

full rationale

This is an empirical training-and-evaluation paper rather than a formal derivation, so most circularity categories do not apply. The training data are synthesized by a teacher model from gold supporting facts (Eqs. 3-6), and the student is optimized with rewards that reward citing those facts (R_trace, R_ans). The central claims--higher QA F1, shorter traces, higher citation F1, and better counterfactual faithfulness--are supported by comparisons against external benchmarks and baselines, not by an equation that re-introduces the target quantity as an input. The closest candidate for circularity is benchmark overlap: D_training is built from HotpotQA_CARE and MuSiQue, while LongBench v1 contains HotpotQA and MuSiQue tasks and Table 3 computes citation F1 against annotated supporting facts of the same kind used during training. However, the paper does not state that the evaluation instances are the same training instances, and no in-text equation reduces a reported number to a fitted parameter. If overlap exists, it would be train/evaluation contamination, an external-validity concern, not an in-paper circular reduction. Self-citations (e.g., the FFN-layer analysis relying on prior work [16]) are secondary and not load-bearing for the core claim. Under the required standard of quoting a specific reduction, no circular step is identifiable.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The method depends on several empirical assumptions: gold supporting facts are reliable, the teacher and verifier are trustworthy, and the synthetic length-diverse contexts transfer to real long-context QA. The primary free parameters are the reward weights and training hyperparameters, which are not extensively sensitivity-tested. No new theoretical entities are introduced.

free parameters (4)

reward weights lambda_1..lambda_4 = 0.1, 0.5, 0.1, 0.3
Set by hand in Appendix A.2; answer correctness and answerability carry 80% of reward mass. No sensitivity analysis is reported, so the main results could depend on this specific weighting.
target context lengths L = 32k, 64k, 128k
Chosen to match long-context evaluation lengths. No analysis of how the length distribution affects learned citation behavior.
GRPO group size G = 4
Standard choice from DeepSeekMath GRPO; no ablation on group size.
SFT/RL learning rates = SFT: 5e-6 (4B), 1e-6 (8B); RL: 1e-6
Conventional tuning values; no sensitivity study.

axioms (6)

domain assumption Gold supporting facts F_i in HotpotQA_CARE and MuSiQue are correct and sufficient for answering the questions.
The traceability reward requires cited content to appear in F_i and the answerability filter uses these facts as the citation source (Section 3.2.1). If the annotations are noisy or non-minimal, the reward penalizes valid alternative evidence.
domain assumption The teacher model Gemini-3.1-preview generates high-quality adaptive citation trajectories.
All training trajectories are synthesized by the teacher (Section 3.2.2). The quality of the student is bounded by the teacher's ability to cite facts at appropriate granularity and timing.
domain assumption GPT-4o reliably judges whether cited content is sufficient to answer the question.
R_ans retains only trajectories for which 'a fixed verifier model can recover the correct answer using only the cited content' (Section 3.1). No accuracy of the verifier itself is reported.
domain assumption Inserting gold context into unrelated distractor documents at random positions creates realistic long-context training instances.
Length-diverse construction in Section 3.2.1 assumes this mimics real long-context evidence distribution; the paper does not validate that the resulting contexts resemble the eval benchmarks.
domain assumption LLM-as-a-judge (GPT-5.4) scores reasoning-trace quality reliably.
Holistic CoT quality in Section 5.3 depends on the judge rubric in Appendix A.7; no human agreement or adversarial calibration is reported.
domain assumption Reduced FFN activation ratios in mid-to-deep layers indicate reduced reliance on parametric knowledge.
Section 5.4 interprets FFN activations through the framework of knowledge neurons and Parammute; the paper does not independently validate this causal interpretation for the trained models.

pith-pipeline@v1.3.0-alltime-deepseek · 21809 in / 11184 out tokens · 136862 ms · 2026-08-01T09:13:45.050963+00:00 · methodology

0 comments

read the original abstract

Large Language Models (LLMs) increasingly leverage long-form reasoning to solve complex tasks, yet their reasoning processes can deviate from the provided context when evidence is incomplete, noisy, or conflicts with parametric knowledge. Existing grounding approaches either append citations after generation or encourage LLMs to retrieve evidence during reasoning, but they often fail to ensure that cited information is sufficient to support intermediate inferences and final answers. To address this limitation, we propose REFACT, an adaptive fact-restatement citation framework that enables LLMs to determine when contextual grounding is needed and selectively restate source facts at appropriate levels of detail for reliable reasoning. To facilitate adaptive citation during reasoning, REFACT first leverages a teacher LLM to construct high-quality citation-aware reasoning trajectories under diverse context conditions with varying evidence lengths, and then optimizes the student LLM through a two-stage SFT-to-RL framework. Experiments on LongBench, LV-Eval, and ConFiQA demonstrate that REFACT improves long-context question answering and counterfactual faithfulness while substantially reducing the number of reasoning tokens. Further analysis reveals that REFACT achieves higher evidence density by preserving more answer-relevant facts with fewer restatements, producing reasoning traces that are more concise yet better grounded. All code and data will be released via https://github.com/NEUIR/REFACT.

Figures

Figures reproduced from arXiv: 2607.20833 by Chaojun Xiao, Ge Yu, Huiyuan Xie, Maosong Sun, Xin Dai, Yu Gu, Zhenghao Liu, Zhensheng Jin.

**Figure 2.** Figure 2: Overview of Our ReFact Framework. reasoning steps with external knowledge [1, 31, 33, 38]. However, these methods primarily treat citations as mechanisms for attribution or evidence injection, without explicitly optimizing whether the cited content provides sufficient and reasoning-relevant support for each reasoning step. 3 Methodology As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Figure 3: Perplexity Scores of Golden Answer Generation. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

**Figure 4.** Figure 4: CoT Quality Evaluation Results. Without such alignment signals, the model is less capable of generating selective and evidence-grounded citations. More detailed analysis is provided in Appendix A.6. After further applying RL, ReFact achieves additional improvements, showing that reinforcement learning effectively guides the model to optimize citation utility and incorporate relevant evidence into the rea… view at source ↗

**Figure 6.** Figure 6: Case study. faithfulness-oriented generation, we adopt three evaluation metrics: Exact Match (EM), Source Preference (PS), and Parametric Override (PO). PS measures the model’s preference for the answer supported by the provided context, while PO measures the frequency with which the generated response follows the original factual answer or its aliases. A higher PO indicates more frequent reliance on par… view at source ↗

**Figure 7.** Figure 7: Average Output Length on LV-Eval Sub-Tasks (lower [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗

**Figure 8.** Figure 8: Generation Prompt for ReFact with Cited Fact Restatement [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗

**Figure 9.** Figure 9: Verifier Prompt for Answerability Evaluation. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗

**Figure 10.** Figure 10: Teacher Prompt for Adaptive Fact-Restatement Data Construction. [PITH_FULL_IMAGE:figures/full_fig_p015_10.png] view at source ↗

**Figure 11.** Figure 11: LLM-as-a-judge prompt used for holistic reasoning-trace evaluation. [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

51 extracted references · 12 linked inside Pith

[1]

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In International conference on learning representations, Vol. 2024. OpenReview.net, Vienna, Austria, 9112–9141. https://openreview.net/forum?id=hSyW5go0v8

2024
[2]

Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. 2024. Longalign: A recipe for long context alignment of large language models. InFindings of the Association for Computational Linguistics: EMNLP 2024. Association for Computational Linguistics, Miami, Florida, USA, 1376–1395. doi:10.18653/v1/2024.findings-emnlp.74

doi:10.18653/v1/2024.findings-emnlp.74 2024
[3]

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al . 2024. Longbench: A bilingual, multitask benchmark for long context understanding. InProceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers). Association for Computational ...

2024
[4]

Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2025. LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)...

2025
[5]

Baolong Bi, Shaohan Huang, Yiwei Wang, Tianchi Yang, Zihan Zhang, Haizhen Huang, Lingrui Mei, Junfeng Fang, Zehao Li, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, and Shenghua Liu. 2025. Context-dpo: Aligning language models for context-faithfulness. InFindings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguis...

2025
[6]

Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2024. Longlora: Efficient fine-tuning of long-context large lan- guage models. InInternational Conference on Learning Representations, Vol. 2024. OpenReview.net, Vienna, Austria, 8220–8238. https://openreview.net/forum? id=6PmJoRfdaK

2024
[7]

Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei
[8]

Xin Dai, Buqiang Xu, Zhenghao Liu, Yukun Yan, Huiyuan Xie, Xiaoyuan Yi, Shuo Wang, and Ge Yu. 2026. Legal𝛿: Enhancing Legal Reasoning in LLMS via Reinforcement Learning with Chain-Of-Thought Guided Information Gain. In ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, IEEE, Barcelona, Spain, 16912–16916

2026
[9]

Constanza Fierro, Reinald Kim Amplayo, Fantine Huot, Nicola De Cao, Joshua Maynez, Shashi Narayan, and Mirella Lapata. 2024. Learning to plan and generate text with citations. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Bangkok, Thailand, 1139...

2024
[10]

Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Cha- ganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, et al
[11]

Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large lan- guage models to generate text with citations. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computa- tional Linguistics, Singapore, 6465–6488. https://aclanthology.org/2023.emnlp- main.398/

2023
[12]

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Compu- tational Linguistics, Online and Punta Cana, Dominican Republic, 5484–5495. https://arxiv.org/abs/2012.14913

Pith/arXiv arXiv 2021
[13]

Google DeepMind. 2026. Gemini 3.1 Pro. Model card. https://deepmind.google/ models/model-cards/gemini-3-1-pro/

2026
[14]

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qi- hao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv:2501.12948 [cs.CL] https://arxiv.org/abs/2501.12948

Pith/arXiv arXiv 2025
[15]

2024.RULER: What’s the real context size of your long-context language models?arXiv:2404.06654 [cs.CL] https://arxiv.org/abs/2404.06654

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024.RULER: What’s the real context size of your long-context language models?arXiv:2404.06654 [cs.CL] https://arxiv.org/abs/2404.06654

Pith/arXiv arXiv 2024
[16]

Pengcheng Huang, Zhenghao Liu, Yukun Yan, Haiyan Zhao, Xiaoyuan Yi, Hao Chen, Zhiyuan Liu, Maosong Sun, Tong Xiao, Ge Yu, et al. 2026. Parammute: Suppressing knowledge-critical ffns for faithful retrieval-augmented generation. Advances in Neural Information Processing Systems38 (2026), 100378–100410

2026
[17]

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv:2410.21276 [cs.CL] https://arxiv.org/abs/2410.21276

Pith/arXiv arXiv 2024
[18]

Jiajie Jin, Xiaoxi Li, Guanting Dong, Yuyao Zhang, Yutao Zhu, Yongkang Wu, Zhonghua Li, Ye Qi, and Zhicheng Dou. 2025. Hierarchical document refine- ment for long-context retrieval-augmented generation. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguisti...

2025
[19]

2023.Measuring faithfulness in chain-of-thought reasoning

Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jack- son Kernion, et al. 2023.Measuring faithfulness in chain-of-thought reasoning. arXiv:2307.13702 [cs.AI] https://arxiv.org/abs/2307.13702

Pith/arXiv arXiv 2023
[20]

Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts.Transactions of the association for computational linguistics12 (2024), 157–173. https://aclanthology.org/2024.tacl-1.9/ Jin et al

2024
[21]

Siyi Liu, Kishaloy Halder, Zheng Qi, Wei Xiao, Nikolaos Pappas, Phu Mon Htut, Neha Anna John, Yassine Benajiba, and Dan Roth. 2025. Towards long context hallucination detection. InFindings of the Association for Computational Linguis- tics: NAACL 2025. Association for Computational Linguistics, Albuquerque, New Mexico, 7827–7835. https://aclanthology.org/...

2025
[22]

2022.Teaching language models to support answers with verified quotes

Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. 2022.Teaching language models to support answers with verified quotes. arXiv:2203.11147 [cs.CL] https://arxiv.org/abs/2203.11147

Pith/arXiv arXiv 2022
[23]

Chunyi Peng, Zhipeng Xu, Zhenghao Liu, Yishan Li, Yukun Yan, Shuo Wang, Yu Gu, Minghe Yu, Ge Yu, and Maosong Sun. 2026. Mixture-of-Retrieval Experts for Reasoning-Guided Multimodal Knowledge Exploitation. InProceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1440–1450

2026
[24]

2023.Question decomposition improves the faithfulness of model- generated reasoning

Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson Deni- son, Danny Hernandez, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamil˙e Lukoši¯ut˙e, et al. 2023.Question decomposition improves the faithfulness of model- generated reasoning. arXiv:2307.11768 [cs.CL] https://arxiv.org/abs/2307.11768

Pith/arXiv arXiv 2023
[25]

2024.Deepseek- math: Pushing the limits of mathematical reasoning in open language models

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al . 2024.Deepseek- math: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300 [cs.CL] https://arxiv.org/abs/2402.03300

Pith/arXiv arXiv 2024
[26]

Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. Hybridflow: A flexible and efficient rlhf framework. InProceedings of the Twentieth European Conference on Computer Systems. Association for Computing Machinery, Rotterdam, The Netherlands, 1279–1297. https://dl.acm.org/doi/10.1145/3689...

arXiv 2025
[27]

2019.Megatron-lm: Training multi-billion pa- rameter language models using model parallelism

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019.Megatron-lm: Training multi-billion pa- rameter language models using model parallelism. arXiv:1909.08053 [cs.CL] https://arxiv.org/abs/1909.08053

Pith/arXiv arXiv 2019
[28]

John Sweller. 1988. Cognitive load during problem solving: Effects on learning.Cognitive science12, 2 (1988), 257–285. https://doi.org/10.1207/ s15516709cog1202_4

1988
[29]

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal
[30]

2025.Qwenlong- l1: Towards long-context large reasoning models with reinforcement learning

Fanqi Wan, Weizhou Shen, Shengyi Liao, Yingcheng Shi, Chenliang Li, Ziyi Yang, Ji Zhang, Fei Huang, Jingren Zhou, and Ming Yan. 2025.Qwenlong- l1: Towards long-context large reasoning models with reinforcement learning. arXiv:2505.17667 [cs.CL] https://arxiv.org/abs/2505.17667

Pith/arXiv arXiv 2025
[31]

Suyuchen Wang, Jinlin Wang, Xinyu Wang, Shiqi Li, Xiangru Tang, Sirui Hong, Xiao-Wen Chang, Chenglin Wu, and Bang Liu. 2025. Improving context fidelity via native retrieval-augmented reasoning. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Suzhou, China, 21205–21218. ht...

arXiv 2025
[32]

Transactions of the Association for Computational Linguistics10 (2022), 539–554

♪ MuSiQue: Multihop Questions via Single-hop Question Composition. Transactions of the Association for Computational Linguistics10 (2022), 539–554. https://aclanthology.org/2022.tacl-1.31/

2022
[33]

Zihao Wang, Anji Liu, Haowei Lin, Jiaqi Li, Xiaojian Ma, and Yitao Liang. 2024. Rat: Retrieval augmented thoughts elicit context-aware reasoning in long-horizon generation. arXiv:2403.05313 [cs.CL] https://arxiv.org/abs/2403.05313

Pith/arXiv arXiv 2024
[34]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reason- ing in large language models.Advances in neural information processing systems 35 (2022), 24824–24837

2022
[35]

2025.Loongrl: Reinforcement learning for advanced reasoning over long contexts

Siyuan Wang, Gaokai Zhang, Li Lyna Zhang, Ning Shang, Fan Yang, Dongyao Chen, and Mao Yang. 2025.Loongrl: Reinforcement learning for advanced reasoning over long contexts. arXiv:2510.19363 [cs.CL] https://arxiv.org/abs/2510.19363

arXiv 2025
[36]

Zhipeng Xu, Zhenghao Liu, Yukun Yan, Shuo Wang, Shi Yu, Zheni Zeng, Chaojun Xiao, Zhiyuan Liu, Ge Yu, and Chenyan Xiong. 2026. ThinkNote: Enhancing knowledge integration and utilization of large language models via constructivist cognition modeling. InFindings of the Association for Computational Linguistics: EACL 2026. 211–229

2026
[37]

2025.Qwen3 technical report

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025.Qwen3 technical report. arXiv:2505.09388 [cs.CL] https://arxiv.org/abs/2505.09388

Pith/arXiv arXiv 2025
[38]

Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Retrieval meets long context large language models. InIn- ternational Conference on Learning Representations, Vol. 2024. OpenReview.net, Vienna, Austria, 49569–49584. https://openreview.net/forum?id=xw5nxFWMlo

2024
[39]

2024.Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k

Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, et al. 2024.Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k. arXiv:2402.05136 [cs.CL] https://arxiv.org/abs/2402.05136

arXiv 2024
[40]

Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, and Sercan Ö Arık. 2024. Chain of agents: Large language models collaborating on long- context tasks.Advances in Neural Information Processing Systems37 (2024), 132208–132237. https://proceedings.neurips.cc/paper_files/paper/2024/hash/ ee71a4b14ec26710b39ee6be113d7750-Abstract-Conference.html

2024
[41]

Cehao Yang, Xueyuan Lin, Chengjin Xu, Xuhui Jiang, Shengjie Ma, Aofan Liu, Hui Xiong, and Jian Guo. 2025. Longfaith: Enhancing long-context reasoning in llms with faithful synthetic data. InFindings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics, Vienna, Austria, 3236–3256. https://aclanthology.org/20...

2025
[42]

Dawei Zhu, Xiyu Wei, Guangxiang Zhao, Wenhao Wu, Haosheng Zou, Junfeng Ran, Xun Wang, Lin Sun, Xiangzheng Zhang, and Sujian Li. 2025. Chain-of- thought matters: Improving long-context language models with reasoning path supervision. InFindings of the Association for Computational Linguistics: EMNLP

2025
[44]

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information processing systems36 (2023), 46595–46623

2023
[47]

5 = all substantive claims are grounded, relevant, and necessary

factuality: Are the chain's substantive factual claims both grounded in CONTEXT and relevant to deriving the answer? Extra facts never increase factuality; grounded but irrelevant facts lower precision, while unsupported, distorted, or contradictory claims receive stronger penalties. 5 = all substantive claims are grounded, relevant, and necessary. 4 = gr...
[48]

5 = every necessary evidence item and hop is present; the answer is fully derivable

completeness: Can the final answer be derived from the evidence and reasoning provided? Completeness concerns necessary evidence and hops, not CONTEXT coverage; extra facts never improve it. 5 = every necessary evidence item and hop is present; the answer is fully derivable. 4 = the answer is derivable, but one minor connection is underexplained. 3 = a ne...
[49]

5 = direct, concise, well organized, and free of wasted steps

structurality: Is the necessary derivation presented clearly, directly, and efficiently? Penalize repetition, redundant quotations, re-reading loops, irrelevant tangents, abandoned speculation, and unnecessary restatement. 5 = direct, concise, well organized, and free of wasted steps. 4 = mostly efficient, with minor repetition or one small detour. 3 = un...
[50]

Unsupported, fabricated, distorted, or contradictory claims are direct hallucinations

hallucination: Higher scores indicate better control of unsupported and unnecessary factual generation. Unsupported, fabricated, distorted, or contradictory claims are direct hallucinations. Grounded but irrelevant facts are not literal fabrications, but still lower this score because they unnecessarily expand the chain. CONTEXT is the only factual source...
[51]

factuality

coherence: Do the stated steps follow logically without contradiction, circular reasoning, or unjustified inference? 5 = every step follows clearly and logically. 4 = mostly logical, with only a small gap or abrupt transition. 3 = understandable but contains noticeable logical jumps. 2 = major unjustified leaps, circularity, or contradictions. 1 = fundame...
[2022]

InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Knowledge neurons in pretrained transformers. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Dublin, Ireland, 8493–8502. https://aclanthology.org/2022.acl-long.581/

2022
[2023]

InProceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers)

Rarr: Researching and revising what language models say, using language models. InProceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers). Association for Computational Linguis- tics, Toronto, Canada, 16477–16508. https://aclanthology.org/2023.acl-long.910/

2023
[2025]

Knowsley

Association for Computational Linguistics, Suzhou, China, 3197–3211. doi:10.18653/v1/2025.findings-emnlp.170 ReFact: Adaptive Fact Restatement for Compact and Faithful Chain-of-Thought Reasoning factrecall hotpotwikiqa loogle_SD multifieldqa Dataset 0 500 1000 1500 2000 2500 3000 3500Avg Output Length 324 379 340 374 522 315 322 347 2,030 2,900 2,947 2,07...

arXiv 2025

[1] [1]

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In International conference on learning representations, Vol. 2024. OpenReview.net, Vienna, Austria, 9112–9141. https://openreview.net/forum?id=hSyW5go0v8

2024

[2] [2]

Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. 2024. Longalign: A recipe for long context alignment of large language models. InFindings of the Association for Computational Linguistics: EMNLP 2024. Association for Computational Linguistics, Miami, Florida, USA, 1376–1395. doi:10.18653/v1/2024.findings-emnlp.74

doi:10.18653/v1/2024.findings-emnlp.74 2024

[3] [3]

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al . 2024. Longbench: A bilingual, multitask benchmark for long context understanding. InProceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers). Association for Computational ...

2024

[4] [4]

Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2025. LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)...

2025

[5] [5]

Baolong Bi, Shaohan Huang, Yiwei Wang, Tianchi Yang, Zihan Zhang, Haizhen Huang, Lingrui Mei, Junfeng Fang, Zehao Li, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, and Shenghua Liu. 2025. Context-dpo: Aligning language models for context-faithfulness. InFindings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguis...

2025

[6] [6]

Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2024. Longlora: Efficient fine-tuning of long-context large lan- guage models. InInternational Conference on Learning Representations, Vol. 2024. OpenReview.net, Vienna, Austria, 8220–8238. https://openreview.net/forum? id=6PmJoRfdaK

2024

[7] [7]

Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei

[8] [8]

Xin Dai, Buqiang Xu, Zhenghao Liu, Yukun Yan, Huiyuan Xie, Xiaoyuan Yi, Shuo Wang, and Ge Yu. 2026. Legal𝛿: Enhancing Legal Reasoning in LLMS via Reinforcement Learning with Chain-Of-Thought Guided Information Gain. In ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, IEEE, Barcelona, Spain, 16912–16916

2026

[9] [9]

Constanza Fierro, Reinald Kim Amplayo, Fantine Huot, Nicola De Cao, Joshua Maynez, Shashi Narayan, and Mirella Lapata. 2024. Learning to plan and generate text with citations. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Bangkok, Thailand, 1139...

2024

[10] [10]

Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Cha- ganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, et al

[11] [11]

Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large lan- guage models to generate text with citations. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computa- tional Linguistics, Singapore, 6465–6488. https://aclanthology.org/2023.emnlp- main.398/

2023

[12] [12]

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Compu- tational Linguistics, Online and Punta Cana, Dominican Republic, 5484–5495. https://arxiv.org/abs/2012.14913

Pith/arXiv arXiv 2021

[13] [13]

Google DeepMind. 2026. Gemini 3.1 Pro. Model card. https://deepmind.google/ models/model-cards/gemini-3-1-pro/

2026

[14] [14]

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qi- hao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv:2501.12948 [cs.CL] https://arxiv.org/abs/2501.12948

Pith/arXiv arXiv 2025

[15] [15]

2024.RULER: What’s the real context size of your long-context language models?arXiv:2404.06654 [cs.CL] https://arxiv.org/abs/2404.06654

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024.RULER: What’s the real context size of your long-context language models?arXiv:2404.06654 [cs.CL] https://arxiv.org/abs/2404.06654

Pith/arXiv arXiv 2024

[16] [16]

Pengcheng Huang, Zhenghao Liu, Yukun Yan, Haiyan Zhao, Xiaoyuan Yi, Hao Chen, Zhiyuan Liu, Maosong Sun, Tong Xiao, Ge Yu, et al. 2026. Parammute: Suppressing knowledge-critical ffns for faithful retrieval-augmented generation. Advances in Neural Information Processing Systems38 (2026), 100378–100410

2026

[17] [17]

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv:2410.21276 [cs.CL] https://arxiv.org/abs/2410.21276

Pith/arXiv arXiv 2024

[18] [18]

Jiajie Jin, Xiaoxi Li, Guanting Dong, Yuyao Zhang, Yutao Zhu, Yongkang Wu, Zhonghua Li, Ye Qi, and Zhicheng Dou. 2025. Hierarchical document refine- ment for long-context retrieval-augmented generation. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguisti...

2025

[19] [19]

2023.Measuring faithfulness in chain-of-thought reasoning

Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jack- son Kernion, et al. 2023.Measuring faithfulness in chain-of-thought reasoning. arXiv:2307.13702 [cs.AI] https://arxiv.org/abs/2307.13702

Pith/arXiv arXiv 2023

[20] [20]

Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts.Transactions of the association for computational linguistics12 (2024), 157–173. https://aclanthology.org/2024.tacl-1.9/ Jin et al

2024

[21] [21]

Siyi Liu, Kishaloy Halder, Zheng Qi, Wei Xiao, Nikolaos Pappas, Phu Mon Htut, Neha Anna John, Yassine Benajiba, and Dan Roth. 2025. Towards long context hallucination detection. InFindings of the Association for Computational Linguis- tics: NAACL 2025. Association for Computational Linguistics, Albuquerque, New Mexico, 7827–7835. https://aclanthology.org/...

2025

[22] [22]

2022.Teaching language models to support answers with verified quotes

Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. 2022.Teaching language models to support answers with verified quotes. arXiv:2203.11147 [cs.CL] https://arxiv.org/abs/2203.11147

Pith/arXiv arXiv 2022

[23] [23]

Chunyi Peng, Zhipeng Xu, Zhenghao Liu, Yishan Li, Yukun Yan, Shuo Wang, Yu Gu, Minghe Yu, Ge Yu, and Maosong Sun. 2026. Mixture-of-Retrieval Experts for Reasoning-Guided Multimodal Knowledge Exploitation. InProceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1440–1450

2026

[24] [24]

2023.Question decomposition improves the faithfulness of model- generated reasoning

Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson Deni- son, Danny Hernandez, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamil˙e Lukoši¯ut˙e, et al. 2023.Question decomposition improves the faithfulness of model- generated reasoning. arXiv:2307.11768 [cs.CL] https://arxiv.org/abs/2307.11768

Pith/arXiv arXiv 2023

[25] [25]

2024.Deepseek- math: Pushing the limits of mathematical reasoning in open language models

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al . 2024.Deepseek- math: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300 [cs.CL] https://arxiv.org/abs/2402.03300

Pith/arXiv arXiv 2024

[26] [26]

Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. Hybridflow: A flexible and efficient rlhf framework. InProceedings of the Twentieth European Conference on Computer Systems. Association for Computing Machinery, Rotterdam, The Netherlands, 1279–1297. https://dl.acm.org/doi/10.1145/3689...

arXiv 2025

[27] [27]

2019.Megatron-lm: Training multi-billion pa- rameter language models using model parallelism

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019.Megatron-lm: Training multi-billion pa- rameter language models using model parallelism. arXiv:1909.08053 [cs.CL] https://arxiv.org/abs/1909.08053

Pith/arXiv arXiv 2019

[28] [28]

John Sweller. 1988. Cognitive load during problem solving: Effects on learning.Cognitive science12, 2 (1988), 257–285. https://doi.org/10.1207/ s15516709cog1202_4

1988

[29] [29]

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal

[30] [30]

2025.Qwenlong- l1: Towards long-context large reasoning models with reinforcement learning

Fanqi Wan, Weizhou Shen, Shengyi Liao, Yingcheng Shi, Chenliang Li, Ziyi Yang, Ji Zhang, Fei Huang, Jingren Zhou, and Ming Yan. 2025.Qwenlong- l1: Towards long-context large reasoning models with reinforcement learning. arXiv:2505.17667 [cs.CL] https://arxiv.org/abs/2505.17667

Pith/arXiv arXiv 2025

[31] [31]

Suyuchen Wang, Jinlin Wang, Xinyu Wang, Shiqi Li, Xiangru Tang, Sirui Hong, Xiao-Wen Chang, Chenglin Wu, and Bang Liu. 2025. Improving context fidelity via native retrieval-augmented reasoning. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Suzhou, China, 21205–21218. ht...

arXiv 2025

[32] [32]

Transactions of the Association for Computational Linguistics10 (2022), 539–554

♪ MuSiQue: Multihop Questions via Single-hop Question Composition. Transactions of the Association for Computational Linguistics10 (2022), 539–554. https://aclanthology.org/2022.tacl-1.31/

2022

[33] [33]

Zihao Wang, Anji Liu, Haowei Lin, Jiaqi Li, Xiaojian Ma, and Yitao Liang. 2024. Rat: Retrieval augmented thoughts elicit context-aware reasoning in long-horizon generation. arXiv:2403.05313 [cs.CL] https://arxiv.org/abs/2403.05313

Pith/arXiv arXiv 2024

[34] [34]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reason- ing in large language models.Advances in neural information processing systems 35 (2022), 24824–24837

2022

[35] [35]

2025.Loongrl: Reinforcement learning for advanced reasoning over long contexts

Siyuan Wang, Gaokai Zhang, Li Lyna Zhang, Ning Shang, Fan Yang, Dongyao Chen, and Mao Yang. 2025.Loongrl: Reinforcement learning for advanced reasoning over long contexts. arXiv:2510.19363 [cs.CL] https://arxiv.org/abs/2510.19363

arXiv 2025

[36] [36]

Zhipeng Xu, Zhenghao Liu, Yukun Yan, Shuo Wang, Shi Yu, Zheni Zeng, Chaojun Xiao, Zhiyuan Liu, Ge Yu, and Chenyan Xiong. 2026. ThinkNote: Enhancing knowledge integration and utilization of large language models via constructivist cognition modeling. InFindings of the Association for Computational Linguistics: EACL 2026. 211–229

2026

[37] [37]

2025.Qwen3 technical report

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025.Qwen3 technical report. arXiv:2505.09388 [cs.CL] https://arxiv.org/abs/2505.09388

Pith/arXiv arXiv 2025

[38] [38]

Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Retrieval meets long context large language models. InIn- ternational Conference on Learning Representations, Vol. 2024. OpenReview.net, Vienna, Austria, 49569–49584. https://openreview.net/forum?id=xw5nxFWMlo

2024

[39] [39]

2024.Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k

Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, et al. 2024.Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k. arXiv:2402.05136 [cs.CL] https://arxiv.org/abs/2402.05136

arXiv 2024

[40] [40]

Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, and Sercan Ö Arık. 2024. Chain of agents: Large language models collaborating on long- context tasks.Advances in Neural Information Processing Systems37 (2024), 132208–132237. https://proceedings.neurips.cc/paper_files/paper/2024/hash/ ee71a4b14ec26710b39ee6be113d7750-Abstract-Conference.html

2024

[41] [41]

Cehao Yang, Xueyuan Lin, Chengjin Xu, Xuhui Jiang, Shengjie Ma, Aofan Liu, Hui Xiong, and Jian Guo. 2025. Longfaith: Enhancing long-context reasoning in llms with faithful synthetic data. InFindings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics, Vienna, Austria, 3236–3256. https://aclanthology.org/20...

2025

[42] [42]

Dawei Zhu, Xiyu Wei, Guangxiang Zhao, Wenhao Wu, Haosheng Zou, Junfeng Ran, Xun Wang, Lin Sun, Xiangzheng Zhang, and Sujian Li. 2025. Chain-of- thought matters: Improving long-context language models with reasoning path supervision. InFindings of the Association for Computational Linguistics: EMNLP

2025

[43] [44]

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information processing systems36 (2023), 46595–46623

2023

[44] [47]

5 = all substantive claims are grounded, relevant, and necessary

factuality: Are the chain's substantive factual claims both grounded in CONTEXT and relevant to deriving the answer? Extra facts never increase factuality; grounded but irrelevant facts lower precision, while unsupported, distorted, or contradictory claims receive stronger penalties. 5 = all substantive claims are grounded, relevant, and necessary. 4 = gr...

[45] [48]

5 = every necessary evidence item and hop is present; the answer is fully derivable

completeness: Can the final answer be derived from the evidence and reasoning provided? Completeness concerns necessary evidence and hops, not CONTEXT coverage; extra facts never improve it. 5 = every necessary evidence item and hop is present; the answer is fully derivable. 4 = the answer is derivable, but one minor connection is underexplained. 3 = a ne...

[46] [49]

5 = direct, concise, well organized, and free of wasted steps

structurality: Is the necessary derivation presented clearly, directly, and efficiently? Penalize repetition, redundant quotations, re-reading loops, irrelevant tangents, abandoned speculation, and unnecessary restatement. 5 = direct, concise, well organized, and free of wasted steps. 4 = mostly efficient, with minor repetition or one small detour. 3 = un...

[47] [50]

Unsupported, fabricated, distorted, or contradictory claims are direct hallucinations

hallucination: Higher scores indicate better control of unsupported and unnecessary factual generation. Unsupported, fabricated, distorted, or contradictory claims are direct hallucinations. Grounded but irrelevant facts are not literal fabrications, but still lower this score because they unnecessarily expand the chain. CONTEXT is the only factual source...

[48] [51]

factuality

coherence: Do the stated steps follow logically without contradiction, circular reasoning, or unjustified inference? 5 = every step follows clearly and logically. 4 = mostly logical, with only a small gap or abrupt transition. 3 = understandable but contains noticeable logical jumps. 2 = major unjustified leaps, circularity, or contradictions. 1 = fundame...

[49] [2022]

InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Knowledge neurons in pretrained transformers. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Dublin, Ireland, 8493–8502. https://aclanthology.org/2022.acl-long.581/

2022

[50] [2023]

InProceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers)

Rarr: Researching and revising what language models say, using language models. InProceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers). Association for Computational Linguis- tics, Toronto, Canada, 16477–16508. https://aclanthology.org/2023.acl-long.910/

2023

[51] [2025]

Knowsley

Association for Computational Linguistics, Suzhou, China, 3197–3211. doi:10.18653/v1/2025.findings-emnlp.170 ReFact: Adaptive Fact Restatement for Compact and Faithful Chain-of-Thought Reasoning factrecall hotpotwikiqa loogle_SD multifieldqa Dataset 0 500 1000 1500 2000 2500 3000 3500Avg Output Length 324 379 340 374 522 315 322 347 2,030 2,900 2,947 2,07...

arXiv 2025