Pith. sign in

REVIEW 9 cited by

ALR$^2$: A Retrieve-then-Reason Framework for Long-context Question Answering

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.03227 v1 pith:XVRURTDE submitted 2024-10-04 cs.CL

Huayang Li , Pat Verga , Priyanka Sen , Bowen Yang , Vijay Viswanathan , Patrick Lewis , Taro Watanabe , Yixuan Su This is my paper · ORCID
classification cs.CL
keywords llmslong-contextreasoningcontextreasonrelevantaccuratelyanswering
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
abstract

The context window of large language models (LLMs) has been extended significantly in recent years. However, while the context length that the LLM can process has grown, the capability of the model to accurately reason over that context degrades noticeably. This occurs because modern LLMs often become overwhelmed by the vast amount of information in the context; when answering questions, the model must identify and reason over relevant evidence sparsely distributed throughout the text. To alleviate the challenge of long-context reasoning, we develop a retrieve-then-reason framework, enabling LLMs to reason over relevant evidence collected during an intermediate retrieval step. We find that modern LLMs struggle to accurately retrieve relevant facts and instead, often hallucinate "retrieved facts", resulting in flawed reasoning and the production of incorrect answers. To address these issues, we introduce ALR$^2$, a method that augments the long-context reasoning capability of LLMs via an explicit two-stage procedure, i.e., aligning LLMs with the objectives of both retrieval and reasoning. We demonstrate the efficacy of ALR$^2$ for mitigating performance degradation in long-context reasoning tasks. Through extensive experiments on long-context QA benchmarks, we find our method to outperform competitive baselines by large margins, achieving at least 8.4 and 7.9 EM gains on the long-context versions of HotpotQA and SQuAD datasets, respectively.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Sense and Sensitivity: Examining the Influence of Semantic Recall on Long Context Code Understanding

    cs.CL 2025-05 conditional novelty 7.0 of 10

    Models keep verbatim code retrieval stable across positions but lose the ability to reason about centrally placed code, and standard benchmarks understate this because they allow pattern matching.

  2. LongReasonArena: A Long Reasoning Benchmark for Large Language Models

    cs.CL 2025-08 conditional novelty 6.0 of 10

    A benchmark where AI models must run long algorithmic computations reveals a linear drop in accuracy as the log of reasoning steps increases, with top models scoring 7-16% on the hardest level.

  3. Positional Biases Shift as Inputs Approach Context Window Limits

    cs.CL 2025-08 conditional novelty 6.0 of 10

    The lost-in-the-middle effect in LLMs appears mainly when inputs fill up to half the model's context window; beyond that, accuracy favors information closest to the end.

  4. SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments

    cs.SE 2025-07 conditional novelty 6.0 of 10

    SetupBench, a 93-instance environment-bootstrap benchmark, finds coding agents succeed on only 34.4-62.4% of setup tasks, with database configuration and repo setup being the hardest.

  5. CAFE: Retrieval Head-based Coarse-to-Fine Information Seeking to Enhance Multi-Document QA Capability

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A two-stage coarse-to-fine attention-based filtering and steering method, CAFE, improves multi-document QA accuracy across three LLMs and five benchmarks without additional training.

  6. Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models

    cs.CL 2025-01 conditional novelty 5.0 of 10

    A new benchmark with hard confounders shows long-context LMs struggle at in-context retrieval, and a retrieve-then-generate fine-tune plus attention-probing decoding substantially improves them.

  7. o1-Coder: an o1 Replication for Coding

    cs.SE 2024-11 conditional novelty 5.0 of 10

    O1-CODER is an early-stage open-source recipe for o1-style code reasoning, with a test-case generator reaching 89.2% pass rate and pseudocode search improving reasoning-path success but not final Pass@1.

  8. Large Language Models Can Self-Improve in Long-context Reasoning

    cs.CL 2024-11 conditional novelty 5.0 of 10

    LLMs can improve their own long-context reasoning by fine-tuning on outputs selected by Minimum Bayes Risk consensus, without external annotations.

  9. OpenRFT: Adapting Reasoning Foundation Model for Domain-specific Tasks with Reinforcement Fine-Tuning

    cs.AI 2024-12 conditional novelty 4.0 of 10

    OpenRFT adapts a reasoning foundation model to eight scientific tasks with 100 samples each via data augmentation, self-distilled reasoning SFT, and PPO with a process reward model, achieving 0.447 vs 0.403 average accuracy.

Pith tools