Pith. sign in

REVIEW 13 cited by

Make Your LLM Fully Utilize the Context

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.16811 v2 pith:NBEJD5XJ submitted 2024-04-25 cs.CL cs.AI

Shengnan An , Zexiong Ma , Zeqi Lin , Nanning Zheng , Jian-Guang Lou This is my paper · ORCID
classification cs.CLcs.AI
keywords contextinformationfilm-7blongtaskstraininglong-contextprobing
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

While many contemporary large language models (LLMs) can process lengthy input, they still struggle to fully utilize information within the long context, known as the lost-in-the-middle challenge. We hypothesize that it stems from insufficient explicit supervision during the long-context training, which fails to emphasize that any position in a long context can hold crucial information. Based on this intuition, our study presents information-intensive (IN2) training, a purely data-driven solution to overcome lost-in-the-middle. Specifically, IN2 training leverages a synthesized long-context question-answer dataset, where the answer requires (1) fine-grained information awareness on a short segment (~128 tokens) within a synthesized long context (4K-32K tokens), and (2) the integration and reasoning of information from two or more short segments. Through applying this information-intensive training on Mistral-7B, we present FILM-7B (FILl-in-the-Middle). To thoroughly assess the ability of FILM-7B for utilizing long contexts, we design three probing tasks that encompass various context styles (document, code, and structured-data context) and information retrieval patterns (forward, backward, and bi-directional retrieval). The probing results demonstrate that FILM-7B can robustly retrieve information from different positions in its 32K context window. Beyond these probing tasks, FILM-7B significantly improves the performance on real-world long-context tasks (e.g., 23.5->26.9 F1 score on NarrativeQA), while maintaining a comparable performance on short-context tasks (e.g., 59.3->59.2 accuracy on MMLU). Github Link: https://github.com/microsoft/FILM.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LLM4MEA: Data-free Model Extraction Attacks on Sequential Recommenders via Large Language Models

    cs.IR 2025-07 conditional novelty 6.0 of 10

    An LLM-driven agent generates synthetic interaction sequences that, when queried against a target sequential recommender, produce surrogate models with higher agreement to the target than random or autoregressive data...

  2. Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Positional Contrastive Decoding, a training-free method that contrasts standard and over-rotated RoPE logits, improves long-context retrieval and QA by a few points.

  3. LCIRC: A Recurrent Compression Approach for Efficient Long-form Context and Query Dependent Modeling in LLMs

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A frozen LLM can process very long contexts by recurrently compressing them with a Perceiver and injecting the compressed memory through gated cross-attention, with query-dependent compression boosting QA performance.

  4. Large Language Models for In-File Vulnerability Localization Can Be "Lost in the End"

    cs.SE 2025-02 conditional novelty 6.0 of 10

    Large language models detect in-file vulnerabilities best when the vulnerable code appears early in the file, a 'lost-in-the-end' effect, and chunking files into smaller blocks can increase recall.

  5. Bootstrap Your Own Context Length

    cs.CL 2024-12 conditional novelty 6.0 of 10

    The authors synthesize long-context training data from short chunks and fine-tune Llama-3 models to reach 1M-token contexts with top open-source RULER scores.

  6. Dehallucinating Parallel Context Extension for Retrieval-Augmented Generation

    cs.CL 2024-12 conditional novelty 5.0 of 10

    DePaC combines context-aware negative training with information-calibrated aggregation to reduce fact fabrication and fact omission in parallel-context RAG systems.

  7. CORD: Balancing COnsistency and Rank Distillation for Robust Retrieval-Augmented Generation

    cs.CL 2024-12 conditional novelty 5.0 of 10

    CORD improves RAG fine-tuning by adding consistency distillation between original and adaptively perturbed passage orders, balancing position robustness with retriever rank information.

  8. Lost in the Middle, and In-Between: Enhancing Language Models' Ability to Reason Over Long Contexts in Multi-Hop QA

    cs.CL 2024-12 conditional novelty 5.0 of 10

    In multi-hop QA, language models answer worse when the required evidence documents are separated by distractor documents, not just when the evidence sits in the middle of the context.

  9. Large Language Models Can Self-Improve in Long-context Reasoning

    cs.CL 2024-11 conditional novelty 5.0 of 10

    LLMs can improve their own long-context reasoning by fine-tuning on outputs selected by Minimum Bayes Risk consensus, without external annotations.

  10. Personality Prediction from Life Stories using Language Models

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A two-step model, sliding-window fine-tuned RoBERTa plus an attention GRU, predicts Big Five scores from life-narrative interviews with higher R2 than several transformer baselines.

  11. Parallel Key-Value Cache Fusion for Position Invariant RAG

    cs.AI 2025-01 conditional novelty 4.0 of 10

    KV-Fusion encodes each retrieved passage in parallel with identical local positions, fuses the key-value caches, and fine-tunes a decoder so answers do not depend on passage order.

  12. LLMQuoter: Enhancing RAG Capabilities Through Efficient Quote Extraction From Large Contexts

    cs.CL 2025-01 reject novelty 4.0 of 10

    LLMQuoter uses a distilled 3B model to extract quotes for RAG; the paper shows gold quotes greatly improve QA, but does not test its own model's quotes end-to-end.

  13. Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directions

    cs.CL 2025-06 conditional novelty 3.0 of 10

    LLM robustness research is organized into adversarial robustness, out-of-distribution robustness, and evaluation, with an accompanying GitHub collection of papers.

Pith tools