Pith. sign in

REVIEW 27 cited by

LooGLE: Can Long-Context Language Models Understand Long Contexts?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.04939 v2 pith:PRJT5G4D submitted 2023-11-08 cs.CL cs.AI

Jiaqi Li , Mengmeng Wang , Zilong Zheng , Muhan Zhang This is my paper · ORCID
classification cs.CLcs.AI
keywords llmslongcontextdependencytaskslooglemodelsshort
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLMs), despite their impressive performance in various language tasks, are typically limited to processing texts within context-window size. This limitation has spurred significant research efforts to enhance LLMs' long-context understanding with high-quality long-sequence benchmarks. However, prior datasets in this regard suffer from shortcomings, such as short context length compared to the context window of modern LLMs; outdated documents that have data leakage problems; and an emphasis on short dependency tasks rather than long dependency tasks. In this paper, we present LooGLE, a Long Context Generic Language Evaluation benchmark for LLMs' long context understanding. LooGLE features relatively new documents post-2022, with over 24,000 tokens per document and 6,000 newly generated questions spanning diverse domains. Human annotators meticulously crafted more than 1,100 high-quality question-answer pairs to meet the long dependency requirements. These pairs underwent thorough cross-validation, yielding the most precise assessment of LLMs' long dependency capabilities. The evaluation of eight state-of-the-art LLMs on LooGLE revealed key findings: (i) commercial models outperformed open-sourced models; (ii) LLMs excelled in short dependency tasks like short question-answering and cloze tasks but struggled with more intricate long dependency tasks; (iii) in-context learning and chaining thoughts offered only marginal improvements; (iv) retrieval-based techniques demonstrated substantial benefits for short question-answering, while strategies for extending context window length had limited impact on long context understanding. As such, LooGLE not only provides a systematic and comprehensive evaluation schema on long-context LLMs, but also sheds light on future development of enhanced models towards "true long-context understanding".

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 27 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions

    cs.CL 2025-07 unverdicted novelty 7.0 of 10

    MemoryAgentBench is a new multi-turn benchmark assessing four memory competencies in LLM agents—accurate retrieval, test-time learning, long-range understanding, and selective forgetting—showing that existing methods ...

  2. Marconi: Prefix Caching for the Era of Hybrid LLMs

    cs.DC 2024-11 conditional novelty 7.0 of 10

    A prefix cache for hybrid attention-SSM LLMs that selectively checkpoints SSM states and evicts by FLOP efficiency, improving token hit rates and time-to-first-token.

  3. Training and Evaluating Diffusion Policies with Long Context Lengths

    cs.RO 2026-06 conditional novelty 6.0 of 10

    Naive long-context Diffusion Policies succeed with UNet+Cross-Attention and sufficient data; variable-history training cuts sample complexity in the low-data regime.

  4. vAttention: Verified Sparse Attention

    cs.LG 2025-10 conditional novelty 6.0 of 10

    vAttention is a sparse attention method that mixes heavy-hitter tokens with a statistically sized random sample to provide (ε, δ)-guaranteed approximation of full attention.

  5. Sticker-TTS: Learn to Utilize Historical Experience with a Sticker-driven Test-Time Scaling Framework

    cs.AI 2025-09 conditional novelty 6.0 of 10

    Sticker-TTS uses three collaborating models to distill and reuse compact summaries (stickers) of past reasoning attempts, improving math benchmark accuracy at a claimed equivalent inference cost.

  6. A Distributed Learned Hash Table

    cs.NI 2025-08 conditional novelty 6.0 of 10

    A learned order-preserving hash inside a Chord-style DHT lets range queries finish in roughly the same cost as single-key lookups, cutting latency and messages by 80-90%+ in tests.

  7. Exploring the Potential of LLMs for Serendipity Evaluation in Recommender Systems

    cs.IR 2025-07 conditional novelty 6.0 of 10

    Basic and multi-model LLM prompts can evaluate recommendation serendipity as well as or better than standard proxy formulas, reaching 21.5% Pearson correlation with user-study ratings.

  8. CROP: Circuit Retrieval and Optimization with Parameter Guidance using LLMs

    cs.LG 2025-07 conditional novelty 6.0 of 10

    LLM-based retrieval-augmented tuning of EDA flow parameters found a 9.9% lower-power configuration on one industrial core than classical optimizers.

  9. Minimizing False Positives in Static Bug Detection via LLM-Enhanced Path Feasibility Analysis

    cs.SE 2025-06 conditional novelty 6.0 of 10

    LLM4PFA combines iterative LLM-based constraint extraction with Z3 solving to reduce false positives in static analysis, achieving 72-96% false positive reduction and 93% recall on a new benchmark.

  10. Hierarchical Tree Search-based User Lifelong Behavior Modeling on Large Language Model

    cs.IR 2025-05 conditional novelty 6.0 of 10

    HiT-LBM chunks lifelong user behavior, uses process-rating-guided tree search to select interest summaries, and fuses them temporally to improve recommendation performance.

  11. NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contexts

    cs.CL 2025-05 conditional novelty 6.0 of 10

    NovelHopQA is a new benchmark that pairs long novel excerpts with 1-4 hop questions and shows LLM accuracy drops consistently with both context length and reasoning depth.

  12. RoToR: Towards More Reliable Responses for Order-Invariant Inputs

    cs.CL 2025-02 conditional novelty 6.0 of 10

    RoToR makes a frozen LLM order-invariant by circularly rotating a single global sort of segment position IDs, and Selective Routing combines it with the original model for mixed lists.

  13. Task-KV: Task-aware KV Cache Optimization via Semantic Differentiation of Attention Heads

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Task-KV identifies 'heterogeneous' attention heads by distance from a per-task semantic center and allocates differentiated KV cache budgets, achieving modest average gains on long-context benchmarks at reduced memory.

  14. Locality-aware Fair Scheduling in LLM Serving

    cs.DC 2025-01 conditional novelty 6.0 of 10

    A deficit-round-robin scheduler layered on longest-prefix-match batching gives LLM serving both prefix-cache locality and bounded fairness between clients.

  15. Systematic Evaluation of Long-Context LLMs on Financial Concepts

    cs.CL 2024-12 conditional novelty 6.0 of 10

    GPT-4o and GPT-4-Turbo degrade significantly on financial news retrieval as context length grows, with performance collapsing on multi-concept tasks beyond 32K tokens.

  16. LIFT: Improving Long Context Understanding Through Long Input Fine-Tuning

    cs.CL 2024-12 conditional novelty 6.0 of 10

    LIFT adapts a short-context LLM by fine-tuning it on overlapping chunks of the test input, improving some long-context benchmarks when combined with in-context learning but degrading others.

  17. KBAlign: Efficient Self Adaptation on Specific Knowledge Bases

    cs.CL 2024-11 conditional novelty 6.0 of 10

    KBAlign is a self-supervised method that generates multi-grained QA pairs from a small text knowledge base and iteratively self-verifies to adapt a RAG model, reaching about 90% of GPT-4-supervised gains on LooGLE F1.

  18. MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    MiniLongBench, a 237-sample compression of LongBench, is claimed to reproduce model rankings with a 0.97 Spearman correlation at 4.5% of the evaluation cost.

  19. 100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?

    cs.CL 2025-05 conditional novelty 5.0 of 10

    100-LongBench and LongScore evaluate LLMs at controlled context lengths and rank models by relative performance drop from a short-context baseline, not by raw accuracy.

  20. The Eye of Sherlock Holmes: Uncovering User Private Attribute Profiling via Vision-Language Model Agentic Framework

    cs.CV 2025-05 conditional novelty 5.0 of 10

    A VLM-LLM agentic pipeline and a new 251-person benchmark show that ordinary personal photo sets can reveal private attributes, including abstract traits like income and MBTI, at rates above human evaluators.

  21. CoDec: Prefix-Shared Decoding Kernel for LLMs

    cs.LG 2025-05 conditional novelty 5.0 of 10

    CoDec combines KV-cache reads across requests that share a prefix, yielding average 1.9x decode-attention speedup and 120.9x less global memory traffic versus FlashDecoding.

  22. Dynamic Chunking and Selection for Reading Comprehension of Ultra-Long Context in Large Language Models

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Dynamic chunking plus question-aware chunk selection improves long-context QA, but the headline numbers are partly inflated by choosing hyperparameters on the test benchmarks.

  23. A Survey on Large Language Model Acceleration based on KV Cache Management

    cs.AI 2024-12 conditional novelty 4.0 of 10

    A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.

  24. XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference

    cs.LG 2024-12 conditional novelty 4.0 of 10

    XKV gives each network layer a personalized KV cache budget based on attention retention, cutting memory 61.6% with comparable accuracy.

  25. A Survey of RWKV

    cs.CL 2024-12 conditional novelty 3.0 of 10

    A review of the RWKV architecture, its versions, applications, benchmarks, and open-source ecosystem; it presents no new experimental results.

  26. An Effective Framework to Help Large Language Models Handle Numeric-involved Long-context Tasks

    cs.CL 2024-11 conditional novelty 3.0 of 10

    A judge-extract-code-conclude workflow beats plain prompting on numeric long-context tasks and cuts API cost, but underperforms CoT on one of two benchmarks.

  27. Structured Memory Mechanisms for Stable Context Representation in Large Language Models

    cs.CL 2025-05 reject novelty 2.0 of 10

    A gated memory module with attention-based reading and forgetting is reported to improve NarrativeQA and dialogue consistency over GPT-2, BART, Longformer, and RETRO.

Pith tools