Pith. sign in

REVIEW 14 cited by

LooGLE: Can Long-Context Language Models Understand Long Contexts?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.04939 v2 pith:PRJT5G4D submitted 2023-11-08 cs.CL cs.AI

Jiaqi Li , Mengmeng Wang , Zilong Zheng , Muhan Zhang This is my paper
classification cs.CLcs.AI
keywords llmslongcontextdependencytaskslooglemodelsshort
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLMs), despite their impressive performance in various language tasks, are typically limited to processing texts within context-window size. This limitation has spurred significant research efforts to enhance LLMs' long-context understanding with high-quality long-sequence benchmarks. However, prior datasets in this regard suffer from shortcomings, such as short context length compared to the context window of modern LLMs; outdated documents that have data leakage problems; and an emphasis on short dependency tasks rather than long dependency tasks. In this paper, we present LooGLE, a Long Context Generic Language Evaluation benchmark for LLMs' long context understanding. LooGLE features relatively new documents post-2022, with over 24,000 tokens per document and 6,000 newly generated questions spanning diverse domains. Human annotators meticulously crafted more than 1,100 high-quality question-answer pairs to meet the long dependency requirements. These pairs underwent thorough cross-validation, yielding the most precise assessment of LLMs' long dependency capabilities. The evaluation of eight state-of-the-art LLMs on LooGLE revealed key findings: (i) commercial models outperformed open-sourced models; (ii) LLMs excelled in short dependency tasks like short question-answering and cloze tasks but struggled with more intricate long dependency tasks; (iii) in-context learning and chaining thoughts offered only marginal improvements; (iv) retrieval-based techniques demonstrated substantial benefits for short question-answering, while strategies for extending context window length had limited impact on long context understanding. As such, LooGLE not only provides a systematic and comprehensive evaluation schema on long-context LLMs, but also sheds light on future development of enhanced models towards "true long-context understanding".

Discussion (0). Sign in to comment.

Forward citations

Cited by 14 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. RULER: What's the Real Context Size of Your Long-Context Language Models?

    cs.CL 2024-04 accept novelty 8.0 of 10

    RULER shows most long-context LMs drop sharply in performance on complex tasks as length and difficulty increase, with only half maintaining results at 32K tokens.

  2. Evaluating Very Long-Term Conversational Memory of LLM Agents

    cs.CL 2024-02 unverdicted novelty 8.0 of 10

    Creates LoCoMo benchmark dataset for very long-term LLM conversational memory and shows current models struggle with lengthy dialogues and long-range temporal dynamics.

  3. Positional Failures in Long-Context LLMs: A Blind Spot in Reasoning Benchmarks

    cs.CL 2026-05 conditional novelty 7.0 of 10

    Audits reveal no reasoning benchmark controls position/filler/length jointly; CRE shows LLMs drop up to 88pp on middle-position tasks at 64K context, with diagnostic probe supporting positional cause.

  4. Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving

    cs.OS 2026-05 unverdicted novelty 7.0 of 10

    Tutti is a GPU-direct SSD-backed KV cache that removes CPU bottlenecks via object abstraction, GPU io_uring, and slack scheduling, delivering near-DRAM performance at 2x higher request rate and 27% lower cost than pri...

  5. When Thoughts Meet Facts: Reusable Reasoning for Long-Context LMs

    cs.CL 2025-10 unverdicted novelty 7.0 of 10

    Thought templates derived from training traces and refined via natural-language feedback improve multi-hop reasoning performance in long-context LMs across benchmarks and can be distilled into smaller models.

  6. Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions

    cs.CL 2025-07 unverdicted novelty 7.0 of 10

    MemoryAgentBench is a new multi-turn benchmark assessing four memory competencies in LLM agents—accurate retrieval, test-time learning, long-range understanding, and selective forgetting—showing that existing methods ...

  7. MicroAgent: Context-Augmented Multi-Agent Framework for Automatic Microservice Decomposition

    cs.SE 2026-06 unverdicted novelty 6.0 of 10

    MicroAgent framework assigns five subtasks to specialized agents with multi-granularity context and analytical tools, achieving 89.2% average accuracy on 10 Java applications and beating prior methods by 24.6%.

  8. Mitigating Position Bias in Transformers via Layer-Specific Positional Embedding Scaling

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    LPES uses per-layer scaling factors optimized by a genetic algorithm with Bézier curves to balance attention and improve long-context LLM performance by up to 11.2% on key-value retrieval.

  9. Training and Evaluating Diffusion Policies with Long Context Lengths

    cs.RO 2026-06 conditional novelty 6.0 of 10

    Naive long-context Diffusion Policies succeed with UNet+Cross-Attention and sufficient data; variable-history training cuts sample complexity in the low-data regime.

  10. Three Heads Are Better Than One: A Multi-perspective Reasoning Framework for Enhanced Vulnerability Detection

    cs.SE 2026-05 conditional novelty 6.0 of 10

    ReasonVul deploys three LLM agents with independent analysis and structured debate to achieve 40% PairAcc and 72.52% F1 on PrimeVul, outperforming baselines by 81% in PairAcc.

  11. ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cache

    cs.DC 2026-04 unverdicted novelty 6.0 of 10

    ForkKV uses copy-on-write disaggregated KV cache with DualRadixTree and ResidualAttention kernels to deliver up to 3x throughput over prior multi-LoRA serving systems with negligible quality loss.

  12. vAttention: Verified Sparse Attention

    cs.LG 2025-10 conditional novelty 6.0 of 10

    vAttention is a sparse attention method that mixes heavy-hitter tokens with a statistically sized random sample to provide (ε, δ)-guaranteed approximation of full attention.

  13. Sticker-TTS: Learn to Utilize Historical Experience with a Sticker-driven Test-Time Scaling Framework

    cs.AI 2025-09 conditional novelty 6.0 of 10

    Sticker-TTS uses three collaborating models to distill and reuse compact summaries (stickers) of past reasoning attempts, improving math benchmark accuracy at a claimed equivalent inference cost.

  14. A Distributed Learned Hash Table

    cs.NI 2025-08 conditional novelty 6.0 of 10

    A learned order-preserving hash inside a Chord-style DHT lets range queries finish in roughly the same cost as single-key lookups, cutting latency and messages by 80-90%+ in tests.

Pith tools