REVIEW 8 cited by
Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large Language Models (LLMs) exhibit positional bias, struggling to utilize information from the middle or end of long contexts. Our study explores LLMs' long-context reasoning by probing their hidden representations. We find that while LLMs encode the position of target information, they often fail to leverage this in generating accurate responses. This reveals a disconnect between information retrieval and utilization, a "know but don't tell" phenomenon. We further analyze the relationship between extraction time and final accuracy, offering insights into the underlying mechanics of transformer models.
Forward citations
Cited by 8 Pith papers
-
Controlling the Risk of Corrupted Contexts for Language Models via Early-Exiting
An early-exit rule with a zero-shot fallback, calibrated by Learn-then-Test risk control, keeps the average loss from corrupted in-context demonstrations under a preset bound.
-
MemoCoder: Automated Function Synthesis using LLM-Supported Agents
A multi-agent LLM framework with a memory of past code fixes (MemoCoder) improves Pass@10 and Pass@50 on MBPP, HumanEval, and LiveCodeBench over zero-shot and self-repair baselines by 3 to 14 percentage points.
-
Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding
Positional Contrastive Decoding, a training-free method that contrasts standard and over-rotated RoPE logits, improves long-context retrieval and QA by a few points.
-
Enhancing Visual Reliance in Text Generation: A Bayesian Perspective on Mitigating Hallucination in Large Vision-Language Models
EVRB is a three-part inference-time method that prunes ambiguous visual tokens, divides the model's output distribution by a text-only prior, and triggers early stopping to reduce hallucination in LVLMs.
-
CAIM: Development and Evaluation of a Cognitive AI Memory Framework for Long-Term Interaction with Intelligent Agents
CAIM, a cognitive-AI-inspired memory framework with ontology-based tagging and relevance filtering, improves retrieval and response correctness for LLM assistants on the Generated Virtual Dataset compared with MemoryB...
-
Extracting Visual Facts from Intermediate Layers for Mitigating Hallucinations in Multimodal Large Language Models
Selecting the intermediate layer where image-conditioned and text-only predictions diverge most, and adding that layer's contrastive visual signal back to the final logits, reduces object hallucinations in four large ...
-
QBD-RankedDataGen: Generating Custom Ranked Datasets for Improving Query-By-Document Search Using LLM-Reranking with Reduced Human Effort
LLM-generated rankings for query-by-document search do not improve BM25 tuning over default parameters unless validated by human ground truth.
-
Reconstructing Context: Evaluating Advanced Chunking Strategies for Retrieval-Augmented Generation
A comparative evaluation on NFCorpus and MSMarco shows contextual retrieval beats late chunking on retrieval metrics at higher computational cost, and neither consistently beats standard early chunking.
Discussion (0). Continue with ORCID to comment.