REVIEW 12 cited by
Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Retrieval-augmented language models (RALMs) represent a substantial advancement in the capabilities of large language models, notably in reducing factual hallucination by leveraging external knowledge sources. However, the reliability of the retrieved information is not always guaranteed. The retrieval of irrelevant data can lead to misguided responses, and potentially causing the model to overlook its inherent knowledge, even when it possesses adequate information to address the query. Moreover, standard RALMs often struggle to assess whether they possess adequate knowledge, both intrinsic and retrieved, to provide an accurate answer. In situations where knowledge is lacking, these systems should ideally respond with "unknown" when the answer is unattainable. In response to these challenges, we introduces Chain-of-Noting (CoN), a novel approach aimed at improving the robustness of RALMs in facing noisy, irrelevant documents and in handling unknown scenarios. The core idea of CoN is to generate sequential reading notes for retrieved documents, enabling a thorough evaluation of their relevance to the given question and integrating this information to formulate the final answer. We employed ChatGPT to create training data for CoN, which was subsequently trained on an LLaMa-2 7B model. Our experiments across four open-domain QA benchmarks show that RALMs equipped with CoN significantly outperform standard RALMs. Notably, CoN achieves an average improvement of +7.9 in EM score given entirely noisy retrieved documents and +10.5 in rejection rates for real-time questions that fall outside the pre-training knowledge scope.
Forward citations
Cited by 12 Pith papers
-
PORTS: Preference-Optimized Retrievers for Tool Selection with Large Language Models
PORTS fine-tunes dense retrievers for LLM tool selection via odds-ratio preference optimization driven by a frozen LLM's log-likelihood of the gold tool call, plus contrastive semantic loss.
-
A-TMA: Decoupling State-Aware Memory Failures in Long-Term Agent Memory
Explicit bank, retrieval, and QA state roles reduce ghost-memory failures on conflict-heavy LTP and improve some temporal scores on LoCoMo, with host-dependent gains.
-
Exploratory Retrieval-Augmented Planning For Continual Embodied Instruction Following
ExRAP couples LLM planning with a temporal knowledge-graph memory and information-based exploration, improving success and efficiency for continual embodied instruction following.
-
Measuring and Augmenting Large Language Models for Solving Capture-the-Flag Challenges
A benchmark and agent for CTF solving, but the agent's retrieval database appears to contain the answers to the test challenges, undermining the reported improvements.
-
Learn to Think: Bootstrapping LLM Reasoning Capability Through Graph Representation Learning
L2T adapts LLM reasoning step by step by graph-labeling thoughts and using a reinforcement-learned GNN to tune prompts and sampling settings.
-
On Memory Construction and Retrieval for Personalized Conversational Agents
SeCom builds conversation memory from LLM-derived topical segments and compresses units with LLMLingua-2 before retrieval, outperforming turn-level, session-level, and summarization baselines on long-term dialogue benchmarks.
-
K-COMP: Retrieval-Augmented Medical Domain Question Answering With Knowledge-Injected Compressor
K-COMP generates entity definitions and a compressed summary from retrieved medical passages, improving retrieval-augmented QA over baseline compressors on MedQuAD, MASH-QA, and BioASQ.
-
Auto-RAG: Autonomous Retrieval-Augmented Generation for Large Language Models
Auto-RAG fine-tunes an LLM on synthesized reasoning traces so it autonomously decides when to retrieve, what to query, and when to stop.
-
Improving Multimodal LLMs Ability In Geometry Problem Solving, Reasoning, And Multistep Scoring
GPSM4K is a new geometry QA dataset with step-by-step solutions; the paper reports that fine-tuning on it improves LVLM scores, with captioning and RAG giving smaller gains.
-
Understanding the Fundamental Design Decisions of Retrieval-Augmented Generation Systems
RAG deployment must be context-aware: QA benefits from 5 to 10 retrieved documents, code generation has no stable optimal document count, and prompting helps code tasks far more than QA tasks.
-
Exploring Reasoning-Infused Text Embedding with Large Language Models for Zero-Shot Dense Retrieval
Reasoning-infused text embedding, which prepends LLM-generated reasoning to queries before embedding, improves zero-shot dense retrieval on BRIGHT.
-
Review-Then-Refine: A Dynamic Framework for Multi-Hop Question Answering with Temporal Adaptability
A two-phase RAG pipeline with dynamic temporal query rewriting and adaptive retrieval improves multi-hop QA accuracy over several tested baselines on four benchmarks.
Discussion (0). Continue with ORCID to comment.