REVIEW 7 cited by
Evaluating Open-Domain Question Answering in the Era of Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Lexical matching remains the de facto evaluation method for open-domain question answering (QA). Unfortunately, lexical matching fails completely when a plausible candidate answer does not appear in the list of gold answers, which is increasingly the case as we shift from extractive to generative models. The recent success of large language models (LLMs) for QA aggravates lexical matching failures since candidate answers become longer, thereby making matching with the gold answers even more challenging. Without accurate evaluation, the true progress in open-domain QA remains unknown. In this paper, we conduct a thorough analysis of various open-domain QA models, including LLMs, by manually evaluating their answers on a subset of NQ-open, a popular benchmark. Our assessments reveal that while the true performance of all models is significantly underestimated, the performance of the InstructGPT (zero-shot) LLM increases by nearly +60%, making it on par with existing top models, and the InstructGPT (few-shot) model actually achieves a new state-of-the-art on NQ-open. We also find that more than 50% of lexical matching failures are attributed to semantically equivalent answers. We further demonstrate that regex matching ranks QA models consistent with human judgments, although still suffering from unnecessary strictness. Finally, we demonstrate that automated evaluation models are a reasonable surrogate for lexical matching in some circumstances, but not for long-form answers generated by LLMs. The automated models struggle in detecting hallucinations in LLM answers and are thus unable to evaluate LLMs. At this time, there appears to be no substitute for human evaluation.
Forward citations
Cited by 7 Pith papers
-
OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inference
Perturbation-based OBD saliency scores for KV cache eviction beat attention-only scoring on LLaMA/Qwen long-context benchmarks.
-
HedraRAG: Coordinating LLM Generation and Database Retrieval in Heterogeneous RAG Serving
HedraRAG uses a graph abstraction and dynamic transformations to pipeline generation and retrieval stages, achieving 1.5x to 5x speedups in heterogeneous RAG serving.
-
Mitigating Object Hallucination via Robust Local Perception Search
A training-free decoding method that uses an MLLM's own local object descriptions as a reward prior, combined with CLIP similarity, to cut object hallucination, especially under adversarial image noise.
-
Investigating Symbolic Triggers of Hallucination in Gemma Models Across HaluEval and TruthfulQA
Symbolic triggers like modifiers and named entities keep Gemma hallucination rates at 64-79% across model scales, suggesting larger models do not eliminate this failure mode.
-
Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injection
ROSI bakes the refusal direction into a model's weight matrices via a rank-one update, raising refusal and jailbreak robustness with minimal measured utility cost.
-
Overview of the Sensemaking Task at the ELOQUENT 2025 Lab: LLMs as Teachers, Students and Evaluators
In the ELOQUENT 2025 Sensemaking task, LLM-based evaluators rated clearly garbled or mismatched question-answer pairs as acceptable, showing that LLM-as-a-Judge scores cannot be trusted.
-
LLMs are Introvert
A psychology-inspired prompting method (SIP-CoT with emotion-guided memory) makes LLM agents reproduce human-like attitudes and behaviors more closely in social simulations, but the evaluation lacks error bars, a name...
Discussion (0). Sign in to comment.