Better Retrieval May Not Lead to Better Question Answering

Ashish Sabharwal; Mihai Surdeanu; Steven Bethard; Tushar Khot; Zhengzhong Liang

Better Retrieval May Not Lead to Better Question Answering

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2205.03685 v1 pith:3P475OIB submitted 2022-05-07 cs.CL

Zhengzhong Liang , Tushar Khot , Steven Bethard , Mihai Surdeanu , Ashish Sabharwal This is my paper · ORCID

classification cs.CL

keywords systemansweringapproachbettercontextimproveopen-domainperformance

verification ladder T0 review T1 audit T2 compute T3 formal

0 comments

read the original abstract

Considerable progress has been made recently in open-domain question answering (QA) problems, which require Information Retrieval (IR) and Reading Comprehension (RC). A popular approach to improve the system's performance is to improve the quality of the retrieved context from the IR stage. In this work we show that for StrategyQA, a challenging open-domain QA dataset that requires multi-hop reasoning, this common approach is surprisingly ineffective -- improving the quality of the retrieved context hardly improves the system's performance. We further analyze the system's behavior to identify potential reasons.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Teaching Smaller Language Models To Generalise To Unseen Compositional Questions (Full Thesis)
cs.CL 2024-11 conditional novelty 7.0 of 10

Smaller language models can generalize to unseen compositional questions when trained and evaluated with retrieval-augmented contexts, and combining Wikipedia retrieval with LLM-generated rationales improves accuracy.

Pith tools