REVIEW 6 cited by
MoreHopQA: More Than Multi-hop Reasoning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Most existing multi-hop datasets are extractive answer datasets, where the answers to the questions can be extracted directly from the provided context. This often leads models to use heuristics or shortcuts instead of performing true multi-hop reasoning. In this paper, we propose a new multi-hop dataset, MoreHopQA, which shifts from extractive to generative answers. Our dataset is created by utilizing three existing multi-hop datasets: HotpotQA, 2WikiMultihopQA, and MuSiQue. Instead of relying solely on factual reasoning, we enhance the existing multi-hop questions by adding another layer of questioning that involves one, two, or all three of the following types of reasoning: commonsense, arithmetic, and symbolic. Our dataset is created through a semi-automated process, resulting in a dataset with 1,118 samples that have undergone human verification. We then use our dataset to evaluate five different large language models: Mistral 7B, Gemma 7B, Llama 3 (8B and 70B), and GPT-4. We also design various cases to analyze the reasoning steps in the question-answering process. Our results show that models perform well on initial multi-hop questions but struggle with our extended questions, indicating that our dataset is more challenging than previous ones. Our analysis of question decomposition reveals that although models can correctly answer questions, only a portion - 38.7% for GPT-4 and 33.4% for Llama3-70B - achieve perfect reasoning, where all corresponding sub-questions are answered correctly. Evaluation code and data are available at https://github.com/Alab-NII/morehopqa
Forward citations
Cited by 6 Pith papers
-
AgenticRAGTracer: A Hop-Aware Benchmark for Diagnosing Multi-Step Retrieval Reasoning in Agentic RAG
AgenticRAGTracer is a hop-level benchmark showing that top LLMs fail on long multi-hop agentic RAG due to collapsed or over-extended reasoning chains.
-
MDBench: A Synthetic Multi-Document Reasoning Benchmark Generated with Knowledge Guidance
MDBench is a synthetically generated, knowledge-guided benchmark for multi-document QA on which frontier LLMs achieve only about 60% exact match.
-
LogiPlan: A Structured Benchmark for Logical Planning and Relational Reasoning in LLMs
LogiPlan introduces a three-task benchmark with controllable graph complexity, showing that although reasoning models excel at plan generation, all models degrade sharply on cycle detection and deep comparison questions.
-
BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domain
BioHopR introduces 1-hop and 2-hop question-answer benchmarks over PrimeKG with multiple correct answers, and shows LLMs achieve low precision, dropping sharply from 1-hop (best 37.93%) to 2-hop (14.57%).
-
CRiT-QA: Evaluating Multi-hop Reasoning with Counterfactual Chains and Distractor Traps
CRiT-QA forces multi-hop QA models to follow counterfactual evidence chains while rejecting type-consistent distractors, revealing large performance collapses relative to standard benchmarks.
-
Reducing Hallucinations in Complex Question Answering using Simple Graph-based Retrieval-Augmented Generation (long version)
A graph-augmented RAG system with vector and graph query tools halves hallucinations and raises factual correctness scores on the MoNaCo complex QA benchmark.
Discussion (0). Sign in to comment.