HopWeaver: Cross-Document Synthesis of High-Quality and Authentic Multi-Hop Questions

Fu Lee Wang; Jianxing Yu; Jiyuan Liu; Yanghui Rao; Yunhe Pang; Zhiyu Shen

arxiv: 2505.15087 · v3 · submitted 2025-05-21 · 💻 cs.CL

HopWeaver: Cross-Document Synthesis of High-Quality and Authentic Multi-Hop Questions

Zhiyu Shen , Jiyuan Liu , Yunhe Pang , Yanghui Rao , Fu Lee Wang , Jianxing Yu This is my paper

Pith reviewed 2026-05-22 14:39 UTC · model grok-4.3

classification 💻 cs.CL

keywords multi-hop question answeringquestion synthesiscross-document reasoningautomatic dataset generationnatural language processingquestion answering datasetssynthetic data

0 comments

The pith

HopWeaver automatically builds authentic multi-hop questions by linking complementary documents across any corpus without human input.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents HopWeaver as a fully automated framework that creates multi-hop questions requiring information from multiple separate documents. It solves the problems of high manual annotation costs and overly simple synthetic questions by first locating pairs of documents that contain related but incomplete information, then building explicit reasoning paths that force integration of facts from both sources. The method generates both bridge questions, which chain facts across documents, and comparison questions, which contrast details from each. Evaluations show the resulting questions reach quality levels comparable to or better than existing human-annotated multi-hop datasets while costing far less to produce. This approach makes it possible to generate large, challenging test sets directly from raw text collections in any domain.

Core claim

HopWeaver is the first cross-document framework that synthesizes authentic multi-hop questions without human intervention. Through a pipeline that identifies complementary documents and constructs authentic reasoning paths, it produces bridge and comparison questions that demand true multi-hop reasoning. A built-in evaluation system confirms these questions achieve quality comparable or superior to human-annotated datasets at lower cost, enabling automatic benchmark creation from any raw corpus.

What carries the argument

The pipeline that identifies complementary documents and constructs authentic reasoning paths to ensure true multi-hop reasoning.

If this is right

Large-scale multi-hop benchmarks can be generated automatically from any raw text collection.
Question-answering models can receive targeted training on authentic cross-document reasoning tasks.
Resource-scarce domains gain access to high-quality evaluation and training data without manual annotation.
Research on advanced reasoning models can iterate faster by creating fresh test sets on demand.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same linking approach could be adapted to generate multi-hop questions inside specialized fields such as legal contracts or medical records.
Repeated application over time might allow models to train on increasingly diverse reasoning patterns without new human labels.
The framework could be extended to produce additional question types beyond bridge and comparison forms.
If quality holds across languages, the method might support creation of multi-hop datasets for low-resource languages.

Load-bearing premise

The automated steps for finding complementary documents and building reasoning paths reliably create questions that cannot be answered from any single document.

What would settle it

Human reviewers examine a random sample of generated questions and determine that a large fraction can be answered correctly using only one of the two source documents.

Figures

Figures reproduced from arXiv: 2505.15087 by Fu Lee Wang, Jianxing Yu, Jiyuan Liu, Yanghui Rao, Yunhe Pang, Zhiyu Shen.

**Figure 1.** Figure 1: Examples of two multi-hop questions synthesized by HopWeaver: Bridge (top) and Comparison (bottom) question. These involve cross-document reasoning via a bridge entity or a shared attribute. MHQA requires a model to connect intermediate entities or concepts across documents to infer answers. However, constructing extensive and highquality MHQA datasets remains costly because manual annotation (Yang et a… view at source ↗

**Figure 2.** Figure 2: HopWeaver: Question Synthesis Framework document ds and a selected complementary document dt (from Dt identified in Step 2), using the bridge entity eb as the pivot. The process consists of the following steps: (a) Sub-Question Generation: To construct the final multi-hop question that requires grounding in both documents, two sequential subquestions are generated: (i) Sub-Question 1 formulated from ds … view at source ↗

**Figure 3.** Figure 3: Fine-Tuning Reranker To enhance the reranking stage (in Section 3.1, Step 2), we fine-tune the reranker using contrastive triples generated through simulating key steps of the bridge question synthesis process ( [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

**Figure 4.** Figure 4: shows that the question synthesized by HopWeaver surpasses the benchmark of human datasets in most dimensions, especially in logical sophistication and information integration, although the top human dataset holds a marginal advantage in conciseness. Fluency Clarity Relevance Conciseness Consistency Question Answerability Answer-Question Consistency Information Integration Reasoning Path Logical Sophistic… view at source ↗

**Figure 5.** Figure 5: An example of a bridge question synthesized [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗

**Figure 6.** Figure 6: An example of a comparison question synthe [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗

**Figure 9.** Figure 9: Heatmap visualizing Fleiss’ Kappa scores [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗

**Figure 7.** Figure 7: Heatmap visualizing AvgSD scores across different LLMs and evaluation dimensions. claude-3-7-sonnet-20250219 deepseek/deepseek-chat-v3-0324 gemini-2.0-flash google/gemma-3-27b-it gpt-4o-2024-11-20 meta-llama/llama-3.3-70b-instruct meta-llama/llama-4-maverick mistralai/mistral-small-3.1-24b-instruct nvidia/llama-3.3-nemotron-super-49b-v1 Model multi_hop_reasoning fluency clarity conciseness relevance consis… view at source ↗

**Figure 8.** Figure 8: Heatmap visualizing Krippendorff’s Alpha [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗

read the original abstract

Multi-Hop Question Answering (MHQA) is crucial for evaluating the model's capability to integrate information from diverse sources. However, creating extensive and high-quality MHQA datasets is challenging: (i) manual annotation is expensive, and (ii) current synthesis methods often produce simplistic questions or require extensive manual guidance. This paper introduces HopWeaver, the first cross-document framework synthesizing authentic multi-hop questions without human intervention. HopWeaver synthesizes bridge and comparison questions through an innovative pipeline that identifies complementary documents and constructs authentic reasoning paths to ensure true multi-hop reasoning. We further present a comprehensive system for evaluating the synthesized multi-hop questions. Empirical evaluations demonstrate that the synthesized questions achieve comparable or superior quality to human-annotated datasets at a lower cost. Our framework provides a valuable tool for the research community: it can automatically generate challenging benchmarks from any raw corpus, which opens new avenues for both evaluation and targeted training to improve the reasoning capabilities of advanced question answering models, especially in domains with scarce resources.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

HopWeaver gives a practical automated pipeline for generating cross-document multi-hop questions from raw text, but the checks that those questions actually require both documents are not strong enough yet.

read the letter

The main takeaway is that HopWeaver automates the creation of bridge and comparison multi-hop questions by selecting complementary documents and building explicit reasoning paths, then claims the output matches or beats human-annotated sets on quality while costing less. This is useful because manual MHQA data is expensive and earlier automatic methods often stayed too simple or needed heavy guidance. The framework can run on any corpus, which matters for low-resource domains where you cannot afford to annotate from scratch. That scalability and the end-to-end automation are the clearest practical advances here. The evaluation system they describe is also more structured than many prior synthesis papers, covering multiple quality axes without human labor at generation time. On the downside, the central claim that the questions force genuine multi-hop reasoning rests on the complementarity step and the path construction, yet the abstract gives no concrete metrics for what counts as complementary or any single-document ablation that would show accuracy drops when one source is removed. An automated evaluation alone can miss cases where surface overlap or shared entities let a model answer from one document. If the full paper does not add those controls or human verification of necessity, the quality comparison to human datasets becomes harder to trust. The work is aimed at QA researchers who need larger or more varied reasoning benchmarks and at groups building training data for models in specialized domains. A reader focused on dataset construction methods will find the pipeline details worth examining even if the multi-hop guarantee needs tightening. I would send it to peer review because the automation problem is real and the results look promising enough to justify referee time, though the authors should expect questions on the reasoning-depth validation.

Referee Report

2 major / 2 minor

Summary. The paper introduces HopWeaver, the first cross-document framework for automatically synthesizing authentic multi-hop questions (bridge and comparison types) without human intervention. It identifies complementary documents from raw corpora and constructs reasoning paths to ensure true multi-hop reasoning, accompanied by a comprehensive automated evaluation system. Empirical results claim that the synthesized questions achieve comparable or superior quality to human-annotated MHQA datasets at lower cost, enabling scalable benchmark generation for any corpus to support model reasoning evaluation and training.

Significance. If the authenticity and quality claims hold, the work would provide a practical, low-cost tool for generating challenging cross-document reasoning benchmarks from arbitrary corpora. This could accelerate progress in MHQA research, particularly in low-resource domains, by reducing dependence on expensive manual annotation and enabling targeted training of models on genuine multi-hop tasks.

major comments (2)

[§4.2] §4.2 (Complementary Document Identification): The complementarity metric is not shown to capture information complementarity (i.e., that each document supplies unique facts necessary for the answer) rather than topical overlap or shared entities. Without this distinction, the pipeline risks producing questions solvable via single-document reasoning or surface cues, directly undermining the central claim of 'authentic' and 'true multi-hop' questions.
[§5] §5 (Evaluation System): The comprehensive evaluation is fully automated and reports quality comparable to human datasets, but contains no single-document ablation (e.g., model accuracy when one document is withheld) or human verification of reasoning necessity. This omission leaves the claim that synthesized questions require cross-document integration untested and is load-bearing for the superiority claim over human-annotated data.

minor comments (2)

[Abstract] The abstract and introduction use 'comprehensive system' and 'empirical evaluations' without naming specific metrics, baselines, or datasets in the opening paragraphs; adding these references early would improve readability.
[§4.3] Notation for reasoning paths and bridge/comparison question templates could be formalized with a small example table to clarify the construction step for readers unfamiliar with MHQA synthesis.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their insightful comments, which have helped us improve the clarity and rigor of our work. We provide detailed responses to each major comment below and indicate the revisions made to the manuscript.

read point-by-point responses

Referee: [§4.2] §4.2 (Complementary Document Identification): The complementarity metric is not shown to capture information complementarity (i.e., that each document supplies unique facts necessary for the answer) rather than topical overlap or shared entities. Without this distinction, the pipeline risks producing questions solvable via single-document reasoning or surface cues, directly undermining the central claim of 'authentic' and 'true multi-hop' questions.

Authors: We value this observation on the complementarity metric. In the original manuscript, §4.2 describes how the metric selects document pairs by measuring the potential for constructing multi-hop questions that bridge information across documents, using extracted entities and relations to prioritize pairs with complementary facts. To more explicitly demonstrate that this goes beyond topical overlap, we have added a new analysis in the revised version showing that randomly selected document pairs with similar topics do not yield valid multi-hop questions under our pipeline, whereas our selected pairs do. This supports the distinction and reinforces the authenticity of the synthesized questions. revision: yes
Referee: [§5] §5 (Evaluation System): The comprehensive evaluation is fully automated and reports quality comparable to human datasets, but contains no single-document ablation (e.g., model accuracy when one document is withheld) or human verification of reasoning necessity. This omission leaves the claim that synthesized questions require cross-document integration untested and is load-bearing for the superiority claim over human-annotated data.

Authors: We agree that including a single-document ablation and some form of verification for reasoning necessity would strengthen the evaluation claims in §5. Although our automated metrics are designed to assess multi-hop characteristics, we have now incorporated a single-document ablation experiment in the revised manuscript. The results show a notable decrease in model performance when one document is withheld, indicating the necessity of cross-document information. For human verification, we conducted a limited human study on a subset of questions to confirm that they require integration from multiple documents, with results reported in the updated section. These additions provide direct evidence supporting our claims. revision: yes

Circularity Check

0 steps flagged

No circularity: independent pipeline with separate evaluation

full rationale

The paper presents HopWeaver as an automated cross-document synthesis pipeline that identifies complementary documents and constructs reasoning paths, followed by a distinct comprehensive evaluation system. No equations, fitted parameters, or predictions are described that reduce to the inputs by construction. The central claims rest on the pipeline's design and empirical comparisons to human-annotated datasets rather than self-definitional loops, self-citation load-bearing premises, or renaming of known results. The method is self-contained against external benchmarks, with evaluation treated as an independent step rather than a tautological output of the synthesis process.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Review is based only on the abstract; no explicit free parameters, axioms, or invented entities are described. The framework is characterized at a high level as an innovative pipeline without technical specifics.

pith-pipeline@v0.9.0 · 5717 in / 1168 out tokens · 49427 ms · 2026-05-22T14:39:14.118990+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Foundation/RealityFromDistinction.lean reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

HopWeaver synthesizes bridge and comparison questions through an innovative pipeline that identifies complementary documents and constructs authentic reasoning paths
IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

We further present a comprehensive system for evaluating the synthesized multi-hop questions

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

EVE: A Domain-Specific LLM Framework for Earth Intelligence
cs.CL 2026-03 unverdicted novelty 6.0

EVE is the first open-source end-to-end system with a domain-adapted 24B LLM that outperforms peers on new Earth Intelligence benchmarks while adding RAG and hallucination detection in a production deployment.

Reference graph

Works this paper leans on

34 extracted references · 34 canonical work pages · cited by 1 Pith paper

[1]

InProceedings of the 33 International Joint Conference on Artificial Intelligence, pages 8038–8047

A survey on neural question generation: Meth- ods, applications, and prospects. InProceedings of the 33 International Joint Conference on Artificial Intelligence, pages 8038–8047. Xu Guo and Yiqiang Chen. 2024. Generative AI for synthetic data generation: Methods, challenges and the future.CoRR, abs/2403.04190. Jie He, Nan Hu, Wanqiu Long, Jiaoyan Chen, a...

work page arXiv 2024
[2]

InProceedings of the 2024 Joint Inter- national Conference on Computational Linguistics, Language Resources and Evaluation, pages 6855– 6866

Explainable multi-hop question generation: An end-to-end approach without intermediate ques- tion labeling. InProceedings of the 2024 Joint Inter- national Conference on Computational Linguistics, Language Resources and Evaluation, pages 6855– 6866. Jiajie Jin, Yutao Zhu, Zhicheng Dou, Guanting Dong, Xinyu Yang, Chenghao Zhang, Tong Zhao, Zhao Yang, and J...

work page 2024
[3]

InProceedings of the 18th Confer- ence of the European Chapter of the Association for Computational Linguistics, pages 139–151

LLM comparative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models. InProceedings of the 18th Confer- ence of the European Chapter of the Association for Computational Linguistics, pages 139–151. Yingzhou Lu, Huazheng Wang, and Wenqi Wei. 2023. Machine learning for synthetic data generation: a review.CoRR, abs/23...

work page arXiv 2023
[4]

Compositional

KGAST: From knowledge graphs to annotated synthetic texts. InProceedings of the 1st Workshop on Knowledge Graphs and Large Language Models, pages 43–55. Maximilian Wich, Christian Widmer, Gerhard Hagerer, and Georg Groh. 2021. Investigating annotator bias in abusive language datasets. InProceedings of the International Conference on Recent Advances in Nat...

work page 2021
[5]

Hop 1(Doc A → Doc B): Didier Lockwood recorded a tribute album in 2000 forStéphane Grappelli

work page 2000
[6]

Hop 2(Doc B → Doc C): Stéphane Grappelli co-founded theQuintette du Hot Club de France in 1934

work page 1934
[7]

medial tendinous margins of the crura... meet ... to form an arch ... known as the median arcuate ligament

Hop 3(Doc C → Answer): The Quintette du Hot Club de France was employed as a house band byLa Grosse Pommein 1937. Verification.Each hop requires information from a distinct document: • Sub-question 1: Who did Didier Lockwood honor with a tribute album in 2000? → Stéphane Grappelli • Sub-question 2: Which quintet did Stéphane Grappelli co-found? → Quintett...

work page 1937
[8]

Identify Structure (from Source): The anatomical structure formed is the median arcuate ligament

work page
[9]

Link to Condition (from Target): Compression by the median arcuate ligament leads to Median arcuate ligament syndrome. Question: What medical condition is attributed to the compression caused by the anatomical structure formed by the meeting of the medial tendinous margins of the diaphragm's crura? Answer: Median arcuate ligament syndrome (MALS) Figure 5:...

work page 2023
[10]

No” indicates a fundamental failure. • Linguistic Dimensions: These evaluate the 16 Source Document (Composer Biography Snippet):

because our synthesized questions are non- parallel and vary significantly in difficulty, making it challenging to establish fair comparative bench- marks necessary for pairwise approaches. The de- tailed criteria for our pointwise evaluation are orga- nized into three main categories: • Multi-Hop QA Rule Dimension: This is a binary (Yes/No) evaluation de...

work page
[11]

Identify Information (from Source): Mihály Mosonyi was born on September 4, 1815

work page
[12]

Identify Information (from Target): Franz Liszt was born on October 22, 1811

work page
[13]

Paris”, “IBM

Compare Dates: 1811 is earlier than 1815 Question: Which composer has an earlier Date of Birth: á or Franz Liszt? Answer: Franz Liszt Figure 6: An example of a comparison question synthe- sized by HopWeaver. This showcases the two entities being compared, the specific attribute, the source evi- dence snippets, and the resulting question. quality of questi...

work page 2023
[14]

How to find the bridge entity (Answer 1) in Document A

work page
[15]

Instructions You are a Polisher module responsible for validating and refining multi-hop questions

How this bridge entity leads to the final answer in Document B] SOURCES: [Document A and Document B, specifying their roles] 24 Bridge Polisher Prompt (POLISHER_PROMPT) Goal Validate and refine multi-hop questions to ensure they genuinely require cross-document reasoning and follow a proper reasoning chain where information from one document is essential ...

work page
[16]

If the question passes all criteria without changes: [PASS]

work page
[17]

If the question needs minor adjustments: [ADJUST] REFINED_REASONING_PATH: [Updated reasoning path] REFINED_QUESTION: [Adjusted question] REFINED_ANSWER: [Updated answer if needed]

work page
[18]

If the question needs significant refinement: [REWORKED] REFINED_REASONING_PATH: [Revised reasoning path] REFINED_QUESTION: [Substantially revised question] REFINED_ANSWER: [Updated answer]

work page
[19]

Population

If the question is fundamentally flawed: [REJECTED] 25 Bridge MHQA Quality Assessment Prompt (MHQA_QUALITY_ASSESSMENT_PROMPT) Goal Conduct arigorous and criticalevaluation of multi-hop questions and their answers across multi- ple quality dimensions. Focus on ensuring questions require genuine cross-document reasoning and are free from logical flaws. A hi...

work page 1990
[20]

If the question needs no modification: [PASS]

work page
[21]

If the question needs fine-tuning: [ADJUST] REFINED_QUESTION: [Unified question with background] REFINED_ANSWER: [Adjusted answer if needed]

work page
[22]

If the question needs substantial rewriting: [REWORKED] REFINED_QUESTION: [Completely rewritten question] REFINED_ANSWER: [New answer] REFINED_FACT_A: [Corrected fact for entity A if needed] REFINED_FACT_B: [Corrected fact for entity B if needed]

work page
[23]

If the question cannot be fixed: [REJECTED] REASON: [Brief explanation of rejection reason] 29 Compare Question Builder Prompt (COMPARE_QUESTION_BUILDER_PROMPT) Goal Imagine you are comparing two documents, Document A (about Entity A) and Document B (a candidate potentially containing a related Entity B).Your task is to:

work page
[24]

Identify the main subject entity within Document B (potential Entity B) and see if it’s relevant to Entity A

work page
[25]

Find if there isat least one specific, comparable attribute pairbetween Entity A and the potential Entity B

work page
[26]

If a suitable comparison pair is found,directly generatea natural languagedirect compari- son question, itscomparative answer, and supportingfull sentence(s)

work page
[27]

If no suitable entity or comparable attribute pair is found, indicate failure. Instructions 1.Analyze Inputs:You are given: - Primary Entity A:{subject_entity_name}(Type:{subject_entity_type}) - Document A Text:{document_a_text} - Entity A’s Attributes List:{attributes_list_str_a} - Candidate Document B Text:{document_b_text} 2.Identify Entity B and Find ...

work page
[28]

Success Output (If a comparable pair was found): PASS entity_a: Name of Entity A (from input) entity_b: Identified Entity B Name (from step 2) attribute_compared: Matched Attribute Name multi_hop_question: Generated DIRECT Comparison Question answer: Concise COMPARATIVE Answer Text fact_entity_a: Extracted Full Sentence(s) for Fact A fact_entity_b: Extrac...

work page
[29]

recall_focused_verify

Failure Output: FAIL 30 Compare Query Generator Prompt (COMPARE_QUERY_GENERATOR_PROMPT) Goal Imagine you are an assistant helping to create interesting comparison questions that might require looking up information in different places (multi-hop).Your task is to analyze a primary entity (Entity A) and its known details. Based on this, decide the bestfirst...

work page
[30]

Structure:A single string containing the chosen path information

work page
[31]

recall_focused_verify

Output Parts (Choose ONE format): Path 1: ("recall_focused_verify"<|>Suggested Entity B Name<|> Chosen Attribute X Name<|>Verification Query) Path 2: ("search_queries"<|>Query 1<|>Query 2<|>Query 3)

work page
[32]

Completion Signal:Append<|COMPLETE|>at the end. 31 Compare QA Quality Assessment Prompt (COMPARE_QA_QUALITY_ASSESSMENT_PROMPT) Goal Conduct arigorous and criticalevaluation of multi-hop comparison questions across multiple quality dimensions. Focus on ensuring questions require genuine cross-document reasoningand are free from logical flaws. A high-qualit...

work page
[33]

Answering requires factual information from at least two different documents

work page
[34]

Yes" only if BOTH conditions are met, otherwise

No single document contains all necessary information about both entities being compared Rate "Yes" only if BOTH conditions are met, otherwise "No" 2.Linguistic Dimensions(Rate as: Very Poor, Poor, Fair, Good, Very Good) -Fluency: Is the question grammatically correct, coherent, and easy to understand? -Clarity: Is the question clearly and precisely expre...

work page

[1] [1]

InProceedings of the 33 International Joint Conference on Artificial Intelligence, pages 8038–8047

A survey on neural question generation: Meth- ods, applications, and prospects. InProceedings of the 33 International Joint Conference on Artificial Intelligence, pages 8038–8047. Xu Guo and Yiqiang Chen. 2024. Generative AI for synthetic data generation: Methods, challenges and the future.CoRR, abs/2403.04190. Jie He, Nan Hu, Wanqiu Long, Jiaoyan Chen, a...

work page arXiv 2024

[2] [2]

InProceedings of the 2024 Joint Inter- national Conference on Computational Linguistics, Language Resources and Evaluation, pages 6855– 6866

Explainable multi-hop question generation: An end-to-end approach without intermediate ques- tion labeling. InProceedings of the 2024 Joint Inter- national Conference on Computational Linguistics, Language Resources and Evaluation, pages 6855– 6866. Jiajie Jin, Yutao Zhu, Zhicheng Dou, Guanting Dong, Xinyu Yang, Chenghao Zhang, Tong Zhao, Zhao Yang, and J...

work page 2024

[3] [3]

InProceedings of the 18th Confer- ence of the European Chapter of the Association for Computational Linguistics, pages 139–151

LLM comparative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models. InProceedings of the 18th Confer- ence of the European Chapter of the Association for Computational Linguistics, pages 139–151. Yingzhou Lu, Huazheng Wang, and Wenqi Wei. 2023. Machine learning for synthetic data generation: a review.CoRR, abs/23...

work page arXiv 2023

[4] [4]

Compositional

KGAST: From knowledge graphs to annotated synthetic texts. InProceedings of the 1st Workshop on Knowledge Graphs and Large Language Models, pages 43–55. Maximilian Wich, Christian Widmer, Gerhard Hagerer, and Georg Groh. 2021. Investigating annotator bias in abusive language datasets. InProceedings of the International Conference on Recent Advances in Nat...

work page 2021

[5] [5]

Hop 1(Doc A → Doc B): Didier Lockwood recorded a tribute album in 2000 forStéphane Grappelli

work page 2000

[6] [6]

Hop 2(Doc B → Doc C): Stéphane Grappelli co-founded theQuintette du Hot Club de France in 1934

work page 1934

[7] [7]

medial tendinous margins of the crura... meet ... to form an arch ... known as the median arcuate ligament

Hop 3(Doc C → Answer): The Quintette du Hot Club de France was employed as a house band byLa Grosse Pommein 1937. Verification.Each hop requires information from a distinct document: • Sub-question 1: Who did Didier Lockwood honor with a tribute album in 2000? → Stéphane Grappelli • Sub-question 2: Which quintet did Stéphane Grappelli co-found? → Quintett...

work page 1937

[8] [8]

Identify Structure (from Source): The anatomical structure formed is the median arcuate ligament

work page

[9] [9]

Link to Condition (from Target): Compression by the median arcuate ligament leads to Median arcuate ligament syndrome. Question: What medical condition is attributed to the compression caused by the anatomical structure formed by the meeting of the medial tendinous margins of the diaphragm's crura? Answer: Median arcuate ligament syndrome (MALS) Figure 5:...

work page 2023

[10] [10]

No” indicates a fundamental failure. • Linguistic Dimensions: These evaluate the 16 Source Document (Composer Biography Snippet):

because our synthesized questions are non- parallel and vary significantly in difficulty, making it challenging to establish fair comparative bench- marks necessary for pairwise approaches. The de- tailed criteria for our pointwise evaluation are orga- nized into three main categories: • Multi-Hop QA Rule Dimension: This is a binary (Yes/No) evaluation de...

work page

[11] [11]

Identify Information (from Source): Mihály Mosonyi was born on September 4, 1815

work page

[12] [12]

Identify Information (from Target): Franz Liszt was born on October 22, 1811

work page

[13] [13]

Paris”, “IBM

Compare Dates: 1811 is earlier than 1815 Question: Which composer has an earlier Date of Birth: á or Franz Liszt? Answer: Franz Liszt Figure 6: An example of a comparison question synthe- sized by HopWeaver. This showcases the two entities being compared, the specific attribute, the source evi- dence snippets, and the resulting question. quality of questi...

work page 2023

[14] [14]

How to find the bridge entity (Answer 1) in Document A

work page

[15] [15]

Instructions You are a Polisher module responsible for validating and refining multi-hop questions

How this bridge entity leads to the final answer in Document B] SOURCES: [Document A and Document B, specifying their roles] 24 Bridge Polisher Prompt (POLISHER_PROMPT) Goal Validate and refine multi-hop questions to ensure they genuinely require cross-document reasoning and follow a proper reasoning chain where information from one document is essential ...

work page

[16] [16]

If the question passes all criteria without changes: [PASS]

work page

[17] [17]

If the question needs minor adjustments: [ADJUST] REFINED_REASONING_PATH: [Updated reasoning path] REFINED_QUESTION: [Adjusted question] REFINED_ANSWER: [Updated answer if needed]

work page

[18] [18]

If the question needs significant refinement: [REWORKED] REFINED_REASONING_PATH: [Revised reasoning path] REFINED_QUESTION: [Substantially revised question] REFINED_ANSWER: [Updated answer]

work page

[19] [19]

Population

If the question is fundamentally flawed: [REJECTED] 25 Bridge MHQA Quality Assessment Prompt (MHQA_QUALITY_ASSESSMENT_PROMPT) Goal Conduct arigorous and criticalevaluation of multi-hop questions and their answers across multi- ple quality dimensions. Focus on ensuring questions require genuine cross-document reasoning and are free from logical flaws. A hi...

work page 1990

[20] [20]

If the question needs no modification: [PASS]

work page

[21] [21]

If the question needs fine-tuning: [ADJUST] REFINED_QUESTION: [Unified question with background] REFINED_ANSWER: [Adjusted answer if needed]

work page

[22] [22]

If the question needs substantial rewriting: [REWORKED] REFINED_QUESTION: [Completely rewritten question] REFINED_ANSWER: [New answer] REFINED_FACT_A: [Corrected fact for entity A if needed] REFINED_FACT_B: [Corrected fact for entity B if needed]

work page

[23] [23]

If the question cannot be fixed: [REJECTED] REASON: [Brief explanation of rejection reason] 29 Compare Question Builder Prompt (COMPARE_QUESTION_BUILDER_PROMPT) Goal Imagine you are comparing two documents, Document A (about Entity A) and Document B (a candidate potentially containing a related Entity B).Your task is to:

work page

[24] [24]

Identify the main subject entity within Document B (potential Entity B) and see if it’s relevant to Entity A

work page

[25] [25]

Find if there isat least one specific, comparable attribute pairbetween Entity A and the potential Entity B

work page

[26] [26]

If a suitable comparison pair is found,directly generatea natural languagedirect compari- son question, itscomparative answer, and supportingfull sentence(s)

work page

[27] [27]

If no suitable entity or comparable attribute pair is found, indicate failure. Instructions 1.Analyze Inputs:You are given: - Primary Entity A:{subject_entity_name}(Type:{subject_entity_type}) - Document A Text:{document_a_text} - Entity A’s Attributes List:{attributes_list_str_a} - Candidate Document B Text:{document_b_text} 2.Identify Entity B and Find ...

work page

[28] [28]

Success Output (If a comparable pair was found): PASS entity_a: Name of Entity A (from input) entity_b: Identified Entity B Name (from step 2) attribute_compared: Matched Attribute Name multi_hop_question: Generated DIRECT Comparison Question answer: Concise COMPARATIVE Answer Text fact_entity_a: Extracted Full Sentence(s) for Fact A fact_entity_b: Extrac...

work page

[29] [29]

recall_focused_verify

Failure Output: FAIL 30 Compare Query Generator Prompt (COMPARE_QUERY_GENERATOR_PROMPT) Goal Imagine you are an assistant helping to create interesting comparison questions that might require looking up information in different places (multi-hop).Your task is to analyze a primary entity (Entity A) and its known details. Based on this, decide the bestfirst...

work page

[30] [30]

Structure:A single string containing the chosen path information

work page

[31] [31]

recall_focused_verify

Output Parts (Choose ONE format): Path 1: ("recall_focused_verify"<|>Suggested Entity B Name<|> Chosen Attribute X Name<|>Verification Query) Path 2: ("search_queries"<|>Query 1<|>Query 2<|>Query 3)

work page

[32] [32]

Completion Signal:Append<|COMPLETE|>at the end. 31 Compare QA Quality Assessment Prompt (COMPARE_QA_QUALITY_ASSESSMENT_PROMPT) Goal Conduct arigorous and criticalevaluation of multi-hop comparison questions across multiple quality dimensions. Focus on ensuring questions require genuine cross-document reasoningand are free from logical flaws. A high-qualit...

work page

[33] [33]

Answering requires factual information from at least two different documents

work page

[34] [34]

Yes" only if BOTH conditions are met, otherwise

No single document contains all necessary information about both entities being compared Rate "Yes" only if BOTH conditions are met, otherwise "No" 2.Linguistic Dimensions(Rate as: Very Poor, Poor, Fair, Good, Very Good) -Fluency: Is the question grammatically correct, coherent, and easy to understand? -Clarity: Is the question clearly and precisely expre...

work page