Pith. sign in

REVIEW 3 major objections 5 minor 23 references

Hybrid-SQuAD: Hybrid Scholarly Question Answering Dataset

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Hybrid-SQuAD is a 10.5K-question scholarly QA benchmark whose answers require combining DBLP, SemOpenAlex, and Wikipedia facts.

desk verdict A genuinely new scholarly hybrid-QA benchmark with a real gold-validation gap; worth reviewing but needs revision. read the letter →

arxiv 2412.02788 v2 pith:3FUGSJTC submitted 2024-12-03 cs.CL cs.AI

Tilahun Abedissa Taffa , Debayan Banerjee , Yaregal Assabie , Ricardo Usbeck This is my paper · ORCID
classification cs.CLcs.AI
keywords hybridquestionansweringscholarlyknowledgegraphsretrieval-augmentedgenerationDBLPSemOpenAlexdatasetLLM-generatedmulti-sourcereasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Hybrid-SQuAD, a 10,581-question benchmark for scholarly question answering in which every question can only be answered by combining facts from two knowledge graphs (DBLP and SemOpenAlex) with Wikipedia text. Existing scholarly QA datasets draw on a single source, so they cannot test systems that must fuse structured and unstructured evidence. The authors claim this is the first large-scale scholarly QA dataset that forces hybrid KG-plus-text reasoning, and they report a retrieval-augmented baseline reaching 69.65 exact match while ChatGPT-3.5 in a zero-shot setting scores 2.6. If the dataset is sound, it gives the field a concrete way to measure multi-source scholarly QA.

What carries the argument

The load-bearing object is the bridging entity: each question embeds a sub-question phrase that must be resolved to a scholarly entity (author, publication, or institution) before evidence can be gathered. The baseline resolves these phrases with ChatGPT-3.5 prompts and SPARQL templates over DBLP, links authors to SemOpenAlex through ORCID, then retrieves Wikipedia text and KG triples; a retrieval-augmented generation (RAG) generator, using FAISS over 200-word chunks embedded with bge-small-en-v1.5 and answered by ChatGPT-3.5 or LLAMA-3-8B, produces the final answer.

What would settle it

Manually answer a random sample of 200 test-set questions using only the released source data; if a substantial fraction of gold answers are wrong or unanswerable, then the reported 69.65 EM measures something other than hybrid QA.

Watch

Extended reading notes

Core claim

The central claim is that hybrid scholarly QA can be benchmarked at scale: the authors construct 10.5K question-answer pairs by prompting ChatGPT-3.5 with paired inputs from DBLP, SemOpenAlex, and Wikipedia, organize the questions into four evidence-traversal paths (KG-to-KG bridge, KG-to-KG comparison, KG-to-text, and KG-to-KG-to-text), and show that a RAG pipeline that links entities, retrieves Wikipedia chunks, and queries KG triples reaches 69.65 exact match and 74.91 F1 on the test set. The same questions answered from the model's internal knowledge alone yield 2.6 EM. The dataset and baseline are offered as the starting point for scholarly hybrid QA research.

Load-bearing premise

The gold answers are correct and actually derivable from the stated sources; only 100 of the 10,581 questions were manually inspected, and that check covered answer categories rather than answer accuracy.

Editorial extensions

If this is right

  • Single-source scholarly QA benchmarks no longer suffice as evidence of general QA ability; Hybrid-SQuAD provides a 10.5K-question test that requires fusion of structured and unstructured evidence.
  • The roughly 67-point gap between zero-shot and RAG performance quantifies how much retrieval contributes when answers are not stored in model weights.
  • The four traversal-path types give a diagnostic: systems can be scored separately on bridge, comparison, KG-text, and KG-KG-text questions.
  • Reported alternative systems (32.0 to 48.9 exact match) all fall below the RAG baseline, suggesting the retrieve-then-generate recipe is currently the most effective documented approach on this benchmark.
  • The dataset can be reused to evaluate entity linking and sub-question decomposition in isolation, not only end-to-end QA.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the gold answers were produced by ChatGPT-3.5, the dataset may contain answer patterns or artifacts that retrieval models exploit; a human re-annotation of a larger sample would show whether the 69.65 EM reflects hybrid reasoning or surface regularities.
  • Editorial inference: the pipeline of pairing KG snapshots with Wikipedia text could be transplanted to other domains, such as biomedical or legal question answering, where facts are split between structured databases and prose.
  • Editorial inference: the dataset's selection criteria (authors with ORCID and Wikipedia pages) skew toward well-documented computer scientists, so performance may not transfer to less-documented scholars or to other academic fields.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Hybrid-SQuAD, a dataset of 10,581 scholarly question-answer pairs that require integrating facts from DBLP, SemOpenAlex, and Wikipedia. The data is generated by ChatGPT-3.5 from per-author records that combine KG triples and Wikipedia text, and the authors identify four evidence-traversal pathways: KG-to-KG bridge, KG-to-KG comparison, KG-to-text, and KG-to-KG-to-text. The paper also proposes a RAG-based baseline with a link-retrieve-generate pipeline, reporting 69.65% exact match on the test set, while zero-shot ChatGPT-3.5 obtains only 2.6% EM. The central claim is that Hybrid-SQuAD is the first large-scale scholarly QA benchmark that forces hybrid KG+text reasoning.

Significance. If the dataset is accurate and the gold answers are correct, Hybrid-SQuAD would fill a real gap: existing scholarly QA datasets such as DBLP-QuAD, SciQA, and QASA are single-source, and general hybrid QA benchmarks like CompMix do not require cross-source reasoning. The paper's strengths include the public release of a 10.5K dataset, a concrete taxonomy of four traversal pathways, and a reproducible-style baseline with both ChatGPT-3.5 and LLaMA generators. The zero-shot numbers, if trustworthy, demonstrate that the dataset is not trivially answerable from parametric knowledge. However, the benchmark's value hinges on the correctness and answerability of the LLM-generated gold answers, and that assumption is not adequately validated in the current manuscript.

major comments (3)
  1. [Section 3.2.2 and Section 4.1] The gold answers are generated entirely by ChatGPT-3.5, and the only reported quality check is a manual analysis of 100 questions that categorizes answer types (Table 2) rather than verifying correctness, completeness, or answerability from the provided sources. The paper itself states that the initial generation produced questions lacking an answer or with single-letter responses, so the pipeline is demonstrably error-prone; re-prompting to replace incomplete answers does not establish that the final gold answers are correct. Since the paper's central contribution is a benchmark, the 69.65 EM baseline and the 2.6 EM zero-shot result measure agreement with an unvalidated LLM output unless a human-annotation study confirms the gold answers. The authors should run a human evaluation on a random sample (with multiple annotators, measuring inter-annotator agreement) to report the share of questions that are answerable and the share of gold answers that are correct, and they should release any corrected gold answers or filter out unanswerable questions.
  2. [Table 4 and footnote 20] The comparison with Efeoglu et al. is not valid as presented because footnote 20 states that this model was evaluated on a different test split. Reporting an EM score of 48.9 for Efeoglu et al. next to 69.65 for the proposed model in the same table implies a direct comparison, but different splits make the difference uninterpretable. The authors should either evaluate Efeoglu et al. on the same Hybrid-SQuAD test split or clearly separate reported results by split and avoid claiming superiority on the basis of incomparable numbers.
  3. [Section 5.1 and Section 5.3] The baseline uses ChatGPT-3.5 both for sub-question extraction and entity identification in the linking phase and for final answer generation, while ChatGPT-3.5 was also the model that generated the question-answer pairs. This creates a potential style-matching bias: the baseline may achieve high EM because it reproduces the answer format and phrasing distribution of the generator rather than because it performs genuine hybrid reasoning. The zero-shot result (2.6 EM) mitigates this concern but does not eliminate it, since the retrieval and prompting pipeline still relies on the same model. The authors should report an analysis of answer-format agreement between generated questions and baseline predictions, and ideally validate a sample of baseline predictions against human judgments of correctness.
minor comments (5)
  1. [Section 3.2.1] There is a typo in the sentence beginning 'Unlike bridging questions, for KG-KG comparison questions, , the data source pool' where an extra comma appears after 'questions'.
  2. [Footnote 10 and Section 6] The text refers to a fine-tuned Flan-T5-Large model, but the footnote links to 'google/flan-t5-small'; please correct this inconsistency so the model size is unambiguous.
  3. [Table 4 and References] The table lists 'Fondi and Fidel 2024' while the reference list gives 'Fondi and Jiomekong Fidel'; use a consistent author name format in the table, text, and bibliography.
  4. [Throughout] The model name is written inconsistently as 'LLAMA', 'LLaMA', and 'LLAMA3.0'; choose one canonical spelling and apply it consistently.
  5. [Section 3 and Section 6] The paper does not specify the exact train/test split sizes, the random seed used for splitting, or the number of authors/entities covered; adding these statistics would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports a dataset construction pipeline and a baseline evaluation; no claim reduces to its inputs by definition or by self-citation.

full rationale

Hybrid-SQuAD is a dataset paper with no mathematical derivation or fitted parameter that is later renamed as a prediction. The gold answer pairs are generated by ChatGPT-3.5 from provided DBLP, SemOpenAlex, and Wikipedia contexts, and the baseline is evaluated against those gold answers. There is no equation in which the output equals the input by construction. The principal methodological concern is that the same LLM (ChatGPT-3.5) is used both to generate the gold QA pairs and as the generator inside the RAG baseline, which could in principle bias answer style or format. However, this is an external validity or benchmark-calibration issue, not a circularity of the paper's own reasoning chain: the baseline must still retrieve external context, and the zero-shot result (2.6 EM) shows the model cannot answer from memory, so the reported improvement is not tautological. Likewise, the gold answers being unvalidated for accuracy (only 100 questions were manually categorized in Section 4.1) threatens correctness of the benchmark, but it does not mean the paper's claims are defined in terms of themselves. The self-citations (Taffa and Usbeck 2023; Banerjee et al. 2023) are background on scholarly QA and are not used to justify the central result. Therefore the paper is not circular; any concerns fall under dataset-quality risk rather than circular reasoning.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper contributes a dataset and baseline; its load-bearing assumptions are about dataset quality rather than about mathematical axioms. The main free parameters are baseline hyperparameters, and the key domain assumptions concern the validity and answerability of the generated QA pairs.

free parameters (2)
  • Retrieval chunk size = 200 words with 10-word overlap
    Chosen for the RAG baseline in Section 5.3; not fitted to data but hand-picked and not varied in an ablation.
  • Top-k retrieved chunks = 5
    Used in the RAG baseline Section 5.3; fixed without reported sensitivity analysis.
assumptions (3)
  • domain assumption Every generated question has an extractive answer that is a contiguous span in the provided sources (per prompt in Listing 1).
    The generation prompt instructs the LLM to make the answer a word or a set of consecutive words in the source; the dataset inherits this as a design constraint that may not hold for all questions.
  • domain assumption LLM-generated question-answer pairs are correct and grounded in the supplied contexts.
    No large-scale human verification is reported; Section 4.1 only analyzes answer types of 100 questions, not correctness.
  • domain assumption The three data sources (DBLP, SemOpenAlex, Wikipedia) cover all facts needed for the questions.
    The dataset only includes authors/institutions that have both ORCID and English Wikipedia pages, so the benchmark is deliberately restricted to this subpopulation (Section 3.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hybrid-SQuAD: Hybrid Scholarly Question Answering Dataset." pith.science (2026). https://pith.science/paper/3FUGSJTC

@misc{pith2026241202788,
  author       = {Pith},
  title        = {Pith review of: Hybrid-SQuAD: Hybrid Scholarly Question Answering Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3FUGSJTC}},
  note         = {Machine review of arXiv:2412.02788}
}
read the original abstract

Existing Scholarly Question Answering (QA) methods typically target homogeneous data sources, relying solely on either text or Knowledge Graphs (KGs). However, scholarly information often spans heterogeneous sources, necessitating the development of QA systems that integrate information from multiple heterogeneous data sources. To address this challenge, we introduce Hybrid-SQuAD (Hybrid Scholarly Question Answering Dataset), a novel large-scale QA dataset designed to facilitate answering questions incorporating both text and KG facts. The dataset consists of 10.5K question-answer pairs generated by a large language model, leveraging the KGs DBLP and SemOpenAlex alongside corresponding text from Wikipedia. In addition, we propose a RAG-based baseline hybrid QA model, achieving an exact match score of 69.65 on the Hybrid-SQuAD test set.

Figures

Figures reproduced from arXiv: 2412.02788 by the authors.

Figure 1
Figure 1. Sample scholarly information from het￾erogeneous data sources: DBLP, SemOpenAlex, and Wikipedia, along with a question and answer pair in Hybrid-SQuAD. Saikh et al., 2022). Additionally, the existing Schol￾arly QA data sets focus on dealing with homo￾geneous data, as listed in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Baseline Model 5.3 Generate The answer generation process for questions involv￾ing KG-Text or KG-KG-Text evidence pathways leverages RAG, combining document retrieval with LLM to ensure precise responses. The RAG model loads the text, divides it into 200-word chunks with a 10-word overlap, generates embeddings using the ‘BAAI/bge-small-en-v1.5’19 model, and stores these embeddings in a FAISS (Douze et al., 2024) vec… view at source ↗
Figure 3
Figure 3. First three words distributions in questions. [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 12 canonical work pages

  1. [1]

    S \"o ren Auer, Dante AC Barone, Cassiano Bartz, Eduardo G Cortes, Mohamad Yaser Jaradeh, Oliver Karras, Manolis Koubarakis, Dmitry Mouromtsev, Dmitrii Pliukhin, Daniil Radyush, et al. 2023. https://www.nature.com/articles/s41598-023-33607-z The SciQA Scientific Question Answering Benchmark for Scholarly Knowledge . Scientific Reports, 13(1):7240

    work page 2023

  2. [2]

    Debayan Banerjee, Sushil Awale, Ricardo Usbeck, and Chris Biemann. 2023. https://ceur-ws.org/Vol-3617/paper-05.pdf Dblp-quad: A question answering dataset over the DBLP scholarly knowledge graph . In Proceedings of the 13th International Workshop on Bibliometric-enhanced Information Retrieval co-located with 45th European Conference on Information Retriev...

    work page 2023

  3. [3]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  4. [4]

    Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.91 H ybrid QA : A dataset of multi-hop question answering over tabular and textual data . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1026--1036, Online. Association for Computational...

    doi:10.18653/v1/2020.findings-emnlp.91 2020

  5. [5]

    Philipp Christmann, Rishiraj Saha Roy, and Gerhard Weikum. 2024. https://doi.org/10.1145/3589335.3651444 Compmix: A benchmark for heterogeneous question answering . In Companion Proceedings of the ACM on Web Conference 2024, WWW '24, page 1091–1094, New York, NY, USA. Association for Computing Machinery

    arXiv 2024

  6. [6]

    Eleftherios Dimitrakis, Konstantinos Sgontzos, and Yannis Tzitzikas. 2020. A survey on question answering systems over linked data and documents. Journal of intelligent information systems, 55(2):233--259

    work page 2020

  7. [7]

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. https://arxiv.org/abs/2401.08281 The faiss library

    arXiv 2024

  8. [8]

    Daria Dzendzik, Jennifer Foster, and Carl Vogel. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.693 English Machine Reading Comprehension Datasets : A Survey . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 8784--8804, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics

    work page doi:10.18653/v1/2021.emnlp-main.693 2021

Show all 23 references
  1. [9]

    Sefika Efeoglu, Nikolas Rauscher, Erik Rubinov, Yuxin Xue, and Schimmler Sonja. 2024. http://dx.doi.org/10.13140/RG.2.2.18260.21123 Large Language Models for Scholarly Question Answering Using Hybrid Data Sources

    2024

  2. [10]

    Michael F \"a rber, David Lamprecht, Johan Krause, Linn Aung, and Peter Haase. 2023. Semopenalex: The scientific landscape in 26 billion rdf triples. In International Semantic Web Conference, pages 94--112. Springer

    2023

  3. [11]

    Yue Feng, Zhen Han, Mingming Sun, and Ping Li. 2022. https://doi.org/10.18653/v1/2022.findings-naacl.12 Multi-hop open-domain question answering over structured and unstructured knowledge . In Findings of the Association for Computational Linguistics: NAACL 2022, pages 151--15...

    2022 doi

  4. [12]

    Fomubad Borista Fondi and Azanzi Jiomekong Fidel. 2024. Integrating sparql and llms for question answering over scholarly data sources. arXiv preprint arXiv:2409.18969

    2024 arXiv

  5. [13]

    Mohamad Yaser Jaradeh, Markus Stocker, and S \"o ren Auer. 2020. https://link.springer.com/chapter/10.1007/978-3-030-54956-5_2 Question Answering on Scholarly Knowledge Graphs . In International Conference on Theory and Practice of Digital Libraries, pages 19--32. Springer

    2020 doi

  6. [14]

    Yoonjoo Lee, Kyungjae Lee, Sunghyun Park, Dasol Hwang, Jaehyeon Kim, Hong-in Lee, and Moontae Lee. 2023. Qasa: advanced question answering on scientific articles. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org

    2023

  7. [15]

    Jens Lehmann, Dhananjay Bhandiwad, Preetam Gattogi, and Sahar Vahdati. 2024. https://doi.org/10.1162/tacl_a_00671 Beyond Boundaries: A Human-like Approach for Question Answering over Structured and Unstructured Information Sources . Transactions of the Association for Computat...

    2024 doi

  8. [16]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\" u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\" a schel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proc...

    2020

  9. [17]

    Michael Ley. 2002. https://doi.org/10.1007/3-540-45735-6\_1 The DBLP computer science bibliography: Evolution, research issues, perspectives . In String Processing and Information Retrieval, 9th International Symposium, SPIRE 2002, Lisbon, Portugal, September 11-13, 2002, Proc...

    2002 doi

  10. [18]

    Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. 2022. Scienceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23(3):289--301

    2022

  11. [19]

    Kanchan Shivashankar and Nadine Steinmetz. 2024. https://arxiv.org/abs/2409.09010 Contri(e)ve: Context + retrieve for scholarly question answering . Preprint, arXiv:2409.09010

    2024 arXiv

  12. [20]

    Tilahun Abedissa Taffa and Ricardo Usbeck. 2023. https://ceur-ws.org/Vol-3592/paper5.pdf Leveraging llms in scholarly knowledge graph question answering . In Joint Proceedings of Scholarly QALD 2023 and SemREC 2023 co-located with 22nd International Semantic Web Conference ISW...

    2023

  13. [21]

    Lingxi Zhang, Jing Zhang, Xirui Ke, Haoyang Li, Xinmei Huang, Zhonghui Shao, Shulin Cao, and Xin Lv. 2023. A survey on complex factual question answering. AI Open, 4:1--12

    2023

  14. [22]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  15. [23]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.