{"id":"57b3148f-389a-49d1-9f23-9fe66c742c22","arxiv_id":"2412.02788","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A 10.5K-question benchmark for scholarly QA forces systems to combine DBLP and SemOpenAlex knowledge graph facts with Wikipedia text.","lead":"Hybrid-SQuAD is a new question-answering dataset with 10.5K questions built from scholarly knowledge graphs (DBLP, SemOpenAlex) and Wikipedia text, designed so that answers require combining structured and unstructured sources. A retrieval-augmented baseline reaches 69.65% exact match, while zero-shot ChatGPT-3.5 scores only 2.6%, suggesting the dataset is hard for current LLMs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold answers are generated by ChatGPT-3.5 with only a 100-question category check (§4.1), not accuracy validation; incorrect or unanswerable gold questions would invalidate both the 69.65 EM baseline and the 2.6 EM zero-shot claim.","rationale":"I agree with the reader's weakest_assumption. The paper's main contribution is the 10.5K-pair dataset; if the gold labels are unreliable, the dataset cannot be used as a benchmark, and the baseline numbers lose their meaning. The authors' own §3.2.2 reveals that initial generation produced malformed outputs, and the fix (re-prompting) was not accompanied by a correctness audit. The 100-question analysis in §4.1 is a categorization exercise, not a validity check. The proposed test—manual verification of a stratified sample on the exact source data—would directly estimate the gold error rate. If it is low, the conditional accept can proceed; if high, the core claim fails until the dataset is cleaned. I also note a secondary concern: §5.1 says the entity linker falls back to the author URI provided with the question, meaning the baseline is given the gold entity; this inflates the 69.65 EM number and would need a separate fix (re-run without gold URIs). However, the gold-answer validity is more fundamental because it affects the dataset itself, not just the baseline. Thus verdict remains CONDITIONAL.","tokens_in":9809,"tokens_out":6475,"duration_ms":63580,"concrete_test":"Select a random sample of 200 QA pairs from the released Hybrid-SQuAD JSON, stratified by the four traversal types in Table 3. For each pair, two independent annotators verify the gold answer against the exact sources used in §3.1 (authors should release the data snapshot; otherwise use versioned dumps from the same dates). Each annotator records: (a) is the question answerable from the cited sources, and (b) is the gold answer correct/complete? Compute inter-annotator agreement and the gold error rate with a 95% Wilson interval. If the error rate exceeds 5%, or if inter-annotator agreement is below 0.8, the unvalidated gold set is a real threat to every reported EM number, and the paper should require re-validation/cleaning before acceptance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The dataset's gold QA pairs are produced entirely by ChatGPT-3.5 (§3.2.2). The only quality control reported is (i) a random check that caught missing/truncated answers, which were re-prompted, and (ii) a manual analysis of 100 questions that classified answer categories (§4.1, Table 2). Neither step verifies that the generated answer is correct, complete, or even answerable from the provided DBLP, SemOpenAlex, and Wikipedia sources. The paper itself states that the initial generation produced questions 'lacking an answer' or with single-letter responses, so the generation pipeline is demonstrably error-prone. If even 5–10% of the 10,581 gold answers are wrong, then the central benchmark claim collapses: the RAG baseline's 69.65 EM and ChatGPT-3.5's 2.6 EM measure agreement with an unvalidated LLM output, not hybrid KG+text reasoning ability. Because the entire paper is a dataset contribution, this is the load-bearing assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Hybrid-SQuAD, a dataset of 10,581 scholarly question-answer pairs that require integrating facts from DBLP, SemOpenAlex, and Wikipedia. The data is generated by ChatGPT-3.5 from per-author records that combine KG triples and Wikipedia text, and the authors identify four evidence-traversal pathways: KG-to-KG bridge, KG-to-KG comparison, KG-to-text, and KG-to-KG-to-text. The paper also proposes a RAG-based baseline with a link-retrieve-generate pipeline, reporting 69.65% exact match on the test set, while zero-shot ChatGPT-3.5 obtains only 2.6% EM. The central claim is that Hybrid-SQuAD is the first large-scale scholarly QA benchmark that forces hybrid KG+text reasoning.","tokens_in":10140,"tokens_out":2850,"duration_ms":32573,"significance":"If the dataset is accurate and the gold answers are correct, Hybrid-SQuAD would fill a real gap: existing scholarly QA datasets such as DBLP-QuAD, SciQA, and QASA are single-source, and general hybrid QA benchmarks like CompMix do not require cross-source reasoning. The paper's strengths include the public release of a 10.5K dataset, a concrete taxonomy of four traversal pathways, and a reproducible-style baseline with both ChatGPT-3.5 and LLaMA generators. The zero-shot numbers, if trustworthy, demonstrate that the dataset is not trivially answerable from parametric knowledge. However, the benchmark's value hinges on the correctness and answerability of the LLM-generated gold answers, and that assumption is not adequately validated in the current manuscript.","major_comments":[{"comment":"The gold answers are generated entirely by ChatGPT-3.5, and the only reported quality check is a manual analysis of 100 questions that categorizes answer types (Table 2) rather than verifying correctness, completeness, or answerability from the provided sources. The paper itself states that the initial generation produced questions lacking an answer or with single-letter responses, so the pipeline is demonstrably error-prone; re-prompting to replace incomplete answers does not establish that the final gold answers are correct. Since the paper's central contribution is a benchmark, the 69.65 EM baseline and the 2.6 EM zero-shot result measure agreement with an unvalidated LLM output unless a human-annotation study confirms the gold answers. The authors should run a human evaluation on a random sample (with multiple annotators, measuring inter-annotator agreement) to report the share of questions that are answerable and the share of gold answers that are correct, and they should release any corrected gold answers or filter out unanswerable questions.","section":"Section 3.2.2 and Section 4.1"},{"comment":"The comparison with Efeoglu et al. is not valid as presented because footnote 20 states that this model was evaluated on a different test split. Reporting an EM score of 48.9 for Efeoglu et al. next to 69.65 for the proposed model in the same table implies a direct comparison, but different splits make the difference uninterpretable. The authors should either evaluate Efeoglu et al. on the same Hybrid-SQuAD test split or clearly separate reported results by split and avoid claiming superiority on the basis of incomparable numbers.","section":"Table 4 and footnote 20"},{"comment":"The baseline uses ChatGPT-3.5 both for sub-question extraction and entity identification in the linking phase and for final answer generation, while ChatGPT-3.5 was also the model that generated the question-answer pairs. This creates a potential style-matching bias: the baseline may achieve high EM because it reproduces the answer format and phrasing distribution of the generator rather than because it performs genuine hybrid reasoning. The zero-shot result (2.6 EM) mitigates this concern but does not eliminate it, since the retrieval and prompting pipeline still relies on the same model. The authors should report an analysis of answer-format agreement between generated questions and baseline predictions, and ideally validate a sample of baseline predictions against human judgments of correctness.","section":"Section 5.1 and Section 5.3"}],"minor_comments":[{"comment":"There is a typo in the sentence beginning 'Unlike bridging questions, for KG-KG comparison questions, , the data source pool' where an extra comma appears after 'questions'.","section":"Section 3.2.1"},{"comment":"The text refers to a fine-tuned Flan-T5-Large model, but the footnote links to 'google/flan-t5-small'; please correct this inconsistency so the model size is unambiguous.","section":"Footnote 10 and Section 6"},{"comment":"The table lists 'Fondi and Fidel 2024' while the reference list gives 'Fondi and Jiomekong Fidel'; use a consistent author name format in the table, text, and bibliography.","section":"Table 4 and References"},{"comment":"The model name is written inconsistently as 'LLAMA', 'LLaMA', and 'LLAMA3.0'; choose one canonical spelling and apply it consistently.","section":"Throughout"},{"comment":"The paper does not specify the exact train/test split sizes, the random seed used for splitting, or the number of authors/entities covered; adding these statistics would improve reproducibility.","section":"Section 3 and Section 6"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the dataset contribution is not yet backed by sufficient gold-answer validation. I would encourage the editor to ask for a human-evaluation study on a random sample with answerability and correctness judgments, and for a corrected comparison table that respects the same test split. If the authors provide those, the paper could become a solid benchmark contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hybrid-SQuAD is a real contribution: a 10.5K QA dataset that forces systems to combine DBLP, SemOpenAlex, and Wikipedia, and the first scholarly benchmark designed so that single-source retrieval is not enough. The four traversal paths are well defined, the examples are convincing, and the zero-shot numbers (2.6 EM) make the case that the dataset is not trivially solvable from parametric memory. The dataset is public, which is a plus.\n\nThe main soft spot is the gold answers. All 10,581 QA pairs are generated by ChatGPT-3.5, and the paper's validation is a 100-question category check, not an accuracy check. The paper itself mentions initial prompts producing missing or truncated answers, so the pipeline is demonstrably error-prone. If even a moderate fraction of gold answers are wrong or unanswerable from the provided sources, the 69.65 EM baseline is measuring agreement with an unvalidated LLM rather than hybrid reasoning. That is the load-bearing assumption, and it needs either a human-validated sample with inter-annotator agreement or at least an error analysis.\n\nSecond issue: Table 4 compares Efeoglu et al.'s model \"evaluated on a different test split\" (footnote). That makes the comparison invalid as reported. The authors should re-evaluate on the same split or clearly separate the numbers. Third, no error bars or significance testing on the EM/F1 differences; minor but worth adding.\n\nI don't see a circularity problem with the baseline using the same model (ChatGPT-3.5) that generated the questions; the retrieval still has to find the right evidence, and the zero-shot results show the model doesn't already know the answers.\n\nWho is this for? Anyone working on scholarly QA, hybrid QA, or RAG. It deserves a serious referee, but it needs revision before acceptance. The dataset and the idea are solid; the evaluation and validation need to catch up.","headline":"A genuinely new scholarly hybrid-QA benchmark with a real gold-validation gap; worth reviewing but needs revision.","tokens_in":10578,"tokens_out":1338,"would_cite":true,"duration_ms":12549,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hybrid-SQuAD is a 10.5K-question scholarly QA benchmark whose answers require combining DBLP, SemOpenAlex, and Wikipedia facts.","keywords":["hybrid question answering","scholarly knowledge graphs","retrieval-augmented generation","DBLP","SemOpenAlex","question answering dataset","LLM-generated dataset","multi-source reasoning"],"falsifier":"Manually answer a random sample of 200 test-set questions using only the released source data; if a substantial fraction of gold answers are wrong or unanswerable, then the reported 69.65 EM measures something other than hybrid QA.","tokens_in":9622,"feed_emoji":"📚","tokens_out":4264,"duration_ms":40145,"temperature":0.7,"pith_summary":"The paper introduces Hybrid-SQuAD, a 10,581-question benchmark for scholarly question answering in which every question can only be answered by combining facts from two knowledge graphs (DBLP and SemOpenAlex) with Wikipedia text. Existing scholarly QA datasets draw on a single source, so they cannot test systems that must fuse structured and unstructured evidence. The authors claim this is the first large-scale scholarly QA dataset that forces hybrid KG-plus-text reasoning, and they report a retrieval-augmented baseline reaching 69.65 exact match while ChatGPT-3.5 in a zero-shot setting scores 2.6. If the dataset is sound, it gives the field a concrete way to measure multi-source scholarly QA.","feed_headline":"Benchmark forces AI to fuse knowledge graphs and Wikipedia","feed_subtitle":"10.5K-question scholarly QA set: retrieval-augmented model hits 69.65 EM, zero-shot ChatGPT only 2.6.","key_machinery":"The load-bearing object is the bridging entity: each question embeds a sub-question phrase that must be resolved to a scholarly entity (author, publication, or institution) before evidence can be gathered. The baseline resolves these phrases with ChatGPT-3.5 prompts and SPARQL templates over DBLP, links authors to SemOpenAlex through ORCID, then retrieves Wikipedia text and KG triples; a retrieval-augmented generation (RAG) generator, using FAISS over 200-word chunks embedded with bge-small-en-v1.5 and answered by ChatGPT-3.5 or LLAMA-3-8B, produces the final answer.","core_discovery":"The central claim is that hybrid scholarly QA can be benchmarked at scale: the authors construct 10.5K question-answer pairs by prompting ChatGPT-3.5 with paired inputs from DBLP, SemOpenAlex, and Wikipedia, organize the questions into four evidence-traversal paths (KG-to-KG bridge, KG-to-KG comparison, KG-to-text, and KG-to-KG-to-text), and show that a RAG pipeline that links entities, retrieves Wikipedia chunks, and queries KG triples reaches 69.65 exact match and 74.91 F1 on the test set. The same questions answered from the model's internal knowledge alone yield 2.6 EM. The dataset and baseline are offered as the starting point for scholarly hybrid QA research.","pith_inferences":["Editorial inference: because the gold answers were produced by ChatGPT-3.5, the dataset may contain answer patterns or artifacts that retrieval models exploit; a human re-annotation of a larger sample would show whether the 69.65 EM reflects hybrid reasoning or surface regularities.","Editorial inference: the pipeline of pairing KG snapshots with Wikipedia text could be transplanted to other domains, such as biomedical or legal question answering, where facts are split between structured databases and prose.","Editorial inference: the dataset's selection criteria (authors with ORCID and Wikipedia pages) skew toward well-documented computer scientists, so performance may not transfer to less-documented scholars or to other academic fields."],"forward_implications":["Single-source scholarly QA benchmarks no longer suffice as evidence of general QA ability; Hybrid-SQuAD provides a 10.5K-question test that requires fusion of structured and unstructured evidence.","The roughly 67-point gap between zero-shot and RAG performance quantifies how much retrieval contributes when answers are not stored in model weights.","The four traversal-path types give a diagnostic: systems can be scored separately on bridge, comparison, KG-text, and KG-KG-text questions.","Reported alternative systems (32.0 to 48.9 exact match) all fall below the RAG baseline, suggesting the retrieve-then-generate recipe is currently the most effective documented approach on this benchmark.","The dataset can be reused to evaluate entity linking and sub-question decomposition in isolation, not only end-to-end QA."],"supporting_citations":[{"why":"Provides DBLP-QuAD, the single-source scholarly KG QA dataset that Hybrid-SQuAD extends and contrasts with.","marker":"Banerjee et al., 2023"},{"why":"Describes SemOpenAlex, the knowledge graph from which author and institution facts are drawn.","marker":"Färber et al., 2023"},{"why":"Defines retrieval-augmented generation, the architecture used by the baseline model.","marker":"Lewis et al., 2020"},{"why":"CompMix is the heterogeneous QA benchmark compared against; Hybrid-SQuAD differs by requiring reasoning across sources.","marker":"Christmann et al., 2024"},{"why":"Introduces DBLP, the bibliographic knowledge graph that provides publication and author data.","marker":"Ley, 2002"},{"why":"A prior hybrid scholarly QA system whose 48.9 EM result is one of the baselines Hybrid-SQuAD's RAG model surpasses.","marker":"Efeoglu et al., 2024"}],"fun_headline_variants":["10.5K Q&A benchmark forces AI to fuse text and KGs","Scholarly QA test: RAG hits 69.65 EM, ChatGPT only 2.6","New hybrid QA dataset spans Wikipedia and knowledge graphs","Benchmark for scholarly QA: text plus KG, 10.5K questions","RAG-based QA scores 69.65 on hybrid scholarly dataset"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gold answers are correct and actually derivable from the stated sources; only 100 of the 10,581 questions were manually inspected, and that check covered answer categories rather than answer accuracy.","fun_headline_variants_meta":{"raw":{"variants":["10.5K Q&A benchmark forces AI to fuse text and KGs","Scholarly QA test: RAG hits 69.65 EM, ChatGPT only 2.6","New hybrid QA dataset spans Wikipedia and knowledge graphs","Benchmark for scholarly QA: text plus KG, 10.5K questions","RAG-based QA scores 69.65 on hybrid scholarly dataset"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000166,"raw_usage":{"total_tokens":1200,"prompt_tokens":837,"completion_tokens":363,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":453,"completion_tokens_details":{"reasoning_tokens":263}},"tokens_in":453,"tokens_out":363,"duration_ms":4347,"temperature":1.0,"reasoning_tokens":263,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:05:46.975575+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually answer a random sample of 200 test-set questions using only the released source data; if a substantial fraction of gold answers are wrong or unanswerable, then the reported 69.65 EM measures something other than hybrid QA.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides DBLP-QuAD, the single-source scholarly KG QA dataset that Hybrid-SQuAD extends and contrasts with."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A prior hybrid scholarly QA system whose 48.9 EM result is one of the baselines Hybrid-SQuAD's RAG model surpasses."}],"review_version":1}