Pith. sign in

REVIEW 5 major objections 5 minor 75 references

GraphRAG out-reasons triplet-only pipelines in a head-to-head QA test

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 19:21 UTC pith:HEF3GTKW

load-bearing objection Useful qualitative tool comparison, but the empirical ranking of GraphRAG over spaCy/CoreNLP is invalid because the pipelines are not matched. the 5 major comments →

arxiv 2509.09272 v1 pith:HEF3GTKW submitted 2025-09-11 cs.AI

Fusing Knowledge and Language: A Comparative Study of Knowledge Graph-Based Question Answering with LLMs

Vaibhav Chaudhary , Neha Soni , Narotam Singh , Amita Kapoor This is my paper
classification cs.AI
keywords knowledge graph question answeringretrieval-augmented generationtriplet extractionspaCyCoreNLP OpenIEGraphRAGLLM evaluationRepliQA
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper compares three ways of turning text into knowledge-graph triplets and feeding them to an LLM for question answering: spaCy, CoreNLP-OpenIE, and GraphRAG. It finds that GraphRAG produces the most coherent, well-reasoned answers on thematic and multi-hop questions, while CoreNLP-OpenIE extracts the broadest set of factual triplets and spaCy gives a clean, high-precision baseline. The authors argue that each method occupies a distinct point in a precision-coverage-reasoning trade-off, and they recommend a hybrid pipeline that combines all three. The result matters because teams building question-answering systems need to know whether to invest in graph construction, extraction breadth, or retrieval-and-reasoning infrastructure.

Core claim

The paper's central discovery is a three-way trade-off among knowledge-graph construction methods for LLM question answering. spaCy, using named-entity recognition and dependency rules, delivers clean, well-formed subject-predicate-object triplets with minimal false positives but limited coverage of idiomatic or complex sentences. CoreNLP-OpenIE extracts the broadest factual coverage, uncovering relations even in non-canonical phrasing, but produces noisy and overly granular triplets that can overwhelm the answering model. GraphRAG, which builds an LLM-generated entity-relation graph with community summaries and graph-aware retrieval, scores highest on thematic and multi-hop reasoning questi

What carries the argument

The comparison is carried by triplet extraction and graph-to-prompt integration. Triplets are subject-predicate-object facts extracted from sentences; spaCy uses linguistic dependency patterns, CoreNLP-OpenIE uses open information extraction, and GraphRAG uses an LLM to build a hierarchical entity-relation graph with community detection and automated summaries. The decisive machinery is GraphRAG's retrieval pipeline: it uses community-level summaries and dynamic query routing rather than passing flat triplets to the answering model, which is why its reasoning advantage appears on thematic and multi-hop questions.

Load-bearing premise

The comparison assumes that differences in answer quality are caused by the triplet-extraction method, but the three pipelines are not built alike: the triplet-based pipelines answer through a flat graph-question-answering chain, while GraphRAG uses its own LLM-generated graph with community summaries and retrieval, so architecture differences could explain the results.

What would settle it

Run the same documents through all three extraction methods, then feed each triplet set through the same answer-generation pipeline, and also feed the GraphRAG graph to that same pipeline. If GraphRAG's reasoning advantage disappears or shrinks, the paper's central ranking is not due to triplet quality. Alternatively, replace GraphRAG's community-summary retrieval with flat triplet prompts and see whether its scores drop toward those of the other two methods.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • For factual coverage, open information extraction yields the most triplets, but more triplets do not automatically improve LLM answers and can introduce noise.
  • For thematic and multi-hop questions, graph-structured context with community summaries produces higher-scoring answers than flat triplet prompts.
  • spaCy provides a lightweight, high-precision baseline but misses complex sentence structures and idiomatic phrasing.
  • Rubric guidance stabilizes weaker pipelines (spaCy and CoreNLP) more than GraphRAG, which is already robust without explicit rubric context.
  • A hybrid pipeline—spaCy for precision filtering, OpenIE for recall, GraphRAG for final reasoning—is the paper's recommended practical design.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reported GraphRAG advantage likely conflates triplet-construction quality with retrieval and prompting architecture; a matched comparison using the same graph and the same answer-generation pipeline is needed before concluding that LLM-built graphs are intrinsically better for reasoning.
  • If the architecture confound is real, teams using triplet-based graphs could gain similar reasoning improvements by adding community-summary retrieval to those graphs, without the higher cost of LLM-based graph construction.
  • The zero exact-match scores on RepliQA suggest current KG-QA pipelines are not ready for tasks demanding verbatim answers; improving answer-form control may matter as much as graph construction.
  • Rubric-induced shifts in lexical metrics hint that evaluation rubrics reward prose style over factual alignment; future benchmarks should separate content fidelity from writing style.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper compares three knowledge-graph-based question-answering pipelines built on spaCy, Stanford CoreNLP-OpenIE, and GraphRAG, all integrated with open-source LLMs. It reports a technical qualitative comparison across setup, customization, hardware, licensing, etc., and an empirical evaluation on a partial Shakespeare play and the RepliQA dataset, using expert and GPT-4 rubric-based scoring plus lexical metrics. The headline claim is that GraphRAG demonstrates superior reasoning abilities, while CoreNLP OpenIE provides the broadest triplet coverage and spaCy offers a lightweight high-precision baseline. The paper also recommends a hybrid pipeline and discusses future directions.

Significance. If the empirical claim were supported, the comparison would be practically useful for practitioners choosing KG construction methods for LLM-based QA. The qualitative comparison of tool characteristics is informative and partly valuable as a practical guide. The paper also has strengths: it uses open-source tools, attempts dual human/LLM evaluation, and includes detailed workflow descriptions. However, the central empirical claim is not supported by the evidence adduced. The study's sample size is tiny, the compared pipelines are not architecture-matched, the LLM-as-judge evaluation has known biases that the paper itself acknowledges, and one presented answer is factually hallucinated. Thus the paper's main contribution reduces to a qualitative survey with an unsupported comparative verdict.

major comments (5)
  1. [§3.1 vs §3.3–3.5] The paper claims in Section 3.1 that the comparison is conducted 'under identical conditions,' but the three pipelines are not architecture-matched. spaCy and CoreNLP use Langchain's GraphQAChain over triplets (Sections 3.3 and 3.4), whereas GraphRAG uses its own local response pipeline with LLM-generated graphs, community summaries, and different retrieval/prompting mechanisms (Section 3.5). Therefore the reported superiority of GraphRAG in Tables 11–16 could be due to retrieval, prompting, or answer-generation differences rather than to the KG construction method. This confound invalidates the paper's causal attribution of GraphRAG's higher scores to its triplet-extraction approach.
  2. [§4.2.1.1, Tables 11–14] The Shakespeare evaluation uses only four questions, and all four are near-identical paraphrases asking about the Helena–Bertram relationship. Scores are averaged over these four answers, with no error bars, no repeated runs, and no test of whether the 0.25–0.5 differences are meaningful. The reported Pearson correlation (r=0.415, p=0.043) is computed over a small and non-independent set of scores; it is not evidence for method-level differences. This sample size is too small to support the paper's comparative conclusions.
  3. [Appendix, CoreNLP response to Q1 with rubric] The appendix contains a CoreNLP with-rubric answer that repeatedly refers to 'A Midsummer Night's Dream,' 'Puck,' and a 'magical potion' instead of 'All's Well That Ends Well.' This is a clear factual hallucination, yet the paper does not flag it, analyze it, or exclude it. This indicates that the rubric-based scoring (both expert and GPT-4) can reward fluent, well-organized text despite severe factual inconsistency, undermining the validity of the scores in Tables 13–14 as measures of QA quality.
  4. [§4.2.1.2, Tables 15–16] The RepliQA evaluation reports metric scores without any variance or significance testing, and the GPT-4 evaluator scores show generated answers outscoring the dataset's ground-truth answers because the rubric rewards 'bigger and fancier answers' (as the paper itself notes). The same rubric is given to the answering LLM and to the GPT-4 evaluator, creating a self-reinforcing bias toward rubric-compliant verbosity. These issues make it impossible to interpret GraphRAG's higher QA-F1, ROUGE-L, and METEOR scores as evidence of superior reasoning.
  5. [Contribution 1, §3] The paper claims a 'fully reproducible framework,' but no code, configuration files, model versions, random seeds, or data splits are provided. The description of the spaCy workflow in Section 3.3 is especially underspecified ('Dependency patterns were defined'), and Section 3.2.1 says only 'the initial part' of the Shakespeare play was used, without quantifying that portion. A reader cannot reproduce or verify the experimental results.
minor comments (5)
  1. [§4.2] Section 4.2.2 appears to be missing; the paper jumps from 4.2.1 to 4.2.3. Re-number the subsections.
  2. [References] References [6] and [8] appear to be the same Ferrucci et al. 2010 'Building Watson' paper; also several citations in the text are not in the reference list or are numbered inconsistently.
  3. [Tables 11–14] The mapping of letter grades to numbers (A=5, B=4, C=3, D=2) and the averaging procedure are described only in prose; this should be stated after each table or in a dedicated evaluation-metrics subsection.
  4. [§4.1.2, Table 5] GitHub star/fork counts are presented as approximate values without a retrieval date or source. Such figures change quickly and should be cited or dated, or omitted.
  5. [Throughout] There are frequent typos and inconsistent capitalization ('spaCy' vs 'Spacy', 'SpaCy', etc.), and some figure captions are uninformative. The paper would benefit from a careful proofreading pass.

Circularity Check

0 steps flagged

No circularity: the paper's claims are empirical comparisons, not derivations; no fitted parameter is relabeled as a prediction, and no load-bearing self-citation is used.

full rationale

The paper is a comparative experimental study, not a derivation: it constructs knowledge graphs with spaCy, CoreNLP-OpenIE, and GraphRAG, then evaluates LLM answers with GPT-4 and an expert using score tables. The central claim that GraphRAG demonstrates superior reasoning is an empirical observation from those scores, not a quantity that reduces to its own inputs by construction. No fitted parameter is later reported as a prediction: the triplet counts and evaluation scores are measured, and the authors do not define the outcome in terms of the method choice. The methodology does differ across pipelines (spaCy and CoreNLP use Langchain's GraphQAChain over extracted triplets, while GraphRAG uses its own local response pipeline with LLM-generated graphs and community summaries, Sections 3.3-3.5), but this is a comparison-validity confound that could explain the reported GraphRAG advantage; it is not circularity. There is no self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in via citation. The use of GPT-4 as judge with a rubric shared with the answering prompt may bias scores, but that is an evaluation-bias concern rather than a definitional equivalence. Honest non-finding: no significant circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

The paper introduces no new entities or fitted parameters. Its conclusions rest on assumptions about evaluator validity, sample representativeness, and the comparability of pipelines that are not independently verified.

axioms (3)
  • domain assumption GPT-4 evaluation scores reflect the quality of KG integration and reasoning.
    The paper uses GPT-4 as an evaluator and treats its scores as evidence for method ranking (Section 4.2.1.1, Tables 11-14).
  • domain assumption The three pipelines differ only in KG construction method, not in QA architecture.
    SpaCy and CoreNLP use Langchain GraphQAChain; GraphRAG uses its own local response pipeline (Sections 3.3, 3.4, 3.5).
  • domain assumption The selected sample (one play excerpt, 10 documents) is representative of complex QA scenarios.
    Section 3.2.1 uses only the initial part of the play; Section 3.2.2 randomly selects 10 documents without a stated seed.

pith-pipeline@v1.3.0-alltime-deepseek · 36027 in / 10097 out tokens · 99255 ms · 2026-08-04T19:21:53.071417+00:00 · methodology

0 comments
read the original abstract

Knowledge graphs, a powerful tool for structuring information through relational triplets, have recently become the new front-runner in enhancing question-answering systems. While traditional Retrieval Augmented Generation (RAG) approaches are proficient in fact-based and local context-based extraction from concise texts, they encounter limitations when addressing the thematic and holistic understanding of complex, extensive texts, requiring a deeper analysis of both text and context. This paper presents a comprehensive technical comparative study of three different methodologies for constructing knowledge graph triplets and integrating them with Large Language Models (LLMs) for question answering: spaCy, Stanford CoreNLP-OpenIE, and GraphRAG, all leveraging open source technologies. We evaluate the effectiveness, feasibility, and adaptability of these methods by analyzing their capabilities, state of development, and their impact on the performance of LLM-based question answering. Experimental results indicate that while OpenIE provides the most comprehensive coverage of triplets, GraphRAG demonstrates superior reasoning abilities among the three. We conclude with a discussion on the strengths and limitations of each method and provide insights into future directions for improving knowledge graph-based question answering.

Figures

Figures reproduced from arXiv: 2509.09272 by Amita Kapoor, Narotam Singh, Neha Soni, Vaibhav Chaudhary.

Figure 1
Figure 1. Figure 1: spaCy workflow 3.4. Workflow for Question Answering using CoreNLP triplets Stanford CoreNLP’s Open Information Extraction (OpenIE) module was used to identify triplets from the text. The CoreNLP Java server was locally hosted. The processing pipeline starts with the tokenization of the text, does PoS tagging on the tokens, and lemmatizes them, followed by dependency parsing and natural language semantic an… view at source ↗
Figure 3
Figure 3. Figure 3: GraphRAG workflow 3.6. Evaluation Criteria The approaches are evaluated across multiple dimensions and compared across several parameters. 3.6.1. The Technical Comparative Study A comprehensive technical study was conducted across multiple parameters to compare spaCy, Stanford CoreNLP, and GraphRAG as distinct approaches for constructing knowledge graphs. The following are the broad parameters that are dec… view at source ↗
Figure 4
Figure 4. Figure 4: Scores comparison (a) Without rubric scores, (b) With rubric scores [PITH_FULL_IMAGE:figures/full_fig_p020_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

75 extracted references · 25 linked inside Pith

  1. [1]

    Baseball: an automatic question -answerer

    Green Jr, Bert F., et al. "Baseball: an automatic question -answerer." Papers presented at the May 9-11, 1961, western joint IRE-AIEE-ACM computer conference. 1961

    1961

  2. [2]

    The Lunar Sciences Natural Language Information System

    Woods, William. "The Lunar Sciences Natural Language Information System." BBN report (1972)

    1972

  3. [3]

    Natural language interfaces to databases–an introduction

    Androutsopoulos, Ion, Graeme D. Ritchie, and Peter Thanisch. "Natural language interfaces to databases–an introduction." Natural Language Engineering 1.1 (1995): 29-81

    1995

  4. [4]

    Shortliffe, E. H. (1977, October). Mycin: A knowledge -based computer program applied to infectious diseases. In Proceedings of the Annual Symposium on Computer Application in Medic al Care (p. 66)

    1977

  5. [5]

    Winograd, T. (1971). Procedures as a representation for data in a computer program for understanding natural language

    1971

  6. [6]

    and Schlaefer, N., 2010

    Ferrucci, D., Brown, E., Chu -Carroll, J., Fan, J., Gondek, D., Kalyanpur, A.A., Lally, A., Murdock, J.W., Nyberg, E., Prager, J. and Schlaefer, N., 2010. Building Watson: An overview of the DeepQA project. AI magazine, 31(3), pp.59-79

    2010

  7. [7]

    Allahyari, S

    M. Allahyari, S. Pouriyeh, M. Assefi, S. Safaei, E. D. Trippe, J. B. Gutierrez, and K. Kochut

  8. [8]

    Building Watson: An overview of the DeepQA project

    Ferrucci, David, et al. "Building Watson: An overview of the DeepQA project." AI magazine 31.3 (2010): 59-79

    2010

  9. [9]

    Answering Reading Comprehension Using Memo ry Networks

    Darshan Kapashi, Pararth Shah(2014)“Answering Reading Comprehension Using Memo ry Networks”, Technical report, Department of Computer Science, Stanford University

    2014

  10. [10]

    End to End Memory Networks

    Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, Rob Fergus(2015) “End to End Memory Networks”, Arxiv e-prints(2015) arXiv:1503.08895v5 [cs.NE]

    Pith/arXiv arXiv 2015

  11. [11]

    Ask Me Anything, Dynamic Memory Networks

    Ankit Kumar, Pete r Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, Richard Socher (2016)“Ask Me Anything, Dynamic Memory Networks”, Arxiv e - prints(2016)- arXiv:1506.07285v5 [cs.CL]

    Pith/arXiv arXiv 2016

  12. [12]

    Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., & Wu, X. (2024). Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering, 36(7), 3580-3599

    2024

  13. [13]

    Conceptnet 5.5: An open multilingual graph of general knowledge,

    R. Speer, J. Chin, and C. Havasi, “Conceptnet 5.5: An open multilingual graph of general knowledge,” in Proceedings of the AAAI conference on artificial intelligence, vol. 31, no. 1, 2017

    2017

  14. [14]

    The unified medical language system(umls): integrating biomedical terminology,

    O.Bodenreider, “The unified medical language system(umls): integrating biomedical terminology,” Nucleic acids research, vol. 32, no. suppl 1, pp. D267–D270, 2004

    2004

  15. [15]

    Imgpedia: a linked dataset with content -based analysis of wikimedia images,

    S. Ferrada, B. Bustos, and A. Hogan, “Imgpedia: a linked dataset with content -based analysis of wikimedia images,” in The Semantic Web–ISWC 2017. Springer, 2017, pp. 84–93

    2017

  16. [16]

    Mmkg: multi-modal knowledge graphs,

    Y. Liu, H. Li, A. Garcia -Duran, M. Niepert, D. Onoro -Rubio, and D. S. Ros enblum, “Mmkg: multi-modal knowledge graphs,” in The Semantic Web: 16th International Conference, ESWC 2019, Portoroˇz, Slovenia, June 2–6, 2019, Proceedings 16. Springer, 2019, pp. 459–474

    2019

  17. [17]

    H., & Ka ng, J

    Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Ka ng, J. (2020). BioBERT: a pre - trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4), 1234-1240

    2020

  18. [18]

    Beltagy, I., Lo, K., & Cohan, A. (2019). SciBERT: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676

    Pith/arXiv arXiv 2019

  19. [19]

    and Riedel, S., 2020

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.T., Rocktäschel, T. and Riedel, S., 2020. Retrieval -augmented generation for knowledge - intensive nlp tasks. Advances in neural information processing systems, 33, pp.9459-9474

    2020

  20. [20]

    and Dong, B., 2023

    Bian, N., Han, X., Sun, L., Lin, H., Lu, Y., He, B., Jiang, S. and Dong, B., 2023. Chatgpt is a knowledgeable but inexperienced solver: An investigation of commonsense problem in large language models. arXiv preprint arXiv:2303.16421

    Pith/arXiv arXiv 2023

  21. [21]

    and Do, Q.V., 2023

    Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W. and Do, Q.V., 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023

    Pith/arXiv arXiv 2023

  22. [22]

    Laskar, M. T. R., Bari, M. S., Rahman, M., Bhuiyan, M. A. H., Joty, S., & Huang, J. X. (2023). A systematic study and comprehensive evaluation of ChatGPT on benchmark datasets. arXiv p reprint arXiv:2305.18486

    Pith/arXiv arXiv 2023

  23. [23]

    Manakul, P., Liusie, A., & Gales, M. J. (2023). Selfcheckgpt: Zero -resource black -box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896

    Pith/arXiv arXiv 2023

  24. [24]

    and Zhou, D., 2022

    Wei, J., Wang, X., Schuurmans, D., Bosm a, M., Xia, F., Chi, E., Le, Q.V. and Zhou, D., 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35, pp.24824-24837

    2022

  25. [25]

    A., Debnath, B., & Chakradhar, S

    Arefeen, M. A., Debnath, B., & Chakradhar, S. (2024). L eancontext: Cost -efficient domain - specific question answering using llms. Natural Language Processing Journal, 7, 100065

    2024

  26. [26]

    Wu, Y., Hu, N., Bi, S., Qi, G., Ren, J., Xie, A., & Song, W. (2023). Retrieve -rewrite-answer: A kg-to-text enhanced llms framewor k for knowledge graph question answering. arXiv preprint arXiv:2309.11206

    Pith/arXiv arXiv 2023

  27. [27]

    F., & Saffari, A

    Baek, J., Aji, A. F., & Saffari, A. (2023). Knowledge -augmented language model prompting for zero-shot knowledge graph question answering. arXiv preprint arXiv:2306.04136

    Pith/arXiv arXiv 2023

  28. [28]

    K., Ding, B., Joty, S., Poria, S., & Bing, L

    Li, X., Zhao, R., Chia, Y. K., Ding, B., Joty, S., Poria, S., & Bing, L. (2023). Chain -of- knowledge: Grounding large language models via dynamic knowledge adapting over heterogeneous sources. arXiv preprint arXiv:2305.13269

    Pith/arXiv arXiv 2023

  29. [29]

    , Wang, S., Lin, C., Gong, Y., Ni, L.M., Shum, H.Y

    Sun, J., Xu, C., Tang, L. , Wang, S., Lin, C., Gong, Y., Ni, L.M., Shum, H.Y. and Guo, J., 2023. Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph. arXiv preprint arXiv:2307.07697

    Pith/arXiv arXiv 2023

  30. [30]

    F., Haffari, G., & Pan, S

    Luo, L., Li, Y. F., Haffari, G., & Pan, S. (2023). Reasoning on graphs: Faithful and interpretable large language model reasoning. arXiv preprint arXiv:2310.01061

    Pith/arXiv arXiv 2023

  31. [31]

    Mavromatis, C., & Karypis, G. (2024). Gnn-rag: Graph neural retrieval for large language model reasoning. arXiv preprint arXiv:2405.20139

    Pith/arXiv arXiv 2024

  32. [32]

    Li, Y., Li, Z., Wang, P., Li, J., Sun, X., Cheng, H., & Yu, J. X. (2023). A survey of graph meets large language model: Progress and future directions. arXiv preprint arXiv:2311.12399

    Pith/arXiv arXiv 2023

  33. [33]

    Text2mol: Cross-modal molecule retrieval with natural language queries

    Carl Edwards, ChengXiang Zhai, and Heng Ji. Text2mol: Cross-modal molecule retrieval with natural language queries. In EMNLP, pages 595–607, 2021

    2021

  34. [34]

    Prompt tuning on graphaugmented low -resource text classification

    Zhihao Wen and Yuan Fang. Prompt tuning on graphaugmented low -resource text classification. arXiv preprint arXiv:2307.10230, 2023

    Pith/arXiv arXiv 2023

  35. [35]

    Learning on large -scale textattributed graphs via variational inference

    Jianan Zhao, Meng Qu, Chaozhuo Li, Hao Yan, Qian Liu, Rui Li, Xing Xie, and Jian Tang. Learning on large -scale textattributed graphs via variational inference. arXiv preprint arXiv:2210.14709, 2022

    Pith/arXiv arXiv 2022

  36. [36]

    Graphformers: Gnn -nested transformers for representation learning on textual graph

    Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanj ay Agrawal, Amit Singh, Guangzhong Sun, and Xing Xie. Graphformers: Gnn -nested transformers for representation learning on textual graph. NeurIPS, 34:2879828810, 2021

    2021

  37. [37]

    Train your own gnn teacher: Graph -aware distillation on textual graphs

    Costas Mavromatis, Vassilis N Ioannidis, Shen Wang, Da Zheng, Soji Adeshina, Jun Ma, Han Zhao, Christos Faloutsos, and George Karypis. Train your own gnn teacher: Graph -aware distillation on textual graphs. arXiv preprint arXiv:2304.10668, 2023

    Pith/arXiv arXiv 2023

  38. [38]

    Pretraining language models with text- attributed heterogeneous graphs

    Tao Zou, Le Yu, Yifei Huang, Leilei Sun, and Bowen Du. Pretraining language models with text- attributed heterogeneous graphs. arXiv preprint arXiv:2310.12580, 2023

    Pith/arXiv arXiv 2023

  39. [39]

    Rajpurkar, P., Jia, R., & Liang, P. (2018). Know what you don't know: Unanswerable questions for SQuAD. arXiv preprint arXiv:1806.03822

    Pith/arXiv arXiv 2018

  40. [40]

    and Toutanova, K., 2019

    Kwiatkowski, T., Palomaki, J., Redf ield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K. and Toutanova, K., 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7, pp.453-466

    2019

  41. [41]

    Mihaylov, T., Clark, P., Khot, T., & Sabharwal, A. (2018). Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789

    Pith/arXiv arXiv 2018

  42. [42]

    W., Sa lakhutdinov, R., & Manning, C

    Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Sa lakhutdinov, R., & Manning, C. D. (2018). HotpotQA: A dataset for diverse, explainable multi -hop question answering. arXiv preprint arXiv:1809.09600

    Pith/arXiv arXiv 2018

  43. [43]

    Zhu, Y., Pang, L., Lan, Y., Shen, H., & Cheng, X. (2021). Adaptive information seeking for open-domain question answering. arXiv preprint arXiv:2109.06747

    Pith/arXiv arXiv 2021

  44. [44]

    Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2018). GLUE: A multi - task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461

    Pith/arXiv arXiv 2018

  45. [45]

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020). Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    Pith/arXiv arXiv 2020

  46. [46]

    and Zhang, H., 2023

    Zheng, L., Chiang, W.L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. and Zhang, H., 2023. Judging llm -as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, pp.46595-46623

    2023

  47. [47]

    Voorhees, E. M. (1999, November). The trec-8 question answering track report. In Trec (Vol. 99, pp. 77-82)

    1999

  48. [48]

    Katz, B. (1988). Using English for indexing and retrieving

    1988

  49. [49]

    Lenat and R

    Douglas B. Lenat and R. V. Guha. 1989. Building Large Knowledge -Based Systems; Representation and Inference in the Cyc Project (1st. ed.). Addi son-Wesley Longman Publishing Co., Inc., USA

    1989

  50. [50]

    Bollacker, C

    K. Bollacker, C. Evans, P. Paritosh, T. Sturge, and J. Taylor. Freebase: A Collaboratively Created Graph Database for Structuring Human Knowledge. In Proceedings of the 2008 ACM SIGMOD International Confer ence on Management of Data, SIGMOD'08, pages 1247 --1250, New York, NY, USA, 2008. ACM

    2008

  51. [51]

    Vrandečić, D., & Krötzsch, M. (2014). Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10), 78-85

    2014

  52. [52]

    (2007, November)

    Auer, S., Bizer, C., Kobilarov, G ., Lehmann, J., Cyganiak, R., & Ives, Z. (2007, November). Dbpedia: A nucleus for a web of open data. In international semantic web conference (pp. 722 -735). Berlin, Heidelberg: Springer Berlin Heidelberg

    2007

  53. [53]

    M., Kasneci, G., & Weikum, G

    Suchanek, F. M., Kasneci, G., & Weikum, G. (2007, May). Yago: a core of semantic knowledge. In Proceedings of the 16th international conference on World Wide Web (pp. 697-706)

    2007

  54. [54]

    (2010, July)

    Carlson, A., Betteridge, J., Kisiel, B., Settles, B., Hruschka, E., & Mitchell, T. (2010, July). Toward an architecture for never-ending language learning. In Proceedings of the AAAI conference on artificial intelligence (Vol. 24, No. 1, pp. 1306-1313)

    2010

  55. [55]

    (2011, February)

    Nakashole, N., Theobald, M., & Weikum, G. (2011, February). Scalable knowledge harvesting with high precision and hi gh recall. In Proceedings of the fourth ACM international conference on Web search and data mining (pp. 227-236)

    2011

  56. [56]

    and Zhang, W., 2014, August

    Dong, X., Gabrilovich, E., Heitz, G., Horn, W., Lao, N., Murphy, K., Strohmann, T., Sun, S. and Zhang, W., 2014, August. Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 601-610)

    2014

  57. [57]

    W., Lee, K., & Toutanova, K

    Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019, June). Bert: Pre -training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) (pp. 4171-4186)

    2019

  58. [58]

    Introducing Chatgpt | openai. OpenAI. (2022, November 30). https://openai.com/index/chatgpt/

    2022

  59. [60]

    Your goal is to create a response that would achieve the highest score possible (A) in each of these categories

    Appendix Prompt for the answering LLM:- You are an expert writer tasked with crafting an answer that will be evaluated based on the following four factors: Content/Ideas, Organization, Style, and Mechanics. Your goal is to create a response that would achieve the highest score possible (A) in each of these categories. Writing Criteria for Top Scores (A):

  60. [61]

    ○ Present insightful, original, and well-developed ideas with comprehensive detail

    Content/Ideas: ○ Demonstrate deep understanding and thorough analysis of the question. ○ Present insightful, original, and well-developed ideas with comprehensive detail. ○ Address all parts of the question fully, providing evidence or examples as needed. ○ Ensure the response follows all literary or academic conventions relevant to the topic

  61. [62]

    ○ Ensure smooth transitions between ideas, keeping the answer focused and coherent

    Organization: ○ Structure the response logically, with a clear introduction, well-organized body paragraphs, and a strong conclusion. ○ Ensure smooth transitions between ideas, keeping the answer focused and coherent. ○ Integrate any quotations, examples, or evidence seamlessly into the discussion

  62. [63]

    ○ Enhance the clarity and impact of the content through effective word choice and sentence flow

    Style: ○ Write in an engaging, sophisticated tone, using varied sentence structures and precise, appropriate vocabulary. ○ Enhance the clarity and impact of the content through effective word choice and sentence flow. ○ Follow the appropriate citation style (e.g., MLA format) without errors

  63. [64]

    ○ Polish the writing so it is clean and professional, with no distracting mistakes

    Mechanics: ○ Ensure your response is free from grammar, punctuation, and spelling errors. ○ Polish the writing so it is clean and professional, with no distracting mistakes. Instructions for Writing the Response:

  64. [65]

    Carefully read the question and think about how to address each part fully, while adhering to the highest standards for each of the four factors mentioned

  65. [66]

    Organize your thoughts clearly before you begin writing, ensuring a logical flow of ideas

  66. [67]

    Use a refined and varied writing style to make your response engaging and clear

  67. [68]

    Proofread your answer to eliminate any errors in grammar, punctuation, or spelling. Write your response to the following question, keeping in mind the criteria for achieving an "A" in Content/Ideas, Organization, Style, and Mechanics: Question: Evaluator Prompt:- You are an expert evaluator responsible for grading responses based on four factors: Content/...

  68. [69]

    Fresh, original concepts with comprehensive coverage of the question

    Content/Ideas: ○ A: Demonstrates deep understanding, insightful ideas, and thorough analysis. Fresh, original concepts with comprehensive coverage of the question. ○ B: Shows solid understanding, with good ideas but may lack depth or originality. Mostly accurate but less detailed. ○ C: Shows partial understanding. Ideas are incomplete, superficial, or som...

  69. [70]

    All ideas are relevant and well-integrated

    Organization: ○ A: Logically organized with smooth transitions and a strong introduction and conclusion. All ideas are relevant and well-integrated. ○ B: Generally organized but may lack clarity in some transitions or sections. The structure is clear, but the introduction or conclusion could be stronger. ○ C: Basic structure with unclear flow. Ideas may j...

  70. [71]

    The style enhances clarity

    Style: ○ A: Engaging and sophisticated tone, varied sentence structure, and appropriate vocabulary. The style enhances clarity. ○ B: Consistent tone and vocabulary, though lacks stylistic variety. Clear but not as engaging. ○ C: Basic and repetitive, lacking refinement. Understandable but not compelling. ○ D: Awkward or unclear, with inappropriate tone or...

  71. [72]

    ○ B: A few minor errors, but not distracting

    Mechanics: ○ A: Minimal or no grammar, punctuation, or spelling errors. ○ B: A few minor errors, but not distracting. ○ C: Some significant errors, occasionally distracting. ○ D: Frequent and severe errors, making it hard to understand. Grading Instructions:

  72. [73]

    Review the response using the four factors above

  73. [74]

    Assign a grade (A, B, C, or D) for each factor

  74. [75]

    All \'s Well That Ends Well,

    Provide an overall rating with a brief explanation based on the overall performance across all factors. Expected Output: ● Content/Ideas: B ● Organization: A ● Style: B ● Mechanics: A ● Overall Rating: B Explanation: The response shows a good understanding with clear organization. However, some ideas are underdeveloped, preventing an A in Content/Ideas, a...

  75. [2017]

    ArXiv e-prints (2017)

    Text Summarization Techniques: A Brief Survey. ArXiv e-prints (2017). arXiv:1707.02268

    Pith/arXiv arXiv 2017