Pith. sign in

REVIEW 4 major objections 5 minor 92 references

This paper claims that a topic-specific knowledge graph, built once, can be reused to generate multiple-choice question sets with controlled difficulty at low cost, and that model rankings on these sets match established benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 21:23 UTC pith:VWCDTXIY

load-bearing objection A useful integrated MCQ-generation pipeline whose main quality claims hold, but the 'adaptive hardness calibration' framing oversells a depth-as-difficulty proxy that is only weakly validated. the 4 major comments →

arxiv 2602.20135 v1 pith:VWCDTXIY submitted 2026-02-23 cs.CL cs.AIcs.IR

KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibration

Mohammad Amanlou , Erfan Shafiee Moghaddam , Yasaman Amou Jafari , Mahdi Noori , Farhan Farsi , Behnam Bahrak This is my paper
classification cs.CL cs.AIcs.IR
keywords multiple-choice question generationknowledge graphsdifficulty calibrationmulti-hop reasoningretrieval-augmented generationLLM evaluationdistractor generationbenchmark generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

KNIGHT is a pipeline that, given a topic, retrieves source text, distills it into a compact knowledge graph of entities and relations, and then generates four-option multiple-choice questions by walking paths of increasing length through that graph. The paper's central claim is that this graph acts as a reusable, compressed representation: once built, it can be queried repeatedly to produce many question variants at different difficulty levels without re-feeding the full source documents. The authors argue that path depth is an intrinsic hardness knob—longer paths yield harder, multi-hop questions—and that this difficulty signal survives validation, matching human error patterns and model uncertainty. They further claim that models ranked by accuracy on KNIGHT-generated sets are ordered the same way as on established broad MCQ benchmarks, supporting KNIGHT's use as a cheap, refreshable benchmark generator.

Core claim

The central claim is that knowledge-graph-guided generation, not retrieval or validation alone, is what produces competitive distractors and controlled difficulty. In their staged comparison, plain and retrieval-only prompting yield near-zero answer entropy and near-ceiling probe accuracy—distractors are trivially wrong. Adding a validator improves item validity but not difficulty; only conditioning on multi-hop graph paths raises entropy, lowers probe accuracy, and separates Level 1 from Level 3. The full KNIGHT pipeline combines this KG guidance with a five-criterion validator, yielding the lowest rates of duplicate, ambiguous, and unanswerable items while preserving the difficulty signal.

What carries the argument

The central object is the depth-bounded topic knowledge graph G=(V,E), built by breadth-first expansion from a seed topic up to a user-chosen depth d_max, with each node carrying a source-validated description and each edge a typed relation. Question generation samples length-d paths (forward or reverse) and verbalizes them into MCQ stems, so d controls the number of reasoning hops. The supporting mechanism is the validator: an LLM-as-critic that enforces five criteria—grammar, single correct key, option uniqueness, answerability from the supplied evidence, and topic relevance. A fixed probe model's predictive entropy over answer choices serves as an automatic, model-agnostic measure of dist

Load-bearing premise

The framework equates question difficulty with knowledge-graph path depth: one hop means Level 1 and three hops mean Level 3. If path length is not a reliable proxy for how hard a question actually is—across different subjects or once linguistic complexity and prior knowledge are factored in—the difficulty-calibration claim, and with it much of the framework's value, fails.

What would settle it

Run KNIGHT on a calculation-heavy subject (e.g., physics or numerical computation) and compare Level 1 vs Level 3 items on human error rates and model accuracy. If Level 3 items are not consistently harder than Level 1, or if the graph-grounding step collapses because facts are not entity-relation triples, then the depth-as-hardness premise is falsified. A more direct test: measure whether human error rates and probe entropy increase monotonically with path length within a single domain, controlling for question length and vocabulary.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A topic KG built once can be reused to generate many question sets at different difficulty levels, hop lengths, and question formats, so marginal cost per additional dataset is small.
  • Level 3 items are consistently harder than Level 1 for both humans and a range of models, and the increase in difficulty does not come with a rise in validity violations.
  • Rankings of models on KNIGHT datasets align with rankings on established broad MCQ suites, suggesting the generated sets capture general QA competence rather than topic-specific quirks.
  • KG-guided prompting is the component that creates plausible distractors; retrieval grounding alone and validation alone do not, which isolates where the value comes from.
  • The five-criteria validator cuts duplicate, ambiguous, and source-unsupported items, and answerability violations serve as a usable proxy for hallucination during generation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, if path depth is a reliable hardness proxy, the same cached graph could drive adaptive test generation, choosing hop length from a learner's or model's observed accuracy in real time.
  • An unstated consequence of the reusable-graph design is testing integrity: new items drawn from different paths do not reuse stems, so refreshable assessments are less vulnerable to memorization.
  • The reported correlation between predictive entropy and human difficulty (r≈0.78) suggests entropy could serve as an automatic calibration signal, enabling closed-loop difficulty tuning without human annotation.
  • The authors' own limitation about low-relational-density domains implies the representation will need hybrid extensions—attaching formulas or procedural steps to nodes—before it transfers to calculation-heavy subjects like physics.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces KNIGHT, an LLM-based framework that constructs a topic-specific knowledge graph from retrieved Wikipedia/Wikidata evidence and then generates multiple-choice questions by traversing graph paths of configurable depth. A five-criterion LLM validator filters the generated items, and the authors report six MCQ datasets across history, biology, and mathematics at two difficulty levels, together with ablations against Plain, RAG, RAG+KG, and RAG+Val baselines. The evaluation covers human audits, automatic fluency checks, topic-relevance scores, predictive-entropy-based distractor analysis, and downstream model ranking compared with MMLU-style benchmarks. The central claims are that the reusable KG representation enables token- and cost-efficient generation, that generated items are high quality on five criteria, and that difficulty-controlled evaluation with model rankings aligned to established benchmarks is achieved.

Significance. If substantiated, KNIGHT would be a practically useful contribution: a topic KG built once and reused for many MCQ-generation runs is a plausible route to low-cost, refreshable, syllabus-aligned evaluation sets, and the released code/PyPI package supports reproducibility. The paper has real strengths: six human audits with n=100 per system, a staged component ablation separating retrieval, KG structure, and validation, an explicit five-criterion validation protocol, and an appendix (E.4) with careful significance testing for topic relevance. However, the manuscript currently overstates the evidence for its two headline capabilities: difficulty control via graph depth, and 'adaptive' hardness calibration. The load-bearing difficulty-calibration premise is supported only by aggregate correlations with no control for confounds, and no adaptive feedback loop is implemented. The topic-relevance presentation also conflicts with the paper's own Table 3.

major comments (4)
  1. [§3.1, Appendix G.2 Eq. (12), Appendix A] The paper's central difficulty-control claim rests on treating KG path depth as an intrinsic hardness knob: Appendix G.2 states 'Depth therefore acts as an intrinsic hardness knob' (Eq. 12), and Section 3.1 uses dmax as the only direct difficulty control. Yet the empirical evidence is correlational: Table 4 shows entropy/probe-accuracy shifts between Level 1 and Level 3, Table 5 shows accuracy drops, and Appendix E.3 reports r≈0.78 between human error rates and model entropy over six aggregate data points. None of these analyses controls for the confounds that naturally accompany longer KG paths, such as rarer entities, more complex syntax, or higher named-entity density. The paper's own limitation (Appendix A) concedes that difficulty is multi-factorial. A convincing demonstration would need, for example, a within-path manipulation that varies depth while holding surface form roughly co
  2. [Title, Abstract, §6] The title and abstract promise 'Adaptive Hardness Calibration,' but no adaptive mechanism is described or evaluated. The only difficulty control is an a priori depth parameter dmax (Algorithm 1) followed by post-generation validation; there is no feedback loop from human or model performance to generation parameters. Section 6 explicitly lists 'incorporating adaptive difficulty tuning via model feedback' as future work. This is a mismatch between the claimed contribution and the implemented system. Either the title/abstract should be revised to describe static difficulty control with post-hoc validation, or an actual calibration loop with evidence of convergence/adaptation should be added.
  3. [§4.3.3, Table 3, Appendix E.4] The main text states that KNIGHT 'maintains strong topical alignment across topics and difficulty levels' and that off-topic rates are low, but Table 3 shows that KNIGHT is consistently worse than RAG+Val on both automatic topic-relevance metrics and the off-topic intersection rate: for example, History L1 off-topic rate is 10.6% for KNIGHT vs 3% for RAG+Val; Biology L1 5.5% vs 1%; Math L1 7.3% vs 4%. Entailment and LLM relevance scores are also lower for KNIGHT in most splits. Appendix E.4's significance tests compare KNIGHT only against a ~100-item GPT-4o-mini baseline, not against RAG+Val, and they do not cover the human TOPIC flags or the 'LLM∩Entailment' off-topic rate. Therefore the paper's claim that KNIGHT achieves high quality across all five criteria is not supported for the topic-relevance criterion in the comparison that matters. Please report the RAG+Val vs KNIGHT comparison
  4. [§4.3.2, Table 2; §5.1, Table 4] Several headline claims rest on small count differences or aggregate statistics without confidence intervals or significance tests. In Table 2, KNIGHT's advantage over RAG+Val on SINGLE_KEY, OPTION_UNIQUENESS, and ANSWERABLE is often only 1–4 flags per 100 items, and no uncertainty is reported. In Table 4, the entropy/probe-accuracy differences that drive the central difficulty and distractor-competition conclusions are reported as single numbers with no error bars, and Appendix E.3's human-entropy correlation is computed over only six aggregate points. Given that these tables carry the paper's main empirical claims, the authors should provide per-split confidence intervals, and where possible significance tests or effect sizes, for the key KNIGHT-vs-baseline comparisons.
minor comments (5)
  1. [Appendix G.3–G.4] The text under G.3 and G.4 largely duplicates Section G.1–G.2 (e.g., the formal definition of node curation appears twice, as Eq. 11 and Eq. 13). Please remove the duplicate material and retain one consolidated curator description.
  2. [References] Several references are duplicated or near-duplicated in the bibliography: [15] and [31] are the same work, and [22] and [42] are the same Wikidata paper. Please de-duplicate.
  3. [Appendix G.5] The claim that 'reverse questions increase model entropy by 15–20% (Section 4)' is not reported in Section 4. Please either point to the specific table/figure or add the supporting result.
  4. [Table 5] The row 'Human (n=200)' is ambiguous. Appendix E.3 describes 200 questions per dataset, but the table appears to aggregate across datasets. Please clarify the exact number of human responses and whether the figure is per-domain or pooled.
  5. [Appendix E.3, Eq. (9)] Reporting a Pearson correlation over six aggregate data points (three domains × two levels) without confidence intervals is misleading. At minimum, report the exact data pairs and a bootstrap interval; preferably compute a per-item or per-dataset correlation with an appropriate multilevel model.

Circularity Check

0 steps flagged

No significant circularity; depth-as-hardness is an empirical premise with acknowledged confounds, not a derivation from its own inputs.

full rationale

The paper's central claims—that KNIGHT produces high-quality MCQs, that Level 3 items are harder than Level 1 items, that distractor entropy tracks difficulty, and that model rankings on KNIGHT align with MMLU-style benchmarks—are empirical evaluations, not consequences of the framework's definitions. Difficulty levels are defined by graph depth (Eq. 12), but the claim that deeper paths are harder is tested externally: probe-model accuracy, human error rates, and within-domain Level 1 vs. Level 3 accuracy all decrease, while entropy increases. These outcomes are not forced by construction; they could have failed. No fitted parameter is renamed as a prediction, and no equation reduces the 'prediction' to the input. The main limitation, stated in Appendix A, is that difficulty is multi-factorial and that graph depth is only a proxy; this is a correctness/validity risk, not circularity. The use of GPT-4o-mini for both generation and validation is a methodological concern about self-confirmation, but human audits provide an independent check, and no load-bearing argument reduces to a self-citation. The paper also openly defers 'adaptive' calibration to future work, so that title term is unsupported but not circularly derived. Overall, the derivation chain is self-contained and empirically testable.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

The framework introduces no new physical or conceptual entity; its load-bearing assumptions are domain-level modeling choices: depth as difficulty, answerability as hallucination, same-LLM validation as quality proxy, and sufficient ontology coverage. The free parameters are numerous hand-set thresholds and temperatures that directly shape graph construction and question quality, but none are fit to an external benchmark to produce the central claims.

free parameters (7)
  • dmax difficulty levels = 1 and 3
    Chosen as easy/hard splits; no principled calibration; used in Algorithm 1 and V_dmax to define Level 1 vs Level 3.
  • eta (description traceability threshold) = 0.35
    Equation (2): node glosses with overlap below 0.35 are discarded; chosen by hand with no sensitivity analysis.
  • retrieval score threshold = 0.15
    Section 3.1: if no retrieved passage scores above 0.15, the system falls back to parametric generation; hand-chosen.
  • lambda_max (Levenshtein near-duplicate threshold) = not reported
    Section 3.1 KG-2: near-duplicate triples removed below this threshold; value omitted, affecting graph deduplication.
  • tau (cosine semantic alias threshold) = not reported
    Algorithm 2: candidate nodes are merged if embedding cosine similarity exceeds tau; value omitted, affecting KG topology.
  • max_branches = 2
    Appendix I.3: limits number of extracted triplets per node; chosen after testing 1-4, directly shapes graph size and question paths.
  • LLM temperatures = 0.4 (description), 0.1 (triplet extraction)
    Appendix I.1: selected via empirical testing; these values directly influence generation and thus all downstream quality numbers.
axioms (5)
  • domain assumption Graph depth is a valid intrinsic hardness knob.
    Appendix G.2 calls depth an 'intrinsic hardness knob' and uses dmax to define difficulty levels; the limitations section admits difficulty is multi-factorial.
  • domain assumption Answerability-from-source is a valid proxy for hallucination.
    Section 4.3.2 treats ANSWERABLE violations as a hallucination proxy; this is an acknowledged operationalization, not a measured ground truth.
  • domain assumption GPT-4o-mini can serve as both generator and validator without systematic bias.
    Section 3.2 uses the same model for generation and validation; human audits cover only 100 items per split, so most items are validated only by the generator itself.
  • domain assumption Wikipedia/Wikidata ontology coverage is sufficient for History, Biology, and Mathematics.
    Section 3.1 KG-4 instantiates the type system with Wikidata; only this instantiation is tested, and generalization to other ontologies is explicitly future work.
  • domain assumption Rank alignment with an average of MMLU, ARC, CSQA, RACE, MedMCQA, and OBQA over six models is meaningful evidence of benchmark utility.
    Section 5.2 Table 5 reports rank order without a correlation coefficient or uncertainty; the claim of alignment rests on visual inspection of six models.

pith-pipeline@v1.3.0-alltime-deepseek · 29334 in / 14630 out tokens · 141702 ms · 2026-08-02T21:23:47.851320+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibration." pith.science (2026). https://pith.science/paper/VWCDTXIY

@misc{pith2026260220135,
  author       = {Pith},
  title        = {Pith review of: KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VWCDTXIY}},
  note         = {Machine review of arXiv:2602.20135}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

With the rise of large language models (LLMs), they have become instrumental in applications such as Retrieval-Augmented Generation (RAG). Yet evaluating these systems remains bottlenecked by the time and cost of building specialized assessment datasets. We introduce KNIGHT, an LLM-based, knowledge-graph-driven framework for generating multiple-choice question (MCQ) datasets from external sources. KNIGHT constructs a topic-specific knowledge graph, a structured and parsimonious summary of entities and relations, that can be reused to generate instructor-controlled difficulty levels, including multi-hop questions, without repeatedly re-feeding the full source text. This knowledge graph acts as a compressed, reusable state, making question generation a cheap read over the graph. We instantiate KNIGHT on Wikipedia/Wikidata while keeping the framework domain- and ontology-agnostic. As a case study, KNIGHT produces six MCQ datasets in History, Biology, and Mathematics. We evaluate quality on five criteria: fluency, unambiguity (single correct answer), topic relevance, option uniqueness, and answerability given the provided sources (as a proxy for hallucination). Results show that KNIGHT enables token- and cost-efficient generation from a reusable graph representation, achieves high quality across these criteria, and yields model rankings aligned with MMLU-style benchmarks, while supporting topic-specific and difficulty-controlled evaluation.

Figures

Figures reproduced from arXiv: 2602.20135 by Behnam Bahrak, Erfan Shafiee Moghaddam, Farhan Farsi, Mahdi Noori, Mohammad Amanlou, Yasaman Amou Jafari.

Figure 1
Figure 1. Figure 1: KNIGHT High-level pipeline. Given a prompt/topic and depth, KNIGHT retrieves evidence, builds a focused KG, generates MCQs, and filters them to produce the final dataset. We treat answerability from retrieved evidence as a proxy for generator hallucination: items judged unanswerable from the sources indicate unsupported or hallucinated content. Since the KG is built once per topic and reused across many ge… view at source ↗
Figure 2
Figure 2. Figure 2: KNIGHT architecture. (Left) A topic/prompt-driven RAG pipeline retrieves evidence, extracts triples, and curates a compact KG under depth budget dmax; (Right) multi-hop paths are sampled to generate questions/distractors and validated for evidence-grounded answerability to form the final MCQA dataset. parametric LMs with retrieved knowledge bases; Lewis et al. [12] show it yields more specific, diverse, an… view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of question lengths for each dataset (histograms), demonstrating an approxi [PITH_FULL_IMAGE:figures/full_fig_p023_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Entropy distributions by topic and difficulty level visualized using boxen plots with [PITH_FULL_IMAGE:figures/full_fig_p028_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: illustrates a sample knowledge-graph traversal starting from the seed node Hafez. At depth 1 (purple path), the algorithm identifies Shiraz as a connected node. Deeper traversals (not shown) discover nodes such as “7th century,” “Iran,” and “>90 million,” which correspond to progressively harder MCQ templates. The Curator module ensures semantic uniqueness by adding entities like “Shiraz” only once, even i… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

92 extracted references · 2 canonical work pages

  1. [1]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Pith/arXiv arXiv 2001

  2. [2]

    A holistic assessment of the carbon footprint of noor, a very large Arabic language model

    Imad Lakim, Ebtesam Almazrouei, Ibrahim Abualhaol, Merouane Debbah, and Julien Launay. A holistic assessment of the carbon footprint of noor, a very large Arabic language model. In Angela Fan, Suzana Ilic, Thomas Wolf, and Matthias Gallé, editors,Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models...

    work page doi:10.18653/v1/2022.b 2022

  3. [3]

    Position: Enough of scaling llms! lets focus on downscaling.arXiv preprint arXiv:2505.00985, 2025

    Yash Goel, Ayan Sengupta, and Tanmoy Chakraborty. Position: Enough of scaling llms! lets focus on downscaling.arXiv preprint arXiv:2505.00985, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Compendium of llm evaluation methods

    Andrei Lopatenko. Compendium of llm evaluation methods. 2024.https://github.com/alo patenko/LLMEvaluation

    2024

  5. [5]

    Ragas: Supercharge your llm application evaluations.https://github.c om/explodinggradients/ragas, 2024

    ExplodingGradients. Ragas: Supercharge your llm application evaluations.https://github.c om/explodinggradients/ragas, 2024

    2024

  6. [6]

    Leaf: Multiple-choicequestion generation

    Kristiyan Vachev, Momchil Hardalov, Georgi Karadzhov, Georgi Georgiev, Ivan Koychev, and Preslav Nakov. Leaf: Multiple-choicequestion generation. InEuropean Conferenceon Information Retrieval, pages 321–328. Springer, 2022

    2022

  7. [7]

    Multiple-choice question generation: Towards an automated assessment framework.arXiv preprint arXiv:2209.11830, 2022

    Vatsal Raina and Mark Gales. Multiple-choice question generation: Towards an automated assessment framework.arXiv preprint arXiv:2209.11830, 2022

    Pith/arXiv arXiv 2022

  8. [8]

    Multiple-choice question generation using large language models: Methodology and educator insights

    Giorgio Biancini, Alessio Ferrato, and Carla Limongelli. Multiple-choice question generation using large language models: Methodology and educator insights. InAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization, pages 584–590, 2024

    2024

  9. [10]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URLhttps: //arxiv.org/abs/2009.03300

    Pith/arXiv arXiv 2021

  10. [11]

    itext2kg: Incremental knowledge graphs construction using large language models

    Yassir Lairgi, Ludovic Moncla, Rémy Cazabet, Khalid Benabdeslem, and Pierre Cléau. itext2kg: Incremental knowledge graphs construction using large language models. InInternational Conference on Web Information Systems Engineering, pages 214–229. Springer, 2024

    2024

  11. [12]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler Kulshreshtha, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. InAdvances in Neural Information Processing Systems (NeurIPS), 2020. URLhttps://arxiv.or...

    Pith/arXiv arXiv 2020

  12. [13]

    InProceedings of the 37th International Conference on Machine Learning (ICML 2020), pages 3929–3938, 2020

    KelvinGuu,KentonLee,ZoraTung,PanupongPasupat,andMing-WeiChang.Realm: Retrieval- augmented language model pre-training. InProceedings of the 37th International Conference on Machine Learning (ICML 2020), pages 3929–3938, 2020. URLhttps://proceedings.mlr.press/ v119/guu20a.html

    2020

  13. [14]

    InProceedings of EMNLP 2021, 2021

    AlexanderFabbri,WojciechKryściński,etal.Sumeval: Re-evaluatingsummarizationevaluation. InProceedings of EMNLP 2021, 2021

    2021

  14. [16]

    A comprehensive survey on automatic knowledge graph construction.ACM Computing Surveys, 56(4):1–62, 2023

    Lingfeng Zhong, Jia Wu, Qian Li, Hao Peng, and Xindong Wu. A comprehensive survey on automatic knowledge graph construction.ACM Computing Surveys, 56(4):1–62, 2023. doi: 10.1145/3618295

    doi:10.1145/3618295 2023

  15. [17]

    Rashid, Anisa Rula, Lukas Schmelzeisen, Juan Sequeda, Steffen Staab, and Antoine Zimmermann

    Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia d’Amato, Gerard de Melo, Claudio Gutierrez, Sabrina Kirrane, Jose Emilio Labra Gayo, Roberto Navigli, Sebastian Neumaier, Axel-Cyrille Ngonga Ngomo, Axel Polleres, Sabbir M. Rashid, Anisa Rula, Lukas Schmelzeisen, Juan Sequeda, Steffen Staab, and Antoine Zimmermann. Knowledge graphs.ACM Computing Surveys...

    doi:10.1145/3447772 2021

  16. [18]

    Danilo Dessì, Francesco Osborne, Diego Reforgiato Recupero, Davide Buscaldi, and Enrico Motta. Generating knowledge graphs by employing natural language processing and machine learning techniques within the scholarly domain.Future Generation Computer Systems, 116: 253–264, 2021

    2021

  17. [19]

    Llms for knowledge graph construction and reasoning: Recent capabilities and future opportunities.World Wide Web, 27(5):58, 2024

    YuqiZhu,XiaohanWang,JingChen,ShuofeiQiao,YixinOu,YunzhiYao,ShuminDeng,Huajun Chen, and Ningyu Zhang. Llms for knowledge graph construction and reasoning: Recent capabilities and future opportunities.World Wide Web, 27(5):58, 2024

    2024

  18. [20]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    OpenAI, Josh Achiam, Steven Adler, and .... Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  19. [21]

    Wiki-based prompts for enhancing relation extraction using language models

    Amirhossein Layegh, Amir H Payberah, Ahmet Soylu, Dumitru Roman, and Mihhail Matskin. Wiki-based prompts for enhancing relation extraction using language models. InProceedings of the 39th ACM/SIGAPP Symposium on Applied Computing, pages 731–740, 2024

    2024

  20. [22]

    Wikidata: A free collaborative knowledgebase.Com- munications of the ACM, 57(10):78–85, 2014

    Denny Vrandečić and Markus Krötzsch. Wikidata: A free collaborative knowledgebase.Com- munications of the ACM, 57(10):78–85, 2014. doi: 10.1145/2629489

    doi:10.1145/2629489 2014

  21. [23]

    Khapra, and Sachindra Joshi

    Sathish Reddy, Dinesh Raghu, Mitesh M. Khapra, and Sachindra Joshi. Generating natural lan- guage question-answer pairs from a knowledge graph using a RNN based question generation model. In Mirella Lapata, Phil Blunsom, and Alexander Koller, editors,Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistic...

    2017

  22. [24]

    Toward subgraph-guided knowledge graph question generation with graph neural networks.IEEE Transactions on Neural Networks and Learning Systems, 2023

    Yu Chen, Lingfei Wu, and Mohammed J Zaki. Toward subgraph-guided knowledge graph question generation with graph neural networks.IEEE Transactions on Neural Networks and Learning Systems, 2023

    2023

  23. [25]

    Multi-hopquestiongeneration with knowledge graph-enhanced language model.Applied Sciences, 13(9):5765, 2023

    ZhenpingLi,ZhenCao,PengfeiLi,YongZhong,andShaoboLi. Multi-hopquestiongeneration with knowledge graph-enhanced language model.Applied Sciences, 13(9):5765, 2023

    2023

  24. [26]

    Difficulty-controllable multi-hop question generation from knowledge graphs

    Vishwajeet Kumar, Yuncheng Hua, Ganesh Ramakrishnan, Guilin Qi, Lianli Gao, and Yuan- Fang Li. Difficulty-controllable multi-hop question generation from knowledge graphs. In Chiara Ghidini, Olaf Hartig, Maria Maleshkova, Vojtech Svátek, Isabel F. Cruz, Aidan Hogan, Jie Song, Maxime Lefrançois, and Fabien Gandon, editors,The Semantic Web – ISWC 2019 – 18t...

    2019

  25. [27]

    Guidingthegrowth: Difficulty-controllablequestiongenerationthroughstep-by-steprewriting

    Yi Cheng, Siyao Li, Bang Liu, Ruihui Zhao, Sujian Li, Chenghua Lin, and Yefeng Zheng. Guidingthegrowth: Difficulty-controllablequestiongenerationthroughstep-by-steprewriting. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors,Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International...

    2021

  26. [28]

    Liquid: aframeworkforlistquestionanswering dataset generation

    SeongyunLee,HyunjaeKim,andJaewooKang. Liquid: aframeworkforlistquestionanswering dataset generation. InProceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13014–13024, 2023

    2023

  27. [29]

    An automatic question usabilityevaluationtoolkit

    Steven Moore, Eamon Costello, Huy A Nguyen, and John Stamper. An automatic question usabilityevaluationtoolkit. InInternationalConferenceonArtificialIntelligenceinEducation,pages 31–46. Springer, 2024

    2024

  28. [30]

    Evaluating the diversity and quality of llm generated content.arXiv preprint arXiv:2504.12522, 2025

    Alexander Shypula, Shuo Li, Botong Zhang, Vishakh Padmakumar, Kayo Yin, and Osbert Bastani. Evaluating the diversity and quality of llm generated content.arXiv preprint arXiv:2504.12522, 2025

    arXiv 2025

  29. [31]

    Towardstrustablelanguagemodels: Investigating information quality of large language models.arXiv preprint arXiv:2401.13086, 2024

    RickRejeleene,XiaoweiXu,andJohnTalburt. Towardstrustablelanguagemodels: Investigating information quality of large language models.arXiv preprint arXiv:2401.13086, 2024

    Pith/arXiv arXiv 2024

  30. [32]

    The curious case of hallucinatory (un)answerability: Finding truths in the hidden states of over-confident large language models

    Aviv Slobodkin, Omer Goldman, Avi Caciularu, Ido Dagan, and Shauli Ravfogel. The curious case of hallucinatory (un)answerability: Finding truths in the hidden states of over-confident large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 36...

    doi:10.18653/v1/2023 2023

  31. [33]

    XiangyuPeng,PrafullaKumarChoubey,CaimingXiong,andChien-ShengWu.Unanswerability evaluation for retrieval augmented generation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8452–8472, Vienna, Austr...

    work page doi:10.18653/v1/2025.acl-long.415 2025

  32. [34]

    Automatic multiple-choice question generation and evaluation systems based on LLM: A study case with university resolutions

    Sérgio Silva Mucciaccia, Thiago Meireles Paixão, Filipe Wall Mutz, Claudine Santos Badue, Alberto Ferreira de Souza, and Thiago Oliveira-Santos. Automatic multiple-choice question generation and evaluation systems based on LLM: A study case with university resolutions. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, a...

    2025

  33. [35]

    Adversarial NLI: A new benchmark for natural language understanding

    Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial NLI: A new benchmark for natural language understanding. InProceedings of ACL, pages 4885–4901, 2020

    2020

  34. [36]

    Bowman, Gabor Angeli, Christopher Potts, and Christopher D

    Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. InProceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 632–642. Association for Computational Linguistics, 2015. URLhttps://aclanthology.org/D15-1075/

    2015

  35. [37]

    Adina Williams, Nikita Nangia, and Samuel R. Bowman. Broad-coverage challenge datasets for sentence understanding. InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 1112–1122. Association for Computational Linguistics, 2018. URLhttps://aclanth...

    2018

  36. [38]

    Unsuper- vised dense information retrieval with contrastive learning

    Gautier Izacard, Lucas Hosseini, Emmanuel De Bézenac, and Vladimir Karpukhin. Unsuper- vised dense information retrieval with contrastive learning. InEMNLP, 2022

    2022

  37. [39]

    The Probabilistic Relevance Framework: BM25 and beyond.Foundations and Trends in Information Retrieval, 2009

    Stephen Robertson and Hugo Zaragoza. The Probabilistic Relevance Framework: BM25 and beyond.Foundations and Trends in Information Retrieval, 2009

    2009

  38. [40]

    Making monolingual sentence embeddings multilingual using knowledge distillation

    Nils Reimers and Iryna Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation. InEMNLP, 2020

    2020

  39. [41]

    Survey of hallucination in natural language generation.ACM Computing Surveys, 2023

    Zihan Ji and et al. Survey of hallucination in natural language generation.ACM Computing Surveys, 2023

    2023

  40. [42]

    Wikidata: A free collaborative knowledgebase.Com- munications of the ACM, 57(10):78–85, 2014

    Denny Vrandečić and Markus Krötzsch. Wikidata: A free collaborative knowledgebase.Com- munications of the ACM, 57(10):78–85, 2014

    2014

  41. [43]

    Large language models as distractor generators for multiple-choice qa

    Hao Yu, Yiming Cui, and Wanxiang Che. Large language models as distractor generators for multiple-choice qa. InProceedings of ACL, 2024

    2024

  42. [44]

    Haladyna, Steven M

    Thomas M. Haladyna, Steven M. Downing, and Michael C. Rodriguez. A review of multiple- choice item-writing guidelines for classroom assessment.Applied Measurement in Education, 15 (3):309–334, 2002

    2002

  43. [45]

    Analyzingquestioncharacteristicsinfluencingchatgpt’sperformancein3000usmle®-style questions.Medical Science Educator, pages 1–11, 2024

    Michael Alfertshofer, Samuel Knoedler, Cosima C Hoch, Sebastian Cotofana, Adriana C Panayi, MartinKauke-Navarro,StefanGTullius,DennisPOrgill,WilliamGAustenJr,BohdanPomahac, etal. Analyzingquestioncharacteristicsinfluencingchatgpt’sperformancein3000usmle®-style questions.Medical Science Educator, pages 1–11, 2024

    2024

  44. [46]

    Can llms solve longer math word problems better?arXiv preprint arXiv:2405.14804, 2024

    Xin Xu, Tong Xiao, Zitong Chao, Zhenya Huang, Can Yang, and Yang Wang. Can llms solve longer math word problems better?arXiv preprint arXiv:2405.14804, 2024

    Pith/arXiv arXiv 2024

  45. [47]

    Do large language models have shared weaknesses in medical question answering?arXiv preprint arXiv:2310.07225, 2023

    Andrew M Bean, Karolina Korgul, Felix Krones, Robert McCraith, and Adam Mahdi. Do large language models have shared weaknesses in medical question answering?arXiv preprint arXiv:2310.07225, 2023

    Pith/arXiv arXiv 2023

  46. [48]

    Self-rag: Learning to retrieve, generate, and critique through self-reflection.arXiv preprint arXiv:2310.11511, 2023

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection.arXiv preprint arXiv:2310.11511, 2023. URLhttps://arxiv.org/abs/2310.11511

    Pith/arXiv arXiv 2023

  47. [49]

    Languagetool: Open-source grammar, style, and spell checker

    LanguageTool Developers. Languagetool: Open-source grammar, style, and spell checker. https://languagetool.org/, 2025. Accessed: 2025-10-06

    2025

  48. [50]

    language-tool-python: Python wrapper for languagetool

    language-tool-python Contributors. language-tool-python: Python wrapper for languagetool. https://pypi.org/project/language-tool-python/, 2025. Accessed: 2025-10-06. 13

    2025

  49. [51]

    Langcheck: Simple, pythonic building blocks to evaluate llm applications.https: //github.com/citadel-ai/langcheck, 2023

    Citadel AI. Langcheck: Simple, pythonic building blocks to evaluate llm applications.https: //github.com/citadel-ai/langcheck, 2023. Accessed: 2025-12-13

    2023

  50. [52]

    Steven M. Downing. The effects of violating standard item-writing principles on tests and students: The consequences are serious.Medical Education, 39(3):291–296, 2005

    2005

  51. [53]

    Rodriguez

    Michael C. Rodriguez. Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research.Educational Measurement: Issues and Practice, 24(2):3–13, 2005

    2005

  52. [54]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  53. [55]

    Click: A benchmark dataset of cultural and linguistic intelligence in korean, 2024

    Eunsu Kim, Juyoung Suk, Philhoon Oh, Haneul Yoo, James Thorne, and Alice Oh. Click: A benchmark dataset of cultural and linguistic intelligence in korean, 2024. URLhttps: //arxiv.org/abs/2403.06412

    Pith/arXiv arXiv 2024

  54. [56]

    Llama: Open and efficient foundation language models

    Hugo Touvron and et al. Llama: Open and efficient foundation language models. InNeurIPS, 2023

    2023

  55. [57]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. InProceedingsofthe2018ConferenceonEmpiricalMethodsinNaturalLanguageProcessing (EMNLP), 2018. URLhttps://arxiv.org/abs/1803.05457

    Pith/arXiv arXiv 2018

  56. [58]

    Commonsenseqa: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL),

    2019

  57. [59]

    Race: Large-scale reading comprehension dataset from examinations

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2017. URLhttps://arxiv.org/ab s/1704.04683

    Pith/arXiv arXiv 2017

  58. [60]

    Medmcqa: A large- scale multi-subject multi-choice dataset for medical domain question answering

    Abhishek Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large- scale multi-subject multi-choice dataset for medical domain question answering. InProceedings of the Conference on Health, Inference, and Learning, volume 174 ofProceedings of Machine Learning Research, pages 248–260, 2022. URLhttps://proceedings.mlr.press/v174/pal22a.html

    2022

  59. [61]

    Can a suit of armor conduct electricity? anewdatasetforopenbookquestionanswering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? anewdatasetforopenbookquestionanswering. InProceedingsofthe2018Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018. URLhttps://arxiv.org/ abs/1809.02789

    Pith/arXiv arXiv 2018

  60. [62]

    Ho, Han Liu, et al

    Yerbolat Myrzakhan, Nelson F. Ho, Han Liu, et al. Open llm leaderboard: Heterogeneous, dynamic, and robust evaluation of llms.arXiv preprint arXiv:2406.07545, 2024

    Pith/arXiv arXiv 2024

  61. [63]

    GPT-4o: System card and model overview.https://openai.com/index/gpt-4o-sys tem-card/, 2024

    OpenAI. GPT-4o: System card and model overview.https://openai.com/index/gpt-4o-sys tem-card/, 2024. Accessed 2025-10-06

    2024

  62. [64]

    Mistral large

    Mistral AI. Mistral large. https://mistral.ai/news/mistral-large/ , 2024. Accessed 2025-10-06

    2024

  63. [65]

    Llama 3 model card and evaluations.https://ai.meta.com/llama/, 2024

    Meta AI. Llama 3 model card and evaluations.https://ai.meta.com/llama/, 2024. Accessed 2025-10-06

    2024

  64. [66]

    Claude 3 model family: Model card and system overview.https://www.anthropic

    Anthropic. Claude 3 model family: Model card and system overview.https://www.anthropic. com/claude, 2024. Accessed 2025-10-06. 14

    2024

  65. [67]

    Qwen technical report: An open large language model family.arXiv preprint arXiv:2309.16609, 2023

    Yuxiao Bai, Weizhe Dai, An Yang, et al. Qwen technical report: An open large language model family.arXiv preprint arXiv:2309.16609, 2023

    Pith/arXiv arXiv 2023

  66. [68]

    Gemma: Openmodelsbuiltfromtheresearchbehind gemini.https://ai.google.dev/gemma, 2024

    GoogleDeepMindandGoogleResearch. Gemma: Openmodelsbuiltfromtheresearchbehind gemini.https://ai.google.dev/gemma, 2024. Accessed 2025-10-06

    2024

  67. [69]

    Electrokinetic ef- fects on flow and ion transport in charge-patterned corrugated nanochannels.arXiv preprint arXiv:2510.22182, 2025

    Thomas Petersen, Pouya Golchin, Jinwoo Im, and Felipe PJ de Barros. Electrokinetic ef- fects on flow and ion transport in charge-patterned corrugated nanochannels.arXiv preprint arXiv:2510.22182, 2025

    Pith/arXiv arXiv 2025

  68. [70]

    Computing the saturation throughput for heterogeneous p-csma in a general wireless network

    Faezeh Dehghan Tarzjani and Bhaskar Krishnamachari. Computing the saturation throughput for heterogeneous p-csma in a general wireless network. In2025 34th International Conference on Computer Communications and Networks (ICCCN), pages 1–7. IEEE, 2025

    2025

  69. [71]

    langchain: Build context-aware reasoning applications.https://github.com/l angchain-ai/langchain, 2025

    LangChain AI. langchain: Build context-aware reasoning applications.https://github.com/l angchain-ai/langchain, 2025

    2025

  70. [72]

    spaCy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing.https://spacy.io, 2017

    Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing.https://spacy.io, 2017

    2017

  71. [73]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi,PierricCistac,TimRault,RémiLouf,MorganFuntowicz,JoeDavison,SamShleifer,Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural lang...

    2020

  72. [74]

    Neo4j developer documentation.https://neo4j.com/docs/

    Neo4j, Inc. Neo4j developer documentation.https://neo4j.com/docs/. Accessed: May 20, 2025

    2025

  73. [75]

    calculation-heavy

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdi- nov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop questionanswering. InProceedingsofthe2018ConferenceonEmpiricalMethodsinNaturalLanguage Processing, pages 2369–2380. Association for Computational Linguistics, 2018. A. Limitations...

    2018

  74. [77]

    GRAMMAR_FLUENCY Is the Question spelled and phrased correctly and clearly?

  75. [78]

    SINGLE_CORRECT_KEY Is exactly one option marked as correct?

  76. [79]

    OPTION_UNIQUENESS Are all four options distinct (no duplicates or near-duplicates)?

  77. [80]

    ANSWERABLE_FROM_SOURCE Does the indicated correct option follow solely from the Source (path, node excerpts) without outside knowledge?

  78. [81]

    [question]

    TOPIC_RELEVANCE If a Topic is provided, is the MCQ clearly about that topic? MCQ Validation User Prompt Purpose: Provides the LLM with the specific MCQ data (“[question]”, “[correct_answer_key]”) and its source- details (including the “[path_representation]”, “[start_node]”, “[end_node]” and etc) to evaluate. It lists the evaluation criteria and specifies...

  79. [82]

    Formally, a questionq satisfies this criterion if it passes both automated grammar checks and human inspection for clarity and style

    Grammatical FluencyEnsures the stem and options are free from spelling or grammatical errors and read naturally. Formally, a questionq satisfies this criterion if it passes both automated grammar checks and human inspection for clarity and style. We quantitatively assess grammatical accuracy of each questionq comprising W words by detecting the number of ...

  80. [83]

    Example of Non-compliance:Which are prime numbers? Options:{2,3,4,5}(with two correct answers: 2 and 3)

    Single Correct KeyExactly one optionok ∈ {o1, o2, o3, o4}is correct: ∃!o k : Correct(o k) = True.(5) This avoids ambiguity in scoring and interpretation. Example of Non-compliance:Which are prime numbers? Options:{2,3,4,5}(with two correct answers: 2 and 3). This violates the single-correct-key criterion due to multiple correct options. Compliant form:Whi...

Showing first 80 references.