{"id":"d4762aed-7d89-4c32-828d-8012294790ed","arxiv_id":"2602.20135","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A reusable per-topic knowledge graph, built once from Wikipedia, lets an LLM generate multi-hop multiple-choice questions whose difficulty is set by path depth, with human-audited quality and model rankings that track MMLU.","lead":"KNIGHT builds a small knowledge graph for a topic and then uses an LLM to write multiple-choice questions from paths in that graph. The pitch is cheaper, faster, and reusable evaluation: one graph per topic, many exam questions, with difficulty set by path length.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Depth-as-hardness is the load-bearing premise, but it is only supported by aggregate correlations that do not control for surface confounds; the paper's own limitation concedes difficulty is multi-factorial, so the difficulty-controlled evaluation claim is conditional.","rationale":"The paper has real strengths: human audits, ablations, released code, and a coherent pipeline. But the central value proposition—difficulty-controlled evaluation from a reusable KG—depends on depth being a valid hardness knob. The authors themselves hedge in Appendix A. The available evidence doesn't isolate depth from surface features and uses aggregate correlations. This doesn't invalidate the framework; it makes the difficulty-calibration claim conditional on further controls. I agree with the reader's weakest assumption. My verdict is UNCHANGED: CONDITIONAL acceptance remains appropriate. If anything, the missing adaptive loop reinforces the condition: the title promises more than the system delivers.","tokens_in":29718,"tokens_out":9255,"duration_ms":88406,"concrete_test":"Pre-register a blinded human difficulty-rating study on a stratified sample from the six datasets (e.g., 50 items per split, 600 items). Raters assign a difficulty score without knowing the level. Fit a mixed-effects model: difficulty ~ dmax + question_length + mean word frequency + named_entity_count + (1|topic). The depth-as-hardness claim is sustained only if the dmax coefficient is positive and significant after controlling for these surface features. As a direct comparison, also include depth-1 items with added irrelevant clauses matched in length to depth-3 items; if raters judge these as hard as real depth-3 items, path depth is not the operative hardness mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 (Alg. 1) makes dmax the only direct control over difficulty; Appendix G.2 states this explicitly: 'Depth therefore acts as an intrinsic hardness knob' (Eq. 12). The abstract's 'difficulty-controlled evaluation' and the title's 'Adaptive Hardness Calibration' both rest on this premise. Yet the manuscript's own Limitations (Appendix A) concede that 'Difficulty is multi-factorial' and that low-relational-density domains (physics, numerical computation) may not suit graph grounding. The empirical evidence for monotonicity is correlational: Table 4 shows entropy/probe-accuracy shifts between Level 1 and Level 3, Table 5 shows accuracy drops across models, and Appendix E.3 reports r≈0.78 between human error rates and model entropy. None of these analyses controls for the confounds that naturally accompany longer KG paths—rarer entities, more complex syntax, more named entities. Moreover, the human correlation is computed over only six aggregate data points, so it cannot isolate depth as the causal driver. The appendix also documents no adaptive calibration loop (no feedback from human/model performance to generation parameters), so the 'Adaptive' part of the title is unsupported independently of the proxy question. If depth is not a monotonic cognitive-hardness knob beyond the three tested domains, the central difficulty-control claim fails even if the generated items are individually high quality.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces KNIGHT, an LLM-based framework that constructs a topic-specific knowledge graph from retrieved Wikipedia/Wikidata evidence and then generates multiple-choice questions by traversing graph paths of configurable depth. A five-criterion LLM validator filters the generated items, and the authors report six MCQ datasets across history, biology, and mathematics at two difficulty levels, together with ablations against Plain, RAG, RAG+KG, and RAG+Val baselines. The evaluation covers human audits, automatic fluency checks, topic-relevance scores, predictive-entropy-based distractor analysis, and downstream model ranking compared with MMLU-style benchmarks. The central claims are that the reusable KG representation enables token- and cost-efficient generation, that generated items are high quality on five criteria, and that difficulty-controlled evaluation with model rankings aligned to established benchmarks is achieved.","tokens_in":29956,"tokens_out":5120,"duration_ms":51499,"significance":"If substantiated, KNIGHT would be a practically useful contribution: a topic KG built once and reused for many MCQ-generation runs is a plausible route to low-cost, refreshable, syllabus-aligned evaluation sets, and the released code/PyPI package supports reproducibility. The paper has real strengths: six human audits with n=100 per system, a staged component ablation separating retrieval, KG structure, and validation, an explicit five-criterion validation protocol, and an appendix (E.4) with careful significance testing for topic relevance. However, the manuscript currently overstates the evidence for its two headline capabilities: difficulty control via graph depth, and 'adaptive' hardness calibration. The load-bearing difficulty-calibration premise is supported only by aggregate correlations with no control for confounds, and no adaptive feedback loop is implemented. The topic-relevance presentation also conflicts with the paper's own Table 3.","major_comments":[{"comment":"The paper's central difficulty-control claim rests on treating KG path depth as an intrinsic hardness knob: Appendix G.2 states 'Depth therefore acts as an intrinsic hardness knob' (Eq. 12), and Section 3.1 uses dmax as the only direct difficulty control. Yet the empirical evidence is correlational: Table 4 shows entropy/probe-accuracy shifts between Level 1 and Level 3, Table 5 shows accuracy drops, and Appendix E.3 reports r≈0.78 between human error rates and model entropy over six aggregate data points. None of these analyses controls for the confounds that naturally accompany longer KG paths, such as rarer entities, more complex syntax, or higher named-entity density. The paper's own limitation (Appendix A) concedes that difficulty is multi-factorial. A convincing demonstration would need, for example, a within-path manipulation that varies depth while holding surface form roughly co","section":"§3.1, Appendix G.2 Eq. (12), Appendix A"},{"comment":"The title and abstract promise 'Adaptive Hardness Calibration,' but no adaptive mechanism is described or evaluated. The only difficulty control is an a priori depth parameter dmax (Algorithm 1) followed by post-generation validation; there is no feedback loop from human or model performance to generation parameters. Section 6 explicitly lists 'incorporating adaptive difficulty tuning via model feedback' as future work. This is a mismatch between the claimed contribution and the implemented system. Either the title/abstract should be revised to describe static difficulty control with post-hoc validation, or an actual calibration loop with evidence of convergence/adaptation should be added.","section":"Title, Abstract, §6"},{"comment":"The main text states that KNIGHT 'maintains strong topical alignment across topics and difficulty levels' and that off-topic rates are low, but Table 3 shows that KNIGHT is consistently worse than RAG+Val on both automatic topic-relevance metrics and the off-topic intersection rate: for example, History L1 off-topic rate is 10.6% for KNIGHT vs 3% for RAG+Val; Biology L1 5.5% vs 1%; Math L1 7.3% vs 4%. Entailment and LLM relevance scores are also lower for KNIGHT in most splits. Appendix E.4's significance tests compare KNIGHT only against a ~100-item GPT-4o-mini baseline, not against RAG+Val, and they do not cover the human TOPIC flags or the 'LLM∩Entailment' off-topic rate. Therefore the paper's claim that KNIGHT achieves high quality across all five criteria is not supported for the topic-relevance criterion in the comparison that matters. Please report the RAG+Val vs KNIGHT comparison","section":"§4.3.3, Table 3, Appendix E.4"},{"comment":"Several headline claims rest on small count differences or aggregate statistics without confidence intervals or significance tests. In Table 2, KNIGHT's advantage over RAG+Val on SINGLE_KEY, OPTION_UNIQUENESS, and ANSWERABLE is often only 1–4 flags per 100 items, and no uncertainty is reported. In Table 4, the entropy/probe-accuracy differences that drive the central difficulty and distractor-competition conclusions are reported as single numbers with no error bars, and Appendix E.3's human-entropy correlation is computed over only six aggregate points. Given that these tables carry the paper's main empirical claims, the authors should provide per-split confidence intervals, and where possible significance tests or effect sizes, for the key KNIGHT-vs-baseline comparisons.","section":"§4.3.2, Table 2; §5.1, Table 4"}],"minor_comments":[{"comment":"The text under G.3 and G.4 largely duplicates Section G.1–G.2 (e.g., the formal definition of node curation appears twice, as Eq. 11 and Eq. 13). Please remove the duplicate material and retain one consolidated curator description.","section":"Appendix G.3–G.4"},{"comment":"Several references are duplicated or near-duplicated in the bibliography: [15] and [31] are the same work, and [22] and [42] are the same Wikidata paper. Please de-duplicate.","section":"References"},{"comment":"The claim that 'reverse questions increase model entropy by 15–20% (Section 4)' is not reported in Section 4. Please either point to the specific table/figure or add the supporting result.","section":"Appendix G.5"},{"comment":"The row 'Human (n=200)' is ambiguous. Appendix E.3 describes 200 questions per dataset, but the table appears to aggregate across datasets. Please clarify the exact number of human responses and whether the figure is per-domain or pooled.","section":"Table 5"},{"comment":"Reporting a Pearson correlation over six aggregate data points (three domains × two levels) without confidence intervals is misleading. At minimum, report the exact data pairs and a bootstrap interval; preferably compute a per-item or per-dataset correlation with an appropriate multilevel model.","section":"Appendix E.3, Eq. (9)"}],"recommendation":"major_revision","confidential_remarks":"The paper describes a genuinely useful system and includes commendable reproducibility artifacts, but the framing currently exceeds the evidence. The most serious issue is that 'adaptive hardness calibration' is not implemented and the depth-as-hardness premise is not tested against obvious confounds. The topic-relevance results in Table 3 also undercut the 'high quality across all criteria' abstract claim. These are fixable with additional analyses and reframing, so I do not recommend rejection, but the revision needs to be substantial."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful systems paper, and its main empirical claims mostly hold up. The integrated pipeline—cached topic KG, multi-hop forward/reverse generation, LLM-based validator—is not present in the cited prior work as a single system, and the ablations show KG structuring plus validation does improve answerability and distractor competition over plain RAG. The human audits on n=100 per split and the probe-entropy results provide real evidence for the quality claims. Credit where due.\n\nThe soft spot is exactly where the title and abstract push hardest: 'adaptive hardness calibration.' The only difficulty control is graph depth (dmax=1 vs 3), and the evidence that depth is a reliable cognitive-hardness knob is correlational. Entropy and probe accuracy move in the right direction, and the human-error correlation is r≈0.78, but that correlation is computed over six aggregate data points and doesn't control for confounds that naturally accompany longer paths—rarer entities, more complex syntax, more named entities. The authors themselves concede in the limitations that difficulty is multi-factorial. There is also no actual adaptive loop anywhere: no feedback from model or human performance feeds back into generation parameters. So 'adaptive' is unsupported and 'calibration' rests on a plausible but under-tested premise.\n\nSmaller issues: the main text reports no confidence intervals or significance tests for most headline numbers. I checked the appendix and E.4 actually does run proper Holm-corrected tests on topic relevance—they're all non-significant with small effect sizes—so that particular concern is partially addressed in the appendix even though the main text tables are shown raw. Reproducibility is incomplete: no commit hash, no dataset release, few-shot prompts omitted.\n\nIf the authors soften the language, add error bars, and release the artifacts, this is a solid contribution to the LLM-evaluation toolkit. I'd send it out; the work is coherent, honestly reported, and the central claim—refreshable, low-cost, topic-specific MCQ generation with quality comparable to RAG—is useful to enough people.","headline":"A useful integrated MCQ-generation pipeline whose main quality claims hold, but the 'adaptive hardness calibration' framing oversells a depth-as-difficulty proxy that is only weakly validated.","tokens_in":30569,"tokens_out":2317,"would_cite":true,"duration_ms":21305,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a topic-specific knowledge graph, built once, can be reused to generate multiple-choice question sets with controlled difficulty at low cost, and that model rankings on these sets match established benchmarks.","keywords":["multiple-choice question generation","knowledge graphs","difficulty calibration","multi-hop reasoning","retrieval-augmented generation","LLM evaluation","distractor generation","benchmark generation"],"falsifier":"Run KNIGHT on a calculation-heavy subject (e.g., physics or numerical computation) and compare Level 1 vs Level 3 items on human error rates and model accuracy. If Level 3 items are not consistently harder than Level 1, or if the graph-grounding step collapses because facts are not entity-relation triples, then the depth-as-hardness premise is falsified. A more direct test: measure whether human error rates and probe entropy increase monotonically with path length within a single domain, controlling for question length and vocabulary.","tokens_in":29522,"feed_emoji":"📚","tokens_out":6455,"duration_ms":53671,"temperature":0.7,"pith_summary":"KNIGHT is a pipeline that, given a topic, retrieves source text, distills it into a compact knowledge graph of entities and relations, and then generates four-option multiple-choice questions by walking paths of increasing length through that graph. The paper's central claim is that this graph acts as a reusable, compressed representation: once built, it can be queried repeatedly to produce many question variants at different difficulty levels without re-feeding the full source documents. The authors argue that path depth is an intrinsic hardness knob—longer paths yield harder, multi-hop questions—and that this difficulty signal survives validation, matching human error patterns and model uncertainty. They further claim that models ranked by accuracy on KNIGHT-generated sets are ordered the same way as on established broad MCQ benchmarks, supporting KNIGHT's use as a cheap, refreshable benchmark generator.","feed_headline":"Knowledge graph reuse makes MCQs cheap and difficulty-tunable","feed_subtitle":"A cached topic graph yields many exam sets; deeper paths make harder questions, and rankings track major benchmarks.","key_machinery":"The central object is the depth-bounded topic knowledge graph G=(V,E), built by breadth-first expansion from a seed topic up to a user-chosen depth d_max, with each node carrying a source-validated description and each edge a typed relation. Question generation samples length-d paths (forward or reverse) and verbalizes them into MCQ stems, so d controls the number of reasoning hops. The supporting mechanism is the validator: an LLM-as-critic that enforces five criteria—grammar, single correct key, option uniqueness, answerability from the supplied evidence, and topic relevance. A fixed probe model's predictive entropy over answer choices serves as an automatic, model-agnostic measure of dist","core_discovery":"The central claim is that knowledge-graph-guided generation, not retrieval or validation alone, is what produces competitive distractors and controlled difficulty. In their staged comparison, plain and retrieval-only prompting yield near-zero answer entropy and near-ceiling probe accuracy—distractors are trivially wrong. Adding a validator improves item validity but not difficulty; only conditioning on multi-hop graph paths raises entropy, lowers probe accuracy, and separates Level 1 from Level 3. The full KNIGHT pipeline combines this KG guidance with a five-criterion validator, yielding the lowest rates of duplicate, ambiguous, and unanswerable items while preserving the difficulty signal.","pith_inferences":["Beyond the paper, if path depth is a reliable hardness proxy, the same cached graph could drive adaptive test generation, choosing hop length from a learner's or model's observed accuracy in real time.","An unstated consequence of the reusable-graph design is testing integrity: new items drawn from different paths do not reuse stems, so refreshable assessments are less vulnerable to memorization.","The reported correlation between predictive entropy and human difficulty (r≈0.78) suggests entropy could serve as an automatic calibration signal, enabling closed-loop difficulty tuning without human annotation.","The authors' own limitation about low-relational-density domains implies the representation will need hybrid extensions—attaching formulas or procedural steps to nodes—before it transfers to calculation-heavy subjects like physics."],"forward_implications":["A topic KG built once can be reused to generate many question sets at different difficulty levels, hop lengths, and question formats, so marginal cost per additional dataset is small.","Level 3 items are consistently harder than Level 1 for both humans and a range of models, and the increase in difficulty does not come with a rise in validity violations.","Rankings of models on KNIGHT datasets align with rankings on established broad MCQ suites, suggesting the generated sets capture general QA competence rather than topic-specific quirks.","KG-guided prompting is the component that creates plausible distractors; retrieval grounding alone and validation alone do not, which isolates where the value comes from.","The five-criteria validator cuts duplicate, ambiguous, and source-unsupported items, and answerability violations serve as a usable proxy for hallucination during generation."],"fun_headline_variants":["Graph-guided MCQs: cheap, tunable, top ranks","Cached knowledge graph yields hard, unbiased MCQs","Reusable KG powers exam questions with real distractors","MCQ factory: KG reuse trims cost, tunes challenge","Knowledge graph reuse: cheap, hard, aligned with MMLU"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The framework equates question difficulty with knowledge-graph path depth: one hop means Level 1 and three hops mean Level 3. If path length is not a reliable proxy for how hard a question actually is—across different subjects or once linguistic complexity and prior knowledge are factored in—the difficulty-calibration claim, and with it much of the framework's value, fails.","fun_headline_variants_meta":{"raw":{"variants":["Graph-guided MCQs: cheap, tunable, top ranks","Cached knowledge graph yields hard, unbiased MCQs","Reusable KG powers exam questions with real distractors","MCQ factory: KG reuse trims cost, tunes challenge","Knowledge graph reuse: cheap, hard, aligned with MMLU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000174,"raw_usage":{"total_tokens":1126,"prompt_tokens":757,"completion_tokens":369,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":286}},"tokens_in":501,"tokens_out":369,"duration_ms":3494,"temperature":1.0,"reasoning_tokens":286,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T21:23:47.851320+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run KNIGHT on a calculation-heavy subject (e.g., physics or numerical computation) and compare Level 1 vs Level 3 items on human error rates and model accuracy. If Level 3 items are not consistently harder than Level 1, or if the graph-grounding step collapses because facts are not entity-relation triples, then the depth-as-hardness premise is falsified. A more direct test: measure whether human error rates and probe entropy increase monotonically with path length within a single domain, controlling for question length and vocabulary.","supporting_citations":[],"review_version":1}