{"id":"18062ee8-0dc2-4332-bda3-9c7cfe1be4c8","arxiv_id":"2412.17032","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MINTQA provides 28,366 multi-hop QA pairs across popular/unpopular and old/new knowledge, with sub-questions, and shows that even the best LLMs achieve only about 62% accuracy even with retrieval.","lead":"MINTQA is a new benchmark of about 28,000 multi-hop questions built from Wikidata, designed to test how well large language models handle rare or newly added knowledge. It evaluates 22 LLMs and finds they struggle especially on new knowledge, and that adding retrieval helps but does not close the gap.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MINTQA-TI's 'new knowledge' label conflates Wikidata first appearance with fact recency: old facts added to Wikidata in 2024 can be labeled 'new', so the headline temporal-knowledge finding lacks direct support.","rationale":"The paper is a serious construction effort: it builds 28,366 QA pairs from Wikidata fact chains, applies connectivity/acyclicity/no-shortcut constraints, validates generated questions via GPT-4o, and reports a 500-instance human check with only 2% insufficient-context cases. Those elements provide real support for MINTQA as a multi-hop QA resource with popularity/newness annotations. The load-bearing weakness is not the construction pipeline but the interpretive step from 'absent from Wikidata 2021-06-21' to 'new knowledge' (Section 3.2). Wikidata is a community-curated database; a triple's first appearance date is not the date the fact became true or became publicly known. LLM pretraining corpora contain enormous amounts of historical and encyclopedic text, so an old fact newly added to Wikidata may already be in the model. Conversely, some genuinely recent facts may never be added to Wikidata promptly. As the paper's own Limitation notes, Wikidata distribution patterns are only a proxy for LLM knowledge; the newness proxy has an additional, unaddressed confound. The reader's weakest assumption pointed at the popularity/newness proxies broadly; this concern sharpens that to the temporal definition and proposes a direct test. Since the reader already returned CONDITIONAL and required proxy validation, this does not move the verdict; it does make the required validation more specific. If the proposed test shows the confound is large, MINTQA-TI would at best measure 'new-to-Wikidata' facts, and the abstract's claim about handling new knowledge would need qualification.","tokens_in":39016,"tokens_out":4324,"duration_ms":39086,"concrete_test":"Sample 300 MINTQA-TI questions labeled 'new' and, for each, determine whether the underlying fact genuinely emerged after mid-2021 (event date, entity inception, or first reliable source) or is an older fact only added to Wikidata between 2021 and 2024 (using Wikidata revision history and external references). Then compare mean model accuracy (e.g., GPT-4o and LLaMA-3.1-70B) between the two groups, controlling for hop count. If accuracy on genuinely new facts is not substantially lower than on late-curated old facts, the benchmark's temporal distinction is confounded and the headline new-knowledge conclusions should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that MINTQA evaluates LLMs on 'new knowledge' depends on the Section 3.2 definition: a fact is 'new' if it appears in the 2024-06-05 Wikidata dump but not in the 2021-06-21 dump. Absence from an older dump is a curation artifact, not a timestamp of the underlying fact. A triple about a pre-existing entity or historical event (e.g., 'Pigeon Bay Domain, country, New Zealand', shown as Type: New in Appendix E) can be added in 2024 because of imports, bot edits, or community work; the fact itself may have been true and documented long before 2021 and may be present in LLM pretraining corpora. Thus MINTQA-TI's low parametric accuracy (GPT-4o: 21.17% on one-hop new questions) could reflect obscurity, entity rarity, or late Wikidata ingestion rather than knowledge that postdates training. The paper's Limitation acknowledges that Wikidata distribution patterns may not track LLM pretraining data, but it does not address this specific conflation of Wikidata first appearance with fact newness. If the temporal proxy is confounded, the abstract's claim that models are 'particularly' limited on new knowledge is not established by MINTQA-TI.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MINTQA, a multi-hop question answering benchmark constructed from Wikidata triples and Wikipedia frequencies, comprising two subsets: MINTQA-POP (17,887 pairs) for popular/unpopular knowledge and MINTQA-TI (10,479 pairs) for old/new knowledge, with one- to four-hop chains and GPT-4o-generated sub-questions. The authors evaluate 22 LLMs across five settings (parametric knowledge, strategy selection, direct retrieval, sub-question generation, and decomposition-with-retrieval) and report that models perform markedly worse on new and unpopular knowledge, with even the best system achieving only 62.33% overall accuracy.","tokens_in":39295,"tokens_out":7594,"duration_ms":64630,"significance":"If the benchmark's construct validity holds, MINTQA would fill a clear gap in multi-hop QA evaluation, and the systematic comparison of 22 models under multiple reasoning strategies would be a valuable community resource. The dataset is released with a permissive license, includes sub-question annotations, and the paper provides a substantial amount of evaluation detail. However, the validity of the core 'new knowledge' claim and the integrity of the multi-hop annotation in retrieval settings are not fully established, so the significance depends on the authors' ability to address the issues below.","major_comments":[{"comment":"The operationalization of 'new knowledge' as triples present in the 2024-06-05 Wikidata dump but absent from the 2021-06-21 dump conflates Wikidata ingestion time with fact recency. A concrete instance is Appendix E, Table 22, where the triple (Pigeon Bay Domain, country, New Zealand) is labeled Type: New, even though this is a stable geographical fact that almost certainly predates 2021 and may well appear in LLM pretraining corpora. As a result, the low parametric accuracy on MINTQA-TI (e.g., GPT-4o at 21.17% for single-hop questions, Section 5) cannot be unambiguously attributed to knowledge that postdates training. The paper's Limitation section acknowledges that Wikidata distribution patterns may not reflect LLM pretraining data, but it does not address this specific conflation. To support the abstract's claim that models are 'particularly' limited on new knowledge, the authors should either re-annotate the 'new' subset using a genuine temporal signal (e.g., entity inception dates or event dates) or explicitly reinterpret MINTQA-TI as measuring 'recently added to Wikidata' and adjust the conclusions accordingly.","section":"Section 3.2; Appendix E, Table 22"},{"comment":"The No Shortcuts check is defined only over Gpop, but the retrieval corpus used in the evaluation is G = Gpop ∪ Gti (Section 7.1). A chain sampled from Gpop can therefore be shortcut by a triple that exists only in Gti (and vice versa for chains from Gti), so the effective number of hops in the retrieval experiments may be lower than annotated. Because the paper's multi-hop validity claims apply to the retrieval-augmented settings (Sections 7 and 9), the No Shortcuts constraint should be verified against the full retrieval corpus, not just the subgraph from which the chain was sampled.","section":"Section 3.1, constraint 4; Section 7.1"},{"comment":"The popularity thresholds (unpopular: [1,10), popular: [50,∞)) are arbitrary and no sensitivity analysis is provided. Since the central dimension of MINTQA-POP is defined by this binarization, the paper should report how the main conclusions (e.g., the performance gap between popular and unpopular questions) change under reasonable alternative thresholds, or justify the chosen cutoffs empirically.","section":"Section 3.1"}],"minor_comments":[{"comment":"The caption and column layout are difficult to parse; the reader cannot easily determine which columns correspond to MINTQA-POP versus MINTQA-TI and how the 'proportion' values map to the actual hop counts. Please clarify the column headers or separate the POP and TI results into distinct tables.","section":"Table 8"},{"comment":"The specific numbers '21.17%' (GPT-4o) and '7.78%' (LLaMA-3.2-1B) for single-hop new-knowledge questions should be accompanied by an explicit pointer to the table that contains these results, since Table 8 in the appendix does not report hop-disaggregated values in an obvious way.","section":"Section 5"},{"comment":"The phrase 'MINTQAQA dataset' contains a typo; it should read 'MINTQA dataset'.","section":"Appendix B"},{"comment":"The label 'Parametric Konwlodege Exploration' contains a spelling error; it should read 'Parametric Knowledge Exploration'.","section":"Figure 2"},{"comment":"The phrase 'LLaMA-3.1-8B sees a 33.41% increase' actually refers to a 33.41 percentage-point increase (from 37.28 to 70.69); please rephrase for precision.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The benchmark resource and the breadth of the evaluation are genuinely useful. The major concerns are fixable: re-annotating or re-framing the 'new' knowledge subset, and checking No Shortcuts against the full retrieval corpus. If the authors provide a convincing re-analysis or a clearly scoped reinterpretation, the paper could become acceptable. I would not reject at this stage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MINTQA is a genuinely useful resource. It is the first multi-hop QA benchmark I know of that jointly varies popularity and temporal recency, and it ships with sub-questions for every hop, 28k+ questions, and an evaluation of 22 LLMs across several axes: parametric answering, strategy selection, retrieval, decomposition, and dynamic retrieval. The popularity axis, based on Wikipedia occurrence frequency, is a defensible proxy for training-data exposure, and the finding that retrieval helps more on unpopular and new questions is plausible and consistent with prior work. The dataset itself, with gold sub-questions and answers, is a real contribution for the RAG and multi-hop reasoning communities.\n\nThe soft spot is the temporal axis. Section 3.2 defines \"new\" as present in the 2024 Wikidata dump but absent from the 2021 dump. That conflates Wikidata first appearance with fact recency. A triple like (Pigeon Bay Domain, country, New Zealand), which the appendix labels \"new,\" is an old fact that was likely added to Wikidata late because of curation or import. It may well be in LLM pretraining corpora. The stress-test note is correct: the low parametric accuracy on MINTQA-TI (e.g., GPT-4o at 21.17% on one-hop new questions) could reflect obscurity, entity rarity, or late Wikidata ingestion rather than knowledge that postdates training. The paper's limitation section acknowledges that Wikidata distributions may not reflect LLM pretraining, but it does not address this specific conflation. I would not trust the temporal headline finding as stated.\n\nOther concerns are minor but worth fixing: the No Shortcuts check is only applied to Gpop, so shortcuts could exist in the full retrieval corpus; question generation and validation both use GPT-4o, though the human check of 500 samples (2% contextual insufficiency) is reassuring; and there are no error bars or significance tests across the 22-model comparison. The GitHub repo could not be verified.\n\nWho should read this? Anyone building benchmarks for RAG or studying what LLMs know, and anyone who wants a multi-hop dataset with sub-questions and controlled knowledge properties. The paper deserves a serious referee. I would send it to review with a request to either validate the temporal proxy against something like pretraining cutoff dates or a temporal QA set like FreshQA, or to reframe the claims around \"sparsely documented in Wikidata\" rather than \"new.\" As a dataset resource, it is worth engaging with; as a demonstration that LLMs fail on genuinely new knowledge, the evidence is not yet there.","headline":"MINTQA is a substantial and useful multi-hop QA dataset, but its 'new knowledge' label rests on a weak Wikidata-first-appearance proxy, so the headline temporal finding needs qualification.","tokens_in":39852,"tokens_out":2123,"would_cite":true,"duration_ms":21502,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MINTQA is a benchmark for multi-hop QA over new and tail knowledge, and it shows that current LLMs, including GPT-4o, fail badly on such queries.","keywords":["multi-hop question answering","long-tail knowledge","new knowledge","retrieval-augmented generation","LLM evaluation","Wikidata benchmark","question decomposition","knowledge boundary"],"falsifier":"Take a matched sample of MINTQA-TI 'new' facts and 'old' facts with the same answer-entity popularity and relation types, and probe an LLM whose training cutoff predates 2021; if accuracy on the two sets is statistically indistinguishable, the new/old split is not isolating knowledge recency but is instead tracking popularity or entity type.","tokens_in":38805,"feed_emoji":"🧩","tokens_out":7078,"duration_ms":58255,"temperature":0.7,"pith_summary":"The paper introduces MINTQA, a benchmark of 28,366 multi-hop question-answer pairs built from Wikidata and Wikipedia, split into MINTQA-POP (17,887 questions on popular versus unpopular knowledge) and MINTQA-TI (10,479 questions on old versus new knowledge). Its purpose is to measure whether LLMs can handle queries that require several reasoning hops and facts that are rare or have only recently appeared. The paper evaluates 22 LLMs and reports that accuracy collapses on new and unpopular knowledge, with GPT-4o falling to 21.17% on single-hop new-knowledge questions and performance degrading further as hop count rises. Retrieval helps, but less on the hardest cases, and even giving models the right sub-questions does not close the gap on new knowledge. The central claim is that current multi-hop QA evaluation misses a distinct capability: coordinating decomposition and retrieval when knowledge is fresh or tail.","feed_headline":"New benchmark: LLMs score as low as 21% on fresh facts","feed_subtitle":"28,366 Wikidata-based questions show retrieval is still needed for rare and recent facts.","key_machinery":"The central object is the fact chain: an ordered sequence of Wikidata triples $(s_1,r_1,o_1), \\dots, (s_n,r_n,o_n)$ with $o_i = s_{i+1}$, no cycles, no chain that is a sub-chain of another, and no shortcut triple connecting non-adjacent entities. Each chain grounds a multi-hop question whose answer is $o_n$, with per-hop sub-questions generated and validated by GPT-4o. The popularity and time splits rest on two proxies (Wikipedia counts for popular versus unpopular, Wikidata dump versions for old versus new), and the evaluation machinery isolates five capabilities: parametric answering, strategy choice, retrieval benefit, sub-question generation, and iterative or dynamic retrieval.","core_discovery":"MINTQA is constructed by chaining connected Wikidata triples into fact chains of one to four hops, with constraints that forbid cycles, shortcuts, and duplicate sub-chains, and by using GPT-4o to turn each chain into a question plus validated sub-questions. Popularity is defined by occurrence frequency in Wikipedia (unpopular: 1 to 9 occurrences; popular: at least 50), and newness by presence in the 2024-06-05 Wikidata dump but absence from the 2021-06-21 dump. Evaluating 22 instruction-tuned LLMs across five protocols, the paper finds that parametric accuracy on MINTQA-TI single-hop questions ranges from 7.78% to 21.17%, that knowledge-boundary awareness improves with model size but is unreliable, that direct retrieval gives roughly a 30-point gain on TI, that self-generated sub-questions hurt TI accuracy, and that decomposition-then-retrieval underperforms direct retrieval on new knowledge. The paper concludes that MINTQA measures a real capability gap in coordinating retrieval and decomposition on fresh and tail knowledge, and that current models, even with retrieval, are far from closing it.","pith_inferences":["If the popularity proxy is imperfect, the headline gap may partly reflect entity-type confounds, because unpopular triples in Wikipedia tend to involve niche entities that are also less likely to appear in training corpora; a controlled test matching entity types would separate the two effects.","A natural extension is to use MINTQA's sub-question pairs as supervision for training a decomposition model, then test whether that training transfers to unseen fresh facts; the paper does not train such a model.","The new/old split could be made dynamic by re-running the construction on later Wikidata dumps, turning current 'new' facts into 'old' ones and letting researchers measure how quickly model knowledge catches up.","The near-ceiling performance with gold sub-questions and gold retrieval suggests the next bottleneck is document-level synthesis, not decomposition or retrieval alone; this goes beyond the paper's explicit claims."],"forward_implications":["Benchmarks that only test popular or static facts overstate LLM multi-hop capability, so including unpopular and newly added facts is necessary to expose the boundary.","Retrieval-augmented generation is not a uniform fix: direct retrieval helps new knowledge, but iterative decomposition-then-retrieval can hurt, indicating the decomposition step itself is a bottleneck.","Giving a model gold sub-questions substantially improves tail-knowledge accuracy but leaves new-knowledge accuracy low, meaning the missing ingredient is the facts themselves, not just the reasoning plan.","Models show systematic overconfidence: smaller models often answer directly or assume the answer is already in hand, while even large models struggle to judge when their own knowledge is insufficient.","Confidence-gated dynamic retrieval does not yet maintain accuracy while cutting retrieval calls, so efficiency gains come at a real accuracy cost on this benchmark."],"supporting_citations":[{"why":"Supplies the GPT-4o-based question generation approach and the retrieval-augmentation baseline that MINTQA extends to multi-hop chains.","marker":"(Maekawa et al., 2024)"},{"why":"Defines the parametric versus non-parametric memory framing and long-tail QA evaluation that MINTQA-POP builds on.","marker":"(Mallen et al., 2022)"},{"why":"Provides the head-to-tail entity popularity analysis that motivates the unpopular-knowledge split.","marker":"(Sun et al., 2023)"},{"why":"MultiHop-RAG is the multi-hop retrieval benchmark MINTQA compares against and extends with new and tail knowledge.","marker":"(Tang and Yang, 2024)"},{"why":"MuSiQue supplies the compositional multi-hop question construction approach that MINTQA adapts to Wikidata fact chains.","marker":"(Trivedi et al., 2021)"},{"why":"Provides the confidence-guided dynamic retrieval method that the paper implements and evaluates.","marker":"(Ni et al., 2024)"},{"why":"Supplies the knowledge-graph linearization used to build the retrieval corpus.","marker":"(Yu et al., 2023)"},{"why":"BM25 is the sparse retriever baseline that performs best on the KG-linearized corpus.","marker":"(Robertson and Zaragoza, 2009)"}],"fun_headline_variants":["New benchmark MINTQA: LLMs score as low as 7.78% on fresh facts","MINTQA: 28k new-knowledge questions stump even the best LLMs","Fresh facts hardest: MINTQA shows top LLMs only hit 21%","Rare knowledge is a wall: MINTQA benchmark exposes LLM gap","From 7% to 21%: MINTQA reveals LLMs' weak grasp of new facts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that Wikipedia mention counts and a triple's absence from an older Wikidata dump really mark the edge of what LLMs know, and if that proxy is wrong the central contrast between new/tail and old/popular knowledge stops measuring actual knowledge.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark MINTQA: LLMs score as low as 7.78% on fresh facts","MINTQA: 28k new-knowledge questions stump even the best LLMs","Fresh facts hardest: MINTQA shows top LLMs only hit 21%","Rare knowledge is a wall: MINTQA benchmark exposes LLM gap","From 7% to 21%: MINTQA reveals LLMs' weak grasp of new facts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000351,"raw_usage":{"total_tokens":1930,"prompt_tokens":980,"completion_tokens":950,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":835}},"tokens_in":596,"tokens_out":950,"duration_ms":7867,"temperature":1.0,"reasoning_tokens":835,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:51:05.001422+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a matched sample of MINTQA-TI 'new' facts and 'old' facts with the same answer-entity popularity and relation types, and probe an LLM whose training cutoff predates 2021; if accuracy on the two sets is statistically indistinguishable, the new/old split is not isolating knowledge recency but is instead tracking popularity or entity type.","supporting_citations":[{"cited_title":"Retrieval Helps or Hurts? A Deeper Dive into the Efficacy of Retrieval Augmentation to Language Models","cited_arxiv_id":"2402.13492","evidence_quote":"Supplies the GPT-4o-based question generation approach and the retrieval-augmentation baseline that MINTQA extends to multi-hop chains."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the parametric versus non-parametric memory framing and long-tail QA evaluation that MINTQA-POP builds on."}],"review_version":1}