{"id":"6a2001cf-c2b2-4e88-8692-6eee205b6095","arxiv_id":"2505.05949","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A benchmark built from fictional news timelines shows LLMs frequently answer with shortcuts instead of deflecting when evidence is insufficient.","lead":"NeoQA is a new benchmark made of fictional news events and Q&A pairs, designed so AI models cannot answer from memory and must use provided evidence. It reveals that current large language models often fail to notice when evidence is missing or subtly mismatched.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evidence-sufficiency labels in NeoQA rest on imperfect NLI filtering; leaked unselected facts could make 'insufficient evidence' instances actually answerable, confounding the shortcut-reasoning result.","rationale":"The reader's weakest assumption identifies the same load-bearing point: the controlled evidence scenarios collapse if the article-to-outline-item mapping is wrong. I agree, and the paper's own numbers make the risk concrete: 7.3% of unselected outline-item/article pairs are NLI-'unknown', and Appendix E's filtering does not eliminate unknown-status unselected items that could carry the answer. A direct underivability check on the final instances is missing, and the human annotation does not cover it. Because the central shortcut-reasoning conclusion depends on these labels, the conditional verdict should stand; the abstract's 'no prior evidence' overclaim is secondary to this correctness risk.","tokens_in":51112,"tokens_out":6197,"duration_ms":67450,"concrete_test":"Sample 200 insufficient-evidence multi-hop instances from the main test split, stratified across 'missing answer' and 'missing bridge' conditions. For each, give annotators the question, answer options, and the exact evidence set used in the experiment, but not the omitted article; ask whether the correct answer is determinable with certainty from that evidence. If more than 5% of instances are judged answerable, the insufficiency labels are unreliable; recompute ADTScore and deflection rates after excluding or relabeling those instances.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The main experimental finding — that models answer as if evidence were sufficient — depends on the gold 'unanswerable' labels being correct. That label requires Requirement 2 (Section 4.3): news articles must convey all selected outline-item facts and exclude non-selected facts. The paper's verification is T5 NLI (Appendix D.2), which agrees on only 92.2% of unselected outline-item/article pairs and returns no label for 7.3% of them. Appendix E's 'conservative' filter then checks only the single selected outline item that is intentionally omitted; it does not purge articles whose unselected outline items are NLI-'unknown'. An article that in fact contains the omitted fact via a non-selected outline item — or via a paraphrase the NLI misses — makes the supposedly insufficient evidence sufficient. Requirement 1's necessity half is likewise unverified post-hoc: the LLM self-answer filter tested sufficiency, not uniqueness. If such leakage occurs, the high 'as-if-sufficient' answer rates in Figure 7 and the negative phi coefficients in Table 14 conflate correct evidence-based answering with shortcut reasoning. The 350-instance human annotation (Table 9) validates reference answers, not the sufficiency/insufficiency labels for constructed evidence sets.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces NeoQA, a fully synthetic question-answering benchmark built from fictional news timelines, with the stated goal of preventing LLMs from relying on parametric knowledge during RAG evaluation. The dataset links each question to atomic outline items and independently generated news articles, allowing automatic construction of instances with sufficient evidence, insufficient evidence, false-premise questions, and uncertain-specificity questions. The authors evaluate seven Phi and Qwen models, introduce the ADTScore metric as a harmonic mean of answerable and deflection accuracy, and report that models frequently answer as if evidence were sufficient when key information is missing, and that they struggle to detect subtle mismatches in false-premise and uncertain-specificity questions. The paper also reports quality checks including LLM filtering, T5 NLI verification, and human annotation of 350 instances.","tokens_in":51349,"tokens_out":4624,"duration_ms":51773,"significance":"If the quality-control assumptions hold, NeoQA is a valuable contribution: it addresses the well-known problem of benchmark staleness in RAG evaluation, provides controlled evidence conditions, and makes a concrete attack on shortcut reasoning. The experimental protocol is extensive and transparent: filtering steps are documented, NLI and human checks are reported, statistical significance is tested, and the dataset is released. The main significance, however, rests on the correctness of the sufficiency/insufficiency labels for evidence sets, and on the claim that parametric knowledge cannot help. Those two load-bearing points are only partially verified, and the paper itself provides evidence of token-distribution leakage (Section 5). With additional validation or appropriately qualified claims, the benchmark would be a solid contribution.","major_comments":[{"comment":"Requirement 2 is only partially verified, and the instance construction does not fully enforce it. The T5 NLI check (Table 8) agrees with the expected label on only 92.2% of unselected outline-item/article pairs and leaves 7.3% of them unlabeled. The conservative filtering described in Appendix E checks, for each insufficient-evidence instance, only the single intentionally omitted outline item; it does not require that all other non-selected outline items be NLI-verified as absent from the retained articles. A news article that entails the answer fact through a different outline item, or through a paraphrase that the NLI model misses, makes a supposedly insufficient evidence set sufficient. Since the central shortcut-reasoning result (Figure 7, Table 14) is defined against these sufficiency labels, this leakage would conflate correct evidence-based answering with shortcut reasoning. I recommend either stronger post-hoc filtering over all non-selected items or a sensitivity analysis that excludes instances with any NLI-unknown unselected items.","section":"Section 4.3 and Appendix E"},{"comment":"The no-evidence accuracy of 53.6% on multi-hop questions, far above the 25% random baseline, demonstrates token-distribution leakage from the synthetic generation process and directly undercuts the claim that NeoQA prevents LLMs from leveraging pretraining knowledge. The paper acknowledges this and hypothesizes that the dataset was sampled from the same token distribution, but this leakage has the same consequence for the controlled experiments as parametric knowledge: a model can produce an \"as-if-sufficient\" answer on an insufficient-evidence instance without doing evidence-based reasoning. The paper should either quantify and remove leakage-prone instances, or substantially weaken the claim that no prior evidence exists in training data, and should discuss how leakage affects the deflection results.","section":"Section 5, GPT-4 Turbo no-evidence probe"},{"comment":"The human annotation validates the reference answer, not the sufficiency or insufficiency labels for the constructed evidence sets. For unanswerable questions, annotators are shown only the relevant outline items rather than the full news-article evidence sets, so the reported 94% agreement (Table 9) does not confirm that the full evidence sets are actually insufficient. Given that the entire controlled-evidence experiment depends on the correctness of these labels, the paper needs an additional human or automated validation at the level of complete instances, or an explicit statement that label validity is assumed rather than verified.","section":"Appendix D.3"},{"comment":"Requirement 1 is only checked for sufficiency, not necessity. The LLM self-answer filter removes questions that cannot be answered from the selected outline items, but it does not check the instruction that no other outline item can answer the question. If an unselected outline item also determines the answer, then removing the selected evidence items may still leave sufficient evidence, again invalidating the insufficient-evidence labels. This is a separate gap from the NLI check on news articles and should be tested directly.","section":"Appendix D.1"}],"minor_comments":[{"comment":"The figure caption contains typos: \"Anwserable\" and \"Unanwserable\" should be \"Answerable\" and \"Unanswerable\".","section":"Figure 2"},{"comment":"The subsection numbering is inconsistent: \"C.3 News Articles\" is immediately followed by \"C.4 Generation\", which appears to be a continuation of C.3 rather than a separate subsection.","section":"Appendix C"},{"comment":"The sentence \"In most cases the outline items where predicted as 'unknown' rather than 'entailed'\" should read \"were predicted\".","section":"Table 8 and Appendix D.2"},{"comment":"The panel labels and color coding in Figure 7 are hard to parse in printed form; adding explicit row and column headers (e.g., \"No Answer\" and \"No Bridge\") directly above each diagram would improve readability.","section":"Figure 7"}],"recommendation":"major_revision","confidential_remarks":"The core idea is strong and the experiments are unusually transparent for a benchmark paper, but the two load-bearing claims—that evidence sufficiency labels are reliable and that parametric knowledge cannot help—are only partially supported. The GPT-4 Turbo no-evidence result in Section 5 is particularly important because it is evidence from the authors' own evaluation that the central premise does not hold cleanly. I would be comfortable with acceptance after the authors either strengthen the evidence-label validation or substantially qualify the central claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"NeoQA is worth your time: it builds a contamination-resistant RAG benchmark on fully fictional timelines with recurring entities, and it is the first to combine controlled evidence sufficiency with false-premise and uncertain-specificity variants. The construction pipeline is elaborate and transparent—sequential event outlines, Wikipedia cross-checks for entity names, NLI filtering, human annotation, and a conservative instance-selection strategy. The experiments cover seven open-weight models plus GPT-4 Turbo probes, with careful attention to prediction patterns for missing answer vs. missing bridge information. The central finding—models often answer as if evidence were sufficient when the bridge entity is missing—is plausible and consistent with prior work on shortcut reasoning. That result alone makes the dataset useful.\n\nWhere I'd push back: the abstract claims no prior evidence can exist in pretraining data, but the paper's own GPT-4 Turbo probe shows 53.6% accuracy on multi-hop questions with no evidence, which the authors attribute to token-distribution leakage. That overclaim should be qualified. More importantly, the controlled 'insufficient evidence' labels depend on Requirement 2—news articles must exclude non-selected outline items—verified only by a T5 NLI model that agrees on 92.2% of unselected pairs and leaves 7.3% unlabeled. The conservative filter drops the unknown cases, which helps, but an article that paraphrases the omitted fact in a way the NLI misses would make the supposedly insufficient evidence sufficient, weakening the shortcut-reasoning interpretation for those instances. The 350-instance human annotation checks reference answers, not the sufficiency labels. This is a real soft spot, but not a fatal one; the direction of the findings is unlikely to reverse entirely.\n\nNet: this is a strong benchmark paper, better executed than most, with an honest limitations section that acknowledges the leakage issue. It deserves a serious referee and likely publication after the abstract is aligned with the body and the evidence-label verification is discussed more carefully. I'd cite it for future RAG evaluation.","headline":"NeoQA is a genuinely useful contamination-resistant RAG benchmark with a solid empirical study, though its abstract overstates leakage resistance and the insufficiency labels rest on imperfect NLI verification.","tokens_in":51851,"tokens_out":2267,"would_cite":true,"duration_ms":22794,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that evidence-based QA benchmarks go stale because LLMs absorb news during pretraining, and that its all-fictional benchmark, NeoQA, neutralizes that advantage — only to reveal that models shortcut past missing evidence…","keywords":["retrieval-augmented generation","evidence-based question answering","fictional timelines","shortcut reasoning","unanswerable questions","parametric knowledge interference","benchmark staleness","ADTScore"],"falsifier":"Take a random sample of the insufficient-evidence instances and have annotators judge whether the news articles do in fact state the information the pipeline claims to have omitted; if a substantial share of instances contain the supposedly missing facts, the deflection labels are wrong and the shortcut-reasoning finding would be an artifact of mislabeled evidence rather than a model behavior.","tokens_in":1843,"feed_emoji":"📰","tokens_out":4016,"duration_ms":82407,"temperature":0.7,"pith_summary":"NeoQA is a benchmark built entirely from fictional news events, designed so that no language model can answer its questions from pretrained memory, only from the evidence documents it is given. The paper's central claim is that this controlled setting exposes a specific failure mode: when key evidence is missing, LLMs often answer anyway, as though the evidence were complete, instead of deflecting. Experiments on seven LLMs show that models handle answerable multi-hop questions well but frequently fail on insufficient-evidence cases, especially when the question's bridge entity is missing from the evidence, and that they struggle to detect subtle false-premise and over-specific questions. The paper also demonstrates empirically that older real-world RAG benchmarks can be solved without retrieval at all, because newer models have internalized the relevant news. If the benchmark works as intended, it gives RAG evaluation a stable, contamination-resistant platform for measuring whether a model can honestly say \"I do not know.\"","feed_headline":"Fake-news benchmark traps LLMs answering without evidence","feed_subtitle":"Models kept answering when key evidence was missing, instead of deflecting — that failure is now measurable.","key_machinery":"The load-bearing mechanism is the fictional timeline: ten sequentially generated events, each with an outline of 20-30 atomic outline items, a knowledge base of fictional named entities checked against Wikipedia, and news articles generated from selected subsets of outline items under four reporting profiles. Questions — multi-hop with a shared bridge entity, time-span, false premise, and uncertain specificity — are grounded in exactly the outline items needed to answer them, so the dataset can automatically pair any question with evidence that is sufficient, insufficient, or distracting. Two requirements enforce this: the selected outline items must be necessary and sufficient for the answer (Requirement 1), and news articles must convey all selected outline facts while excluding unselected ones (Requirement 2). The evaluation metric, ADTScore, is the harmonic mean $$\\text{ADTScore} = \\frac{2\\, \\mathrm{acc}_a \\, \\mathrm{acc}_u}{\\mathrm{acc}_a + \\mathrm{acc}_u}$$ of accuracy on answerable instances and accuracy on unanswerable instances requiring deflection, which penalizes a model that answers well but refuses to say \"I do not know.\"","core_discovery":"NeoQA establishes that when questions and evidence are both drawn from fictional timelines, so that parametric knowledge cannot help, LLMs frequently resort to shortcut reasoning: instead of verifying that the evidence supports the question, they answer as if nothing were missing. When the evidence omitted the answer information, models tended to pick a misleading distractor (52.9%-77.9% of errors), and when it omitted only the bridge-entity information, models answered as if the evidence were sufficient in 69.7%-90.7% of errors. The paper also reports a significant negative association between accuracy on answerable multi-hop questions and correct deflection on subtle variants (false premise, uncertain specificity, and bridge-entity omission), suggesting that the same reasoning strength that produces correct answers can drive over-confidence when evidence is incomplete. The companion RealTimeQA experiment shows that a model's accuracy on dated news quizzes rises with question age, confirming that benchmarks based on real events lose their ability to measure evidence-based reasoning as models absorb those events during pretraining.","pith_inferences":["Because NeoQA's entities are fictional, the benchmark can double as a contamination detector: if a future model answers NeoQA questions above chance without evidence, that would signal leakage of the benchmark itself rather than genuine world knowledge.","The benchmark's controlled evidence design could be transplanted to other evaluation settings where parametric interference distorts measurement, such as tool-use planning, long-context summarization, or counterfactual reasoning tasks.","The near-random no-evidence accuracy on time-span questions but above-chance accuracy on multi-hop questions hints that some question phrasings leak statistical cues from the generating model; a systematic probe of how much signal resides in the question rather than the evidence could sharpen the benchmark further."],"forward_implications":["Real-world RAG benchmarks are vulnerable to parametric knowledge leakage: accuracy that supposedly measures retrieval can instead measure memorization, so evaluations on dated events need either fictional worlds or frequent regeneration.","A model can score well on answerable questions while failing on unanswerable ones, so RAG evaluation should report balanced measures like ADTScore rather than raw answer accuracy.","When evidence is incomplete, the most dangerous case is not missing everything but missing the bridge entity while the answer is present: models then answer confidently on the basis of partial support, which is precisely the behavior that undermines trust in deployed systems.","Adding irrelevant documents degrades performance most sharply within the first 20 documents, indicating that retrieval noise, not context length alone, drives much of the accuracy loss in long-context RAG."],"supporting_citations":[{"why":"Supplies RealTimeQA, the weekly news-quiz dataset whose pre-cutoff accuracy decay motivates the paper's staleness argument.","marker":"Kasai et al. (2024)"},{"why":"HotpotQA provides the bridge-entity multi-hop question paradigm that NeoQA adapts, and the real-world contrast for its shortcut-reasoning discussion.","marker":"Yang et al. (2018)"},{"why":"The T5-XXL NLI model enforces Requirement 2 by checking that news articles entail selected outline items and exclude unselected ones.","marker":"Honovich et al. (2022)"},{"why":"Michelangelo is the closest prior benchmark with out-of-pretraining data and IDK questions; NeoQA extends it with recurring entities and subtle mismatches.","marker":"Vodrahalli et al. (2024)"},{"why":"Supports the design choice of always including an explicit unanswerable option to help models deflect.","marker":"Slobodkin et al. (2023)"},{"why":"Frames retrieval-augmented generation as the task that NeoQA is built to evaluate.","marker":"Lewis et al. (2020)"},{"why":"Provides the contrastive methodology of reusing the same question with varying evidence, which NeoQA adopts for its controlled instances.","marker":"Schuster et al. (2021)"}],"fun_headline_variants":["Fictional news events expose LLM evidence gaps","LLMs flunk evidence check on made-up news stories","LLMs can't say 'not enough evidence' in new benchmark","NeoQA: Fake news events test evidence-based QA","New benchmark forces LLMs to reason from evidence alone"],"cache_read_input_tokens":54016,"weakest_assumption_plain":"The entire controlled-evidence design rests on the claim that news articles contain every fact from the selected outline items and nothing from the non-selected ones; if that mapping is wrong, the labels \"sufficient\" and \"insufficient\" are wrong, and every deflection result collapses.","fun_headline_variants_meta":{"raw":{"variants":["Fictional news events expose LLM evidence gaps","LLMs flunk evidence check on made-up news stories","LLMs can't say 'not enough evidence' in new benchmark","NeoQA: Fake news events test evidence-based QA","New benchmark forces LLMs to reason from evidence alone"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001497,"raw_usage":{"total_tokens":6010,"prompt_tokens":948,"completion_tokens":5062,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":4982}},"tokens_in":564,"tokens_out":5062,"duration_ms":34277,"temperature":1.0,"reasoning_tokens":4982,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:51:40.366701+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the insufficient-evidence instances and have annotators judge whether the news articles do in fact state the information the pipeline claims to have omitted; if a substantial share of instances contain the supposedly missing facts, the deflection labels are wrong and the shortcut-reasoning finding would be an artifact of mislabeled evidence rather than a model behavior.","supporting_citations":[],"review_version":1}