{"id":"ab93dee7-c0d1-4c5b-a321-13cb2ec669aa","arxiv_id":"2501.12975","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"OnionEval reports that small LLMs detect atomic fact hallucinations well but fail in layered contexts, a result undermined by prompt confounds and numeric inconsistencies.","lead":"The paper introduces OnionEval, a benchmark that wraps simple facts in layered stories to test how well small language models spot false statements. The authors report that small models excel at isolated facts but collapse once context is added, a conclusion this review finds confounded by prompt changes and inconsistent data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The context effect is confounded with a prompt change: atomic and context prompts differ in task frame, answer polarity, and added CoT instruction, so the claimed SLLM context deficit is not evidenced.","rationale":"The load-bearing step in the paper's argument is the comparison between atomic-layer accuracy and context-layer accuracy in Table 2. If that comparison is invalid, the central claim in §4.2—that SLLMs suffer a specific context-induced hallucination deficit—has no support. The appendix shows the comparison is not controlled: the atomic and context prompts differ in the judgment target (statement vs Mike's belief), in the meaning of \"yes\" (wrong vs correct), and in the presence of \"Think step by step.\" The paper's own §3.4 claims the discrimination task stays consistent, but the appendix contradicts it. This concern is empirical and can be settled by a matched-prompt ablation. I agree with the reader's weakest_assumption. I also note secondary numerical inconsistencies that make the reported near-zero accuracies hard to trust, but the prompt mismatch alone is sufficient to undermine the headline claim. The limitation section does not flag any of this. Since the central claim is not supported by the evidence as presented, the verdict should remain REJECT; the underlying layered-benchmark idea could be salvageable, but this paper would need a controlled prompt design and corrected numbers.","tokens_in":11017,"tokens_out":4727,"duration_ms":49515,"concrete_test":"Re-run Llama3.1-8B and Llama3.2-3B on a fixed sample (e.g., 200 items) under four conditions: (A) atomic prompt as in Table 4; (B) first-layer prompt as in Table 5; (C) first-layer context with the atomic framing \"Judge whether the statement in the context is wrong; answer yes if it is wrong\" and without \"Think step by step\"; (D) atomic prompt with the context framing/polarity and \"Think step by step.\" If accuracy in (C) is close to (A), or accuracy in (D) drops to the near-zero range, the reported context effect is an artifact of prompt differences rather than context. Report per-condition accuracy and recompute CI from conditions (C) and (D).","verdict_should_be":"REJECT","load_bearing_attack":"The central claim in §4.2 that SLLMs collapse on context-wrapped hallucination detection rests on comparing accuracy across the atomic layer and context layers. But Appendix A shows the comparison is not matched. Table 4 (atomic) asks \"Judge whether the statement I give you is wrong. If it is wrong, reply yes,\" while Tables 5 and 6 (context layers) ask \"judge if Mike's opinion is correct or not,\" add \"Think step by step,\" and invert the response mapping (yes = wrong vs yes = correct). This changes the object of judgment (a statement vs an agent's belief), the polarity of \"yes,\" and the reasoning instruction all at once. Because the context layers pair a hallucinated author's statement with Mike's true belief, a model that is sensitive to any of these prompt properties can score near zero or near perfect for reasons unrelated to context reasoning. The paper's own §3.4 asserts that the binary discrimination process \"remains consistent\" across layers, but the appendix contradicts that assertion. The Limitation section (§6) does not flag the mismatch. Secondary numerical inconsistencies (Table 2 Layer 1 for Llama3.1-8B is 0.15%, while §4.3 says 0.03%; Llama3.2-3B Layer 1 is 0.9% vs 23.92%) further weaken confidence that the reported near-zero values are stable measurements. Without matched prompts, the central claim that context wrapping causes the drop is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OnionEval, a benchmark for fact-conflicting hallucination detection in small LLMs, built from atomic knowledge-graph facts that are wrapped in two progressively richer narrative contexts. It introduces a Context Influence (CI) score and reports accuracy and CI results for Llama, Gemma, and Qwen models from 3B to 72B. The authors claim that SLLMs perform comparably to large models on atomic facts but collapse when context is added, and that chain-of-thought prompting substantially repairs the deficit.","tokens_in":11263,"tokens_out":7647,"duration_ms":80922,"significance":"OnionEval addresses a genuine gap: few hallucination benchmarks focus on small LLMs or manipulate context level in a controlled way. The paper makes its dataset and code public, includes human cross-validation, evaluates multiple model families, and reports zero-temperature runs. However, the central comparison is confounded by a prompt mismatch between the atomic and context layers, and the reported numbers contain internal contradictions. As presented, the headline claim that context wrapping causes SLLM accuracy to collapse is not supported by the evidence.","major_comments":[{"comment":"The paper's main claim in §4.2 that SLLMs' accuracy collapses under context rests on comparing the atomic layer with the context layers. The prompts are not matched: the atomic prompt asks whether a statement is wrong and maps 'yes' to wrong, whereas the context prompts ask whether Mike's opinion is correct, map 'yes' to correct, add 'Think step by step,' and change the judged object from a statement to an agent's belief. Thus the drop from 98.49% to 0.15% for Llama3.1-8B could be caused by flipped polarity, the added CoT instruction, or the reframing, rather than by contextual information. Moreover, in the displayed context examples Mike's opinion is always the true atomic fact, so the correct answer is always 'yes' and accuracy could reflect a response bias instead of contextual reasoning. This design contradicts §3.4's assertion that the binary discrimination process 'remains consistent' across layers. The central claim can only be tested with prompts that are identical in task frame, answer polarity, and reasoning instruction, with context as the sole manipulated variable.","section":"§3.4, Appendix A (Tables 4–6)"},{"comment":"The baseline first-layer accuracy for Llama3.2-3B is 0.9% in Table 2 but 23.92% in §4.3, and for Llama3.1-8B it is 0.15% in Table 2 but 0.03% in §4.3. Because the mitigation results are presented as improvements over these baselines (e.g., 'from 0.03% to 69.45%'), the inconsistencies make the reported gains impossible to verify. The authors need to report one consistent set of baseline numbers and re-derive all mitigation comparisons from it.","section":"§4.3 vs. Table 2 and Table 3"},{"comment":"The CI scores are reported as point estimates without confidence intervals, error bars, or significance tests, yet the text contrasts scores such as 99.66 and 10.56 as evidence of a systematic size effect. With only one model per size in each family, and with the underlying accuracies already affected by the prompt mismatch, the quantitative CI comparisons are not established. The score also depends on the arbitrary choice of n=2 layers, and no robustness check is provided.","section":"§3.5 and §4.2 (CI score)"}],"minor_comments":[{"comment":"'An Unified' should be 'A Unified', and 'Introdcution' is a typo for 'Introduction'.","section":"Title and Section 1"},{"comment":"The header 'L AYER' contains an unwanted space; it should read 'LAYER'.","section":"Table 2"},{"comment":"The text refers to 'Llama3.2-8B', but the model list and Table 2 include Llama3.2-3B; the model designation should be consistent.","section":"§4.3"},{"comment":"The phrase 'as a controlled variable for comparison' should be 'as a control condition'.","section":"§4.2"},{"comment":"Several reference entries have incomplete URL/venue information, and the Langley (2000) entry is not cited in the body of the paper.","section":"References"}],"recommendation":"reject","confidential_remarks":"The central empirical finding is invalidated by the prompt mismatch between the atomic and context layers, and the baseline inconsistencies in §4.3 make the mitigation results unverifiable. The dataset may be salvageable, but correcting the design requires re-running the entire evaluation with matched prompts and reconciling the reported numbers. I would not invite a revision of the current manuscript; a corrected study could be submitted as new work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The layered-context idea is genuinely new: nobody else wraps atomic factuality questions in nested narrative contexts with a metric like CI, and the dataset (3,356 questions, 19 categories, human validation) is real work. The CoT mitigation result, if it survived matching, would be genuinely useful for practitioners deploying small models. So the artifact deserves credit.\n\nThe problem is that the central claim—SLLMs collapse on context-wrapped hallucination detection—is not supported as stated. The atomic prompt asks \"Is the statement wrong? yes/no,\" while both context prompts ask \"Is Mike correct? yes/no\" and add \"Think step by step.\" That flips the polarity convention (yes=wrong vs. yes=correct), changes the object of judgment (statement vs. agent's belief), and adds an instruction that is known to change outputs. Any of those could produce the near-zero accuracies; the paper gives the reader no way to separate context effect from prompt effect. This is not a minor nuisance. The paper's own §3.4 says the discrimination process \"remains consistent\" across layers, but the appendix contradicts it, and the Limitation section never mentions the mismatch.\n\nThere are also internal arithmetic problems. Table 2 reports Llama3.2-3B layer-1 accuracy 0.9%, but Section 4.3 uses 23.92% as that same baseline. Llama3.1-8B is 0.15% in Table 2 and 0.03% in the text. The CI score for Llama3.2-3B is reported as 78.99, but plugging the table's accuracies into their own formula gives about 98.25. No error bars, no significance tests. These are the kinds of inconsistencies that make a referee suspect the near-zero values are fragile measurements, not stable findings.\n\nOn citations: the related-work coverage is adequate and there is no obvious self-citation padding. The GitHub link exists but there is no commit hash or release version, so reproducibility is currently unverifiable.\n\nAll that said, this is a fixable paper. Matched prompts, corrected numbers, and error analysis would turn it into a usable benchmark. As it stands, the main conclusion is unsupported, but the artifact is real and the question is relevant. If I were an editor, I'd send it to review rather than desk reject, with a clear request for major revision. The authors need to redo the experiments with a clean comparison or reframe the claims to what the current design actually shows.","headline":"The benchmark idea is worth a look, but the headline accuracy drop is confounded by a systematic prompt change across layers, and the numbers don't reconcile.","tokens_in":11857,"tokens_out":2704,"would_cite":false,"duration_ms":27493,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OnionEval is a layered benchmark showing that small language models detect atomic fact hallucinations accurately but collapse when the same facts are placed in narrative context; chain-of-thought prompting largely restores the loss.","keywords":["fact-conflicting hallucination","small language models","context influence score","layered evaluation benchmark","chain-of-thought prompting","hallucination mitigation","LLM evaluation","atomic facts"],"falsifier":"Run a matched-polarity control: present context-layer items with the atomic prompt's format ('Judge whether the statement I give you is wrong. If it is wrong, reply yes') while keeping the same story context, and present atomic items with the context prompt's polarity ('Answer yes if Mike is correct'). If small-model accuracy stays high under matched polarity, the claimed context effect is an artifact of the prompt change; if it still collapses, the context-attribution claim survives.","tokens_in":10753,"feed_emoji":"🧅","tokens_out":8495,"duration_ms":73886,"temperature":0.7,"pith_summary":"OnionEval is a layered benchmark for measuring fact-conflicting hallucination in small language models (SLLMs), the 1–10 billion parameter tier of LLMs. The paper claims that SLLMs detect false atomic statements almost as well as large models, but their accuracy collapses when the same statement is embedded in a narrative context, dropping from values like 98.49% to 0.15% in one tested model. To quantify this, the paper introduces the Context Influence (CI) score, which combines the atomic error rate with the error increase added by each context layer. The paper also reports that a simple Chain-of-Thought prompt largely repairs the collapse, whereas few-shot prompting and retrieval-augmented generation do not. If the claim holds, small models cannot be trusted for fact-checking in contextual settings without added reasoning guidance, which matters for edge and mobile deployments.","feed_headline":"Small LLMs collapse on fact checks once context is added","feed_subtitle":"OnionEval's layered tests show ~98% atomic accuracy drop to near 0% in narrative context; CoT recovers most of it.","key_machinery":"The central object is OnionEval, a dataset of 3,356 yes/no questions built from 515 entities across 19 categories. Atomic facts are extracted from a knowledge graph, and GPT-4 with human cross-validation generates hallucinated counterfactual versions of each fact. Each atomic hallucination is then wrapped into two progressively richer narrative layers: a book-reading scenario, and that same scenario relocated to a coffee shop. The load-bearing metric is the Context Influence score, defined as $CI = \\rho_h + (\\Delta_1 + \\Delta_2)/n$, where $\\rho_h$ is the model's hallucination rate on atomic facts and $\\Delta_i$ is the drop in hallucination rate from the atomic layer to layer $i$; this score is intended to measure how much context alone degrades fact-conflicting hallucination detection. The binary accuracy metric also counts 'unmatched' answers that do not yield a clean yes/no.","core_discovery":"The paper's central claim is that small language models have significantly lower performance on fact-conflicting hallucinations once context is present, and that they are markedly more susceptible to context-wrapping effects than larger models. On isolated atomic facts, small models perform comparably to large ones—Llama3.2-3B reaches 90.71% accuracy and Llama3.1-8B reaches 98.49%, close to or above larger siblings—but after one layer of narrative context the same models fall to near zero (0.9% and 0.15% respectively), while the 70B and 405B models stay near 89–91%. The paper interprets this as evidence that SLLMs excel at factual analysis but lack robust context reasoning. It further claims that chain-of-thought prompting recovers most of the lost accuracy (e.g., Llama3.1-8B from 0.03% to 69.45% on first-layer detection), supporting the reasoning-deficit interpretation over a knowledge-deficit one, since RAG does not help.","pith_inferences":["If the prompt-polarity mismatch between layers is confirmed, the near-zero layer-1 scores may overstate the effect; a matched-polarity rerun would tell whether context alone, or the flipped yes/no convention, drives the collapse.","The extremely low context-layer accuracy suggests small models may be following the instruction to judge a character's belief rather than the statement's truth; separating belief-attribution from factuality could change the benchmark's interpretation.","The CI score, being asymmetrically high for small models, could be used as a diagnostic for when pruning or distillation sacrifices context reasoning, even when atomic fact accuracy is preserved.","Because recent models within a family show much lower CI, release time and training data may confound the SLLM-vs-LLM comparison; a controlled comparison across checkpoints of the same architecture would sharpen the claim."],"forward_implications":["Small language models cannot be treated as reliable fact-checkers in any application where claims appear inside narrative or conversational context, unless a reasoning prompt is added.","Chain-of-thought prompting is a low-cost, effective mitigation for context-induced hallucination in SLLMs, and few-shot prompting helps only at larger sizes within the small tier.","RAG does not fix context-induced hallucination in small models, indicating the failure is about reasoning guidance rather than missing parametric knowledge.","The CI score gives a single number for how much context a given model can withstand, and the framework can be extended to additional context layers for longer-input scenarios.","Within each model family, newer or distilled small models show better atomic accuracy but still high context susceptibility, so training-time choices do not automatically close the context gap."],"supporting_citations":[{"why":"HaluEval results motivate the paper's opening question by showing SLLMs near large models on QA but far below on General; the benchmark's accuracy comparisons start from this observation.","marker":"(Li et al., 2023)"},{"why":"FactCHD results show smaller models doing well on comparison data but poorly on set-operation data, used to frame the claim that SLLM performance varies by context-demanding task.","marker":"(Chen et al., 2024)"},{"why":"Defines atomic facts and the decomposition method adopted in OnionEval's atomic fact extraction.","marker":"(Min et al., 2023)"},{"why":"Provides the hallucination taxonomy used to classify fact-conflicting hallucination and the survey framing for the benchmark methodology.","marker":"(Huang et al., 2024)"},{"why":"Documents that context can influence and induce hallucination in summarization, the direct precedent for treating context as a variable in hallucination evaluation.","marker":"(Flemings et al., 2024)"},{"why":"Chain-of-thought prompting is the mitigation strategy the paper tests and finds effective; its success is used to argue the small-model deficit is a reasoning deficit.","marker":"(Wei et al., 2023)"},{"why":"Few-shot prompting is the baseline mitigation tested in Section 4.3; its poor scaling in the 3B model informs the conclusion that few-shot is hint-based rather than reasoning-guided.","marker":"(Brown et al., 2020)"},{"why":"RAG framework description; the paper's finding that RAG does not help is interpreted against this background as evidence against a knowledge-deficit explanation.","marker":"(Gao et al., 2024)"}],"fun_headline_variants":["Context kills small LLM fact checks: atomic fine, narrative falls to ~0%","Small models ace atomic facts but tank with context, CoT rescues","OnionEval: small LLMs lose 98% accuracy when facts gain context","Why small LLMs flunk fact checks in context: reasoning, not knowledge","Chain-of-thought revives small LLMs from context-induced hallucination"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The atomic and context layers are assumed to test the same detection skill, but their prompts differ in answer polarity and in whether the model judges a statement or Mike's belief; if that prompt change, rather than context per se, causes the collapse, the paper's central conclusion fails.","fun_headline_variants_meta":{"raw":{"variants":["Context kills small LLM fact checks: atomic fine, narrative falls to ~0%","Small models ace atomic facts but tank with context, CoT rescues","OnionEval: small LLMs lose 98% accuracy when facts gain context","Why small LLMs flunk fact checks in context: reasoning, not knowledge","Chain-of-thought revives small LLMs from context-induced hallucination"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000673,"raw_usage":{"total_tokens":3071,"prompt_tokens":956,"completion_tokens":2115,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":2012}},"tokens_in":572,"tokens_out":2115,"duration_ms":14183,"temperature":1.0,"reasoning_tokens":2012,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:34:09.115852+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a matched-polarity control: present context-layer items with the atomic prompt's format ('Judge whether the statement I give you is wrong. If it is wrong, reply yes') while keeping the same story context, and present atomic items with the context prompt's polarity ('Answer yes if Mike is correct'). If small-model accuracy stays high under matched polarity, the claimed context effect is an artifact of the prompt change; if it still collapses, the context-attribution claim survives.","supporting_citations":[],"review_version":1}