Pith. sign in

REVIEW 3 major objections 5 minor 17 references

OnionEval: An Unified Evaluation of Fact-conflicting Hallucination for Small-Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OnionEval is a layered benchmark showing that small language models detect atomic fact hallucinations accurately but collapse when the same facts are placed in narrative context; chain-of-thought prompting largely restores the loss.

desk verdict The benchmark idea is worth a look, but the headline accuracy drop is confounded by a systematic prompt change across layers, and the numbers don't reconcile. read the letter →

arxiv 2501.12975 v1 pith:E7AY5UHF submitted 2025-01-22 cs.CL

Chongren Sun , Yuran Li , Di Wu , Benoit Boulet This is my paper · ORCID
classification cs.CL
keywords fact-conflictinghallucinationsmalllanguagemodelscontextinfluencescorelayeredevaluationbenchmarkchain-of-thoughtpromptingmitigationLLMatomicfacts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OnionEval is a layered benchmark for measuring fact-conflicting hallucination in small language models (SLLMs), the 1–10 billion parameter tier of LLMs. The paper claims that SLLMs detect false atomic statements almost as well as large models, but their accuracy collapses when the same statement is embedded in a narrative context, dropping from values like 98.49% to 0.15% in one tested model. To quantify this, the paper introduces the Context Influence (CI) score, which combines the atomic error rate with the error increase added by each context layer. The paper also reports that a simple Chain-of-Thought prompt largely repairs the collapse, whereas few-shot prompting and retrieval-augmented generation do not. If the claim holds, small models cannot be trusted for fact-checking in contextual settings without added reasoning guidance, which matters for edge and mobile deployments.

What carries the argument

The central object is OnionEval, a dataset of 3,356 yes/no questions built from 515 entities across 19 categories. Atomic facts are extracted from a knowledge graph, and GPT-4 with human cross-validation generates hallucinated counterfactual versions of each fact. Each atomic hallucination is then wrapped into two progressively richer narrative layers: a book-reading scenario, and that same scenario relocated to a coffee shop. The load-bearing metric is the Context Influence score, defined as $CI = \rho_h + (\Delta_1 + \Delta_2)/n$, where $\rho_h$ is the model's hallucination rate on atomic facts and $\Delta_i$ is the drop in hallucination rate from the atomic layer to layer $i$; this score is intended to measure how much context alone degrades fact-conflicting hallucination detection. The binary accuracy metric also counts 'unmatched' answers that do not yield a clean yes/no.

What would settle it

Run a matched-polarity control: present context-layer items with the atomic prompt's format ('Judge whether the statement I give you is wrong. If it is wrong, reply yes') while keeping the same story context, and present atomic items with the context prompt's polarity ('Answer yes if Mike is correct'). If small-model accuracy stays high under matched polarity, the claimed context effect is an artifact of the prompt change; if it still collapses, the context-attribution claim survives.

Watch

Extended reading notes

Core claim

The paper's central claim is that small language models have significantly lower performance on fact-conflicting hallucinations once context is present, and that they are markedly more susceptible to context-wrapping effects than larger models. On isolated atomic facts, small models perform comparably to large ones—Llama3.2-3B reaches 90.71% accuracy and Llama3.1-8B reaches 98.49%, close to or above larger siblings—but after one layer of narrative context the same models fall to near zero (0.9% and 0.15% respectively), while the 70B and 405B models stay near 89–91%. The paper interprets this as evidence that SLLMs excel at factual analysis but lack robust context reasoning. It further claims that chain-of-thought prompting recovers most of the lost accuracy (e.g., Llama3.1-8B from 0.03% to 69.45% on first-layer detection), supporting the reasoning-deficit interpretation over a knowledge-deficit one, since RAG does not help.

Load-bearing premise

The atomic and context layers are assumed to test the same detection skill, but their prompts differ in answer polarity and in whether the model judges a statement or Mike's belief; if that prompt change, rather than context per se, causes the collapse, the paper's central conclusion fails.

Editorial extensions

If this is right

  • Small language models cannot be treated as reliable fact-checkers in any application where claims appear inside narrative or conversational context, unless a reasoning prompt is added.
  • Chain-of-thought prompting is a low-cost, effective mitigation for context-induced hallucination in SLLMs, and few-shot prompting helps only at larger sizes within the small tier.
  • RAG does not fix context-induced hallucination in small models, indicating the failure is about reasoning guidance rather than missing parametric knowledge.
  • The CI score gives a single number for how much context a given model can withstand, and the framework can be extended to additional context layers for longer-input scenarios.
  • Within each model family, newer or distilled small models show better atomic accuracy but still high context susceptibility, so training-time choices do not automatically close the context gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the prompt-polarity mismatch between layers is confirmed, the near-zero layer-1 scores may overstate the effect; a matched-polarity rerun would tell whether context alone, or the flipped yes/no convention, drives the collapse.
  • The extremely low context-layer accuracy suggests small models may be following the instruction to judge a character's belief rather than the statement's truth; separating belief-attribution from factuality could change the benchmark's interpretation.
  • The CI score, being asymmetrically high for small models, could be used as a diagnostic for when pruning or distillation sacrifices context reasoning, even when atomic fact accuracy is preserved.
  • Because recent models within a family show much lower CI, release time and training data may confound the SLLM-vs-LLM comparison; a controlled comparison across checkpoints of the same architecture would sharpen the claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes OnionEval, a benchmark for fact-conflicting hallucination detection in small LLMs, built from atomic knowledge-graph facts that are wrapped in two progressively richer narrative contexts. It introduces a Context Influence (CI) score and reports accuracy and CI results for Llama, Gemma, and Qwen models from 3B to 72B. The authors claim that SLLMs perform comparably to large models on atomic facts but collapse when context is added, and that chain-of-thought prompting substantially repairs the deficit.

Significance. OnionEval addresses a genuine gap: few hallucination benchmarks focus on small LLMs or manipulate context level in a controlled way. The paper makes its dataset and code public, includes human cross-validation, evaluates multiple model families, and reports zero-temperature runs. However, the central comparison is confounded by a prompt mismatch between the atomic and context layers, and the reported numbers contain internal contradictions. As presented, the headline claim that context wrapping causes SLLM accuracy to collapse is not supported by the evidence.

major comments (3)
  1. [§3.4, Appendix A (Tables 4–6)] The paper's main claim in §4.2 that SLLMs' accuracy collapses under context rests on comparing the atomic layer with the context layers. The prompts are not matched: the atomic prompt asks whether a statement is wrong and maps 'yes' to wrong, whereas the context prompts ask whether Mike's opinion is correct, map 'yes' to correct, add 'Think step by step,' and change the judged object from a statement to an agent's belief. Thus the drop from 98.49% to 0.15% for Llama3.1-8B could be caused by flipped polarity, the added CoT instruction, or the reframing, rather than by contextual information. Moreover, in the displayed context examples Mike's opinion is always the true atomic fact, so the correct answer is always 'yes' and accuracy could reflect a response bias instead of contextual reasoning. This design contradicts §3.4's assertion that the binary discrimination process 'remains consistent' across layers. The central claim can only be tested with prompts that are identical in task frame, answer polarity, and reasoning instruction, with context as the sole manipulated variable.
  2. [§4.3 vs. Table 2 and Table 3] The baseline first-layer accuracy for Llama3.2-3B is 0.9% in Table 2 but 23.92% in §4.3, and for Llama3.1-8B it is 0.15% in Table 2 but 0.03% in §4.3. Because the mitigation results are presented as improvements over these baselines (e.g., 'from 0.03% to 69.45%'), the inconsistencies make the reported gains impossible to verify. The authors need to report one consistent set of baseline numbers and re-derive all mitigation comparisons from it.
  3. [§3.5 and §4.2 (CI score)] The CI scores are reported as point estimates without confidence intervals, error bars, or significance tests, yet the text contrasts scores such as 99.66 and 10.56 as evidence of a systematic size effect. With only one model per size in each family, and with the underlying accuracies already affected by the prompt mismatch, the quantitative CI comparisons are not established. The score also depends on the arbitrary choice of n=2 layers, and no robustness check is provided.
minor comments (5)
  1. [Title and Section 1] 'An Unified' should be 'A Unified', and 'Introdcution' is a typo for 'Introduction'.
  2. [Table 2] The header 'L AYER' contains an unwanted space; it should read 'LAYER'.
  3. [§4.3] The text refers to 'Llama3.2-8B', but the model list and Table 2 include Llama3.2-3B; the model designation should be consistent.
  4. [§4.2] The phrase 'as a controlled variable for comparison' should be 'as a control condition'.
  5. [References] Several reference entries have incomplete URL/venue information, and the Langley (2000) entry is not cited in the body of the paper.

Circularity Check

1 steps flagged · score 2.0 of 10

No load-bearing circularity: the central accuracy-drop result is independently measured, but the CI 'context-influence' metric algebraically reduces to the context-layer error rate, so the metric's influence claim is a definitional restatement.

  1. self definitional [Section 3.5 (Metrics Explanation), definitions of rho_h, Delta_1, Delta_2, and CI]
    "The CI score is defined as the sum of the atomic hallucination rate (ρh), which represent model’s hallucination rate across the whole benchmark on atomic facts, and the sum of Δ1 and Δ2, which are the differences for the first and second layers to compare with ρh, divided by n, number of layers. ... ρh = 1 − ρatomic (atomic hallucination rate) ... Δ1 = ρatomic − ρfirst-layer (first layer difference) ... Δ2 = ρatomic − ρsecond-layer (second layer difference) ... CI = ρh + (Σ_{i=1}^n Δ_i)/n (composite score, where n = 2)"

    Substituting the three definitions into CI cancels ρatomic identically: CI = (1−ρatomic) + [(ρatomic−ρfirst-layer)+(ρatomic−ρsecond-layer)]/2 = 1 − (ρfirst-layer+ρsecond-layer)/2. Thus the proposed 'context-influence' metric is, by construction, exactly the average hallucination rate of the two context layers; the atomic baseline that the definition claims to compare against drops out. The later statement that 'CI scores ... indicate that small language models (SLLMs) are significantly more susceptible to context wrapping effects' therefore restates the measured context-layer error rate under a new name rather than quantifying influence relative to atomic facts. This is a definitional reduction, even though the raw accuracy drop is measured independently.

full rationale

The main empirical claim—that SLLMs drop from roughly 90% atomic accuracy to near-zero on context-wrapped layers—is a direct accuracy measurement, not a fitted parameter, and it does not depend on the CI formula. There is also no load-bearing self-citation, no imported uniqueness theorem, and no ansatz smuggled in via citation. The one circularity-adjacent step is the CI metric: because the atomic term cancels in the algebra, CI is exactly the context-layer error rate, so the 'context-influence' label is a renaming of the input measurement rather than an independently derived influence estimate. Separately, the atomic and context prompts in Appendix A are not matched—the polarity of 'yes' flips, the task changes from judging a statement to judging Mike's belief, and 'Think step by step' is added—and Section 4.3 reports different Layer-1 baselines (0.03 vs 0.15 for Llama3.1-8B; 23.92 vs 0.9 for Llama3.2-3B). These are serious threats to the validity of the central comparison, but they are experimental confounds rather than circular derivation.

Assumptions & free parameters 2 free parameters · 5 assumptions · 2 invented entities

The central claims do not rest on fitted parameters, but the benchmark design involves several hand-chosen components (layer count, equal layer weights, prompt templates) and hidden assumptions about task validity and API fidelity. These choices determine the headline result: the near-zero context-layer accuracies could be an artifact of prompt polarity and instruction mismatch rather than a measured property of small models.

free parameters (2)
  • number of context layers (n) = 2
    Chosen because 'two layers of context wrapping already bring significance to the result' (Section 3.5), i.e., selected post hoc based on the observed outcome; the CI score depends linearly on n.
  • CI equal weighting per layer (1/n) = 1/2
    The CI score averages delta-1 and delta-2 with equal weights; no justification is given for equal weighting over alternative schemes.
assumptions (5)
  • domain assumption Atomic facts from Google Knowledge Graph are ground truth for the hallucination questions.
    The benchmark's correctness relies on these facts being true and on GPT-4-generated adversarial statements being false; invoked in Sections 3.2 and 3.3.
  • domain assumption The accuracy drop between atomic and context layers is caused solely by context wrapping, not by prompt task or instruction changes.
    This is the load-bearing premise behind the 'context reasoning' conclusion; the prompts in Appendix A show that task framing, answer polarity, and instructions differ across layers.
  • domain assumption Models served via the Fireworks API faithfully represent the open-weights models (Llama, Gemma, Qwen).
    All evaluations used third-party API endpoints; the paper does not verify parity with local checkpoints or quantization effects (Section 4.1).
  • domain assumption The yes/no mapping is unambiguous for the binary classification task.
    The atomic prompt maps 'yes' to 'the statement is wrong', while the context prompt maps 'yes' to 'Mike's opinion is correct'; a systematic polarity inversion would produce near-zero accuracy without any reasoning failure (Section 3.5, Appendix A).
  • domain assumption Unmatched answers are rare and can be counted uniformly as errors.
    The accuracy formula counts both wrong and unmatched answers as errors, but no inter-annotator agreement or handling of partially correct responses is specified (Section 3.5).
invented entities (2)
  • Context Influence (CI) score
    purpose: Quantify the influence of context on fact-conflicting hallucination detection.
    A new composite metric defined as rho_h plus the average of delta-1 and delta-2; it has no external validation and its interpretation depends entirely on the benchmark's prompt design (Section 3.5).
  • OnionEval benchmark independent evidence
    purpose: Evaluate SLLMs' fact-conflicting hallucination across layered context levels.
    A new dataset of 3,356 QA pairs with a public GitHub repository, so other researchers can run it; however, its validity as a benchmark rests on the paper's own GPT-4 and human annotation pipeline and has not been independently audited (Section 3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of OnionEval: An Unified Evaluation of Fact-conflicting Hallucination for Small-Large Language Models." pith.science (2026). https://pith.science/paper/E7AY5UHF

@misc{pith2026250112975,
  author       = {Pith},
  title        = {Pith review of: OnionEval: An Unified Evaluation of Fact-conflicting Hallucination for Small-Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E7AY5UHF}},
  note         = {Machine review of arXiv:2501.12975}
}
read the original abstract

Large Language Models (LLMs) are highly capable but require significant computational resources for both training and inference. Within the LLM family, smaller models (those with fewer than 10 billion parameters) also perform well across various tasks. However, these smaller models share similar limitations to their larger counterparts, including the tendency to hallucinate. Despite the existence of many benchmarks to evaluate hallucination in LLMs, few have specifically focused on small LLMs (SLLMs). Additionally, SLLMs show widely varying performance across different benchmarks. In this paper, we introduce OnionEval, a multi-layer structured framework with a specific metric called the context-influence score (CI), designed to effectively assess the fact-conflicting hallucination tendencies of small LLMs across different contextual levels. Our experimental results reveal a key feature of SLLMs: they excel in factual analysis but face challenges with context reasoning. Further investigation shows that a simple Chain-of-Thought strategy can significantly reduce these limitations, improving the practical usefulness of SLLMs in real-world applications.

Figures

Figures reproduced from arXiv: 2501.12975 by the authors.

Figure 1
Figure 1. Halueval benchmark, SLLMs (in red rectangle) have sim￾ilar performance on QA dataset, but largely lower performance on General dataset. Larger models’ evaluation are more stable. This bring one question by us: why SLLMs perform vastly different on different benchmarks? Sun et al., 2024). However, these models require substan￾tial computational resources for both training and inference, limiting their accessibility a… view at source ↗
Figure 2
Figure 2. The construction pipeline of OnionEval. From left to right, they are: entity sampling, atmoic fact extraction, and hallucination generation (KG) is a structured representation of real-world entities and their relationships, typically modeled as a directed graph. It consists of three main components: a set of entities (E), a set of relations (R), and a set of triplets (T). Each triplet is represented as (h, r, t), wh… view at source ↗
Figure 4
Figure 4. Two levels of context wrapping. The blue word is the entity. The red words represent atomic hallucination words. And, the green words are atomic facts. Each input is used as part of context of next level’s input 4. Experiment 4.1. Experiment setting Evaluation Models. We evaluated three model families: the Llama 3 (instructed) family, which includes Llama￾3 8B, Llama-3.1 8B, Llama-3.1 70B, and Llama-3.2 3B; the Gemm… view at source ↗
Figures from the paper (1 more)
Figure 3
Figure 3. Figure 3: Context Influence score for models in three families: Llama, Gemma, Qwen. Each subfigure displays the respective Context influence score for different models in the family. From left to right in each figure, the model size increases. However, Context influence score is…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 15 linked inside Pith

  1. [4]

    Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Liu, T., Chang, B., Sun, X., Li, L., and Sui, Z

    URL https://arxiv.org/abs/2309.11495. Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Liu, T., Chang, B., Sun, X., Li, L., and Sui, Z. A survey on in-context learning,

    arXiv

  2. [5]

    Flemings, J., Zhang, W., Jiang, B., Takhirov, Z., and Annavaram, M

    URL https://arxiv.org/abs/2301.00234. Flemings, J., Zhang, W., Jiang, B., Takhirov, Z., and Annavaram, M. Characterizing context influence and hallucination in summarization,

    arXiv

  3. [6]

    Gao, Y ., Xiong, Y ., Gao, X., Jia, K., Pan, J., Bi, Y ., Dai, Y ., Sun, J., Wang, M., and Wang, H

    URL https: //arxiv.org/abs/2410.03026. Gao, Y ., Xiong, Y ., Gao, X., Jia, K., Pan, J., Bi, Y ., Dai, Y ., Sun, J., Wang, M., and Wang, H. Retrieval-augmented generation for large language models: A survey,

    arXiv

  4. [7]

    URL https://arxiv.org/abs/2312.10997. Hadi, M. U., Qureshi, R., Shah, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Wu, J., Mirjalili, S., et al. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints,

    arXiv

  5. [8]

    doi: 10.1145/3703155

    ISSN 1046-8188. doi: 10.1145/3703155. URL https: //doi.org/10.1145/3703155. Just Accepted. Langley, P. Crafting papers on machine learning. In Langley, P. (ed.),Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207–1216, Stan- ford, CA,

    doi:10.1145/3703155

  6. [10]

    Truthfulqa: Measuring how models mimic human falsehoods

    Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958,

    arXiv

  7. [11]

    Luo, J., Li, T., Wu, D., Jenkin, M., Liu, S., and Dudek, G

    URL https:// arxiv.org/abs/2409.15790. Luo, J., Li, T., Wu, D., Jenkin, M., Liu, S., and Dudek, G. Hallucination detection and hallucination mitigation: An investigation. arXiv preprint arXiv:2401.08358,

    arXiv

  8. [12]

    W., Iyyer, M., Zettlemoyer, L., and Hajishirzi, H

    Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-t., Koh, P. W., Iyyer, M., Zettlemoyer, L., and Hajishirzi, H. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251,

    arXiv

Show all 17 references
  1. [13]

    Nguyen, C

    URL https:// arxiv.org/abs/2307.06908. Nguyen, C. V ., Shen, X., Aponte, R., Xia, Y ., Basu, S., Hu, Z., Chen, J., Parmar, M., Kunapuli, S., Barrow, J., Wu, J., Singh, A., Wang, Y ., Gu, J., Dernoncourt, F., Ahmed, N. K., Lipka, N., Zhang, R., Chen, X., Yu, T., Kim, S., Deilam...

    arXiv

  2. [14]

    Rawte, V ., Tonmoy, S

    URL https://arxiv.org/ abs/2410.20011. Rawte, V ., Tonmoy, S. M. T. I., Rajbangshi, K., Nag, S., Chadha, A., Sheth, A. P., and Das, A. Factoid: Fac- tual entailment for hallucination detection,

    arXiv

  3. [15]

    Sun, C., Singh, A., and Samuel, S

    URL https://arxiv.org/abs/2403.19113. Sun, C., Singh, A., and Samuel, S. Evaluation of control modalities in highly automated vehicles: A virtual reality simulationbased study. IEEE Transactions on Intelli- gent V ehicles, pp. 1–11,

    arXiv

  4. [16]

    doi: 10.1109/TIV .2024. 3454608. Tonmoy, S., Zaman, S., Jain, V ., Rani, A., Rawte, V ., Chadha, A., and Das, A. A comprehensive survey of hal- lucination mitigation techniques in large language models. arXiv preprint arXiv:2401.01313,

    2024 arXiv

  5. [17]

    Zhang, Y ., Li, Y ., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y ., Chen, Y ., et al

    URL https://arxiv.org/abs/ 2201.11903. Zhang, Y ., Li, Y ., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y ., Chen, Y ., et al. Siren’s song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219,

    arXiv

  6. [2000]

    Li, J., Cheng, X., Zhao, W

    Morgan Kaufmann. Li, J., Cheng, X., Zhao, W. X., Nie, J.-Y ., and Wen, J.- R. Halueval: A large-scale hallucination evaluation benchmark for large language models. arXiv preprint arXiv:2305.11747,

    arXiv

  7. [2020]

    Chen, X., Song, D., Gui, H., Wang, C., Zhang, N., Jiang, Y ., Huang, F., Lv, C., Zhang, D., and Chen, H

    URL https:// arxiv.org/abs/2005.14165. Chen, X., Song, D., Gui, H., Wang, C., Zhang, N., Jiang, Y ., Huang, F., Lv, C., Zhang, D., and Chen, H. Factchd: Benchmarking fact-conflicting hallucination detection,

    2005 arXiv

  8. [2023]

    2310.03368

    doi: 10.48550/arXiv. 2310.03368. URL https://doi.org/10.48550/ arXiv.2310.03368. Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., and Weston, J. Chain-of-verification reduces hallucination in large language models,

    doi arXiv

  9. [2024]

    Cheng, Q., Sun, T., Zhang, W., Wang, S., Liu, X., Zhang, M., He, J., Huang, M., Yin, Z., Chen, K., and Qiu, X

    URL https://arxiv.org/abs/ 2310.12086. Cheng, Q., Sun, T., Zhang, W., Wang, S., Liu, X., Zhang, M., He, J., Huang, M., Yin, Z., Chen, K., and Qiu, X. Evaluating hallucinations in chinese large language mod- els. CoRR, abs/2310.03368,

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.