Pith. sign in

REVIEW 5 major objections 5 minor 38 references

CondAmbigQA: A Benchmark and Dataset for Conditional Ambiguous Question Answering

T0 review · 5 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Apparent LLM hallucinations often stem from ambiguous queries, and making the hidden interpretation explicit improves QA accuracy by 11.75%.

desk verdict A genuinely useful dataset and annotation framework undermined by a circular evaluation loop and internally inconsistent headline numbers. read the letter →

arxiv 2502.01523 v2 pith:PUPIMXXK submitted 2025-02-03 cs.CL

Zongxi Li , Yang Li , Haoran Xie , S. Joe Qin This is my paper · ORCID
classification cs.CL
keywords ambiguousquestionansweringconditionsretrieval-augmentedgenerationhallucinationLLMevaluationbenchmarkdatasetdisambiguationcitationaccuracy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that when a user asks an ambiguous question, an LLM's answer is often not wrong—the model simply picked one of several equally valid interpretations without saying which one. It introduces "conditions," explicit contextual constraints that pin down one valid interpretation, and builds CondAmbigQA, a 2,000-query benchmark in which each query comes with retrieved encyclopedia fragments and condition–answer–citation triples. Across several open and proprietary models, having the model identify conditions before answering improves answer accuracy by 11.75% over standard retrieval-augmented generation, and supplying the gold conditions adds another 7.15%. The claim is that apparent hallucinations in QA are often expectation mismatches caused by query ambiguity, and that explicit condition reasoning gives models and users a shared frame for resolving them.

What carries the argument

The machinery is the condition, defined as a set of contextual constraints that must be satisfied for an answer to be correct within a particular scope. Conditions are generated by retrieving encyclopedia fragments for a query, having an LLM draft condition-answer pairs from those fragments, and having human annotators refine and validate them. Each condition is tied to a ground-truth answer and to the specific fragments that support it, so the triple (condition, answer, citations) does the work of separating "wrong answer" from "right answer under a different interpretation." The evaluation then scores condition identification, answer correctness, and citation precision, with a combined score that penalizes models for generating too many or too few condition-answer pairs.

What would settle it

Take a random sample of CondAmbigQA queries, give independent annotators only the query and the retrieved fragments (no model-drafted conditions), and ask them to list every plausible interpretation and answer. If a substantial share of gold condition-answer pairs omit interpretations those annotators identify, or if model rankings change when scored against the independent annotations, the claim that explicit conditions resolve inherent ambiguity is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that much of what is called hallucination in question answering is really the model answering a different, but co-valid, interpretation of the same query. The paper formalizes the missing context as conditions and shows that when a model states the condition it is answering under—either by discovering it from retrieved passages or by being given it—both answer accuracy and citation accuracy improve consistently across model sizes. The evidence includes a condition-quality/answer-quality correlation (Pearson 0.598, Spearman 0.637, p < 0.001) and a closed-book ablation in which condition reasoning raises answer scores by an average of 135% over direct answering. The benchmark's design makes this visible by pairing every query with multiple condition-answer-citation triples instead of forcing one answer.

Load-bearing premise

The load-bearing premise is that the condition-answer triples produced by the retrieval-based annotation process and validated by a small human team faithfully and completely capture the reasonable interpretations of each query; if those gold conditions are incomplete or biased toward the drafting model's reading, the measured gains show agreement with that view rather than genuine ambiguity resolution.

Editorial extensions

If this is right

  • QA systems that first identify and state a condition before answering should show higher factual accuracy and better source attribution than standard RAG systems, across both large and small models.
  • Supplying the correct condition helps even models that fail to discover it themselves, which means improving condition identification is a concrete target for retrieval and reasoning pipelines.
  • The condition–answer–citation structure gives an evaluation that distinguishes ambiguity resolution from factual correctness, so benchmark scores can separate a model's world knowledge from its ability to handle ambiguous queries.
  • Because the condition-based approach transferred to an existing long-form QA benchmark with about a 10% gain, the method is not limited to the new dataset.
  • Closed-book ablation results suggest that condition reasoning contributes beyond simple retrieval, since models with self-generated conditions substantially outperform direct answering even without retrieved passages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the central claim holds, user-facing assistants could usefully ask a clarifying question or display the condition they are assuming before answering, turning silent expectation mismatches into explicit choices; the paper does not test interactive clarification.
  • A natural extension is to probe interpretations that do not appear in the retrieved fragments, since the benchmark's conditions are bounded by what the encyclopedia passages contain; an independent-annotation study could reveal how much ambiguity the retrieval step misses.
  • The benchmark's condition-answer triples could be used as training supervision for smaller models, and if that training narrows the gap between API and local models, it would strengthen the paper's claim that condition reasoning, not raw scale, is the active ingredient.
  • The citation score could double as a groundedness check: a fluent answer with low citation precision signals that the model is not actually using the retrieved evidence, which may explain some failures that look like hallucination.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces CondAmbigQA, a benchmark and dataset of 2,000 ambiguous question-answering instances derived from ALCE-ASQA/AmbigNQ, each annotated with a set of condition-answer-citation triples where the conditions represent explicit contextual constraints that resolve query ambiguity. The annotation process uses GPT-4o to draft initial triples and a six-person human team to validate and calibrate them. The paper proposes G-Eval-based metrics (Condition, Answer, Citation, and Combined scores), evaluates seven (or eight) models under three settings — closed-book, standard RAG, and condition-based RAG with either self-generated or ground-truth conditions — and reports consistent improvements from condition-aware answering, plus a closed-book ablation and an external generalization experiment on ALCE-ASQA. The central claim is that incorporating explicit conditions before answering improves response quality over standard RAG and that apparent hallucinations often stem from query ambiguity rather than model failure.

Significance. If the claims hold, CondAmbigQA would be a useful resource for studying ambiguity-driven hallucination, and the condition-based framing could inform future QA system design. The dataset itself is released under CC BY-SA 4.0, which is a concrete contribution, and the paper reports a detailed annotation workflow with cost and time estimates. The experimental direction — that conditioning on explicit interpretations improves answer quality — is plausible and broadly consistent across the models tested. However, the magnitude and even the interpretation of the reported improvements are currently not verifiable because several load-bearing evaluation components are underspecified or circular: the headline improvement percentages are untraceable to the tables, the Combined Score formula is not defined, the LLM-as-judge metric is validated on only 20 examples, and the ground-truth conditions are generated by the same model that is the top performer and that also defines the reference used by the judge. These issues affect the core quantitative claims rather than just the presentation.

major comments (5)
  1. [Abstract and Sections 5.5–5.6] The abstract's headline numbers — an 11.75% improvement from condition reasoning and an additional 7.15% gain when conditions are explicitly provided — do not appear in any table or equation. Section 5.5 reports an average absolute gain of 0.20 on Answer Score, and Section 5.6 (Table 3) reports relative improvements ranging from +93% to +400%, none of which matches 11.75% or 7.15%. In addition, the Limitations section states GPT-4o's combined score as 0.701 and Qwen2.5 as 0.469, while Table 2 lists 0.662 and 0.360 respectively. The central quantitative claims are therefore not reproducible from the manuscript as written. Please provide exact formulas, a dedicated results table, and reconcile these inconsistencies.
  2. [Section 4.1, Combined Score] The Combined Score is used throughout the results (Table 2, Figure 2, Limitations) and drives the model rankings, but it is never defined mathematically. The text states only that it is a weighted average of Condition, Answer, and Citation scores adjusted by penalties for overgeneration, undergeneration, and single-answer output; no weights, penalty values, or normalization steps are given. Without an exact specification, the combined-score comparisons cannot be reproduced or interpreted.
  3. [Section 4.1 and Appendix E] The entire evaluation relies on G-Eval as an LLM judge, yet its agreement with human judgment is validated on only 20 randomly sampled examples (Table 9). With n=20, the reported Pearson correlations (0.88 and 0.83) have wide confidence intervals, and the Answer Quality Spearman correlation of 0.68 is considerably weaker. Furthermore, the human rubric uses holistic 10-point scores for condition quality and answer quality, which may not align with the specific step-by-step criteria in Appendix D; this mapping is not described. This is insufficient evidence for the reliability of the metric on 2,000 instances across eight models.
  4. [Sections 3.3, 4.1, and 5.1] The evaluation loop is circular: GPT-4o drafted the ground-truth condition-answer-citation triples (Section 3.3), the primary metric is G-Eval as implemented in DeepEval with those same triples as the reference (Section 4.1, Appendix D), and GPT-4o is the top-scoring model (Section 5.1). The measured improvements from condition guidance may therefore reflect how well models imitate GPT-4o's annotation style and output format rather than how well they resolve genuine ambiguity. Human calibration reduces but does not remove this concern. Please re-evaluate with (a) human-written conditions for a subset of queries, (b) a judge model that is not GPT-based and is blind to model identity, and (c) a format-controlled analysis.
  5. [Section 5.5 and Appendix D] The comparison between condition-based RAG and standard RAG is not controlled. The standard RAG prompt (Appendix D) explicitly instructs the model only to answer based on the fragments and cites up to five sources, with no request to state conditions or separate answers by condition. The evaluation criteria, however, require comparing extracted conditions against expected conditions and 'heavily penalise omission of critical details in the condition.' Thus the standard-RAG baseline is penalized for missing a required output structure, which likely inflates the gap. A controlled baseline should use the same output format (e.g., an empty or fixed condition block) or a format-invariant metric.
minor comments (5)
  1. [Section 5.8 and Abstract] The paper uses inconsistent percentage conventions. Section 5.8 describes a change from 0.374 to 0.471 as a 'substantial gain of 10%', but that is an absolute percentage-point increase; the relative gain is approximately 26%. Similarly, the abstract's 11.75% and 7.15% should state whether they are relative or absolute improvements.
  2. [Equation (1)] Citation Score as defined is precision-only: it divides by the number of model citations. A model that outputs a single correct citation receives a perfect score of 1.0, and the formula is undefined when the model outputs zero citations. Please add a recall term or F1-style combination and specify the zero-citation convention.
  3. [Section 3.3] The reported Cohen's κ of approximately 0.72 is not accompanied by a description of what is being measured for agreement (which annotators, how many items, and what codings). Please specify the inter-annotator agreement protocol.
  4. [Section 3.2 and Figure 1] The retrieval setup is underspecified for reproducibility. The text mentions a 'Wikipedia FAISS index', but does not state which retriever (e.g., DPR, BM25, sentence-level embeddings), chunk size, or number of retrieved fragments per query is used. Since the benchmark is retrieval-included, this is needed to reconstruct the dataset and to compare against ALCE-ASQA's DPR passages.
  5. [Section 4.2] The alternative setting, where models are given ground-truth conditions, is described in a single sentence. Since this setting underlies the central improvement claims, the exact prompt (e.g., the 'Modified Condition-based Prompt' in Appendix D) and output format expectations should be spelled out in the main text or at least explicitly cross-referenced.

Circularity Check

2 steps flagged · score 6.0 of 10

Evaluation loop: GPT-4o drafts the ground-truth conditions, is the top-scoring model, and G-Eval judges against those same conditions; the metric also rewards condition-separated answers by construction.

  1. fitted input called prediction [Section 3.3 (Annotation Process) and Section 4.1 (Evaluation Metrics); Table 2]
    "GPT-4o processes each query using predefined dataset-construction prompts to draft initial condition-answer pairs. ... The G-Eval framework evaluates whether the model has accurately identified and clearly articulated all relevant conditions."

    The ground-truth conditions and answers used as the G-Eval reference are drafted by GPT-4o, and GPT-4o is also the best-scoring evaluated model (Table 2: Condition Score 0.552 vs. next-best 0.302). G-Eval is an LLM-as-judge that compares model outputs against those GPT-4o-derived references. Thus GPT-4o's high scores, and the condition-aware gap over other models, partly measure how closely a model reproduces GPT-4o's own annotation choices rather than independent ambiguity resolution. Human validation weakens but does not sever the loop, since the humans primarily review and calibrate GPT-4o's proposals.

  2. self definitional [Section 4.1 (Combined Score) and Limitations; Appendix D (Standard RAG Prompt)]
    "Our evaluation penalises unsupported/incoherent generations and rewards condition-separated answers, but explicit adversarial-evidence detection is out of scope."

    The evaluation rubric explicitly rewards condition-separated answers, while the standard RAG baseline prompt (Appendix D) instructs the model only to 'Answer the question based solely on the provided fragments. Cite up to FIVE sources' — it never asks for conditions. The Combined Score aggregates Condition Score, penalizes single-answer outputs, and adjusts for the number of condition-answer pairs. Therefore the reported improvement of condition-aware RAG over standard RAG is partly guaranteed by the evaluation definition: the baseline is not instructed to produce the structure that the metric rewards, and the Condition Score penalizes omission of conditions by construction.

full rationale

The central benchmark result is partially circular, but not wholly so. Ground-truth condition-answer-citation triples are drafted by GPT-4o and then human-validated by six NLP researchers; 60% of GPT-4o drafts required modification or calibration, so human input provides some independent check. The external ALCE-ASQA generalization experiment also offers a modest independent anchor, and the correlation between condition quality and answer performance (Pearson 0.598, Spearman 0.637) is a real computed relationship. However, the primary evaluation loop remains: GPT-4o authors the reference annotations, GPT-4o is the top-scoring model, and the G-Eval judge is an LLM comparing outputs to those same GPT-4o-derived references. The metric further rewards condition-separated answers while the baseline is not prompted to produce them, making part of the improvement definitional. No load-bearing self-citation chain or imported uniqueness theorem appears. Internal inconsistencies (Limitations quotes GPT-4o combined score 0.701 vs. Table 2's 0.662; abstract headline gains of 11.75% and 7.15% do not appear in the result tables) further undermine the exact quantitative claims, but these are correctness risks rather than circularity. Overall: one or more 'predictions' reduce partly by construction, so the score is 6.

Assumptions & free parameters 1 free parameters · 3 assumptions · 1 invented entities

The central claim rests on the fidelity of the GPT-4o-with-human-check annotation process, the reliability of an LLM-based evaluation metric, and the completeness of Wikipedia retrieval. The only free parameter that materially affects results is the undocumented combined-score calibration.

free parameters (1)
  • Combined Score calibration weights and penalties = not specified
    Section 4.1 defines the Combined Score as a weighted average with penalties for over-/under-generation, but the exact weights and penalty magnitudes are never given, so rankings depend on undocumented choices.
assumptions (3)
  • domain assumption Retrieved Wikipedia fragments contain sufficient and correct information to identify all meaningful interpretations of each ambiguous query.
    Section 3.3 derives conditions from retrieval results; if retrieval misses an interpretation, the dataset is incomplete by construction.
  • domain assumption G-Eval scores from an LLM correlate with human judgment sufficiently well to evaluate all 2,000 instances.
    Appendix E validates G-Eval on only 20 examples; with Pearson 0.88 and Spearman 0.83, the metric is treated as reliable despite the very small sample.
  • domain assumption Human validation of GPT-4o-generated condition-answer pairs removes model-specific bias.
    Section 3.3 describes human review and calibration, but with six annotators and Cohen's kappa about 0.72, the independence from GPT-4o is partial.
invented entities (1)
  • condition
    purpose: Explicit contextual constraint that delimits a valid answer to an ambiguous query, used as the core unit of annotation and evaluation.
    The concept is defined by the authors and instantiated only in their dataset. No external, falsifiable prediction is made that would validate conditions as a natural kind beyond this benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CondAmbigQA: A Benchmark and Dataset for Conditional Ambiguous Question Answering." pith.science (2026). https://pith.science/paper/PUPIMXXK

@misc{pith2026250201523,
  author       = {Pith},
  title        = {Pith review of: CondAmbigQA: A Benchmark and Dataset for Conditional Ambiguous Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PUPIMXXK}},
  note         = {Machine review of arXiv:2502.01523}
}
read the original abstract

Users often assume that large language models (LLMs) share their cognitive alignment of context and intent, leading them to omit critical information in question-answering (QA) and produce ambiguous queries. Responses based on misaligned assumptions may be perceived as hallucinations. Therefore, identifying possible implicit assumptions is crucial in QA. To address this fundamental challenge, we propose Conditional Ambiguous Question-Answering (CondAmbigQA), a benchmark comprising 2,000 ambiguous queries and condition-aware evaluation metrics. Our study pioneers "conditions" as explicit contextual constraints that resolve ambiguities in QA tasks through retrieval-based annotation, where retrieved Wikipedia fragments help identify possible interpretations for a given query and annotate answers accordingly. Experiments demonstrate that models considering conditions before answering improve answer accuracy by 11.75%, with an additional 7.15% gain when conditions are explicitly provided. These results highlight that apparent hallucinations may stem from inherent query ambiguity rather than model failure, and demonstrate the effectiveness of condition reasoning in QA, providing researchers with tools for rigorous evaluation.

Figures

Figures reproduced from arXiv: 2502.01523 by the authors.

Figure 1
Figure 1. Annotation workflow adopted in CondAmbigQA dataset construction. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Model performance on four metrics. In partic [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Comparison of score distributions across met [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Model performance in Answer Score and Citation Score, comparing answering without conditions, [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Relationship between condition and answer [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 34 canonical work pages

  1. [1]

    Read each question and annotation carefully

    work page

  2. [2]

    Analyse each question for: a) ambiguity - explain different interpretations b) multiple possible answers - provide examples

    work page

  3. [3]

    Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer

    Hallucination-free? assessing the reliability of leading ai legal research tools.arXiv preprint arXiv:2405.20362. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering am- biguous open-domain questions. InProceedings of 11 the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 5783...

    arXiv 2020

  4. [4]

    Use format: <analysis> <question_number>Number</question_number> <question_text>Text</question_text> <ambiguity_analysis>Results</ambiguity_analysis> <multiple_answers>Results</multiple_answers> </analysis>

    work page

  5. [5]

    Last Man Standing

    Corrective retrieval augmented generation. arXiv preprint arXiv:2401.15884. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report.arXiv preprint arXiv:2407.10671. Jing Yao, Xiaoyuan Yi, and Xing Xie. 2024. Clave: An adaptive framework for evaluating valu...

    arXiv 2024

  6. [8]

    Consider: question clarity, vague terms, context sufficiency, subjective elements

    work page

  7. [10]

    [INSERT QUESTION HERE]

    Compare with previous annotations Table 5: Instruction template used to analyse queries from ASQA. We use GPT-4o to identify data samples where ambiguity is truly impactful. 13 C Dataset Prompts Dataset Prompts (Part 1) Question Answering: You are tasked with providing a structured answer to a question based on the given text fragments. Your goal is to pr...

    work page

  8. [11]

    [Impact on Interpretation 1] [Based on Fragment X, Y]

    work page

Show all 38 references
  1. [12]

    [INSERT QUESTION HERE]

    [Impact on Interpretation 2] [Based on Fragment Z, A] Contextual considerations: [How different backgrounds might affect understanding] [Repeat the Ambiguity point structure for as many ambiguities as necessary] </analysis> Explain how each ambiguity leads to different valid a...

  2. [13]

    Do not speculate beyond the given information

    Condition: Summarise the context of the question strictly using the provided fragments. Do not speculate beyond the given information

  3. [14]

    Use only what is explicitly stated

    Ground-truth: Provide the exact answer to the question based on the retrieved fragments. Use only what is explicitly stated

  4. [15]

    Include the title and text of the fragments that were used

    Citations: List the relevant fragments that support your answer. Include the title and text of the fragments that were used

  5. [16]

    condition

    Reason: Explain how the answer was derived solely from the fragments, and mention why any gaps in information were left unfilled. Fragments: retrieved fragments Output format: “condition”: “<summary based on fragments>”, “ground truth”: [“<answer derived from fragments>”], “ci...

  6. [17]

    Identify up to FIVE key conditions related to the question based solely on the provided fragments

  7. [18]

    For each condition, provide a corresponding detailed answer

  8. [19]

    Cite the sources (fragment numbers) that support each condition and answer

  9. [20]

    Modified Condition-based Prompt: Question: {question} Context fragments: {Fragment 1 - {title}: {text}}

    Output the results in JSON format with the following structure. Modified Condition-based Prompt: Question: {question} Context fragments: {Fragment 1 - {title}: {text}} ... Conditions to address: Condition 1: {condition} ... IMPORTANT: Respond with ONLY the following JSON forma...

  10. [21]

    Answer the question based solely on the provided fragments

  11. [22]

    Condition Correctness

    Cite up to FIVE sources (fragment numbers) that support your answer. Evaluation Metrics - Condition Correctness: - Name: “Condition Correctness” - Criteria: “Determine whether the actual condition is factually correct based on the expected condition.” - Evaluation steps:

  12. [23]

    Check whether the facts in ’actual condition’ contradicts any facts in ’expected condition’

  13. [24]

    Heavily penalise omission of critical details in the condition

  14. [25]

    Answer Correctness

    Ensure that the condition is clear and unambiguous. Evaluation Metrics - Answer Correctness: - Name: “Answer Correctness” - Criteria: “Determine whether the actual answer is factually correct based on the expected answers.” - Evaluation steps:

  15. [26]

    Check whether the facts in ’actual answer’ contradicts any facts in ’expected answers’

  16. [27]

    Heavily penalise omission of critical details in the answer

  17. [28]

    Which is bigger Kansas City or St. Louis?

    Ensure that the answer directly addresses the question without irrelevant information. Table 8: Evaluation prompts. The models are prompted according to these instructions and their outputs are evaluated using theG-Evalfunction as implemented in theDeepEvalpackage. 16 Metric P...

  18. [29]

    Louis (2.8 million) is larger than the Kansas City metropolitan area (2.2 million)

    Metropolitan area comparison: Greater St. Louis (2.8 million) is larger than the Kansas City metropolitan area (2.2 million)

  19. [30]

    480,000 by 2017) than St

    City proper comparison: Kansas City has a larger city proper population (approx. 480,000 by 2017) than St. Louis. For the colour TV question, multiple valid per- spectives include:

    2017

  20. [31]

    Technological introduction: Color TV was officially introduced in December 1953 with the approval of the NTSC standard, with the first national broadcast on January 1, 1954

    1953

  21. [32]

    F.2 Performance Patterns and Failure Modes Our analysis reveals distinct patterns of perfor- mance as summarised in Table 10

    Widespread adoption: Color TV became widely adopted in the mid-1960s, with NBC’s 1965 transition to colour programming catalysing industry-wide changes. F.2 Performance Patterns and Failure Modes Our analysis reveals distinct patterns of perfor- mance as summarised in Table 10...

    1965

  22. [33]

    Influence of both cities in their respec- tive metropolitan areas

    Condition Misidentification: Smaller mod- els frequently generate conditions that miss the core ambiguity. For example, Gemma2’s response to the city comparison query in- cluded “Influence of both cities in their respec- tive metropolitan areas” rather than explicitly addressi...

  23. [34]

    the Kansas City metropolitan area is larger than Greater St. Louis,

    Factual Inaccuracy: Models sometimes pro- vide incorrect information. DeepSeek incor- rectly stated, “the Kansas City metropolitan area is larger than Greater St. Louis,” contra- dicting available data

  24. [35]

    When were color TVs first made available to the public in the U.S.?

    Citation Failures: Most models, particularly local ones, struggle with citation accuracy. Even when answers contain correct informa- tion, they often cite wrong fragments, reduc- ing their reliability and trustworthiness. Using balanced scoring metrics, we established performa...

  25. [36]

    the Kansas City metropolitan area is larger than Greater St. Louis,

    Factual accuracy: DeepSeek incorrectly claimed that “the Kansas City metropolitan area is larger than Greater St. Louis,” contra- dicting established facts

  26. [37]

    Historical Growth

    Condition comprehensiveness: DeepSeek failed to adequately address both interpre- tations of the city comparison question, fo- cusing on superficial aspects like “Historical Growth” rather than comprehensive size com- parisons

  27. [38]

    When comparing the size of Kansas City and St. Louis, it is important to consider the metropolitan areas... The Greater St. Louis area is generally considered larger

    Answer depth: While DeepSeek provided some accurate information (e.g., the date of the first color broadcast), its answers lacked the contextual depth and nuance found in API model responses. These findings suggest that while specialized reasoning training provides some benefi...

    2017

  28. [2023]

    Association for Computational Lin- guistics

    Who are all the stochastic parrots imitating? they should tell us! InProceedings of the 13th In- ternational Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Lin- guistics (Volume 2: Short P...

    2024 arXiv

  29. [2024]

    Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang

    Llms will always hallucinate, and we need to live with this.arXiv preprint arXiv:2409.05746. Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. Longrope: Extending llm con- text window beyond 2 million tokens.arXiv...

    2024 arXiv

  30. [2025]

    Stephanie Lin, Jacob Hilton, and Owain Evans

    OpenReview.net. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. InProceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. A...

    2022

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.