Pith. sign in

REVIEW 4 major objections 6 minor 34 references

Clinical RAG systems can pass every standard accuracy check while attributing one drug’s real trial evidence to a completely different queried drug.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 03:40 UTC pith:YAA6NTHW

load-bearing objection Real, evaluation-invisible entity-attribution failure in clinical RAG, with solid multi-method support; the 98.7% EAV recall is thinner than the headline suggests. the 4 major comments →

arxiv 2607.09349 v1 pith:YAA6NTHW submitted 2026-07-10 cs.CL cs.AIcs.LG

Deceptive Grounding: Entity Attribution Failure in Clinical Retrieval-Augmented Generation

Cedric Caruzzo , Donggeun Yoo , Tae Soo Kim This is my paper
classification cs.CL cs.AIcs.LG
keywords deceptive groundingentity attributionclinical RAGfaithfulness evaluationretrieval-augmented generationmedical LLMsentity-attribution verificationhallucination detection
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper shows that retrieval-augmented generation can produce answers that look perfect under every usual automated test—no hallucinations, high faithfulness, real citations—yet still assign another drug’s clinical evidence to the drug the user actually asked about. The authors name this failure deceptive grounding and show it is invisible to existing evaluation frameworks because every claim is factually sourced; only the entity link is wrong. Across thirteen models under controlled adversarial retrieval, rates range from 8% to 87%, and medical or biomedical fine-tuning makes the problem worse rather than better. A production clinical system measured 7.8% overall, rising to 13.6% for recently approved drugs. They identify a two-stage mechanism and introduce entity-attribution verification, which catches the failure at high precision and recall, a check no current framework performs.

Core claim

A clinical RAG response can pass hallucination detection, faithfulness scoring, and citation verification while presenting drug Y’s real clinical evidence as evidence about the queried drug X. This deceptive grounding is evaluation-invisible by design. Under peak adversarial retrieval, rates span 8–87% across thirteen models (medical and biomedical models up to 86.7%); production measurement finds 7.8% overall and 13.6% for recently approved drugs. Entity-attribution verification detects it at 97.0% precision and 98.7% DG recall on an IPW-adjusted human gold standard.

What carries the argument

Deceptive grounding (DG): faithful relay of retrieved evidence attributed to the wrong entity. The carrying mechanism is a two-stage permission gate—Stage 1 opens via shared disease context or document framing; Stage 2 routes to entity-attribution failure when completing clinical details (trial names, NCT numbers, outcomes) are present in the wrong-entity documents, or to confabulation when they are removed. Completing-information ablation eliminates entity-attribution failure entirely while total incorrect responses remain high.

Load-bearing premise

The full-scale rates and the 98.7% recall claim rest on an automated judge whose calibration was checked on a modest human sample after the same model also wrote the synthetic documents used in the benchmark.

What would settle it

Re-adjudicate a large random sample of the full factorial responses with independent human experts; if the discontinuous jump at completing information, the drop of entity-attribution failure to 0% under ablation, or the production 7.8% rate disappear under that re-labeling, the central claims fail.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Clinical RAG benchmarks must add entity-attribution verification; faithfulness and citation metrics are structurally blind to this failure.
  • Prioritizing entity-specific retrieval for the queried drug is the highest-leverage single intervention, suppressing DG to at most 6.4% in the controlled conditions.
  • Medical and biomedical fine-tuning amplify rather than reduce risk, so training must enforce entity-identity constraints before evidence synthesis.
  • Recently approved drugs carry elevated risk (13.6%) because entity-specific retrieval is sparsest.
  • A complete detector needs both entity-attribution verification (for DG) and an NER-based check (for pure entity substitution).

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same silent misattribution pattern is likely wherever RAG retrieves near-neighbor entities in a dense space—companies, statutes, chemical compounds, or historical figures—not only drugs.
  • Regulatory checklists for AI medical devices that omit entity-attribution accuracy leave a safety gap already measurable in production systems.
  • Instruction-based anchoring helps models whose Stage 1 is driven by parametric priors but does little for models that open Stage 1 via document framing, pointing to architecture-specific fixes.
  • The large gap between adversarial and naturalistic rates implies the practical monitoring target is retrieval pipelines that accidentally surface highly completing documents about the wrong entity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper defines deceptive grounding (DG): a clinical RAG failure in which a model faithfully relays real evidence about alternate entity Y as evidence about queried entity X, thereby passing standard hallucination, faithfulness, and citation checks. Using a 2D factorial retrieval benchmark over 264 drug–disease–alternate triples and 13 models, the authors report peak adversarial DG rates of 8–87% (medical/biomedical models up to 86.7%), a two-stage mechanism supported by a completing-information ablation (entity-attribution failure 67%→0% with failures shifting to confabulation), label-substitution and noticing experiments, and entity-attribution verification (EAV) at 97.0% precision / 98.7% DG recall on an IPW-adjusted human gold standard (n=88). Production measurement on 740 drug–disease pairs finds 7.8% overall DG (13.6% for recently approved drugs).

Significance. If the core results hold, the paper identifies a clinically material, evaluation-invisible failure mode that existing RAG safety frameworks are structurally unable to catch. Strengths include a clean formal definition and three-tier taxonomy; a controlled factorial design with a load-bearing single-variable CIT ablation; multi-model scale (13 models) with profile taxonomy; human adjudication that independently confirms directional findings R1–R4; and a pre-registered production measurement with negative controls. The work is directly actionable for clinical RAG evaluation (add entity-attribution checks) and retrieval design (prioritize entity-specific evidence). These contributions are substantial for clinical NLP and RAG safety even if absolute detection metrics require tempering.

major comments (4)
  1. [§5.4 / Abstract / Contribution 3] §5.4, Abstract, and Contribution 3: The headline EAV claim of 98.7% DG/EAF recall rests on an IPW-adjusted human gold standard with neff≈18 EAF-positive cases (n=88 adjudicated; CI [82.4, 100.0]). That interval is compatible with materially lower true recall and does not securely distinguish a near-perfect detector from a useful but incomplete one. Precision (97.0% [87.7, 99.0]) and 0.0% FPR on clean controls are better supported. Either expand independent human adjudication of EAF-positive cases to a sample that can underwrite a tight recall claim, or reframe Contribution 3 / the abstract to lead with precision, specificity, and complementarity to NER-based ES detection rather than near-perfect recall.
  2. [§4 / Appendix B / Tables 1–2] §4 and Appendix B: Kimi-K2.5 both generates the synthetic Cy documents that embed CITs and serves as the primary full-scale judge. Human adjudication and a second-model judge break circularity for directional conclusions R1–R4, but full-scale absolute DG rates in Tables 1–2 remain Kimi-judged. The dual role should be treated as a calibration risk for absolute rates: report full-scale rates under the second judge (or a larger human-adjudicated subsample stratified by condition) as a sensitivity analysis, and state clearly in the main text which claims are judge-invariant versus Kimi-dependent.
  3. [Table 2 / §5.2 / Abstract] Table 2 and §5.2: Absolute peak DG% for non-L1 models are lower bounds because CITs were elicited from L1-16B-A3B’s pharmacological prior (Appendix C). This is acknowledged in the appendix and a table footnote, but the abstract and main-text cross-model claims (“8–87%”, “medical models reach up to 86.7%”) read as calibrated absolute rates. Promote the lower-bound framing into the abstract and the first paragraph of §5.2, and emphasize calibration-independent quantities (Cx protection ratios, gradient shapes, profile assignments, medical vs. general rank order) as the primary cross-model evidence.
  4. [§5.3 / Appendix D] §5.3 / Appendix D: The Stage-1 causal claim from activation patching is significant only for L1-16B-A3B (+13.7 pp vs. 0% random); Llama-3.1-70B is underpowered (near-floor baseline X-attribution). The two-stage gate is still well supported by the CIT ablation (Stage 2) and cross-model behavioral patterns, but the manuscript should not present pharmacological-class residual-stream patching as a general Stage-1 causal proof across architectures. Restrict the causal Stage-1 claim to L1 and treat class probing for other models as representational characterization.
minor comments (6)
  1. [§4 / Appendix B] Inter-rater κ=0.415 (Appendix B) is moderate; the complementary blind-spot analysis (Kimi ES definitional gap vs. Llama phrase-level EAF) is useful and should be summarized briefly in the main §4 measurement paragraph so readers do not treat κ alone as a red flag.
  2. [Table 3] Table 3: Make explicit in the table caption that EAV’s 0.0% ES recall is by design (no X-attributed claim to evaluate), so the “comprehensive system = EAV + NER” recommendation is not an afterthought.
  3. [§1 / Figure 1] Figure 1 and the rituximab/FOP example are clear; ensure NCT03188666 / garetosmab attribution is consistent wherever the example is reused so the pedagogical case cannot be misread as a factual error about the trial.
  4. [§5.5 / Abstract] Production §5.5: The 5.5× gap between worst-case naturalistic stratum (13.2%) and controlled absent×synthetic_Y (73.1%) is important; consider a short sentence in the abstract or conclusion so readers do not equate adversarial susceptibility with deployment prevalence.
  5. [§3 / Appendix E] Minor terminology: Tier-1 is variously called DG, EAF, and (in production pre-registration) entity_substitution; a single glossary mapping paper taxonomy to pre-registration labels would reduce confusion (Appendix E already notes the mismatch).
  6. [References] References include Gemma 4 and Kimi K2.5 with 2026 dates and some Hugging Face URLs; ensure citation completeness and stable identifiers for camera-ready.

Circularity Check

1 steps flagged

No load-bearing derivation circularity; only a partially mitigated measurement-loop risk (Kimi generates Cy docs and judges) that the paper itself flags and breaks for directional claims via human gold standard.

specific steps
  1. other [§4 Measurement; Appendix B 'Independence of Kimi document generation and judgment']
    "We address potential circularity from using Kimi-K2.5 for both document generation and judgment in Appendix B; the human gold standard confirms all directional conclusions independently of Kimi calibration. ... Kimi-K2.5 generates synthetic Cy documents (temperature=0; structured factual writing task) and judges responses (entity-attribution reasoning task)."

    Same model family writes the adversarial Cy documents (embedding CITs as Y’s evidence) and later scores whether subject-model responses misattribute that content to X. In principle this could couple document style to judge sensitivity. It is not self-definitional of DG rates (generation and judgment are different tasks; responses are from 13 other models), and human adjudication plus a second judge break the loop for R1–R4. Absolute full-scale rates still rest on Kimi alone—measurement contamination, not a prediction forced by construction.

full rationale

This is an empirical measurement paper, not a first-principles derivation. DG is defined structurally (Y-evidence presented as X, claims entailed by D), then rates, mechanism, and detection are measured—not deduced from the definition. The evaluation-invisibility claim is true by construction of that definition relative to faithfulness/hallucination/citation checks; that is the contribution (naming a blind spot), not a circular empirical result. Absolute DG rates are measurements under constructed stimuli, not fitted parameters renamed as predictions. CIT elicitation from L1’s prior is intentional stimulus design for studying that prior; ablation (67%→0% EAF when CITs removed), label-substitution, activation patching, production measurement (740 pairs), and human adjudication are independent of any self-fit. No uniqueness theorems, no self-citation load-bearing chain, no ansatz smuggled via overlapping authors. The only circularity-adjacent issue is dual use of Kimi-K2.5 for synthetic Cy document generation and EA V judgment—acknowledged in §4/Appendix B and partially broken by Llama inter-rater and n=88 human gold standard confirming R1–R4. That is measurement contamination risk, not a derivation that reduces to its inputs. Score 2 for that minor, non-load-bearing loop; central existence, mechanism, and production claims stand independently.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 4 invented entities

This is an empirical evaluation paper, not a first-principles derivation. Load-bearing content is operational definitions (DG/EAF/ES/PKC), the factorial retrieval design, judge-as-metric assumptions, and the two-stage mechanism as an explanatory model. Free parameters are design choices (CIT thresholds, temperatures, strata) rather than fitted physical constants. Invented entities are named constructs for the failure and detector, not physical objects.

free parameters (4)
  • CIT inclusion threshold (≥3/8 samples)
    Defines which trial/outcome strings count as completing information for prior_completing documents; changes the Cy tier boundary that drives R1.
  • CIT elicitation temperature (0.7) and k=8 samples on L1-16B-A3B
    Anchors the adversarial stimulus to one model’s pharmacological prior; absolute DG rates for other models become lower bounds.
  • Human gold-standard sample size and IPW design (n=88 adjudicated, neff≈55)
    Calibrates reported EAV precision/recall; EAF recall CI rests on a small effective EAF stratum.
  • Activation-patching layer/token choices (e.g., L1 layer 13)
    Selected at peak class-silhouette depth; affects the Stage-1 causal estimate magnitude.
axioms (4)
  • domain assumption Standard faithfulness/entailment metrics verify claim support by document content without requiring entity identity match between claim and document primary entity.
    Stated in §1–3 as the reason DG is evaluation-invisible; foundational to the claim that existing frameworks miss DG by design.
  • ad hoc to paper A response is deceptive grounding iff claims from Y≠X documents are presented as evidence about X while remaining factually entailed by retrieved D.
    Formal definition in §3; partitions Tier-1 DG from ES and confabulation and determines what EAV counts as FAIL.
  • domain assumption LLM-as-judge labels (with human adjudication on a stratified subsample) are adequate primary metrics for large-scale DG rates.
    §4 and Appendix B; common in LLM evaluation but load-bearing here because full matrices are Kimi-judged.
  • ad hoc to paper Completing Information Targets (trial names, NCT numbers, outcomes) mark the causal boundary between entity-attribution failure and confabulation under fixed Stage-1 conditions.
    Operationalized in the factorial design and confirmed by CIT ablation (§5.1, §5.3).
invented entities (4)
  • Deceptive grounding (DG / Tier-1 EAF) independent evidence
    purpose: Name the evaluation-invisible failure of attributing real Y evidence to queried X.
    Core construct; independent handle is behavioral measurement under controlled retrieval plus human labels, not an external physical entity.
  • Two-stage permission gate (Stage 1 context/prior; Stage 2 completing-info path) no independent evidence
    purpose: Explain cross-model gradients and the DG vs confabulation dissociation.
    Explanatory mechanism supported by ablation/patching but still a paper-introduced causal model of model behavior.
  • Entity-attribution verification (EAV) independent evidence
    purpose: Operational detector: check that supporting document’s primary entity matches the queried entity.
    Proposed missing evaluation criterion; performance reported against human gold standard in this paper.
  • Failure profiles P1–P6 no independent evidence
    purpose: Taxonomize gradient shapes across 13 models.
    Descriptive clustering of observed behaviors; not independently validated outside this benchmark.

pith-pipeline@v1.1.0-grok45 · 26354 in / 3606 out tokens · 41554 ms · 2026-07-13T03:40:57.213198+00:00 · methodology

0 comments
read the original abstract

Retrieval-augmented generation evaluation checks whether model claims are factually grounded in retrieved documents. It does not check whether retrieved evidence is attributed to the correct entity. A clinical RAG response can pass every automated check (zero hallucinations, near-perfect faithfulness, real citations) while presenting drug Y's clinical evidence as evidence about queried drug X. We term this deceptive grounding (DG): a failure invisible to faithfulness, hallucination, and citation checks because every claim is sourced from a real document, about the wrong entity. Using a controlled factorial benchmark across 13 models, we find DG rates spanning 8-87% at peak adversarial conditions. Medical and biomedical fine-tuned models reach up to 86.7%; domain specialization amplifies the failure rather than mitigating it. A controlled ablation identifies the mechanism: removing entity-specific clinical evidence from retrieved documents eliminates entity-attribution failure entirely, shifting all failures to confabulation. The two failure modes respond to the same trigger, taking different paths. Production measurement across 740 drug-disease pairs finds 7.8% overall DG in a deployed RAG system, rising to 13.6% for recently approved drugs. Entity-attribution verification (checking that cited evidence applies to the queried entity) detects DG at 97.0% precision and 98.7% DG recall (IPW-adjusted human gold standard); no existing framework implements it.

Figures

Figures reproduced from arXiv: 2607.09349 by Cedric Caruzzo, Donggeun Yoo, Tae Soo Kim.

Figure 1
Figure 1. Figure 1: The deceptive grounding failure structure. A query about drug X in disease C returns retrieved documents about Y . The model accurately relays Y ’s clinical evidence, presents it as evi￾dence about X, and passes all three standard automated checks (hallucination detection, faithfulness scoring, citation accuracy). Only entity-attribution verification reveals the failure. The rituximab/FOP example above ins… view at source ↗
Figure 2
Figure 2. Figure 2: Controlled 2D factorial benchmark design. Rows: Cx (queried-drug retrieval complete￾ness). Columns: Cy (alternate-drug document content). High-risk cells (absent × prior_completing and absent × synthetic_Y) are shaded. 264 triples × 15 conditions = 3,960 responses per schema variant [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Two-stage permission gate. Stage 1 opens when disease-context overlap activates a parametric attribution prior (most profiles) or when document framing establishes discourse context (inverted-gradient profiles). Stage 2 determines failure mode: completing information channels attribution to deceptive grounding; its absence forces confabulation. Ablation confirms: removing completing information eliminates … view at source ↗
Figure 4
Figure 4. Figure 4: Cy gradient by failure profile. DG rate (solid) and confabulation rate (dashed) across Cy conditions at absent Cx for six representative models. Thick lines show cross-model median; thin lines are individual traces. The crossover pattern—confabulation dominant at low Cy specificity, DG dominant at completing Cy conditions—is the behavioral signature of the two-stage permission gate ( [PITH_FULL_IMAGE:figu… view at source ↗
Figure 5
Figure 5. Figure 5: Cross-model DG rate comparison across all 13 models (10-tool schema). Peak DG rates [PITH_FULL_IMAGE:figures/full_fig_p019_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Complete cross-model DG rate matrices for 13 models (4-tool schema). Profile labels (P1– [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Pharmacological class organization in internal representations (class probing). Solid lines: [PITH_FULL_IMAGE:figures/full_fig_p022_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 2 canonical work pages · 2 internal anchors

  1. [1]

    Evaluation of Attribution Bias in Generator-Aware Retrieval-Augmented Large Language Models

    Amin Abolghasemi, Leif Azzopardi, Seyyed Hadi Hashemi, Maarten de Rijke, and Suzan Verberne. Evaluation of attribution bias in generator-aware retrieval-augmented large language models. In Findings of the Association for Computational Linguistics: ACL 2025 , pages 21105--21124, Vienna, Austria, 2025. Association for Computational Linguistics. doi:10.18653...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.18653/v1/2025.findings-acl.1087 2025

  2. [2]

    Self-RAG : Learning to retrieve, generate, and critique through self-reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG : Learning to retrieve, generate, and critique through self-reflection. In International Conference on Learning Representations (ICLR), 2024. arXiv:2310.11511

    Pith/arXiv arXiv 2024

  3. [3]

    Merging facts, crafting fallacies: Evaluating the contradictory nature of aggregated factual claims in long-form generations

    Cheng-Han Chiang and Hung-yi Lee. Merging facts, crafting fallacies: Evaluating the contradictory nature of aggregated factual claims in long-form generations. In Findings of the Association for Computational Linguistics: ACL 2024 , pages 2734--2751, Bangkok, Thailand, 2024. Association for Computational Linguistics. doi:10.18653/v1/2024.findings-acl.160....

    Pith/arXiv arXiv doi:10.18653/v1/2024.findings-acl.160 2024

  4. [4]

    Med42-v2 : A suite of clinical LLMs , 2024

    Cl \'e ment Christophe, Praveen K Kanithi, Tathagata Raha, Shadab Khan, and Marco AF Pimentel. Med42-v2 : A suite of clinical LLMs , 2024

    2024

  5. [5]

    The Llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Ponti, and Siva Reddy

    Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Osmar Zaiane, Mo Yu, Edoardo M. Ponti, and Siva Reddy. FaithDial : A faithful benchmark for information-seeking dialogue. Transactions of the Association for Computational Linguistics, 10: 0 1473--1490, 2022. doi:10.1162/tacl_a_00529. arXiv:2204.10757

    Pith/arXiv arXiv doi:10.1162/tacl_a_00529 2022

  7. [7]

    RAGAs : Automated evaluation of retrieval augmented generation

    Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. RAGAs : Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations (EACL), 2024. arXiv:2309.15217

    Pith/arXiv arXiv 2024

  8. [8]

    Language models are surprisingly fragile to drug names in biomedical benchmarks, 2024

    Jack Gallifant, Shan Chen, Pedro Moreira, Nikolaj Munch, Mingye Gao, Jackson Pond, Leo Anthony Celi, Hugo Aerts, Thomas Hartvigsen, and Danielle Bitterman. Language models are surprisingly fragile to drug names in biomedical benchmarks, 2024. arXiv:2406.12066

    Pith/arXiv arXiv 2024

  9. [9]

    RARR : Researching and revising what language models say, using language models

    Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. RARR : Researching and revising what language models say, using language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), pages 16477--16508, 20...

    Pith/arXiv arXiv 2023

  10. [10]

    Gemma Team . Gemma 4. https://huggingface.co/google/gemma-4-27B-it, 2026 a

    2026

  11. [11]

    Gemma Team . Gemma 4. https://huggingface.co/google/gemma-4-31B-it, 2026 b

    2026

  12. [12]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43 0 (2): 0 1--55, 2025. doi:10.1145/3703155. arXiv:2311.05232

    Pith/arXiv arXiv doi:10.1145/3703155 2025

  13. [13]

    Survey of hallucination in natural language generation

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55 0 (248), 2023. doi:10.1145/3571730. arXiv:2202.03629

    Pith/arXiv arXiv doi:10.1145/3571730 2023

  14. [14]

    WiCE : Real-world entailment for claims in wikipedia

    Ryo Kamoi, Tanya Goyal, Juan Diego Rodriguez, and Greg Durrett. WiCE : Real-world entailment for claims in wikipedia. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. arXiv:2303.01432

    Pith/arXiv arXiv 2023

  15. [15]

    Kimi K2.5 : Visual agentic intelligence

    Kimi Team . Kimi K2.5 : Visual agentic intelligence. arXiv preprint arXiv:2602.02276, 2026

    Pith/arXiv arXiv 2026

  16. [16]

    Entity-based knowledge conflicts in question answering

    Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. arXiv:2109.05052

    Pith/arXiv arXiv 2021

  17. [17]

    L1 : The first clinical language model by lunit

    Lunit. L1 : The first clinical language model by lunit. https://huggingface.co/learning-unit/L1-16B-A3B, 2026

    2026

  18. [18]

    When not to trust language models: Investigating effectiveness of parametric and non-parametric memories

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), volume 1, 2023. arXiv:2212.10511

    Pith/arXiv arXiv 2023

  19. [19]

    On faithfulness and factuality in abstractive summarization

    Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020. arXiv:2005.00661

    Pith/arXiv arXiv 2020

  20. [20]

    FActScoring : Fine-grained atomic evaluation of factual precision in long-form text generation

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScoring : Fine-grained atomic evaluation of factual precision in long-form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. arXiv:2305.14251

    Pith/arXiv arXiv 2023

  21. [21]

    Capabilities of GPT-4 on medical challenge problems, 2023

    Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of GPT-4 on medical challenge problems, 2023. arXiv:2303.13375

    Pith/arXiv arXiv 2023

  22. [22]

    gpt-oss-120b & gpt-oss-20b model card, 2025

    OpenAI . gpt-oss-120b & gpt-oss-20b model card, 2025

    2025

  23. [23]

    OpenBioLLMs : Advancing open-source large language models for healthcare and life sciences

    Ankit Pal and Malaikannan Sankarasubbu. OpenBioLLMs : Advancing open-source large language models for healthcare and life sciences. https://huggingface.co/aaditya/OpenBioLLM-Llama3-70B, 2024

    2024

  24. [24]

    Qwen3.5 : Accelerating productivity with native multimodal agents

    Qwen Team . Qwen3.5 : Accelerating productivity with native multimodal agents. https://huggingface.co/Qwen/Qwen3.5-122B-A10B, 2026

    2026

  25. [25]

    ARES : An automated evaluation framework for retrieval-augmented generation systems

    Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. ARES : An automated evaluation framework for retrieval-augmented generation systems. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 338--354, Mexico City, Mex...

    Pith/arXiv arXiv doi:10.18653/v1/2024.naacl-long.20 2024

  26. [26]

    Large language models can be easily distracted by irrelevant context

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Sch \"a rli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In Proceedings of the 40th International Conference on Machine Learning (ICML), 2023. arXiv:2302.00093

    Pith/arXiv arXiv 2023

  27. [27]

    Sara Mahdavi, et al

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, et al. Large language models encode clinical knowledge. Nature, 620: 0 172--180, 2023. doi:10.1038/s41586-023-06291-2

    doi:10.1038/s41586-023-06291-2 2023

  28. [28]

    Groundedness in Retrieval-augmented Long-form Generation: An Empirical Study

    Alessandro Stolfo. Groundedness in retrieval-augmented long-form generation: An empirical study. In Findings of the Association for Computational Linguistics: NAACL 2024 , pages 1537--1552, Mexico City, Mexico, 2024. Association for Computational Linguistics. doi:10.18653/v1/2024.findings-naacl.100. arXiv:2404.07060

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.18653/v1/2024.findings-naacl.100 2024

  29. [29]

    Food and Drug Administration

    U.S. Food and Drug Administration . Artificial intelligence/machine learning ( AI/ML )-based software as a medical device ( SaMD ) action plan. Technical Report, January 2021

    2021

  30. [30]

    Xiong, Shannon Zejiang Shen, Yoon Kim, and Monica Agrawal

    Lionel Wong, Ayman Ali, Raymond M. Xiong, Shannon Zejiang Shen, Yoon Kim, and Monica Agrawal. Position: Retrieval-augmented systems can be dangerous medical communicators. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, 2025. Position paper track. arXiv:2502.14898

    Pith/arXiv arXiv 2025

  31. [31]

    Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts

    Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. In International Conference on Learning Representations (ICLR), 2024. Spotlight. arXiv:2305.13300

    Pith/arXiv arXiv 2024

  32. [32]

    Benchmarking retrieval-augmented generation for medicine

    Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. Benchmarking retrieval-augmented generation for medicine. In Findings of the Association for Computational Linguistics: ACL 2024, 2024. arXiv:2402.13178

    Pith/arXiv arXiv 2024

  33. [33]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    Dalal, et al

    Cyril Zakka, Rohan Shad, Akash Chaurasia, Alex R. Dalal, et al. Almanac---retrieval-augmented language models for clinical medicine. NEJM AI, 1 0 (2), 2024. doi:10.1056/AIoa2300068

    doi:10.1056/aioa2300068 2024