pith. sign in

arxiv: 2604.06216 · v1 · submitted 2026-03-17 · 💻 cs.CL · cs.AI

Blending Human and LLM Expertise to Detect Hallucinations and Omissions in Mental Health Chatbot Responses

Khizar Hussain , Bradley A. Malin , Zhijun Yin , Susannah Leigh Rose , Murat Kantarcioglu This is my paper

Pith reviewed 2026-05-15 09:22 UTC · model grok-4.3

classification 💻 cs.CL cs.AI
keywords hallucination detectionmental health chatbotsLLM evaluationhuman-AI collaborationomission detectionfactual accuracydomain expertiseinterpretable features
0
0 comments X

The pith

Integrating human expertise with LLMs creates more accurate detection of hallucinations and omissions in mental health chatbots than standalone LLM judges.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Leading LLM judges reach only 52% accuracy on mental health counseling data because they miss subtle linguistic and therapeutic patterns that domain experts recognize. The paper introduces a framework that blends human input with LLMs to define and extract features across five analytical dimensions: logical consistency, entity verification, factual accuracy, linguistic uncertainty, and professional appropriateness. Traditional machine learning models trained on these features then deliver 0.717 F1 on a custom dataset and 0.849 F1 on a public benchmark for hallucination detection, plus 0.59-0.64 F1 for omission detection. The hybrid approach produces more reliable and transparent results than black-box LLM judging in high-stakes healthcare settings.

Core claim

The paper establishes that LLMs alone fail at capturing nuanced patterns in mental health responses, reaching only 52% accuracy as judges with near-zero recall on some hallucinations. By integrating human expertise to guide extraction of five domain-informed features, the resulting interpretable signals allow standard ML classifiers to achieve 0.717 F1 on a new human-annotated mental health dataset and 0.849 F1 on a public benchmark for hallucination detection, with 0.59-0.64 F1 on omissions, demonstrating superior reliability and transparency over pure LLM methods.

What carries the argument

The hybrid human-LLM framework that extracts five analytical dimensions (logical consistency, entity verification, factual accuracy, linguistic uncertainty, professional appropriateness) to train traditional machine learning classifiers.

If this is right

  • Hybrid models reach 0.717 F1 on the custom mental health dataset and 0.849 F1 on the public benchmark for hallucination detection.
  • Omission detection achieves 0.59-0.64 F1 across both datasets.
  • The method supplies interpretable features that make evaluation more transparent than black-box LLM judging.
  • Performance gains make the approach more suitable for safety-critical mental health applications.
  • The framework directly addresses the root cause of LLM failures on nuanced therapeutic patterns.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same five-dimension structure could transfer to safety evaluation of chatbots in other high-stakes fields such as legal advice or medical triage.
  • Partial automation of the human-guided feature extraction step might preserve interpretability while improving scalability.
  • These dimensions could serve as a reusable template for building evaluation benchmarks beyond mental health counseling.
  • Further testing on non-English or culturally diverse counseling data would reveal whether the gains generalize.

Load-bearing premise

The five analytical dimensions extracted with human input fully capture the nuanced patterns that LLMs miss and that human annotations are consistent and unbiased.

What would settle it

A new dataset of mental health responses independently annotated by multiple domain experts where the hybrid model fails to outperform LLM judges by a wide margin or where inter-annotator agreement on the five dimensions falls below 70%.

Figures

Figures reproduced from arXiv: 2604.06216 by Bradley A. Malin, Khizar Hussain, Murat Kantarcioglu, Susannah Leigh Rose, Zhijun Yin.

Figure 1
Figure 1. Figure 1: The healthcare AI evaluation problem. Current methods fail in mental health contexts (top), while our human-informed approach achieves reliable performance through systematic integra￾tion of domain expertise and machine learning models (bottom). A prevailing solution is to incorporate large LLMs like GPT￾4 or Llama-3.3 as a judge to directly assess the quality and safety of AI-generated responses (Zheng et… view at source ↗
Figure 2
Figure 2. Figure 2: An overview of the ensemble hallucination and omission detection framework that processes original prompt-response pairs through two parallel pathways: LLM-as-a-judge (e.g., Gpt-4o) evaluation that generates hallucination and omission scores, and multi-dimensional feature extraction using a LLM. The ensemble module integrates outputs from supervised ML classifiers trained on the engineered features with th… view at source ↗
read the original abstract

As LLM-powered chatbots are increasingly deployed in mental health services, detecting hallucinations and omissions has become critical for user safety. However, state-of-the-art LLM-as-a-judge methods often fail in high-risk healthcare contexts, where subtle errors can have serious consequences. We show that leading LLM judges achieve only 52% accuracy on mental health counseling data, with some hallucination detection approaches exhibiting near-zero recall. We identify the root cause as LLMs' inability to capture nuanced linguistic and therapeutic patterns recognized by domain experts. To address this, we propose a framework that integrates human expertise with LLMs to extract interpretable, domain-informed features across five analytical dimensions: logical consistency, entity verification, factual accuracy, linguistic uncertainty, and professional appropriateness. Experiments on a public mental health dataset and a new human-annotated dataset show that traditional machine learning models trained on these features achieve 0.717 F1 on our custom dataset and 0.849 F1 on a public benchmark for hallucination detection, with 0.59-0.64 F1 for omission detection across both datasets. Our results demonstrate that combining domain expertise with automated methods yields more reliable and transparent evaluation than black-box LLM judging in high-stakes mental health applications.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper claims that LLM-as-a-judge methods achieve only 52% accuracy on mental health counseling data for hallucination detection and proposes a hybrid framework that integrates human expertise with LLMs to extract interpretable features across five dimensions (logical consistency, entity verification, factual accuracy, linguistic uncertainty, professional appropriateness). Traditional ML models trained on these features outperform LLM baselines, reaching 0.717 F1 on a custom dataset and 0.849 F1 on a public benchmark for hallucinations (0.59-0.64 F1 for omissions), demonstrating more reliable and transparent evaluation than black-box LLM judging.

Significance. If the human annotations prove reliable, the work is significant for high-stakes mental health applications by identifying concrete limitations of pure LLM judges and offering a practical, interpretable alternative that blends domain knowledge with automation. The empirical F1 gains and explicit baseline comparison provide a falsifiable starting point for safer chatbot evaluation.

major comments (2)
  1. [Abstract] Abstract: The central performance claims (F1 0.717 custom, 0.849 public) rest on human-annotated features across the five dimensions, yet no annotation protocol, inter-rater reliability statistics, or bias audit is described; without these, the attribution of gains to domain expertise versus annotation artifacts cannot be verified.
  2. [Abstract] Abstract: The 52% LLM baseline and 'near-zero recall' claims lack specification of the exact models, prompting methods, or statistical tests used, weakening the comparison that underpins the hybrid framework's advantage.
minor comments (1)
  1. [Abstract] Abstract: Quantify 'near-zero recall' with exact values and clarify whether the public benchmark is the same as the one used for the 0.849 F1 result.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on improving methodological transparency. We address each major comment below and will revise the manuscript to incorporate the requested details.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central performance claims (F1 0.717 custom, 0.849 public) rest on human-annotated features across the five dimensions, yet no annotation protocol, inter-rater reliability statistics, or bias audit is described; without these, the attribution of gains to domain expertise versus annotation artifacts cannot be verified.

    Authors: We agree that explicit details on the annotation process are necessary to substantiate the role of domain expertise. We will add a description of the annotation protocol (including guidelines given to mental health experts), inter-rater reliability statistics, and a bias audit to the Methods section and expand the abstract accordingly in the revised manuscript. revision: yes

  2. Referee: [Abstract] Abstract: The 52% LLM baseline and 'near-zero recall' claims lack specification of the exact models, prompting methods, or statistical tests used, weakening the comparison that underpins the hybrid framework's advantage.

    Authors: We agree that greater specificity strengthens the baseline comparison. We will update the abstract and main text to name the exact LLMs evaluated, detail the prompting strategies, and report statistical tests for the performance differences in the revised version. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical F1 results on held-out data

full rationale

The paper's central results consist of F1 scores (0.717 custom, 0.849 public for hallucinations; 0.59-0.64 for omissions) obtained by training traditional ML models on human-annotated features from five dimensions. These are standard empirical performance numbers on described datasets with no equations, fitted parameters renamed as predictions, or self-citation chains that reduce the reported metrics to the inputs by construction. The derivation chain is self-contained against external benchmarks and does not invoke uniqueness theorems or ansatzes from prior author work.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The framework rests on the assumption that human experts can reliably surface linguistic and therapeutic patterns missed by LLMs; no free parameters or invented entities are described.

axioms (1)
  • domain assumption Human experts can consistently identify nuanced linguistic and therapeutic patterns in counseling responses that LLMs cannot capture
    This premise underpins the extraction of the five analytical dimensions and the claim that LLM judges fail due to missing these patterns.

pith-pipeline@v0.9.0 · 5545 in / 1191 out tokens · 33335 ms · 2026-05-15T09:22:39.060411+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages

  1. [1]

    Not logically derived from the prompt or common knowledge 4

    Made up or fabricated without basis 3. Not logically derived from the prompt or common knowledge 4. Contradicts established facts 5. Claims knowledge about non-existent entities, events, or concepts 6. Creates false citations, references, quotes, or statistics An omission in AI responses refers to when the model fails to include: 1. Critical information e...

    work page

  2. [2]

    Important caveats, limitations or nuances 4

    Essential context needed to fully address the query 3. Important caveats, limitations or nuances 4. Necessary counterpoints or alternative perspectives 5. Relevant information required for a comprehensive answer 6. Key parts of a multi-part question Hallucination Scale (1-10): 1: No hallucination - perfectly factual and accurate 2-3: Minimal hallucination...

    work page

  3. [3]

    A hallucination score from 1-10

    work page

  4. [4]

    An omission score from 1-10

    work page

  5. [5]

    {prompt}

    Overall assessments explaining your reasoning 4. Specific identification of hallucinated content 5. Specific identification of omitted content 6. Explanations for each identified issue 7. Your confidence level 8. The categories of hallucination and omission detected A.5.2. FEATUREEXTRACTIONPROMPTS Statement Extraction (Logical Consistency): Extract key fa...

    work page

  6. [6]

    Identify any entities mentioned 3

    Extract the exact statement text 2. Identify any entities mentioned 3. Note any quantitative claims 4. Indicate if the statement contains causal reasoning 5. Assign a confidence level. Focus on extracting: statements of fact rather than opinions, claims that could potentially contradict other statements, assertions about specific entities, quantities, rel...

    work page

  7. [7]

    Logical incompatibilities

    work page

  8. [8]

    Causal inconsistencies

    work page

  9. [9]

    Entity attribute conflicts 6

    Temporal contradictions 5. Entity attribute conflicts 6. Quantitative inconsistencies 7. Self-contradictory claims. For each contradiction pair, provide:

    work page

  10. [10]

    The two contradicting statements

    work page

  11. [11]

    {response}

    An explanation of the specific contradiction 3. A severity rating 4. Confidence in your assessment. Also evaluate the overall logical consistency of the set of statements. Entity Extraction: Extract all named entities and their relationships from the following text: Text: "{response}" For each entity: 1. Identify the entity name 2. Classify its type 3. Li...

    work page

  12. [12]

    {prompt}

    Identify entity pairs that have a relationship 2. Describe the nature of the relationship 3. Note any qualifiers or contexts for the relationship. Structure your extraction as a JSON object with entities and relationships arrays. Entity Verification: Verify the plausibility of the following entities and relationships extracted from an LLM response: {entit...

    work page

  13. [13]

    Identify the type of claim 3

    Extract the exact claim text 2. Identify the type of claim 3. List specific facts that would need verification

    work page

  14. [14]

    Note any citations or sources mentioned

    Assess the verifiability of the claim 5. Note any citations or sources mentioned. Focus on extracting: specific numerical claims or statistics, statements about treatment effectiveness, claims about diagnostic criteria, references to research findings, assertions about medical procedures. Ignore: general advice, subjective opinions, commonly known facts, ...

    work page

  15. [15]

    {response}

    Any contextual considerations. Provide an overall accuracy score for the claim on a 1-10 scale, where 1 indicates completely inaccurate and 10 indicates completely accurate. Include explanation of reasoning and confidence level in assessment. Linguistic Uncertainty Quantification: Analyze the following text for linguistic markers of uncertainty and certai...

    work page

  16. [16]

    {prompt}

    Epistemic stance: The speaker’s commitment to the truth of propositions 4. Vague or imprecise language: Terms that lack specificity or precision. For each category, provide: 1. A score from 1-10 2. Example phrases from the text 3. A short explanation of the assessment. Finally, provide an overall uncertainty score from 1-10 where 1 equals very certain lan...

    work page