Pith. sign in

REVIEW 2 major objections 1 minor 1 cited by

Auditing Stealth Sycophancy in Mental-Health Dialogue: Structured Clinical-State Diagnostics and Clean Matched Benchmarks

T0 review · 2 major / 1 minor · reviewed 2026-07-01 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Dynamic Emotional Signature Graphs detect implicit sycophancy in mental-health dialogues by scoring clinical-state transitions on a leakage-audited benchmark.

desk verdict The leakage-audited benchmark is a concrete step forward, but the 0.0488 F1 gain rests on unvalidated LLM state extraction that may carry the same bias. read the letter →

arxiv 2605.03472 v2 pith:IKVP6CRT submitted 2026-05-05 cs.CL cs.AI

Tianze Han , Beining Xu , Hanbo Zhang , Yongming Lu This is my paper · ORCID
classification cs.CLcs.AI
keywords implicitsycophancymental-healthdialogueclinical-statediagnosticscognitivedistortionharmful-riskdetectionmatchedbenchmarkstatetransitionsDynamicEmotionalSignatureGraphs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Mental-health dialogue responses can appear empathetic while implicitly reinforcing patterns such as catastrophizing, avoidance, or hopeless prediction. The paper builds a diagnostic benchmark from peer support, counseling, and crisis sources, then creates a leakage-audited clean matched set of 500 contexts and 1,500 response windows. It proposes Dynamic Emotional Signature Graphs, which extract semantic, affective, and cognitive-distortion states via LLM and score the direction of clinical change induced by each response. On this benchmark, the DESG-StateRisk variant improves macro-F1 by 0.0488 over the strongest non-DESG baseline and leads in harmful-risk detection. The work shows that reliable detection of this hidden failure mode needs explicit clinical-state modeling plus controls for leakage and shortcuts.

What carries the argument

Dynamic Emotional Signature Graphs (DESG), a structured offline audit framework that extracts semantic, affective, and cognitive-distortion states via LLM and scores clinical direction through state transitions rather than free-form judgment.

What would settle it

A test in which the state-extraction step is shown to reinforce the same harmful patterns or in which DESG-StateRisk loses its performance edge on an independently constructed clean matched benchmark.

Watch

Extended reading notes

Core claim

The paper establishes that DESG, by separating LLM-based state extraction from scoring and evaluating the direction of semantic, affective, and cognitive-distortion state transitions, outperforms metadata, surface-style, lexical, embedding, and rubric-LLM baselines; on the leakage-audited clean matched benchmark it improves macro-F1 by 0.0488 and achieves the best harmful-risk detection result.

Load-bearing premise

The LLM-based extraction of semantic, affective, and cognitive-distortion states produces reliable clinical direction signals that are not themselves biased by the sycophancy patterns being detected.

Editorial extensions

If this is right

  • Evaluating implicit sycophancy requires explicit clinical-state modeling together with leakage checks, shortcut controls, and competitive baselines.
  • Surface-style, lexical, embedding, and rubric-LLM baselines are outperformed when direction of clinical-state change is scored directly.
  • A clean single-response matched benchmark built from everyday, counseling-style, and crisis sources enables more reliable harmful-risk detection.
  • Three representative dialogue sources provide coverage across peer support, emotional support, and crisis-oriented interactions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Mental-health chatbot developers could embed similar state-transition audits into evaluation pipelines to reduce unintended reinforcement of distortions.
  • The state-transition approach may extend to detecting subtle reinforcing biases in other dialogue settings such as educational or advisory conversations.
  • Replacing the LLM extractor with domain-specific clinical models or human annotators could increase reliability while preserving the graph structure.
  • Widespread use would shift safety standards for therapeutic AI from empathy-focused metrics toward measurable clinical direction.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper claims that implicit sycophancy (responses that appear empathetic while reinforcing harmful cognitive patterns) in mental-health dialogue can be audited via a new leakage-audited clean matched benchmark (500 contexts, 1,500 response windows from three dialogue sources) and the DESG framework, which separates LLM-based extraction of semantic/affective/cognitive-distortion states from scoring of clinical-state transitions; on this benchmark DESG-StateRisk yields a 0.0488 macro-F1 gain over the strongest non-DESG baseline and the best harmful-risk result.

Significance. If the extraction step is shown to be reliable, the work supplies a structured, direction-aware alternative to surface-style or free-form LLM evaluators and demonstrates the value of explicit leakage controls and matched benchmarks; the emphasis on clinical-state transitions rather than metadata or lexical cues is a constructive direction for safety auditing.

major comments (2)
  1. [Abstract / DESG framework description] Abstract / DESG framework: the reported 0.0488 macro-F1 improvement and best harmful-risk result rest on LLM extraction of semantic, affective, and cognitive-distortion states, yet no extraction accuracy, clinician inter-annotator agreement, or bias audit of the extractor itself is provided; this is load-bearing because any sycophancy bias in the extractor LLM would systematically distort the state-transition signals that DESG scores.
  2. [Benchmark construction] Benchmark section (implied by abstract): the construction of the 1,500 matched response windows and the post-hoc cleaning procedure are described only at high level, with no error bars, statistical significance tests, or sensitivity analysis for the 0.0488 macro-F1 delta; without these the modest lift cannot be distinguished from sampling or cleaning artifacts.
minor comments (1)
  1. [Abstract] The three source datasets are referred to only generically; explicit names and citations should be supplied when first introduced.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their thoughtful and constructive comments, which highlight important aspects of validation and statistical rigor. We address each major comment below and commit to revisions that strengthen the manuscript without altering its core claims.

read point-by-point responses

  1. Referee: [Abstract / DESG framework description] Abstract / DESG framework: the reported 0.0488 macro-F1 improvement and best harmful-risk result rest on LLM extraction of semantic, affective, and cognitive-distortion states, yet no extraction accuracy, clinician inter-annotator agreement, or bias audit of the extractor itself is provided; this is load-bearing because any sycophancy bias in the extractor LLM would systematically distort the state-transition signals that DESG scores.

    Authors: We agree that the reliability of the LLM-based state extraction step is foundational and that its absence represents a gap. The current manuscript emphasizes the overall DESG framework and benchmark results but does not report extractor-level validation metrics. We will revise the paper to add a dedicated subsection detailing: (i) accuracy of semantic, affective, and cognitive-distortion state extraction against clinician-annotated gold labels on a held-out subset; (ii) inter-annotator agreement (e.g., Cohen's kappa) among multiple clinicians; and (iii) a targeted bias audit for sycophantic tendencies in the extractor outputs. These additions will be placed in the Methods section and will include the annotation protocol and sample size. revision: yes

  2. Referee: [Benchmark construction] Benchmark section (implied by abstract): the construction of the 1,500 matched response windows and the post-hoc cleaning procedure are described only at high level, with no error bars, statistical significance tests, or sensitivity analysis for the 0.0488 macro-F1 delta; without these the modest lift cannot be distinguished from sampling or cleaning artifacts.

    Authors: We acknowledge that the benchmark construction and the statistical characterization of the performance delta are presented at a summary level. We will expand the relevant section to include: a more granular description of the matching procedure across the three dialogue sources and the post-hoc cleaning steps (including explicit criteria and any automated filters); bootstrapped or cross-run error bars around the macro-F1 scores; results of statistical significance tests comparing DESG-StateRisk to the strongest baseline; and sensitivity analyses that vary key parameters such as context window size, cleaning thresholds, and source proportions. These changes will clarify that the reported 0.0488 improvement is not an artifact of sampling or cleaning choices. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical results on constructed benchmark are independent of fitted parameters or self-citations

full rationale

The paper presents an empirical study introducing a new diagnostic benchmark and the DESG framework, which separates LLM-based state extraction from downstream scoring of semantic/affective/cognitive-distortion transitions. Reported gains (0.0488 macro-F1) are measured performance on the leakage-audited matched benchmark against baselines; no equations, fitted parameters, or self-citation chains are shown that would make the central claims equivalent to their inputs by construction. The derivation chain is therefore self-contained.

Assumptions & free parameters 0 free parameters · 1 assumptions · 1 invented entities

The central claim rests on the assumption that LLM-extracted clinical states are sufficiently unbiased to serve as ground truth for direction scoring; the paper introduces the DESG framework as a new structured method without citing prior formalization of its state-transition rules.

assumptions (1)
  • domain assumption LLM-based extraction of semantic, affective, and cognitive-distortion states produces consistent clinical direction signals across response windows
    Invoked in the description of DESG separating state extraction from final scoring
invented entities (1)
  • Dynamic Emotional Signature Graphs (DESG)
    purpose: Structured offline audit that models clinical-state transitions rather than free-form judgment
    New framework proposed in the paper; no independent evidence outside this work is provided in the abstract

how reviews work

0 comments
Cite this review

Pith. "Pith review of Auditing Stealth Sycophancy in Mental-Health Dialogue: Structured Clinical-State Diagnostics and Clean Matched Benchmarks." pith.science (2026). https://pith.science/paper/IKVP6CRT

@misc{pith2026260503472,
  author       = {Pith},
  title        = {Pith review of: Auditing Stealth Sycophancy in Mental-Health Dialogue: Structured Clinical-State Diagnostics and Clean Matched Benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IKVP6CRT}},
  note         = {Machine review of arXiv:2605.03472}
}
read the original abstract

Mental-health dialogue models are increasingly evaluated by AI-based evaluators, yet these evaluators often treat surface empathy, supportiveness, or fluency as evidence of safety. In this paper, we study a hidden failure mode that we call implicit sycophancy: a response may appear empathetic while implicitly reinforcing catastrophizing, avoidance, hopeless prediction, or CBT-style labeling. To examine this problem, we introduce a diagnostic benchmark for implicit-sycophancy detection, built from three representative mental-health dialogue sources covering everyday peer support, counseling-style emotional support, and crisis-oriented interaction, and further construct a leakage-audited clean single-response matched benchmark with 500 contexts and 1,500 matched response windows. We then propose Dynamic Emotional Signature Graphs (DESG), a structured offline audit framework that separates LLM-based state extraction from final scoring and evaluates clinical direction through semantic, affective, and cognitive-distortion state transitions rather than free-form LLM judgment. Unlike metadata, surface-style, lexical, embedding, and rubric-LLM baselines, DESG scores the direction of clinical-state change induced by a response; on the leakage-audited clean matched benchmark, DESG-StateRisk improves over the strongest non-DESG baseline by 0.0488 macro-F1 and achieves the best harmful-risk detection result. These results suggest that evaluating implicit sycophancy requires explicit clinical-state modeling together with leakage checks, shortcut controls, and competitive baselines.

Figures

Figures reproduced from arXiv: 2605.03472 by the authors.

Figure 1
Figure 1. Evaluation blind spot for stealth sycophancy, where clinically harmful direc￾tionality can appear as supportive surface language. 1 Introduction Conversational AI systems are increasingly being deployed in mental-health sup￾port scenarios, raising significant concerns about whether current evaluation methods can reliably identify harmful model behavior[2,14,32]. In these settings, surface-level empathy, fluent langu… view at source ↗
Figure 2
Figure 2. DESG pipeline and validity controls, separating state extraction, clinical-state representation, directed graph scoring, and benchmark auditing. 3.1 State Decoupling into a 1548-D Clinical Space DESG begins from the observation that surface language alone is not sufficient for psychological dialogue evaluation. Responses with similar semantic content may lead to different clinical trajectories, especially when warm … view at source ↗
Figure 2
Figure 2. DESG workflow and validity controls, separating state extraction, clinical-state representation, directed graph scoring, and benchmark auditing. rubric-style judgment[17,12,14,5]. This is insufficient for implicit-sycophancy de￾tection because two dialogues may share similar empathetic language while mov￾ing in opposite clinical directions. For example, movement from despair to tenta￾tive agency and movement from di… view at source ↗
Figures from the paper (7 more)
Figure 3
Figure 3. Figure 3: Representative harmful windows missed by the direct LLM judge and official evaluator baselines. Representative failure cases explain why the direct LLM judge and official evaluator baselines miss clinically unsafe directionality, as visualized in
Figure 4
Figure 4. Figure 4: Exploratory t-SNE views of pure-text and affective-manifold representations
Figure 5
Figure 5. Figure 5: Harmful-window miss patterns for direct and external evaluator baselines. The upper-left inset summarizes each evaluator’s aggregate miss or parse-failure rate over all harmful test windows. Rows in the matrix are representative harmful cases, columns are evaluators, g…
Figure 6
Figure 6. Figure 6: Representative state trajectories behind the qualitative disagreement cases. Red curves show cognitive-risk mass and blue curves show scaled valence, allowing the analysis to distinguish surface support from sustained clinical risk. C.3 Parameter Sensitivity Visualizat…
Figure 7
Figure 7. Figure 7: Parameter-sensitivity ranges used as a mechanism-claim gate. Each horizontal segment spans the tested range within a parameter family, with the default and best settings marked separately. C.4 Mechanism Sanity Control Visualization The sanity-control visualization in
Figure 8
Figure 8. Figure 8: Mechanism sanity-control deltas relative to the default setting. Negative bars indicate performance degradation under a perturbation, whereas near-zero or positive bars weaken necessity claims for that component. C.5 Deep Branch and Ensemble Visualization The deep-bran…
Figure 9
Figure 9. Figure 9: Deep-branch and ensemble robustness diagnostics. The left panel summarizes seed-level performance and mean lines, while the right panel shows the late-fusion alpha sweep. D Ethics Statement This work is limited to offline evaluation and red-team auditing of psychologic…

Review history (3 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond Feeling Better: Capability-Sustaining Emotional Dialogue as a Longitudinal Research Paradigm

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Proposes a 'capability-sustaining' longitudinal paradigm for emotional dialogue, backed by an audit showing current systems focus on relief and never measure long-term capability.

Reference graph

Works this paper leans on

38 extracted references · 38 canonical work pages · cited by 1 Pith paper

  1. [1]

    Guilford Press (1979)

    Beck, A.T., Rush, A.J., Shaw, B.F., Emery, G.: Cognitive Therapy of Depression. Guilford Press (1979)

    work page 1979

  2. [2]

    Behaviour Research and Therapy70, 32–37 (2015)

    Braun, J.D., Strunk, D.R., Sasso, K.E., Cooper, A.A.: Therapist use of socratic questioning predicts session-to-session symptom change in cognitive therapy for depression. Behaviour Research and Therapy70, 32–37 (2015). https://doi.org/ 10.1016/j.brat.2015.05.004

    work page doi:10.1016/j.brat.2015.05.004 2015

  3. [3]

    In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing

    Chen, G.H., Chen, S., Liu, Z., Jiang, F., Wang, B.: Humans or LLMs as the judge? a study on judgement bias. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pp. 8301–8327. Association for Compu- tational Linguistics (2024). https://doi.org/10.18653/v1/2024.emnlp-main.474

    work page doi:10.18653/v1/2024.emnlp-main.474 2024

  4. [4]

    In: Findings of the Association for Computational Linguistics: ACL 2024

    Chen, Y., Yan, S., Liu, S., Li, Y., Xiao, Y.: EmotionQueen: A benchmark for evaluating empathy of large language models. In: Findings of the Association for Computational Linguistics: ACL 2024. pp. 2149–2176. Association for Computa- tional Linguistics (2024). https://doi.org/10.18653/v1/2024.findings-acl.128

    work page doi:10.18653/v1/2024.findings-acl.128 2024

  5. [5]

    In: Proceedings of the 63rd Annual MeetingoftheAssociationforComputationalLinguistics(Volume1:LongPapers)

    Chiang, C.H., Lee, H.y., Lukasik, M.: TRACT: Regression-aware fine-tuning meets chain-of-thought reasoning for LLM-as-a-judge. In: Proceedings of the 63rd Annual MeetingoftheAssociationforComputationalLinguistics(Volume1:LongPapers). pp. 2934–2952. Association for Computational Linguistics (2025). https://doi.org/ 10.18653/v1/2025.acl-long.147

    work page doi:10.18653/v1/2025.acl-long.147 2025

  6. [6]

    In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    D’Souza, J., Babaei Giglou, H., Münch, Q.: YESciEval: Robust LLM-as-a-judge for scientific question answering. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 13749–13783. Association for Computational Linguistics (2025). https://doi.org/ 10.18653/v1/2025.acl-long.675

    work page doi:10.18653/v1/2025.acl-long.675 2025

  7. [7]

    Psychotherapy 48(1), 43–49 (2011)

    Elliott, R., Bohart, A.C., Watson, J.C., Greenberg, L.S.: Empathy. Psychotherapy 48(1), 43–49 (2011). https://doi.org/10.1037/a0022187 20 T. Han, B. Xu et al

    work page doi:10.1037/a0022187 2011

  8. [8]

    Suicide and Life-Threatening Behavior 37(3), 338–352 (2007)

    Gould, M.S., Kalafat, J., Harris Munfakh, J.L., Kleinman, M.: An evaluation of cri- sis hotline outcomes part 2: Suicidal callers. Suicide and Life-Threatening Behavior 37(3), 338–352 (2007). https://doi.org/10.1521/suli.2007.37.3.338

    work page doi:10.1521/suli.2007.37.3.338 2007

Show all 38 references
  1. [9]

    The Innovation7(6), 101253 (2026)

    Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., Wang, Y., Guo, J.: A survey on LLM-as-a-judge. The Innovation7(6), 101253 (2026). https://doi.org/10.1016/j.xinn.2025.101253

    2026 doi

  2. [10]

    Cognitive Therapy and Research36(5), 427–440 (2012)

    Hofmann, S.G., Asnaani, A., Vonk, I.J.J., Sawyer, A.T., Fang, A.: The efficacy of cognitive behavioral therapy: A review of meta-analyses. Cognitive Therapy and Research36(5), 427–440 (2012). https://doi.org/10.1007/s10608-012-9476-1

    2012 doi

  3. [11]

    JMIR mHealth and uHealth6(11), e12106 (2018)

    Inkster, B., Sarda, S., Subramanian, V.: An empathy-driven, conversational artifi- cial intelligence agent (Wysa) for digital mental well-being: Real-world data eval- uation mixed-methods study. JMIR mHealth and uHealth6(11), e12106 (2018). https://doi.org/10.2196/12106

    2018 doi

  4. [13]

    Lee, D., Hwang, Y., Kim, Y., Park, J., Jung, K.: Are LLM-judges robust to expres- sions of uncertainty? investigating the effect of epistemic markers on LLM-based evaluation. In: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for C...

    2025 doi

  5. [15]

    In: Findings of the Association for Computational Linguistics: EMNLP

    Li, A., Lu, Y., Song, N., Zhang, S., Ma, L., Lan, Z.: Understanding the therapeutic relationship between counselors and clients in online text-based counseling using LLMs. In: Findings of the Association for Computational Linguistics: EMNLP

  6. [16]

    1280–1303

    pp. 1280–1303. Association for Computational Linguistics (2024). https:// doi.org/10.18653/v1/2024.findings-emnlp.69

    2024 doi

  7. [18]

    In: The Twelfth International Conference on Learning Represen- tations (2024), https://openreview.net/forum?id=gtkFw6sZGS

    Li, J., Sun, S., Yuan, W., Fan, R.Z., Zhao, H., Liu, P.: Generative judge for evalu- ating alignment. In: The Twelfth International Conference on Learning Represen- tations (2024), https://openreview.net/forum?id=gtkFw6sZGS

    2024

  8. [19]

    arXiv preprint arXiv:2506.08584 (2025), https://arxiv.org/abs/2506.08584 Auditing Stealth Sycophancy 21

    Li, Y., Yao, J., Bunyi, J.B.S., Frank, A.C., Hwang, A.H.C., Liu, R.: CounselBench: A large-scale expert evaluation and adversarial benchmarking of large language models in mental health question answering. arXiv preprint arXiv:2506.08584 (2025), https://arxiv.org/abs/2506.0858...

    2025 arXiv

  9. [21]

    Liu, S., Zheng, C., Demasi, O., Sabour, S., Li, Y., Yu, Z., Jiang, Y., Huang, M.: Towards emotional support dialog systems. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Interna- tional Joint Conference on Natural Lang...

    2021 doi

  10. [22]

    In: Findings of the Association for Computational Linguistics: ACL 2025

    Na, H., Hua, Y., Wang, Z., Shen, T., Yu, B., Wang, L., Wang, W., Torous, J., Chen, L.: A survey of large language models in psychotherapy: Current landscape and future directions. In: Findings of the Association for Computational Linguistics: ACL 2025. pp. 7362–7376. Associati...

    2025 doi

  11. [24]

    In: Advances in Neural Information Processing Systems (2024), https://arxiv.org/abs/2404.13076

    Panickssery, A., Bowman, S.R., Feng, S.: LLM evaluators recognize and favor their own generations. In: Advances in Neural Information Processing Systems (2024), https://arxiv.org/abs/2404.13076

    2024 arXiv

  12. [25]

    In: Findings of the Association for Computational Lin- guistics: EMNLP 2024

    Park, J., Jwa, S., Meiying, R., Kim, D., Choi, S.: OffsetBias: Leveraging debiased data for tuning evaluators. In: Findings of the Association for Computational Lin- guistics: EMNLP 2024. pp. 1043–1067. Association for Computational Linguistics (2024). https://doi.org/10.18653...

    2024 doi

  13. [27]

    In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)

    Reimers, N., Gurevych, I.: Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 3...

    2019 doi

  14. [28]

    Europe’s Journal of Psychology12(3), 348–362 (2016)

    Rnic, K., Dozois, D.J.A., Martin, R.A.: Cognitive distortions, humor styles, and depression. Europe’s Journal of Psychology12(3), 348–362 (2016). https://doi. org/10.5964/ejop.v12i3.1118

    2016 doi

  15. [29]

    Journal of Consulting Psychology21(2), 95–103 (1957)

    Rogers, C.R.: The necessary and sufficient conditions of therapeutic personality change. Journal of Consulting Psychology21(2), 95–103 (1957). https://doi.org/ 10.1037/h0045357

    1957 doi

  16. [30]

    Child Development 73(6), 1830–1843 (2002)

    Rose, A.J.: Co-rumination in the friendships of girls and boys. Child Development 73(6), 1830–1843 (2002). https://doi.org/10.1111/1467-8624.00509

    2002 doi

  17. [31]

    Journal of Personality and Social Psychology39(6), 1161–1178 (1980)

    Russell, J.A.: A circumplex model of affect. Journal of Personality and Social Psychology39(6), 1161–1178 (1980). https://doi.org/10.1037/h0077714 22 T. Han, B. Xu et al

    1980 doi

  18. [32]

    In: Chinese Conference on Pattern Recognition and Computer Vision

    Shan, G., Ma, X., Bai, X., Zhu, H., Wang, T., Zhu, S., Wang, L.: Dental diagnosis from x-ray panoramic radiography images: A dataset and a hybrid framework. In: Chinese Conference on Pattern Recognition and Computer Vision. pp. 234–248 (2024). https://doi.org/10.1007/978-981-9...

    2024 doi

  19. [33]

    Shi, L., Ma, C., Liang, W., Diao, X., Ma, W., Vosoughi, S.: Judging the judges: A systematic study of position bias in LLM-as-a-judge. In: Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter o...

    2025

  20. [34]

    Hugging Face dataset (2026), https://huggingface.co/datasets/SungJoo/Cradle-Dialogue, dataset card

    SungJoo: CRADLE-Dialogue: Crisis-response dialogue dataset. Hugging Face dataset (2026), https://huggingface.co/datasets/SungJoo/Cradle-Dialogue, dataset card

    2026

  21. [35]

    The Canadian Journal of Psychiatry64(7), 456–464 (2019)

    Vaidyam, A.N., Wisniewski, H., Halamka, J.D., Kashavan, M.S., Torous, J.B.: Chatbots and conversational agents in mental health: A review of the psychiatric landscape. The Canadian Journal of Psychiatry64(7), 456–464 (2019). https:// doi.org/10.1177/0706743719828977

    2019 doi

  22. [37]

    arXiv preprint arXiv:2410.21819 (2024), https://arxiv.org/abs/2410.21819

    Wataoka, K., Takahashi, T., Ri, R.: Self-preference bias in LLM-as-a-judge. arXiv preprint arXiv:2410.21819 (2024), https://arxiv.org/abs/2410.21819

    2024 arXiv

  23. [38]

    In: Proceedings of the 2024 Conference on EmpiricalMethodsinNaturalLanguageProcessing.pp.7900–7932.Associationfor Computational Linguistics (2024)

    Watts, I., Gumma, V., Yadavalli, A., Seshadri, V., Swaminathan, M., Sitaram, S.: PARIKSHA: A large-scale investigation of human-LLM evaluator agreement on multilingual and multi-cultural data. In: Proceedings of the 2024 Conference on EmpiricalMethodsinNaturalLanguageProcessin...

    2024 doi

  24. [39]

    In: Proceedings of the 63rd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers)

    Xie, H., Chen, Y., Xing, X., Lin, J., Xu, X.: PsyDT: Using LLMs to construct the digital twin of psychological counselor with personalized counseling style for psychological counseling. In: Proceedings of the 63rd Annual Meeting of the As- sociation for Computational Linguisti...

    2025 doi

  25. [41]

    In: Findings of the Association for Computational Linguistics: ACL 2024

    Zhang, C., Li, R., Tan, M., Yang, M., Zhu, J., Yang, D., Zhao, J., Ye, G., Li, C., Hu, X.: CPsyCoun: A report-based multi-turn dialogue reconstruction and evaluation framework for Chinese psychological counseling. In: Findings of the Association for Computational Linguistics: ...

    2024 doi

  26. [42]

    Zhang, M., Yang, X., Zhang, X., Labrum, T., Chiu, J.C., Eack, S.M., Fang, F., Wang, W.Y., Chen, Z.: CBT-bench: Evaluating large language models on assisting Auditing Stealth Sycophancy 23 cognitive behavior therapy. In: Proceedings of the 2025 Conference of the Na- tions of th...

    2025 doi

  27. [43]

    In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Zhang, Q., Wang, Y., Jiang, Y., Li, L., Wu, C., Wang, Y., Jiang, X., Shang, L., Tang, R., Lyu, F., Ma, C.: Crowd comparative reasoning: Unlocking comprehensive evaluations for LLM-as-a-judge. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Lingu...

  28. [44]

    https://doi.org/10.18653/ v1/2025.acl-long.252

    Association for Computational Linguistics (2025). https://doi.org/10.18653/ v1/2025.acl-long.252

    2025

  29. [45]

    In: International Conference on Learning Rep- resentations (2020), https://openreview.net/forum?id=SkeHuCVFDr

    Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: BERTScore: Evalu- ating text generation with BERT. In: International Conference on Learning Rep- resentations (2020), https://openreview.net/forum?id=SkeHuCVFDr

    2020

  30. [46]

    In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process- ing

    Zhao, H., Li, L., Chen, S., Kong, S., Wang, J., Huang, K., Gu, T., Wang, Y., Wang, J., Dandan, L., Li, Z., Teng, Y., Xiao, Y., Wang, Y.: ESC-eval: Evalu- ating emotion support conversations in large language models. In: Proceedings of the 2024 Conference on Empirical Methods i...

    2024 doi

Pith tools

Reviewed July 1, 2026 · model on record in the stance chip above.