REVIEW
CompAIssionate Other: Identity Assumptions in LLM Responses to Distress
Human-Computer Interaction · Submitted 7 Nov 2026 · 111 references
Pith's verdict: Solid audit of a subtle harm—LLMs make unsupported identity-attributions in distress responses, but the finding is real and well-supported.
T0 · AI review
Why Pith says this
Highlight any text to comment on it.
Referee report
The paper
The paper claims
Large language models routinely attribute distress to a disclosed marginalized identity even when the writer says nothing connecting the two.
The reading
What carries the argument
The central mechanism is a counterfactual audit design: each post from Reddit SI and SH-Detection is prepended with a one-sentence identity disclosure ('I'm [identity].') while the post text itself is held constant. Responses are evaluated by an LLM judge (GPT-5-nano) on multiple dimensions, with the primary measure being 'distress attribution'—whether the response presumes the disclosed identity is a cause of the writer's distress. The authors also introduce the concept of 'compassionate othering' from medical education literature to characterize the harm of imposing identity-linked narratives within otherwise empathetic language.
Extended reading notes
Core claim
The paper's central discovery is that LLMs can construct explanatory narratives around a disclosed social identity when responding to distress, even when the writer's own account of their circumstances does not connect the two. Using counterfactual interventions where a single identity sentence is prepended to unchanged posts, the authors find that marginalized identity disclosures elevate 'distress attribution'—the model presuming identity is a cause of distress—compared with majority identity disclosures or no disclosure. This disparity persists across two datasets and six models, with the strongest effects for immigrant, trans, and Black identities. The findings hold while safety behaviors (refusal, risk recognition, crisis resource provision) remain nearly identical across disclosure conditions, revealing a dimension of harm that standard safety evaluations do not capture.
Load-bearing premise
The paper's primary measure of distress attribution relies on an LLM judge (GPT-5-nano) to decide whether a response assumes identity is a cause of distress, which can be ambiguous when models suggest plausible associations without asserting causality.
Editorial extensions
If this is right
- Evaluations of AI mental health support must include checks for unsupported identity-linked explanations, as standard safety metrics (refusal, risk recognition, crisis resources) do not capture this form of harm.
- Reducing demographic disparities in refusal or interactional risk does not guarantee equitable treatment; models may still impose identity-based interpretations that misrepresent marginalized users' experiences.
- Structured response protocols that reduce distress attribution (like the clinician-guideline prompt) also reduce identity engagement more broadly, suggesting a trade-off that requires careful calibration.
- Developers should audit for unsupported identity attribution before deployment and after model updates, and incorporate user feedback to distinguish contextually appropriate personalization from harmful generalization.
- The pattern of compassionate othering may erode trust and cause therapeutic alliance ruptures in ongoing interactions, particularly during disclosures of suicidal ideation where continued engagement is critical.
Reading between the lines
- The paper's findings suggest that current fine-tuning for safety and empathy may reduce overt toxicity but not subtle interpretive biases, which could be more common and harder to detect; post-deployment monitoring should include qualitative analysis of how models frame users' identities.
- An immediate testable extension is a controlled user study where participants from marginalized groups review model responses and rate whether they feel their experiences are accurately represented or presumptuously explained, providing evidence to ground mitigation strategies.
- The pattern might generalize to other personal information beyond social identity, such as medical conditions or life events; future work could examine whether disclosing a diagnosis or hardship similarly redirects models' explanations.
- The paper's reliance on single-turn responses leaves open whether users would correct or clarify the model's attribution in follow-up turns, which would be a natural way to distinguish harmful presumption from helpful exploration.
Original abstract
Individuals experiencing suicidal ideation (SI) or psychological distress may disclose their social identities when seeking support from LLMs. Attending to identity-related stressors can be important for providing contextually appropriate and empathetic support. However, disclosing an identity does not necessarily mean that it is a cause of distress. In this work, we audit how LLMs interpret distress through disclosed identities. Using paired counterfactual interventions, we evaluate six LLMs across two SI and distress datasets, 16 identity and control conditions, and two prompting protocols. Our findings suggest that models can transform identity disclosures into explanatory narratives, attributing distress to presumed identity-related adversity. These assumptions frequently appear within otherwise empathetic responses, particularly when marginalized identities are disclosed. We contextualize our findings through the lens of compassionate othering to characterize these patterns as a form of harm in which empathetic responses impose identity-linked assumptions on individual suffering. These responses can misrepresent individuals' experiences by treating their identities as explanations for distress without supporting evidence. Our findings highlight the need to mitigate these harms by grounding interpretations of distress in individuals' disclosures and avoiding unsupported identity-linked assumptions.
The compiled record
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
The abstract, sentence by sentence
Individuals experiencing suicidal ideation (SI) or psychological distress may disclose their social identities when seeking support from LLMs. Attending to identity-related stressors can be important for providing contextually appropriate and empathetic support. However, disclosing an identity does not necessarily mean that it is a cause of distress. In this work, we audit how LLMs interpret distress through disclosed identities. Using paired counterfactual interventions, we evaluate six LLMs across two SI and distress datasets, 16 identity and control conditions, and two prompting protocols. Our findings suggest that models can transform identity disclosures into explanatory narratives, attributing distress to presumed identity-related adversity. These assumptions frequently appear within otherwise empathetic responses, particularly when marginalized identities are disclosed. We contextualize our findings through the lens of compassionate othering to characterize these patterns as a form of harm in which empathetic responses impose identity-linked assumptions on individual suffering. These responses can misrepresent individuals' experiences by treating their identities as explanations for distress without supporting evidence. Our findings highlight the need to mitigate these harms by grounding interpretations of distress in individuals' disclosures and avoiding unsupported identity-linked assumptions.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Individuals experiencing suicidal ideation (SI) or psychological distress may disclose their social identities when seeking support from LLMs. | background 1.00 | 0.08 | 0.02 | hedged possibility 1.00 |
| s1 | Attending to identity-related stressors can be important for providing contextually appropriate and empathetic support. | background 1.00 | 0.05 | 0.03 | hedged possibility 0.51 |
| s2 | However, disclosing an identity does not necessarily mean that it is a cause of distress. | background 0.92 | 0.05 | 0.06 | asserted 0.78 |
| s3 | In this work, we audit how LLMs interpret distress through disclosed identities. | contribution 1.00 | 0.90 | 0.03 | asserted 0.87 |
| s4 | Using paired counterfactual interventions, we evaluate six LLMs across two SI and distress datasets, 16 identity and control conditions, and two prompting protocols. | method 1.00 | 0.95 | 0.03 | asserted 0.98 |
| s5 | Our findings suggest that models can transform identity disclosures into explanatory narratives, attributing distress to presumed identity-related adversity. | result 1.00 | 0.94 | 0.67 | hedged possibility 0.98 |
| s6 | These assumptions frequently appear within otherwise empathetic responses, particularly when marginalized identities are disclosed. | result 0.98 | 0.88 | 0.16 | asserted 0.87 |
| s7 | We contextualize our findings through the lens of compassionate othering to characterize these patterns as a form of harm in which empathetic responses impose identity-linked assumptions on individual suffering. | interpretation 1.00 | 0.55 | 0.48 | asserted 0.97 |
| s8 | These responses can misrepresent individuals' experiences by treating their identities as explanations for distress without supporting evidence. | interpretation 0.88 | 0.62 | 0.20 | asserted 0.56 |
| s9 | Our findings highlight the need to mitigate these harms by grounding interpretations of distress in individuals' disclosures and avoiding unsupported identity-linked assumptions. | future work 0.95 | 0.40 | 0.55 | asserted 0.89 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
Figures
Figures reproduced from arXiv:2611.10808 by the authors.
References
Discussion
Disagree with the review, or know something it missed? Leave a comment, highlight any passage above to comment on it, or challenge this review formally.