Pith. sign in

REVIEW 5 major objections 7 minor 34 references

Words Like Knives: Backstory-Personalized Modeling and Detection of Violent Communication

T0 review · 5 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Relationship backstory shifts how humans judge conversational conflict, but LLMs fail to use it.

desk verdict Useful new simulated dataset and a plausible human-vs-LLM gap, but the human effect stats need a clustering-aware redo before the headline claim holds. read the letter →

arxiv 2505.21451 v2 pith:SD5LC4R3 submitted 2025-05-27 cs.CL

Jocelyn Shen , Akhila Yerukola , Xuhui Zhou , Cynthia Breazeal , Maarten Sap , Hae Won Park This is my paper
classification cs.CL
keywords conversationalbreakdowndetectionnonviolentcommunicationrelationshipbackstoryAI-mediatedLLMperceptionofconflictsyntheticdialoguecorpusemotionalimpactpredictionpersonalization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that whether a message in an intimate conversation counts as a breakdown depends on the relationship history of the people saying it, not just on the words. Using nonviolent communication theory, the authors generate 5,772 simulated dialogues between friends, family, and romantic partners, each paired with a positive and a negative backstory, and collect turn-level human ratings on a subset. They find that backstory polarity significantly changes human judgments of how problematic a turn is and of the speaker's intentions, with sympathy as the mediating mechanism. They then show that LLMs, while benefiting from the full conversation, largely fail to integrate the backstory, and consistently overestimate how positively a message will make the listener feel. If right, the work implies that AI systems meant to mediate or reframe human conflict must be personalized to the relationship context to be trustworthy.

What carries the argument

The paper's central mechanism is the paired backstory-polarity manipulation built on Nonviolent Communication (NVC) theory. NVC supplies five 'violent' communication forms (moralistic judgment, comparison, denial of responsibility, demand/threat, deserve/punitive thinking) and five nonviolent forms (observation, feeling, need, request, empathy) that define the labeling scheme. For each of 5,772 GPT-4-simulated conversations, the authors generate two backstories about the same dyad, one casting the focal character as sympathetic and one as blameworthy, then measure whether ratings of the identical turns shift between conditions. The same data structure feeds the LLM experiments, which vary context in three levels (turn only; turn plus conversation; turn plus conversation plus backstory) to isolate what models can and cannot exploit.

What would settle it

A study using a corpus of real or self-reported conflict conversations between intimates, with the same turn-level rating tasks: if human ratings do not shift with backstory polarity, or if some LLM's detection accuracy improves when given backstory, the paper's central asymmetry claim would be contradicted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that communication breakdowns in close relationships are not intrinsic to utterances: the same turn is rated as significantly more problematic, less understandable, more negatively intentioned, and less sympathetic when it is accompanied by a negative relationship backstory than by a positive one (turn-level t(1083)=3.73, p=.0002; conversation-level d=0.55), and this effect is mediated by sympathy toward the speaker (β1=0.31, β2=-0.46). Against that human baseline, GPT-4o, Llama-4, and Gemini-1.5-pro all fail to meaningfully use the backstory in the problematic-detection task: adding backstory produces no significant improvement in the positive condition and can hurt performance in the negative condition, while all models consistently overpredict positive emotional impact relative to human listeners.

Load-bearing premise

The findings assume that GPT-4-simulated dialogues with deliberately injected conflict markers are naturalistic enough to stand in for real intimate relationship conflicts, which the paper's limitations section acknowledges may not hold.

Editorial extensions

If this is right

  • If backstory polarity shifts human judgments of problematicness, then 'how problematic a message is' is not a stable property of the text; any conflict-detection benchmark that omits relationship context is measuring something narrower than real-world harm.
  • AI-mediated communication systems that ignore relationship history will systematically misjudge whether a turn needs reframing.
  • Adding backstory to LLM prompts is not enough: current models either fail to improve or overcorrect, so personalization requires training or inference-time adaptation beyond prompt context.
  • Models' consistent overprediction of positive emotional impact means they will tend to green-light hurtful messages as harmless in intimate settings.
  • The PersonaConflicts Corpus provides a controlled testbed for future relationship-contextualized detection and generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the effect generalizes, real-world conflict mediation tools should elicit the user's own perceived relationship history, not an externally supplied backstory, to calibrate what counts as harmful.
  • The backstory-polarity manipulation could be used adversarially: the same dialogue can be framed to make a victim look abusive or an abuser sympathetic, so any deployed mediator must be alert to framing bias.
  • A natural next test is to replace simulated dialogues with real or self-reported intimate conflicts and check whether human backstory effects and model failures replicate; if they do, the corpus is a valid proxy.
  • The finding that negative backstories sometimes hurt model performance suggests models may anchor on the character's overall valence rather than integrating the backstory into turn-level reasoning; this could be tested by counterfactual backstories.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper introduces the PersonaConflicts Corpus, a set of 5,772 GPT-4-simulated conflict and non-conflict dialogues between friends, family, and romantic partners, each paired with two deliberately opposing relationship backstories (positive and negative). The authors run a between-subjects human study on a 120-conversation subset with turn-level and conversation-level ratings grounded in Nonviolent Communication (NVC) theory, and then evaluate GPT-4o, LLaMA-4, and Gemini-1.5-pro on two tasks: predicting whether a turn is problematic and predicting its emotional impact on the listener, under three context conditions (turn only, turn+conversation, turn+conversation+backstory). The main claims are that backstory polarity significantly shifts human perception of communication quality (Section 6), that sympathy mediates this effect, and that LLMs fail to meaningfully leverage backstory, while generally overestimating the positive emotional impact of messages (Section 7).

Significance. If the central human-perception result survives a clustering-aware reanalysis, the paper makes a useful contribution: it provides a theory-grounded, publicly released corpus for studying relationship-contextualized conflict, a controlled between-subjects design that is relatively rare in NLP crowdsourcing, and a systematic comparison of multiple LLMs under controlled context manipulations. The authors are also candid in the Limitations section about the synthetic nature of the data and the moderate inter-annotator agreement. However, the current inferential statistics do not yet establish the paper's central quantitative claim, and the model-comparison significance reporting is underspecified. The strengths of the paper—corpus release, controlled stimuli, and the use of NVC as an interpretable annotation scheme—are real, but they are attached to a load-bearing statistical analysis that needs to be redone.

major comments (5)
  1. [Section 6 (RQ1 analysis)] The independent t-tests reported for the human study—t(1083) for turn-level problematicness and t(232) for conversation-level outcomes—treat turns and conversations as independent observations. The data are hierarchically structured: 91 participants rated multiple conversations, with between-subjects assignment to backstory condition, and turns are nested within conversations. Ratings from the same participant and same conversation are likely correlated, so the effective sample size is far smaller than the degrees of freedom suggest and the reported p-values are anti-conservative. The turn-level effect is small (Cohen's d = 0.23), so it may not survive a mixed-effects model with random intercepts for participant and conversation. The authors need to reanalyze Section 6 with multilevel models (or at least cluster-robust standard errors), report the intraclass correlation, and provide effect sizes with confidence intervals. This is essential because the paper's central claim that backstory polarity significantly shifts human perception rests directly on these tests.
  2. [Section 5.1 (Believability filter)] The paper states that 'we filter only on believable stories to ensure validity of our results' but does not report how many of the 120 conversations or 240 backstory variants were excluded by this post-hoc believability filter, nor the criteria for exclusion. Because the believability ratings come from the same annotators who provide the outcome ratings used in Sections 6 and 7, filtering on these ratings can introduce selection bias and is not a pre-registered inclusion rule. The authors should report the exact number of excluded items, and ideally show that the main results in Sections 6 and 7 are robust to including all data or to using a pre-specified believability threshold.
  3. [Section 7.2 and Table 2] The significance stars in Table 2 are said to denote 'statistical difference from the prior condition,' but the manuscript never describes the statistical test used, how it accounts for turns nested within conversations, or whether any multiple-comparison correction was applied. The F1 differences are often small (e.g., GPT-4o positive backstory turn+convo = 45.96 vs. turn+convo+backstory = 48.07), so without a clearly specified test and confidence intervals, the claim that 'models fail to meaningfully leverage backstory' is not quantitatively supported. The authors should specify the test (e.g., paired bootstrap at the conversation level), report confidence intervals, and clarify whether all three context conditions were evaluated on the same annotated subset.
  4. [Section 6 (Mediation analysis)] The mediation analysis using structural equation modeling is reported only as two path coefficients (β1 = 0.31, β2 = −0.46) in Figure 7, with no standard errors, p-values, confidence intervals, or a test of the indirect effect. The conclusion that 'sympathy mediates the relationship between backstory type and perceived problematicness' therefore cannot be evaluated from the manuscript as written. The authors need to provide the full model specification, the indirect effect estimate with a bootstrap confidence interval, and the sample size used for the SEM.
  5. [Section 5.2 and Table 1] Inter-annotator agreement on the turn-level labels used as gold for the LLM evaluation is moderate (Krippendorff's alpha = 0.34–0.46). Since the model F1 scores in Table 2 are computed against human gold labels obtained by averaging only two annotators, the model comparisons inherit substantial label noise. The authors should discuss how this reliability level affects the model-evaluation conclusions, and ideally report results against each annotator separately or use an agreement-weighted aggregation to show that the conclusions are robust to annotation noise.
minor comments (7)
  1. [Section 2.2] There is a typo: 'inerpersonal' should be 'interpersonal' in the last sentence of the section.
  2. [Section 5.2] The phrase 'which we hypothesize can be do to variations' contains a typo; it should be 'can be due to variations.'
  3. [Appendix A.4] In the backstory generation prompt, 'a number fo how frequently' should be 'a number of how frequently.'
  4. [Author affiliations] The affiliation line for the Allen Institute reads 'Seattle, W A, USA'; this should be 'WA, USA.'
  5. [Section 7.2] The phrase 'in alignment with human perception' is confusing in context: the authors have just shown that humans do shift their perceptions with backstory, whereas models largely do not. Please rephrase to clarify that the model behavior is not aligned with human behavior.
  6. [Section 7.1 and Table 2] The manuscript should state explicitly whether the LLM experiments are run on the full corpus or only on the 120 human-annotated conversations; the current text says human gold labels are obtained for 'each task,' which implies the annotated subset, but this is not stated.
  7. [Figure 6 and Table 2] The bold and underline formatting in Table 2 and the layout of Figure 6 should be explained in the captions; the current captions do not define the highlighting.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports empirical human and model evaluations without fitted inputs or derivation chains that reduce to their own assumptions.

full rationale

This paper is an empirical study, not a derivation. The central claims are that backstory polarity shifts human perception of conflict and that LLMs do not meaningfully leverage backstories. The backstories are generated externally by GPT-4 using explicit prompts, then presented to human annotators; human ratings are collected independently and compared across between-subjects conditions. LLM evaluations are zero-shot with no training on the human labels, so there is no fitted parameter later renamed as a prediction. The NVC framework is used as a labeling scheme rather than as an output that is derived from the data. Some prior works by the same authors are cited for simulation infrastructure (e.g., SOTOPIA) and for contextual reasoning, but these citations are not load-bearing for the empirical findings; they provide tools and related context, not the conclusions. The paper's limitations acknowledge the synthetic nature of the data and moderate inter-annotator agreement, but those are validity concerns, not circularity. No equation or statistical construction in the paper makes an output equal to an input by definition. Therefore no significant circularity is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper's central claims rest on three domain assumptions: the NVC taxonomy is a valid scheme for labeling communication breakdowns, GPT-4 simulated dialogues with injected markers are naturalistic proxies for real conflicts, and crowdsourced ratings with moderate agreement are reliable gold labels. No fitted parameters or invented entities underpin the results.

assumptions (3)
  • domain assumption The five NVC violent communication types are a faithful taxonomy of conversational breakdowns.
    Section 3 grounds labels in Rosenberg and Chopra (2015) and injects only these types into simulated conflict dialogues; if the taxonomy misses other breakdown forms, dataset coverage is incomplete.
  • domain assumption LLM-simulated dialogues with injected violent communication markers approximate real relationship conflicts.
    Section 5.1 and Limitations acknowledge synthetic conversations may lack ecological validity; the entire human and model evaluation uses these dialogues.
  • domain assumption Crowdsourced MTurk judgments are reliable gold labels for problematicity and emotional impact despite moderate agreement.
    Section 5.2 reports Krippendorff's alpha 0.34 to 0.46; the study aggregates two annotators per conversation and uses these as gold labels for model evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Words Like Knives: Backstory-Personalized Modeling and Detection of Violent Communication." pith.science (2026). https://pith.science/paper/SD5LC4R3

@misc{pith2026250521451,
  author       = {Pith},
  title        = {Pith review of: Words Like Knives: Backstory-Personalized Modeling and Detection of Violent Communication},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SD5LC4R3}},
  note         = {Machine review of arXiv:2505.21451}
}
read the original abstract

Conversational breakdowns in close relationships are deeply shaped by personal histories and emotional context, yet most NLP research treats conflict detection as a general task, overlooking the relational dynamics that influence how messages are perceived. In this work, we leverage nonviolent communication (NVC) theory to evaluate LLMs in detecting conversational breakdowns and assessing how relationship backstory influences both human and model perception of conflicts. Given the sensitivity and scarcity of real-world datasets featuring conflict between familiar social partners with rich personal backstories, we contribute the PersonaConflicts Corpus, a dataset of N=5,772 naturalistic simulated dialogues spanning diverse conflict scenarios between friends, family members, and romantic partners. Through a controlled human study, we annotate a subset of dialogues and obtain fine-grained labels of communication breakdown types on individual turns, and assess the impact of backstory on human and model perception of conflict in conversation. We find that the polarity of relationship backstories significantly shifted human perception of communication breakdowns and impressions of the social partners, yet models struggle to meaningfully leverage those backstories in the detection task. Additionally, we find that models consistently overestimate how positively a message will make a listener feel. Our findings underscore the critical role of personalization to relationship contexts in enabling LLMs to serve as effective mediators in human communication for authentic connection.

Figures

Figures reproduced from arXiv: 2505.21451 by the authors.

Figure 1
Figure 1. Conversation turns can be perceived as more [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. We use Nonviolent Communication Theory to ground labels for communication types. Only violent [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of our simulation framework for generating conflict and non-conflict conversations and [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Example of turn-level annotation interface [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Distribution of believability scores for simu [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Human study results comparing impact of neg/pos backstory on perception of conflict and characters. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Sympathy mediates the relationship between [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 33 canonical work pages

  1. [2]

    The conversation should have a rise and fall, rather than repeating the same points over and over again

    work page

  2. [3]

    The conversation does NOT need to have a resolution

    work page

  3. [4]

    Use realistic emotional speech patterns — trailing off, pausing, short bursts

    work page

  4. [9]

    babe", "honey

    Depending on the relationship, characters should use pet names or titles (e.g. "babe", "honey", "sweetie", "Mom", "Dad")

    work page

  5. [10]

    ### ### Format the output as: Turn #1 (speaker 1's first name): dialogue Turn #2 (speaker 2's first name): dialogue A.3 Simulation Prompt (Conflict) Let's think step by step

    Remember, keep the dialogues neutral and ambiguous, leaving opening for multiple meanings depending on the backstory of the characters. ### ### Format the output as: Turn #1 (speaker 1's first name): dialogue Turn #2 (speaker 2's first name): dialogue A.3 Simulation Prompt (Conflict) Let's think step by step. Generate a 10 to maximum 15 turn conversation ...

    work page

  6. [11]

    The conversation should sound human

    Avoid sounding like a therapist or a robot. The conversation should sound human

    work page

  7. [12]

    Both characters should respond irrationally and emotionally

    work page

  8. [13]

    Each turn should be short

    work page

Show all 34 references
  1. [15]

    You’re such an idiot for doing that

    Judgment – Definition: Assigns fault or labels someone as bad/wrong. Example: “You’re such an idiot for doing that.” (Moralistic judgment)

  2. [16]

    Your work isn’t as good as X’s work

    Comparison – Definition: Unfavorably contrasts a person to another, causing inferiority/shame. Example: “Your work isn’t as good as X’s work.” “No one else is as dramatic as you”

  3. [17]

    I hit you because you provoked me

    Deflection of Responsibility – Definition: Denies ownership of one’s actions or feelings; blames external forces. Example: “I hit you because you provoked me.” “It’s your fault I’m in a crappy mood” “I feel like you don’t love me anymore”

  4. [18]

    You must do this, or you’ll be sorry

    Demand/Threat – Definition: Pressure or order with implied punishment or guilt if not obeyed. Example: “You must do this, or you’ll be sorry.” “You better fix your problem.”

  5. [19]

    deserve,

    Deserve/Punitive – Definition: Uses “deserve,” rewards, or punishment language to judge behavior. Example: “She messed up, so she deserves whatever happens to her.” ### Read through the overall conversation guidelines carefully. These are important:

  6. [20]

    Not every turn should be a conflict-inducing statement (ONLY 1-2 TURNS AT MOST FROM EACH CHARACTER)

  7. [21]

    The conflict should be extremely subtle, rather than overtly and obviously offensive

  8. [22]

    Joe recently lost a friend

    Make sure the conflict statement is appropriate to the magnitude of the scenario (e.g. "Joe recently lost a friend", bad example: "Oh come on, it's not like you lost your mom")

  9. [23]

    The conversation should be contextualized to the scenario and the character profiles

  10. [24]

    The conflict should have a rise and fall, rather than repeating the same points over and over again

  11. [25]

    Each character should respond to the other person’s attacks without backing down

  12. [26]

    BOTH characters should be responsible for the conflict

    Remember a conflict happen between TWO people. BOTH characters should be responsible for the conflict

  13. [27]

    The conflict does NOT need to have a resolution -- it can be cut off in the middle

  14. [28]

    Use realistic emotional speech patterns — trailing off, pausing, short bursts of anger

  15. [29]

    Use INFORMAL language

  16. [33]

    John, you should

    Do NOT keep referring to the other person's name (bad example: "John, you should...", "It's not like that, Mary"). In realistic dialogue, people often don't refer to each other's names

  17. [34]

    babe", "honey

    Depending on the relationship, characters should use pet names or titles (e.g. "babe", "honey", "sweetie", "Mom", "Dad") ### ### Format the output as: Turn #1 (speaker 1's first name): dialogue Turn #2 (speaker 2's first name): dialogue A.4 Backstory Generation Let's think ste...

  18. [35]

    For example, instead of stating a number fo how frequently they interact, show through the story

    Make sure the backstory reads as an actual narrative with flow, rather than a list of attributes. For example, instead of stating a number fo how frequently they interact, show through the story

  19. [36]

    The scenario happened AFTER the backstory

    Do NOT reference the scenario or conversation in the backstory. The scenario happened AFTER the backstory

  20. [37]

    Be as creative as possible

  21. [38]

    Do not simply state what the relationship between the characters is like, but SHOW through narrative and life events

  22. [39]

    backstory_one

    Each backstory should be at most a paragraph. This conversation happened in this scenario: {original_scenario} ### Speakers: {agent_1_name} and {agent_2_name} --- {agent_1_name} Profile --- {agent_1_data} --- {agent_2_name} Profile --- {agent_2_data} ### {agent_1_name} and {ag...

    2022

  23. [1985]

    Journal of personality and social psychology, 49:1246–65

    Emotional Communication in Close Relation- ships. Journal of personality and social psychology, 49:1246–65. Odette N. Gould and Sylvia MacNeil Gautreau

  24. [2014]

    flashbulb

    Empathy and Conversational Enjoyment in Younger and Older Adults. Experimental Aging Re- search, 40(1):60–80. Publisher: Routledge _eprint: https://doi.org/10.1080/0361073X.2014.857559. Bhanu Prakash Reddy Guda, Aparna Garimella, and Niyati Chhaya. 2021. EmpathBERT: A BERT-bas...

    2014

  25. [2019]

    PLoS ONE, 14(11):e0225102–e0225102

    Empathic concern and personal distress de- pend on situational but not dispositional factors. PLoS ONE, 14(11):e0225102–e0225102. Publisher: Public Library of Science. James D. Finch and Jinho D. Choi. 2024. Diverse and Effective Synthetic Data Generation for Adaptable Zero-Sh...

    2024

  26. [2025]

    ArXiv:2502.12813 [cs]

    Simulating User Diversity in Task-Oriented Dialogue Systems using Large Language Models. ArXiv:2502.12813 [cs]. Lisa P. Argyle, Christopher A. Bail, Ethan C. Busby, Joshua R. Gubler, Thomas Howe, Christopher Ryt- ting, Taylor Sorensen, and David Wingate. 2023. Leveraging AI fo...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.