Caring Without Feeling: Affective Dynamics as the Control Layer of Human-AI Agent Collaboration
Pith reviewed 2026-06-30 23:28 UTC · model grok-4.3
The pith
Affective dynamics function as the control layer through which humans and AI agents negotiate capability, uncertainty and responsibility.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Affective dynamics are the processes through which affective cues, emotion-like behaviour and perceived agent affect shape trust calibration, delegation decisions, error correction, dependence and governance. The framework treats affect not as an internal property of AI but as a coordination layer through which humans and agents negotiate capability, uncertainty and responsibility. This provides a foundation for calibrated measurement, purposeful design and informed governance.
What carries the argument
Affective dynamics as the control layer: interaction loops in which model-generated affective signals enter human-agent exchanges that govern reliance, repair and oversight.
If this is right
- Measurement instruments can be developed to track how specific affective signals alter reliance and repair loops.
- Design choices for agents can intentionally modulate affective cues to support appropriate delegation and oversight.
- Governance frameworks can incorporate affective coordination as a factor when assigning responsibility between humans and agents.
- Error patterns and dependence levels become predictable from the structure of affective interaction loops.
Where Pith is reading between the lines
- The framework suggests empirical tests that vary the timing and intensity of affective signals across multi-session agent tasks to identify thresholds that shift human oversight.
- It implies that similar coordination mechanisms may operate in non-AI automation settings, offering a bridge to existing human-factors research on supervisory control.
- Safety evaluations of autonomous agents would need to include interactional measures of perceived affect rather than relying solely on model internals.
Load-bearing premise
The fragmented literatures on affective computing, simulated empathy in large language models, trust in automation and AI safety can be synthesized into one integrated account of how affective cues operate in agentic collaboration.
What would settle it
An experiment in which humans delegate consequential tasks to autonomous agents but show no measurable change in delegation rate, trust ratings or error-correction behaviour when affective signals are added or removed would falsify the central claim.
read the original abstract
AI agents that plan, retain memory across sessions, invoke external tools and act with partial autonomy are transforming human--AI collaboration. Research on affective computing, simulated empathy in large language models, trust in automation and AI safety has illuminated important design principles, yet these literatures remain fragmented. No integrated account explains how affective cues operate within agentic collaboration -- settings in which humans delegate, monitor and correct consequential tasks. This Review synthesises computational and interactional mechanisms of affective dynamics: the processes through which affective cues, emotion-like behaviour and perceived agent affect shape trust calibration, delegation decisions, error correction, dependence and governance. We trace how model-generated affective signals enter interaction loops that govern reliance, repair and oversight, and propose a framework that treats affect not as an internal property of AI but as a coordination layer through which humans and agents negotiate capability, uncertainty and responsibility. The framework provides a foundation for calibrated measurement, purposeful design and informed governance.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript is a review paper that synthesizes research from affective computing, simulated empathy in large language models, trust in automation, and AI safety. It proposes a framework in which affective dynamics function as the control layer of human-AI agent collaboration, shaping trust calibration, delegation decisions, error correction, dependence, and governance. The central claim is that model-generated affective signals enter interaction loops that allow humans and agents to negotiate capability, uncertainty, and responsibility, providing a foundation for measurement, design, and governance rather than a validated empirical model.
Significance. If the synthesis holds, the framework would offer an integrated perspective on affective cues in agentic settings that could inform the design of collaborative AI systems and support more calibrated oversight mechanisms. By reframing affect as a coordination layer rather than an internal property, the work could help bridge currently fragmented literatures and guide future empirical studies on reliance and repair in autonomous agent interactions.
minor comments (3)
- [Abstract] Abstract: The statement that 'no integrated account explains how affective cues operate within agentic collaboration' would benefit from a short clause indicating the specific gaps (e.g., lack of joint treatment of delegation and oversight) to make the motivation more precise.
- [Framework] Framework section: The claim that the framework 'treats affect not as an internal property of AI but as a coordination layer' is central; adding one or two concrete examples of how existing affective-computing mechanisms map onto this layer would strengthen the integration without requiring new data.
- [Governance] Governance discussion: The section on informed governance references AI safety literature but could note one or two existing regulatory frameworks (e.g., EU AI Act provisions on transparency) to illustrate applicability.
Simulated Author's Rebuttal
We thank the referee for the constructive summary and positive assessment of the manuscript. The recommendation for minor revision is noted. No specific major comments appear in the report, so our response addresses the overall evaluation.
Circularity Check
No significant circularity
full rationale
The manuscript is a conceptual review paper that synthesizes fragmented literatures on affective computing, simulated empathy, trust in automation, and AI safety into a proposed coordination-layer framework. It contains no equations, derivations, fitted parameters, or empirical predictions. All central claims are scoped as providing a foundation for measurement and design rather than deriving new results from self-referential steps or self-citations. The argument structure is self-contained as an integrative synthesis without any reduction of outputs to inputs by construction.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption The literatures on affective computing, simulated empathy in large language models, trust in automation and AI safety are fragmented and require integration for agentic collaboration.
Reference graph
Works this paper leans on
-
[1]
S.et al.Generative agents: Interac- tive simulacra of human behavior.Proceed- ings of the 36th Annual ACM Symposium on User Interface Software and Technology 1–22 (2023)
Park, J. S.et al.Generative agents: Interac- tive simulacra of human behavior.Proceed- ings of the 36th Annual ACM Symposium on User Interface Software and Technology 1–22 (2023)
2023
-
[2]
Wang, L.et al.A survey on large language model based autonomous agents.Frontiers of Computer Science18, 186345 (2024)
2024
-
[3]
Xi, Z.et al.The rise and potential of large language model based agents: A sur- vey.Science China Information Sciences 68, 121101 (2025)
2025
-
[4]
Lee, J. D. & See, K. A. Trust in automation: Designing for appropriate reliance.Human factors46, 50–80 (2004)
2004
-
[5]
Sharma, M.et al.Towards understand- ing sycophancy in language models.The Twelfth International Conference on Learn- ing Representations(2024)
2024
-
[6]
Cheng, M.et al.Sycophantic ai decreases prosocial intentions and promotes depen- dence.Science391, eaec8352 (2026). 16
2026
-
[7]
& Campos-Castillo, C
Laestadius, L., Bishop, A., Gonzalez, M., Illenˇ c´ ık, D. & Campos-Castillo, C. Too human and not human enough: A grounded theory analysis of mental health harms from emotional dependence on the social chatbot replika.New Media & Society26, 5923–5941 (2024)
2024
-
[8]
Amershi, S.et al.Guidelines for human- ai interaction.Proceedings of the 2019 CHI Conference on Human Factors in Comput- ing Systems1–13 (2019)
2019
-
[9]
Smuha, N. A. Regulation 2024/1689 of the eur. parl. & council of june 13, 2024 (eu arti- ficial intelligence act).International Legal Materials64, 1234–1381 (2025)
2024
-
[10]
& Verduyn, P
Kuppens, P. & Verduyn, P. Emotion dynam- ics.Current opinion in psychology17, 22–26 (2017)
2017
-
[11]
Calvo, R. A. & D’Mello, S. Affect detec- tion: An interdisciplinary review of models, methods, and their applications.IEEE Transactions on affective computing1, 18– 37 (2010)
2010
-
[12]
& Sengers, P
Boehner, K., DePaula, R., Dourish, P. & Sengers, P. How emotion is made and measured.International Journal of Human- Computer Studies65, 275–291 (2007)
2007
-
[13]
& Moon, Y
Nass, C. & Moon, Y. Machines and mindlessness: Social responses to computers. Journal of social issues56, 81–103 (2000)
2000
-
[14]
Extended Abstracts of the CHI Conference on Human Factors in Computing Systems 1–15 (2024)
Cohn, M.et al.Believing anthropomor- phism: Examining the role of anthropomor- phic cues on trust in large language models. Extended Abstracts of the CHI Conference on Human Factors in Computing Systems 1–15 (2024)
2024
-
[15]
Advances in neural information processing systems35, 27730–27744 (2022)
Ouyang, L.et al.Training language models to follow instructions with human feedback. Advances in neural information processing systems35, 27730–27744 (2022)
2022
-
[16]
& Lvovsky, M
Elyoseph, Z., Hadar-Shoval, D., Asraf, K. & Lvovsky, M. Chatgpt outperforms humans in emotional awareness evaluations.Fron- tiers in psychology14, 1199058 (2023)
2023
-
[17]
Schlegel, K., Sommer, N. R. & Mortillaro, M. Large language models are proficient in solving and creating emotional intelligence tests.Communications Psychology3, 80 (2025)
2025
-
[18]
W.et al.Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum.JAMA internal medicine183, 589–596 (2023)
Ayers, J. W.et al.Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum.JAMA internal medicine183, 589–596 (2023)
2023
-
[19]
Sorin, V.et al.Large language models and empathy: systematic review.Journal of medical Internet research26, e52597 (2024)
2024
-
[20]
& Nass, C
Reeves, B. & Nass, C. The media equation: How people treat computers, television, and new media like real people.Cambridge, UK 10, 19–36 (1996)
1996
-
[21]
& Cacioppo, J
Epley, N., Waytz, A. & Cacioppo, J. T. On seeing human: a three-factor theory of anthropomorphism.Psychological review 114, 864 (2007)
2007
-
[22]
Hurst, A.et al.Gpt-4o system card.arXiv preprint arXiv:2410.21276(2024)
work page internal anchor Pith review Pith/arXiv arXiv 2024
- [23]
-
[24]
R., Gabriel, I., Summerfield, C., Vidgen, B
Kirk, H. R., Gabriel, I., Summerfield, C., Vidgen, B. & Hale, S. A. Why human–ai relationships need socioaffective alignment. Humanities and Social Sciences Communi- cations12, 1–9 (2025)
2025
-
[25]
& Sharot, T
Glickman, M. & Sharot, T. How human– ai feedback loops alter human perceptual, emotional and social judgements.Nature Human Behaviour9, 345–359 (2025)
2025
-
[26]
Demszky, D.et al.Goemotions: A dataset of fine-grained emotions.Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics4040–4054 (2020). 17
2020
-
[27]
& Hovy, E
Poria, S., Majumder, N., Mihalcea, R. & Hovy, E. Emotion recognition in conver- sation: Research challenges, datasets, and recent advances.IEEE Access7, 100943– 100953 (2019)
2019
-
[28]
& Carvalho, J
Pereira, P., Moniz, H. & Carvalho, J. P. Deep emotion recognition in textual con- versations: a survey.Artificial Intelligence Review58(2025)
2025
-
[29]
F., Adolphs, R., Marsella, S., Martinez, A
Barrett, L. F., Adolphs, R., Marsella, S., Martinez, A. M. & Pollak, S. D. Emo- tional expressions reconsidered: Challenges to inferring emotion from human facial movements.Psychological science in the public interest20, 1–68 (2019)
2019
-
[30]
D’mello, S. K. & Kory, J. A review and meta-analysis of multimodal affect detection systems.ACM computing surveys (CSUR) 47, 1–36 (2015)
2015
-
[31]
Feng, S.et al.Affect recognition in conver- sations using large language models.Pro- ceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue259–273 (2024)
2024
-
[32]
Lian, Z.et al.Affectgpt: A new dataset, model, and benchmark for emotion under- standing with multimodal large language models.Proceedings of the 42nd Inter- national Conference on Machine Learning (2025)
2025
-
[33]
Kang, D.et al.Can large language models be good emotional supporters? mitigating preference bias on emotional support con- versation.Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics15232–15261 (2024)
2024
-
[34]
Liu, S.et al.Towards emotional support dia- log systems.Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Pro- cessing3469–3483 (2021)
2021
-
[35]
W., Miner, A
Sharma, A., Lin, I. W., Miner, A. S., Atkins, D. C. & Althoff, T. Human–ai collabora- tion enables more empathic conversations in text-based peer-to-peer mental health sup- port.Nature Machine Intelligence5, 46–57 (2023)
2023
-
[36]
Chen, D.et al.Patient perceptions of empathy in physician and artificial intelli- gence chatbot responses to patient questions about cancer.NPJ Digital Medicine8, 275 (2025)
2025
- [37]
-
[38]
Wang, X., Li, C., Chang, Y., Wang, J. & Wu, Y. Negativeprompt: Leveraging psy- chology for large language models enhance- ment via negative emotional stimuli.Pro- ceedings of the Thirty-Third International Joint Conference on Artificial Intelligence 6504–6512 (2024)
2024
-
[39]
& Sekine, S
Yin, Z., Wang, H., Horio, K., Kawahara, D. & Sekine, S. Should we respect llms? a cross- lingual study on the influence of prompt politeness on llm performance.Proceedings of the Second Workshop on Social Influence in Conversations9–35 (2024)
2024
-
[40]
& Suh, B
Kim, K., Jeon, H., Ryu, J. & Suh, B. Will llms sink or swim? exploring decision- making under pressure.Findings of the Association for Computational Linguistics: EMNLP 202411425–11450 (2024)
2024
-
[41]
& Suhr, A
Sclar, M., Choi, Y., Tsvetkov, Y. & Suhr, A. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting.The Twelfth International Con- ference on Learning Representations(2024)
2024
-
[42]
& Jurgens, D
Zheng, M., Pei, J., Logeswaran, L., Lee, M. & Jurgens, D. When a helpful assis- tant is not really helpful: Personas in system prompts do not improve performances of large language models.Findings of the Association for Computational Linguistics: 18 EMNLP 202415126–15154 (2024)
2024
-
[43]
NeurIPS 2023 Foundation Models for Deci- sion Making Workshop(2024)
Ruan, Y.et al.Identifying the risks of lm agents with an lm-emulated sandbox. NeurIPS 2023 Foundation Models for Deci- sion Making Workshop(2024)
2023
-
[44]
Hua, W.et al.Trustagent: Towards safe and trustworthy llm-based agents.Findings of the Association for Computational Linguis- tics: EMNLP 202410000–10016 (2024)
2024
-
[45]
Tseng, Y.-M.et al.Two tales of persona in llms: A survey of role-playing and person- alization.Findings of the Association for Computational Linguistics: EMNLP 2024 16612–16631 (2024)
2024
-
[46]
Jiang, H.et al.Personallm: Investigating the ability of large language models to express personality traits.Findings of the Associa- tion for Computational Linguistics: NAACL 20243605–3627 (2024)
2024
-
[47]
Zhang, Z.et al.A survey on the memory mechanism of large language model-based agents.ACM Transactions on Information Systems43, 1–47 (2025)
2025
-
[48]
Packer, C.et al.Memgpt: towards llms as operating systems. (2023)
2023
-
[49]
Bai, Y.et al.Constitutional ai: Harm- lessness from ai feedback.arXiv preprint arXiv:2212.08073(2022)
work page internal anchor Pith review Pith/arXiv arXiv 2022
-
[50]
Glaese, A.et al.Improving alignment of dialogue agents via targeted human judge- ments.arXiv preprint arXiv:2209.14375 (2022)
work page internal anchor Pith review Pith/arXiv arXiv 2022
-
[51]
Findings of the Association for Computa- tional Linguistics: ACL 202313387–13434 (2023)
Perez, E.et al.Discovering language model behaviors with model-written evaluations. Findings of the Association for Computa- tional Linguistics: ACL 202313387–13434 (2023)
2023
-
[52]
Hoff, K. A. & Bashir, M. Trust in automa- tion: Integrating empirical evidence on fac- tors that influence trust.Human factors57, 407–434 (2015)
2015
-
[53]
C., Davis, J
Mayer, R. C., Davis, J. H. & Schoorman, F. D. An integrative model of organizational trust.Academy of management review20, 709–734 (1995)
1995
-
[54]
Klein, S. H. The effects of human-like social cues on social responses towards text-based conversational agents—a meta- analysis.Humanities and Social Sciences Communications12, 1–16 (2025)
2025
-
[55]
& Epley, N
Waytz, A., Heafner, J. & Epley, N. The mind in the machine: Anthropomorphism increases trust in an autonomous vehicle. Journal of experimental social psychology 52, 113–117 (2014)
2014
-
[56]
J.et al.Almost human: Anthropomorphism increases trust resilience in cognitive agents.Journal of Experimental Psychology: Applied22, 331 (2016)
De Visser, E. J.et al.Almost human: Anthropomorphism increases trust resilience in cognitive agents.Journal of Experimental Psychology: Applied22, 331 (2016)
2016
-
[57]
& Schreck, P
Klingbeil, A., Gr¨ utzner, C. & Schreck, P. Trust and reliance on ai—an experimental study on the extent and costs of overreliance on ai.Computers in Human Behavior160, 108352 (2024)
2024
-
[58]
S., Liao, Q
Kim, S. S., Liao, Q. V., Vorvoreanu, M., Ballard, S. & Vaughan, J. W. I’m Not Sure, But...: Examining the impact of large language models’ uncertainty expression on user reliance and trust.Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency822–835 (2024)
2024
-
[59]
& Lee, Y.-C
Xu, Z., Song, T. & Lee, Y.-C. Confronting verbalized uncertainty: Understanding how llm’s verbalized uncertainty influences users in ai-assisted decision-making.International Journal of Human-Computer Studies197, 103455 (2025)
2025
-
[60]
Artificial empathy in healthcare chatbots: Does it feel authentic?Computers in Human Behavior: Artificial Humans2, 100067 (2024)
Seitz, L. Artificial empathy in healthcare chatbots: Does it feel authentic?Computers in Human Behavior: Artificial Humans2, 100067 (2024)
2024
-
[61]
& Gadiraju, U
He, G., Demartini, G. & Gadiraju, U. Plan- then-execute: An empirical study of user 19 trust and team performance when using llm agents as a daily assistant.Proceed- ings of the 2025 CHI Conference on Human Factors in Computing Systems1–22 (2025)
2025
-
[62]
Parasuraman, R., Sheridan, T. B. & Wick- ens, C. D. A model for types and lev- els of human interaction with automation. IEEE Transactions on systems, man, and cybernetics-Part A: Systems and Humans 30, 286–297 (2000)
2000
-
[63]
Y., Wan, S
Bo, J. Y., Wan, S. & Anderson, A. To rely or not to rely? evaluating interventions for appropriate reliance on large language mod- els.Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems 1–23 (2025)
2025
-
[64]
Lai, V.et al.Human-ai collaboration via conditional delegation: A case study of con- tent moderation.Proceedings of the 2022 CHI Conference on Human Factors in Com- puting Systems1–18 (2022)
2022
-
[65]
Salikutluk, V.et al.An evaluation of situa- tional autonomy for human-ai collaboration in a shared workspace setting.Proceed- ings of the 2024 CHI Conference on Human Factors in Computing Systems1–17 (2024)
2024
-
[66]
& Taddeo, M
Tsamados, A., Floridi, L. & Taddeo, M. Human control of ai systems: from supervi- sion to teaming.AI and Ethics5, 1535–1548 (2025)
2025
-
[67]
& Tauber, E
Nass, C., Steuer, J. & Tauber, E. R. Com- puters are social actors.Proceedings of the SIGCHI Conference on Human Factors in Computing Systems72–78 (1994)
1994
-
[68]
Why Language Models Hallucinate
Kalai, A. T., Nachum, O., Vempala, S. S. & Zhang, E. Why language models hal- lucinate.arXiv preprint arXiv:2509.04664 (2025)
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[69]
Zhang, Y., Liao, Q. V. & Bellamy, R. K. Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making.Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency295–305 (2020)
2020
-
[70]
S., Vaughan, J
Kim, S. S., Vaughan, J. W., Liao, Q. V., Lombrozo, T. & Russakovsky, O. Fostering appropriate reliance on large language mod- els: The role of explanations, sources, and inconsistencies.Proceedings of the 2025 CHI Conference on Human Factors in Comput- ing Systems1–19 (2025)
2025
-
[71]
Sharma, N.et al.Feedback by design: Understanding and overcoming user feed- back barriers in conversational agents.Pro- ceedings of the 2026 CHI Conference on Human Factors in Computing Systems1–22 (2026)
2026
-
[72]
sorry, it was my fault
Zhang, X., Lee, S. K., Kim, W. & Hahn, S. “sorry, it was my fault”: Repairing trust in human-robot interactions.International Journal of Human-Computer Studies175, 103031 (2023)
2023
-
[73]
Srikumar, M.et al.Prioritizing real-time failure detection in ai agents.Partnership on AI(2025)
2025
-
[74]
& Shang, S
Xu, W., Huang, C., Gao, S. & Shang, S. Llm-based agents for tool learning: A survey: W. xu et al.Data Science and Engineering1–31 (2025)
2025
-
[75]
& Wakslak, C
Yin, Y., Jia, N. & Wakslak, C. J. Ai can help people feel heard, but an ai label diminishes this impact.Proceedings of the National Academy of Sciences121, e2319112121 (2024)
2024
-
[77]
Fang, C. M.et al.How ai and human behaviors shape psychosocial effects of extended chatbot use: A longitudinal ran- domized controlled study.arXiv preprint arXiv:2503.17473(2025)
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[78]
& Barco, A
K¨ uhne, R., Peter, J., de Jong, C. & Barco, A. How does children’s anthropomorphism of a social robot develop over time? a six- wave panel study.International Journal of 20 Social Robotics16, 1665–1679 (2024)
2024
-
[79]
& Todorovic, M
Pu, L., Moyle, W., Jones, C. & Todorovic, M. The effectiveness of social robots for older adults: a systematic review and meta- analysis of randomized controlled studies. The gerontologist59, e37–e51 (2019)
2019
-
[80]
Berridge, C., Zhou, Y., Robillard, J. M. & Kaye, J. Companion robots to mitigate lone- liness among older adults: Perceptions of benefit and possible deception.Frontiers in Psychology14, 1106633 (2023)
2023
-
[81]
& West, J
Peter, S., Riemer, K. & West, J. D. The benefits and dangers of anthropomor- phic conversational agents.Proceedings of the National Academy of Sciences122, e2415898122 (2025)
2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.