REVIEW 1 major objections 5 minor 2 cited by
Project Riley: Multimodal Multi-Agent LLM Collaboration with Emotional Reasoning and Voting
T0 review · 1 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Project Riley claims that staging a debate among five emotional agent personas—Joy, Sadness, Fear, Anger, Disgust—before every chatbot answer makes the reply more emotionally appropriate and clear, and its user-test results support that…
desk verdict Project Riley is a plausible engineering combination of known components whose central effectiveness claim is not yet supported by the evaluation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The working mechanism is the emotion-persona debate combined with justified voting and late-stage synthesis. Five agents, each prompted as one Ekman-style basic emotion, independently generate answers, then read and critique one another over two discussion rounds, then finalise their positions; a reasoning model then votes on the candidate answers with a written justification for each vote, and a final synthesis produces a response segmented into REASONING, THOUGHTS, and FINAL ANSWER. This is the machinery that supposedly turns a single shared text LLM into a plural, emotionally differentiated system. In Armando, the additional load-bearing mechanism is conditional retrieval: embeddings of authoritative documents are compared with the cumulative conversational context, and the top matches are injected immediately before synthesis so the final answer is grounded without being diluted by the debate.
What would settle it
Run a user study that compares the full Riley pipeline against three ablations: a single LLM given the same final synthesis prompt with no debate; five agents with identical neutral personas debating and voting; and the full five-emotion pipeline with emotion labels removed from the output. If ratings of emotional appropriateness and clarity do not separate the full pipeline from the neutral-persona or label-stripped versions, the claimed emotion mechanism is not doing the work.
Extended reading notes
Core claim
The central claim is that a symbolic layer of emotional roles and deliberation can make a single text LLM behave as if it had distinct emotional reasoning. Each of the five agents keeps its own conversation history, answers the user's query at Round 0, reads and critiques the other agents' answers in later rounds, and finalises its own response; from these candidates, reasoning models vote with written justifications, and the synthesis stage emits a response that either follows the winning emotion or blends perspectives on a tie, structured as REASONING, THOUGHTS, and FINAL ANSWER. User-testing with 17 participants across five life themes supports the claim only partially: emotional appropriateness and clarity scored well (appropriateness reached 4.71 on a 5-point scale for job loss and difficult decisions), while naturalness and human-likeness lagged (best theme 3.88) and visualisation of the process received modest scores. In the Armando variant, the paper's claim is that placing RAG enrichment immediately before final synthesis—rather than before the multi-agent debate—preserves factual accuracy while keeping the emotional tone calibrated.
Load-bearing premise
The load-bearing premise is that prompting one shared text model with five different emotional personas produces five genuinely distinct emotional perspectives, so the debate and vote improve the answer rather than just adding length and visible labels.
Editorial extensions
If this is right
- If the architecture works as claimed, emotional range becomes a configuration choice: the same pipeline can be re-prompted for Plutchik's wheel, Panksepp's affective systems, or the circumplex model without retraining.
- Logs of the per-agent dialogue, votes, and justifications make the chatbot's reasoning visible and downloadable, giving users an inspectability that conventional single-model chatbots do not offer.
- In emergency chatbots, grounding the final synthesis with retrieved official documents—rather than grounding the debate—is claimed to prevent factual dilution, so RAG placement is part of the design, not an afterthought.
- The user-test pattern predicts that the approach will perform best in goal-focused scenarios and will need separate tuning for interpersonal-loss scenarios, where empathy and naturalness scores were lowest.
- Abliterated models, which bypass safety-alignment filters, were empirically chosen because they produce more genuinely emotion-consistent outputs; if this holds, safety-aligned models may be too inhibited for faithful emotion simulation.
Reading between the lines
- Editorial inference: because all five agents share the same base LLM and differ only in prompt, the reported benefits could come from ensemble aggregation, response length, or the visible emotion labels rather than from genuine emotional differentiation; this is precisely the comparison the paper lists as future work.
- Editorial inference: the voting protocol is structurally similar to social-choice aggregation in multi-agent LLM ensembles, so a natural ablation is to run the same debate with five neutral, non-emotional personas; if ratings do not drop, the emotion framing is not the active ingredient.
- Editorial inference: the RAG-placement result suggests a transferable design principle—in retrieval-augmented empathetic dialogue, ground the final answer, not the deliberation, because early factual context can be attenuated by multi-round debate; this could be tested in non-emergency counselling domains.
- Editorial inference: user-assigned emotion labels in open responses (Joy in anxiety contexts, Fear in job loss) may reflect the system's reassuring tone rather than the actual winning agent's perspective; hiding the labels in the output would reveal whether perceived emotional alignment is driven by content or by labelling.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes Project Riley, a conversational AI architecture that models five emotions (Joy, Sadness, Fear, Anger, Disgust) as separate agents, each implemented as a distinct persona prompt over the same underlying text LLM. The agents generate responses, engage in multi-round discussion, vote through a separate reasoning model, and a final reasoning stage synthesizes the output. A multimodal variant includes a vision LLM for image input. The paper also presents Armando, a RAG-based emergency-response variant that grounds answers in retrieved documents and omits the thought display for speed. Evaluation consists of a 17-participant user study with Likert-scale questionnaires across three dimensions (Emotional Appropriateness, Clarity and Utility, Naturalness and Human-likeness) and an anecdotal comparison of three prompts in Armando with and without RAG. The paper claims 'strong performance' in structured scenarios, particularly for emotional alignment and communicative clarity.
Significance. If the claimed effects were rigorously established, the architecture would offer a practical method for injecting affective reasoning into conversational agents and a novel application of multi-agent debate with voting. The paper's strengths include a clearly described pipeline, a functional local prototype, explicit logging, and an honest acknowledgment in Section 7 that the core premise—that five distinct emotional agents produce meaningfully different reasoning—has not been empirically tested. The RAG variant is a sensible application with a plausible design. However, the significance is currently limited by the preliminary evaluation: no control conditions, no statistical testing, and no quantitative comparison of the Armando RAG system. The architecture is a useful system description, but the evidence does not support the strength of the paper's central empirical claims.
major comments (1)
- [Section 3 (Proposed Architecture) and Abstract] The paper repeatedly describes 'five distinct emotional agents' and claims to be the 'first framework that leverages generative AI to orchestrate structured affective reasoning through independent emotional agents'. Since all five text agents are the same base model with different system prompts, they are not independent in the usual sense (no separate weights, no fine-tuning). The distinction may still be meaningful from a prompting perspective, but the current evidence does not show that the agents produce genuinely differentiated emotional reasoning. The novelty claim should be tempered or supported by an analysis showing that the five agents' outputs are indeed distinct and that the distinction matters for the final answer.
minor comments (5)
- [Section 5, Table 2 and Table 3] The open-ended question 'Which emotion do you predominantly identify in the final response?' is analyzed as frequency counts in Table 3, but no coding scheme, inter-rater reliability, or procedure for handling free-text responses is described; this limits the reproducibility of that analysis.
- [Section 6, title] The heading 'Armando: A emergency response chatbot' has a grammatical error; it should read 'An emergency response chatbot'.
- [References] Several references are listed as 'Preprint' without arXiv identifiers, DOIs, or access dates (e.g., [6], [7], [8], [11], [13], [16]); this hinders verification of the related-work claims.
- [Section 3 (Model names)] The model names are typeset awkwardly with spaces inserted (e.g., 'huihui_ai/llama3.2-abliterate:3b' appears as 'h u i h u i _ a i / l l a m a 3 . 2 - a b l i t e r a t e : 3 b' and 'mxbai-embed-large' as 'm x b a i - e m b e d - l a r g e'); this should be fixed for readability.
- [Section 4 (Prototype)] The paper states that abliterated models 'consistently produced more genuine and emotionally resonant outputs' based on 'empirical testing', but no details of this testing are provided; either add a reference or describe the informal evaluation.
Circularity Check
No derivation-level circularity; one self-referential evaluation metric for emotional alignment.
-
self definitional
[Section 5 (Prototype Evaluation, Table 2 and Discussion); Section 4 (Final Synthesis)]
"The final output is clearly segmented into REASONING (analytical assessment), THOUGHTS (representing Riley’s internal cognitive processes), and FINAL ANSWER ... At the end of the process, all interactions, dialogues, discussions, and relevant information should be made available to the user. ... "Do you believe the answer(s) of the emotion(s) with the most votes were the most appropriate?" ... the alignment of system-selected emotions with user expectations was strongest in Family or Intergenerational Conflicts (4.35)."
The paper's evidence for 'emotional alignment' is the user rating of the question 'Do you believe the answer(s) of the emotion(s) with the most votes were the most appropriate?'. The set of emotions 'with the most votes' is an output of the system's own Voting and Analysis phase, and the final response plus the reasoning/voting process is shown to the user before they answer. The reported 'alignment of system-selected emotions with user expectations' therefore measures how often participants ratify the system's self-announced winner, not whether the response is emotionally appropriate by an independent standard.
full rationale
This paper contains no mathematical derivation, no fitted parameters, and no load-bearing self-citations; all cited prior work is external, and no author-imposed uniqueness theorem is invoked. The Armando RAG comparison is a direct retrieval demonstration rather than a prediction, since the RAG variant is defined as retrieving the document that contains the requested facts. The central architecture claims are supported by a deployed prototype and logged interactions, so most circularity patterns do not apply. The one genuine issue is the self-referential measurement of 'emotional alignment': the questionnaire asks participants to approve the emotion(s) that received the most votes after the system has displayed its emotional reasoning and voting process, so high agreement partly reflects endorsement of the system's own label. This is a measurement circularity affecting one evaluation dimension, not a reduction of the whole derivation to its inputs. Section 7 also concedes that whether the five emotional agents produce meaningfully different responses compared to a shared emotion-conditioned model is still future work, which is a validity limitation rather than circularity. Overall, the paper is not fundamentally circular, but this one alignment metric warrants a low-moderate circularity score.
Assumptions & free parameters
free parameters (3)
- Emotion set (Anger, Joy, Sadness, Fear, Disgust) =
5 emotions from Inside Out
- Multi-round dialogue depth =
Rounds 0-3 (initial plus three discussion rounds)
- Abliterated model variants =
llama3.2-abliterate:3b; deepseek-r1-abliterated:8b
assumptions (4)
- domain assumption Ekman's basic emotions are a valid and sufficient foundation for simulating emotionally appropriate responses.
- domain assumption Prompts can make a shared text LLM behave as authentically distinct emotional agents.
- domain assumption Abliterated models are more emotionally expressive without unacceptable safety costs.
- domain assumption User Likert self-reports on five selected themes are a valid proxy for emotional appropriateness.
Cite this review
Pith. "Pith review of Project Riley: Multimodal Multi-Agent LLM Collaboration with Emotional Reasoning and Voting." pith.science (2026). https://pith.science/paper/G2KKFWC3
@misc{pith2026250520521,
author = {Pith},
title = {Pith review of: Project Riley: Multimodal Multi-Agent LLM Collaboration with Emotional Reasoning and Voting},
year = {2026},
howpublished = {\url{https://pith.science/paper/G2KKFWC3}},
note = {Machine review of arXiv:2505.20521}
}
read the original abstract
This paper presents Project Riley, a novel multimodal and multi-model conversational AI architecture oriented towards the simulation of reasoning influenced by emotional states. Drawing inspiration from Pixar's Inside Out, the system comprises five distinct emotional agents - Joy, Sadness, Fear, Anger, and Disgust - that engage in structured multi-round dialogues to generate, criticise, and iteratively refine responses. A final reasoning mechanism synthesises the contributions of these agents into a coherent output that either reflects the dominant emotion or integrates multiple perspectives. The architecture incorporates both textual and visual large language models (LLMs), alongside advanced reasoning and self-refinement processes. A functional prototype was deployed locally in an offline environment, optimised for emotional expressiveness and computational efficiency. From this initial prototype, another one emerged, called Armando, which was developed for use in emergency contexts, delivering emotionally calibrated and factually accurate information through the integration of Retrieval-Augmented Generation (RAG) and cumulative context tracking. The Project Riley prototype was evaluated through user testing, in which participants interacted with the chatbot and completed a structured questionnaire assessing three dimensions: Emotional Appropriateness, Clarity and Utility, and Naturalness and Human-likeness. The results indicate strong performance in structured scenarios, particularly with respect to emotional alignment and communicative clarity.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 2 Pith papers
-
MASCOT: Towards Multi-Agent Socio-Collaborative Companion Systems
MASCOT combines per-agent RLAIF persona training with a meta-agent director to improve role consistency and reduce redundancy in multi-agent emotional-support dialogues.
-
Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectives
A survey proposes a macro-meso-micro value framework for agentic AI alignment and maps applications, methods, and benchmarks onto it.
Reference graph
Works this paper leans on
- [1]
- [2]
-
[3]
Psychologists write about ‘Inside Out’ experience, July 2015
Anne Brice. Psychologists write about ‘Inside Out’ experience, July 2015
work page 2015
-
[4]
What ’Inside Out 2’ got right about anxiety, per a psychologist : Short Wave, June 2024
Regina Barber. What ’Inside Out 2’ got right about anxiety, per a psychologist : Short Wave, June 2024
work page 2024
-
[5]
Jeremiah Ratican and James Hutson. The six emotional dimension (6de) model: A multidimensional approach to analyzing human emotions and unlocking the potential of emotionally intelligent artificial intelligence (ai) via large language models (llm).DS Journal of Artificial Intelligence and Robotics, 1(1):44–51, September 2023
work page 2023
-
[6]
Nadine: An llm-driven intelligent social robot with affective capabilities and human-like memory
Hangyeol Kang, Maher Ben Moussa, and Nadia Magnenat- Thalmann. Nadine: An llm-driven intelligent social robot with affective capabilities and human-like memory. Preprint, 2024
work page 2024
-
[7]
Eq-negotiator: An emotion- reasoning llm agent in credit dialogues, 2025
Yuhan Liu and Yunbo Long. Eq-negotiator: An emotion- reasoning llm agent in credit dialogues, 2025. , Preprint
work page 2025
-
[8]
Abdur Rasool, Muhammad Irfan Shahzad, Hafsa Aslam, Vincent Chan, and Muhammad Ali Arshad. Emotion-aware embedding fusion in llms (flan-t5, llama 2, deepseek-r1, and chatgpt 4) for intelligent response generation. October
Show all 33 references
-
[9]
Speak from heart: An emotion-guided llm-based multimodal method for emotional dialogue generation
Chenxiao Liu, Zheyong Xie, Sirui Zhao, Jin Zhou, Tong Xu, Minglei Li, and Enhong Chen. Speak from heart: An emotion-guided llm-based multimodal method for emotional dialogue generation. InProceedings of the 2024 International Conference on Multimedia Retrieval, ICMR ’24, pages...
2024
-
[10]
Reconcile: Round-table conference improves reasoning via consensus among diverse llms
Justin Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse llms. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7066–7085. Associatio...
2024
-
[11]
Sumeet Ramesh Motwani, Chandler Smith, Rocktim Jyoti Das, Rafael Rafailov, Ivan Laptev, Philip H. S. Torr, Fabio Pizzati, Ronald Clark, and Christian Schroeder de Witt. Malt: Improving reasoning with multi-agent llm training. December 2024. , Preprint
2024
-
[12]
Enhancing emotions in positive way: Llm-based ai using cognitive behavioral therapy for emotional support
Jiseon Yang, Dahee Lim, Semin Lee, Seoyoung Lee, and Uran Oh. Enhancing emotions in positive way: Llm-based ai using cognitive behavioral therapy for emotional support. International Journal of Advanced Smart Convergence, 14(1):247–256, March 2025
2025
-
[13]
Exploring emotion-sensitive llm- based conversational ai
Antonin Brun, Ruying Liu, Aryan Shukla, Frances Watson, and Jonathan Gratch. Exploring emotion-sensitive llm- based conversational ai. February 2025. , Preprint
2025
-
[14]
An electoral approach to diversify llm-based multi-agent collective decision-making
Xiutian Zhao, Ke Wang, and Wei Peng. An electoral approach to diversify llm-based multi-agent collective decision-making. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2712–2727. Association for Computational Linguistics, 2024
2024
-
[15]
Minimizing hallucinations and communication costs: Adversarial debate and voting mechanisms in llm-based multi-agents.Applied Sciences, 15(7):3676, March 2025
Yi Yang, Yitong Ma, Hao Feng, Yiming Cheng, and Zhu Han. Minimizing hallucinations and communication costs: Adversarial debate and voting mechanisms in llm-based multi-agents.Applied Sciences, 15(7):3676, March 2025
2025
-
[16]
Towards reasoning in large language models via multi-agent peer review collaboration
Zhenran Xu, Senbao Shi, Baotian Hu, Jindi Yu, Dongfang Li, Min Zhang, and Yuxiang Wu. Towards reasoning in large language models via multi-agent peer review collaboration. November 2023. , Preprint
2023
-
[17]
Paul Ekman and Wallace V. Friesen. Constants across cultures in the face and emotion.Journal of Personality and Social Psychology, 17(2):124–129, 1971
1971
-
[18]
Universal Emotions
Paul Ekman. Universal Emotions
-
[19]
Inside out: Behind-the-scenes science with dacher keltner, phd.Eye on Psi Chi Magazine, 20(3):20– 23, 2016
Bradley Cannon. Inside out: Behind-the-scenes science with dacher keltner, phd.Eye on Psi Chi Magazine, 20(3):20– 23, 2016
2016
-
[20]
How Pixar Picked the 5 Core Emotions of Inside Out’s Star.Wired, 2015
Caitlin Roper. How Pixar Picked the 5 Core Emotions of Inside Out’s Star.Wired, 2015
2015
-
[21]
A psychoevolutionary theory of emotions
Robert Plutchik. A psychoevolutionary theory of emotions. Social Science Information, 21(4–5):529–553, July 1982
1982
-
[22]
Izard.Human Emotions
Carroll E. Izard.Human Emotions. Springer US, 1977
1977
-
[23]
Toward a general psychobiological theory of emotions.Behavioral and Brain Sciences, 5(3):407–422, September 1982
Jaak Panksepp. Toward a general psychobiological theory of emotions.Behavioral and Brain Sciences, 5(3):407–422, September 1982
1982
-
[24]
James A. Russell. A circumplex model of affect.Journal of Personality and Social Psychology, 39(6):1161–1178, December 1980
1980
-
[25]
Russell.An Approach to Environmental Psychology
Albert Mehrabian and James A. Russell.An Approach to Environmental Psychology. The MIT Press, 1980
1980
-
[26]
ollama, April 2025
Ollama. ollama, April 2025. original-date: 2023-06- 26T19:39:32Z
2025
-
[27]
Uncensor any llm with abliteration, June 2024
Maxime Labonne. Uncensor any llm with abliteration, June 2024
2024
-
[28]
huihui ai/llama3.2-abliterate
Meta and huhui ai. huihui ai/llama3.2-abliterate
-
[30]
huihui ai/deepseek-r1- abliterated/model
Deepseek and huihui ai. huihui ai/deepseek-r1- abliterated/model. 12 Ortigoso et al
-
[31]
Not ´ ıcias falsas sobre apag˜ ao geram pˆ anico
SAPO24. Not ´ ıcias falsas sobre apag˜ ao geram pˆ anico. N˜ ao, n˜ ao foi um hidroavi˜ ao e ainda n˜ ao se sabe a causa, April 2025. Ana Rita Ortigosowas born in Portugal, in 2002. She received the B.Sc. degree in Computer Engineering from the Polytechnic University of Leiria...
2025
-
[1999]
Software-Defined Things in Support of the Elderly
He received the B.Sc. degree in Computer Engineering, with a specialisation in Information Technology in Polytechnic University of Leiria, Portugal. He is currently pursuing, in the same school, the M.Sc. degree in Cybersecurity and Informatics. Since 2023, he has been a Resea...
2023
-
[2004]
He is also a member of the Computer Science and Communications Research Centre (CIIC) at the same institution
Since 2004, he has been with the School of Technology and Management (ESTG) at the Polytechnic Institute of Leiria, where he is currently a Coordinator Professor in the Department of Computer Science Engineering. He is also a member of the Computer Science and Communications R...
2004
-
[2023]
degree in Cybersecurity and Computer Forensics at the same institution
She is currently pursuing the M.Sc. degree in Cybersecurity and Computer Forensics at the same institution. Since 2023, she has been a research fellow at Computer Science and Communication Research Centre (CIIC), Polytechnic University of Leiria. Her research interests include...
2023
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.