Who Plays Which Role When? Communication Role Dynamics for Peer Recognition and Team Performance Prediction
Pith reviewed 2026-06-30 00:55 UTC · model grok-4.3
The pith
Eight communication roles drawn from education literature predict peer recognition in student teams and performance gains in deliberation data better than lexical or LLM baselines.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Labeling messages with the eight education-derived communication roles allows the authors to describe how role usage evolves across project stages and to build predictors of peer recognition that surpass lexical, conversational, and direct LLM-prompting baselines; the identical role constructs further improve prediction of post-deliberation performance gains on the DeliData corpus.
What carries the argument
The eight-role taxonomy of communication behaviors, which assigns each message to one of eight categories grounded in prior education research on group interaction.
If this is right
- Students enact a wider variety of roles as the project advances.
- Different roles reach peak frequency at distinct phases of the team lifecycle.
- Role-based features improve accuracy when predicting which students receive peer recognition.
- The same role labels raise accuracy when predicting whether deliberation improves team performance on the DeliData corpus.
Where Pith is reading between the lines
- Interventions that prompt underused roles at specific project stages could raise recognition equity.
- The taxonomy might serve as an early-warning signal for teams that will underperform after discussion.
- Similar role tracking could be tested on professional collaboration platforms without new category invention.
Load-bearing premise
The eight roles identified in education literature capture the behaviors that matter for peer recognition and team performance in both the Slack student teams and the DeliData deliberation setting.
What would settle it
A new set of expert annotations on the same Slack corpus or DeliData messages that shows the eight roles cannot be applied reliably, or a prediction model using only lexical and conversational features that matches or exceeds the role-based accuracy on held-out teams.
Figures
read the original abstract
Team roles offer an interpretable lens on collaboration, yet computational studies of roles often rely on domain-specific personas or data-driven clustering rather than theory-grounded taxonomies. We operationalize a taxonomy of eight communication roles grounded in education literature and annotate a corpus of 6,307 Slack messages from 55 students across 18 teams in a semester-long computer science course project. We evaluate whether LLMs can approximate expert labels, enabling scalable, taxonomy-driven role annotation. Using these role labels, we characterize role dynamics over teams' lifecycles, finding that different roles peak at different moments and that students enact a more diverse set of roles as projects progress. To evaluate the utility of our role constructs, we use them to predict peer recognition, outperforming lexical, conversational, and LLM-prompting baselines. To assess generalizability beyond the educational context, we apply the same role constructs to a public dataset (DeliData) to predict team performance improvement after deliberation, again exceeding prior performance.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper operationalizes an eight-role communication taxonomy from education literature, annotates 6,307 Slack messages from 55 CS students in 18 teams, evaluates LLM approximation to expert labels, characterizes role dynamics over project lifecycles (diversity increasing over time, roles peaking at different stages), and deploys the labels to predict peer recognition (outperforming lexical, conversational, and LLM baselines) and, on the public DeliData corpus, to predict post-deliberation team performance improvement (again exceeding prior results).
Significance. If the taxonomy applicability, annotation reliability, and statistical outperformance claims hold after verification, the work supplies a theory-grounded, interpretable alternative to purely data-driven role clustering for collaboration analysis; explicit use of the public DeliData dataset for cross-domain testing is a clear strength that supports the generalizability argument.
major comments (2)
- Abstract and §4 (results): the claims of outperformance on peer recognition and DeliData performance prediction are presented without statistical details, baseline definitions, error bars, or inter-annotator agreement metrics for the 6,307-message annotation; these omissions are load-bearing because they prevent verification that the reported gains are reliable and not artifacts of the chosen evaluation protocol.
- §3 (taxonomy operationalization): no evidence is supplied on how the eight education-literature roles were adapted or validated for Slack messages in CS project teams or for the DeliData deliberation corpus, nor on inter-annotator agreement for the new annotations; because the central claims rest on the taxonomy being relevant and sufficient in these domains, this gap directly affects the soundness of the dynamics characterization and prediction results.
minor comments (2)
- Notation for role labels and dynamics metrics should be defined consistently in the main text rather than only in supplementary material.
- Figure captions for lifecycle plots should explicitly state the time normalization used (e.g., week number or project phase).
Simulated Author's Rebuttal
We thank the referee for the constructive feedback on the need for greater statistical rigor and transparency in taxonomy operationalization. We will revise the manuscript accordingly to strengthen verifiability while preserving the core contributions.
read point-by-point responses
-
Referee: Abstract and §4 (results): the claims of outperformance on peer recognition and DeliData performance prediction are presented without statistical details, baseline definitions, error bars, or inter-annotator agreement metrics for the 6,307-message annotation; these omissions are load-bearing because they prevent verification that the reported gains are reliable and not artifacts of the chosen evaluation protocol.
Authors: We agree that the current version omits key statistical details. The revised manuscript will add: (i) explicit definitions and implementations of all baselines (lexical, conversational, and LLM-prompting), (ii) inter-annotator agreement metrics (e.g., Cohen's kappa or Fleiss' kappa) for the expert annotations on the 6,307 messages, (iii) error bars or confidence intervals on all performance metrics, and (iv) statistical significance tests (e.g., paired t-tests or McNemar's test) comparing our role-based predictors against baselines. These additions will be placed in §4 and referenced in the abstract. revision: yes
-
Referee: §3 (taxonomy operationalization): no evidence is supplied on how the eight education-literature roles were adapted or validated for Slack messages in CS project teams or for the DeliData deliberation corpus, nor on inter-annotator agreement for the new annotations; because the central claims rest on the taxonomy being relevant and sufficient in these domains, this gap directly affects the soundness of the dynamics characterization and prediction results.
Authors: We acknowledge the gap in reporting. The revision will expand §3 with: (i) a detailed description of how each of the eight roles was mapped and adapted from the education literature to Slack project communication and to the DeliData deliberation setting (including example message annotations), (ii) any pilot studies or expert validation steps used to confirm applicability, and (iii) the inter-annotator agreement statistics for the expert-labeled subset. If the original annotation process lacked multiple annotators, we will note this limitation and, where feasible, report agreement on a re-annotated sample. revision: yes
Circularity Check
No circularity: taxonomy and predictions remain independent
full rationale
The paper grounds its eight-role taxonomy in external education literature, performs separate annotation of the Slack corpus, and applies the resulting labels to predict distinct outcomes (peer recognition; DeliData performance improvement). No equations, fitted parameters, or self-citations are shown to reduce any prediction to its own inputs by construction. The derivation chain from taxonomy application to outcome prediction is self-contained against external benchmarks and does not exhibit self-definitional, fitted-input, or self-citation circularity.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption The eight-role taxonomy from education literature applies directly to computer science student team communication on Slack.
Reference graph
Works this paper leans on
-
[1]
Behavioral Sciences , year=
Team Role Adoption and Distribution in Engineering Project Meetings , author=. Behavioral Sciences , year=
-
[2]
Behavioral Sciences , year=
Interaction Diagrams: Development of a Method for Observing Group Interactions , author=. Behavioral Sciences , year=
-
[3]
You Are What You Say: Using Meeting Participants' Speech to Detect their Roles and Expertise
Banerjee, Satanjeev and Rudnicky, Alexander. You Are What You Say: Using Meeting Participants' Speech to Detect their Roles and Expertise. Proceedings of the Analyzing Conversations in Text and Speech. 2006
2006
-
[4]
Weakly Supervised Role Identification in Teamwork Interactions
Yang, Diyi and Wen, Miaomiao and Ros \'e , Carolyn. Weakly Supervised Role Identification in Teamwork Interactions. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2015. doi:10.3115/v1/P15-1161
-
[5]
Cao, Hancheng and Yang, Vivian and Chen, Victor and Lee, Yu Jin and Stone, Lydia and Diarrassouba, N'godjigui Junior and Whiting, Mark E. and Bernstein, Michael S. , title =. Proc. ACM Hum.-Comput. Interact. , month = jan, articleno =. 2021 , issue_date =. doi:10.1145/3432929 , abstract =
-
[6]
Behavioral Sciences , author =
Communication. Behavioral Sciences , author =. 2020 , pmid =. doi:10.3390/bs10070111 , abstract =
-
[7]
Katz, Andrew and Wei, Siqing and Nanda, Gaurav and Brinton, Christopher and Ohland, Matthew , month = may, year =. Exploring the. doi:10.48550/arXiv.2305.11882 , abstract =
-
[8]
Lee, Junyoung and Koh, Elizabeth , editor =. Teamwork. Artificial. 2023 , keywords =. doi:10.1007/978-3-031-36336-8_39 , abstract =
-
[9]
, editor =
Reitter, David and Moore, Johanna D. , editor =. Predicting. Proceedings of the 45th. 2007 , pages =
2007
-
[10]
Enayet, Ayesha and Sukthankar, Gita , editor =. Improving the. Proceedings of the 17th. 2023 , pages =. doi:10.18653/v1/2023.eacl-main.258 , abstract =
-
[11]
Zhang, Justine and Chang, Jonathan and Danescu-Niculescu-Mizil, Cristian and Dixon, Lucas and Hua, Yiqing and Taraborelli, Dario and Thain, Nithum , editor =. Conversations. Proceedings of the 56th. 2018 , pages =. doi:10.18653/v1/P18-1125 , abstract =
-
[12]
Rong-Chang Chen and Shih-Ying Chen and Jyun-You Fan and Yen-Ting Chen , keywords =. Grouping Partners for Cooperative Learning Using Genetic Algorithm and Social Network Analysis , journal =. 2012 , note =. doi:https://doi.org/10.1016/j.proeng.2012.01.589 , url =
-
[13]
Applied Sciences , VOLUME =
Kim, Woori and Yoo, Yongseok , TITLE =. Applied Sciences , VOLUME =. 2022 , NUMBER =
2022
-
[14]
Shannon L. Marlow and Christina N. Lacerenza and Jensine Paoletti and C. Shawn Burke and Eduardo Salas , keywords =. Does team communication represent a one-size-fits-all approach?: A meta-analysis of team communication and performance , journal =. 2018 , issn =. doi:https://doi.org/10.1016/j.obhdp.2017.08.001 , url =
-
[15]
Hundhausen, Christopher and Conrad, Phill and Adesope, Olusola and Tariq, Ahsun , title =. ACM Trans. Comput. Educ. , month = jul, articleno =. 2023 , issue_date =. doi:10.1145/3593592 , abstract =
-
[16]
Proceedings of the 54th ACM Technical Symposium on Computer Science Education V
Hundhausen, Christopher and Conrad, Phill and Adesope, Olusola and Tariq, Ahsun and Sbai, Samir and Lu, Andrew , title =. Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 , pages =. 2023 , isbn =. doi:10.1145/3545945.3569790 , abstract =
-
[17]
Recognizing Multi-Party Epistemic Dialogue Acts During Collaborative Game-Based Learning Using Large Language Models , volume =
Acosta, Halim and Lee, Seung and Bae, Haesol and Feng, Chen and Rowe, Jonathan and Glazewski, Krista and Hmelo-Silver, Cindy and Mott, Bradford and Lester, James , year =. Recognizing Multi-Party Epistemic Dialogue Acts During Collaborative Game-Based Learning Using Large Language Models , volume =. International Journal of Artificial Intelligence in Educ...
-
[18]
Katuka, Gloria and Bex, Richard and Celepkolu, Mehmet and Boyer, Kristy and Wiebe, Eric and Mott, Bradford and Lester, James , year =
-
[19]
Kumaran, Vikram and Rowe, Jonathan and Mott, Bradford and Chaturvedi, Snigdha and Lester, James. Improving Classroom Dialogue Act Recognition from Limited Labeled Data with Self-Supervised Contrastive Learning Classifiers. Findings of the Association for Computational Linguistics: ACL 2023. 2023. doi:10.18653/v1/2023.findings-acl.698
-
[20]
Mixed Membership M arkov Models for Unsupervised Conversation Modeling
Paul, Michael J. Mixed Membership M arkov Models for Unsupervised Conversation Modeling. Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. 2012
2012
-
[21]
Annual Meeting of the Association for Computational Linguistics , year=
Learning Latent Personas of Film Characters , author=. Annual Meeting of the Association for Computational Linguistics , year=
-
[22]
Journal of Social Issues , author =
Functional roles of group members , volume =. Journal of Social Issues , author =. 1948 , note =. doi:10.1111/j.1540-4560.1948.tb01783.x , abstract =
-
[23]
Human Resource Management International Digest , author =
Management. Human Resource Management International Digest , author =. 2011 , file =. doi:10.1108/hrmid.2011.04419cae.002 , number =
-
[24]
Miranda. Thematic review of approaches to design group learning activities in higher education: The development of a comprehensive framework , journal =. 2016 , issn =. doi:https://doi.org/10.1016/j.edurev.2016.01.001 , url =
-
[25]
Collaborative Learning in Engineering Education
Emma Mercier and Goldstein, Molly H and Preethi Baligar and Jephthah Rajarathinam , Robin. Collaborative Learning in Engineering Education. International Handbook of Engineering Education Research. 2023. doi:10.4324/9781003287483-23
-
[26]
Advances in Engineering Education , author =
Design and. Advances in Engineering Education , author =. 2010 , note =
2010
-
[27]
Journal of Marketing Education , author =
Assessing. Journal of Marketing Education , author =. 2014 , keywords =. doi:10.1177/0273475313499023 , abstract =
-
[28]
John E. Mathieu and Scott I. Tannenbaum and Michael R. Kukenberger and Jamie S. Donsbach and George M. Alliger , title =. Group & Organization Management , volume =. 2015 , doi =. https://doi.org/10.1177/1059601114562000 , abstract =
-
[29]
and Mai, Minh Chau and Hancock, Jeffrey T
Jia, Chenyan and Lam, Michelle S. and Mai, Minh Chau and Hancock, Jeffrey T. and Bernstein, Michael S. , title =. Proc. ACM Hum.-Comput. Interact. , month = apr, articleno =. 2024 , issue_date =. doi:10.1145/3641002 , abstract =
-
[30]
Conversational Markers of Constructive Discussions
Niculae, Vlad and Danescu-Niculescu-Mizil, Cristian. Conversational Markers of Constructive Discussions. Proceedings of the 2016 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies. 2016. doi:10.18653/v1/N16-1070
-
[31]
Nguyen, Thien Hai and Shirai, Kiyoaki and Velcin, Julien , title =. Expert Syst. Appl. , month = dec, pages =. 2016 , issue_date =. doi:10.1016/j.eswa.2015.07.052 , abstract =
-
[32]
Assessing the Language of Chat for Teamwork Dialogue , urldate =
Antonette Shibani and Elizabeth Koh and Vivian Lai and Kyong Jin Shim , journal =. Assessing the Language of Chat for Teamwork Dialogue , urldate =
-
[33]
Detecting Disagreement in Conversations using Pseudo-Monologic Rhetorical Structure
Allen, Kelsey and Carenini, Giuseppe and Ng, Raymond. Detecting Disagreement in Conversations using Pseudo-Monologic Rhetorical Structure. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ( EMNLP ). 2014. doi:10.3115/v1/D14-1124
-
[34]
Roles and Success in W ikipedia Talk Pages: Identifying Latent Patterns of Behavior
Maki, Keith and Yoder, Michael and Jo, Yohan and Ros \'e , Carolyn. Roles and Success in W ikipedia Talk Pages: Identifying Latent Patterns of Behavior. Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2017
2017
-
[35]
Lu, Keming and Yu, Bowen and Zhou, Chang and Zhou, Jingren. Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024. doi:10.18653/v1/2024.acl-long.423
-
[36]
The Teams Corpus and Entrainment in Multi-Party Spoken Dialogues
Litman, Diane and Paletz, Susannah and Rahimi, Zahra and Allegretti, Stefani and Rice, Caitlin. The Teams Corpus and Entrainment in Multi-Party Spoken Dialogues. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2016. doi:10.18653/v1/D16-1149
-
[37]
LLM -Based Agent Society Investigation: Collaboration and Confrontation in Avalon Gameplay
Lan, Yihuai and Hu, Zhiqiang and Wang, Lei and Wang, Yang and Ye, Deheng and Zhao, Peilin and Lim, Ee-Peng and Xiong, Hui and Wang, Hao. LLM -Based Agent Society Investigation: Collaboration and Confrontation in Avalon Gameplay. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.emnlp-main.7
-
[38]
Advancing Collaborative Debates with Role Differentiation through Multi-Agent Reinforcement Learning
Li, Haoran and Su, Ziyi and Xue, Yun and Tian, Zhiliang and Song, Yiping and Huang, Minlie. Advancing Collaborative Debates with Role Differentiation through Multi-Agent Reinforcement Learning. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025. doi:10.18653/v1/2025.acl-long.1105
-
[39]
Proceedings of the ACM on Human-Computer Interaction , volume=
DeliData: A dataset for deliberation in multi-party problem solving , author=. Proceedings of the ACM on Human-Computer Interaction , volume=. 2023 , publisher=
2023
-
[40]
Li, Wenkai and Liu, Jiarui and Liu, Andy and Zhou, Xuhui and Diab, Mona T. and Sap, Maarten. BIG 5- CHAT : Shaping LLM Personalities Through Training on Human-Grounded Data. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025. doi:10.18653/v1/2025.acl-long.999
-
[41]
(2024) Chatdev: Communicative agents for software development
Qian, Chen and Liu, Wei and Liu, Hongzhang and Chen, Nuo and Dang, Yufan and Li, Jiahao and Yang, Cheng and Chen, Weize and Su, Yusheng and Cong, Xin and Xu, Juyuan and Li, Dahai and Liu, Zhiyuan and Sun, Maosong. C hat D ev: Communicative Agents for Software Development. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguist...
-
[42]
P. C. Wason , title =. Quarterly Journal of Experimental Psychology , volume =. 1968 , doi =
1968
-
[43]
A Century of Work Teams in the Journal of Applied Psychology , volume =
Mathieu, John and Hollenbeck, John and Knippenberg, Daan and Ilgen, Daniel , year =. A Century of Work Teams in the Journal of Applied Psychology , volume =. Journal of Applied Psychology , doi =
-
[44]
Modeling Functional Roles Dynamics in Small Group Interactions , year=
Dong, Wen and Lepri, Bruno and Pianesi, Fabio and Pentland, Alex , journal=. Modeling Functional Roles Dynamics in Small Group Interactions , year=
-
[45]
Social Network Analysis and Mining , year=
Improving team performance prediction in MMOGs with temporal communication networks , author=. Social Network Analysis and Mining , year=
-
[46]
Developing a Team Performance Prediction Model: A Rough Sets Approach
Omar, Mazni and Syed-Abdullah, Sharifah-Lailee and Mohd Hussin, Naimah. Developing a Team Performance Prediction Model: A Rough Sets Approach. Informatics Engineering and Information Science. 2011
2011
-
[47]
A. Aranzabal and E. Epelde and M. Artetxe , keywords =. Team formation on the basis of Belbin’s roles to enhance students’ performance in project based learning , journal =. 2022 , issn =. doi:https://doi.org/10.1016/j.ece.2021.09.001 , url =
-
[48]
, title =
Song, Yifan and Vithani, Ritika and Shi, Wenxuan Wendy and Bailey, Brian P. , title =. Proceedings of the 57th ACM Technical Symposium on Computer Science Education V.1 , pages =. 2026 , isbn =
2026
-
[49]
Jahanbakhsh, Farnaz and Fu, Wai-Tat and Karahalios, Karrie and Marinov, Darko and Bailey, Brian , title =. 2017 , isbn =. doi:10.1145/3025453.3026011 , booktitle =
-
[50]
Wegmann, Anna and Broek, Tijs A. Van Den and Nguyen, Dong. What ' s Mine becomes Yours: Defining, Annotating and Detecting Context-Dependent Paraphrases in News Interview Dialogs. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.emnlp-main.52
-
[51]
Human Communication Research , volume =
Krippendorff, Klaus , title =. Human Communication Research , volume =. 2004 , month =. doi:10.1111/j.1468-2958.2004.tb00738.x , url =
-
[52]
Writing Strategies for Science Communication: Data and Computational Analysis
August, Tal and Kim, Lauren and Reinecke, Katharina and Smith, Noah A. Writing Strategies for Science Communication: Data and Computational Analysis. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020. doi:10.18653/v1/2020.emnlp-main.429
-
[53]
Shanyun He and Xinyue Shi and Tae-Hee Choi and Junqing Zhai , keywords =. How do students'roles in collaborative learning affect collaborative problem-solving competency? A systematic review of research , journal =. 2023 , issn =. doi:https://doi.org/10.1016/j.tsc.2023.101423 , url =
-
[54]
Aho and Jeffrey D
Alfred V. Aho and Jeffrey D. Ullman , title =. 1972
1972
-
[55]
Publications Manual , year = "1983", publisher =
1983
-
[56]
Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243
-
[57]
Scalable training of
Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of
-
[58]
Dan Gusfield , title =. 1997
1997
-
[59]
Tetreault , title =
Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =
2015
-
[60]
A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =
Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.