REVIEW 5 major objections 7 minor 49 references
Physiologically-Informed Predictability of a Teammate's Future Actions Forecasts Team Performance
T0 review · 5 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A team-level predictability score—how well two teammates' physiology and behavior forecast the third's next control action—is positively correlated with team performance in a three-person virtual-reality spacecraft task.
desk verdict Interesting idea, solid VR task, but the predictability metric is not shown to be out-of-sample, so the main correlation claim outruns the evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a leave-one-out predictive model built on a multi-head attention transformer. For each ring event, the encoder receives one second of spacecraft trajectory and the behavioral and physiological signals of two co-pilots; cross-modal attention layers fuse those modalities, self-attention layers extract intra-modal patterns, and a decoder with masked attention generates the third co-pilot's controller action over the next 0.5 seconds. Predictability is defined as the Pearson correlation $r$ between the predicted and actual actions for each co-pilot, then averaged over the three co-pilots to form the team biomarker; speech data are excluded from the input so that the result is not driven by the one modality whose synchrony already correlates with performance.
What would settle it
The claim would be settled by re-estimating the team predictability scores with an explicit held-out protocol—for example, training on trials from two sessions and predicting the third session, or leaving entire teams out—and checking whether the per-person Pearson $r$ values and the team-level $\beta = 3.20$ correlation survive; if out-of-sample predictability drops to near zero, the biomarker reflects the model memorizing its training data rather than real predictability.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that a team-level predictability score—the average, across the three co-pilots, of the Pearson correlation between a co-pilot's true controller action and a transformer model's prediction of that action from the other two co-pilots' multi-modal physiology and behavior—predicts team performance. In the Apollo Distributed Control Task, teams with higher predictability passed more rings ($\beta = 3.20$, $P < 0.001$, across 10 teams), while inter-subject synchrony in pupil size, EEG, controller actions, and speech was mostly not significantly or only weakly related to performance. The paper interprets this as evidence that high-performing team members become mutually anticipatable, and that this anticipatability is visible in their physiology and behavior before the team outcome is known.
Load-bearing premise
The load-bearing premise is that the reported predictability scores measure genuine prediction of new data; the paper does not say how the transformer model's training and test data were separated, so the correlation with team performance is only trustworthy if the model generalizes beyond the examples it was fit on.
Editorial extensions
If this is right
- In this task, team performance is indexed by how forecastable each member's next controller action is from teammates' combined physiology and behavior, rather than by how synchronized teammates are.
- Synchrony metrics alone—EEG, pupil, controller-action, and speech-event inter-subject correlations—are not reliable performance biomarkers in this setting.
- The predictability score offers a continuous, quantitative outcome for team training, role assignment, or feedback systems in collaborative sensorimotor tasks.
- Because the main result is computed without speech input, the predictability-performance link is not explained by the one modality whose synchrony already predicted performance.
Reading between the lines
- A direct extension the paper does not pursue is an intervention test: if raising mutual predictability through practice, shared displays, or communication protocols also raises rings passed, that would turn the observed correlation into a causal handle on team performance.
- The same leave-one-out predictability metric could be ported to human-AI teams, treating an AI teammate's actions as the target to measure whether an artificial agent behaves in a way human teammates can anticipate.
- A modality-ablation study—removing EEG, pupil, or controller inputs one at a time—would reveal which signals carry the predictability effect, a question the paper leaves open.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports a virtual-reality triadic collaborative task (Apollo Distributed Control Task, ADCT) in which 17 teams of three performed three sessions each. The authors collected EEG, pupillometry, eye gaze, speech, and controller inputs and computed inter-subject correlations (ISC) across modalities, finding that speech synchrony is positively associated with team performance while physiological synchronies are not. The main contribution is a 'predictability' metric: a multi-head attention transformer is trained to predict one co-pilot's future controller action from the other two co-pilots' multimodal data and spacecraft trajectory, and the team-level predictability (average Pearson r between predicted and true actions) is reported to correlate positively with team performance (β=3.20, P<0.001). The paper argues that this predictability is a biomarker of team performance, challenging the emphasis on synchrony.
Significance. If the predictability metric were validated as an out-of-sample forecast, it would provide a novel, quantitative biomarker for team coordination and a template for multimodal team analysis; the ADCT dataset with synchronized multimodal recordings across repeated sessions is a valuable resource. The study also contributes a useful negative result on physiological synchrony. However, the central claim is not yet established because the paper does not show that the predictability scores are out-of-sample, does not compare against trajectory-only or self-history baselines, uses a small sample with unexplained exclusions, and reports the headline estimate without confidence intervals or a full model specification. These issues are load-bearing for the main conclusion.
major comments (5)
- [4.9–4.10] The model evaluation in Section 4.10 does not demonstrate that the reported Pearson r is based on out-of-sample predictions. Section 4.9 mentions only that a validation dataset was used to monitor loss during training, and Section 4.10 computes the correlation between concatenated target actions and model predictions without stating that the predicted epochs were held out from training. If the scored epochs are the same ones used for fitting, the predictability scores are inflated by memorization, and the team-level correlation in Fig. 3d (β=3.20, P<0.001 in Section 2.5) may reflect a training artifact rather than genuine predictability. Please specify the train/validation/test split, the number of epochs in each, and report the correlation on held-out data.
- [4.9–4.10] No baseline or ablation is reported for the forecasting model. The input combines spacecraft trajectory with the two co-pilots' multimodal data, but the paper never tests whether the physiological and behavioral inputs add predictive power beyond the trajectory alone, or whether the target's own recent actions suffice. Without such controls, the predictability metric could be driven by the deterministic relationship between ring position and required corrective action, which is available from the environment rather than from teammates. The claim that this is a 'physiologically-informed' biomarker requires showing that the multimodal teammate data improves prediction over trajectory-only and self-history baselines.
- [4.1, Fig. 3c–d] The predictability analysis is based on n=10 teams, yet the Methods section reports only one team excluded for incomplete sessions and lists modality-specific exclusions (pupil: 4 teams, EEG: 9 teams, speech: 10 teams). The paper does not explain how 17 usable teams reduce to 10 for the predictability analysis, nor whether these exclusions are related to team performance. If data-quality issues are more common in low- or high-performing teams, the correlation between predictability and performance could be biased. Please provide a participant-flow diagram showing the sample size for each analysis and a sensitivity analysis with alternative exclusion criteria.
- [2.5] The headline association (β=3.20, P<0.001) is reported without confidence intervals, standard errors, or the number of observations used in the GLMM. With only 10 teams, the apparent correlation in Fig. 3d may be sensitive to a few influential points. The paper should report the GLMM specification explicitly (fixed and random effects as used, not just the general form in Eq. 6), include confidence intervals, and show team-level scatters with per-session points to assess robustness. A nonparametric or robust correlation at the team level would also be helpful.
- [2.4] The rationale for excluding speech data from the model input is not convincing: the authors state that speech event synchrony significantly correlates with team performance and therefore they excluded speech to 'avoid potential bias.' Excluding a modality because it correlates with the outcome removes a potentially informative predictor and may itself bias the predictability metric. The supplementary comparison with speech included should be summarized in the main text, or the exclusion should be justified on other grounds (e.g., technical artifacts or preprocessing failures).
minor comments (7)
- [2.2] The repeated-measures ANOVA results report degrees of freedom such as F(2,32) for N=17 teams, but the error degrees of freedom are not consistently derived; please clarify the within-subject design and report effect sizes or at least mean squared errors.
- [4.9, Eq. (4)] The text states that 8-head attention is used, but Eq. (4) shows Concat(head1,...,head4); please correct the inconsistency.
- [4.9] Hyperparameters for training (loss function, optimizer, learning rate, batch size, number of epochs, early stopping criterion) are not reported; without these, the forecasting model is not reproducible.
- [4.10] The Pearson correlation is computed on target and predicted controller actions that are largely discrete ternary values (-1, 0, 1); a rank-based metric or classification accuracy would be more appropriate and less sensitive to the action baseline rate.
- [4.7, Eq. (1)] Equation (1) is typeset with garbled fragments ('qPn', '\sqrt{}'); please fix the formatting so the formula is readable.
- [2.1] The claim that 'the total number of rings passed by each team increased monotonically' is based on group means; individual teams may not all show monotonic improvement, so please soften to 'on average' and present individual trajectories.
- [2.4] The paper states that co-pilots make about 0.3 remote controller actions in the 0.5-second output window; report the exact distribution and how this threshold was chosen, since the rarity of actions affects the interpretability of the correlation.
Circularity Check
The predictability biomarker is partially circular: epochs are defined per ring-passing event, so high-performing teams contribute more training samples; with no reported out-of-sample split, the Fig. 3d correlation partly reflects in-sample fit to an outcome-selected epoch pool.
-
fitted input called prediction
[Section 2 (performance definition), Section 2.4, Sections 4.9–4.10, Section 4.11]
"Each experimental session included 45 trials, each consisting of 15 rings. Team performance was quantitatively evaluated by the team’s total number of ring obstacles successfully navigated. ... First, we epoched multi-modal physiological and behavioral data from 1.5 seconds before each ring-passing event. ... We evaluated the predictive model’s performance by computing the Pearson correlation coefficient r between the prediction and the target."
Because an epoch is defined only for a ring-passing event, the number of epochs a team contributes equals (up to trial completion) the team’s performance score—rings passed. The transformer in Section 4.9 is trained on this epoch pool, and Section 4.10 computes predictability by correlating the model’s predictions with the target actions on the same data, with no train/test split specified. Consequently, high-performing teams dominate the training set, and the team-level predictability score (Section 4.11) is partly a measure of how well the model fits the very teams that supplied more training epochs because they performed better. Regressing that score on performance (Fig. 3d, β=3.20) therefore re-discovers, in part, a training-set-weighting artifact rather than an independent forecast.
full rationale
No additional circular steps were found. The ISC analyses use standard inter-subject correlation and CorrCA; citations to prior Sajda-lab work are not load-bearing for the predictability claim. The predictability metric is not defined in terms of team performance, so the finding is not a tautology. However, the main result is partially circular under the procedure as reported: the epoch sampling is chosen by successful ring passage (the performance variable), and the model evaluation is not described as out-of-sample. This inflates the predictability-performance correlation by construction. If the authors can show a held-out (e.g., leave-one-team/session-out) evaluation with epochs balanced by outcome, the central claim would have independent content; as written, the derivation chain reduces in part to a fitted-input artifact.
Assumptions & free parameters
free parameters (5)
- Multi-head attention model weights =
not reported (trained on task data)
- Model hyperparameters (8 heads, dk=dv=dm/h=64) =
8, 64
- Epoch window boundaries (input 1s, output 0.5s before ring) =
1.0 s / 0.5 s
- Controller action discretization threshold =
±0.5
- GLMM fixed-effect coefficient for predictability =
beta = 3.20
assumptions (4)
- ad hoc to paper Predictability of a teammate's future controller actions from other teammates' multimodal data is a valid operationalization of team coordination.
- ad hoc to paper The transformer model captures cross-modal dependencies without target leakage from the held-out teammate.
- domain assumption Independent Component Analysis and manual channel rejection adequately clean EEG.
- standard math Pearson correlation is an appropriate accuracy metric for predicted controller actions.
Cite this review
Pith. "Pith review of Physiologically-Informed Predictability of a Teammate's Future Actions Forecasts Team Performance." pith.science (2026). https://pith.science/paper/LXBBD3OG
@misc{pith2026250115328,
author = {Pith},
title = {Pith review of: Physiologically-Informed Predictability of a Teammate's Future Actions Forecasts Team Performance},
year = {2026},
howpublished = {\url{https://pith.science/paper/LXBBD3OG}},
note = {Machine review of arXiv:2501.15328}
}
read the original abstract
In collaborative environments, a deep understanding of multi-human teaming dynamics is essential for optimizing performance. However, the relationship between individuals' behavioral and physiological markers and their combined influence on overall team performance remains poorly understood. To explore this, we designed a triadic human collaborative sensorimotor task in virtual reality (VR) and introduced a novel predictability metric to examine team dynamics and performance. Our findings reveal a strong connection between team performance and the predictability of a team member's future actions based on other team members' behavioral and physiological data. Contrary to conventional wisdom that high-performing teams are highly synchronized, our results suggest that physiological and behavioral synchronizations among team members have a limited correlation with team performance. These insights provide a new quantitative framework for understanding multi-human teaming, paving the way for deeper insights into team dynamics and performance.
Reference graph
Works this paper leans on
-
[1]
Starring Tom Hanks, Kevin Bacon, and Bill Paxton
(1995) Apollo 13. Starring Tom Hanks, Kevin Bacon, and Bill Paxton
work page 1995
-
[2]
Cognitive processing 16:325–332
Abney DH, Paxton A, Dale R, et al (2015) Movement dynamics reflect a func- tional role for weak coupling and role structure in dyadic problem solving. Cognitive processing 16:325–332
work page 2015
-
[3]
Small Group Research 44(6):680–711
Bradley BH, Baur JE, Banford CG, et al (2013) Team players and collective performance: How agreeableness affects team performance over time. Small Group Research 44(6):680–711
work page 2013
-
[4]
Journal of the American statistical Association 88(421):9–25
Breslow NE, Clayton DG (1993) Approximate inference in generalized linear mixed models. Journal of the American statistical Association 88(421):9–25
work page 1993
-
[5]
Chen B, Vondrick C, Lipson H (2021) Visual behavior modelling for robotic theory of mind. Scientific Reports 11(1):424
work page 2021
-
[6]
In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 879–888
Dabbish L, Kraut R, Patton J (2012) Communication and commitment in an online game team. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 879–888
work page 2012
-
[7]
Journal of biomedical informatics 96:103250
Dias RD, Zenati MA, Stevens R, et al (2019) Physiological synchronization and entropy as measures of team cognitive load. Journal of biomedical informatics 96:103250
work page 2019
-
[8]
Current biology 27(9):1375– 1380
Dikker S, Wan L, Davidesco I, et al (2017) Brain-to-brain synchrony tracks real- world dynamic group interactions in the classroom. Current biology 27(9):1375– 1380
2017
Show all 49 references
-
[9]
Dmochowski JP, Sajda P, Dias J, et al (2012) Correlated components of ongo- ing eeg point to emotionally laden attention–a possible marker of engagement? Frontiers in human neuroscience 6:112
2012
-
[10]
Frontiers in Human Neuroscience 14:332 17
Dunbar TA, Gorman JC (2020) Using communication to modulate neural synchronization in teams. Frontiers in Human Neuroscience 14:332 17
2020
-
[11]
Administrative science quarterly 44(2):350–383
Edmondson A (1999) Psychological safety and learning behavior in work teams. Administrative science quarterly 44(2):350–383
1999
-
[12]
Proceedings of the National Academy of Sciences of the United States of America 116(13):6482 – 6490
Faller J, Cummings J, Saproo S, et al (2019) Regulation of arousal via online neurofeedback improves human performance in a demanding sensory-motor task. Proceedings of the National Academy of Sciences of the United States of America 116(13):6482 – 6490. https://doi.org/10.107...
2019 doi
-
[13]
Scientific reports 10(1):8484
Gordon I, Gilboa A, Cohen S, et al (2020) Physiological and behavioral synchrony predict group cohesion and performance. Scientific reports 10(1):8484
2020
-
[14]
Frontiers in Neuroscience 7(267):1–13
Gramfort A, Luessi M, Larson E, et al (2013) MEG and EEG data analysis with MNE-Python. Frontiers in Neuroscience 7(267):1–13. https://doi.org/10.3389/ fnins.2013.00267
2013
-
[15]
In: International Conference on Augmented Reality, Virtual Reality and Computer Graphics, Springer, pp 22–33
Hansen A, Larsen KB, Nielsen HH, et al (2020) Asymmetrical multiplayer versus single player: Effects on game experience in a virtual reality edutainment game. In: International Conference on Augmented Reality, Virtual Reality and Computer Graphics, Springer, pp 22–33
2020
-
[16]
Computers in Human Behavior 145:107765
Harris-Watson AM, Larson LE, Lauharatanahirun N, et al (2023) Social per- ception in human-ai teams: Warmth and competence predict receptivity to ai teammates. Computers in Human Behavior 145:107765
2023
-
[17]
Personnel Psychology 56(3):633–669
Harrison DA, Mohammed S, McGrath JE, et al (2003) Time matters in team performance: Effects of member familiarity, entrainment, and task discontinuity on speed and quality. Personnel Psychology 56(3):633–669
2003
-
[18]
International Journal of Psychophysiology 40(3):221–232
Henning RA, Boucsein W, Gil MC (2001) Social–physiological compliance as a determinant of team performance. International Journal of Psychophysiology 40(3):221–232
2001
-
[19]
British Journal of Educational Technology 42(4):616–623
Jang Y, Ryu S (2011) Exploring game experiences and game leadership in massively multiplayer online role-playing games. British Journal of Educational Technology 42(4):616–623
2011
-
[20]
Frontiers in neuroinformatics 4:669
Kauppi JP, J¨ a¨ askel¨ ainen IP, Sams M, et al (2010) Inter-subject correlation of brain hemodynamic responses during watching a movie: localization in space and frequency. Frontiers in neuroinformatics 4:669
2010
-
[21]
Scientific Data 11(1):214
Keles U, Dubois J, Le KJ, et al (2024) Multimodal single-neuron, intracranial eeg, and fmri brain responses during movie watching in human patients. Scientific Data 11(1):214
2024
-
[22]
In: 2023 IEEE 33rd International Workshop on Machine Learning for Signal Processing (MLSP), IEEE, pp 1–6 18
Madsen AG, Lehn-Schiøler WT, J´ onsd´ ottir´A, et al (2023) Concept-based explain- ability for an eeg transformer model. In: 2023 IEEE 33rd International Workshop on Machine Learning for Signal Processing (MLSP), IEEE, pp 1–6 18
2023
-
[23]
PNAS nexus 1(1):pgac020
Madsen J, Parra LC (2022) Cognitive processing of a common stimulus synchro- nizes brains, hearts, and eyes. PNAS nexus 1(1):pgac020
2022
-
[24]
Advances in neural information processing systems 8
Makeig S, Bell A, Jung TP, et al (1995) Independent component analysis of electroencephalographic data. Advances in neural information processing systems 8
1995
-
[25]
Frontiers in public health 5:267
McCraty R (2017) New frontiers in heart rate variability and social coherence research: techniques, technologies, and implications for improving group dynamics and outcomes. Frontiers in public health 5:267
2017
-
[26]
IEEE Access 9:70242–70260
Metcalfe JS, Perelman BS, Boothe DL, et al (2021) Systemic oversimplification limits the potential for human-ai partnership. IEEE Access 9:70242–70260
2021
-
[27]
PloS one 15(10):e0241011
Moore SM, Geuss MN (2020) Familiarity with teammate’s attitudes improves team performance in virtual reality. PloS one 15(10):e0241011
2020
-
[28]
The Journal of the American Society of Anesthesiologists 106(5):907–915
Morgan PJ, Pittini R, Regehr G, et al (2007) Evaluating teamwork in a simulated obstetric environment. The Journal of the American Society of Anesthesiologists 106(5):907–915
2007
-
[29]
arXiv preprint arXiv:180108881
Parra LC, Haufe S, Dmochowski JP (2018) Correlated components analysis-extracting reliable dimensions in multivariate data. arXiv preprint arXiv:180108881
2018
-
[30]
Journal of Management 32(4):575–594
Pearsall MJ, Ellis AP (2006) The effects of critical team member assertiveness on team performance and satisfaction. Journal of Management 32(4):575–594
2006
-
[31]
In: 2013 IEEE 15th Conference on Business Informatics, IEEE, pp 55–62
Pobiedina N, Neidhardt J, Moreno MdCC, et al (2013) On successful team for- mation: Statistical analysis of a multiplayer online game. In: 2013 IEEE 15th Conference on Business Informatics, IEEE, pp 55–62
2013
-
[32]
Scientific reports 7(1):43916
Poulsen AT, Kamronn S, Dmochowski J, et al (2017) Eeg in the classroom: Synchronised neural recordings during video presentation. Scientific reports 7(1):43916
2017
-
[33]
Social cognitive and affective neuroscience 16(1- 2):43–57
Reinero DA, Dikker S, Van Bavel JJ (2021) Inter-brain synchrony in teams predicts collective performance. Social cognitive and affective neuroscience 16(1- 2):43–57
2021
-
[34]
Houston, we have a problem
Rerup C (2001) “Houston, we have a problem”: Anticipation and improvisation as sources of organizational resilience. Snider Entrepreneurial Center, Wharton School Philadelphia, PA
2001
-
[35]
PLoS computational biology 16(10):e1008228 19
Sainburg T, Thielk M, Gentner TQ (2020) Finding, visualizing, and quantify- ing latent structure across diverse animal vocal repertoires. PLoS computational biology 16(10):e1008228 19
2020
-
[36]
Royal Society open science 5(6):180329
Sapienza A, Zeng Y, Bessi A, et al (2018) Individual performance in team-based online games. Royal Society open science 5(6):180329
2018
-
[37]
In: 9th Python in Science Conference
Seabold S, Perktold J (2010) statsmodels: Econometric and statistical modeling with python. In: 9th Python in Science Conference
2010
-
[38]
9th Python in Science Conference
Skipper S, Josef P (2010) statsmodels: Econometric and statistical modeling with python. 9th Python in Science Conference
2010
-
[39]
Human brain mapping 41(4):1072–1083
ˇSpil´ akov´ a B, Shaw DJ, Czek´ oov´ a K, et al (2020) Getting into sync: Data-driven analyses reveal patterns of neural coupling that distinguish among different social exchanges. Human brain mapping 41(4):1072–1083
2020
-
[40]
Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care 5(1):50–54
Stevens R, Galloway T, Gorman J, et al (2016) Toward objective measures of team dynamics during healthcare simulation training. Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care 5(1):50–54. https://doi.org/10.1177/2327857916051010, URL h...
2016 doi
-
[41]
Neuroimage 152:425–436
Szymanski C, Pesquita A, Brennan AA, et al (2017) Teams on the same wave- length perform better: Inter-brain phase synchronization constitutes a neural substrate for social facilitation. Neuroimage 152:425–436
2017
-
[42]
International journal of production economics 111(1):42–55
Van der Vaart T, Van Donk DP (2008) A critical review of survey-based research in supply chain integration. International journal of production economics 111(1):42–55
2008
-
[43]
Presence 22(3):202– 215
Varlet M, Filippeschi A, Ben-Sadoun G, et al (2013) Virtual reality as a tool to learn interpersonal coordination: Example of team rowing. Presence 22(3):202– 215
2013
-
[44]
Advances in neural information processing systems 30
Vaswani A, Shazeer N, Parmar N, et al (2017) Attention is all you need. Advances in neural information processing systems 30
2017
-
[45]
Journal of Nonverbal Behavior 40:335–361
Vicaria IM, Dickens L (2016) Meta-analyses of the intra-and interpersonal out- comes of interpersonal coordination. Journal of Nonverbal Behavior 40:335–361
2016
-
[46]
IEEE Transactions on Visualization and Computer Graphics 27(5):2524–2534
Weissker T, Froehlich B (2021) Group navigation for guided tours in dis- tributed virtual environments. IEEE Transactions on Visualization and Computer Graphics 27(5):2524–2534
2021
-
[47]
Behavioral and brain functions 7:1–15
Winkler I, Haufe S, Tangermann M (2011) Automatic classification of artifactual ica-components for artifact removal in eeg signals. Behavioral and brain functions 7:1–15
2011
-
[48]
Scientific Reports 13(1):3591 20
Wohltjen S, Toth B, Boncz A, et al (2023) Synchrony to a beat predicts synchrony with other minds. Scientific Reports 13(1):3591 20
2023
-
[49]
Proceedings of the National Academy of Sciences 117(37):23066–23072 21
Xie H, Karipidis II, Howell A, et al (2020) Finding the neural correlates of col- laboration using a three-person fmri hyperscanning paradigm. Proceedings of the National Academy of Sciences 117(37):23066–23072 21
2020
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.