{"id":"876e6de3-c35b-4b1a-9d41-0a0c910342e7","arxiv_id":"2411.15405","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ML-SPEAK learns a map from Big Five and dominance traits to individual turn-taking parameters, and predicts held-out team conversations better than the baselines tested.","lead":"ML-SPEAK is a machine learning model that predicts which member of a team will speak next, using personality traits to estimate each person's speaking style. The authors report that it predicts turn-taking better than earlier models on both simulated data and real student team conversations, and it surfaces interpretable trait-to-behavior links.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Study 3's forward selection uses the same 20 sliding-window data trials for both model selection and evaluation, so the claimed advantage of the extraversion-agreeableness-emotional stability model may be an artifact of selection bias.","rationale":"The reader's formal weakest assumption is misspecification of Eq. (1), which the paper itself acknowledges in the Limitations. I agree that is a real limitation, but it does not target the strongest empirical claim as precisely as the Study 3 evaluation protocol does. The reader's rationale does mention selection on test folds and non-independent sliding-window trials, so there is overlap, but that issue is not the reader's stated weakest assumption. My concern is load-bearing because the headline real-data result rests on a comparison that was selected and evaluated on the same overlapping test folds; this is a fixable methodological flaw, not a claim of misconduct. The synthetic studies remain internally coherent and support the method's mechanics, so the appropriate verdict stays CONDITIONAL rather than moving to ACCEPT or REJECT. A nested cross-validation rerun would settle whether the trait-combination advantage survives without selection bias.","tokens_in":23039,"tokens_out":5149,"duration_ms":51359,"concrete_test":"Re-run Study 3 with nested cross-validation. Split the 20 teams into 5 folds. For each outer fold, run the full forward-selection procedure (uni-, bi-, tri-variate) using only the 16 training teams, choosing the final trait set by an inner validation split or 4-fold inner CV rather than by test loss. Then train the selected model on those 16 teams and evaluate it once on the 4 held-out teams. Pool the outer-fold test losses and compare with the baselines using a permutation test over team assignments. If the extraversion-agreeableness-emotional stability model no longer significantly beats the same-traits and SPEAK baselines, the reported Study 3 advantage is explained by selection bias on the shared test folds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central real-world claim (Study 3) is that the three-trait ML-SPEAK model 'performed significantly better than the other baseline models.' The evidence is compromised by the evaluation protocol. The Study 3 Procedure generates 20 data trials by sliding the train/validation/test demarcations one team at a time over a randomized team order, so the trials share teams and are not independent. The Analysis then uses these same 20 trials for forward selection: at each step the authors keep any trait or trait pair whose median loss difference is below zero and select the model with the lowest p-value from paired, one-sided Wilcoxon tests on those trials. The final three-trait model is then compared with baselines on the very test folds that were used to choose it. This creates optimistic selection bias: choosing the best of many candidate trait sets on the test data inflates apparent performance, and the reported Kruskal-Wallis chi-squared = 75.87 with pairwise Wilcoxon comparisons do not account for the selection step. Consistent with this worry, the step adding emotional stability to the two-trait model was not itself significant (W = 91, p = 0.31), yet the model was retained because it had the best median loss. Since the paper's headline claim about trait-to-behavior relationships depends on this comparison, the real-data result is not currently established.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ML-SPEAK, a neural-network-augmented version of the Stasser-Taylor SPEAK model of conversational turn-taking. Each speaker's next-turn likelihood is modeled as a baseline term pi_i plus an exponentially decaying memory term d_i, normalized across team members, and a small feedforward network maps personality traits to the pair (pi_i, d_i). The model is trained by maximum likelihood on observed turn sequences. The paper reports three studies: two synthetic evaluations and one application to real student teams. In Study 3, forward selection over personality traits yields a three-trait model (extraversion, agreeableness, emotional stability) that is claimed to predict held-out turn-taking significantly better than several baselines. The authors also visualize learned trait-to-parameter relationships and discuss implications for team composition.","tokens_in":23308,"tokens_out":8708,"duration_ms":80724,"significance":"If the real-data claim were properly established, this would be a useful contribution: it extends a classic turn-taking model to learn trait-to-parameter maps and can generate full conversation sequences from trait profiles without observing prior conversations. The manuscript has notable strengths: code and data are shared, the synthetic studies vary model components and data properties systematically, and Study 1 uses known generating functions as an external benchmark, which is a legitimate check of the fitting procedure. However, the headline real-data result is currently compromised by the evaluation protocol: model selection and final evaluation use the same overlapping test folds, and the statistical tests do not account for the resulting dependence. The paper's broader conclusions therefore outrun the evidence as presented.","major_comments":[{"comment":"The forward-selection procedure and the final model comparison use the same 20 test folds. At each selection step the authors retain models with median loss difference below zero and choose the lowest p-value from paired one-sided Wilcoxon tests on those trials; the selected three-trait model is then compared with baselines on the same folds. This is selection on the test data, so the reported Kruskal-Wallis chi-squared = 75.87 and the pairwise Wilcoxon comparisons are optimistically biased. The non-significant step adding emotional stability (W = 91, p = 0.31) illustrates that the selected model's advantage over the two-trait model is not established. Please use a selection-aware evaluation, for example performing trait selection on training folds only, or reporting all candidate models with an appropriate multiple-comparison correction.","section":"Study 3, Analysis and Results"},{"comment":"The 20 data trials in Study 3 are generated by sliding the train/validation/test demarcations over a randomized team order, so the same teams appear in multiple trials and the test folds overlap. Paired Wilcoxon tests over these 20 trials therefore treat dependent observations as independent, and the effective sample size is at most the 20 teams. The same problem appears in Studies 1 and 2, where a single test set is reused across all 20 training trials. Moreover, the Kruskal-Wallis tests compare models on loss values from the same data trials, violating the independence assumption of that test. Please use methods that respect the pairing and clustering, such as team-level bootstrap or non-overlapping folds, and report the number of independent test units.","section":"Study 3, Procedure; Study 1 and Study 2, Procedure"},{"comment":"The synthetic data are generated from the same functional family as the model, namely Eq. (1), so the strong Study 1 results are consistency checks rather than evidence of real-world utility. In particular, baselines that omit the memory term are disadvantaged by construction. The authors partly acknowledge this in the Limitations, but the text should state more directly that the synthetic experiments validate parameter recovery and model comparison within the assumed model family, not the adequacy of that family for real conversations. In addition, the SPEAK baseline's d value is calibrated to the median ratio learned by ML-SPEAK, so the comparison between ML-SPEAK and SPEAK does not cleanly isolate the benefit of learning trait-to-parameter associations from data.","section":"Study 1 and Study 2; Computational Methods, Eq. (1)"},{"comment":"The paper acknowledges that the restriction of speaking likelihood to the form in Eq. (1) 'may limit the model's ability to learn more complex speaking patterns.' This is load-bearing because all Study 3 trait-to-behavior relationships are estimated under this fixed functional form; if real turn-taking depends on conversational content, floor states, or changing task context, the learned pi and d functions may be artifacts of misspecification rather than genuine trait effects. Please add robustness checks against alternative generating processes (for instance, content-dependent or floor-state models) or explicitly restrict the conclusions to the class of processes described by Eq. (1).","section":"Limitations and Future Directions"}],"minor_comments":[{"comment":"The text refers to a 'paired, one-sided Wilcoxon Rank sum test'; this should be a paired one-sided Wilcoxon signed-rank test.","section":"Study 3, Analysis"},{"comment":"The same Kruskal-Wallis chi-squared value (52.459) is reported for several distinct comparisons, including Data type = Mem, Data type = SamePi-Mem, Simple traits, and Complex traits; please verify that these are not copy-paste errors.","section":"Study 2, Results"},{"comment":"Please report the neural network training details (activation functions, optimizer, learning rate, regularization, early stopping, number of initializations) or cite the repository location where these details are fully specified.","section":"Computational Methods"},{"comment":"Placeholders such as '[citation to be added after review]' and '[Grant number to be added after review]' must be completed before publication.","section":"Various"},{"comment":"There is a typo in the text: 'extraverison' should be 'extraversion'.","section":"General Discussion"},{"comment":"The author note states that the paper 'is awaiting peer review and has not yet been published'; such a line should be removed from a submitted manuscript.","section":"Author Note"}],"recommendation":"major_revision","confidential_remarks":"The method is promising and the open-data/open-code practices are a strength, but the Study 3 evaluation protocol needs substantive rework before the central real-data claim can be accepted. The duplicate chi-squared statistics and incomplete placeholders suggest the manuscript was not fully proofread. I recommend major revision rather than rejection because the load-bearing issues are addressable with nested/selection-aware evaluation and independence-respecting statistical tests."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real, clearly-presented extension of the SPEAK turn-taking model — it learns a map from personality traits to per-speaker (pi, d) parameters and can predict turn sequences for unseen teams from traits alone. The synthetic studies are internally coherent, and the extraversion result in Study 3 replicates a solid literature. But the headline real-data claim is not yet established: the forward selection in Study 3 picks the trait set on the same 20 sliding-window test folds that are later used for the baseline comparison, and those trials are not independent because teams are reused. That's selection bias plus anti-conservative p-values.\n\nWhat's new: relative to Stasser & Taylor's team-level SPEAK, Padilha & Carletta's fixed trait-behavior links, and Basu et al.'s team-specific influence model, learning trait-to-parameter maps that transfer to new teams is a genuine contribution. The model itself is simple enough to interpret: each speaker's next-turn logit is pi_i plus a decaying memory term d_i, normalized across members. Study 1 shows the method can recover the generating functions on synthetic data and outperforms the baselines. Study 2 is a thorough sensitivity analysis. The code and data are promised on OSF, which helps.\n\nSoft spots, in order of severity. First, the Study 3 selection procedure: they generate 20 trials by sliding the train/validation/test split one team at a time over a randomized team order, so the trials share teams. Then they do forward selection using those same trials — at each step keeping any candidate with median loss difference below zero and choosing the one with the lowest paired Wilcoxon p-value. The final three-trait model is then compared to baselines on the same test folds that were used to choose it. The step that added emotional stability to extraversion+agreeableness was not itself significant (W=91, p=0.31), yet the model was retained for best median loss. So the reported advantage over baselines is optimistically biased, and the pairwise Wilcoxon tests ignore the dependence across trials. This doesn't sink the method, but it means the real-data trait-to-behavior relationships are not supported as presented.\n\nSecond, the dataset is described in detail only by a citation that is still a placeholder, and there are other placeholders (grant number). That's fine for a preprint but needs to be fixed for review. Third, the synthetic studies are self-consistency checks: they validate the model against its own generating process, not against an independent process, so the strong synthetic results don't by themselves transfer to real conversations. The authors acknowledge the functional-form restriction in the Limitations, which is good.\n\nWho's it for: team science researchers, computational social scientists, and method developers. The central idea is worth a serious referee. I'd send it to peer review, but the Study 3 evaluation protocol needs to be redone — either select on a separate validation split, or use a nested procedure, and account for the non-independence of trials. Until then, treat the real-data comparisons as exploratory.","headline":"Genuine new method for trait-to-turn-taking mapping, but Study 3's model selection on the same test folds leaves the real-data claim unproven.","tokens_in":23846,"tokens_out":3307,"would_cite":false,"duration_ms":29429,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a small neural network can learn how personality traits shape conversational turn-taking and predict a new team's speaking sequence from member traits alone.","keywords":["conversational turn-taking","personality traits","team communication","SPEAK model","machine learning","team composition","computational modeling","Big Five"],"falsifier":"Train ML-SPEAK and a comparison model whose memory term includes floor-state or content features on transcripts containing long ABA floor exchanges, then compare held-out per-turn log-likelihood; if the state-aware model consistently wins, the fixed form in Eq. (1) is not sufficient to support the paper's predictive claim.","tokens_in":22824,"feed_emoji":"💬","tokens_out":9853,"duration_ms":84930,"temperature":0.7,"pith_summary":"The paper claims that a small amount of structure—each speaker's turn likelihood is a personal baseline plus an exponentially decaying boost after their last turn—is enough for a machine-learning model to map personality traits onto conversational turn-taking and to predict whole speaking sequences for teams it has never observed. The authors train a two-layer network to turn trait scores into each speaker's two parameters by maximizing the probability of observed conversations, then use those parameters to simulate new conversations. In synthetic tests the model beats the earlier SPEAK model and other baselines, and on real student-team data the trait combination of extraversion, agreeableness, and emotional stability yields the best predictions. If the claim holds, team researchers could forecast communication dynamics from trait composition alone and see which trait combinations drive equal participation rather than dominance.","feed_headline":"Forecast who speaks next from three personality traits","feed_subtitle":"ML-SPEAK learns the trait-to-turn-taking link and predicts conversations for teams it has never observed.","key_machinery":"The load-bearing object is the per-speaker parameter pair $\\theta_i = (\\pi_i, d_i)$: $\\pi_i$ is the stable baseline likelihood of speaking, and $d_i$ controls how strongly a recent turn boosts the chance of speaking again, with that boost decaying exponentially over subsequent turns. The model normalizes the raw likelihoods $\\ell_i(t)$ across all members to get next-speaker probabilities, forbidding back-to-back turns by the same speaker. A small feedforward network with one hidden layer of ten neurons produces $\\theta_i$ from trait scores, trained by maximum likelihood on observed conversations, so that, once trained, the whole conversation can be simulated from trait values alone. Because the network's outputs are directly interpretable, the same machinery also reveals how each trait shifts baseline speaking and interactivity.","core_discovery":"The central claim is that the stochastic process governing who speaks next in a small group can be predicted from member traits alone once the trait-to-parameter mapping is learned. Each member $i$ has a raw speaking likelihood $\\ell_i(t)$ that is their baseline $\\pi_i$ plus a memory term $d_i$ decaying exponentially with the number of turns since they last spoke, and these raw values are normalized to a probability distribution over members, with the just-spoken member excluded. A two-layer feedforward network maps trait scores to the pair $\\theta_i = (\\pi_i, d_i)$, and its weights are fit to maximize the log-likelihood of the observed turn sequences. The paper reports that this ML-SPEAK model outperforms the SPEAK model and all tested baselines on synthetic data, and that, on real team conversations, a model trained on extraversion, agreeableness, and emotional stability predicts speaking sequences better than every baseline.","pith_inferences":["An extension the paper leaves implicit: if the trait-to-parameter map is stable across settings, the same architecture could be retrained on abilities, motivation, language proficiency, or other individual differences, and on in-person rather than video conversations.","Replacing the fixed exponential memory with a learned memory function, which the limitations section suggests, could reveal speakers whose likelihood of speaking rises as they stay silent longer; the present model cannot represent that.","A direct stress test would compare ML-SPEAK against a model with floor-state or content-aware memory terms; if the richer model improves held-out prediction, content-independent turn-taking is missing part of the mechanism.","The predictive direction could be inverted into a team-design tool—searching trait compositions for a target pattern such as equal participation—a use the authors gesture toward but do not implement."],"forward_implications":["A new team's likely speaking-turn sequence can be simulated before the team ever talks, as long as each member's trait scores are known.","The split between a stable baseline ($\\pi_i$) and a sequence-dependent memory ($d_i$) lets the model expose trait effects that aggregate measures like total speaking time cannot.","On real data, extraversion alone beat a no-individual-differences baseline, and adding agreeableness and emotional stability improved prediction further, though the further gain was not statistically significant.","Predictive performance held across group sizes and missing-meeting patterns, but conversation lengths below roughly 150 turns degraded learning of trait-to-behavior mappings.","The learned mapping generalizes to unobserved teams, unlike earlier approaches whose parameters were tied to the specific speakers in the training data."],"supporting_citations":[{"why":"Supplies the SPEAK conversation model and the baseline-likelihood-plus-memory structure that Eq. (1) extends to individual differences.","marker":"Stasser and Taylor (1991)"},{"why":"Provides the floor-state analysis showing that context-based and second-order probabilities outperform first-order Markov models, motivating the memory component.","marker":"Parker (1988)"},{"why":"Documents that variance in speaking turns relates to collective intelligence, motivating turn-taking patterns as a team outcome worth predicting.","marker":"Woolley et al. (2010)"},{"why":"Provides the IPIP Big-Five markers used to measure extraversion, agreeableness, conscientiousness, emotional stability, and openness in Study 3.","marker":"Goldberg (1992)"},{"why":"Provides the California Psychological Inventory dominance scale used as a trait input in Study 3.","marker":"Gough (1996)"},{"why":"Supports the paper's interpretation that trait combinations such as extraversion and emotional stability can be more informative than single traits.","marker":"Judge and Erez (2007)"},{"why":"Is the earlier learned-parameter influence model that the paper contrasts with, since it cannot carry learned parameters to new teams.","marker":"Basu et al. (2001)"}],"fun_headline_variants":["ML-SPEAK forecasts team turn-taking from three personality traits","Predict who speaks next with ML-SPEAK using only personality traits","Trait-based model predicts conversation turn patterns better than baselines","Machine learning maps personality to speaking order in teams","ML-SPEAK learns trait-to-turn-taking link to forecast unobserved teams"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The model assumes every speaker's next-turn likelihood has the fixed form of a personal baseline plus an exponentially decaying boost from their last turn, with no dependence on conversational content, floor states, or changing task context; if real turn-taking is driven largely by those omitted factors, the learned trait-to-behavior relationships could be artifacts of misspecification.","fun_headline_variants_meta":{"raw":{"variants":["ML-SPEAK forecasts team turn-taking from three personality traits","Predict who speaks next with ML-SPEAK using only personality traits","Trait-based model predicts conversation turn patterns better than baselines","Machine learning maps personality to speaking order in teams","ML-SPEAK learns trait-to-turn-taking link to forecast unobserved teams"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1518,"prompt_tokens":989,"completion_tokens":529,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":442}},"tokens_in":605,"tokens_out":529,"duration_ms":4961,"temperature":1.0,"reasoning_tokens":442,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:20:56.519459+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train ML-SPEAK and a comparison model whose memory term includes floor-state or content features on transcripts containing long ABA floor exchanges, then compare held-out per-turn log-likelihood; if the state-aware model consistently wins, the fixed form in Eq. (1) is not sufficient to support the paper's predictive claim.","supporting_citations":[],"review_version":1}