{"id":"439aa24d-7b6e-44ba-bd64-626780fb6909","arxiv_id":"2505.03423","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A pilot study applies the NOVA multimodal analysis tool to simulated teacher-parent conversations, finding positive student reception but weak quantitative evidence for predicting expert ratings.","lead":"This paper reports a training seminar in which student teachers practiced simulated parent-teacher conversations and received AI-generated feedback on speech, gaze, and facial expressions. It finds that students welcomed the feedback, but the quantitative evidence for its predictive value is weak and the abstract overstates the results.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's core quantitative claim is unsupported: §4.4 reports no correlation coefficients or significance tests, and 56% accuracy on n=29 with a 16% SD is not shown to exceed the 50% majority baseline, especially if FeatureWiz selection was not nested inside CV.","rationale":"The reader identified the gaze feature transfer as the weakest assumption; I agree that this is a real validity risk, and it deserves scrutiny. However, I find a more upstream and more load-bearing concern: the paper's headline quantitative result is the claimed correlation between communication features and conversation quality, yet §4.4 never reports a correlation, test statistic, or uncertainty estimate that would support the word 'significant'. On n=29, 56% accuracy with a 16% standard deviation is within noise of the 50% baseline, and feature selection before cross-validation could make even that number optimistic. If this concern lands, the central claim that AI-based feedback is objective and actionable loses its main quantitative support; what remains is a coherent qualitative acceptance study that does not establish that the extracted features are valid predictors of counselling quality. This does not change the reader's conditional verdict: the paper can be read as an exploratory design-and-acceptance contribution, but not as evidence of feature-rating prediction until the missing statistics are supplied. I therefore recommend keeping the conditional verdict rather than hardening or softening it.","tokens_in":12283,"tokens_out":4480,"duration_ms":48568,"concrete_test":"Re-run the §4.4 pipeline with FeatureWiz selection nested inside each fold of the five-fold cross-validation and with the rating dichotomization documented in advance. Report Spearman correlations with bootstrap confidence intervals between each selected feature and the dichotomized expert rating, and run an exact binomial test of the resulting accuracy against the 50% majority baseline. If any correlation confidence interval includes zero, or the accuracy confidence interval lower bound is at or below 50%, the abstract's 'significant correlations' and 'objective, actionable insights' wording should be removed or substantially downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing quantitative pillar of the central claim is that measurable paraverbal and nonverbal features predict expert-rated conversation quality, which is what makes AI feedback 'objective' and 'actionable'. Section 4.4 is the only place this is assessed, but it does not report the correlations promised in the abstract: no coefficients, p-values, or confidence intervals are given for any feature-rating relationship. The evidence offered is 56.0% (±16.0%) accuracy for XGB and 55.3% (±6.9%) for LR/SVC on n=29. That is not shown to exceed the 50% majority-class baseline acknowledged in §6: the standard deviations overlap the baseline, and no binomial test or cross-validation confidence interval is provided. In addition, the FeatureWiz feature set appears to be selected on the full dataset before the five-fold split, which can leak label information into feature selection and inflate accuracy; the dichotomization of the five-point rating is also not specified in §4.4. Without at least one credible statistical test, the quantitative evidence cannot carry the abstract's 'significant correlations' claim, and the feedback advice derived from feature differences in §5.1 lacks empirical grounding. The qualitative acceptance finding is a separate, better-supported result, but it does not substitute for the missing feature-quality evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a four-day block seminar for prospective teachers in which simulated teacher-parent conversations are recorded, analyzed with an AI-based multimodal toolchain (NOVA/DISCOVER, WhisperX, OpenFace, LibreFace, openSMILE), and used to generate feedback. Three research questions are posed: (I1) whether nonverbal and paraverbal features correlate with expert-rated conversation quality, (I2) how students perceive AI-based feedback, and (I3) whether AI models can identify conversation phases and communication techniques. The paper reports classifier accuracies around 55–56% for predicting a dichotomized expert rating, qualitative interview findings showing generally positive student reception, and inter-rater agreement for verbal annotations. The central claim is that AI-based feedback provides objective, actionable insights for teacher training.","tokens_in":12499,"tokens_out":3581,"duration_ms":37192,"significance":"If the quantitative claims were adequately supported, the paper would be a useful contribution to an emerging area: using multimodal AI feedback in teacher education. The qualitative part is a strength: the interview study reports concrete student preferences, including the combination of video playback and tabular data, and it gives authentic quotations, which lends credibility to the user-acceptance findings. The paper also describes the full pipeline in enough detail for replication, and it is transparent about limitations such as the small sample size and the transfer of gaze-model hyperparameters from a medical dataset. However, the load-bearing quantitative claim of 'significant correlations' is not backed by the reported statistics, so the paper's central assertion of objective, data-grounded feedback remains unsubstantiated.","major_comments":[{"comment":"The abstract and the discussion state that significant correlations exist between nonverbal and paraverbal features and conversation quality, but §4.4 reports no correlation coefficients, p-values, confidence intervals, or significance tests for any feature-rating relationship. The evidence offered is a five-fold stratified cross-validation with XGB accuracy of 56.0% ± 16.0% and LR/SVC accuracy of 55.3% ± 6.9% on n = 29 sessions. With this sample size, 56.0% is not shown to exceed the 50% majority baseline; the reported standard deviations overlap the baseline and no binomial test or cross-validation confidence interval is provided. The phrase 'significant correlations' in the abstract is therefore unsupported by the reported statistics, and the feedback recommendations in §5.1 that are derived from feature differences lack empirical grounding.","section":"§4.4 and Abstract"},{"comment":"The FeatureWiz feature set appears to be selected on the full dataset before the five-fold split, because §4.4 lists the selected features (question, statement, sentiment, gaze, smile, happiness, sadness, anger) and then describes cross-validation using that fixed set. This can leak label information into the feature-selection step and inflate accuracy estimates. Feature selection should be nested inside the cross-validation loop or otherwise restricted to training folds. In addition, the dichotomization of the five-point expert rating is not specified: the discussion mentions a 50% majority-class baseline for rating '4 = helpful', but the paper does not state how the five-point scale was converted to a binary target.","section":"§4.4 and §6"},{"comment":"The gaze-clustering hyperparameters were chosen on a private medical doctor-patient dataset so that gaze in the 'main direction' correlated with patient satisfaction, and were then applied without re-tuning to teacher-parent conversations. Since gaze and mutual gaze appear in the FeatureWiz-selected feature set, the validity of these features is load-bearing for the quantitative claim. The paper provides no evidence that this transfer is valid for the teacher-parent setting, and no internal validation or calibration of the gaze feature is reported. This is particularly concerning given that the gaze feature is among the few that contribute to the reported classifier results.","section":"§4.3"},{"comment":"The third research question, 'Training AI models to identify conversation phases and techniques,' is not actually answered. §4.4 reports only inter-rater agreement for annotations (66.9% for phases, 72.4% for techniques) and coincidence matrices; no trained model, no model accuracy, and no comparison to baselines is presented. The discussion in §6 correctly states that the annotation guidelines need to be improved first, but the paper nevertheless lists I3 as a guiding investigation and the conclusion does not clearly state that this part of the study was not completed. The reader is left without any evidence about the trainability of models for verbal skill identification.","section":"§4.4 and §1 (I3)"}],"minor_comments":[{"comment":"The feature list contains the typo 'happines' for 'happiness'.","section":"§4.4"},{"comment":"The sentence beginning 'The students’ first received the collective feedback' should read 'The students first received'.","section":"§5"},{"comment":"There is a typo in 'the former to forms of feedback', which should be 'the former two forms of feedback'.","section":"§5.2"},{"comment":"The quotation 'Apparently I am beeing perceived as more positively than i thought' contains two spelling/capitalization errors ('beeing' and 'i'), which should be corrected even in translated quotations.","section":"§5.2"},{"comment":"The phrase 'three preceeding frames' should be 'three preceding frames'.","section":"§4.3"},{"comment":"The section header 'NOV A Play Back' contains an unwanted space and should be 'NOVA Playback'.","section":"§5.1"},{"comment":"The camera resolution '2160p25' is later contrasted with '360p25' and '540p25'; these terms are inconsistently formatted and should be clarified (e.g., as 4K, 360p, 540p).","section":"§3"},{"comment":"The relative-deviation formula is defined twice in slightly different forms; the presentation would be clearer if the formula were given once as relative = (absolute - mean) / mean.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid qualitative component but the quantitative foundation of the central claim is not established. The authors should be asked to either report proper correlation and significance analyses, or substantially soften the claims in the abstract and conclusion. The FeatureWiz leakage issue is fixable, but if the corrected analysis no longer shows predictive value, the paper may need to be reframed around the qualitative findings and the technical pipeline. I also note the unusually high proportion of self-citations to the NOVA/DISCOVER toolchain; while these are justified by the method, the paper would benefit from a clearer validation of each feature source against the target domain."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's real contribution is the qualitative user study; its headline quantitative claim is not supported by the reported numbers. The abstract says \"significant correlations\" between nonverbal/paraverbal features and conversation quality, but the evaluation section reports only classifier accuracies from a five-fold CV. No correlation coefficients, confidence intervals, or significance tests appear anywhere. The best accuracy is 56.0% ±16.0% on n=29 (XGB with FeatureWiz features), which is not shown to exceed the 50% majority baseline; the standard deviation alone makes that obvious. The Discussion says it is \"above dummy classification level\" but offers no test. Also, the feature selection by FeatureWiz appears to be done on the full dataset before the CV split, which can leak label information. The abstract also says \"training of AI models to identify conversation phases and techniques,\" but the paper only reports inter-rater agreement for human annotation; no such model is trained. So the quantitative pillar of the abstract is absent.\n\nWhat the paper does well: the seminar design and the qualitative interview study are solid. The students' perceptions of the four feedback formats (tabular data, parallel coordinates, radar charts, NOVA playback) are presented with direct quotes, and the findings are sensible: they prefer video playback plus tabular data, want more interpretation, and find radar charts need explanation. That is a useful, honest piece of work for teacher-education and educational-technology researchers. The authors also acknowledge the small sample and the need for larger data.\n\nSoft spots beyond the missing statistics: the gaze feature is built on a clustering tuned on a private medical dataset and transferred without re-tuning; the paper admits this, but it weakens the claim that gaze \"main direction\" captures engagement in teacher-parent talk. The five-point rating is dichotomized somewhere, but the cut point is never specified. The FeatureWiz feature set itself was selected on the full data, and no nested selection is described. These are fixable with a revision that honestly reports what was done and rephrases the abstract.\n\nWho is this for? People building AI-assisted feedback systems for interpersonal skills training, and teacher-education researchers. A serious referee should see it, but the authors need to either supply the missing statistics or downgrade the claims. I'd send it to review, with a request for major revision before acceptance.","headline":"The qualitative findings are worth a look, but the abstract's 'significant correlations' claim is not backed by any reported statistics.","tokens_in":13086,"tokens_out":1968,"would_cite":false,"duration_ms":19765,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that AI-derived speech and gaze features can predict expert-rated quality in simulated teacher-parent conversations, and that students find the resulting feedback useful.","keywords":["counselling competence","AI-based feedback","teacher-parent conversation","multimodal communication analysis","paraverbal features","nonverbal features","teacher education","qualitative interview study"],"falsifier":"Have experts label eye contact in a sample of the recorded teacher-parent conversations, then compare those labels with the gaze-in-main-direction feature; if agreement is low, or if the feature's correlation with expert ratings disappears after the gaze clustering is re-tuned on teacher-parent data, the gaze-based part of the feedback claim is falsified. As a second check, run the classifier with and without the gaze features and see whether accuracy and selected features change.","tokens_in":12039,"feed_emoji":"🎓","tokens_out":11717,"duration_ms":109230,"temperature":0.7,"pith_summary":"This paper sets out to show that AI analysis of recorded conversations can supply objective, useful feedback on the counselling skills of prospective teachers. In a four-day seminar, student teachers conducted simulated conversations with actors playing parents, and a multimodal analysis pipeline extracted paraverbal features (pitch, loudness, speaking rate, sentiment, shares of questions and statements) and nonverbal features (gaze, mutual gaze, smiling, happiness, sadness, anger) from the recordings. The authors report that these machine-measured features correlate with expert ratings of conversation quality, with the best classifier reaching 56.0% accuracy against a 50% majority-class baseline, and that students interviewed after the seminar responded positively to the AI feedback, especially when synchronized video playback was combined with numeric comparison tables. If these results hold, teacher training gains concrete, repeatable targets for communication skills that are usually judged holistically and subjectively.","feed_headline":"AI feedback reads teacher-parent talks to train counselling","feed_subtitle":"Speech, gaze, and smile signals predicted expert ratings of simulated parent talks, and students welcomed the feedback.","key_machinery":"The mechanism is a multimodal feedback loop built around an analysis-and-visualization tool that fuses automatic transcription with frame-level audio and facial features. The transcription splits each recording into speaking and listening segments; on spoken frames the pipeline computes paraverbal features such as segment duration, words per segment, word length, speaking rate, statement and question shares, sentiment, pitch, and loudness, and across all frames it computes gaze direction clustered into a 'main direction' versus other, mutual gaze, smiling, mutual smiling, happiness, sadness, and anger. These features are aggregated over the session, displayed in several complementary forms, and compared with expert ratings. The load-bearing piece is the gaze feature: the system assumes that the cluster of gaze angles called 'main direction' represents looking at the interlocutor, and gaze and mutual gaze enter both the selected feature set and the feedback advice, but the clustering hyperparameters were chosen on a private medical dataset and applied without re-tuning.","core_discovery":"On the paper's own terms, the discovery is that an AI-based feedback loop can work for teacher-parent counselling training. The authors designed and ran an iterative block seminar four times with 29 student teachers; recordings of the simulated conversations were processed into session-level statistics for paraverbal and nonverbal behaviour, then shown to students in tables, parallel-coordinate plots, radar charts, and synchronized video playback together with expert ratings. The quantitative results show predictive capability of these features for expert rating above chance (56.0% accuracy versus a 50% dummy baseline), and the qualitative interviews show that students perceived the AI feedback positively, valuing the combination of video excerpts and numeric comparison. The verbal component is treated as groundwork: inter-rater agreement for conversation phases was 66.9% and for communication techniques 72.4%, below the 80% level the authors say is needed before automatic verbal feedback can be attempted.","pith_inferences":["A direct validation of the gaze feature in teacher-parent conversations is missing: if the gaze clustering were re-tuned or validated on this setting, the feature selection and accuracy might change, so the reported correlation should be treated as conditional on that transfer.","The small sample of 29 students over four seminars makes the 56% accuracy a proof-of-concept; a natural extension is to fix the feature set and classifier before the next seminar and test whether the same features re-emerge.","Students' wish to see the parent's mood over time suggests a testable design: displaying reciprocal affect dynamics rather than session averages may make feedback more actionable and could show whether teacher techniques shift parent affect during the conversation.","If such feedback works for teacher-parent counselling, the same pipeline could be adapted to other professional communication trainings, such as medical consultations or social work interviews, where the underlying paraverbal and nonverbal features are similar."],"forward_implications":["If the correlations hold, teacher training programmes can define concrete, machine-measurable communication targets such as pace, questioning, gaze, and smiling instead of relying only on holistic expert impressions.","Students' preference for video-plus-table feedback gives a concrete design rule: combine synchronized playback of the recorded conversation with session-level numeric comparisons.","The verbal-skill annotations, once agreement is raised from 66.9% and 72.4% toward 80%, would allow automatic detection of conversation phases and techniques, extending feedback from paraverbal and nonverbal signals to what is said.","The system is intended to support experts rather than replace them: at 56% accuracy the AI feedback is presented alongside instructor ratings and expert interpretation."],"supporting_citations":[{"why":"Describes the multimodal analysis and visualization tool used to extract and present the paraverbal and nonverbal feedback features.","marker":"[6]"},{"why":"Provides the data-driven extraction and exploration backbone on which the feedback pipeline is built.","marker":"[38]"},{"why":"The speech transcription with word-level timestamps that splits sessions into speaking and listening frames and anchors the paraverbal feature computations.","marker":"[2]"},{"why":"The face-tracking tool that supplies gaze-angle regression, the input for the gaze and mutual-gaze features.","marker":"[3]"},{"why":"The facial-expression model that predicts the categorical emotions (happiness, sadness, anger) used as feedback features.","marker":"[11]"},{"why":"The audio feature extractor that computes pitch and loudness aligned to spoken frames.","marker":"[22]"},{"why":"The automated feature-selection step that chose the feature set on which the best classifier reached 56.0% accuracy.","marker":"[39]"},{"why":"Provides the satisfaction questionnaire used when tuning the gaze-clustering hyperparameters on the private medical dataset; it is the basis for the gaze validity assumption.","marker":"[13]"},{"why":"Defines the conversation-phase scheme that students are taught and that anchors the verbal annotation and feedback.","marker":"[9]"},{"why":"Defines the communication techniques (paraphrasing, verbalising, structuring) that constitute the verbal skills under study.","marker":"[25]"}],"fun_headline_variants":["AI feedback on nonverbal cues sharpens counselling skills","Smiles and gaze decoded by AI to train teacher counselling","AI feedback loop helps future teachers master counselling","Teacher-parent conversations get AI feedback for training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that clustering a person's gaze directions into a 'main direction' correctly captures looking at the conversation partner in teacher-parent video conversations; the clustering was tuned on a private medical dataset and transferred without re-tuning, so if that transfer fails, the gaze-based findings and feedback advice lose their grounding.","fun_headline_variants_meta":{"raw":{"variants":["AI feedback on nonverbal cues sharpens counselling skills","Smiles and gaze decoded by AI to train teacher counselling","AI feedback loop helps future teachers master counselling","Teacher-parent conversations get AI feedback for training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000157,"raw_usage":{"total_tokens":1175,"prompt_tokens":851,"completion_tokens":324,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":265}},"tokens_in":467,"tokens_out":324,"duration_ms":3895,"temperature":1.0,"reasoning_tokens":265,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:51:19.516683+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have experts label eye contact in a sample of the recorded teacher-parent conversations, then compare those labels with the gaze-in-main-direction feature; if agreement is low, or if the feature's correlation with expert ratings disappears after the gaze clustering is re-tuned on teacher-parent data, the gaze-based part of the feedback claim is falsified. As a second check, run the classifier with and without the gaze features and see whether accuracy and selected features change.","supporting_citations":[{"cited_title":"DISCOVER: A Data-driven Interactive System for Comprehensive Observation, Visualization, and ExploRation of Human Behaviour","cited_arxiv_id":"2407.13408","evidence_quote":"Provides the data-driven extraction and exploration backbone on which the feedback pipeline is built."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The face-tracking tool that supplies gaze-angle regression, the input for the gaze and mutual-gaze features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The facial-expression model that predicts the categorical emotions (happiness, sadness, anger) used as feedback features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The automated feature-selection step that chose the feature set on which the best classifier reached 56.0% accuracy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the satisfaction questionnaire used when tuning the gaze-clustering hyperparameters on the private medical dataset; it is the basis for the gaze validity assumption."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the conversation-phase scheme that students are taught and that anchors the verbal annotation and feedback."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the communication techniques (paraphrasing, verbalising, structuring) that constitute the verbal skills under study."}],"review_version":1}