{"id":"d0901c65-298c-446b-b420-506ed230a498","arxiv_id":"2504.15093","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Adding acoustic embeddings to BERT raised F1 for social-cognitive collaborative problem-solving classes but not affective classes, and only with transformer models.","lead":"A study of 78 secondary students found that adding audio features to a language model improved detection of some collaborative problem-solving behaviors, but not others. The authors argue that multimodality and model choice are only helpful for certain indicators, not a universal upgrade.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Instance-level splitting of multi-coded utterances before the 80:20 split risks train/test leakage; with a single split and no significance test, the reported AudiBERT advantage (0.587 vs 0.573 F1) is not yet established.","rationale":"The central claim is narrow and modest: adding acoustic embeddings to a transformer-based text model improves diagnosis of social-cognitive CPS classes. For that claim to hold, the comparison must be unbiased. The strongest threat is not a modeling error but data partitioning: multi-coded utterances are split into instances before the train/test split, creating a possible train/test overlap at the source-utterance level. The reader flagged exactly this risk. Because the reported advantage is 0.014 F1 on a single split and the paper explicitly acknowledges the need for multifold cross-validation and statistical analysis, the evidence is not yet robust. I do not see an internal inconsistency in the architecture, and the direction of the observed effect is plausible, so the appropriate response is to maintain the CONDITIONAL verdict while requiring a grouped-split reanalysis. No change to the reader's verdict is needed.","tokens_in":13243,"tokens_out":4717,"duration_ms":43000,"concrete_test":"Re-run the full pipeline with a grouped stratified split: assign all instances derived from the same source utterance to the same fold, then perform repeated 5-fold or 10-fold cross-validation. Report paired significance tests (e.g., McNemar or bootstrap) for BERT vs AudiBERT on the social-cognitive dimension. If the F1 gap (0.587 vs 0.573) reverses or shrinks below noise under grouping, the multimodality claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 states that utterances with multiple coded indicators were split into individual instances, each tagged to a single indicator, before the 80:20 stratified train-test split described in Section 3.5. If this instance-level split is performed before the train/test split, the same source utterance can appear in both training and testing partitions under different labels. The central evidence for multimodality is AudiBERT's weighted F1 gain over BERT on the social-cognitive dimension (0.587 vs 0.573, Table 2) from a single split. If the test set contains near-duplicates of training utterances, the text encoder can memorize surface forms rather than learn generalizable classes, inflating performance for both models and potentially changing the relative ranking. This is especially relevant for the small classes where AudiBERT shows improvement. The paper itself acknowledges in Section 5.1 that multifold cross-validation and statistical analysis are needed before robust conclusions can be drawn. The load-bearing condition is independence of training and test instances at the source-utterance level; that condition is not currently demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether adding acoustic embeddings to transformer-based text models improves automated diagnosis of collaborative problem solving (CPS) subskills and affective states. Using transcripts and audio from 78 secondary students, the authors compare random forest with TF-IDF, a multimodal random forest, fine-tuned BERT, and an audio-assisted BERT variant (AudiBERT) on two coding dimensions with 80:20 stratified splits. They report that transformer models outperform traditional ones, and that AudiBERT achieves a small weighted F1 gain over BERT on the social-cognitive dimension (0.587 vs 0.573) while not improving over BERT on the affective dimension. The authors conclude that multimodality and model selection should not be assumed to help all CPS classes, and they advocate human-AI complementarity and ensemble approaches.","tokens_in":13446,"tokens_out":2506,"duration_ms":25276,"significance":"If the evaluation is valid, the paper provides useful empirical evidence on the value of multimodality in LLM-based CPS diagnosis, using a real classroom setting, a documented coding framework, high inter-rater reliability (Cohen's kappa 0.847), and publicly shared code and appendices. The findings are more nuanced than blanket claims about multimodal superiority and could inform practical choices in AIED systems. However, the central comparison is currently undermined by evaluation-design concerns: a single split, very small test classes, possible train/test leakage from instance-level splitting, and a confound between architecture and modality. Because these issues directly affect the main claim, the contribution is conditional rather than established.","major_comments":[{"comment":"The data processing pipeline creates a risk of train/test leakage. Section 3.4 states that utterances with multiple coded indicators were split into individual instances, each tagged to a single indicator, and Section 3.5 then describes the 80:20 stratified train-test split. If the split operates on these instances rather than on source utterances, the same original utterance can appear in both training and testing under different labels. Since BERT and AudiBERT are text-based, they could memorize surface forms of such near-duplicates, inflating reported performance and potentially changing the relative ordering of models, especially for rare classes. The paper should either confirm that all instances derived from one source utterance were kept on the same side of the split, or re-run the evaluation with an utterance-level grouping before splitting.","section":"Section 3.4 and 3.5"},{"comment":"The main comparison rests on a single fixed-seed 80:20 split with no confidence intervals or significance tests, and several test classes contain only 4, 6, or 8 instances (e.g., SS7, SS4, AS3). The reported AudiBERT advantage over BERT on the social-cognitive dimension is 0.014 weighted F1, which is within the range that could plausibly arise from sampling noise on such small classes. The authors acknowledge in Section 5.1 that multifold cross-validation and statistical analysis are needed before robust conclusions can be drawn; this acknowledgment should be reflected in the results by adding bootstrap confidence intervals, repeated stratified cross-validation, or significance tests, and by reporting per-class performance for the small classes.","section":"Section 3.5 and Table 2"},{"comment":"The BERT and AudiBERT comparison is confounded by architecture. The BERT baseline is fine-tuned directly, whereas AudiBERT feeds BERT text embeddings and Wav2Vec2.0 audio embeddings into separate BiLSTM layers followed by self-attention and concatenation. Thus AudiBERT differs from BERT not only by adding audio but also by adding recurrent and attention components over the text encoder. The observed improvement on social-cognitive classes could stem from these architectural additions rather than from multimodality. To support the claim that audio fusion is the cause, the authors should compare AudiBERT against a text-only model with the same BiLSTM/self-attention post-processing, or ablate the audio branch.","section":"Section 3.1"}],"minor_comments":[{"comment":"The model name 'AudiBERT' is written as 'Audibert' in the first paragraph of the Discussion; please make the capitalization consistent.","section":"Section 5"},{"comment":"The reported average Word Error Rate of 0.553 is quite high and could affect the quality of the text modality. The paper should briefly discuss how ASR errors may interact with the comparison between BERT and AudiBERT, or provide a sensitivity check on a subset with lower WER.","section":"Section 3.3"},{"comment":"The row-normalized confusion matrices are described only in general terms. Because several classes have very few test instances, the figure would be more informative if accompanied by the actual counts and per-class F1 scores, ideally in a supplementary table.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for AIED and presents an interesting real-world dataset, but the central claim about AudiBERT's superiority is not yet robust because of the leakage risk and the architecture/modality confound. The authors can address these issues with additional experiments or a clearer description of the split; if the leakage is confirmed, the reported effect may disappear, so I would not recommend acceptance without this clarification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, this is a real applied comparison: AudiBERT (BERT plus Wav2Vec2 audio features) against text-only BERT and traditional baselines for diagnosing collaborative problem-solving subskills from secondary students' transcribed audio. Second, the headline claim—that multimodality helps social-cognitive CPS classes (F1 .587 vs .573)—is not yet established. The evaluation leaks at the utterance-instance level and rests on a single split.\n\nWhat is good: the paper is honest and careful about what it claims. It transfers an existing architecture (AudiBERT from Toto et al.) to a new domain, and the indicator-level analysis is genuinely useful. The finding that multimodality helps some social-cognitive classes but not others, and not affect, is a nice counter to the 'multimodal always helps' narrative. The authors explicitly list limitations: small classes, no cross-validation, need for statistical tests. There is no circularity; they test on held-out human labels. Code is provided.\n\nNow the soft spots. Section 3.4 states utterances with multiple coded indicators were split into individual instances, each tagged to one indicator, and Section 3.5 then does the 80:20 stratified split. If the split is at the instance level, the same source utterance can appear in both training and testing under different labels. That is a textbook way to inflate text-encoder performance through surface-form memorization. The authors never say they grouped by source utterance. Given the AudiBERT advantage is 0.014 F1 on one split, and test sizes for classes like SS4 (n=6) and SS7 (n=4) are tiny, the reported benefit could easily be noise or leakage. The paper's own Section 5.1 concedes that multifold cross-validation and statistical analysis are needed.\n\nThis is not a takedown. The central message—don't assume multimodality helps every CPS indicator—probably survives regardless, since traditional models did worse with audio and affective results did not improve. But the specific claim that AudiBERT beats BERT should not be cited without a grouped split and repeated cross-validation with significance testing.\n\nWho should read it: AIED and multimodal learning analytics people, especially those designing automated CPS diagnosis. It is a decent empirical contribution for that community, and a useful example of why evaluation hygiene matters. It deserves a serious referee, but I would want revision before acceptance.","headline":"A useful, honest applied comparison of multimodal vs text-only transformers for CPS diagnosis, but the AudiBERT gain over BERT rests on a single split with likely utterance-level leakage and needs grouped cross-validation before the 0.014 F1 difference is believed.","tokens_in":13919,"tokens_out":2315,"would_cite":false,"duration_ms":21638,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Audio input gives BERT a small edge in diagnosing collaborative problem solving.","keywords":["collaborative problem solving","multimodal learning analytics","transformer models","BERT","acoustic embeddings","large language models","artificial intelligence in education","CPS diagnosis"],"falsifier":"Re-run the BERT versus AudiBERT comparison using a split that keeps all instances of the same source utterance in the same fold (e.g., group by utterance before splitting, then stratify), and check whether AudiBERT's weighted F1 advantage on social-cognitive classes persists and whether the confusion matrices still differ on the sparse classes.","tokens_in":13073,"feed_emoji":"🎓","tokens_out":3645,"duration_ms":30460,"temperature":0.7,"pith_summary":"This paper asks whether adding acoustic information to a transformer-based language model improves automated diagnosis of students' collaborative problem solving (CPS) skills from recorded discussions. On data from 78 secondary students working in triads, the multimodal AudiBERT model—text embeddings from BERT plus acoustic embeddings from Wav2Vec2.0—outperformed text-only BERT on weighted F1 for social-cognitive CPS classes (0.587 vs 0.573), but not consistently on affective classes. The authors' larger point is cautionary: multimodality and model choice should not be assumed universally beneficial; their value depends on the indicator type, label complexity, and class distribution. The paper also shows transformer models beat traditional TF-IDF baselines, but adding acoustic features to the traditional model hurt performance.","feed_headline":"Audio input gives BERT a small edge in CPS diagnosis","feed_subtitle":"Adding acoustic embeddings improves detection of some social-cognitive skills but not affective states.","key_machinery":"The central object is AudiBERT, a dual self-attentive audio-textual framework in which BERT encodes transcribed utterances and Wav2Vec2.0 encodes the raw audio; each embedding stream passes through its own BiLSTM and a self-attention layer, and the two attended representations are concatenated to classify each utterance into CPS subskills or affective states. This fusion mechanism is what lets the model exploit acoustic-prosodic information beyond the spoken words. The comparison baselines are a random forest on TF-IDF, the same random forest with 11 openSMILE acoustic features appended, and a fine-tuned text-only BERT.","core_discovery":"The paper claims that a multimodal transformer that fuses textual and acoustic embeddings, AudiBERT, improves the automated diagnosis of social-cognitive collaborative problem-solving classes compared with a unimodal text-only BERT, while the same addition of acoustic features to a traditional random forest on TF-IDF does not help. The improvement is modest on weighted F1 and appears for specific subskills (e.g., sense-making, formulating a solution, reaching a solution, maintaining roles, using scripting) and for the sparse positive-affect class, while the affective dimension overall favours the unimodal text model on accuracy, precision, and recall. The authors therefore argue that the value of multimodality is conditional: it is tied to the choice of model architecture and to the alignment between the acoustic features and the human-coded labels.","pith_inferences":["If the evaluation's split leakage is real, the reported F1 gains may shrink under a strict utterance-level split; the qualitative conclusion about conditionality may survive, but the class-level claims would need re-testing.","AudiBERT's advantage on sparse classes hints that acoustic features act as a regularizer or provide supplementary signal when text examples are few; one could test this by ablating the audio encoder and varying class size.","The authors' human-AI complementarity suggestion implies a practical deployment: use models only for classes that are semantically clear and route the rest to human coders; a concrete next step would be to measure total coding time and agreement under such a mixed workflow."],"forward_implications":["If the central claim holds, practitioners can expect transformer-based models (BERT and AudiBERT) to outperform traditional TF-IDF random forests for CPS diagnosis, especially for sparse classes.","Adding acoustic features helps a transformer model but not a traditional model, implying that the model architecture determines whether multimodal data is useful.","AudiBERT's gains concentrate in specific social-cognitive subskills and the sparse positive-affect class, suggesting that acoustic information matters where semantic content alone is insufficient.","Multimodality should not be assumed a universal improvement; its value depends on the indicator type, the complexity of the labels, and the class distribution of the dataset."],"supporting_citations":[{"why":"Provides the AudiBERT architecture that the paper adapts, an audio-textual dual self-attentive framework.","marker":"[56]"},{"why":"Supplies the BERT text encoder used as the unimodal baseline and the text component of AudiBERT.","marker":"[11]"},{"why":"Provides the 11 openSMILE acoustic features and the prior finding that adding acoustic-prosodic features decreased accuracy, which the paper contrasts.","marker":"[48]"},{"why":"Prior work using BERT on transcribed CPS speech that this study extends to multimodality.","marker":"[41]"},{"why":"Supplies the CPS subskill and indicator framework that the authors adapt for their annotation scheme.","marker":"[54]"},{"why":"The openSMILE tool used to extract acoustic-prosodic features for the traditional and AudiBERT models.","marker":"[14]"}],"fun_headline_variants":["Multimodal edge is real but narrowly targeted","Audio helps BERT for some CPS skills, not others","Text-plus-audio aids only transformer CPS diagnosis","Multimodality: modest boost, conditional on model","Audible gains: BERT+audio beats text-only in CPS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation splits the data after utterances with multiple coded indicators have been expanded into separate instances, so the 80:20 stratified split may place instances from the same source utterance in both training and testing; if so, the text encoder could memorize surface forms and inflate the reported performance difference.","fun_headline_variants_meta":{"raw":{"variants":["Multimodal edge is real but narrowly targeted","Audio helps BERT for some CPS skills, not others","Text-plus-audio aids only transformer CPS diagnosis","Multimodality: modest boost, conditional on model","Audible gains: BERT+audio beats text-only in CPS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1393,"prompt_tokens":986,"completion_tokens":407,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":328}},"tokens_in":602,"tokens_out":407,"duration_ms":3877,"temperature":1.0,"reasoning_tokens":328,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:33:28.569342+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the BERT versus AudiBERT comparison using a split that keeps all instances of the same source utterance in the same fold (e.g., group by utterance before splitting, then stratify), and check whether AudiBERT's weighted F1 advantage on social-cognitive classes persists and whether the confusion matrices still differ on the sparse classes.","supporting_citations":[{"cited_title":"User Modeling and User-Adapted Interaction 31(4), 713–751 (2021)","cited_arxiv_id":null,"evidence_quote":"Provides the 11 openSMILE acoustic features and the prior finding that adding acoustic-prosodic features decreased accuracy, which the paper contrasts."},{"cited_title":"In: Proceedings of the 14th International Conference on Educational Data Mining, EDM 2021","cited_arxiv_id":null,"evidence_quote":"Prior work using BERT on transcribed CPS speech that this study extends to multimodality."},{"cited_title":"In: ACM Learning Analytics & Knowledge conference (LAK’24)","cited_arxiv_id":null,"evidence_quote":"Supplies the CPS subskill and indicator framework that the authors adapt for their annotation scheme."}],"review_version":1}