{"id":"9a5261d1-6109-4b69-81cb-a92b9ef14d2d","arxiv_id":"2507.07280","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Interruption detection in small-group dialogue remains accurate in simulated multi-group background noise when acoustic and textual features are combined, while an overlap-based heuristic collapses.","lead":"This paper tests whether AI can detect interruptions in small-group conversations when other groups are talking in the background. The model using speech text and acoustic cues keeps working in simulated multi-group audio, while the common shortcut of treating overlapping speech as an interruption fails.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper infers 'meaningful patterns beyond overlapping speech' without directly evaluating interruptions that do not overlap; a stratified F1 on the 103 non-overlap interruptions would test this load-bearing claim.","rationale":"I read the paper's central contribution as the claim that a learned model can detect interruptions in multi-group audio better than an overlap-based operationalization, and that it does so using features beyond simple overlap. The experimental comparison to the overlap heuristic is a reasonable first demonstration, and the leave-one-group-out evaluation is appropriate for ten groups. The drop of the heuristic baseline from .819 to .524 as groups are added is a robust qualitative phenomenon. However, the paper's Section 6 interpretation goes beyond the data: outperforming a fixed heuristic does not establish that the model detects non-overlap interruptions. The clean-condition F1 being equal to the heuristic, .819, is consistent with the model learning the same overlap cue, calibrated to the target group. The 103 non-overlap interruptions are identified but never used as an evaluation subset. This is an omitted analysis that directly bears on the headline claim, and it can be settled with existing annotations. The reader's concern about undocumented spatial effects is valid but secondary: it affects external validity and reproducibility, while the stratified analysis affects internal validity of the beyond-overlap claim. No code or data release, and the unsupported state-of-the-art phrasing, further support keeping the verdict conditional.","tokens_in":850,"tokens_out":1930,"duration_ms":131038,"concrete_test":"Using the existing Oracle timestamps and the model's per-utterance predictions, stratify the clean-condition evaluation by whether the human-annotated interruption overlaps with another utterance in the same group, as in Figure 3. Report F1, precision, and recall separately for overlapping and non-overlapping interruptions, and specifically for the 103 non-overlap instances. If F1 on the non-overlap subset is at or near chance, with chance F1 about .56 for clean segments, the claim that the model learns patterns beyond overlapping speech is not supported. For the noisy conditions, apply the same stratification using the target group's Oracle timestamps to verify the robustness claim is not driven solely by within-group overlap cues.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 6 that the model learns meaningful patterns beyond overlapping speech is not directly measured. The evidence is that a supervised model outperforms a fixed overlap heuristic in the noisy condition, .792 vs .524 F1. But a supervised model can beat a hand-coded rule by calibrating its threshold; this does not show it detects interruptions that lack overlap. In the clean condition the supervised model's F1, .819, equals the overlap heuristic, .819, and Figure 3 reports 103 interruptions with no overlapping speech. No analysis reports model performance on that subset. If the model's noisy-condition advantage comes only from ignoring cross-group overlap while still relying on within-group overlap cues, the beyond-overlap claim is unsupported. The simulation-fidelity issue raised in the reader's verdict is real but is a transferability concern; the missing stratified analysis is an internal-validity gap in the paper's headline interpretation. The limitations section acknowledges the simulated setting, but does not acknowledge that the beyond-overlap claim lacks a direct test.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies interruption detection in small-group collaborative dialogues, comparing a clean single-group condition with a simulated multi-group noisy condition. Using the Weights Task Dataset, the authors extract BERT-small text embeddings and eGeMAPS acoustic features, train random forest classifiers with leave-one-group-out cross-validation and Hyperopt-based model selection, and evaluate three conditions: clean audio with clean segments, noisy audio with noisy segments, and noisy audio with clean segments. The central empirical result is that a supervised model maintains F1 around 0.75-0.79 under simulated multi-group noise, while an overlap-based heuristic baseline drops from F1 0.819 in the clean analogue to 0.524 in the noisy analogue. The paper concludes that interruption detection can be decoupled from overlapping speech and that the model learns meaningful patterns beyond overlap, with implications for classroom deployment.","tokens_in":8820,"tokens_out":4385,"duration_ms":53975,"significance":"If the results hold, the paper makes a useful contribution to multimodal dialogue analysis in educational settings: it demonstrates that the common operationalization of interruptions as overlapping speech fails when multiple groups converse simultaneously, and it provides initial evidence that acoustic and textual features retain signal under background speech. The leave-one-group-out evaluation is appropriate given the ten-group dataset, and the comparison against a heuristic baseline is a sensible way to isolate the effect of overlap. The feature ablation and the use of an existing labeled dataset are additional strengths. However, the headline interpretation that the model 'learns meaningful patterns beyond overlapping speech' is not directly tested, and the model-selection procedure and noise simulation are described too coarsely to fully support the absolute performance claims. These gaps are addressable with additional analysis rather than being fundamental flaws.","major_comments":[{"comment":"The conclusion that 'the model learns meaningful patterns beyond overlapping speech' is not directly supported by any reported experiment. Figure 3 shows that 103 interruptions have no overlapping speech, but no analysis reports model performance on that subset. In the clean condition the supervised model's F1 of 0.819 (Table 1) equals the overlap heuristic's F1 of 0.819, and in the noisy condition the model could outperform the heuristic simply by learning to ignore cross-group overlap while still relying on within-group overlap cues. Please report stratified precision, recall, and F1 for the non-overlap interruptions (the 103 instances in Figure 3) and for the overlap interruptions, and compare those numbers against the chance F1 values reported in Section 4. Without this analysis, the load-bearing claim in Section 6 is unsupported.","section":"Section 6 (Conclusion) and Figure 3"},{"comment":"The hyperparameter search is not described as nested inside the leave-one-group-out cross-validation folds. The text says Hyperopt was used 'to identify the best model and hyperparameters using F1 score as the guiding metric' and that leave-one-group-out CV was then used, but it does not state whether hyperparameter selection was performed on the training folds only. If hyperparameters were chosen using the evaluation folds, the reported F1 scores in Tables 1-3 are optimistically biased. Please specify the inner validation procedure or report the fixed hyperparameters and justify that no selection occurred on held-out groups.","section":"Section 3.4 (Model Search)"},{"comment":"No significance tests are reported for the central comparisons. With only ten groups and standard deviations that often overlap (e.g., Table 1, textual F1 .798 (SD=.055) vs. acoustic F1 .821 (SD=.029); Table 3, Textual+Acoustic .792 (SD=.056) vs. Textual .782 (SD=.052)), the differences between feature sets and between experimental conditions may not be reliable. Please report paired statistical tests across the ten group folds for the main comparisons: clean vs. noisy audio, and textual vs. acoustic vs. combined features. This is particularly important for the feature-robustness claims in Section 5.","section":"Section 4 (Results) and Tables 1-3"},{"comment":"The simulation of multi-group audio is underspecified. The manuscript states only that 'spatial effects' were used to approximate a shared acoustic environment, without specifying how the audio files were overlaid, what spatial effects were applied (e.g., reverberation, source positions, gains), what signal-to-noise ratios resulted, or whether the simulation was validated against real multi-group recordings. Because all noisy-condition results and the classroom-deployment claim depend on this simulation, please provide the simulation parameters and, if possible, a sensitivity analysis over noise levels or a comparison with real recordings. The limitations paragraph in Section 6 acknowledges the simulation but does not supply the missing detail.","section":"Section 3.1 (Multi-group recordings)"}],"minor_comments":[{"comment":"The sentence 'we used spatial effects to simulate how sound travels in a physical space when overlaying these groups’ audio files in order to approximated a shared acoustic environment' contains a typo: 'approximated' should be 'approximate.'","section":"Section 3.1"},{"comment":"The discussion of Figure 4 says 'even when only 2 other groups are present,' but the number of total groups in that condition should be stated explicitly (e.g., the target group plus two others, for three total groups). The x-axis of Figure 4 should also be labeled clearly with the total number of groups.","section":"Section 4 (Baseline)"},{"comment":"The clean analogue of the heuristic baseline uses Oracle timestamps, while the supervised model uses automatic VAD-based segments. This means the clean F1 comparison (0.819 vs. 0.819) is not a perfectly controlled comparison. The authors should note this asymmetry or, ideally, also compute the overlap baseline using the automatic clean segments.","section":"Section 4 (Baseline)"},{"comment":"The permutation importance table has inconsistent formatting: some cells contain negative values and dashes, and the caption does not explain what the dash indicates. Please clarify the units and the sign convention, and define the feature name abbreviations in the table or caption.","section":"Table 4"},{"comment":"The statement that 'the acoustic features were able to provide necessary information for a model to learn main-speaker overlap' is vague; the authors should specify what 'main-speaker overlap' means in the feature analysis and how the permutation importance results support this claim.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for an AI-in-education or multimodal dialogue venue. The core idea is sound and the heuristic-baseline degradation is compelling, but the headline 'beyond overlapping speech' claim requires a direct stratified test on the non-overlap interruptions, and the model-selection and noise-simulation details need to be pinned down. These are fixable in revision; I would not reject on the current evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper has a genuinely useful new evaluation—interruption detection under simulated multi-group background noise—and the leave-one-group-out results are plausible. But the abstract and conclusion overreach: the model is called 'state-of-the-art' without any comparison to prior systems, and the claim that it learns 'meaningful patterns beyond overlapping speech' is not directly tested. The stress-test note is right: the paper never reports model performance on the 103 non-overlapping interruptions. In the clean condition the model ties the overlap heuristic (.819), so the noisy-condition advantage could come entirely from ignoring cross-group overlap while still using within-group overlap cues. A stratified F1 on the non-overlap subset would settle it.\n\nWhat's new and good: the task itself—evaluating interruption detection when multiple groups talk concurrently—is new relative to the single-conversation literature cited here. The three experiments cleanly separate the effects of segmentation vs. audio noise, and the feature ablations give a useful picture of which features survive. Leave-one-group-out is the right call for ten groups. The limitations section honestly flags the simulated setting and limited feature set.\n\nSoft spots, in order: (1) The missing stratified analysis above is the main internal-validity gap. (2) The Hyperopt search isn't described as nested inside the CV folds; with ten groups that's a real selection-bias risk. (3) The noise simulation is underspecified—'spatial effects' is vague, and no code or data are released, so reproducibility is limited. (4) 'State-of-the-art' is unjustified with no comparison. These are fixable, and none of them sink the core demonstration that an overlap-based operationalization collapses in multi-group audio while a trained model doesn't.\n\nWho's it for: researchers in educational AI, classroom analytics, and multi-party dialogue. A serious referee should see it, but it needs revision before publication. I'd recommend the editor send it out.","headline":"Useful new evaluation of interruption detection under multi-group noise, but the 'beyond overlap' claim isn't directly tested and the tone overstates the findings.","tokens_in":9356,"tokens_out":2165,"would_cite":false,"duration_ms":24042,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A learned classifier keeps interruption detection near .79 F1 when ten groups talk at once, while an overlap-defined baseline collapses to .52.","keywords":["interruption detection","collaborative learning","overlapping speech","background noise","acoustic features","prosody","random forest","classroom monitoring"],"falsifier":"Record several real triads working simultaneously in one classroom with actual room acoustics, run the same pipeline, and compare interruption-detection F1 against the authors' .79 with clean timestamps and .75 with noisy segmentation; a drop near the overlap baseline's .52 would show the simulated noise overstated acoustic robustness. A lighter test is to measure whether the overlap heuristic degrades with real co-located groups as steeply as Figure 4 predicts.","tokens_in":8418,"feed_emoji":"🗣️","tokens_out":4298,"duration_ms":45306,"temperature":0.7,"pith_summary":"The paper sets out to show that automatic interruption detection in small-group collaboration can survive the background chatter of other groups working in the same room. Its central result is that a random-forest classifier using transcribed text and acoustic features keeps detecting interruptions when audio from up to ten groups is mixed together, while treating \"interruption\" as \"overlapping speech\" collapses in the same conditions. On clean audio both approaches look similar, with F1 around .82; once other groups' speech is overlaid, the overlap heuristic falls to .52, while the learned model stays at .75–.79 depending on whether segmentation is also degraded. The paper further argues that this gap shows interruptions carry linguistic and prosodic signatures beyond mere overlap, and that acoustic features hold up under noise better than text. This matters because AI classroom monitors that track many small groups at once need a definition of interruption that does not depend on a quiet room.","feed_headline":"Learned model beats overlap at spotting interruptions in noisy groups","feed_subtitle":"With ten groups talking at once, text plus prosody keeps F1 near .79; overlap alone drops to .52.","key_machinery":"The load-bearing object is the feature set: 512-dimensional BERT-small embeddings of ASR transcripts paired with 88 eGeMAPS acoustic features (loudness, spectral shape, voice length) extracted per utterance with openSMILE, fed to a random-forest classifier under leave-one-group-out cross-validation. The experiments vary segmentation and audio cleanliness independently to separate noise's effect on segmentation from its effect on feature quality. A second load-bearing object is the heuristic baseline: labeling any utterance whose Oracle timestamps overlap another utterance's start as an interruption, which isolates exactly what happens when overlap serves as the operational definition of interruption. The contrast between the baseline's collapse and the classifier's stability is what carries the argument.","core_discovery":"The paper's central claim is that a fully automated pipeline can identify human-annotated interruptions in collaborative dialogue even when recordings contain simultaneous speech from multiple groups, and that it does so by learning patterns beyond overlap. In the authors' experiments, interruption detection with combined textual and acoustic features reaches F1 of .819 on clean audio, .751 on noisy audio segmented noisily, and .792 on noisy audio with clean timestamps; the overlap-as-interruption baseline matches the clean result at .819 but falls to .524 in the noisy analogue. The authors conclude that interrupting speakers are marked by loudness and spectral-shape changes, not only by talking at the same time as someone else, and that retaining acoustic features is key to robustness.","pith_inferences":["Beyond the paper's experiments, the reliance on loudness features suggests real deployment would need per-room calibration: a classroom with high HVAC noise or distant groups could shift the acoustic distribution more than the model saw.","The findings imply interruption labels in multiparty corpora should be annotated independently of overlap status, since 103 interruptions in this data never overlap; re-labeling with overlap as a feature rather than a definition would let future work test cooperative versus competitive interruptions.","A testable extension is to feed the same feature pipeline audio from real multi-group recordings with speaker diarization and compare against the simulated overlay; equal performance would validate the simulation and support cocktail-party style evaluation tasks."],"forward_implications":["In a classroom with multiple concurrent groups, interruption detection should not be defined by overlapping speech; the paper's results put a lower bound of roughly .75 F1 for an automated pipeline under simulated multi-group noise.","Acoustic features such as loudness and spectral shape carry interruption signal that survives background speech, so future classroom dialogue systems should keep prosodic features even when ASR text is noisy.","Noise degrades automatic segmentation, producing longer, coarser utterances; the paper shows this segmentation hit is partly separable from feature-quality loss by using clean timestamps.","Interruptions remain the best-detected of the four CPS indicators under noisy audio, suggesting the phenomenon has distinctive acoustic-linguistic markers worth modeling separately.","Because the model never sees overlap features directly, its continued performance is evidence that learned representations encode interruption behavior beyond simultaneous speech."],"supporting_citations":[{"why":"Supplies the Weights Task Dataset of ten triads with recordings and human/ASR transcripts used for all experiments.","marker":"[9]"},{"why":"Provides the collaborative-problem-solving indicator framework, including the interruption labels the models predict.","marker":"[15]"},{"why":"Defines the segmentation and multimodal feature extraction pipeline that this paper follows, including the random-forest choice.","marker":"[3]"},{"why":"Defines the 88-feature eGeMAPS acoustic parameter set used for prosodic feature extraction.","marker":"[4]"},{"why":"Provides openSMILE, the tool that extracts the acoustic features from each utterance.","marker":"[5]"},{"why":"Supplies the BERT-small model that converts ASR transcriptions into 512-dimensional textual embeddings.","marker":"[17]"},{"why":"Provides the Google WebRTC voice activity detector used for automatic utterance segmentation.","marker":"[8]"}],"fun_headline_variants":["Overlap alone fails to spot interruptions in multi-group chatter","Text plus prosody beats overlap for interruptions in class noise","Interruption detection survives overlapping groups via text and prosody","Robust interruption ID in multi-group noise: text + prosody wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The noisy-condition results all rest on synthetic multi-group audio made by overlaying single-group recordings with spatial effects meant to imitate a shared room; if that simulation does not capture real classroom reverberation, geometry, and group spacing, the reported F1 scores may not transfer to deployment.","fun_headline_variants_meta":{"raw":{"variants":["Overlap alone fails to spot interruptions in multi-group chatter","Text plus prosody beats overlap for interruptions in class noise","Interruption detection survives overlapping groups via text and prosody","Robust interruption ID in multi-group noise: text + prosody wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001005,"raw_usage":{"total_tokens":4200,"prompt_tokens":847,"completion_tokens":3353,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":3283}},"tokens_in":463,"tokens_out":3353,"duration_ms":24903,"temperature":1.0,"reasoning_tokens":3283,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:44:23.825589+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record several real triads working simultaneously in one classroom with actual room acoustics, run the same pipeline, and compare interruption-detection F1 against the authors' .79 with clean timestamps and .75 with noisy segmentation; a drop near the overlap baseline's .52 would show the simulated noise overstated acoustic robustness. A lighter test is to measure whether the overlap heuristic degrades with real co-located groups as steeply as Figure 4 predicts.","supporting_citations":[{"cited_title":"Journal of open humanities data10(1) (2024)","cited_arxiv_id":null,"evidence_quote":"Supplies the Weights Task Dataset of ten triads with recordings and human/ASR transcripts used for all experiments."},{"cited_title":"Computers & Ed- ucation143, 103672 (Jan 2020)","cited_arxiv_id":null,"evidence_quote":"Provides the collaborative-problem-solving indicator framework, including the interruption labels the models predict."},{"cited_title":"In: International Conference on Artificial Intelligence in Education","cited_arxiv_id":null,"evidence_quote":"Defines the segmentation and multimodal feature extraction pipeline that this paper follows, including the random-forest choice."},{"cited_title":"IEEE Transactions on Affective Computing7(2), 190–202 (Apr 2016)","cited_arxiv_id":null,"evidence_quote":"Defines the 88-feature eGeMAPS acoustic parameter set used for prosodic feature extraction."},{"cited_title":"In: Proceedings of the 18th ACM international conference on Multimedia","cited_arxiv_id":null,"evidence_quote":"Provides openSMILE, the tool that extracts the acoustic features from each utterance."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Google WebRTC voice activity detector used for automatic utterance segmentation."}],"review_version":1}