VIP-MINGLE: A Corpus for Videoconference and In-Person Multimodal Interaction in Group Language Engagement

Abhinay K Bodi; Andrew Chang; David Poeppel; Dhiwahar A Kennady; Dustin Freeman; Junrui Huang; Sumanth B H Karanam; Venu G Kadamba; Wenxin Deng

REVIEW 5 major objections 5 minor 35 references

VIP-MINGLE: A Corpus for Videoconference and In-Person Multimodal Interaction in Group Language Engagement

T0 review · 5 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Videoconferencing reshapes group conversation, not merely degrades it

desk verdict The corpus is the contribution; the behavioral-shift claims need a data audit before they carry weight. read the letter →

arxiv 2607.13614 v1 pith:DPRY2PKN submitted 2026-07-15 cs.HC cs.MM

Andrew Chang , Abhinay K Bodi , Wenxin Deng , Junrui Huang , Venu G Kadamba , Sumanth B H Karanam , Dhiwahar A Kennady , David Poeppel

show 1 more author

Dustin Freeman

This is my paper · ORCID

classification cs.HCcs.MM

keywords multimodalcorpusgroupconversationvideoconferencingin-personinteractionturn-takingspeakerdiarizationfacialexpressiondomainshift

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VIP-MINGLE is a new corpus built to answer a deceptively simple question: when the same people do the same collaborative task face-to-face and over a video link, what actually changes? The paper's claim is that the change is substantial and qualitative. Videoconference sessions show longer turn-taking gaps, shorter utterances, syntactically simpler speech, and less salient facial expressions overall, though a few localized expressions increase. Human raters also find in-person sessions more enjoyable, even though the two settings are rated equally fluid. If right, this means communication-medium effects can be isolated from speaker and task confounds for the first time in an open dataset, and it warns that models built on one setting will not silently transfer to the other.

What carries the argument

The engine of the design is within-subject pairing: the same group, the same task, two settings. Because speaker identity, group chemistry, and task demands are held constant, any systematic difference between the two sessions can be attributed to the communication medium. Supporting this are per-participant signal-processing pipelines — speaker diarization to separate individual voices, speech transcription with word-level timestamps, facial-action-unit and emotion extraction from video, and human annotation of 10-second clips — all aligned to a common timeline. The turn-taking metrics specifically come from a standard model of conversation timing applied to diarized speech, so the quality

What would settle it

Take a random sample of, say, 30 in-person and 30 videoconference sessions from the released corpus, have human annotators mark true speaker turns on the raw audio, and compute the diarization error rate separately for each setting. If the in-person error rate is dramatically higher and re-estimating standard turn-taking gaps and utterance durations from the human labels eliminates the significant differences, the central temporal claim is refuted. A second falsifier: check whether the higher in-person interruption rate survives human annotation, since the annotation study already measured int

Watch

Extended reading notes

Core claim

The central discovery is a corpus plus a set of paired comparisons. The corpus records 32 groups (105 participants) in both settings on the same trivia-style collaborative task, with order counterbalanced, yielding about 59 hours of raw audio/video, per-participant processed features, psychometric baselines, and 7,077 segment-level human ratings. The analysis shows that switching from in-person to videoconferencing shifts entire distributions: turn-taking gaps lengthen, utterance durations shrink, dependency distance — a syntactic-complexity measure — drops, and facial-expression profiles change qualitatively, not just in amplitude. Ratings show lower enjoyment but unchanged fluidity in vide

Load-bearing premise

The in-person individual audio tracks are assumed to be correctly separated by automated speaker diarization even though the microphones were closer to interfering speakers than the usual 3:1 guideline recommends, and no error rate for that separation is reported; if the separation is unreliable during overlapping speech, the measured differences in turn-taking gaps and utterance durations could be artifacts of the separation software rather than real effects of the communica

Editorial extensions

If this is right

Models trained on traditional in-person meeting corpora will likely need adaptation before they work on videoconference data; the corpus gives a benchmark for measuring that domain shift.
The longer gaps and shorter turns in video suggest that videoconferencing systems could improve perceived engagement by adjusting for turn-taking delays or making interruption cues more visible.
Because in-person conversation was rated more enjoyable despite more interruptions, 'clean' remote exchanges may be missing interactional dynamics that users value; interface design should consider restoring them.
The released raw recordings let others extract novel features (e.g., spatial cues from the 360° camera) to test whether the observed shifts replicate with newer models.
The significant behavioral differences across modalities motivate building multimodal, domain-aware models rather than assuming one-size-fits-all conversation understanding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct validation step the paper leaves implicit: compute diarization error rates on a hand-labeled subset of the in-person recordings. If errors are concentrated in overlapping speech, the reported gap/utterance differences could be an artifact of speaker-boundary estimation rather than a medium effect.
The 360° in-person video is a hidden asset: it can be used to test whether the lower enjoyment in video is driven by reduced gaze alignment or mutual attention, an explanation the paper does not pursue.
The syntactic-complexity drop without a change in lexical diversity suggests the effect may be about on-line production under video latency rather than vocabulary; a follow-up could examine disfluencies and filler words in the transcripts.
The equal-fluidity, lower-enjoyment dissociation implies that enjoyment is carried by non-temporal channels — expression intensity, gaze, or interruption patterns — which the corpus is well suited to disentangle in a mediation analysis.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a corpus paper, and the corpus is the contribution. VIP-MINGLE looks like the first open, paired within-subject dataset of group conversation in both in-person and videoconference settings, with ~59 hours of raw audio/video, processed features, psychometrics, and 7,077 annotated segments. If the release matches the description, that fills a real gap: AMI/ICSI are in-person only, CANDOR/RoomReader are remote only, and cross-setting comparisons out there use unpaired or unavailable data.

The design is genuinely good: same groups do the same task in both settings, order counterbalanced, individual laptop audio plus lavalier mics, raw data and scripts on Zenodo. The self-citations aren't circular; they supply annotation criteria and modeling context rather than the outcome. The behavioral findings also replicate known directions—longer gaps, shorter utterances, simpler syntax in videoconference—so the paper is consistent with prior work rather than inventing a new effect.

Where it gets soft is measurement quality. Section 3.1's central claims come from pyannote speaker diarization applied to in-person recordings where the target-to-interfering distance ratio was below the 3:1 rule of thumb and crosstalk was only mitigated by manual gain reduction. No diarization error rate is reported. In multiparty recordings with overlap, diarization errors systematically bias utterance duration and gap distributions, and since the videoconference audio is cleanly separated by room, any error asymmetry between settings will masquerade as a setting effect. The authors don't flag this in the limitations, which is a real omission. The good news is they release raw audio, so this is an audit problem rather than an unfixable one—reviewers can demand a DER estimate on a held-out subset, or the authors can add it.

There are smaller issues in the same spirit: no WER for Whisper transcripts, no inter-annotator reliability for the annotations despite mention of excluding low-reliability annotators, uncorrected multiple comparisons over the AU/emotion tests, and a causal flourish in Section 4 that connects gaps/interruptions to 'Zoom fatigue'—the design can't support that. None of these undercut the corpus's value; they undercut the strength of the exploratory claims.

Who's this for? People building or evaluating conversational AI across settings, and researchers studying videoconferencing effects. A serious referee should see it, with the expectation that measurement-quality audits are added; the release-with-data means those audits can actually be done.

Referee Report

5 major / 5 minor

Summary. The paper presents VIP-MINGLE, a multimodal corpus of 32 groups and 105 participants who each took part in paired in-person and videoconference sessions of a standardized collaborative game. The dataset release includes raw audio/video, psychometric baselines, processed features (pyannote diarization, Whisper transcripts, OpenFace/DeepFace facial features), and crowd-sourced segment-level annotations. The authors report exploratory comparisons showing longer turn-taking gaps, shorter utterance durations, lower syntactic complexity, altered facial-expression patterns, and lower enjoyment in videoconference sessions, using these differences to argue that videoconferencing is a qualitatively different communicative medium rather than a degraded version of in-person interaction.

Significance. If the corpus and its processing pipeline are validated, VIP-MINGLE is a valuable community resource. Its main strength is the within-subject design: the same groups perform the same task in both settings, which removes many speaker- and task-level confounds present in unpaired cross-setting comparisons. The decision to release raw recordings in addition to processed features is also commendable, because it makes the automatic-pipeline outputs auditable and re-computable. The key caveat is that the central behavioral-shift claims rest on automatic diarization, transcription, and annotation whose measurement quality is not documented in the manuscript. Since the raw data are available, this is a fixable validation gap rather than a fatal design flaw, but it must be addressed before the reported comparisons can be interpreted as evidence about communication-medium effects.

major comments (5)

[§2.3.1, §3.1] The central speech-timing effects (longer turn-taking gaps and shorter utterance durations in videoconference, Fig. 2A–B) are computed from pyannote diarization of in-person recordings under conditions explicitly described as violating the 3:1 target-to-interferer rule. No diarization error rate (DER), precision/recall against manual segmentation, or per-speaker overlap statistics are reported. Because the videoconference condition is intrinsically room-separated, any systematic diarization error in the in-person condition—such as missed short utterances or split/merged turns—will shift the gap and duration distributions and can masquerade as a setting effect. The authors should provide DER on a held-out subset, overlap statistics, and a comparison of automatically derived turn boundaries against manual annotations. This is necessary to support the §3.1 claim.
[§2.3.2, §3.2] The language-complexity analysis in §3.2 uses Whisper transcripts as the basis for MDD and other text metrics, but no word error rate (WER) or transcript-quality assessment is reported. Moreover, the speaker-attributed transcripts are obtained by aligning Whisper segments to diarized speaker labels; misalignment or diarization errors will add noise or bias to per-speaker text. The authors should report WER on a sampled subset and describe how overlapping speech and alignment failures were handled. This is particularly relevant to the MDD difference in Fig. 2C, which is the only significant language-complexity result.
[§4] The annotation analysis relies on 7,077 clips rated by 192 annotators, with only a reference to prior work [10] for excluding low-reliability annotators. No inter-annotator agreement coefficients are reported for the enjoyment ratings or for the multi-label event annotations (interruptions, gaps), and the number/percentage of excluded annotators is not given. Without these, the enjoyment and event-rate comparisons in Fig. 3 are not independently verifiable. Please report Krippendorff's alpha or ICC for the Likert ratings and per-label agreement for the event annotations, along with the exclusion criteria and counts.
[§2] The headline corpus-scale numbers are internally inconsistent. Thirty-two groups × two settings × an average session duration of 21 minutes amounts to approximately 22.4 hours of wall-clock session recordings, not the stated ≈59 hours. If 'recording hours' counts individual audio/video tracks or streams rather than session time, that should be stated explicitly, and the relationship between 'average session duration' and 'recording hours' should be defined. This statistic is central to the corpus's contribution and must be unambiguous.
[§3.3, Fig. 2D–E] The facial-expression comparisons test a large number of action units and emotion categories with uncorrected Wilcoxon signed-rank tests. Given the multiple comparisons, the few significant differences could include false positives. The exploratory framing in the text is helpful, but the abstract's claim of 'significant behavioral distribution shifts' would be better supported by reporting multiple-comparison correction (or explicitly acknowledging the uncorrected exploratory nature) and effect sizes for the significant AUs/emotions.

minor comments (5)

[§2.3.2] Minor formatting: '16 kHZ' should be '16 kHz.' Also, §2.3.1 mentions encoding diarized audio as 'mono, 32 kHz audio-only MP4 files,' while §2.3.2 says audio was standardized to 16 kHz mono WAV; please clarify the final format(s) and sampling rates shipped in the corpus.
[§3.1] The mixed-effects models are described only by β, SE, and p. Please report the model specification explicitly—random effects (group/participant/session), fixed effects, and how order/counterbalancing was entered—so the coefficients are interpretable and reproducible.
[§4] The annotation procedure says annotators viewed 120 clips each, but it is unclear how this maps to 7,077 rated clips across 192 annotators. A sentence describing the annotation design (how many clips per segment, overlap, quality-control clips) would improve transparency.
[Fig. 3] The event-label term 'gaps' in the annotation task should be defined consistently with the Heldner–Edlund gap measure used in §3.1; it is currently unclear whether they refer to the same construct or to a coarser subjective judgment.
[References] Reference [22] is listed as an 'Advanced Online Publication' from 2026; please verify the publication year and provide the final DOI if available.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: VIP-MINGLE is a corpus/observation paper; its behavioral-shift findings are direct empirical contrasts, not outputs derived from fitted inputs or self-cited uniqueness claims.

full rationale

The paper's chain is data collection → processing (pyannote diarization, Whisper, OpenFace, DeepFace) → pairwise statistical comparisons. There is no fitted model parameter that is later reported as a prediction; the Section 3/4 results (longer gaps, shorter utterances, lower syntactic complexity, changed AU patterns, lower enjoyment) are directly measured from the released recordings and annotations. Self-citations [10,11] motivate multimodal modeling and supply an annotator-reliability exclusion criterion in Section 4, but they do not determine the outcome values or the sign/size of the setting contrasts; the actual ratings and features are external to those citations. The main validity risk—in-person speaker diarization without a reported DER under sub-3:1 target-to-interferer ratio (Section 2.3.1)—is a measurement-error concern, not a circularity: it is testable because raw audio is released. No equation or definition makes an output equal to an input, no parameter is fit then renamed, and no uniqueness theorem is imported from the authors' prior work. The paper is therefore self-contained as a corpus contribution and its analyses are not circular.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The corpus claim rests on empirical collection and measurement; there are no fitted free parameters. The main burdens are unvalidated assumptions about the feature-extraction tools, the naturalness of the recorded in-person condition, and annotator reliability.

assumptions (5)

domain assumption Pre-trained feature extractors (pyannote diarization, Whisper large, OpenFace, DeepFace) yield valid participant-level measurements on this corpus without further validation.
Sections 2.3.1–2.3.3 use these tools as ground truth; no DER, WER, or frame-level validation is reported, so any tool error propagates into the comparative statistics.
domain assumption Recording the in-person condition with laptops, webcams and microphones (via Zoom) does not alter behavior relative to naturally occurring in-person interaction.
Section 2.2 describes dimmed screens and muted speakers, but the physical presence of recording devices and the same platform in both conditions could impose remote-like constraints; no manipulation check is reported.
domain assumption The counterbalanced within-subject design fully controls for order effects, practice, and task repetition.
Each group performs the identical task twice; although order is counterbalanced, no analysis of session-order effects or carryover is presented (Sections 2.1, 3).
domain assumption The 7,077 human annotations are reliable despite excluding low-reliability annotators.
Section 4 reports exclusion of low-reliability annotators referencing [10] but provides no inter-annotator agreement, ICC, or kappa in this paper.
domain assumption The Family Feud task elicits representative group conversation in both settings.
Acknowledged in Section 5 as a limitation; internal comparisons are still valid, but external generalization is assumed in the framing.

how reviews work

0 comments

Cite this review

Pith. "Pith review of VIP-MINGLE: A Corpus for Videoconference and In-Person Multimodal Interaction in Group Language Engagement." pith.science (2026). https://pith.science/paper/DPRY2PKN

@misc{pith2026260713614,
  author       = {Pith},
  title        = {Pith review of: VIP-MINGLE: A Corpus for Videoconference and In-Person Multimodal Interaction in Group Language Engagement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DPRY2PKN}},
  note         = {Machine review of arXiv:2607.13614}
}

read the original abstract

Group conversations are a fundamental yet complex form of social interaction central to human cognition and telecommunication technology. While understanding and facilitating these interactions has been a long-standing goal, findings are often isolated within specific in-person or videoconferencing settings due to a scarcity of datasets that bridge the two. We introduce VIP-MINGLE, a multimodal dataset comprising 59 hours of recordings (32 groups, 105 participants), featuring paired within-subject sessions in both settings. The dataset includes raw audio/video, psychometric data, processed multimodal features (e.g., diarized speech, facial expressions, transcriptions), and time-resolved human annotations. Our analysis reveals significant behavioral distribution shifts across multiple modalities between settings, reinforcing the need for a cross-setting corpus. VIP-MINGLE serves as a critical resource for developing robust models of group conversations across settings.

Figures

Figures reproduced from arXiv: 2607.13614 by the authors.

**Figure 1.** Schematic of the VIP-MINGLE data, signal processing pipeline and multimodal features across in-person and videoconference settings. Solid boxes: provided raw recordings and processed features; Dashed box: derivable features. 2.1. Participants, Procedure and Task Participants (age: 17–28; gender: 29 men, 69 women, 7 other) were recruited from New York University. Participants received course credit for their time, a… view at source ↗

**Figure 2.** Pairwise behavioral feature comparisons between settings: (A, B) speech temporal features, (C) language (syntactic) complexity, and (D, E) facial expressions (note: OpenFace does not support intensity data for AU28). The results show a broad spectrum of substantial behavioral shifts. Specifically, videoconferencing elicits shorter, less complex speech with longer pauses. Facially, it is associated with generally l… view at source ↗

**Figure 3.** Human-rated conversational qualities and annotated events. Ratings and annotations were performed at the segment level and averaged per session for statistical analysis. (A) In-person sessions scored significantly higher in enjoyment than videoconferencing, despite no significant difference in conversational fluidity. (B) In-person sessions exhibited a higher frequency of interruptions and gaps, suggesting that such… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 1 canonical work pages

[10]

Zoom exhaustion & fatigue scale,

G. Fauville, M. Luo, A. C. Queiroz, J. N. Bailenson, and J. Han- cock, “Zoom exhaustion & fatigue scale,”Computers in Human Behavior Reports, vol. 4, p. 100119, 2021

2021
[1]

Videoconferencing plat- forms have evolved from occasional alternatives into a primary medium for professional collaboration, education, and social in- teraction

Introduction & Relevant Works The rapid digitization of the workplace has fundamentally transformed human communication. Videoconferencing plat- forms have evolved from occasional alternatives into a primary medium for professional collaboration, education, and social in- teraction. As this hybrid reality stabilizes, there is a need for systems that can u...
[2]

Family Feud

VIP-MINGLE Corpus Unique to VIP-MINGLE is its within-subject design. Every group participates in two distinct communication settings: in- person and videoconference (remote), allowing for direct intra- group comparison across modalities. It includes both group- level and participant-level multimodal features (e.g., raw au- dio/video, diarized speech, tran...

arXiv 2026
[3]

Even within a representative subset of all derivable features, our exploratory results show substantial behavioral changes rather than mere scaling differ- ences

Comparing In-Person and Videoconference Multimodal Features To investigate whether videoconferencing simply degrades in- person communication or features a substantially different be- havioral pattern, we conducted pairwise comparisons of multi- modal features between settings. Even within a representative subset of all derivable features, our exploratory...
[4]

Rated Conversational Interactions & Annotated Events Videos were segmented into 10-second clips and annotated by 192 qualified annotators on Qualtrics (after excluding those with low reliability [10]), with each annotator viewing 120 clips. Participants were crowdsourced from New York Univer- sity for course credits (aged 18–25; 121 women, 44 men, 27 othe...
[5]

Specifically, videoconferencing elicits shorter, syntac- tically simpler speech with longer pauses

Conclusion & Limitations Our exploratory analyses highlight the need for having a paired conversation corpus, as the conversational behaviors (both mul- timodal features and annotations) vary substantially across set- tings. Specifically, videoconferencing elicits shorter, syntac- tically simpler speech with longer pauses. In terms of facial expressions, ...
[6]

The authors remain fully responsible for the accuracy and integrity of the final content

Generative AI Use Disclosure Generative AI tools were used to assist with data analysis code and manuscript editing. The authors remain fully responsible for the accuracy and integrity of the final content
[7]

are supported by NYU Discovery Research Fund for Human Health

Acknowledgments A.C., D.P., and D.F. are supported by NYU Discovery Research Fund for Human Health. A.C. is supported by Leon Levy Schol- arships in Neuroscience, Leon Levy Foundation and New York Academy of Sciences. The funders have no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. This work wa...

Show all 35 references

[8]

The AMI meet- ing corpus,

W. Kraaij, T. Hain, M. Lincoln, and W. Post, “The AMI meet- ing corpus,” inProc. International Conference on Methods and Techniques in Behavioral Research, 2005, pp. 1–4

2005
[9]

The ICSI meet- ing corpus,

A. Janin, D. Baron, J. Edwards, D. Ellis, D. Gelbart, N. Morgan, B. Peskin, T. Pfau, E. Shriberg, A. Stolckeet al., “The ICSI meet- ing corpus,” in2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP’03)., vol. 1. IEEE, 2003...

2003
[11]

Zoom disrupts the rhythm of conversation

J. E. Boland, P. Fonseca, I. Mermelstein, and M. Williamson, “Zoom disrupts the rhythm of conversation.”Journal of Experi- mental Psychology: General, vol. 151, no. 6, p. 1272, 2022

2022
[12]

Video-conferencing usage dynamics and nonverbal mech- anisms exacerbate Zoom Fatigue, particularly for women,

G. Fauville, M. Luo, A. Queiroz, A. Lee, J. Bailenson, and J. Han- cock, “Video-conferencing usage dynamics and nonverbal mech- anisms exacerbate Zoom Fatigue, particularly for women,”Com- puters in Human Behavior Reports, vol. 10, p. 100271, 2023

2023
[13]

The CANDOR corpus: Insights from a large multimodal dataset of naturalistic conversation,

A. Reece, G. Cooney, P. Bull, C. Chung, B. Dawson, C. Fitz- patrick, T. Glazer, D. Knox, A. Liebscher, and S. Marin, “The CANDOR corpus: Insights from a large multimodal dataset of naturalistic conversation,”Science Advances, vol. 9, no. 13, p. eadf3197, 2023

2023
[14]

RoomReader: A multimodal corpus of online multiparty conversational interactions,

J. Reverdy, S. O’Connor Russell, L. Duquenne, D. Garaialde, B. R. Cowan, and N. Harte, “RoomReader: A multimodal corpus of online multiparty conversational interactions,” inPro- ceedings of the Thirteenth Language Resources and Evaluation Conference. Marseille, France: Europea...

2022
[15]

A corpus study on the difference of turn-taking in online audio, online video, and face-to-face conver- sation,

Y . Tian, S. Liu, and J. Wang, “A corpus study on the difference of turn-taking in online audio, online video, and face-to-face conver- sation,”Language and Speech, vol. 67, no. 3, pp. 593–616, 2024

2024
[16]

Virtual (zoom) interactions alter conversational behavior and in- terbrain coherence,

S. Balters, J. G. Miller, R. Li, G. Hawthorne, and A. L. Reiss, “Virtual (zoom) interactions alter conversational behavior and in- terbrain coherence,”Journal of Neuroscience, vol. 43, no. 14, pp. 2568–2578, 2023

2023
[17]

Multimodal machine learning can predict videoconference fluidity and enjoyment,

A. Chang, V . Akkaraju, R. M. Cogliano, D. Poeppel, and D. Free- man, “Multimodal machine learning can predict videoconference fluidity and enjoyment,” inICASSP 2025 - 2025 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025, pp. 1–5

2025
[18]

Multimodal fusion with semi-supervised learning minimizes annotation quantity for modeling videoconfer- ence conversation experience,

A. Chang, C. Hu, J. Qi, Z. Wei, K. Zhang, V . Akkaraju, D. Poep- pel, and D. Freeman, “Multimodal fusion with semi-supervised learning minimizes annotation quantity for modeling videoconfer- ence conversation experience,” inProc. Interspeech 2025, 2025, pp. 4313–4317

2025
[19]

Multimodal, multiparty modeling of collaborative problem solv- ing performance,

S. K. Subburaj, A. E. Stewart, A. Ramesh Rao, and S. K. D’Mello, “Multimodal, multiparty modeling of collaborative problem solv- ing performance,” inProceedings of the 2020 International Con- ference on Multimodal Interaction, 2020, pp. 423–432

2020
[20]

Multimodal model- ing of collaborative problem-solving facets in triads,

A. E. Stewart, Z. Keirn, and S. K. D’Mello, “Multimodal model- ing of collaborative problem-solving facets in triads,”User Mod- eling and User-Adapted Interaction, vol. 31, no. 4, pp. 713–751, 2021

2021
[21]

Turn-taking in human face-to-face interaction is multimodal: gaze direction and manual gestures aid the coordination of turn transitions,

K. H. Kendrick, J. Holler, and S. C. Levinson, “Turn-taking in human face-to-face interaction is multimodal: gaze direction and manual gestures aid the coordination of turn transitions,”Philo- sophical Transactions of the Royal Society B, vol. 378, no. 1875, p. 20210473, 2023

2023
[22]

A very brief measure of the big-five personality domains,

S. D. Gosling, P. J. Rentfrow, and W. B. Swann Jr, “A very brief measure of the big-five personality domains,”Journal of Research in personality, vol. 37, no. 6, pp. 504–528, 2003

2003
[23]

Development and validation of brief measures of positive and negative affect: the PANAS scales

D. Watson, L. A. Clark, and A. Tellegen, “Development and validation of brief measures of positive and negative affect: the PANAS scales.”Journal of Personality and Social Psychology, vol. 54, no. 6, p. 1063, 1988

1988
[24]

Modeling collaborative mul- timodal behavior in group dialogues: The MULTISIMO corpus,

M. Koutsombogera and C. V ogel, “Modeling collaborative mul- timodal behavior in group dialogues: The MULTISIMO corpus,” inProceedings of the Eleventh International Conference on Lan- guage Resources and Evaluation (LREC 2018), 2018

2018
[25]

pyannote.audio 2.1 speaker diarization pipeline: prin- ciple, benchmark, and recipe,

H. Bredin, “pyannote.audio 2.1 speaker diarization pipeline: prin- ciple, benchmark, and recipe,” inProc. INTERSPEECH 2023, 2023

2023
[26]

Powerset multi-class cross entropy loss for neural speaker diarization,

A. Plaquet and H. Bredin, “Powerset multi-class cross entropy loss for neural speaker diarization,” inProc. INTERSPEECH 2023, 2023

2023
[27]

Robust speech recognition via large- scale weak supervision,

A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large- scale weak supervision,” 2022. [Online]. Available: https: //arxiv.org/abs/2212.04356

2022 arXiv
[28]

Open- face 2.0: Facial behavior analysis toolkit,

T. Baltrusaitis, A. Zadeh, Y . C. Lim, and L.-P. Morency, “Open- face 2.0: Facial behavior analysis toolkit,” in2018 13th IEEE In- ternational Conference on Automatic Face & Gesture Recognition (FG 2018), 2018, pp. 59–66

2018
[29]

Boosted lightface: A hybrid dnn and gbm model for boosted facial recognition,

S. I. Serengil and A. Ozpinar, “Boosted lightface: A hybrid dnn and gbm model for boosted facial recognition,”Gazi University Journal of Science, no. Advanced Online Publication, 2026. [Online]. Available: https://dergipark.org.tr/en/pub/gujs/article/ 1794891

2026
[30]

Pauses, gaps and overlaps in conver- sations,

M. Heldner and J. Edlund, “Pauses, gaps and overlaps in conver- sations,”Journal of Phonetics, vol. 38, no. 4, pp. 555–568, 2010

2010
[31]

Textdescriptives: A python package for calculating a large variety of metrics from text,

L. Hansen, L. R. Olsen, and K. Enevoldsen, “Textdescriptives: A python package for calculating a large variety of metrics from text,”Journal of Open Source Software, vol. 8, no. 84, p. 5153,
[33]

LexicalRichness: A small module to compute textual lexical richness,

L. Shen, “LexicalRichness: A small module to compute textual lexical richness,” 2022. [Online]. Available: https: //github.com/LSYS/lexicalrichness

2022
[34]

minicons: Enabling flexible behavioral and represen- tational analyses of transformer language models,

K. Misra, “minicons: Enabling flexible behavioral and represen- tational analyses of transformer language models,”arXiv preprint arXiv:2203.13112, 2022

2022 arXiv
[35]

Nonverbal overload: A theoretical argument for the causes of Zoom fatigue

J. N. Bailenson, “Nonverbal overload: A theoretical argument for the causes of Zoom fatigue.”Technology, Mind, and Behavior, vol. 2, no. 1, p. 1, 2021

2021
[2023]

Available: https://doi.org/10.21105/joss.05153

[Online]. Available: https://doi.org/10.21105/joss.05153

doi

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [10]

Zoom exhaustion & fatigue scale,

G. Fauville, M. Luo, A. C. Queiroz, J. N. Bailenson, and J. Han- cock, “Zoom exhaustion & fatigue scale,”Computers in Human Behavior Reports, vol. 4, p. 100119, 2021

2021

[2] [1]

Videoconferencing plat- forms have evolved from occasional alternatives into a primary medium for professional collaboration, education, and social in- teraction

Introduction & Relevant Works The rapid digitization of the workplace has fundamentally transformed human communication. Videoconferencing plat- forms have evolved from occasional alternatives into a primary medium for professional collaboration, education, and social in- teraction. As this hybrid reality stabilizes, there is a need for systems that can u...

[3] [2]

Family Feud

VIP-MINGLE Corpus Unique to VIP-MINGLE is its within-subject design. Every group participates in two distinct communication settings: in- person and videoconference (remote), allowing for direct intra- group comparison across modalities. It includes both group- level and participant-level multimodal features (e.g., raw au- dio/video, diarized speech, tran...

arXiv 2026

[4] [3]

Even within a representative subset of all derivable features, our exploratory results show substantial behavioral changes rather than mere scaling differ- ences

Comparing In-Person and Videoconference Multimodal Features To investigate whether videoconferencing simply degrades in- person communication or features a substantially different be- havioral pattern, we conducted pairwise comparisons of multi- modal features between settings. Even within a representative subset of all derivable features, our exploratory...

[5] [4]

Rated Conversational Interactions & Annotated Events Videos were segmented into 10-second clips and annotated by 192 qualified annotators on Qualtrics (after excluding those with low reliability [10]), with each annotator viewing 120 clips. Participants were crowdsourced from New York Univer- sity for course credits (aged 18–25; 121 women, 44 men, 27 othe...

[6] [5]

Specifically, videoconferencing elicits shorter, syntac- tically simpler speech with longer pauses

Conclusion & Limitations Our exploratory analyses highlight the need for having a paired conversation corpus, as the conversational behaviors (both mul- timodal features and annotations) vary substantially across set- tings. Specifically, videoconferencing elicits shorter, syntac- tically simpler speech with longer pauses. In terms of facial expressions, ...

[7] [6]

The authors remain fully responsible for the accuracy and integrity of the final content

Generative AI Use Disclosure Generative AI tools were used to assist with data analysis code and manuscript editing. The authors remain fully responsible for the accuracy and integrity of the final content

[8] [7]

are supported by NYU Discovery Research Fund for Human Health

Acknowledgments A.C., D.P., and D.F. are supported by NYU Discovery Research Fund for Human Health. A.C. is supported by Leon Levy Schol- arships in Neuroscience, Leon Levy Foundation and New York Academy of Sciences. The funders have no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. This work wa...