{"id":"a95eca5b-a6b9-4d5a-b224-f6b42673e67d","arxiv_id":"2412.17295","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Friends-MMC is a multi-modal multi-party conversation dataset from the TV show Friends with face and speaker annotations, and baselines show that speaker identification benefits from combining visual and textual cues.","lead":"The authors introduce Friends-MMC, a new dataset of 24,000+ utterances from the TV show Friends with video, face tracks, and speaker labels, plus two tasks: speaker identification and response prediction. They show that a modular baseline combining visual and textual models outperforms fine-tuned multimodal LLMs on speaker identification, and that speaker information improves response prediction.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The face-labeling threshold is tuned on the test season, so training-data construction is not cleanly separated from test; the magnitude on reported accuracy is unquantified and should be settled by retuning on a held-out season.","rationale":"The reader's weakest assumption identifies the threshold selection on the test season as the central methodological flaw. I agree: the paper explicitly states t=0.6 'is set to maximize the validation accuracy described in the following paragraph,' and that validation is performed on Season 3 against C1C annotations—the same season used as the test set. This is a genuine train/test separation violation in the data-construction pipeline. The concern is load-bearing because the entire empirical contribution depends on the test set being a clean evaluation of the proposed baseline and comparisons. If t is retuned on a held-out season and the reported accuracy changes materially, the headline results (83.21% vs. LLaVA 46.30%, Emu 61.76%, GPT-4o 66.36%) could shift. Even if the effect is small, the lack of sensitivity analysis means the reader cannot verify the magnitude. I also note a secondary concern: the human-performance comparison is cross-setting (human saw only static frames and text, while the best model used video+audio+text), so 'rivaling human performance' is overstated as stated. However, the threshold issue is the more fundamental benchmark-integrity problem, and the reader's conditional verdict is appropriate. The proposed concrete test—retuning t on a non-test season and rerunning the baseline—would directly settle whether the concern lands. If the accuracy is stable, the paper could be accepted with a clarification; if not, the reported numbers need revision. Therefore I recommend no change to the reader's CONDITIONAL verdict.","tokens_in":14883,"tokens_out":9729,"duration_ms":101764,"concrete_test":"Rebuild the training set with t chosen on a held-out season (e.g., manually annotate a small validation split from Season 4, or use only train-season data to select t), keep the test set's human-annotated C1C labels unchanged, and rerun the full speaker-identification baseline (M1(CNN)+M2 and M1(TalkNet)+M2) on the 5-turn test set. If the accuracy drop is within random variation (<1 point), the threshold tuning is not load-bearing; if it drops by more than 2 points or changes the ranking against LLaVA/Emu, the original evaluation is compromised and the reported numbers must be re-estimated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The automatic face-labeling threshold t=0.6 is selected by maximizing face-labeling accuracy on Season 3, which is exactly the test season (see 'Automatic Face Labelling' and 'Test & Test-noisy Set'). Although the test set itself uses human-annotated C1C faces rather than auto-labels, t controls which face tracks are retained and labeled in the training set (Seasons 1, 2, and 4-10). Thus the training-data construction pipeline is tuned against the test-season distribution. If t is overfit to Season 3, the training set's face tracks/labels are not representative of a clean train split, and the downstream visual model M1 (CNN or TalkNet) may be indirectly adapted to test-season appearance. The paper reports no sensitivity analysis for t, so the magnitude of this effect is unknown. This is a benchmark-integrity issue: hyperparameter selection should be restricted to training-season data or a separate validation season; otherwise the reported 83.21% for M1(TalkNet)+M2 is not a clean evaluation. Additionally, the 'rivaling human' claim compares a video+audio+text model (83.21%) to a human study that only saw static frames+text (82.25%); this cross-setting comparison is not apples-to-apples and should be flagged, though it is secondary to the threshold concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Friends-MMC, a large multi-modal multi-party conversation dataset built from the TV series Friends, containing 24,000+ unique utterances paired with video clips, speaker labels, face bounding boxes, and face names. It proposes two tasks: conversation speaker identification and conversation response prediction. For speaker identification, the paper designs a modular baseline combining a visual model (CNN for frames or TalkNet for video), a same-speaker text model (DeBERTa-v3), and a quadratic binary optimization solver; it reports 83.21% accuracy on the 5-turn clean test set, outperforming fine-tuned LLaVA, Emu, and GPT-4o, and close to human performance. For response prediction, it fine-tunes Llama2-7B and Emu-14B and shows that speaker information, especially ground-truth speaker names, improves response selection accuracy.","tokens_in":15133,"tokens_out":6273,"duration_ms":55469,"significance":"The dataset fills a genuine gap: existing multi-modal dialogue datasets are mostly two-party and bystander-style, whereas Friends-MMC supports multi-party, situated conversation understanding with speaker and face annotations. The paper is transparent about its construction pipeline and releases code and data, which is commendable. The modular baseline is clearly described, and the ablations (M1 vs. M1+M2) are consistent with the claim that both visual and textual modalities contribute. The response-prediction analysis of speaker information, including random/shuffled controls, is a useful contribution. However, the benchmark-integrity issue with threshold tuning tempers the significance; if resolved, the dataset would be a solid and reusable resource.","major_comments":[{"comment":"The face-labeling threshold t = 0.6 is selected by maximizing the validation accuracy on Season 3, which is the same season used as the test set. Although the test set uses human-annotated C1C faces, t controls which face tracks are kept and labeled in the training set (Seasons 1, 2, 4–10). Tuning this threshold on the test-season distribution leaks information about the test set into training-data construction, potentially inflating the reported downstream accuracies (Table 2). Please retune t on a held-out season (or on training-season data only), report the sensitivity of the results to t, and re-evaluate the main claims with the resulting threshold. If the effect is small, that should be demonstrated quantitatively.","section":"Automatic Face Labelling / Test & Test-noisy Set"},{"comment":"The claim that M1(TalkNet)+M2 (83.21%) 'rivals' human performance (82.25%) is not an apples-to-apples comparison. The human study provided only static frames plus text (no audio or video), whereas TalkNet uses video and audio. The appropriate human comparison for the video+text setting would require a video/audio setting or a clear caveat. Please either temper the claim or report human performance under the same video+audio setting.","section":"Main Results / Table 2"},{"comment":"The statement 'regardless of which pre-trained model is used, or the length of dialogue context, in all cases adding speaker information always improves the performance' is contradicted by the Emu-14B row for 5 turns, where the 'Random' condition (29.35) is lower than 'No' (30.49). Please correct this claim or clarify that random speaker names are considered a form of speaker information that may not help.","section":"Experiment Results / Table 3"}],"minor_comments":[{"comment":"The objective is written as f(X) = (1−α)X^T A X + α X B; as written, X^T A X is l×l and X B is m×m, so the objective is not a scalar. Presumably the intended scalar objective uses traces (e.g., tr(X^T A X) + α tr(X B)). Please correct the equation to make the optimization problem well-defined.","section":"Quadratic Binary Optimization Problem Solver"},{"comment":"The abstract uses 'Friend-MMC' where 'Friends-MMC' is intended; please fix the typo.","section":"Abstract"},{"comment":"The appendix contains 'paramter-efficient'; it should be 'parameter-efficient'.","section":"Appendix / Details of Baseline Methods"},{"comment":"The notation 'LLaV A' appears with an unintended space; please use 'LLaVA' consistently throughout the paper.","section":"Table 2 caption"},{"comment":"There is a discrepancy between the main text, which says 80 dialogue sessions were sampled, and the appendix instructions, which refer to two folders of 20 pieces and annotation of 60.jpg–79.jpg. Please clarify the sampling and annotation procedure.","section":"Human Performance appendix"}],"recommendation":"major_revision","confidential_remarks":"The threshold-tuning issue is the key risk for this benchmark paper. The dataset and code release are valuable, but the evaluation protocol must be clean before acceptance. The other two major comments are also worth addressing, but they are secondary to the threshold concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Karen,\n\nThe short version: Friends-MMC is a genuinely useful new resource, and the paper is solid enough to referee, but the face-labeling threshold is tuned on the very season used for testing, and that needs to be fixed before I'd trust the headline numbers.\n\nWhat's actually new: a multi-party conversation dataset where every utterance has video, face tracks, face names, and a speaker label. That combination is absent from MELD, VSTAR, and OpenViDial, so it fills a real gap. The two tasks (speaker identification and response prediction) are sensible, and the baseline is a reasonable modular design: TalkNet/CNN for visual speaking-face detection, DeBERTa-v3 for same-speaker relations, and a quadratic solver to fuse them. The paper is clearly written, the ablations are consistent with the claims, and the code and data are public.\n\nThe main soft spot is the one the stress-test flagged. The threshold t=0.6 in automatic face labelling is explicitly set to maximize face-label accuracy on Season 3, and Season 3 is the test set. Even though the test set itself uses human-annotated faces, t controls which face tracks and labels enter the training set, so test-season information leaks into training-data construction. That's a benchmark-integrity issue, and the paper gives no sensitivity analysis, so we don't know how much it matters. The fix is straightforward: retune t on training seasons or a separate validation season and show the results barely move. That should be a required revision.\n\nTwo smaller issues. First, the 'rivaling human performance' claim compares a video+audio+text model (83.21%) to a human study that only saw static frames with text; that's not apples-to-apples. Second, the task forces prediction among faces in the candidate set, so a speaker who never appears cannot be predicted; that's a design choice, but it should be stated more prominently.\n\nI don't see any fatal flaws. The dataset is a genuine contribution, the baseline is reproducible, and the paper is honest about limitations in the appendix. The threshold issue is real but fixable, and the human comparison is easy to correct. This deserves a serious referee, not a desk reject. I'd recommend major revision, with the threshold retuning as the main condition.","headline":"Useful new multimodal dataset, but the face-label threshold is tuned on the test season; fix that and this is a solid contribution.","tokens_in":15688,"tokens_out":3112,"would_cite":true,"duration_ms":28502,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper builds Friends-MMC, a multi-modal multi-party conversation dataset from the TV series Friends, and shows that a modular baseline with a quadratic binary optimization solver identifies speakers at 83.21% accuracy, beating…","keywords":["multi-modal conversation","multi-party conversation","speaker identification","response prediction","active speaker detection","quadratic binary optimization","Friends dataset","situated dialogue"],"falsifier":"Re-run face labeling with the similarity threshold tuned on seasons 1-2 instead of season 3 and measure the 5-turn test accuracy on season 3; a material drop below 83.21% would show the reported result depends on tuning the training-data construction to the test season.","tokens_in":14679,"feed_emoji":"🎬","tokens_out":5653,"duration_ms":49331,"temperature":0.7,"pith_summary":"This paper introduces multi-modal multi-party conversation (MMC) as a research area and releases Friends-MMC, a dataset of more than 24,000 utterances from the TV series Friends, each paired with video, audio, speaker labels, face bounding boxes, and face names. It defines two tasks: conversation speaker identification and conversation response prediction. The paper claims that a modular baseline—visual active-speaker detection plus a text model for same-speaker relations plus a quadratic binary optimization solver—identifies the speaker of each turn at 83.21% accuracy on the clean 5-turn test set, beating fine-tuned multimodal pretrained models (LLaVA 46.30%, Emu 61.76%, GPT-4o 66.36%) and roughly matching human accuracy (82.25%). It also claims that adding speaker information, even when predicted automatically, improves response prediction in both text-only and multimodal generative models. The dataset and code are released to encourage further work.","feed_headline":"TV sitcom benchmark identifies speakers better than GPT-4o","feed_subtitle":"New Friends dataset links 24,000+ utterances to speakers and faces; the baseline hits 83% accuracy, near human 82%.","key_machinery":"The load-bearing object is the three-module baseline with the quadratic binary optimization solver at its center. From the visual model it builds a vision reward matrix $B \\in \\mathbb{R}^{l \\times m}$, where $B_{ij}$ is the probability that face $c_i$ is the speaking face for turn $u_j$; from the text model it builds a text reward matrix $A = p_{\\text{sim}} - \\text{mean}(p_{\\text{sim}})$, the mean-subtracted pairwise same-speaker probabilities. The solver maximizes $f(X) = (1-\\alpha)X^T A X + \\alpha X B$ over binary assignment matrices $X$, with the constraint that each turn gets exactly one speaker, and $\\alpha$ weighting the two modalities. The mean subtraction prevents the solver from assigning all turns to one speaker, and $\\alpha$ is chosen on a held-out validation set.","core_discovery":"The central claim is that speaker identification in situated multi-party dialogue can be solved accurately by modular component models plus global optimization, without end-to-end multimodal fusion. The baseline uses a visual model (TalkNet for video or a CNN for single frames) to score each visible face as the current speaker, a text model (DeBERTa-v3) to estimate whether each pair of utterances was spoken by the same person, and a quadratic binary optimization solver that assigns every turn exactly one speaker by balancing the two reward matrices. On the clean 5-turn test set this reaches 83.21% accuracy, above LLaVA-13B (46.30%), Emu-14B (61.76%), and GPT-4o (66.36%), and slightly above the measured human accuracy (82.25%). For response prediction, the paper shows that feeding ground-truth or automatically inferred speaker names into Llama2-7B and Emu-14B consistently improves the accuracy of selecting the true next utterance among ten candidates, and argues that the main benefit is speaker names acting as a local-context indicator rather than a global speaking style signal.","pith_inferences":["The paper leaves open whether the reported test accuracy survives a cleaner validation split: the face-labeling threshold t=0.6 was tuned on Season 3, which is also the test season, so the training-data construction may be indirectly optimized for the test distribution.","A natural extension would be to replace the discrete quadratic binary solver with a differentiable neural assignment layer, allowing end-to-end training of the visual and textual modules while preserving the global constraint.","The modular approach could transfer to real-world meeting analysis, where face tracks and dialogue overlap, though active-speaker detection in meetings is typically more challenging than in a sitcom with clear face tracks.","The response-selection evaluation, which picks one of ten candidates by perplexity, may reward shallow cues; a generative evaluation with human judgments could reveal whether the speaker-information gains persist in free-form response generation."],"forward_implications":["Speaker identification in situated multi-party conversation can be decomposed into per-face speaking probability and pairwise same-speaker relations, then solved globally, without requiring end-to-end multimodal pretrained fusion.","The text-based same-speaker model provides a meaningful boost, especially when the speaker is not visible in the current clip or frame, and the benefit grows when the visual model is stronger.","Including speaker information, even from an automatic speaker-identification system, improves response prediction in both text-only and multimodal generative models.","Speaker names act primarily as a local-context indicator, not a global speaking style tag, so models should use speaker identity to anchor whose perspective the next utterance takes.","The released dataset enables future work on character-centered dialogue understanding in a natural, situated, multi-party setting."],"supporting_citations":[{"why":"Supplies the C1C human-labeled face tracks for Season 3, used both as test ground truth and as the validation target for the automatic face-labeling threshold.","marker":"Kalogeiton and Zisserman 2020"},{"why":"Provides TalkNet, the active-speaker detection model used as the video visual model M1.","marker":"Tao et al. 2021"},{"why":"Provides DeBERTa-v3, the text encoder used in M2 to predict whether two utterances are spoken by the same person.","marker":"He, Gao, and Chen 2021"},{"why":"The Ubuntu Dialogue Corpus is used to pre-train the text model before fine-tuning on Friends-MMC.","marker":"Lowe et al. 2015"},{"why":"FaceNet-512 feature embeddings are used to construct character prototypes and to label face tracks automatically.","marker":"Schroff, Kalenichenko, and Philbin 2015"},{"why":"The S3FD face detector is used in video preprocessing to detect faces in each frame before track construction.","marker":"Zhang et al. 2017"}],"fun_headline_variants":["New Friends-MMC dataset: baseline beats GPT-4o on speaker ID","Optimization solver outranks GPT-4o on multi-party speaker ID","Friends-MMC: 24k utterances, faces, and a solver that tops LLMs","Speaker ID in multi-party chat: simple solver beats GPT-4o"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The automatic face-labeling threshold t=0.6 is chosen by validating on Season 3, the same season that forms the test set, so the reported test accuracy may be inflated by tuning training-data construction to the test distribution.","fun_headline_variants_meta":{"raw":{"variants":["New Friends-MMC dataset: baseline beats GPT-4o on speaker ID","Optimization solver outranks GPT-4o on multi-party speaker ID","Friends-MMC: 24k utterances, faces, and a solver that tops LLMs","Speaker ID in multi-party chat: simple solver beats GPT-4o"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000819,"raw_usage":{"total_tokens":3628,"prompt_tokens":1032,"completion_tokens":2596,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":2511}},"tokens_in":648,"tokens_out":2596,"duration_ms":17322,"temperature":1.0,"reasoning_tokens":2511,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:36:52.325665+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run face labeling with the similarity threshold tuned on seasons 1-2 instead of season 3 and measure the 5-turn test accuracy on season 3; a material drop below 83.21% would show the reported result depends on tuning the training-data construction to the test season.","supporting_citations":[{"cited_title":"S.; and Zisserman, A","cited_arxiv_id":null,"evidence_quote":"Supplies the C1C human-labeled face tracks for Season 3, used both as test ground truth and as the validation target for the automatic face-labeling threshold."},{"cited_title":"K.; Qian, X.; Shou, M","cited_arxiv_id":null,"evidence_quote":"Provides TalkNet, the active-speaker detection model used as the video visual model M1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Ubuntu Dialogue Corpus is used to pre-train the text model before fine-tuning on Friends-MMC."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FaceNet-512 feature embeddings are used to construct character prototypes and to label face tracks automatically."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The S3FD face detector is used in video preprocessing to detect faces in each frame before track construction."}],"review_version":1}