{"id":"23df23b2-c96d-402e-9c72-122ecb25fac7","arxiv_id":"2501.11269","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"On the new XMP parallel multi-party dialogue dataset, current multilingual LLMs generate low-quality responses; adding more or mixed-language training data yields only marginal, often negative, cross-lingual transfer.","lead":"A new dataset of podcast conversations in six languages tests whether multilingual AI models can handle multi-party dialogue. The models perform poorly, and fine-tuning with several languages mixed usually helps less than expected or hurts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"R3's 'multilingual mixing is detrimental' claim is confounded: Table 5 does not hold training budget, optimizer steps, or target-language exposure constant, and reports no variance; a controlled reproduction is needed.","rationale":"Reader's rejection centers on R3's confound, and I agree. The paper's empirical contribution is XMP; the claims are R1-R3. R3 is the most surprising and most directly contradicts previous work, so it is the load-bearing element of the paper's significance. But Table 5 compares 'single' and 'mixed' fine-tuning without specifying how total data size, number of steps, LR schedule, or target-language proportion were matched. The Appendix E command shown sets num_train_epochs=2.0 on one dataset; no mixed-run command is included. If mixed runs were concatenated, total data and steps double; if steps were kept fixed, target-language examples halve. Both possibilities allow the observed 8B drop to be an artifact of optimization rather than evidence of cross-lingual interference. The 70B results, with many deltas under 0.5, lack seeds and variance and cannot support 'mixing is usually detrimental' either. I am not claiming the conclusion is false; it is unsupported by the reported design. The concrete test above would settle it. I also note R1 relies on absolute low scores without a dyadic or human baseline, but the R3 confound is the most decisive issue for the headline claim. Verdict should remain REJECT, so I mark UNCHANGED.","tokens_in":20032,"tokens_out":8183,"duration_ms":85707,"concrete_test":"Reproduce the Table 5 matrix for LLaMA-3.1-8B with three matched conditions per target language: (1) single-language SFT on the target split; (2) mixed SFT on target+aux with total optimizer steps equal to (1), achieved by subsampling or epoch adjustment; (3) monolingual SFT on the target split upsampled or duplicated to the same total steps, which isolates the effect of additional same-language data. Use at least 5 seeds, identical LR/warmup/batch, and report mean and 95% CI for F1, BLEU-4, and ROUGE-L. If condition (2) is not consistently below conditions (1) and (3), R3's 'mixing is detrimental' claim fails. Optionally repeat for 70B to test whether the reported mitigation is real.","verdict_should_be":"UNCHANGED","load_bearing_attack":"R3 (the claimed reversal of multilingual complementarity) is the paper's most surprising finding, and Table 5 is its only support. The single-language vs mixed-language SFT comparison assumes that language composition is the only changed variable. The paper does not report how mixed datasets were assembled or how optimization was matched: the only training command fixes num_train_epochs=2.0 for one dataset, and mixed runs (e.g., DE+FR) are not specified. If mixed training concatenates splits and keeps epochs fixed, the mixed model sees roughly twice the data and twice the optimizer steps with a different cosine schedule; if optimizer steps are fixed instead, the target language contributes half as many examples. Both scenarios confound language mixing with training-budget and data-exposure effects. The consistent 8B degradation in Table 5 is therefore equally compatible with underfitting or optimization mismatch as with cross-lingual interference, and the scattered 70B deltas (mostly <0.5) are within plausible seed-to-seed noise, which the paper never reports. Without a controlled comparison, R3's 'detrimental' conclusion is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces XMP, a parallel multilingual multi-party dialogue dataset built by translating English podcast transcripts into Chinese, Japanese, German, French, and Italian, with about 312k conversations. It evaluates Qwen-2.5 7B and LLaMA-3.1 8B/70B on a response-generation task and claims three findings: R1, that MLLMs fail to generalize to multi-party dialogue; R2, that fine-tuning on XMP improves performance only marginally, with the 70B model gaining at most 1% absolute over the 8B model; and R3, that multilingual complementarity is inconsistent or even detrimental in this setting. The paper also reports translation-quality checks, dataset statistics, and case studies of model errors.","tokens_in":20210,"tokens_out":6089,"duration_ms":55940,"significance":"If the negative findings were rigorously established, they would be noteworthy because they would challenge the expectation that cross-lingual fine-tuning benefits transfer to complex generation tasks. The dataset itself is a potentially useful resource: it is large, covers six languages, and the authors report human and automatic translation-quality checks. The empirical claims, however, are not currently supported at the level needed for the paper's central conclusions, and one quantitative claim is contradicted by the paper's own table.","major_comments":[{"comment":"The single-language versus mixed-language fine-tuning comparison does not control the training budget. Appendix E documents only the train_en command with num_train_epochs=2.0; no commands or dataset-composition details are given for the mixed runs such as 'DE, FR'. If mixed training concatenates the two single-language splits and keeps epochs fixed, the mixed model is trained on roughly twice as many examples and twice as many optimizer steps with a different cosine schedule; if optimizer steps are instead held fixed, the target language contributes only half the examples per epoch. Both scenarios confound language mixing with training-data exposure and optimization length. Under this design, the consistent degradation of the 8B model in Table 5 is equally compatible with underfitting or data imbalance as with cross-lingual interference, and the scattered 70B deltas, mostly below 0.5 points, are within plausible seed-to-seed noise that the paper never reports. R3 is therefore not established by the reported experiments.","section":"§6.3, Table 5; Appendix E"},{"comment":"R2 as stated is not supported. First, Table 3 reports only the fine-tuned models; there is no zero-shot or in-context-learning baseline under the same evaluation protocol, so the paper cannot claim that fine-tuning 'improves only marginally.' The abstract's wording conflates the effect of fine-tuning with the effect of model scale. Second, the quantitative gloss 'at most a 1% absolute gain' is contradicted by the table: LLaMA-3.1 70B exceeds LLaMA-3.1 8B by 1.31 F1 on English (21.23 vs. 19.92). The claim should either be revised to 'typically within 1% on most metrics' and then verified, or the table and claim must be reconciled.","section":"§6.1, Table 3; Abstract R2"},{"comment":"R1, that MLLMs fail to generalize to multi-party dialogue, is supported only by flat or declining in-context-learning curves on roughly 1,000 samples and by low absolute n-gram scores. These observations do not distinguish 'the model cannot handle multi-party structure' from 'n-gram metrics are insensitive for open-ended generation with many valid continuations' or 'ICL is not the right regime for this task.' No human-performance baseline, chance-level reference, or zero-shot baseline under the final evaluation setup is provided, and no significance test accompanies the differences in Figure 5. The claim should be reworded or supplemented with a controlled comparison, such as human judgments or a simple template-based baseline.","section":"§3, Appendix A, Figure 5; R1"},{"comment":"No measure of variance is reported for any result. The headline deltas in Table 5 are as small as +0.04 or -0.00 F1, and several 70B improvements, such as DE+FR on FR with +0.66 F1, could plausibly flip sign under a different random seed. Without multiple seeds, confidence intervals, or significance tests, qualitative conclusions such as 'detrimental,' 'marginal,' or 'inconsistent' are not supported by these point estimates.","section":"Tables 3 and 5; §6.3"}],"minor_comments":[{"comment":"The sentence describing the cross-entropy loss is incomplete: 'the cross-entropy loss is computed between and the ground truth' should identify the model output R_hat as the first argument.","section":"§5.2, Eq. (1)"},{"comment":"The acronym MLLM is used for 'multilingual large language model,' but MLLM more commonly denotes 'multimodal large language model'; please define and disambiguate at first use.","section":"Abstract and Introduction"},{"comment":"The caption says that differences are 'highlighted,' but no highlighting is visible in the submitted text; please indicate explicitly that the superscript numbers are deltas relative to LLaMA-3.1 70B.","section":"Table 3 caption"},{"comment":"The three template types are distinguished only by color or line style; in grayscale the Type 1, Type 2, and Type 3 curves are difficult to separate. Adding markers would improve readability.","section":"Figure 5"},{"comment":"BERTScore is reported only for LLaMA-3.1 8B, but the surrounding discussion generalizes to 'the models'; please clarify that the semantic-divergence observation is currently demonstrated for one model only.","section":"§6.2, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The dataset contribution is real and the translation-quality checks are a genuine strength. However, the three headline findings are not currently supported: the R3 experiment needs controlled training budgets and variance reporting, R2 needs a no-fine-tuning baseline and a corrected quantitative statement, and R1 needs a more careful operationalization. If the authors can supply these controls, the paper may be salvageable; if not, the empirical conclusions should be substantially weakened or removed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this paper's real contribution is the XMP dataset, and the headline claims about fine-tuning and multilingual mixing are not established by the experiments as reported. The dataset effort is genuine: 312k multi-party dialogues, sourced from NPR podcasts, translated into five languages plus English, with human and automatic quality checks that look reasonable. That is a new resource for a genuinely under-tested scenario, and it deserves attention.\n\nWhat the paper does well: it is upfront about the negative results, it provides detailed appendices on data collection and translation evaluation, and it ships training commands. The observation that ICL does not help, and that even English performance is low on this task, is worth knowing even if it is not deeply analyzed. The XMP dataset is the first parallel multilingual multi-party dialogue resource I know of, and that alone justifies a serious look.\n\nThe soft spots are where the paper makes its three research claims. R1 (MLLMs fail to generalize) is plausible but rests on absolute metric values without any calibration—there is no reference system, no human performance, no chance-level baseline, so \"fails\" is doing a lot of work. R2 (fine-tuning improves only marginally) is undercut by the absence of a non-fine-tuned baseline in Section 6.1; Table 3 shows only fine-tuned models, so we cannot see the gain from fine-tuning at all. R3 (mixing languages is detrimental) is the most consequential claim and the weakest supported. Table 5 compares single-language and mixed-language SFT, but the training commands in Appendix E only show single-dataset runs; mixed runs are not specified, and there is no statement about whether total data, optimizer steps, or target-language exposure were held constant. The stress-test note is right: with epochs fixed, the mixed run sees twice the data; with steps fixed, the target language sees half the data. Either way, the observed degradation is confounded with training budget and data composition. On top of that, there are no error bars or significance tests anywhere, and many of the 70B deltas are under 0.5 point, which is seed-to-seed noise territory.\n\nNone of this means the resource is worthless. It means the paper's empirical conclusions require a controlled re-analysis. For peer review, I would send it out: the dataset is worth refereeing, and the negative results, if they survive controls, are useful for the conversational AI community. But I would ask for the R2 and R3 experiments to be redone with matched budgets, multiple seeds, and variance reported. That is a heavy but feasible revision.","headline":"XMP is a genuinely new dataset, but the headline negative claims about fine-tuning and cross-lingual mixing are confounded and need controlled re-runs before they can be believed.","tokens_in":20721,"tokens_out":2111,"would_cite":true,"duration_ms":20171,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that multilingual LLMs fail to generalize to multi-party dialogue, and that fine-tuning on a new six-language dataset of 312k podcast conversations gives only marginal gains while mixing languages during training usually…","keywords":["multi-party dialogue","multilingual dataset","response generation","multilingual complementarity","cross-lingual transfer","LLM fine-tuning","XMP dataset","low-rank adaptation"],"falsifier":"Run the mixed-language condition with the same total number of training examples and the same number of target-language examples as the single-language condition (for example, subsample each language to match the single-language size and train for the same number of steps). If the performance drop disappears or reverses, the paper's R3 claim about cross-lingual interference is not supported; if the drop persists under matched data, the claim survives.","tokens_in":19838,"feed_emoji":"🎙️","tokens_out":5976,"duration_ms":50507,"temperature":0.7,"pith_summary":"This paper asks whether multilingual large language models can produce the next utterance in conversations among three or more speakers, and answers that on a new benchmark they cannot. To test this, the authors built XMP, a parallel six-language dataset of 312k multi-party podcast dialogues, the first of its kind. They report three findings: models generalize poorly to this setting, supervised fine-tuning on XMP gives only marginal gains (the 70B model beats the 8B by at most one absolute point), and mixing languages during fine-tuning usually hurts rather than helps, reversing the 'multilingual complementarity' effect seen in simpler tasks. The dataset and negative results matter because real-world multilingual use includes meetings and group discussions, not only two-person chat.","feed_headline":"Multilingual LLMs fail to generalize to multi-party dialogue","feed_subtitle":"A 312k-conversation, six-language benchmark shows fine-tuning barely helps and mixing languages often hurts","key_machinery":"The load-bearing object is XMP, a parallel multilingual dataset for multi-party dialogue built from public podcast transcripts, with each English dialogue translated into Chinese, Japanese, German, French, and Italian, yielding 312k conversations averaging 3.64 speakers and 8.47 turns. The paper also relies on the notion of 'multilingual complementarity' — the claimed ability of a model to use training data from several languages to improve on one target language — and on a low-rank-adaptation fine-tuning setup using LLaMA 3.1 8B/70B and Qwen 2.5 7B. The dataset supplies the missing testbed: a complex, multi-speaker generation task where prior multilingual benchmarks (structured tasks or two-party dialogue) cannot reveal whether model abilities generalize.","core_discovery":"On the paper's own terms, the central discovery is that the capabilities of multilingual LLMs do not transfer to multi-party dialogue: the models cannot reliably track who is speaking, what the topic is, or what a given speaker would naturally say next. Fine-tuning on XMP, a purpose-built parallel multilingual dataset of 312k podcast conversations in English, Chinese, Japanese, German, French, and Italian, recovers only a small fraction of this ability; the 70B LLaMA model's advantage over the 8B model stays within about one absolute point on most metrics. The multilingual complementarity effect, previously observed in structured tasks where training on multiple languages improves a target language, largely diminishes or reverses in this setting: mixing English with another language during supervised fine-tuning usually degrades the target language, with only isolated and marginal gains in the 70B model. The authors interpret the failure as a genuine limitation of conversational understanding rather than an artifact of the test: context-removal experiments show performance depends mostly on the last two turns, and case studies show the model losing track of speakers and producing generic, content-free replies.","pith_inferences":["Extension: if the single- versus mixed-language runs are not matched for total training data or target-language proportion, the observed 'detrimental' effect of mixing may be an artifact of doubled data or diluted target examples; a matched-data experiment would settle this.","Extension: the context-removal result suggests a testable fix — explicitly encoding speaker identities and turn structure (for example, role embeddings or graph attention over speakers) might recover the multi-party ability that plain sequence fine-tuning misses.","Extension: since the dataset is machine-translated via Google Translate, some of the cross-lingual differences could reflect translationese rather than model limitations; a human-translated subset would be a sharper test.","Extension: the paper's negative results imply that dialog systems for real group settings (meetings, multiplayer games) may need dedicated multi-party training objectives rather than relying on scale or cross-lingual transfer."],"forward_implications":["Multilingual LLMs should not be assumed to handle multi-party dialogue; at minimum, benchmarks should include multi-speaker settings before claiming general multilingual chat ability.","Fine-tuning on a high-quality parallel dataset does not automatically restore this ability, and scale alone (70B vs 8B) offers only marginal gains of about one absolute point.","Training on multiple languages during supervised fine-tuning is usually detrimental in this setting, so multilingual complementarity is not a universal property of LLM training.","Models generate responses mostly from the last few turns, not the full dialogue history, so improving multi-party performance requires better long-range conversational modeling.","High n-gram scores for Japanese and Chinese do not reflect semantic quality; BERTScore shows Chinese lagging, so token-based metrics can mislead in multilingual dialogue evaluation."],"supporting_citations":[{"why":"XDailyDialog, the existing multilingual two-party dialogue corpus that XMP contrasts with and extends.","marker":"(Liu et al., 2023)"},{"why":"Study of cross-lingual data sharing during LM fine-tuning; its experimental setup is followed for the ICL and language-mixing comparisons.","marker":"(Choenni et al., 2023)"},{"why":"Reports that mixing English improves target-language performance, the claim that R3 directly tests and reverses.","marker":"(Whitehouse et al., 2023)"},{"why":"Introduces the low-rank adaptation method used for all supervised fine-tuning experiments.","marker":"(Hu et al., 2021)"},{"why":"Ubuntu Dialogue Corpus, a multi-party dialogue dataset compared against XMP in the task-type table.","marker":"(Lowe et al., 2015)"},{"why":"Reports that machine translation systems can outperform LLMs in translation quality, motivating the choice of Google Translate and the translation-quality evaluation.","marker":"(Singh et al., 2024)"},{"why":"XCOPA, a multilingual structured benchmark used to illustrate the gap between structured tasks and complex generation.","marker":"(Ponti et al., 2020)"},{"why":"Primer on multilingual models, cited as prior evidence that models leverage cross-lingual similarity.","marker":"(Doddapaneni et al., 2021)"}],"fun_headline_variants":["Multilingual LLMs fail to handle multi-party dialogue","Fine-tuning barely helps multilingual multi-party dialogue","Language mixing worsens multilingual dialogue performance","Multi-party dialogue exposes LLM multilingual limits","LLMs can't track speakers in multilingual group chats"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison between single-language and mixed-language fine-tuning assumes that only the language mixture changes, but the paper does not state whether total training data size or the proportion of target-language examples is held constant, and the training commands concatenate dataset splits, so mixed runs may train on roughly twice as many examples.","fun_headline_variants_meta":{"raw":{"variants":["Multilingual LLMs fail to handle multi-party dialogue","Fine-tuning barely helps multilingual multi-party dialogue","Language mixing worsens multilingual dialogue performance","Multi-party dialogue exposes LLM multilingual limits","LLMs can't track speakers in multilingual group chats"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000575,"raw_usage":{"total_tokens":2736,"prompt_tokens":988,"completion_tokens":1748,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":1679}},"tokens_in":604,"tokens_out":1748,"duration_ms":12878,"temperature":1.0,"reasoning_tokens":1679,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:27:25.023398+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the mixed-language condition with the same total number of training examples and the same number of target-language examples as the single-language condition (for example, subsample each language to match the single-language size and train for the same number of steps). If the performance drop disappears or reverses, the paper's R3 claim about cross-lingual interference is not supported; if the drop persists under matched data, the claim survives.","supporting_citations":[],"review_version":1}