{"id":"2b2c33aa-b07c-4440-806a-0a9311d750c4","arxiv_id":"2508.01656","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A systematic benchmark of multilingual authorship attribution shows fine-tuned LLM detectors exceed 0.9 macro F1 in-language but transfer poorly across languages, with Russian training generalizing better than English.","lead":"This paper defines and evaluates the task of attributing machine-generated texts to specific LLMs or humans across 18 languages. It shows that fine-tuned multilingual models work well in-language, but cross-lingual transfer remains difficult and language-dependent.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Article-level train/test leakage in MULTITuDE v3 could inflate in-language scores, exaggerating the cross-lingual difficulty gap; split disjointness is never reported.","rationale":"The reader's weakest assumption was that MULTITuDE v3 is representative of real-world multilingual machine-generated text; my concern is more specific and more damaging to the headline comparison. The paper's central claim depends on the gap between same-language and cross-lingual attribution performance. If the official split is not article-disjoint, same-language scores can be inflated by a trivial article-overlap shortcut, while cross-lingual scores cannot benefit from that shortcut. This would make cross-lingual transfer look harder than it is. I did not find evidence in the paper that article-level disjointness was enforced, and the limitations section does not mention it. The check I propose is concrete and can be settled directly from the public dataset. I am not rejecting the paper: the dataset coverage, the 18-language evaluation, and the consistent cross-lingual drops even between related languages are genuine contributions, and the central qualitative finding may survive the check. But because the reported numbers, not just the interpretation, could be affected, the paper should remain conditional until this split issue is resolved. The reader's dataset-representativeness concern is related but not the same mechanism, hence 'partial' agreement.","tokens_in":27780,"tokens_out":8523,"duration_ms":112737,"concrete_test":"Using the released MULTITuDE v3 data, count train/test pairs that share a source article ID or normalized headline within each language, and also check for cross-language article overlap (e.g., parallel stories). If the overlap is nonzero, rerun the RQ1 evaluation and the internal rows of Table 3 on article-disjoint splits, grouping instances by article ID and splitting by article. Compare macro-F1: if scores drop by more than a few points, the internal/external gap motivating the cross-lingual claim is partly a split artifact; if scores are stable, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central evidence for the claim that cross-lingual transferability is a major challenge is the contrast between high in-language performance (Table 2, plus the internal rows of Table 3) and much lower cross-lingual performance. That contrast is only valid if the MULTITuDE v3 train/test split cannot be exploited through article identity. MULTITuDE v3 is built from source news articles: for each article there is a human-written text plus seven LLM-generated texts prompted by the same headline. The paper reports per-language instance counts and refers to the train/test splits, but it never states that the splits are disjoint at the article or headline level. If the split is instance-level random, the same article can appear in both training and test within a language, allowing a classifier to use headline, topic, or named-entity overlap as a shortcut rather than learning true generator attribution. Cross-lingual test sets do not share article identity with the source-language training set, so this shortcut is available in-language but largely unavailable cross-lingually. That asymmetry would inflate the internal scores and hence exaggerate the paper's central finding that cross-lingual transfer remains hard. It could also distort the Russian-vs-English transfer comparison if the Russian test split happens to contain more article overlap than the English split. Because this issue concerns internal validity rather than dataset representativeness, it is more load-bearing than the reader's stated weakest assumption.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines and investigates multilingual authorship attribution (ML-MGT) and cross-lingual authorship attribution (CL-MGT), evaluating seven methods over 18 languages and 8 author classes (7 LLMs plus human) from the MULTITuDE v3 dataset. The main empirical findings are that fine-tuned and contrastive methods (mdok, OTBDetector) achieve strong in-language macro-F1 scores, that cross-lingual transfer is generally poor and especially weak from English, that training on Russian transfers better than English or Spanish, and that generator identity affects attribution difficulty. The paper concludes that current monolingual AA methods are not readily language-agnostic and that multilingual and cross-lingual AA constitute open benchmark problems.","tokens_in":28010,"tokens_out":7388,"duration_ms":88213,"significance":"If its conclusions hold, the paper provides a useful first systematic evaluation of multilingual AA and establishes a falsifiable baseline: training on English alone is insufficient for cross-lingual attribution, and transfer depends on language family and script. The study's strengths include the formal problem definitions, the broad and balanced coverage of 18 languages representing multiple families and scripts, the comparison of seven diverse methods, the detailed supplementary per-generator results, and the transparent reporting of dataset selection and computational resources. However, the central claim about the difficulty of cross-lingual transfer is currently under-supported by two internal-validity concerns: the train/test split is not shown to be article/headline-disjoint, and all quantitative conclusions rest on single runs without significance tests or confidence intervals.","major_comments":[{"comment":"The paper never states whether the MULTITuDE v3 train/test split is disjoint at the article or headline level. Because MULTITuDE v3 contains, for each human news article, seven LLM-generated texts prompted by the same headline, an instance-level random split can place several variants of the same underlying article in both training and test. This would let models exploit headline, topic, or named-entity overlap as a shortcut in the in-language conditions of Table 2 and in the 'Internal' rows of Table 3, while cross-lingual test languages share no article identity with the training languages. Such an asymmetry would inflate the in-language scores and exaggerate the cross-lingual difficulty gap, which is the paper's central finding. The authors should either explicitly confirm that the splits are article-disjoint (with supporting statistics) or re-run the key experiments using an article-level split and report how the results change.","section":"Multilingual Data and Generator Models; Table 1"},{"comment":"All results in Tables 2, 3, and 4 are single-run macro-F1 numbers, and the language-level paragraph states that 'training on Russian ... has a significantly greater impact than other languages' with +0.25 vs. English and +0.12 vs. Spanish. No standard deviations, confidence intervals, significance tests, or effect sizes are provided, and the phrase 'in terms of overall best results' is not defined. The observed differences may be within run-to-run variance, especially for the smaller statistical methods. The authors should report multiple seeds with variance estimates and provide a statistical test (or at least a clear definition of the reported aggregate) before making claims of significance.","section":"Cross-lingual Transferability Evaluation; Table 3"},{"comment":"The claim that Russian training yields the best cross-lingual transfer is based primarily on the 'all' column of Table 3, which averages macro F1 over all 18 test languages. Of these, eight are Slavic (5 Slavic-Latin plus 3 Slavic-Cyrillic), so the average is strongly influenced by languages closely related to Russian. This confound is not controlled for in the headline comparison. The paper should either report family-balanced averages (e.g., one average per family in Table 4, then an average over families) or restrict the claim to the per-family evidence shown in Table 4, which already tells a more nuanced story.","section":"Cross-lingual Transferability Evaluation; Tables 3 and 4"}],"minor_comments":[{"comment":"In the paragraph beginning 'As expected, detectors based on a multilngual pretraining', 'multilngual' should be 'multilingual.'","section":"Results, Multilingual Suitability Evaluation"},{"comment":"Several reference author names contain spurious spaces after initial letters (e.g., 'V asilatos', 'Y oo', 'V enkatraman', 'V erma'); these should be corrected.","section":"References"},{"comment":"The sentence 'The fourth task corresponds to to investigate the impact of the various LLM generators' contains a duplicated 'to' and should be rephrased.","section":"Experimental Setup, RQ3"},{"comment":"The statement 'the F1 score of OTBDetector only reduces by 3%' is ambiguous about whether the reduction is absolute or relative; it should be clarified (e.g., 0.93 to 0.90 macro F1 is an absolute drop of 0.03).","section":"Results, Multilingual Suitability Evaluation"},{"comment":"The two best-performing methods, mdok and OTBDetector, are developed by the same research group that curates MULTITuDE, and one author is a co-creator of the dataset. This is not itself a technical flaw, but the paper should explicitly disclose this home-field advantage and state whether any methods were selected or tuned after observing results on the test set.","section":"Methods and Dataset Selection"},{"comment":"Table S3 reports weighted F1 while the main tables report macro F1; the paper should specify why these metrics differ and warn readers that the two are not directly comparable.","section":"Supplementary Results, Table S3"}],"recommendation":"major_revision","confidential_remarks":"The main concern is internal validity: the train/test split leakage at the article level is plausibly load-bearing for the paper's central contrast between in-language and cross-lingual performance. This is fixable by re-running with article-level splits or by providing evidence that the current splits are article-disjoint. The lack of statistical testing is a separate but important weakness. The authors' dual role as dataset and method creators should be disclosed, though I see no evidence of misconduct. If the authors can address the split-integrity issue, the paper could become a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something genuinely useful: it formalizes ML-MGT and CL-MGT, then runs the first systematic evaluation of authorship attribution methods across 18 languages and 8 generator classes. The finding that fine-tuned detectors work well in-language but transfer poorly cross-lingually is consistent and backed by large gaps, not noise. The methodological adaptation of seven existing detectors is careful, and the comparisons across language families and scripts are informative. Credit is due for the breadth and for choosing MULTITuDE v3, which has more consistent generator/domain coverage than alternatives.\n\nThe soft spots are real but not fatal. First, there are no error bars or significance tests anywhere, yet the text says training on Russian has a \"significantly greater impact\" than English or Spanish. That word needs statistical support, not just a big point estimate. Second, the authors are also the creators of MULTITuDE and of two of the best-performing systems (mdok and OTBDetector). That is not disqualifying, but combined with no code release it makes independent verification harder than it should be.\n\nThere is one concern that bothers me more than the reader's representativeness worry, and it concerns internal validity. MULTITuDE v3 is built from source news articles: each article yields a human text plus seven LLM outputs prompted by the same headline. The paper never states that the train/test splits are disjoint at the article or headline level. If the split is instance-level random, the same article can appear in both training and test within a language, letting a classifier exploit topic, headline, or named-entity overlap as a shortcut. That shortcut would inflate in-language scores while cross-lingual test sets remain unaffected, because the test languages are unseen. The result would exaggerate the paper's central claim that cross-lingual transfer is hard. This is a checkable, deterministic issue, not a matter of taste, and it should be resolved before the findings are treated as definitive.\n\nThe paper deserves a serious referee. The task formalization and benchmark value are solid, and if the split is indeed article-disjoint, the conclusions are likely to hold. I would ask the authors to report the split construction, add significance tests or error bars, and release the code. The audience is anyone working on MGT detection or attribution, especially with multilingual or cross-lingual ambitions.","headline":"A solid first benchmark for multilingual machine-generated text authorship attribution, but the central in-language vs cross-lingual contrast rests on a split-disjointness detail the paper never reports.","tokens_in":28552,"tokens_out":1739,"would_cite":true,"duration_ms":21782,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Existing authorship-attribution methods can be adapted to multilingual text, but they transfer poorly across language families and writing scripts.","keywords":["machine-generated text attribution","multilingual authorship attribution","cross-lingual transferability","LLM-generated text detection","language families and scripts","contrastive learning","fine-tuned multilingual encoders","generator attribution"],"falsifier":"Build an equivalent balanced corpus with the same seven generators and human class but in held-out languages from other families or from a different domain, train the best detector on Russian and on English, and measure macro-F1 on the distant-script targets. The paper's central claims would fail if Russian's transfer advantage disappeared, or if a script-distant target matched Slavic-Cyrillic performance, since both results would show that family and script distance are not the controlling factors.","tokens_in":27580,"feed_emoji":"🌐","tokens_out":11048,"duration_ms":118688,"temperature":0.7,"pith_summary":"Authorship attribution—deciding which of several large language models, or a human, wrote a given text—has mostly been studied in English. The paper formalizes two new tasks: multilingual attribution (ML-MGT), where training and test texts span many languages, and cross-lingual attribution (CL-MGT), where test languages are unseen during training. Using a balanced 18-language news corpus built from seven LLM generators plus a human class, it evaluates seven adapted detectors and finds that fine-tuned and contrastive models can reach macro-F1 above 0.9 when all languages are seen, while cross-lingual transfer drops sharply, especially between dissimilar language families or writing scripts. The authors argue that attribution methods are not language-agnostic: the linguistic properties of the target language and the identity of the generator jointly determine success. If the finding holds, deployed attribution systems trained only on English cannot be trusted for multilingual content.","feed_headline":"18 languages, 7 generators: attribution transfers poorly","feed_subtitle":"Fine-tuned detectors handle seen languages well, but lose accuracy on unseen families and scripts.","key_machinery":"The central machinery is a controlled train/test protocol built on two problem definitions. ML-MGT formalizes attribution as a multi-class mapping from text to one human plus seven machine classes; CL-MGT restricts the training languages to a proper subset of the test languages. The protocol uses macro-averaged F1—the mean per-class F1, so all eight classes count equally—and applies four training conditions: all 18 languages jointly, English only, Spanish only, Russian only, and a combined English-Spanish-Russian condition. The adapted detectors include a fine-tuned decoder, a contrastive learner built on a multilingual encoder, fine-tuned multilingual and English-only encoders, a statistical ensemble, and two zero-shot statistical detectors. Comparing these conditions reveals which parts of the performance gap come from language family, writing script, or generator identity.","core_discovery":"On the paper's own terms, the core discovery is that multilingual authorship attribution is feasible but cross-lingual transferability is the bottleneck. The authors define ML-MGT as an 8-way classification task (7 LLMs plus human) over a language set and CL-MGT as the same task with a disjoint subset of languages at test time. Across 18 languages from eight families and five scripts, the best adapted models—a fine-tuned decoder and a contrastive learner—achieve 0.90 to 0.93 macro F1 in the fully multilingual setting, whereas zero-shot statistical detectors stay between 0.16 and 0.45. Transfer results differ: Russian-trained models generalize best, with the top decoder reaching 0.68 overall macro F1 (plus 0.25 over English-trained and plus 0.12 over Spanish-trained), English-trained models generalize worst, and Greek, Arabic, and Chinese remain hard targets for every training language. The paper concludes that existing methods are not language-agnostic and that generator identity—for example, the predictable catch-all behavior of certain models under uncertainty—shapes attribution errors as much as language does.","pith_inferences":["One consequence the paper leaves implicit is that training data for attribution should be selected by typological diversity rather than by resource abundance: Russian's strong transfer suggests morphologically rich languages carry reusable signals that high-resource analytic languages like English do not.","The paper's morphological explanation for Russian's advantage is testable: one could correlate the typological distance between each training language and each target language with the observed macro-F1 drop; a strong correlation would turn language selection into a predictive design choice.","The persistent weakness on Arabic, Greek, and Chinese across all training languages suggests a practical system should include at least one representative of each writing script in the training set or add script-specific adapters, rather than trusting a single multilingual model to transfer.","The catch-all classes point to a calibration problem: an abstention or none-of-the-above option, learned from the uncertainty patterns in the confusion matrices, might improve practical attribution more than adding another generator to the class set."],"forward_implications":["Trained on all 18 languages at once, the best two detectors keep macro-F1 above 0.90 in every tested language, so multilingual attribution is achievable when target languages are available in training.","Off-the-shelf statistical detectors designed for binary machine-text detection are effectively unusable for attribution, with macro-F1 scores between 0.16 and 0.45 and near-collapse on non-Latin scripts.","Training on English is the weakest cross-lingual strategy; training on Russian transfers best overall, with a large gap in overall macro-F1 for the top decoder (0.68 versus 0.36).","Combining English, Spanish, and Russian training helps Germanic and Romance targets but still leaves Greek, Arabic, and Chinese below about 0.5 macro F1, so multi-language training does not solve cross-lingual attribution.","Generator identity drives errors: two architecturally related models are the most confused pair in unseen-language settings, and two other models become default catch-all predictions when the detector is uncertain."],"supporting_citations":[{"why":"Supplies the 18-language balanced corpus of human and LLM-generated news articles used for all experiments.","marker":"(Macko et al. 2025)"},{"why":"Source of the human-written articles in that corpus.","marker":"(Varab and Schluter 2021)"},{"why":"The multilingual decoder detector that the paper adapts into its top-performing attribution method.","marker":"(Macko 2025)"},{"why":"The contrastive method adapted here as the second-best attribution method.","marker":"(La Cava and Tagarelli 2025)"},{"why":"The multilingual encoder used by the fine-tuned and contrastive detectors.","marker":"(Conneau et al. 2020)"},{"why":"The English-only encoder whose poor transfer anchors the cross-lingual comparison.","marker":"(Liu et al. 2019)"},{"why":"Fast-DetectGPT, one of the zero-shot statistical baselines.","marker":"(Bao et al. 2023)"},{"why":"Binoculars, the zero-shot detector used both as a baseline and as a feature in the statistical ensemble.","marker":"(Hans et al. 2024)"}],"fun_headline_variants":["Across 18 languages, AI attribution fails to generalize","Multilingual text attribution: seen or unseen languages gap","Transfer gap: detectors know seen languages, miss unseen","Attribution accuracy drops sharply on unfamiliar language families"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the balanced 18-language, seven-generator news dataset represents real-world multilingual machine-generated text well enough that the observed transfer failures are about language and generator identity, not about the dataset's domain or language choice.","fun_headline_variants_meta":{"raw":{"variants":["Across 18 languages, AI attribution fails to generalize","Multilingual text attribution: seen or unseen languages gap","Transfer gap: detectors know seen languages, miss unseen","Attribution accuracy drops sharply on unfamiliar language families"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000174,"raw_usage":{"total_tokens":1310,"prompt_tokens":999,"completion_tokens":311,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":249}},"tokens_in":615,"tokens_out":311,"duration_ms":4592,"temperature":1.0,"reasoning_tokens":249,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T05:27:07.265772+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build an equivalent balanced corpus with the same seven generators and human class but in held-out languages from other families or from a different domain, train the best detector on Russian and on English, and measure macro-F1 on the distant-script targets. The paper's central claims would fail if Russian's transfer advantage disappeared, or if a script-distant target matched Slavic-Cyrillic performance, since both results would show that family and script distance are not the controlling factors.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The multilingual encoder used by the fine-tuned and contrastive detectors."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Fast-DetectGPT, one of the zero-shot statistical baselines."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Binoculars, the zero-shot detector used both as a baseline and as a feature in the statistical ensemble."}],"review_version":1}