{"id":"689e57bb-4ef1-4e78-a91e-fa336da0b6e5","arxiv_id":"2412.15498","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"mT5 fine-tuned on Spanish suicide tweets translated into five languages outperforms mBERT and XLM-R across six languages, but the evaluation is under-specified and the F1 'above 85%' claim is contradicted for two languages.","lead":"The authors fine-tune three multilingual AI models on machine-translated Spanish tweets about suicide to detect suicidal text in six languages, and report that mT5 performs best. The claim is hard to verify because the test set is not described and no code or data are released.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation rests on an undescribed self-cited test set (LSL) that may share its translation pipeline with the training data; if so, the reported F1 scores do not measure cross-lingual generalization.","rationale":"The reader's weakest assumption correctly identifies the LSL test set as the load-bearing uncertainty. I agree that the paper provides no information about LSL's construction, and that the self-citation to a paper titled 'Automatically translating suicide-related language' raises a concrete risk of circularity: if LSL was created by translating the same Spanish tweets into the target languages, then the evaluation only demonstrates performance on synthetic translationese, not on genuinely multilingual suicidal text. This concern is more severe than the abstract's F1 inconsistency because it directly undermines the central generalization claim. The mT5 superiority over mBERT and XML-R is plausible and internally consistent, but it is only meaningful if the test set is valid. The manuscript also shows signs of incompleteness—Section 8 ends mid-sentence with 'Finally,'—which reinforces the need for a revised version that fully specifies the test set, reports error bars, and reconciles the abstract with Table 2. Therefore, the conditional verdict is appropriate; providing the LSL details or replacing it with an independent native-language test set would be the deciding check.","tokens_in":11177,"tokens_out":3807,"duration_ms":19763,"concrete_test":"Contact the authors or retrieve LSL from the COLING 2025 proceedings and verify: (1) whether any LSL test instance is a machine translation (via SeamlessM4T or any other model) of one of the 2,068 Spanish tweets in Valeriano et al. (2020), or is otherwise overlapping with the training/validation splits; (2) whether the non-Spanish LSL subsets are sourced from native speakers' text rather than translated from Spanish. If LSL is disjoint and native, recompute Table 2 with 95% confidence intervals and report per-language test sizes. If overlap or translation-from-Spanish is found, re-run the best model on a new human-annotated test set of naturally occurring suicidal and non-suicidal posts in each target language; only then can the cross-lingual claim be assessed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that mT5 fine-tuned on machine-translated Spanish tweets detects suicidal text in six languages with F1 above 85%—requires a valid, independent test set in each language. The paper's only test set is 'Lexicography Saves Lives' (LSL), cited as Schoene et al. (2025), an author self-citation. The paper gives no size, source, annotation procedure, or language distribution for LSL. The title of the cited paper ('Automatically translating suicide-related language') raises the possibility that LSL was produced by the same or a similar machine-translation pipeline from the same Spanish Valeriano et al. (2020) corpus used for training. If the test examples are translations of (or derived from) the training tweets, then the evaluation measures the model's ability to classify translationese from a known distribution, not cross-lingual generalization to natural suicidal text in English, German, Italian, Catalan, or Portuguese. This would make the reported F1 scores uninformative for the central claim. A secondary issue: Table 2 lists mT5 F1 = 83.2 for Italian and 84.8 for Portuguese, below the abstract's 'above 85%'. The overstatement is minor relative to the test-set concern, but it indicates the reported numbers are not the ones the paper advertises.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multilingual approach to detecting suicidal text on social media. The authors use a Spanish corpus of 2,068 tweets (498 positive, 1,570 negative) from Valeriano et al. (2020), machine-translate it into English, Catalan, German, Italian, and Portuguese using SeamlessM4T, and then fine-tune mBERT, XLM-R (referred to as XML-R), and mT5 on this multilingual data. They report that mT5 performs best, with F1 scores 'above 85%' in the abstract, and they include a perplexity-based analysis of translation quality. The evaluation is carried out on a test set called Lexicography Saves Lives (LSL), cited to Schoene et al. (2025), which is not described in the paper. The paper also includes a 10-fold cross-validation experiment on the training data and discusses limitations and ethical considerations.","tokens_in":11544,"tokens_out":3940,"duration_ms":30916,"significance":"If the central claim is verifiable, the work would be a useful step toward multilingual suicide risk detection in low-resource languages, and the idea of augmenting training data via machine translation is a pragmatic and transferable approach. The comparison of three strong multilingual transformer baselines with documented hyperparameters is a reasonable contribution to the applied NLP literature. However, the evaluation currently rests on an undescribed test set from the same research group, and the reported headline numbers are not fully consistent with the results table. These issues must be resolved before the contribution can be assessed.","major_comments":[{"comment":"The test set 'Lexicography Saves Lives (LSL)' is introduced with only a citation to Schoene et al. (2025) and no description of its size, source, labeling procedure, or language composition. Because the paper's central claim is cross-lingual generalization to natural suicidal text in six languages, the evaluation must be based on an independent, well-characterized test set. If LSL was created by translating the same Spanish corpus, or by using the same SeamlessM4T pipeline as the training data, then the reported F1 scores would measure performance on translationese from the training distribution rather than generalization to genuine target-language text. The authors must provide a full description of LSL, including its construction, statistics per language, and how independence from the training data is ensured. Ideally, the authors should also evaluate on an additional external benchmark in each target language.","section":"Section 3.1"},{"comment":"The abstract and discussion state that mT5 achieves 'F1 scores above 85%', but Table 2 reports mT5 F1 values of 83.2 for Italian and 84.8 for Portuguese. Section 4.1 also says scores 'surpassing 85% in most cases', which is technically true but the abstract's unqualified claim overstates the results. Please revise the abstract and Sections 4.1 and 6 to reflect the actual observed range, or explicitly qualify which languages are above 85%.","section":"Abstract, Section 4.1, Section 6"},{"comment":"The perplexity-based evaluation of translation quality is not internally consistent. Table 3 uses different language models for different target languages (roberta-large for English, xlm-roberta-xl for the others), so the perplexity values are not comparable across languages. Furthermore, perplexity is at best a fluency measure and does not capture semantic fidelity of suicide-related content, which is the critical property for this task. The abstract's statement that 'English and Spanish translations also displayed high quality based on perplexity' is also unsupported: Spanish is the source language and no Spanish perplexity is reported. Please provide a uniform, task-appropriate evaluation or substantially temper the claims about translation quality.","section":"Section 5, Table 3"}],"minor_comments":[{"comment":"The Ethical Considerations section ends mid-sentence with 'Finally,' and is incomplete. This should be finished or removed before publication.","section":"Section 8"},{"comment":"The text says the experiments 'enlisted the capabilities of four cutting-edge language models' but only three models are evaluated. The typo should be corrected.","section":"Section 3.3"},{"comment":"The model name 'XML-R' should be 'XLM-R' or 'XLM-RoBERTa' consistently, and 'mBERT' is typically written as 'mBERT' or 'multilingual BERT' but should be consistent with the reference.","section":"Throughout"},{"comment":"The paper does not specify whether the F1 scores are macro-averaged, micro-averaged, or class-weighted. Given the class imbalance (24% positive, 76% negative), the averaging method matters for interpreting the results. Please state it.","section":"Table 2"},{"comment":"The claim that 'no research has been found exploring multilingual language models for suicide risk detection' is very strong and difficult to verify; consider softening it to 'to our knowledge' and citing any recent related work.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"The central problem is the self-cited, undescribed test set (LSL). The authors of this manuscript include the authors of the LSL paper, and the construction of LSL is not described here. This raises a circularity risk that cannot be assessed from the manuscript alone. If the authors can demonstrate that LSL is independent and correctly labeled, the paper may be salvageable. The discrepancy between the abstract and Table 2 also needs correction. The incomplete Ethical Considerations section suggests the manuscript was not finalized."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a plausible fine-tuning study with a serious evaluation gap. What's actually new: a Spanish suicide-ideation corpus machine-translated into five languages with SeamlessM4T, plus a three-model comparison (mBERT, XML-R, mT5) across six languages. The hyperparameter table is clear, and the perplexity check on translation quality is a reasonable sanity check. That part is solid enough.\n\nThe load-bearing problem is the test set. The only test set is LSL, cited as Schoene et al. (2025), with no information about its size, source, labeling, or language composition. It is a self-citation, and the cited paper's title ('Automatically translating suicide-related language') raises the real concern that LSL was produced with the same or similar translation pipeline from the same Spanish source. If that is the case, the reported F1 scores measure classification of translationese from a known distribution, not cross-lingual generalization to natural text. This is the difference between supporting the central claim and not. The abstract also overstates the results: Italian F1 is 83.2 and Portuguese 84.8, both below 'above 85%'. There are no error bars or variance estimates. The ethical considerations section ends mid-sentence, which suggests an incomplete draft.\n\nTo be fair, the authors are transparent that the training data are translations, and they acknowledge translation quality and ethical concerns as limitations. The 'first multilingual model' claim is weakly supported by the literature review, but that is secondary.\n\nThis paper deserves a serious referee because the translated dataset and the comparison could be useful, but the referee must demand a full description of LSL and its construction, plus code or data release. If the test-set concern is confirmed, the central claim collapses. If LSL is independent, the paper is a modest but solid contribution. I would bring it to a reading group to discuss evaluation independence, but I would not cite it until the test-set issue is resolved.","headline":"A useful translated dataset and model comparison, but the central claim hinges on an undescribed self-cited test set that may share the training pipeline.","tokens_in":11972,"tokens_out":3300,"would_cite":false,"duration_ms":26912,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a single multilingual transformer fine-tuned on machine-translated Spanish tweets can detect suicidal text in six languages, with mT5 outperforming mBERT and XLM-RoBERTa in every language.","keywords":["multilingual suicide detection","suicidal ideation","machine translation","cross-lingual transfer","mT5","mBERT","XLM-RoBERTa","social media NLP"],"falsifier":"If the LSL test set contains any of the same tweets, or machine-translated versions of them, used to fine-tune the models, or if its labels were created by the same annotation process as the training data, then the reported F1 scores do not establish detection of suicidal text in genuinely new posts; a reader can check by computing tweet overlaps and by testing mT5 on freshly annotated native-language tweets in the six languages.","tokens_in":10949,"feed_emoji":"💬","tokens_out":6217,"duration_ms":49536,"temperature":0.7,"pith_summary":"The paper claims that a single multilingual transformer can detect suicidal text in six languages — Spanish, English, German, Catalan, Portuguese, and Italian — when fine-tuned on a Spanish Twitter corpus machine-translated into those languages. The authors compare mBERT, XLM-RoBERTa, and mT5, and report that mT5 is best in every language, with F1 scores from 83.2 to 88.1, which they summarize as above 85%. The point of the work is that machine translation plus a multilingual model can substitute for expensive per-language annotation, making automated suicide-risk screening more accessible to non-English-speaking online communities. If the claim holds, it would mean that low-resource languages can inherit a suicide-detection capability from a single labeled source dataset, with translation quality as the main cost.","feed_headline":"mT5 beats two rivals at spotting suicidal tweets in six languages","feed_subtitle":"Fine-tuned on translated Spanish tweets, one model reaches 83–88% F1 in all six languages and beats both BERT-style baselines.","key_machinery":"The machinery is the combination of SeamlessM4T, a multilingual translation model, and three pretrained transformers — mBERT (multilingual BERT), XLM-RoBERTa, and mT5, a text-to-text transformer. Training data are the 2,068 Spanish tweets from Valeriano et al. (2020), translated into English, Catalan, German, Italian, and Portuguese; each model is fine-tuned on the multilingual set and evaluated on the Lexicography Saves Lives (LSL) test set. Translation quality is measured by perplexity under monolingual language models, and robustness is checked with 10-fold cross-validation. The load-bearing piece is the translated corpus: it is what lets one labeled dataset stand in for six languages.","core_discovery":"The central discovery, as the authors state it, is that mT5 outperforms mBERT and XLM-RoBERTa on suicide-text classification in all six languages, with the largest gains over the BERT-style baselines appearing consistently across languages rather than only in the source language. English and Spanish score highest, Italian and Portuguese lowest, and the ordering of the three models is stable from language to language, which the authors read as evidence that each model's strengths transfer independent of linguistic particulars. The paper also reports translation-quality scores via perplexity, finding English and Portuguese translations most fluent and German and Italian least, and concludes that a multilingual model fine-tuned on translated Spanish tweets is a viable path to broad linguistic coverage in suicide-risk detection.","pith_inferences":["Editorial inference: the translated corpus is only six languages, but the same recipe — source corpus plus SeamlessM4T plus mT5 — could be pressed into service for any language SeamlessM4T supports, so the paper's method, if sound, is a template rather than a finished product.","Editorial inference: because all non-Spanish training data are machine translations, the models may be learning translationese — stylistic artifacts of the translation engine — rather than natural suicidal phrasing in each language; testing on human-written native tweets is the way to find out.","Editorial inference: a practical deployment would need per-language confidence thresholds and a human-in-the-loop protocol, since equal F1 scores can hide different error patterns across languages and false positives carry real clinical cost."],"forward_implications":["mT5 should be the default choice among the three tested models for multilingual suicide-text detection, since it leads every language and every metric.","Machine translation of a single annotated dataset can create usable training data for languages that lack native suicide-ideation corpora.","English and Spanish performance suggests the approach is strongest for languages close to the source; Italian and Portuguese likely need additional native data or better translations.","Perplexity scores identify German and Italian as the weakest translation links, so translation quality is the likely ceiling on detection in those languages.","Fine-tuned multilingual transformers can transfer a sensitive classification task like suicide risk across languages without per-language annotation."],"supporting_citations":[{"why":"Supplies the source corpus of 2,068 Spanish tweets whose manual labels define the classification task.","marker":"Valeriano et al. (2020)"},{"why":"Is the LSL test set on which all reported accuracy, F1, and AUC numbers are computed.","marker":"Schoene et al. (2025)"},{"why":"Provides SeamlessM4T, the model that translates the Spanish corpus into the five other languages.","marker":"Barrault et al. (2023)"},{"why":"Defines mBERT, the weakly performing baseline in the comparison.","marker":"Devlin et al. (2019)"},{"why":"Defines XLM-RoBERTa (XML-R), the middle-performing model in the comparison.","marker":"Liu et al. (2019)"},{"why":"Defines mT5, the best-performing model and the paper's recommended architecture.","marker":"Xue et al. (2020)"}],"fun_headline_variants":["mT5 beats BERT and XLM-R in six languages","Cross-lingual mT5 tops suicide-tweet detection in six languages","mT5 outperforms rivals for suicide texts in six languages","Multilingual suicide detection: mT5 wins across 6 languages"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported six-language results stand on the LSL test set, and the paper provides no information about how that test set was built, labeled, or whether it is independent of the translated training tweets; if the test set is not independent or is mislabeled, the F1 scores do not show cross-lingual generalization.","fun_headline_variants_meta":{"raw":{"variants":["mT5 beats BERT and XLM-R in six languages","Cross-lingual mT5 tops suicide-tweet detection in six languages","mT5 outperforms rivals for suicide texts in six languages","Multilingual suicide detection: mT5 wins across 6 languages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000981,"raw_usage":{"total_tokens":4121,"prompt_tokens":859,"completion_tokens":3262,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":475,"completion_tokens_details":{"reasoning_tokens":3186}},"tokens_in":475,"tokens_out":3262,"duration_ms":20588,"temperature":1.0,"reasoning_tokens":3186,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:22:36.533794+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If the LSL test set contains any of the same tweets, or machine-translated versions of them, used to fine-tune the models, or if its labels were created by the same annotation process as the training data, then the reported F1 scores do not establish detection of suicidal text in genuinely new posts; a reader can check by computing tweet overlaps and by testing mT5 on freshly annotated native-language tweets in the six languages.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the source corpus of 2,068 Spanish tweets whose manual labels define the classification task."},{"cited_title":"Ortega, Rodolfo Joel Zevallos, and Laura Ihle","cited_arxiv_id":null,"evidence_quote":"Is the LSL test set on which all reported accuracy, F1, and AUC numbers are computed."}],"review_version":1}