{"id":"757ef7f3-ad55-4bae-a3e8-850b9cb1a720","arxiv_id":"2505.14181","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new English-Chinese slang translation benchmark with polysemy labels, plus a deep-thinking model that detects, explains, and translates slang, outperforming vanilla and fine-tuned LLMs.","lead":"The paper introduces a new benchmark and model for translating English slang into Chinese, where the model first detects the slang, explains it, and then translates it. It shows that adding this structured 'thinking' step substantially improves translation quality compared with standard fine-tuning on the same data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SlangOWL's claimed translation gains may reflect alignment with GPT-4o's annotation and evaluation preferences rather than human-judged slang translation quality; only 200 samples were human-validated, and no human translation evaluation is reported.","rationale":"The reader's weakest assumption is LLM-generated annotation quality; my concern is adjacent but more specific: the annotation pipeline and the evaluation pipeline share the same model family (GPT-4o), creating a circularity that the 200-sample human check in Section 3.3 does not break because it covers only detection and explanation, not translations. This concern is load-bearing because the central claim is about translation quality, and the most direct evidence for 'significantly surpasses' on the general test comes from the GPT-4o-based GRF metric, not from BLEU or CometK. The proposed human evaluation would settle whether the measured advantage is real. I do not see an internal inconsistency that would justify rejection; rather, the result needs an additional validation step, which is exactly the conditionality the reader already assigned. Therefore I leave the verdict unchanged while sharpening the reason for the condition.","tokens_in":20423,"tokens_out":9787,"duration_ms":96945,"concrete_test":"Take a random sample of 300 hard-test source sentences; have three professional bilingual translators independently score anonymized outputs from SlangOWL-14B, SFT-14B, and GPT-4o on adequacy and preservation of slang meaning, without access to automatic scores. Compute mean human preference and the Spearman correlation between human scores and GPT-4o GRB/GRF. If SlangOWL is not preferred by humans, or if GRB/GRF correlate weakly with human judgments on this subset, the central claim is unsupported. Additionally, recompute SlangOWL versus SFT BLEU/COMET on the subset where human annotators confirm the gold slang term and explanation; if the gap disappears, the benchmark labels are driving the result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that SlangDIT's gold labels and evaluation metrics validly measure interpretative slang translation. Section 3.2 constructs the gold standard entirely through Qwen2.5-72b, Llama3.3-70b, and GPT-4o: these models decide what counts as slang, extract the slang terms, write the cross-lingual explanations, and annotate polysemy. Section 3.3 validates only 200 random samples, and only for slang detection and explanation, not for the Chinese translations. Meanwhile, Appendix C uses GPT-4o as the GRB/GRF translation evaluator, and SlangOWL is trained on thoughts distilled from DeepSeek-R1-Distill-Qwen-32B that are conditioned on these same LLM-generated labels. The evaluation is therefore circular in a specific sense: a model that learns to reproduce the annotation models' judgments will be rewarded by GPT-4o evaluators, and by BLEU/COMET only insofar as the gold references reflect those judgments. On the general test, the only statistically significant translation gains over SFT-14B are on GRF (77.52 vs. 76.00); BLEU and CometK differences are not marked significant. On the hard test, BLEU improves by +2.23, but the hard-test gold explanations and polysemy labels are GPT-4o-annotated. Without a human-judged translation evaluation, the evidence conflates 'better agreement with the auto-annotation pipeline' with 'better slang translation,' so the central claim is not yet established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SlangDIT, a new English-Chinese benchmark for interpretative slang translation, consisting of three sub-tasks: slang detection, cross-lingual slang explanation, and slang translation. The dataset contains over 25k sentence pairs derived from movie subtitles, annotated with slang terms, Chinese explanations, polysemy labels, and translations, using a pipeline of Qwen2.5-72b, Llama3.3-70b, and GPT-4o, with a 200-sample human validation of slang detection and explanation. The authors also propose SlangOWL, a 'deep thinking' model that explicitly reasons about whether a sentence contains slang, whether the slang is polysemous, what the slang means in context, and finally translates the sentence. SlangOWL is trained on thought traces distilled from DeepSeek-R1-Distill-Qwen-32B, conditioned on the gold labels. Experiments on three backbone LLMs (Qwen2.5-7B/14B-Instruct and Llama-3.1-8B-Instruct) report consistent improvements over vanilla and SFT baselines on the general and hard test sets across BLEU, Comet, CometK, and GPT-4o-based GRB/GRF metrics.","tokens_in":20755,"tokens_out":7740,"duration_ms":99471,"significance":"If the benchmark and evaluation are valid, SlangDIT fills a genuine gap by linking slang detection and explanation to translation, and the proposed deep-thinking approach is a plausible way to improve slang translation. Strengths include the scale of the dataset, the held-out test sets with no slang-term overlap, the inclusion of polysemy annotation, the use of multiple backbones, and the public release of the data and code. The main results are consistent in direction across model sizes and metrics, and the hard test set provides a challenging polysemy-focused evaluation. However, the current evidence is weakened by the heavy reliance on LLM-generated gold labels and LLM-as-judge evaluation without adequate human validation of translations, and by the incomplete reporting of statistical significance details. These issues directly affect whether the central claim of the abstract—that SlangOWL significantly surpasses vanilla and SFT models—is established.","major_comments":[{"comment":"The gold annotations for slang detection, explanation, and polysemy are produced entirely by Qwen2.5-72b, Llama3.3-70b, and GPT-4o (§3.2), and only 200 samples are human-validated, covering only slang detection and explanation (§3.3). No human evaluation of the translations is reported; instead, GRB and GRF are computed by GPT-4o (Appendix C). Because SlangOWL is trained on thoughts distilled from DeepSeek-R1-Distill-Qwen-32B that are conditioned on these same LLM-generated labels, the consistent gains over SFT baselines may partly reflect alignment with the annotation/evaluation pipeline rather than human-judged translation quality. This bears directly on the abstract's central claim that SlangOWL 'significantly surpasses' vanilla and SFT models. The authors should add a human translation evaluation on a representative sample (e.g., 100–200 hard-test instances) with adequacy and idiomaticity judgments, and report inter-annotator agreement.","section":"§3.2–3.3, Appendix C"},{"comment":"The statistical significance annotation ('†'/'††') is under-specified. The paper only states that a t-test with p<0.05 and p<0.01 was used, without reporting the number of samples, whether the test is paired over sentences or bootstrap-based, or whether any correction for multiple comparisons was applied across the many metrics and models. Without these details, the dagger annotations are not verifiable. Moreover, on the general test the BLEU and CometK differences between SlangOWL-14B and SFT-14B (24.94 vs. 24.51 and 71.38 vs. 71.29) are not marked significant, so the claim of consistent significant superiority is only partially supported by the data.","section":"§5.1, Tables 3–4"},{"comment":"The construction of the hard test set and its references is ambiguous. The text says that 1,165 instances are randomly sampled from the polysemous slang terms, and that for the remaining 85% GPT-4o generates possible meanings and translation pairs. It is not clear whether the 1,165 hard-test instances also have GPT-4o-generated translations or whether they come from the original MSCTD corpus with human translations. If the hard-test references are GPT-4o-generated, the BLEU/Comet improvements in Table 4 partly measure agreement with GPT-4o output rather than human translation quality. The authors should clarify the exact provenance of the hard-test Chinese references and, ideally, include human-written references for at least a subset of the hard test.","section":"§3.2, Table 4"}],"minor_comments":[{"comment":"The model name is spelled inconsistently as 'SlangOWL' and 'SLangOWL' in the abstract and text; please standardize it.","section":"Abstract, §1"},{"comment":"The phrase 'the main reason is lacking of a benchmark' should be 'the main reason is the lack of a benchmark'.","section":"§1"},{"comment":"'we filter the sentences that contains repetitive slang terms' should be 'contain'.","section":"§3.2"},{"comment":"The metric name is given as 'CometKivi' in the text but 'CometK' in Table 3; please use one consistent spelling (e.g., CometKiwi).","section":"§5.1, Table 3"},{"comment":"The sentence 'Please refer to Appendix B for details' appears both in Section 5.1 and at the beginning of Appendix B itself; the latter is a copy-paste error and should be replaced with the actual details.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper introduces a potentially useful benchmark and a plausible method, but the evaluation needs substantial strengthening before publication. The reliance on GPT-4o for both annotation and evaluation creates a risk of model-alignment artifacts, and the lack of human translation evaluation is a key gap that should be addressed with a focused human study. The statistical reporting also needs more rigor. The self-citations to MSCTD and DRT-o1 from the same group are not a concern per se, but the authors should ensure the hard-test data provenance is described unambiguously."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's what I'd flag about the SlangDIT paper. The benchmark is a real contribution: it's the first to tie slang detection, cross-lingual explanation, polysemy annotation, and translation together in one English–Chinese testbed, at a scale that dwarfs CLIX and OpenSubtitles-Slang. The authors also do some curation right: they filter by model agreement, split so slang terms don't overlap between train and test, and include no-slang sentences to make detection meaningful.\n\nThe method part is standard long-CoT distillation: they distill thoughts from DeepSeek-R1-Distill-Qwen-32B and fine-tune three backbones. The results are consistent in direction—SlangOWL beats SFT-without-thinking on most metrics at all scales, and the hard polysemous testset shows a clear BLEU edge. The case study is genuinely illustrative, and the comparison to stronger models (GPT-4o, Google) in the appendix is a nice extra.\n\nThe soft spots are real but not fatal. Slang terms, explanations, and polysemy labels are all machine-generated, with only 200 samples human-checked, and no human evaluation of translations anywhere. The auxiliary task scores are therefore partly circular: the model is trained and evaluated on the same LLM judgments. The translation references themselves are human movie subtitles, which mitigates circularity for the translation claim, but the GPT-4o reference-free evaluator may still reward fluency that matches its own preferences. Also, the significance claim is shakier than the abstract implies: on the general test, the BLEU and CometK differences aren't significant, only GRB/GRF are. That's a modest caveat, but the paper should report confidence intervals and be more careful with \"significantly surpasses.\"\n\nWho benefits: anyone working on informal language translation or on benchmarks built from LLM annotation. It's a useful resource despite the evaluation gaps. I'd send it to serious referees, with a request that revisions add human translation evaluation, bootstrap or CI-based significance testing, and an annotation-bias analysis (e.g., how much of the gain is alignment with GPT-4o). Net: worth engaging; the central claim is plausible, not yet proven.","headline":"A large, thoughtfully split benchmark for slang translation, whose central claim is plausible but needs human evaluation and tighter statistics.","tokens_in":21257,"tokens_out":3365,"would_cite":true,"duration_ms":30423,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":null,"created_at":"2026-08-07T15:38:15.246230+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":null,"supporting_citations":[],"review_version":1}