Pith. sign in

REVIEW 3 major objections 5 minor 26 references

SlangDIT: Benchmarking LLMs in Interpretative Slang Translation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new English-Chinese slang translation benchmark with polysemy labels, plus a deep-thinking model that detects, explains, and translates slang, outperforming vanilla and fine-tuned LLMs.

desk verdict A large, thoughtfully split benchmark for slang translation, whose central claim is plausible but needs human evaluation and tighter statistics. read the letter →

arxiv 2505.14181 v1 pith:FU2ITS7P submitted 2025-05-20 cs.CL

Yunlong Liang , Fandong Meng , Jiaan Wang , Jie Zhou This is my paper · ORCID
classification cs.CL
keywords slangtranslationexplanationllmsslangowldetectionmodelssentence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Slang is hard for translators because its meaning depends on the context and is often not literal. The paper builds a dataset from movie subtitles, using large language models to mark slang terms and write Chinese explanations for them. The dataset contains over 25,000 English-Chinese sentence pairs where at least one slang term appears, and it is split so that no slang term appears in more than one of the train, validation, and test sets. The authors then train a model, SlangOWL, that first thinks about whether the sentence contains slang, whether that slang has multiple meanings, and what it means in the current context. Only after this reasoning does the model produce a translation. On held-out tests, SlangOWL outperforms both uninstructed LLMs and models fine-tuned on the same final answers but without the thinking step. The improvements appear across several metrics, including BLEU, COMET, and scores from GPT-4o. The model also matches or beats much larger systems such as GPT-4o on some of the harder test examples, which involve slang terms with multiple meanings.
Extended reading notes

Core claim

The central claim is that the deep thinking approach of SlangOWL, which explicitly performs slang detection, polysemy judgment, cross-lingual explanation, and translation, significantly surpasses vanilla LLMs and supervised fine-tuned models without thinking. The paper states this in the abstract: 'our deep thinking approach indeed enhances the performance of LLMs where the proposed SLangOWL significantly surpasses the vanilla models and supervised fine-tuned models without thinking.' If correct, the improvement is consistent across model scales and metrics on both the general and hard test sets.

Load-bearing premise

The gold-standard annotations (slang terms, cross-lingual explanations, and polysemy labels) are generated automatically by LLMs (Qwen2.5-72b, Llama3.3-70b, GPT-4o) and filtered by model agreement, with only 200 samples human-validated. If these LLM-generated labels carry systematic errors or model-specific biases, the benchmark's validity and the measured performance improvements are compromised. This assumption enters in Section 3.2 (Annotation Procedure) and Section 3.3 (Annotation Quality Assessment).

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces SlangDIT, a new English-Chinese benchmark for interpretative slang translation, consisting of three sub-tasks: slang detection, cross-lingual slang explanation, and slang translation. The dataset contains over 25k sentence pairs derived from movie subtitles, annotated with slang terms, Chinese explanations, polysemy labels, and translations, using a pipeline of Qwen2.5-72b, Llama3.3-70b, and GPT-4o, with a 200-sample human validation of slang detection and explanation. The authors also propose SlangOWL, a 'deep thinking' model that explicitly reasons about whether a sentence contains slang, whether the slang is polysemous, what the slang means in context, and finally translates the sentence. SlangOWL is trained on thought traces distilled from DeepSeek-R1-Distill-Qwen-32B, conditioned on the gold labels. Experiments on three backbone LLMs (Qwen2.5-7B/14B-Instruct and Llama-3.1-8B-Instruct) report consistent improvements over vanilla and SFT baselines on the general and hard test sets across BLEU, Comet, CometK, and GPT-4o-based GRB/GRF metrics.

Significance. If the benchmark and evaluation are valid, SlangDIT fills a genuine gap by linking slang detection and explanation to translation, and the proposed deep-thinking approach is a plausible way to improve slang translation. Strengths include the scale of the dataset, the held-out test sets with no slang-term overlap, the inclusion of polysemy annotation, the use of multiple backbones, and the public release of the data and code. The main results are consistent in direction across model sizes and metrics, and the hard test set provides a challenging polysemy-focused evaluation. However, the current evidence is weakened by the heavy reliance on LLM-generated gold labels and LLM-as-judge evaluation without adequate human validation of translations, and by the incomplete reporting of statistical significance details. These issues directly affect whether the central claim of the abstract—that SlangOWL significantly surpasses vanilla and SFT models—is established.

major comments (3)
  1. [§3.2–3.3, Appendix C] The gold annotations for slang detection, explanation, and polysemy are produced entirely by Qwen2.5-72b, Llama3.3-70b, and GPT-4o (§3.2), and only 200 samples are human-validated, covering only slang detection and explanation (§3.3). No human evaluation of the translations is reported; instead, GRB and GRF are computed by GPT-4o (Appendix C). Because SlangOWL is trained on thoughts distilled from DeepSeek-R1-Distill-Qwen-32B that are conditioned on these same LLM-generated labels, the consistent gains over SFT baselines may partly reflect alignment with the annotation/evaluation pipeline rather than human-judged translation quality. This bears directly on the abstract's central claim that SlangOWL 'significantly surpasses' vanilla and SFT models. The authors should add a human translation evaluation on a representative sample (e.g., 100–200 hard-test instances) with adequacy and idiomaticity judgments, and report inter-annotator agreement.
  2. [§5.1, Tables 3–4] The statistical significance annotation ('†'/'††') is under-specified. The paper only states that a t-test with p<0.05 and p<0.01 was used, without reporting the number of samples, whether the test is paired over sentences or bootstrap-based, or whether any correction for multiple comparisons was applied across the many metrics and models. Without these details, the dagger annotations are not verifiable. Moreover, on the general test the BLEU and CometK differences between SlangOWL-14B and SFT-14B (24.94 vs. 24.51 and 71.38 vs. 71.29) are not marked significant, so the claim of consistent significant superiority is only partially supported by the data.
  3. [§3.2, Table 4] The construction of the hard test set and its references is ambiguous. The text says that 1,165 instances are randomly sampled from the polysemous slang terms, and that for the remaining 85% GPT-4o generates possible meanings and translation pairs. It is not clear whether the 1,165 hard-test instances also have GPT-4o-generated translations or whether they come from the original MSCTD corpus with human translations. If the hard-test references are GPT-4o-generated, the BLEU/Comet improvements in Table 4 partly measure agreement with GPT-4o output rather than human translation quality. The authors should clarify the exact provenance of the hard-test Chinese references and, ideally, include human-written references for at least a subset of the hard test.
minor comments (5)
  1. [Abstract, §1] The model name is spelled inconsistently as 'SlangOWL' and 'SLangOWL' in the abstract and text; please standardize it.
  2. [§1] The phrase 'the main reason is lacking of a benchmark' should be 'the main reason is the lack of a benchmark'.
  3. [§3.2] 'we filter the sentences that contains repetitive slang terms' should be 'contain'.
  4. [§5.1, Table 3] The metric name is given as 'CometKivi' in the text but 'CometK' in Table 3; please use one consistent spelling (e.g., CometKiwi).
  5. [Appendix B] The sentence 'Please refer to Appendix B for details' appears both in Section 5.1 and at the beginning of Appendix B itself; the latter is a copy-paste error and should be replaced with the actual details.

Circularity Check

1 steps flagged · score 4.0 of 10

Annotator–evaluator overlap (GPT-4o) creates a partial evaluation loop; no by-construction circularity in the core benchmark/training derivation.

  1. other [§3.2 Polysemy Annotation; §5.1 Metrics (Appendix C)]
    "To explore the context impact on the slang term, we utilize GPT-4o to annotate whether each slang term is polysemous. … Besides, following recent studies … we also use GPT-4o as evaluators in reference-based and reference-free manner, which we refer to as GRB and GRF, respectively."

    The hard testset is built from exactly the polysemous slang terms whose labels GPT-4o created (Section 3.2), and the largest advertised hard-test translation gain is on GRF (SlangOWL-14B 82.35 vs. SFT-14B 79.64), a score assigned by GPT-4o. SlangOWL's thought data are distilled from DeepSeek-R1-Distill-Qwen-32B using those same gold labels, so the model is explicitly trained to align with the annotation pipeline. GRF therefore rewards agreement with the annotator/evaluator rather than with an independent human judgment. This is an evaluation loop, not an equation-level reduction: BLEU, COMET and CometKiwi are external, and 200 detection/explanation samples were human-validated, so the loop is partial.

full rationale

The paper's central derivation chain is empirical, not mathematical. SlangDIT is a constructed benchmark, and SlangOWL's training is a standard distillation from DeepSeek-R1-Distill-Qwen-32B conditioned on gold labels; test sets are held out and slang terms do not overlap across splits. I do not find a step where an equation reduces to its own input or a fitted parameter is renamed a prediction. The nontrivial circularity risk is concentrated in the evaluator/annotator overlap: GPT-4o contributes to the annotation of exactly the phenomenon (polysemy) that defines the hard test, and GPT-4o also serves as the GRB/GRF grader, so the hard-test GRF margin partly measures how well the model has learned to reproduce GPT-4o's annotation preferences. This is mitigated by independent metrics: BLEU improves on the hard test (21.41 vs. 19.18), COMET/CometKiwi are external learned metrics, and 200 detection/explanation samples were human-validated, though translations were not separately human-evaluated. Self-citations (MSCTD, DRT-o1) are used as data/baselines and are not load-bearing. The central claim therefore has substantial independent content; the loop is partial, so the score is 4.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper is an empirical benchmark and method paper, not a mathematical derivation. The only hand-chosen numeric values are the analysis thresholds in Section 5.3. The assumptions are domain-level about data, annotation, and evaluation.

free parameters (2)
  • CSU ROUGE-L threshold = 0.4
    Used to define 'correct slang understanding' in the analysis of Section 5.3; hand-chosen, not fitted.
  • GoodT thresholds = GRB > 70, GRF > 80
    Used to define 'good translation' in the analysis of Section 5.3; hand-chosen for the analysis only.
assumptions (3)
  • domain assumption Movie subtitles from MSCTD provide natural, diverse slang contexts and reliable Chinese references.
    Data source chosen in Section 3.1. If the underlying translations are poor or aligned, the benchmark quality suffers.
  • domain assumption LLM-based judgments (Qwen2.5-72b, Llama3.3-70b, GPT-4o) can accurately identify slang and generate explanations.
    The annotation pipeline in Section 3.2 relies entirely on LLM outputs, with only 200 samples human-validated in Section 3.3.
  • domain assumption BLEU, ROUGE, Comet, CometKiwi, and GPT-4o scores reflect translation and explanation quality for slang.
    Metrics used in Section 5.1. No human translation evaluation is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SlangDIT: Benchmarking LLMs in Interpretative Slang Translation." pith.science (2026). https://pith.science/paper/FU2ITS7P

@misc{pith2026250514181,
  author       = {Pith},
  title        = {Pith review of: SlangDIT: Benchmarking LLMs in Interpretative Slang Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FU2ITS7P}},
  note         = {Machine review of arXiv:2505.14181}
}
read the original abstract

The challenge of slang translation lies in capturing context-dependent semantic extensions, as slang terms often convey meanings beyond their literal interpretation. While slang detection, explanation, and translation have been studied as isolated tasks in the era of large language models (LLMs), their intrinsic interdependence remains underexplored. The main reason is lacking of a benchmark where the two tasks can be a prerequisite for the third one, which can facilitate idiomatic translation. In this paper, we introduce the interpretative slang translation task (named SlangDIT) consisting of three sub-tasks: slang detection, cross-lingual slang explanation, and slang translation within the current context, aiming to generate more accurate translation with the help of slang detection and slang explanation. To this end, we construct a SlangDIT dataset, containing over 25k English-Chinese sentence pairs. Each source sentence mentions at least one slang term and is labeled with corresponding cross-lingual slang explanation. Based on the benchmark, we propose a deep thinking model, named SlangOWL. It firstly identifies whether the sentence contains a slang, and then judges whether the slang is polysemous and analyze its possible meaning. Further, the SlangOWL provides the best explanation of the slang term targeting on the current context. Finally, according to the whole thought, the SlangOWL offers a suitable translation. Our experiments on LLMs (\emph{e.g.}, Qwen2.5 and LLama-3.1), show that our deep thinking approach indeed enhances the performance of LLMs where the proposed SLangOWL significantly surpasses the vanilla models and supervised fine-tuned models without thinking.

Figures

Figures reproduced from arXiv: 2505.14181 by the authors.

Figure 1
Figure 1. Some examples of the SlangDIT benchmark. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The generated deep thinking thought (training example) by DeepSeek-R1-Distill-Qwen-32B. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Two prompts used in GRB and GRF during evaluation via GPT-4o where “[src]”,“[ref]” and “[hyp]” mean [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: The prompt used in prompting vanilla models where “[sentence]” means the source sentence. [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Case Study [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: The prompt used in generating deep thinking thought by DeepSeek-R1-Distill-Qwen-32B. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: The system prompt used in the section of Annotation Procedure. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: The prompt used in the section of Annotation Procedure. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: The prompt used in the section of Annotation Procedure. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: The prompt used in the section of Annotation Procedure. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: The prompt used in the section of Annotation Procedure. [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 26 canonical work pages

  1. [1]

    - If slang is present, identify the specific slang term(s)

    Identify Slang: - Examine the sentence to determine if it contains any slang terms or expressions. - If slang is present, identify the specific slang term(s)

    work page

  2. [2]

    Provide Explanation: - For each identified slang term, provide a brief explanation of its meaning and usage in Chinese

    work page

  3. [3]

    Translate the Sentence: - Translate the entire sentence into the target language, ensuring that the meaning of the slang is accurately conveyed in th e translation

    work page

  4. [4]

    No slang

    Output the Results: - If slang is present, output the identified slang term(s), their explanation(s), and the translated sentence. - If no slang is present, output "No slang" and the translated sentence. Examples: - Input Sentence: "She's always spilling the tea about her coworkers." Output: \n - Slang: "Spilling the tea" - Explanation: 这个俚语的意思是分享八卦或泄露秘密。...

    work page

  5. [5]

    No slang or non slang usage: If there is no slang in the sentence, or if it contains slang but is not slang in the current co ntext, please explain why the English is translated into the above Chinese

    work page

  6. [6]

    Then, based on the above analysis, provide reasons for translating the sentence into the Chinese version mentioned above

    Contains slang: (analyzed in the following two situations) -Non polysemous slang: If the slang contained in the sentence is not a polysemous word, please analyze the source and purpose of the slang, and explain its specific meaning in the current sentence. Then, based on the above analysis, provide reasons for translating the sentence into the Chinese ver...

    work page

  7. [9]

    That party was lit!

    Example: Sentence: "That party was lit!" Answer: Yes Please analyze the following sentence and note that you should only output 'no' if no slang identified, or ‘yes’ with slang term. [sentence] ''' PROMPT of slang extraction = ''' Y our task is to analyze the given English sentence and determine whether it contains any slang

    work page

  8. [10]

    Analyze the Sentence: Read the sentence carefully to identify any informal or non-standard language

    work page

Show all 26 references
  1. [11]

    Slang is typically informal language that may not be found in standard dictionaries and is often used in casual conversation

    Identify Slang: Determine if any word or phrase in the sentence qualifies as slang. Slang is typically informal language that may not be found in standard dictionaries and is often used in casual conversation

  2. [12]

    That party was lit!

    Example: Sentence: "That party was lit!" Slang: "Lit” Please analyze the following sentence and note that you should only output 'no slang' if no slang identified, or the identified slang term. [sentence] ''' Figure 8: The prompt used in the section of Annotation Procedure. PR...

  3. [13]

    像狗一样 - 字面意思，指行为或表现像狗。

  4. [14]

    过得很糟糕 - 俚语，指生活或处境非常糟糕。

  5. [15]

    忠诚 - 引申含义，比喻对某人或某事非常忠诚。 [user]: Fuckin' 是一词多义词语，请仅输出其在不同上下文下的可能中文翻译 [assistant]:

  6. [16]

    他妈的 - 字面意思，用于表达愤怒、惊讶或强调

  7. [17]

    真是 - 俚语，为形容词，表示非常或极其，用于加强语气，表示强烈的情感或态度。 [user]: beat the devil out of it 是一词多义词语，请仅输出其在不同上下文下的可能中文翻译 [assistant]:

  8. [18]

    使劲地清理 - 字面意思，指用力地清除灰尘或污垢，通常用于清洁地毯或衣物。

  9. [19]

    狠狠地打 - 俚语，指用力地击打某物。这个词组常用于描述某人对某物进行激烈的打击或处理。'' PROMPT = '''[SlangTerm] 是一词多义词语，请仿照例子仅输出其在不同上下文下的可能中文翻译''' Figure 10: The prompt used in the section of Annotation Procedure. SYSTEM_PROMPT of generating translation for each meaning = '''你是一名人工翻译专家，非常擅长为一词多义词生成样例，现在需要你为该词的每一种含 义生...

  10. [20]

    副手 - 引申含义，指职位或权力上仅次于最高领导的人。

  11. [21]

    厕所 - 俚语，在某些俚语或非正式用法中，指厕所。也用来委婉地指排便。这个词组通常在非正式场合使用，避免直接提到排便这一行为。 [assistant]:

  12. [22]

    She finished the race in Number two position.\n 她在比赛中获得了第二名。

  13. [23]

    As the CEO's Number two, he has a lot of responsibilities.\n 作为首席执行官的副手，他有很多责任。

  14. [24]

    He said he needed to go Number two before the long trip.\n 他说在长途旅行前他需要上厕所。 [user]: dropped an a-bomb 在不同上下文中有如下含义，请为每一种含义生成一个英中翻译句对，英语句子中需包含当前词 dropped an a-bomb 或 dropped an a- bomb 的变形词：

  15. [25]

    投下原子弹 - 字面意思，指将原子弹投放到目标区域。

  16. [26]

    爆料 - 俚语，指透露出令人震惊或重大的消息。 [assistant]:

  17. [27]

    The United States dropped an a-bomb on Hiroshima in 1945, causing massive destruction.\n 美国在1945年向广岛投下了一颗原子弹，造 成了巨大的破坏。

    1945

  18. [28]

    During the interview, the celebrity dropped an a-bomb by announcing her unexpected retirement.\n 在采访中，这位名人爆料宣布了她 意想不到的退役消息。''' PROMPT = '''[SlangTerm] 在不同上下文中有如下含义，请为每一种含义生成一个英中翻译句对，英语句子中需包含当前词 [SlangTerm] 或 [SlangTerm] 的 变形词： [SlangExplanation]''' Figure 11: The prompt used i...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.