Pith. sign in

REVIEW 3 major objections 4 minor 21 references

Toxicity-Aware Few-Shot Prompting for Low-Resource Singlish Translation

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A two-stage few-shot prompting pipeline with human-curated examples and embedding-based model selection preserves toxic Singlish tone in LLM translations, with GPT-4o mini the best of the four models tested.

desk verdict Solid pipeline, but the paper's headline comparative claim is not actually tested by the human data; the evidence behind it is an unvalidated embedding proxy. read the letter →

arxiv 2507.11966 v1 pith:7RVRRAWF submitted 2025-07-16 cs.CL cs.AIcs.CY

Ziyu Ge , Gabriel Chua , Leanne Tan , Roy Ka-Wei Lee This is my paper
classification cs.CLcs.AIcs.CY
keywords Singlishcode-mixedtranslationtoxicitypreservationfew-shotpromptinglow-resourcemachineLLMsafetysemanticsimilaritypromptoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that large language models can translate toxic Singlish into Chinese, Malay, and Tamil without sanitizing the offensive tone, despite scarce parallel data. It proposes a two-stage framework: humans iteratively curate and rank twenty Singlish–target few-shot examples so the model learns the right slang and tone, then embedding-based semantic similarity via direct and back-translation selects the best model–prompt configuration. Applying this to a Singlish safety corpus, the paper reports that GPT-4o mini is the most effective of the four LLMs tested, and that human ratings for Chinese and Malay come within 0.2 points of human gold translations. If this holds, the framework offers a low-annotation path to culturally sensitive content moderation in multilingual settings.

What carries the argument

The load-bearing mechanism is the reference-free selection score: cosine similarity in text-embedding-3-large space, computed both for the direct translation (Singlish source vs target) and for the back-translation (source vs re-translated source). This score replaces gold references, letting the authors rank models and prompt sizes without parallel corpora. The other half is the curated few-shot pool: twenty Singlish sentences, balanced between benign and harmful, refined through three rounds of annotator selection, which supply the tone and slang anchors in the final prompt.

What would settle it

Have independent annotators rate only tone and toxicity preservation for the same 200 translated sentences, then check whether those ratings reproduce the GPT-4o mini ranking and whether they correlate with text-embedding-3-large cosine similarities; near-zero correlation, or a different model topping human ratings, would show the selection step is not measuring toxicity preservation.

Watch

Extended reading notes

Core claim

The central claim is that the two-stage, human-in-the-loop pipeline preserves both semantic content and harmful tone in low-resource translation, and that GPT-4o mini is the strongest model for this task among Gemini 2.0 Flash, Grok 3 Beta Mini, DeepSeek-R1, and GPT-4o mini. Under the final pipeline, Chinese and Malay translations approach gold-reference quality, while Tamil lags, which the authors attribute to linguistic distance and to limited annotator diversity.

Load-bearing premise

The load-bearing premise is that cosine similarity between an embedding of the Singlish source and embeddings of its translations (or back-translations) captures whether meaning and toxic tone were preserved; if the embedding model misses code-mixed slang or subtle tone shifts, the model and prompt choices could be wrong.

Editorial extensions

If this is right

  • With GPT-4o mini and the curated few-shot pool, Singlish-to-Chinese and Singlish-to-Malay translations receive human ratings within 0.2 points of human gold translations, so the pipeline is usable where parallel data is scarce.
  • The reference-free selection step makes model and prompt choice possible without gold references, which is the main bottleneck in low-resource settings.
  • The optimal number of few-shot examples is language-dependent: roughly 15 for Chinese, 10 for Malay, and 20 for Tamil, so prompt size should be tuned per target language.
  • Tamil remains a hard case; even the best configuration is rated 2.49 versus 3.30 for gold references, with sanitized and emotionally flat outputs, so linguistically distant targets need more than prompt curation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's reported pairs, the same two-stage design should transfer to other code-mixed creoles such as Hinglish or Taglish; the main cost is recruiting annotators who can judge tone, not building parallel data.
  • The embedding-similarity proxy could be stress-tested by comparing it against a dedicated toxicity classifier; if the two diverge on implicit or context-dependent harm, the selection step would need a second signal.
  • A risk the paper does not fully resolve is that back-translation similarity can in principle be gamed by models that round-trip smoothly while changing meaning, so a human-checked tone audit on a held-out set is a cheap addition.
  • If LLMs keep improving on low-resource translation, the per-language k tuning and human three-round curation could become automated, but the paper's core insight that toxicity must be an explicit objective, not a side effect of fluency, would remain.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a two-stage human-in-the-loop framework for translating toxic Singlish content into Chinese, Malay, and Tamil while preserving informal and offensive tone. Stage 1 builds a 20-example few-shot pool through a three-round annotation procedure; Stage 2 selects among four LLMs and optimizes the number k of retrieved examples using cosine similarity between text-embedding-3-large embeddings, comparing direct translation similarity and back-translation similarity. A human evaluation then rates 200 outputs of the chosen GPT-4o mini pipeline on a 1-5 scale for meaning and tone fidelity. The paper claims that the approach improves retention of toxic language signals, that GPT-4o mini is the most effective model, and that the framework supports culturally sensitive moderation for low-resource languages.

Significance. The problem is real and understudied: general MT systems and safety filters often flatten culturally embedded markers of harm in code-mixed low-resource varieties. The paper contributes a concrete annotation workflow, a reference-free evaluation procedure, and a controlled-access corpus, and it is candid about annotator diversity and the limits of embedding-based similarity. If the comparative claims were fully supported, the framework would be a useful template for toxicity-preserving translation in content moderation. However, the current human evaluation only rates the final configuration in isolation; the evidence for the central comparative claims comes from an unvalidated embedding proxy. This gap is fixable with additional human evaluation on matched baselines, so the work is promising but not yet conclusive.

major comments (3)
  1. [Section 3, Table 2] The human evaluation does not contain the comparator needed for the paper's comparative claims. The 200 machine translations and the 20 gold-reference translations are not matched on the same source sentences; the gold set is the curated few-shot pool, so the observed closeness (Chinese 3.83 vs 4.07; Malay 4.09 vs 4.08) may reflect source difficulty rather than translation quality. More importantly, no human rating is collected for a zero-shot baseline, for other models, or for alternative k values. Consequently, the conclusion that the approach improved the retention of toxic language signals and that GPT-4o mini emerged as the most effective model is not independently supported by the human data. The authors should add a human evaluation on the same 200 sources for at least the zero-shot baseline, and ideally one other model, with ratings separated by meaning fidelity, tone, and toxicity.
  2. [Sections 2.2.1-2.2.3, Tables 1 and 3] Model selection and k optimization both rely on cosine similarity from text-embedding-3-large as a proxy for semantic fidelity and toxicity retention, and this same proxy is then used as the main quantitative evidence for improvement. This is circular in the sense that GPT-4o mini and the per-language k are chosen because they maximize the same metric later reported as the gain; the metric itself is never validated against human judgments. The authors concede in Section 4 that embedding similarity 'may overlook subtle shifts in tone.' I request either a validation study showing correlation between the embedding metric and human ratings on a held-out set, or a human-rated comparison of models and k values, at least for toxicity-specific dimensions.
  3. [Section 3 and Section 4] Toxicity retention is not directly measured anywhere in the paper. The human evaluation uses a single holistic score for 'meaning and tone,' and the automatic metrics are semantic-similarity proxies; neither provides a toxicity label, severity judgment, or comparison of offensive content before and after translation. The paper's own Limitations section states that the study does not explicitly measure subtle or implicit toxicity. A reader therefore cannot verify the headline claim that the pipeline improved the retention of toxic language signals; the final human scores could be high even if profanity were consistently softened, because a polite but semantically faithful translation can still receive an acceptable holistic tone rating in isolation. The evaluation should include a direct toxicity-preservation task, e.g., expert ratings of offensiveness or severity on matched source and target items, or a validated toxicity classifier applied to both sides.
minor comments (4)
  1. [Section 2.2.3 and Appendix B.2] The pointer 'see Appendix 3' should be 'Table 3,' and Appendix B.2 contains the typo 'Results are shown is Table 3.'
  2. [Table 4] The columns 'Jaccard R1/R2/R3' are not defined; specify what sets or annotation decisions the Jaccard similarity is computed over.
  3. [Appendix B.3] The DSPy results mix a baseline score of 0.672 with COPRO scores reported as percentages (60.6-62.1%); clarify the metric and scale so the 'marginal improvements' claim is interpretable.
  4. [Section 2.1.2] The relationship between the custom-submission counts (6.4 for Chinese, 8.8 for Malay, 5.6 for Tamil) and the final pool retention rates (nine LLM-generated examples for Chinese and Tamil, two for Malay) should be made explicit, since it is unclear whether the counts are cumulative across rounds, per-sentence averages, or something else.

Circularity Check

0 steps flagged · score 2.0 of 10

No derivation step reduces to its inputs; the overlap between the embedding metric used for model/k selection and the comparative support is an evidentiary limitation, not a circularity.

full rationale

The paper's derivation chain is empirically grounded rather than definitional. The few-shot pool is curated by human annotation (Section 2.1), the model comparison uses embedding cosine similarity (Section 2.2.1, Table 1), and the final pipeline is independently rated by humans (Section 3, Table 2). No equation is defined in terms of the target claim, no fitted parameter is later renamed as a prediction, and no load-bearing self-citation or imported uniqueness theorem appears. The only concern is that the comparative statements ('GPT-4o mini emerging as the most effective model', 'improved the retention of toxic language signals') are supported by the same text-embedding-3-large similarity metric used to select the model and k values, while the human evaluation rates only the final GPT-4o mini pipeline and does not rate other models or k values. That is an evidentiary gap about the validity of the proxy, and the authors explicitly concede in Section 4 that embedding-based similarity 'may overlook subtle shifts in tone.' It does not make the derivation circular, because the metric is an empirical proxy rather than an input that defines the outcome, and the human ratings provide independent evidence for the absolute quality of the chosen configuration. Score 2 reflects the minor but non-circular concern.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper rests on three domain assumptions rather than a mathematical derivation: embedding similarity as a fidelity proxy, representativeness of the 20-sentence pool, and the validity of 1-5 human ratings as a toxicity-preservation measure. The only fitted numeric parameter is k per language. No new entities are introduced.

free parameters (2)
  • k, number of few-shot examples per language = 15 (Chinese), 10 (Malay), 20 (Tamil)
    Selected by maximizing embedding-based direct semantic similarity on the 20 curated examples (Table 3, Appendix B.2); no held-out validation.
  • Few-shot pool size = 20 Singlish sentences
    Chosen by the authors, balanced between benign and harmful content, not optimized.
assumptions (3)
  • domain assumption Cosine similarity between embeddings of source and translated or back-translated text is a valid proxy for semantic fidelity and toxicity preservation.
    Used in Section 2.2.1 for direct and back-translation similarity; the authors acknowledge in Limitations that embedding-based similarity may overlook subtle tone shifts.
  • domain assumption The 20 selected Singlish sentences are representative of toxic and slang-heavy content in the LionGuard source.
    The few-shot pool, model selection, and k tuning all use this small sample; no distributional analysis is provided.
  • domain assumption Human 1-5 ratings of meaning and tone measure toxicity-preserving translation quality.
    Section 3 uses these ratings as ground truth; Limitations state implicit toxicity is not measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Toxicity-Aware Few-Shot Prompting for Low-Resource Singlish Translation." pith.science (2026). https://pith.science/paper/7RVRRAWF

@misc{pith2026250711966,
  author       = {Pith},
  title        = {Pith review of: Toxicity-Aware Few-Shot Prompting for Low-Resource Singlish Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7RVRRAWF}},
  note         = {Machine review of arXiv:2507.11966}
}
read the original abstract

As online communication increasingly incorporates under-represented languages and colloquial dialects, standard translation systems often fail to preserve local slang, code-mixing, and culturally embedded markers of harmful speech. Translating toxic content between low-resource language pairs poses additional challenges due to scarce parallel data and safety filters that sanitize offensive expressions. In this work, we propose a reproducible, two-stage framework for toxicity-preserving translation, demonstrated on a code-mixed Singlish safety corpus. First, we perform human-verified few-shot prompt engineering: we iteratively curate and rank annotator-selected Singlish-target examples to capture nuanced slang, tone, and toxicity. Second, we optimize model-prompt pairs by benchmarking several large language models using semantic similarity via direct and back-translation. Quantitative human evaluation confirms the effectiveness and efficiency of our pipeline. Beyond improving translation quality, our framework contributes to the safety of multicultural LLMs by supporting culturally sensitive moderation and benchmarking in low-resource contexts. By positioning Singlish as a testbed for inclusive NLP, we underscore the importance of preserving sociolinguistic nuance in real-world applications such as content moderation and regional platform governance.

Figures

Figures reproduced from arXiv: 2507.11966 by the authors.

Figure 1
Figure 1. The proposed framework for toxicity-preserving translation. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Screenshot of Annotation Platform – Round 1: Initial Translation Selection [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Screenshot of Annotation Platform – Round 2: Top Translation Re-evaluation [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Screenshot of Annotation Platform – Round 3: Final Choice [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Box plots of annotator ratings for Chinese, Malay, and Tamil translations across [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 10 canonical work pages

  1. [1]

    Unsupervised neural machine translation

    Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. arXiv preprint arXiv:1710.11041,

    arXiv

  2. [3]

    B.3 DSPy We utilized DSPy Khattab et al

    8 k SG → ZH SG → MS SG → TA Baseline 66.62 72.89 30.80 k = 5 69.76 73.57 31.82 k = 10 70.10 72.79 32.15 k = 15 70.23 73.63 32.10 k = 20 70.09 73.74 32.27 Table 3: Semantic similarity between Singlish (SG) and target translations—Chinese (ZH), Malay (MS), and Tamil (TA)—across different numbers of few-shot examplesk. B.3 DSPy We utilized DSPy Khattab et al...

    work page 2024

  3. [6]

    URL https://arxiv.org/abs/2407.10995. Google. Introducing gemini 2.0: our new ai model for the agen- tic era,

    work page Pith review arXiv

  4. [7]

    Accessed: 2025-05-07

    URL https://blog.google/technology/google-deepmind/ google-gemini-ai-update-december-2024/ . Accessed: 2025-05-07. Barry Haddow, Rachel Bawden, Antonio Valerio Miceli Barone, Jindˇrich Helcl, and Alexan- dra Birch. Survey of low-resource machine translation. Computational Linguistics, 48(3): 673–732, September

    work page 2024

  5. [10]

    URL https://arxiv.org/abs/2310. 03714. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. Unsuper- vised machine translation using monolingual corpora only.arXiv preprint arXiv:1711.00043,

    arXiv

  6. [11]

    Adam Lopez

    URL https: //arxiv.org/abs/2303.16634. Adam Lopez. Statistical machine translation. ACM Computing Surveys (CSUR), 40(3):1–49,

    arXiv

  7. [12]

    New embedding models and api updates, 2024b

    OpenAI. New embedding models and api updates, 2024b. URL https://openai.com/ index/new-embedding-models-and-api-updates/ . Accessed: 2025-05-12. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguist...

    work page 2025

  8. [13]

    Gpt-4o mini: advancing cost-efficient intelligence, 2024a

    OpenAI. Gpt-4o mini: advancing cost-efficient intelligence, 2024a. URL https://openai. com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ . Accessed: 2025-05-

    work page 2025

Show all 21 references
  1. [17]

    org/abs/2309.07423

    URL https://arxiv. org/abs/2309.07423. Bhaskarjit Sarmah, Kriti Dutta, Anna Grigoryan, Sachin Tiwari, Stefano Pasquali, and Dhagash Mehta. A comparative study of dspy teleprompter algorithms for aligning large language models evaluation metrics to human evaluation. arXiv prepr...

    arXiv

  2. [18]

    Felix Stahlberg

    URL https://arxiv.org/abs/2412.15298. Felix Stahlberg. Neural machine translation: A review. Journal of Artificial Intelligence Research, 69:343–418,

    arXiv

  3. [19]

    doi: 10.18653/v1/2023.acl-long.859

    Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.859. URL https://aclanthology.org/2023.acl-long.859/. Xing Wang, Zhengdong Lu, Zhaopeng Tu, Hang Li, Deyi Xiong, and Min Zhang. Neural machine translation advised by statistical machine translation. In P...

    2023 doi

  4. [20]

    {sentence}

    URL https://x.ai/news/grok-3. Accessed: 2025-05-12. 7 A Ethical Considerations The experiments of the proposed framework involved curating and annotating harmful content, including hate speech and explicit language, to support research in LLM safety. Native speakers were engag...

    2025

  5. [2002]

    Word embeddings for code- mixed language processing

    Adithya Pratapa, Monojit Choudhury, and Sunayana Sitaram. Word embeddings for code- mixed language processing. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, ...

    2018

  6. [2005]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,

    1901

  7. [2008]

    What talking you?: Translating code-mixed messaging texts to english

    Lynnette Hui Xian Ng and Luo Qi Chan. What talking you?: Translating code-mixed messaging texts to english. arXiv preprint arXiv:2411.05253,

    arXiv

  8. [2018]

    doi: 10.18653/v1/D18-1344

    Association for Computational Linguistics. doi: 10.18653/v1/D18-1344. URL https://aclanthology. org/D18-1344/. Nathaniel R. Robinson, Perez Ogayo, David R. Mortensen, and Graham Neubig. Chatgpt mt: Competitive for high- (but not low-) resource languages,

    doi

  9. [2020]

    Toxicity in multilingual machine translation at scale

    Marta R Costa-juss `a, Eric Smith, Christophe Ropers, Daniel Licht, Jean Maillard, Javier Ferrando, and Carlos Escolano. Toxicity in multilingual machine translation at scale. arXiv preprint arXiv:2210.03070,

    arXiv

  10. [2022]

    URL https://aclanthology.org/ 2022.cl-3.6/

    doi: 10.1162/coli a 00446. URL https://aclanthology.org/ 2022.cl-3.6/. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. How good are gpt models at machine translation? a comprehe...

    2022 arXiv

  11. [2023]

    doi: 10.18653/v1/2023.wmt-1.41

    Association for Computa- tional Linguistics. doi: 10.18653/v1/2023.wmt-1.41. URL https://aclanthology.org/ 2023.wmt-1.41/. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moa...

    2023 doi

  12. [2024]

    Jessica Foo and Shaun Khoo

    URL https://arxiv.org/abs/2404.13813. Jessica Foo and Shaun Khoo. Lionguard: Building a contextualized moderation classifier to tackle localized unsafe content,

    arXiv

  13. [2025]

    Maxim Enis and Mark Hopkins

    URL https://arxiv.org/abs/2501.12948. Maxim Enis and Mark Hopkins. From llm to nmt: Advancing low-resource machine translation with claude,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.