Pith. sign in

REVIEW 4 major objections 6 minor 22 references

TULUN: Transparent and Adaptable Low-resource Machine Translation

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TULUN claims that prompting an LLM with glossary entries and similar past translations turns a base machine-translation draft into a domain-adapted translation for low-resource languages, without any fine-tuning.

desk verdict Useful applied MT tool with honest evaluation, but the FLORES claim needs a leakage check before I'd trust the 2.8-point average. read the letter →

arxiv 2505.18683 v1 pith:SMRPHCYO submitted 2025-05-24 cs.CL

Raphaël Merx , Hanna Suominen , Lois Hong , Nick Thieberger , Trevor Cohn , Ekaterina Vylomova This is my paper · ORCID
classification cs.CL
keywords machinetranslationlow-resourcelanguagesLLMpost-editingglossarymemoryterminology-awaredomainadaptationopen-sourceplatform
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that low-resource machine translation can be adapted to specialized domains without model fine-tuning. The proposed system, TULUN, runs a base MT model and then asks a large language model to post-edit the draft, supplying it with glossary entries and similar past translations retrieved from a translation memory. On medical translation into Tetun and disaster-relief translation into Bislama, this two-stage pipeline gains 16.90–22.41 ChrF++ points over the base MT systems; on six low-resource languages from FLORES-200 it averages 2.83 ChrF++ points above NLLB-54B, with one language showing a small loss. If correct, the result matters because it turns curation of terminology resources into a practical substitute for training or fine-tuning, which small organizations cannot usually do.

What carries the argument

The load-bearing mechanism is a retrieval-augmented post-editing pipeline with no weight updates. A base MT model produces a draft; then the LLM is prompted with glossary entries whose source tokens overlap the input, retrieved by unigram and bigram overlap, and the top $N$ translation-memory sentences according to BM25, a standard ranking function based on token frequency and document length. The prompt includes few-shot examples and chain-of-thought instructions, and the LLM's revised translation becomes the output. The user-facing interface makes the retrieved evidence visible and lets non-technical users edit the glossary and memory, which the authors argue is what makes the system adaptable and transparent.

What would settle it

Re-run the FLORES-200 experiment after removing from the translation-memory pool every sentence pair with an exact match or high n-gram overlap with the devtest set; if the average 2.83 ChrF++ gain falls to near zero, the benchmark gain is retrieval of near-duplicate answers rather than a generalizable post-editing ability. A complementary test on the real-world data is to ablate the glossary and the translation memory separately to see which resource drives the improvement.

Watch

Extended reading notes

Core claim

The central claim is that LLM post-editing, fed with user-supplied glossary matches and translation-memory examples, is sufficient to produce large domain-adaptation gains on low-resource languages. The evidence is a comparison between NMT-only baselines and the TULUN pipeline: 16.90–22.41 ChrF++ average gains on the two real-world tasks, an average 2.83 ChrF++ gain over NLLB-54B across six FLORES-200 languages, and qualitative cases where the post-editor repairs a domain term or a hallucinated span. The authors also claim the system is transparent and usable, reporting a System Usability Scale score of 81.25 and usefulness ratings of 4–5 out of 5.

Load-bearing premise

The main load-bearing premise is that the translation-memory sentences used in the FLORES evaluation are genuinely similar to the input but not near-copies of the test sentences, and that the post-editing LLM can actually follow the glossary and memory it is given.

Editorial extensions

If this is right

  • Organizations that hold a domain glossary and a modest translation memory can adapt MT to a specialized domain without training models or hiring engineers, provided they can call an LLM API.
  • The largest gains should appear exactly where the base MT is weakest and domain terminology matters, so specialized-domain use should benefit more than general benchmark text.
  • Because glossary and memory entries are editable and take effect immediately, translation quality can be improved incrementally by the people who use the system.
  • Showing users the glossary matches and memory hits that guided each edit gives them a way to audit the system's output, supporting human oversight in sensitive domains such as health and disaster response.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: filtering the FLORES translation-memory pool to remove near-duplicates of the devtest set would separate a genuine post-editing skill from answer retrieval; if the 2.83-point average survives that filter, the generalizability claim is much stronger.
  • Editorial inference: the Rundi drop suggests a confidence gate that applies the LLM's rewrite only when the glossary-based change is likely correct could make the pipeline at least neutral on every language.
  • Editorial inference: because the MT draft and the LLM editor are separate components that can be swapped independently, the reported results can be read as a lower bound that should climb as either component improves.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents TULUN, an open-source web-based platform that combines a neural machine translation (NMT) engine with LLM-based post-editing, using user-provided glossaries and translation memories retrieved at inference time. The system is designed to be transparent, adaptable, and usable by non-technical organizations, with no model fine-tuning required. The authors report two sets of evaluations: (1) applied medical and disaster-relief translation tasks for Tetun and Bislama, where TULUN improves over baseline NMT systems by 16.90-22.41 ChrF++ points, and (2) a FLORES-200 benchmark covering six low-resource languages, where TULUN improves over NLLB-54B by an average of 2.8 ChrF++ points. They also report a usability study with an average SUS score of 81.25. The paper's central quantitative claims are plausible but rest on evaluation details that are not fully verified, particularly possible test-set overlap with the retrieved translation memory and insufficiently controlled baselines.

Significance. If the reported gains hold under closer scrutiny, TULUN would be a practically valuable contribution: it lowers the barrier to domain-adapted MT for low-resource languages, is transparent about which glossary and translation-memory entries influenced the output, and is released as open source with real deployment partners. The idea of combining NMT with LLM post-editing guided by retrieval is well-motivated and builds on prior work. The strongest evidence is the specialized-domain evaluation, where the system is used in real organizational workflows with domain-specific resources. However, the benchmark evaluation on FLORES and the domain evaluations both share a possible retrieval-leakage risk that is not addressed in the manuscript, and the LLM-only baselines do not control for the glossary and translation-memory context. These issues are load-bearing for the paper's headline accuracy claims, so the significance can only be fully assessed after the authors supply the missing controls and contamination checks.

major comments (4)
  1. [§4.2 (Data)] The FLORES evaluation populates the post-editing prompt with parallel sentences from allenai/nllb and evaluates on FLORES-200 devtest, but the paper reports no check for exact or near-duplicate overlap between the retrieved translation-memory sentences and the 1,012 test sentences. Because FLORES-200 devtest source sentences are public English text and the NLLB-derived corpus is a large web-mined parallel corpus, this is not a remote possibility; if any test sentence appears in the translation memory with the same target translation, the LLM post-editor could copy the reference, inflating the reported +2.83 average and the per-language deltas in Table 2. The negative Rundi result does not rule out contamination, since overlap could be language-specific and still move the average. I ask the authors to (a) report exact-match and fuzzy/paraphrase overlap statistics between the allenai/nllb translation-memory sentences and the FLORES-200 devtest set, (b) rerun the evaluation with any overlapping sentences removed from the retrieval pool, and (c) clarify whether Table 2's caption phrase 'sentences from the NLLB training set' is equivalent to the allenai/nllb corpus described in the text, since that distinction matters for contamination assessment.
  2. [§4.1.1] For the Tetun and Bislama evaluations, the paper states that 451 and 841 sentences are 'reserved' for evaluation but does not explicitly state that these held-out sentences are excluded from the translation memory used for BM25 retrieval during post-editing. If a test sentence is retained in the retrieval pool, the LLM prompt may contain the reference translation, which would directly invalidate the reported gains of 16.90-22.41 ChrF++ points in Table 1. Please state clearly whether the evaluation subsets were removed from the translation memory before retrieval, and if so, describe the procedure; if not, the experiment needs to be rerun with strict exclusion.
  3. [§4.2 (Models) and Table 1] The 'LLM only' baseline is not a controlled comparison for isolating the effect of LLM post-editing. The proposed system provides the LLM with a glossary and retrieved translation-memory sentences, while the LLM-only baseline uses only fixed examples. The comparison therefore conflates the benefit of the additional in-context resources with the benefit of the post-editing step itself. To support the claim that 'LLM-based post-editing' is the active ingredient, the authors should add an LLM-only condition that receives the same glossary and translation-memory context (without the initial NMT hypothesis), and ideally an NMT+APE condition with the post-editor but without glossary/TM retrieval. Without these controls, the relative contributions of the NMT hypothesis, the LLM, and the retrieved resources remain unidentified.
  4. [§4.1.2 and Ethics statement] The usability and usefulness claims rest on responses from two participants, both of whom are acknowledged to be co-authors of the paper. The reported SUS score of 81.25 and '5/5' usefulness rating are therefore not independent evidence of user-friendliness, and the paper's abstract and introduction present these as validated system characteristics. I recommend reporting this as an anecdotal pilot study, tempering the corresponding claims, and, if possible, collecting responses from independent end users. This does not affect the accuracy evaluation but it is a substantive part of the paper's contribution.
minor comments (6)
  1. [Throughout] All evaluation numbers in Tables 1 and 2 appear to be single runs with no variance estimates, decoding temperature, or number of seeds reported. Given the LLM's stochasticity, a small number of repeated runs with mean and standard deviation would substantially strengthen the empirical claims.
  2. [§5 (Discussion)] The sentence 'resulting in +0.25 points compared to NLLB' for Rundi is ambiguous: it is not clear whether this +0.25 is the difference obtained without the glossary, and it would help to show the value in Table 2's format or as an explicit baseline row.
  3. [§4.2 (Data)] The data statement mentions that the Tetun and Bislama datasets are 'available upon request,' but no contact or data-sharing details are provided beyond institutional names. For reproducibility, the authors should provide a clear access procedure or a hosted anonymized version where ethically permissible.
  4. [Appendix A] In the example prompt, the sentence 'The sentences will be written' is incomplete and appears to be a cut-off explanation; this should be rewritten to describe the example format fully.
  5. [Table 1] The table header uses 'opus-mt-en-**' as a placeholder that obscures the two different model names used for Tetun (opus-mt-en-tdt) and Bislama (opus-mt-en-bi); listing both explicitly would improve clarity.
  6. [§1 and Table 2] The abstract says 'average improvement of 2.8 ChrF points over NLLB-54B,' while Table 2 and §4.2 report an average of 2.83 ChrF++ points; the paper should consistently refer to ChrF++ and avoid rounding the metric name.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: all central claims are empirical comparisons against external systems and benchmarks.

full rationale

TULUN does not propose a mathematical derivation or fitted model; its results are empirical ChrF++ scores. The reported gains in Tables 1 and 2 compare a pipeline (NMT + LLM post-editing with glossary/TM retrieval) against external baselines (MADLAD-400-10B, OPUS-MT, NLLB-54B, Gemini) and an external benchmark (FLORES-200 devtest). The glossaries come from partner organizations, GATITOS, or user input, not from the system's own output; the translation memories are organization-owned corpora or allenai/nllb data, again external to the evaluation procedure. No parameter is fitted to the test sets and then renamed as a prediction. The only self-citation, Merx et al. (2024) on the scarcity of Tetun health corpora, is motivational and not load-bearing. The Discussion candidly reports a negative result for Rundi and an ablation without glossary, which is inconsistent with the idea that the headline gains are forced by construction. The possible near-duplicate overlap between the allenai/nllb translation memory and FLORES-200 devtest is a data-leakage/soundness risk, not a circularity; it does not show that any stated prediction reduces to its inputs. Accordingly, no circular step can be quoted from the paper, and the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no invented theoretical entities and no numerical parameter fitted to the reported outcomes. The quantitative claims depend on external data resources (partner glossaries and memories, GATITOS, NLLB-derived parallel sentences) and on the behavior of proprietary MT and LLM services, plus a few unreported configuration choices such as the retrieval count and prompt details. This shifts the burden from derivation to empirical data hygiene.

free parameters (3)
  • Translation memory retrieval count N = not reported
    The admin UI lets users set the number of BM25 matches injected into the prompt; the value used for the reported tables is not stated.
  • Few-shot example configuration in post-editing prompt = not reported
    Appendix A shows an example prompt but the exact number and selection of few-shot chain-of-thought examples for Tables 1 and 2 are not specified.
  • Glossary n-gram match range = 1-2 grams
    The retrieval method uses {1,2}-gram overlap with source tokens; this is a fixed design choice rather than a fitted value.
assumptions (4)
  • domain assumption Partner glossaries and translation memories are accurate enough to guide post-editing.
    No quality audit of the Tetun medical glossary (2,698 entries), Bislama dictionary (5,769 entries), or translation memories is reported; evaluation treats them as reliable in-domain context (Section 4.1.1).
  • domain assumption NLLB-derived parallel sentences used as translation memory do not overlap with FLORES-200 devtest.
    Section 4.2 populates the prompt with sentences from allenai/n-llb and evaluates on FLORES devtest; no deduplication or leakage check is reported.
  • domain assumption Gemini 2.0 Flash can follow post-editing instructions for each low-resource language.
    The method depends on LLM instruction-following; the paper itself concedes in Section 5 that for extremely low-resource languages with rare scripts this may fail and cause hallucinations.
  • domain assumption English spaCy tokenization and BM25 retrieval surface the most useful glossary and memory entries.
    Section 3.2 defines retrieval but no retrieval-quality evaluation is reported; degraded retrieval would reduce post-editing gains.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TULUN: Transparent and Adaptable Low-resource Machine Translation." pith.science (2026). https://pith.science/paper/SMRPHCYO

@misc{pith2026250518683,
  author       = {Pith},
  title        = {Pith review of: TULUN: Transparent and Adaptable Low-resource Machine Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SMRPHCYO}},
  note         = {Machine review of arXiv:2505.18683}
}
read the original abstract

Machine translation (MT) systems that support low-resource languages often struggle on specialized domains. While researchers have proposed various techniques for domain adaptation, these approaches typically require model fine-tuning, making them impractical for non-technical users and small organizations. To address this gap, we propose Tulun, a versatile solution for terminology-aware translation, combining neural MT with large language model (LLM)-based post-editing guided by existing glossaries and translation memories. Our open-source web-based platform enables users to easily create, edit, and leverage terminology resources, fostering a collaborative human-machine translation process that respects and incorporates domain expertise while increasing MT accuracy. Evaluations show effectiveness in both real-world and benchmark scenarios: on medical and disaster relief translation tasks for Tetun and Bislama, our system achieves improvements of 16.90-22.41 ChrF++ points over baseline MT systems. Across six low-resource languages on the FLORES dataset, Tulun outperforms both standalone MT and LLM approaches, achieving an average improvement of 2.8 ChrF points over NLLB-54B.

Figures

Figures reproduced from arXiv: 2505.18683 by the authors.

Figure 1
Figure 1. System overview with example translation [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Translation View with the MT text, post-edited [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 20 canonical work pages

  1. [1]

    I think that I would like to use this system frequently. 5 5

    work page

  2. [2]

    I found the system unnecessarily complex. 1 1

    work page

  3. [3]

    I thought the system was easy to use. 5 5

    work page

  4. [4]

    I think that I would need the support of a technical person to be able to use this system. 1 2

    work page

  5. [5]

    Low-Resource Machine Translation through Retrieval-Augmented LLM Prompting: A Study on the Mambai Language. In Proceedings of the 2nd Workshop on Resources and Technologies for Indige- nous, Endangered and Lesser-resourced Languages in Eurasia (EURALI) at LREC-COLING 2024, pages 1–11, Torino, Italia. ELRA and ICCL. Yasmin Moslem, Rejwanul Haque, John D. K...

    work page 2024

  6. [6]

    I thought there was too much inconsistency in this system. 1 2

    work page

  7. [7]

    I would imagine that most people would learn to use this system very quickly. 5 3

    work page

  8. [8]

    I found the system very cumbersome to use. 1 2

    work page

Show all 22 references
  1. [9]

    I felt very confident using the system. 5 4

  2. [10]

    2 2 Table 3: Usability ratings Statement R1 R2

    I needed to learn a lot of things before I could get going with this system. 2 2 Table 3: Usability ratings Statement R1 R2

  3. [11]

    I found the various functions in this system were well integrated. 1 2

  4. [17]

    Using this system improves the quality of my translations. 5 4

  5. [18]

    Using this system increases my productiv- ity when translating documents. 5 4

  6. [19]

    Using this system enhances my effective- ness in maintaining terminology consistency. 4 4

  7. [20]

    Using this system makes it easier to trans- late technical/specialized content. 4 5

  8. [21]

    The glossary and translation memory fea- tures are useful for my translation work. 5 5

  9. [22]

    5 5 Table 4: Usefulness ratings

    Overall, I find this system useful for my translation tasks. 5 5 Table 4: Usefulness ratings

  10. [594]

    Steven Bird

    Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/10447310802205776. Steven Bird. 2024. Must NLP be Extractive? In Pro- ceedings of the 62nd Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 14915–14929, Bangkok, Thaila...

    2024 doi

  11. [2022]

    Computational Linguistics, 48(3):673–732

    Survey of Low-Resource Machine Translation. Computational Linguistics, 48(3):673–732. Place: Cambridge, MA Publisher: MIT Press. Alexander Jones, Isaac Caswell, Orhan Firat, and Ishank Saxena. 2023. GATITOS: Using a New Multilin- gual Lexicon for Low-resource Machine Translati...

    2023

  12. [2023]

    Studies in Health Technology and Informatics, 302:408–412

    Scale Up Multilingualism in Health Emer- gency Learning: Developing an Automated Tran- scription and Translation Tool. Studies in Health Technology and Informatics, 302:408–412. Viswanath Venkatesh, Michael G. Morris, Gordon B. Davis, and Fred D. Davis. 2003. User Acceptance o...

    2003

  13. [2024]

    In Findings of the Associ- ation for Computational Linguistics: NAACL 2024, pages 3022–3039, Mexico City, Mexico

    Retrieving Examples from Memory for Re- trieval Augmented Neural Machine Translation: A Systematic Comparison. In Findings of the Associ- ation for Computational Linguistics: NAACL 2024, pages 3022–3039, Mexico City, Mexico. Association for Computational Linguistics. john Broo...

    2024

  14. [2025]

    arXiv preprint

    New Trends for Modern Machine Transla- tion with Large Reasoning Models. arXiv preprint. ArXiv:2503.10351 [cs]. Qingyu Lu, Liang Ding, Kanjian Zhang, Jinxia Zhang, and Dacheng Tao. 2025. MQM-APE: Toward high- quality error annotation predictors with automatic post-editing in L...

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.