Pith. sign in

REVIEW 4 major objections 5 minor 30 references

A New NMT Model for Translating Clinical Texts from English to Spanish

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A decoder guided by bilingual lexicons and UMLS phrases outperforms two NMT baselines on clinical English-to-Spanish translation.

desk verdict A useful new EHR corpus and a sensible lexicon-integrated NMT idea, undercut by an ambiguous decoding setup and a claim that its own Table 2 contradicts. read the letter →

arxiv 2508.18607 v1 pith:CPLZMNRU submitted 2025-08-26 cs.CL

Rumeng Li , Xun Wang , Hong Yu This is my paper · ORCID
classification cs.CL
keywords clinicalmachinetranslationelectronichealthrecordsneuralunknownwordsbilinguallexiconUMLSEnglish-Spanishwordrepetition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a neural machine translation system can translate English electronic health record narratives into Spanish accurately even when the in-domain parallel corpus is small, as long as the decoder is supplemented with external medical vocabulary. Its proposed model, NOOV, blends a bilingual lexicon learned from parallel text with a phrase look-up table drawn from biomedical knowledge resources, attacking both the unknown-word problem and the tendency of attention-based decoders to repeat words. On a newly built English-Spanish EHR corpus, the paper reports that NOOV outperforms a subword BPE system and a hybrid word-character system on BLEU and on human adequacy and fluency ratings in every training configuration tested. The result matters because translated records could improve communication between Spanish-speaking patients and their U.S. clinicians.

What carries the argument

The mechanism is the lexicon-and-phrase-assisted decoder: output at each step is $w'_i = \arg\max((1-\alpha) v(o'_i) + \alpha \mathrm{prob}_i)$, where $v(o'_i)$ is the LSTM's softmax and $\mathrm{prob}_i$ is an attention-weighted sum of lexical translation probabilities from a Giza++-derived lexicon built only from sentence pairs whose source shares a word with the current input. If $\arg\max(\mathrm{prob}_i) = \arg\max(\mathrm{prob}_{i+1})$, indicating an alignment-induced repetition, the model looks up the source word and current target word in the UMLS phrase table and selects the phrase with the longest matching source span. This carries the argument because it is exactly the insertion point for external vocabulary and repetition control.

What would settle it

Rebuild the context-aware lexicon for the EHR test set twice, once with training pairs only and once with the test sentence's gold parallel pair included, and re-run the Medline-pretrain/EHR-finetune condition; if the training-only BLEU falls to the hybrid baseline's 31.06 or below, the reported NOOV gain is an artifact of test leakage.

Watch

Extended reading notes

Core claim

NOOV adds an extra probability distribution to the standard attention-based decoder. At each target step, it computes a lexicon distribution by taking attention weights over source words and weighting each source word's Giza++ lexical translation probabilities, then interpolates that distribution with the decoder's softmax output using a tuned weight alpha. When two consecutive decoder outputs would both be aligned to the same source word, the model consults a 465,256-entry phrase table extracted from UMLS and prefers the candidate whose source phrase is the longest match in the current sentence. The authors claim that this design yields better translation of rare medical terms and fewer repeated words, and report that it beats both baselines across Medline, Medline+EHR, and Medline-pretrain/EHR-finetune settings, with EHR BLEU of 34.71 versus 30.03 and 31.06, human adequacy 3.26 versus 2.78 and 2.97, and fluency 2.84 versus 2.64 and 2.56.

Load-bearing premise

The entire empirical advantage rests on the context-aware lexicon being built only from training sentence pairs: if a test sentence's own gold translation is used to assemble its lexicon, the decoder receives information that would not exist in deployment.

Editorial extensions

If this is right

  • The same decoder-side lexicon injection could be retargeted to other low-resource medical language pairs without retraining the whole system from scratch.
  • Because the lexicon probabilities are built from attention weights, any improvement in attention quality should directly improve the lexicon channel.
  • The phrase-table correction shortens the effective sequence the decoder must produce, so longer sentences, the model's known weak point, may benefit most when a matching UMLS phrase covers a long span.
  • The newly built parallel EHR corpus provides a gold standard for future clinical machine translation evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the test sentence's parallel reference were inadvertently included in the local lexicon construction, the reported BLEU and human scores would overstate real performance; an explicit test-side exclusion is needed to confirm the magnitude.
  • The context-aware lexicon only helps when the test sentence shares vocabulary with training pairs; on sentences with completely novel clinical terms the model would fall back to the decoder alone, so segmenting test results by source-vocabulary overlap would show where the gain lives.
  • The same interpolation trick could be adapted to other biomedical term mappings and evaluated on whether more lexicon coverage yields monotonically better clinical translation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes NOOV, an NMT system that augments an attention-based encoder-decoder with a Giza++-derived bilingual lexicon and a UMLS-derived phrase look-up table, with the goal of handling out-of-vocabulary words in English-to-Spanish translation of EHR narratives. The authors also introduce a small parallel EHR corpus of 3,020 sentence pairs and compare NOOV against subword NMT and hybrid word-character NMT under three training settings, reporting BLEU scores and a single-physician human evaluation of adequacy and fluency. The manuscript claims that NOOV beats both baselines in all experiments.

Significance. If the reported results are correct, the idea of integrating a context-aware lexicon and a biomedical phrase table directly into the decoder would be a useful contribution for low-resource clinical MT, and the new EHR parallel corpus could be a resource for the community. The manuscript also has a clear strength in that the proposed mechanism is concretely specified in Eqs. (1)-(2) and the appendix gives qualitative examples. However, the evaluation as presented does not establish the central claims: the results in Table 2 contradict the 'all experiments' wording, the lexicon construction leaves open a potential test-set leakage path, and no statistical significance or inter-annotator evidence is provided.

major comments (4)
  1. [Section 3, Table 2] The central claim that 'the proposed NOOV model beats the two baseline systems in all experiments' is directly contradicted by Table 2. In the Medline-only training setting on the EHR test set, NOOV receives 6.92 BLEU, while subword NMT receives 7.26 and hybrid NMT receives 11.06. This is not a small gap; it is a large deficit in exactly the low-resource setting that the paper's motivation emphasizes. The claim needs to be corrected, and the analysis should explain why NOOV fails in this setting or the conclusions must be substantially weakened.
  2. [Section 2, context-aware lexicon construction] The context-aware lexicon construction is not specified in a way that rules out test-set leakage. The paper states: 'At decoding, when translating a source sentence sent_s_k, we collect all the sentence pairs sent_s_j:sent_t_j whose sent_s_j shares at least one word with sent_s_k.' It never states that the test sentence's own parallel reference is excluded from this set, nor that sentence pairs from the test set are withheld. If the current test pair is included, then Eq. (2)'s prob_i(V^t) receives probability mass from the gold target translation, and the decoder output w'_i = arg max((1-alpha) v(o'_i) + alpha prob_i) is biased toward the reference. This would inflate the reported BLEU and human scores in all settings. The authors must state explicitly that the local lexicon is built only from training (or development) sentence pairs, and describe how this is enforced.
  3. [Section 3, evaluation] The automatic evaluation reports BLEU as a single number per system and setting with no error bars, confidence intervals, bootstrap resampling, or significance tests. Several reported differences are small, for example 35.18 vs. 34.70 in the Medline column, and could easily be within noise. The human evaluation is based on one physician, with no reported inter-annotator agreement, no sample-size information, and no breakdown by the three experimental settings. Without significance testing or at least a reproducibility-oriented protocol, the claim that NOOV improves accuracy and fluency is not statistically supported.
  4. [Section 2, Eq. (2) and repetition mechanism] The mechanism for reducing word repetition is not evaluated directly. The paper argues that the phrase look-up table helps when consecutive attention weights focus on the same source word, but it does not report repetition rates, examples of repaired repetitions, or an ablation without the phrase-table component. Without such an analysis, the causal claim that the phrase look-up table specifically reduces the word-repeat problem is not demonstrated.
minor comments (5)
  1. [Throughout] The manuscript contains several typos and formatting artifacts: 'Unites States' in the introduction, 'gradidentclip' and 'Admda' in the implementation details, 'datastes' in the experimental setup, and the placeholder 'XXXXXXXXModels' in the Table 2 header. These should be corrected.
  2. [Section 3, Implementation Details] The sentence 'For the implementation we used Pytorch 1.' is incomplete; the PyTorch version should be stated.
  3. [Appendix, Figure 1] The appendix says 'Figure 1 demonstrates that ...' but the figure itself is not included in the manuscript text provided. If the figure is supposed to be part of the submission, it needs to be inserted; if not, the reference is dangling.
  4. [Section 3, human evaluation] The identity of the human evaluator is described as 'an English-speaking physician in the US whose first language is Spanish.' The report should clarify whether the evaluator is a professional translator or a clinician, and should report the number of sentences scored and the exact scoring instructions.
  5. [Section 2, Eq. (2)] The notation is occasionally inconsistent: the text uses both 'output_i = alpha * prob_i(V^t) + (1-alpha) v(o'_i)' and 'w'_i = arg max((1-alpha) v(o'_i) + alpha prob_i)'. These are equivalent only if the weights are understood consistently, but the sign of alpha in the two expressions appears reversed relative to the prose description. Please unify the notation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity in NOOV derivation; the lexicon and UMLS phrase table are external resources, with evaluation-validity caveats noted separately.

full rationale

The central derivation is not circular. NOOV's bilingual lexicon is produced by Giza++ from parallel medical corpora and the phrase look-up table is extracted from UMLS; both are external resources, not functions of the EHR test references. Equation (2) mixes the LSTM decoder distribution v(o'_i) with a lexicon-derived distribution prob_i, a standard externally-grounded interpolation rather than a definitional identity with the gold translation. The paper invokes no uniqueness theorem and no load-bearing self-citation: the cited Giza++, BPE, and hybrid word-character methods are independent prior work. The design therefore has no step in which the predicted output is equivalent, by construction, to its own input. Two non-circularity caveats should be noted: (i) Section 2's local lexicon construction says 'we collect all the sentence pairs sent_s_j:sent_t_j' without explicitly excluding test pairs, so if the test pair is included the reported gains would reflect leakage rather than prediction; this is an evaluation-validity ambiguity, not a derivation circularity. (ii) Table 2 shows NOOV (6.92 BLEU) below both baselines (7.26 and 11.06) in the Medline-trained/EHR-test setting, contradicting the 'all experiments' claim; this is an internal-consistency and correctness concern, not circularity.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the reliability of external resources (Giza++ and UMLS), a small single-hospital EHR corpus, a hand-tuned interpolation weight (alpha = 0.2), and unstated test-set hygiene in local lexicon construction. No new entities are invented.

free parameters (1)
  • alpha (lexicon-decoder interpolation weight) = 0.2
    Tuned from 0 to 1 in increments of 0.2 on the development set (Section 3) and fixed at 0.2; directly controls the balance between the lexicon probability and the decoder output.
assumptions (5)
  • domain assumption Giza++ lexical translation probabilities are reliable for clinical sub-language terms.
    The lexicon is built with Giza++ and used as decoder supervision for rare/unknown words (Section 2); its validity for EHR jargon is not demonstrated.
  • domain assumption UMLS concept identifiers map English and Spanish terms as valid translation pairs.
    The phrase look-up table is extracted from UMLS 2017AB by matching CUIs (Section 2); cross-language CUI equivalence is assumed to imply correct translations.
  • domain assumption Attention weights approximate source-target word alignments well enough for the lexicon weighting.
    The method uses attention weights to combine lexicon probabilities (Section 2); this is a standard NMT assumption but not validated for the noisy, de-identified EHR text.
  • domain assumption The 3,020-sentence EHR corpus is representative of the target clinical translation task.
    Data come from 57 type-2-diabetes discharge summaries from one hospital, translated by a single professional translator (Section 3); no sampling or inter-annotator justification is given.
  • domain assumption Test sentence pairs are not used when constructing the local context-aware lexicon.
    The local lexicon construction (Section 2) does not state that test pairs are excluded; if included, evaluation would be circular.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A New NMT Model for Translating Clinical Texts from English to Spanish." pith.science (2026). https://pith.science/paper/CPLZMNRU

@misc{pith2026250818607,
  author       = {Pith},
  title        = {Pith review of: A New NMT Model for Translating Clinical Texts from English to Spanish},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CPLZMNRU}},
  note         = {Machine review of arXiv:2508.18607}
}
read the original abstract

Translating electronic health record (EHR) narratives from English to Spanish is a clinically important yet challenging task due to the lack of a parallel-aligned corpus and the abundant unknown words contained. To address such challenges, we propose \textbf{NOOV} (for No OOV), a new neural machine translation (NMT) system that requires little in-domain parallel-aligned corpus for training. NOOV integrates a bilingual lexicon automatically learned from parallel-aligned corpora and a phrase look-up table extracted from a large biomedical knowledge resource, to alleviate both the unknown word problem and the word-repeat challenge in NMT, enhancing better phrase generation of NMT systems. Evaluation shows that NOOV is able to generate better translation of EHR with improvement in both accuracy and fluency.

Figures

Figures reproduced from arXiv: 2508.18607 by the authors.

Figure 1
Figure 1. Effects of sentence length on translation results [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 17 canonical work pages

  1. [1]

    Incorporating discrete translation lexicons into neural machine translation

    Philip Arthur, Graham Neubig, and Satoshi Nakamura. Incorporating discrete translation lexicons into neural machine translation. arXiv preprint arXiv:1606.02006, 2016

    arXiv 2016

  2. [2]

    Neural machine translation by jointly learning to align and translate

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014

    arXiv 2014

  3. [3]

    Synthetic and natural noise both break neural machine translation

    Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine translation. arXiv preprint arXiv:1711.02173, 2017

    arXiv 2017

  4. [4]

    The unified medical language system (umls): integrating biomedical terminology

    Olivier Bodenreider. The unified medical language system (umls): integrating biomedical terminology. Nucleic acids research, 32(suppl_1):D267–D270, 2004

    2004

  5. [5]

    Findings of the 2014 workshop on statistical machine translation

    Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, et al. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the ninth workshop on statistical machine translation, pages 12–58, 2014

    work page 2014

  6. [6]

    Learning phrase representations using rnn encoder- decoder for statistical machine translation

    Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder- decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014

    arXiv 2014

  7. [7]

    Improving statistical machine translation in the medical domain using the unified medical language system

    Matthias Eck, Stephan V ogel, and Alex Waibel. Improving statistical machine translation in the medical domain using the unified medical language system. In Proceedings of the 20th inter- national conference on Computational Linguistics , page 792. Association for Computational Linguistics, 2004

    work page 2004

  8. [8]

    Implicit Distortion and Fertility Models for Attention-based Encoder-Decoder NMT Model

    Shi Feng, Shujie Liu, Mu Li, and Ming Zhou. Implicit distortion and fertility models for attention-based encoder-decoder nmt model. arXiv preprint arXiv:1601.03317, 2016

    work page Pith review arXiv 2016

Show all 30 references
  1. [9]

    Convolutional sequence to sequence learning

    Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. arXiv preprint arXiv:1705.03122, 2017

    2017 arXiv

  2. [10]

    Domain adaptation and attention-based unknown word replacement in chinese-to-japanese neural machine transla- tion

    Kazuma Hashimoto, Akiko Eriguchi, and Yoshimasa Tsuruoka. Domain adaptation and attention-based unknown word replacement in chinese-to-japanese neural machine transla- tion. In Proceedings of the 3rd Workshop on Asian Translation (WAT2016) , pages 75–83, 2016

    2016

  3. [11]

    Montreal neural machine translation systems for wmt’15

    Sébastien Jean, Orhan Firat, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. Montreal neural machine translation systems for wmt’15. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 134–140, 2015

    2015

  4. [12]

    Recurrent continuous translation models

    Nal Kalchbrenner and Phil Blunsom. Recurrent continuous translation models. InProceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , pages 1700–1709, 2013

    2013

  5. [13]

    Fully character-level neural machine translation without explicit segmentation

    Jason Lee, Kyunghyun Cho, and Thomas Hofmann. Fully character-level neural machine translation without explicit segmentation. arXiv preprint arXiv:1610.03017, 2016

    2016 arXiv

  6. [14]

    Towards zero unknown word in neural machine translation

    Xiaoqing Li, Jiajun Zhang, and Chengqing Zong. Towards zero unknown word in neural machine translation. In IJCAI, pages 2852–2858, 2016

    2016

  7. [15]

    Decoding-history-based adaptive control of attention for neural machine translation

    Junyang Lin, Shuming Ma, Qi Su, and Xu Sun. Decoding-history-based adaptive control of attention for neural machine translation. arXiv preprint arXiv:1802.01812, 2018

    2018 arXiv

  8. [16]

    Character-based neural machine translation

    Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W Black. Character-based neural machine translation. arXiv preprint arXiv:1511.04586, 2015. 5

    2015 arXiv

  9. [17]

    Babelmesh: development of a cross-language tool for medline/pubmed

    Fang Liu, Paul Fontelo, and Michael Ackerman. Babelmesh: development of a cross-language tool for medline/pubmed. In AMIA Annual Symposium Proceedings, volume 2006, page 1012. American Medical Informatics Association, 2006

    2006

  10. [18]

    Translating electronic health record notes from english to spanish: A preliminary study

    Weisong Liu and Shu Cai. Translating electronic health record notes from english to spanish: A preliminary study. Proceedings of BioNLP 15, pages 134–140, 2015

    2015

  11. [19]

    Achieving open vocabulary neural machine translation with hybrid word-character models

    Minh-Thang Luong and Christopher D Manning. Achieving open vocabulary neural machine translation with hybrid word-character models. arXiv preprint arXiv:1604.00788, 2016

    2016 arXiv

  12. [20]

    Ad- dressing the rare word problem in neural machine translation

    Minh-Thang Luong, Ilya Sutskever, Quoc V Le, Oriol Vinyals, and Wojciech Zaremba. Ad- dressing the rare word problem in neural machine translation. arXiv preprint arXiv:1410.8206, 2014

    2014 arXiv

  13. [21]

    Translating the foundational model of anatomy into french using knowledge- based and lexical methods

    Tayeb Merabti, Lina F Soualmia, Julien Grosjean, Olivier Palombi, Jean-Michel Müller, and Stéfan J Darmoni. Translating the foundational model of anatomy into french using knowledge- based and lexical methods. BMC medical informatics and decision making , 11(1):65, 2011

    2011

  14. [22]

    A systematic comparison of various statistical alignment models

    Franz Josef Och and Hermann Ney. A systematic comparison of various statistical alignment models. Computational linguistics, 29(1):19–51, 2003

    2003

  15. [23]

    Language use in the united states: 2011

    Camille Ryan. Language use in the united states: 2011. American community survey reports, 22:1–16, 2013

    2011

  16. [24]

    Neural machine translation of rare words with subword units

    Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015

    2015 arXiv

  17. [25]

    Sequence to sequence learning with neural networks

    Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems , pages 3104–3112, 2014

    2014

  18. [26]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- tion Processing Systems, pages 6000–6010, 2017

    2017

  19. [27]

    Neural-based machine translation for medical text domain

    Krzysztof Wołk and Krzysztof Marasek. Neural-based machine translation for medical text domain. based on european medicines agency leaflet texts. arXiv preprint arXiv:1509.08644, 2015

    2015 arXiv

  20. [28]

    Findings of the wmt 2017 biomedical translation shared task

    Antonio Jimeno Yepes, Aurélie Névéol, Mariana Neves, Karin Verspoor, Ondrej Bojar, Arthur Boyer, Cristian Grozea, Barry Haddow, Madeleine Kittner, Yvonne Lichtblau, et al. Findings of the wmt 2017 biomedical translation shared task. In Proceedings of the Second Conference on M...

    2017

  21. [29]

    Can multilingual machine translation help make medical record content more comprehensible to patients?Studies in health technology and informatics , 160(Pt 1):73–77, 2010

    Qing Zeng-Treitler, Hyeoneui Kim, Graciela Rosemblat, and Alla Keselman. Can multilingual machine translation help make medical record content more comprehensible to patients?Studies in health technology and informatics , 160(Pt 1):73–77, 2010

    2010

  22. [30]

    polyuria

    Shenjian Zhao and Zhihua Zhang. Deep character-level neural machine translation by learning morphology. 2016. 6 A Supplemental material In this Appendix, we report the error analysis for a better understanding of the proposed system’s performance. Our NOOV model addresses the ...

    2016

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.