{"id":"17908b4b-b656-456c-b41f-7a6cbc6db11e","arxiv_id":"2508.18607","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A lexicon- and phrase-table-enhanced NMT model reports BLEU and human-eval gains over two baselines for English-to-Spanish EHR translation, but one experiment contradicts the claimed all-around improvement.","lead":"This paper proposes NOOV, a neural machine translation system that adds a bilingual lexicon and a UMLS phrase table to the decoder to translate English electronic health records into Spanish. The result could matter for patient access to medical records, but the paper's claimed universal improvement is not supported by its own experiments.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Context-aware lexicon may include test-set parallel pairs, leaking gold target words into decoding and inflating NOOV's reported gains.","rationale":"The reader's weakest assumption is that the context-aware lexicon may include the test sentence's own parallel reference, inflating NOOV's scores. I agree; this is the most load-bearing concern because the paper's central claim of universal superiority rests entirely on the validity of the evaluation, and the method description in Section 2 is ambiguous about whether test pairs are excluded. No code or data are provided, so the ambiguity cannot be resolved by inspection. The concern is concrete: if test pairs are included, the decoder receives gold target information at inference, which would artificially raise BLEU and human ratings. The proposed concrete test directly addresses this by rebuilding the lexicon from training-only pairs. I also note the independent internal contradiction in Table 2, where NOOV loses on EHR in the Medline-only setting, which further undermines the 'all experiments' claim. This does not change the reader's REJECT verdict; it reinforces it. The paper is not acceptable in its current form because the central claim is overstated and the evaluation is compromised by a potentially fatal data-leakage ambiguity.","tokens_in":7041,"tokens_out":5455,"duration_ms":48291,"concrete_test":"Re-run the three experimental settings with the context-aware lexicon constructed only from the training split, explicitly excluding development and test sentence pairs, then compare the resulting BLEU scores against Table 2; if NOOV's advantage over the baselines shrinks or reverses, the reported gains were inflated by test-set leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that NOOV beats both baselines in all experiments depends on the context-aware lexicon being built only from training (or development) data. Section 2 states: 'At decoding, when translating a source sentence sent_s_k, we collect all the sentence pairs sent_s_j:sent_t_j whose sent_s_j shares at least one word with sent_s_k.' The paper never specifies that these sentence pairs exclude the test set, nor that the test sentence's own parallel reference is withheld. If test pairs are included, then at inference the local lexicon for a test sentence is constructed from its own gold translation, so the term prob_i(V^t) in Eq. 2 would assign probability mass to the correct target words, and the decoder output w'_i = arg max((1-α)v(o'_i) + α*prob_i) would be biased toward the reference. This would inflate the reported BLEU and human adequacy/fluency scores in all three experimental settings, especially on the small EHR test set (595 sentences) where word overlap with similar test sentences is high. The omission of an explicit exclusion statement makes the evaluation ambiguous and the central claim unverifiable. Additionally, Table 2 already contradicts the 'all experiments' wording: in the Medline-only setting on EHR, NOOV scores 6.92 BLEU versus 7.26 for subword NMT and 11.06 for hybrid NMT, so NOOV does not beat the baselines there. This internal inconsistency reinforces the concern that the paper's positive conclusions are not reliably supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NOOV, an NMT system that augments an attention-based encoder-decoder with a Giza++-derived bilingual lexicon and a UMLS-derived phrase look-up table, with the goal of handling out-of-vocabulary words in English-to-Spanish translation of EHR narratives. The authors also introduce a small parallel EHR corpus of 3,020 sentence pairs and compare NOOV against subword NMT and hybrid word-character NMT under three training settings, reporting BLEU scores and a single-physician human evaluation of adequacy and fluency. The manuscript claims that NOOV beats both baselines in all experiments.","tokens_in":7329,"tokens_out":4100,"duration_ms":41795,"significance":"If the reported results are correct, the idea of integrating a context-aware lexicon and a biomedical phrase table directly into the decoder would be a useful contribution for low-resource clinical MT, and the new EHR parallel corpus could be a resource for the community. The manuscript also has a clear strength in that the proposed mechanism is concretely specified in Eqs. (1)-(2) and the appendix gives qualitative examples. However, the evaluation as presented does not establish the central claims: the results in Table 2 contradict the 'all experiments' wording, the lexicon construction leaves open a potential test-set leakage path, and no statistical significance or inter-annotator evidence is provided.","major_comments":[{"comment":"The central claim that 'the proposed NOOV model beats the two baseline systems in all experiments' is directly contradicted by Table 2. In the Medline-only training setting on the EHR test set, NOOV receives 6.92 BLEU, while subword NMT receives 7.26 and hybrid NMT receives 11.06. This is not a small gap; it is a large deficit in exactly the low-resource setting that the paper's motivation emphasizes. The claim needs to be corrected, and the analysis should explain why NOOV fails in this setting or the conclusions must be substantially weakened.","section":"Section 3, Table 2"},{"comment":"The context-aware lexicon construction is not specified in a way that rules out test-set leakage. The paper states: 'At decoding, when translating a source sentence sent_s_k, we collect all the sentence pairs sent_s_j:sent_t_j whose sent_s_j shares at least one word with sent_s_k.' It never states that the test sentence's own parallel reference is excluded from this set, nor that sentence pairs from the test set are withheld. If the current test pair is included, then Eq. (2)'s prob_i(V^t) receives probability mass from the gold target translation, and the decoder output w'_i = arg max((1-alpha) v(o'_i) + alpha prob_i) is biased toward the reference. This would inflate the reported BLEU and human scores in all settings. The authors must state explicitly that the local lexicon is built only from training (or development) sentence pairs, and describe how this is enforced.","section":"Section 2, context-aware lexicon construction"},{"comment":"The automatic evaluation reports BLEU as a single number per system and setting with no error bars, confidence intervals, bootstrap resampling, or significance tests. Several reported differences are small, for example 35.18 vs. 34.70 in the Medline column, and could easily be within noise. The human evaluation is based on one physician, with no reported inter-annotator agreement, no sample-size information, and no breakdown by the three experimental settings. Without significance testing or at least a reproducibility-oriented protocol, the claim that NOOV improves accuracy and fluency is not statistically supported.","section":"Section 3, evaluation"},{"comment":"The mechanism for reducing word repetition is not evaluated directly. The paper argues that the phrase look-up table helps when consecutive attention weights focus on the same source word, but it does not report repetition rates, examples of repaired repetitions, or an ablation without the phrase-table component. Without such an analysis, the causal claim that the phrase look-up table specifically reduces the word-repeat problem is not demonstrated.","section":"Section 2, Eq. (2) and repetition mechanism"}],"minor_comments":[{"comment":"The manuscript contains several typos and formatting artifacts: 'Unites States' in the introduction, 'gradidentclip' and 'Admda' in the implementation details, 'datastes' in the experimental setup, and the placeholder 'XXXXXXXXModels' in the Table 2 header. These should be corrected.","section":"Throughout"},{"comment":"The sentence 'For the implementation we used Pytorch 1.' is incomplete; the PyTorch version should be stated.","section":"Section 3, Implementation Details"},{"comment":"The appendix says 'Figure 1 demonstrates that ...' but the figure itself is not included in the manuscript text provided. If the figure is supposed to be part of the submission, it needs to be inserted; if not, the reference is dangling.","section":"Appendix, Figure 1"},{"comment":"The identity of the human evaluator is described as 'an English-speaking physician in the US whose first language is Spanish.' The report should clarify whether the evaluator is a professional translator or a clinician, and should report the number of sentences scored and the exact scoring instructions.","section":"Section 3, human evaluation"},{"comment":"The notation is occasionally inconsistent: the text uses both 'output_i = alpha * prob_i(V^t) + (1-alpha) v(o'_i)' and 'w'_i = arg max((1-alpha) v(o'_i) + alpha prob_i)'. These are equivalent only if the weights are understood consistently, but the sign of alpha in the two expressions appears reversed relative to the prose description. Please unify the notation.","section":"Section 2, Eq. (2)"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to be an older submission that has not been updated for the current literature or for the standards of statistical reporting now expected in MT evaluation. The combination of the Table 2 contradiction, the unresolved leakage question in the context-aware lexicon, and the absence of significance testing make the central claims unverifiable as written. Even if the leakage were ruled out, the low-resource setting result would require a major rewrite of the paper's conclusions. This is beyond a minor or major revision in the usual sense."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things about arXiv:2508.18607. First, it ships a genuinely useful resource: a 3,020-sentence English-Spanish parallel corpus of EHR discharge summaries, professionally translated and back-translated. That is a real contribution to a subfield that lacks in-domain parallel data. Second, the paper's claim that NOOV beats both baselines in all experiments is false on its face: in the Medline-only setting, Table 2 shows NOOV at 6.92 BLEU on EHR versus 7.26 for subword NMT and 11.06 for hybrid NMT. That is a loss, not a beating. The sentence and the table cannot both be right.\n\nWhat is new and good: the idea of integrating a Giza++-derived bilingual lexicon and a UMLS phrase table into the decoder, rather than post-processing, is a sensible extension of earlier lexicon-integrated NMT. The context-aware local lexicon construction is interesting. The human evaluation, while limited, does show NOOV ahead on adequacy and fluency. The error analysis examples are illustrative and honest about failures.\n\nWhere the paper falls down, in order of severity. The leakage ambiguity is the worst. Section 2 says that at decoding, for each test sentence, \"we collect all the sentence pairs sent_s_j:sent_t_j\" sharing at least one word. It never states that test pairs are excluded. If the test sentence's own parallel reference is in that collection, then the decoder is getting gold target words through the lexicon term in Eq. 2. That would inflate the reported gains and make the whole evaluation circular. The paper needs an explicit statement that the local lexicon uses only training (or dev) pairs. Without that, the central result is unverifiable.\n\nSecond, the BLEU claim is overstated as noted. Third, there are no significance tests or error bars, and the human evaluation is one rater. No code or data are provided, so the corpus is not citable until it is released.\n\nThis is not a desk-reject-level paper. The underlying idea is coherent, the corpus fills a gap, and the methodology issues are fixable. A serious referee would send this back for major revision: fix the leak question, correct the \"all experiments\" claim, release the corpus, and add basic significance testing. As it stands, I would not cite it yet.","headline":"A useful new EHR corpus and a sensible lexicon-integrated NMT idea, undercut by an ambiguous decoding setup and a claim that its own Table 2 contradicts.","tokens_in":7838,"tokens_out":3411,"would_cite":false,"duration_ms":30611,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A decoder guided by bilingual lexicons and UMLS phrases outperforms two NMT baselines on clinical English-to-Spanish translation.","keywords":["clinical machine translation","electronic health records","neural machine translation","unknown words","bilingual lexicon","UMLS","English-Spanish","word repetition"],"falsifier":"Rebuild the context-aware lexicon for the EHR test set twice, once with training pairs only and once with the test sentence's gold parallel pair included, and re-run the Medline-pretrain/EHR-finetune condition; if the training-only BLEU falls to the hybrid baseline's 31.06 or below, the reported NOOV gain is an artifact of test leakage.","tokens_in":6837,"feed_emoji":"🩺","tokens_out":5380,"duration_ms":50743,"temperature":0.7,"pith_summary":"The paper tries to establish that a neural machine translation system can translate English electronic health record narratives into Spanish accurately even when the in-domain parallel corpus is small, as long as the decoder is supplemented with external medical vocabulary. Its proposed model, NOOV, blends a bilingual lexicon learned from parallel text with a phrase look-up table drawn from biomedical knowledge resources, attacking both the unknown-word problem and the tendency of attention-based decoders to repeat words. On a newly built English-Spanish EHR corpus, the paper reports that NOOV outperforms a subword BPE system and a hybrid word-character system on BLEU and on human adequacy and fluency ratings in every training configuration tested. The result matters because translated records could improve communication between Spanish-speaking patients and their U.S. clinicians.","feed_headline":"Lexicons and phrase tables lift clinical translation scores","feed_subtitle":"NOOV beats two NMT baselines on BLEU and human ratings for English-to-Spanish EHR notes.","key_machinery":"The mechanism is the lexicon-and-phrase-assisted decoder: output at each step is $w'_i = \\arg\\max((1-\\alpha) v(o'_i) + \\alpha \\mathrm{prob}_i)$, where $v(o'_i)$ is the LSTM's softmax and $\\mathrm{prob}_i$ is an attention-weighted sum of lexical translation probabilities from a Giza++-derived lexicon built only from sentence pairs whose source shares a word with the current input. If $\\arg\\max(\\mathrm{prob}_i) = \\arg\\max(\\mathrm{prob}_{i+1})$, indicating an alignment-induced repetition, the model looks up the source word and current target word in the UMLS phrase table and selects the phrase with the longest matching source span. This carries the argument because it is exactly the insertion point for external vocabulary and repetition control.","core_discovery":"NOOV adds an extra probability distribution to the standard attention-based decoder. At each target step, it computes a lexicon distribution by taking attention weights over source words and weighting each source word's Giza++ lexical translation probabilities, then interpolates that distribution with the decoder's softmax output using a tuned weight alpha. When two consecutive decoder outputs would both be aligned to the same source word, the model consults a 465,256-entry phrase table extracted from UMLS and prefers the candidate whose source phrase is the longest match in the current sentence. The authors claim that this design yields better translation of rare medical terms and fewer repeated words, and report that it beats both baselines across Medline, Medline+EHR, and Medline-pretrain/EHR-finetune settings, with EHR BLEU of 34.71 versus 30.03 and 31.06, human adequacy 3.26 versus 2.78 and 2.97, and fluency 2.84 versus 2.64 and 2.56.","pith_inferences":["If the test sentence's parallel reference were inadvertently included in the local lexicon construction, the reported BLEU and human scores would overstate real performance; an explicit test-side exclusion is needed to confirm the magnitude.","The context-aware lexicon only helps when the test sentence shares vocabulary with training pairs; on sentences with completely novel clinical terms the model would fall back to the decoder alone, so segmenting test results by source-vocabulary overlap would show where the gain lives.","The same interpolation trick could be adapted to other biomedical term mappings and evaluated on whether more lexicon coverage yields monotonically better clinical translation."],"forward_implications":["The same decoder-side lexicon injection could be retargeted to other low-resource medical language pairs without retraining the whole system from scratch.","Because the lexicon probabilities are built from attention weights, any improvement in attention quality should directly improve the lexicon channel.","The phrase-table correction shortens the effective sequence the decoder must produce, so longer sentences, the model's known weak point, may benefit most when a matching UMLS phrase covers a long span.","The newly built parallel EHR corpus provides a gold standard for future clinical machine translation evaluation."],"supporting_citations":[{"why":"Supplies the Giza++ alignment tool and lexical translation probability model used to build the learned bilingual lexicon.","marker":"[22]"},{"why":"Defines UMLS and provides the 2017AB terminology from which the 465,256-pair English-Spanish phrase look-up table is extracted.","marker":"[4]"},{"why":"Contributes the ESPAC MedlinePlus parallel biomedical corpus used for training and the earlier medical-domain translation work this paper extends.","marker":"[18]"},{"why":"Defines the subword (BPE) neural machine translation baseline that NOOV is compared against.","marker":"[24]"},{"why":"Defines the hybrid word-character neural machine translation baseline that NOOV is compared against.","marker":"[19]"}],"fun_headline_variants":["NOOV NMT adds lexicon and phrase tables for clinical translation","New NMT model beats baselines on English-Spanish EHR notes","Lexicon and phrase tables boost clinical NMT translation","NOOV NMT improves EHR translation with lexicon and phrase tables"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire empirical advantage rests on the context-aware lexicon being built only from training sentence pairs: if a test sentence's own gold translation is used to assemble its lexicon, the decoder receives information that would not exist in deployment.","fun_headline_variants_meta":{"raw":{"variants":["NOOV NMT adds lexicon and phrase tables for clinical translation","New NMT model beats baselines on English-Spanish EHR notes","Lexicon and phrase tables boost clinical NMT translation","NOOV NMT improves EHR translation with lexicon and phrase tables"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000474,"raw_usage":{"total_tokens":2320,"prompt_tokens":876,"completion_tokens":1444,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":1374}},"tokens_in":492,"tokens_out":1444,"duration_ms":8668,"temperature":1.0,"reasoning_tokens":1374,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:55:07.135703+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rebuild the context-aware lexicon for the EHR test set twice, once with training pairs only and once with the test sentence's gold parallel pair included, and re-run the Medline-pretrain/EHR-finetune condition; if the training-only BLEU falls to the hybrid baseline's 31.06 or below, the reported NOOV gain is an artifact of test leakage.","supporting_citations":[{"cited_title":"A systematic comparison of various statistical alignment models","cited_arxiv_id":null,"evidence_quote":"Supplies the Giza++ alignment tool and lexical translation probability model used to build the learned bilingual lexicon."},{"cited_title":"Translating electronic health record notes from english to spanish: A preliminary study","cited_arxiv_id":null,"evidence_quote":"Contributes the ESPAC MedlinePlus parallel biomedical corpus used for training and the earlier medical-domain translation work this paper extends."}],"review_version":2}