{"id":"e4450d81-fd36-42c6-9082-6e18db0a29d7","arxiv_id":"2506.18318","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-task mT5 model that predicts named entities, translates them, and translates the full sentence outperforms mT5 and mBART baselines in BLEU on three of four SemEval 2025 Task 2 language pairs.","lead":"This paper fine-tunes a multilingual translation model to do three things at once: identify named entities, translate them, and translate the full sentence. It reports BLEU gains over simple baselines on three of four language pairs in the SemEval 2025 entity-aware translation task.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run BLEU point estimates with no error bars or significance tests leave the claimed gains underdetermined; multi-seed reruns are needed.","rationale":"The reader's CONDITIONAL verdict is appropriate. The reader focused on whether the automatically produced entity labels are accurate enough to train on; that is a real internal-validity threat. I see an even more direct threat to the stated numerical claim: the absence of any variance or significance information around the BLEU scores. The test sets are small, the margins are mixed, and no code or exact split is provided, so the reported ordering of systems could be a single-seed artifact. The proposed multi-seed rerun with paired bootstrap would settle this directly; if the gains replicate, the label-quality question remains but the headline survives. If they do not, the headline should be downgraded. Since neither we nor the reader can currently resolve this, the verdict remains CONDITIONAL rather than ACCEPT or REJECT. I credit the paper for a clear task setup and a plausible method, but the evidence bar for the central claim is lower than the strength of the claim.","tokens_in":5031,"tokens_out":8042,"duration_ms":85863,"concrete_test":"Run the identical fine-tuning (mT5-large, 50 epochs, lr=5e-5, batch 16) for at least 5 seeds on both the baseline and the multi-task model for all four language pairs, keeping the same train/dev/test split. Compute per-sentence BLEU (sacreBLEU, same tokenization and reference handling) for each run; report mean ± 95% CI and a paired bootstrap (e.g., 10,000 resamples) of the baseline-vs-multitask difference per language. If the en-de and en-es 95% CIs overlap zero or the paired bootstrap is not significant at p<0.05, the claimed 'highest scores in most cases' is not supported by the current evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is the BLEU advantage of multi-task learning over the mT5 baseline (Table 2). The paper reports one number per system per language pair, with no seeds, no standard deviations, and no significance tests, on test sets of only 748–1107 sentences (Table 1). The observed deltas are +1.68 (en-de), +3.06 (en-es), +5.34 (en-it), and -1.10 (en-fr). BLEU has substantial variance at this test-set size, and a single training run can move by more than a point from seed to seed. The French result shows the method is not uniformly better, so 'highest in most cases' could flip with a small perturbation. The unmeasured Qwen/AWESOME label quality is a mechanism-level concern, but even perfect labels would not establish the headline if the BLEU differences are within noise. This is the load-bearing issue because the paper's central claim is the numerical superiority itself.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-task learning framework for entity-aware machine translation (EAMT). The authors fine-tune mT5-large on the SemEval 2025 Task 2 data, training the model to output three segments separated by <SEP>: a list of named entities, their target-language translations, and the full translation with XML boundary tags around entity mentions. Source–target entity alignments are constructed by combining Qwen2.5-VL-72B outputs that are kept only when the entity appears in the source sequence with AWESOME token-level alignments. On the four language pairs, the method achieves BLEU scores of 47.69 (en-de), 48.51 (en-fr), 48.83 (en-it), and 54.18 (en-es), compared with an mT5-large baseline of 46.01, 49.61, 43.49, and 51.12, and an mBART baseline of 40.79, 42.01, 35.40, and 45.11. The central claim is that multi-task learning achieves the highest scores in most cases, with the en-fr pair being the exception.","tokens_in":5206,"tokens_out":4574,"duration_ms":46656,"significance":"If the reported gains are reproducible, this is a practical and useful result: a simple modification of the fine-tuning output format yields BLEU improvements of 1.7–5.3 points over a strong mT5-large baseline on three of four language pairs, with no change at inference time. The proposed recipe is easy to describe and could be adopted by other EAMT systems. The paper is less convincing as a scientific study, however, because the headline numbers are single-run point estimates without uncertainty quantification, and the entity-label construction pipeline is neither fully specified nor validated. Nevertheless, the core idea is sound and the experimental setting is clearly tied to a recognized shared task.","major_comments":[{"comment":"The central empirical claim—that multi-task learning achieves the highest BLEU in most cases—rests entirely on single-run point estimates. Table 2 reports one BLEU score per system per language pair, with no standard deviations, no number of random seeds, and no significance or bootstrap confidence intervals. The test sets contain only 748–1,107 sentences (Table 1), and BLEU at this scale can easily vary by more than the smallest reported gap (1.68 BLEU, en-de) across training runs. The en-fr result (48.51 vs. 49.61) shows the method is not uniformly better, so the conclusion that the method is effective could flip under seed variation. Please provide multiple seeds or otherwise report confidence intervals and a significance test for the differences.","section":"§4.2, Table 2"},{"comment":"The entity-alignment pipeline that produces the training labels is under-specified and its quality is never measured. The text states only that Qwen2.5-VL-72B outputs are retained when the entity appears in the source sequence and that these are combined with AWESOME token-level alignments. The prompt used, the combination rule (union, intersection, or confidence threshold), the coverage of the alignments, and the agreement between the two methods are not reported. If the resulting labels are noisy, the multi-task model is trained on corrupted entity annotations, and the observed BLEU gains cannot be attributed to entity-aware learning; the en-fr degradation could be a consequence of such noise. Please report label statistics and a sample-based accuracy estimate, e.g., against the reference mentions or a small human-annotated set.","section":"§3, Data preparation"},{"comment":"The mBART baseline is not described. The paper does not state which mBART variant is used, whether it was fine-tuned on the same SemEval training data, or which hyperparameters were used. Since Table 2 includes mBART as one of the three compared approaches, this row is currently uninterpretable. This is not the main baseline, but the comparison should be either properly specified or removed.","section":"§4.1, Table 2"},{"comment":"The paper says the data is partitioned into training, validation, and test sets, but does not explain how the validation set is used. There is no mention of early stopping, checkpoint selection, or any model selection criterion, despite training for 50 epochs. Please clarify whether the reported scores come from the final epoch or from a selected checkpoint, and how the validation set influences the results.","section":"§4.1, Experimental Settings"}],"minor_comments":[{"comment":"There are typographical issues in the tables: \"T rain Size\", \"T able 1\", \"T able 2\" and \"mBAR TmT5 large\" should be corrected to \"Train Size\", \"Table 1\", \"Table 2\", and \"mBART / mT5-large\".","section":"§4.1, Table 1"},{"comment":"The fine-tuning loss is not described. It is presumably standard cross-entropy over the whole target sequence, but the paper does not state whether the NER segment, the entity-translation segment, and the final translation segment are weighted equally or trained jointly through a single unified objective. Please state the loss formulation explicitly.","section":"§3, Multitask learning Design"},{"comment":"The example output lists \"Europe | Allied Forces\" as the NER result, while the source sentence is \"Who was the overall Commander of Allied Forces in Europe?\" The ordering of the two entities appears arbitrary; please clarify whether entity order is canonicalized or preserved from the source, as this affects the reproducibility of the data construction.","section":"§3, Data preparation"},{"comment":"The paper does not release code or the constructed aligned dataset. Given that the method is simple and the paper is short, releasing the preprocessing scripts and the generated entity-alignment labels would materially improve reproducibility and allow others to verify label quality.","section":"§5, Conclusion"}],"recommendation":"major_revision","confidential_remarks":"This is a compact system-description paper. The main risk is statistical: the central claim is supported by single-run BLEU differences of about 1.7–5.3 points, and the authors do not report variance or significance. If the authors can supply multi-seed results, confidence intervals or significance tests, and a basic validation of the alignment labels, the contribution would be acceptable for an empirical MT venue. If the journal expects deeper analysis, the novelty may be on the lighter side, but the negative en-fr result and the unvalidated data construction should be addressed in any case."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent shared-task system description, not a methodological breakthrough. The new pieces are the Qwen+AWESOME alignment pipeline and the three-part <SEP> output format that forces NER and entity translation before the full translation. The headline numbers (47.69 en-de, 48.83 en-it, 54.18 en-es) beat the mT5 baseline by 1.7-5.3 BLEU, but they are single runs with no seeds, variance, or significance tests. On test sets of 748-1107 sentences, one or two BLEU points can be noise, so the French result (48.51 vs 49.61) is a reminder that 'highest in most cases' might flip under reruns.\n\nThe paper deserves credit for being honest and well-scoped. It correctly places itself against Xie et al. and Rikters & Miwa, and admits the French underperformance. The data-construction step is genuinely creative: Qwen2.5-VL is used to propose source-target entity alignments, filtered by requiring the entity to appear in the source, then combined with AWESOME token alignments. That is a concrete engineering choice worth borrowing. The three-part output format with entity translation output separately before the full translated text is a clean prompting scheme for mT5, and stripping the tags after generation is sensible.\n\nThe main soft spot is the evaluation. Single-run BLEU on the official test set is common for shared-task descriptions, but it leaves the central numerical claim underdetermined. The absence of an entity-level metric hurts more here: the task data includes the entity mention in the target, so computing entity translation accuracy would directly test the claimed benefit. The alignment step is also unmeasured; we never learn how many source entities were correctly identified, so the multi-task training signal could be noisy. These are not fatal, but they limit what we can conclude from the numbers. Minor: no code or data release.\n\nThis paper is for readers looking for engineering ideas for entity-aware MT, not for someone seeking strong evidence about which method wins. It deserves a serious referee; the review should ask for multi-seed reruns and an entity-level evaluation. I'd accept it into a workshop or system-description track with light revisions.","headline":"A useful shared-task system description with a creative alignment pipeline and a sensible multi-task output format, but single-run BLEU without error bars or an entity-level metric leaves the claimed gains underdetermined; worth a serious referee for its engineering details.","tokens_in":5771,"tokens_out":4081,"would_cite":false,"duration_ms":37105,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a single fine-tuned text-to-text model can perform entity-aware translation by first listing entities, then translating them, then producing the full tagged sentence—a format that raises BLEU by up to 5.3 points on…","keywords":["entity-aware machine translation","multi-task learning","named entity recognition","text-to-text model","fine-tuning","BLEU","XML tagging"],"falsifier":"Fix every training detail except the target template and replace the three-part output with plain translations; if BLEU on English–German, English–Italian, and English–Spanish stays within noise, the entity-aware template is not the cause of the reported gains. Separately, hand-annotate a random subset of the training sentences for entity mentions and compare against the labels the method used; low agreement would mean the model was trained on corrupted entity supervision.","tokens_in":4820,"feed_emoji":"🌐","tokens_out":11323,"duration_ms":102754,"temperature":0.7,"pith_summary":"This paper claims that the hard part of entity-aware translation—finding the entities, deciding how to translate them, and placing them correctly in the sentence—can be handled by a single fine-tuned text-to-text model rather than by chaining a separate named-entity recognizer into a translator. The proposed target sequence has three parts separated by a special token: a list of named entities, their target-language translations, and the full translation with entity spans wrapped in XML tags. Fine-tuning a 580-million-parameter multilingual text-to-text Transformer on this format beats its translation-only baseline by 1.7 BLEU on English–German, 5.3 on English–Italian, and 3.1 on English–Spanish, while English–French drops by 1.1 BLEU. The conclusion a sympathetic reader draws is that forcing the model to commit to entities and their translations before generating the sentence is a viable way to inject entity knowledge into translation.","feed_headline":"Entity-first multitask training lifts BLEU by up to 5.3 points","feed_subtitle":"Entities first, then their translations, then the sentence: the template wins on three language pairs.","key_machinery":"The central object is the three-part target template used during fine-tuning. Each example's output is a sequence of three blocks separated by the special token `<SEP>`: first, the list of named entities found in the source; second, their target-language translations, with multiple entities separated by `|`; third, the full translation of the source in which each entity span is wrapped in the XML tags `<entity>` and `</entity>`. This template carries the argument because it makes entity awareness a generation-order constraint: the model must decide which spans are entities and how to translate them before it writes the final sentence, and the XML tags in the final block give the decoder explicit anchors. The same sequence-to-sequence format lets the model be trained on named-entity recognition and translation with one standard language-modeling loss.","core_discovery":"On the paper's own terms, the discovery is that entity-aware translation can be achieved by changing only the target text used in fine-tuning. The same mT5-large model, asked to output a list of recognized entities, then their translations, then the full translation with `<entity>...</entity>` marks, outperforms the plain mT5 baseline on three of four tested language pairs and outperforms mBART on all four. The authors credit the tight coupling of named-entity recognition and translation: the model must identify and translate entities before writing the full sentence, and the XML tags anchor attention on the relevant spans. The method builds on earlier entity-aware multi-task training, with the main changes being the explicit three-part output template and the label-construction pipeline that aligns entity mentions using a large vision-language model and token-level alignment.","pith_inferences":["A testable extension the paper does not run is to ablate the three output parts separately; removing the NER list, the entity translations, or the tags one at a time would show which part carries the BLEU gain.","The paper never measures whether its automatically built entity labels are correct, so part of the observed gain could come from denoising rather than true entity awareness; a manual label audit would separate these possibilities.","The entity-first ordering suggests a broader generation-planning principle: when a few tokens carry most of a sentence's meaning, making the model produce those tokens first can serve as a cheap explicit plan. This idea is not claimed in the paper.","The template could naturally be tested on low-resource language pairs without entity dictionaries, where the model would have to induce entity translations from parallel text alone; the current evaluation does not cover that setting."],"forward_implications":["Entity-aware translation can be obtained by reformatting the training target alone, so no extra model, dictionary, or retrieval step is needed at inference time.","The same three-part template should transfer to any text-to-text multilingual model, suggesting the gains are not specific to mT5's internals.","Because the XML tags are stripped after generation, the method still returns ordinary translations with only a deterministic cleanup step.","The English–French result shows the method can also regress, so per-language-pair validation is needed before deployment."],"supporting_citations":[{"why":"Defines the task, the JSON data format, and the public and private test sets that all reported BLEU scores come from.","marker":"[3]"},{"why":"Supplies token-level source–target alignments used to build the entity labels for training.","marker":"[4]"},{"why":"Introduces the XML boundary-tagging strategy for entities that the translation portion of the target template adopts.","marker":"[5]"},{"why":"Defines BLEU, the metric used for every reported translation-quality score.","marker":"[6]"},{"why":"Shows the predecessor multi-task setup in which entities detected externally are aligned to target tokens and wrapped in XML tags for T5 training.","marker":"[8]"},{"why":"Provides the end-to-end entity-aware NMT framing that this paper extends by adding the three-part output template.","marker":"[12]"},{"why":"Supplies the mT5-large pretrained multilingual text-to-text backbone that is fine-tuned in all experiments.","marker":"[13]"},{"why":"Supplies the large vision-language model used to extract source–target entity alignments, filtered by requiring entities to appear in the source.","marker":"[14]"}],"fun_headline_variants":["Entity-first output template lifts BLEU by up to 5.3","Multi-task entity template beats mT5 on 3 of 4 pairs","Entity-aware MT improved via multi-task fine-tuning","Structured entity output boosts translation quality","mT5-large entity-first fine-tuning outperforms mBART"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automatically constructed entity labels—combining a large vision-language model's alignments with token-level word alignment—are accurate enough to teach the model which spans are entities and how to translate them, and the paper never measures that accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Entity-first output template lifts BLEU by up to 5.3","Multi-task entity template beats mT5 on 3 of 4 pairs","Entity-aware MT improved via multi-task fine-tuning","Structured entity output boosts translation quality","mT5-large entity-first fine-tuning outperforms mBART"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000581,"raw_usage":{"total_tokens":2662,"prompt_tokens":795,"completion_tokens":1867,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":411,"completion_tokens_details":{"reasoning_tokens":1782}},"tokens_in":411,"tokens_out":1867,"duration_ms":17066,"temperature":1.0,"reasoning_tokens":1782,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:51:35.443404+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fix every training detail except the target template and replace the three-part output with plain translations; if BLEU on English–German, English–Italian, and English–Spanish stays within noise, the entity-aware template is not the cause of the reported gains. Separately, hand-annotate a random subset of the training sentences for entity mentions and compare against the labels the method used; low agreement would mean the model was trained on corrupted entity supervision.","supporting_citations":[{"cited_title":"In: Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025)","cited_arxiv_id":null,"evidence_quote":"Defines the task, the JSON data format, and the public and private test sets that all reported BLEU scores come from."},{"cited_title":"In: Conference of the European Chapter of the Association for Computational Linguistics (EACL) (2021)","cited_arxiv_id":null,"evidence_quote":"Supplies token-level source–target alignments used to build the entity labels for training."},{"cited_title":"In: Proceedings of the 22nd annual conference of the european association for machine translation","cited_arxiv_id":null,"evidence_quote":"Introduces the XML boundary-tagging strategy for entities that the translation portion of the target template adopts."},{"cited_title":"In: Mahamood, S., Minh, N.L., Ippolito, D","cited_arxiv_id":null,"evidence_quote":"Shows the predecessor multi-task setup in which entities detected externally are aligned to target tokens and wrapped in XML tags for T5 training."},{"cited_title":"Machine Learning 111(3), 1181–1203 (2022)","cited_arxiv_id":null,"evidence_quote":"Provides the end-to-end entity-aware NMT framing that this paper extends by adding the three-part output template."}],"review_version":2}