{"id":"865c4ebe-ce94-4c25-aaf3-05f6ec2b517a","arxiv_id":"2501.03008","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"ProNMT combines QE scores with a pronoun-likelihood reward to fine-tune NMT models, but the reported pronoun gains are partly circular and the best configuration was selected on the test set.","lead":"This paper proposes ProNMT, a training method that fine-tunes a translation model using quality estimation scores plus a pronoun-specific reward, and reports large gains in English-to-German pronoun translation. The gains are questionable because the evaluation metrics overlap with the training objectives and the best settings were chosen using the test set.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pronoun-quality claim rests on PGL, which is both the training objective and shown by Table 3 to be gameable; no ContraPro or human validation.","rationale":"The reader's weakest assumption identifies the correct Achilles heel: PGL is the training reward of Eq. (2) and is reported as the headline pronoun metric. This is not merely a missing external benchmark; the paper's own Table 3 demonstrates the metric is gameable. The ONLY-PGL row reaches PGL 0.9675 while COMET collapses to 16.79 and BLEU to 0.07, so a model can maximize the reported pronoun metric while destroying translation quality. Since the paper's novelty and title are about pronoun translation, and the pronoun claim is supported only by PGL, the central contribution is not established. The test-set selection of alpha and beta and the duplicate Table 2 row are additional protocol failures, but they mainly affect the overall-quality comparison; the PGL issue alone is sufficient to reject. A ContraPro evaluation would provide the missing external check and would likely show the claimed pronoun gain does not survive.","tokens_in":8904,"tokens_out":10680,"duration_ms":99322,"concrete_test":"Obtain the released checkpoints or reproduce the best ProNMT and the ONLY-PGL baseline, then evaluate both on the ContraPro EN->DE test set and report standard pronoun-accuracy. If the ProNMT checkpoint does not beat the context-agnostic ProNMT and the pre-trained NLLB model on ContraPro, the claim of improved pronoun translation fails. A secondary check is to compute the Spearman correlation between PGL and ContraPro accuracy over a sample of 200 test sentences; near-zero or negative correlation confirms that PGL is not a valid proxy for pronoun translation quality.","verdict_should_be":"REJECT","load_bearing_attack":"Section 4.3.2 adopts PGL as the headline pronoun-quality metric, but PGL is exactly the quantity optimized by the reward in Eq. (2): a candidate is preferred when the model assigns high probability to the reference pronoun token, and the reported PGL is the average of that same training reward on the test set. The paper never validates PGL against ContraPro or human judgments. More seriously, its own Table 3 shows the metric is gameable: the 'ONLY PGL REWARD' row reaches PGL 0.9675 while COMET collapses to 16.79 and BLEU to 0.07. A model can be trained to emit the reference pronoun with very high confidence while producing near-incoherent translations, so a high PGL does not entail correct pronoun translation in any useful sense. The reported PGL improvements (e.g., 0.4183 vs 0.3672 in Table 3, and 0.8543 in Table 2) therefore demonstrate that the training objective is being optimized, not that pronoun translation has improved. The central claim of pronoun-specific gains rests on a circular metric, and the paper's own ablations provide direct evidence of the failure mode.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ProNMT, an iterative fine-tuning framework for neural machine translation that uses a linear combination of a quality-estimation (QE) reward and a pronoun-specific reward to select candidate translations and fine-tune a pre-trained NLLB-200 model on a filtered English-to-German Europarl corpus. The pronoun reward is the Pronoun Generation Likelihood (PGL), defined as the model's probability of generating the reference pronoun token, with a positive sign if the generated pronoun matches the reference and a negative sign otherwise. The authors report that the context-aware ProNMT configuration achieves higher COMET, BLEU, QE, and PGL scores than a context-agnostic baseline, and they conclude that the framework consistently improves both pronoun translation and overall translation quality.","tokens_in":9137,"tokens_out":4803,"duration_ms":49178,"significance":"The core idea is timely and useful: using a reference-free QE model plus a pronoun-specific reward within an iterative SFT loop is a plausible recipe for improving pronoun translation without additional human annotation. The paper also deserves credit for including ablations that isolate PGL-only and QE-only rewards, which help illustrate the trade-off between pronoun likelihood and overall translation quality. However, as presented, the empirical claims are not established. The headline pronoun metric is the same quantity used as the training reward, the QE metric is the same model used as a reward component, and hyperparameters are selected on the test set. These issues are load-bearing for the paper's central conclusion.","major_comments":[{"comment":"The reported pronoun metric PGL is exactly the RPGL component of the training reward in Eq. (2), and the test-set PGL values in Table 3 are averages of that same reward on the test set. The model is therefore trained to maximize the metric used to claim pronoun gains. The paper's own ablation makes the problem concrete: the 'ONLY PGL REWARD' row in Table 3 reaches PGL 0.9675 while COMET collapses to 16.79 and BLEU to 0.07, showing that PGL can be maximized while producing near-incoherent translations. Consequently, the reported improvement from 0.3672 to 0.4183 (Table 3) does not demonstrate better pronoun translation. The authors should validate on an external benchmark such as ContraPro or with human judgments; Section 4.1 dismisses ContraPro, but the replacement metric is not independently validated.","section":"Section 3.2, Eq. (2); Section 4.3.2; Table 3"},{"comment":"Table 2 reports COMET, BLEU, QE, and PGL for four alpha/beta combinations evaluated directly on the test set, and Table 3 then presents the configuration with 'highest COMET and BLEU scores' as the headline ProNMT result. This is test-set model selection: the hyperparameters are chosen by looking at test performance, which invalidates the claim that ProNMT 'consistently outperforms' baselines on unseen data. The authors should select alpha and beta on a validation set, or report all configurations with confidence intervals and make clear how the reported configuration was chosen.","section":"Table 2 and Table 3"},{"comment":"The evaluation metric labeled 'QE' is computed with 'wmt21-comet-qe-da', which is the same model used as Rtranslation in Eq. (2). Thus the QE column is not an independent measure of translation quality; it is another view of the training reward. The abstract and conclusion state that ProNMT improves performance 'across multiple metrics, including COMET, BLEU and QE models,' but the QE improvement is partly circular. COMET and BLEU are external, and they are the strongest evidence in the paper, but the QE-based claim should be removed or re-framed.","section":"Section 4.3.2; Section 5"},{"comment":"The evaluation is performed on a self-constructed filtered Europarl subset restricted to English 'it' aligned to German 'er', 'sie', or 'es', and the reward requires exact match to a single reference pronoun. The authors acknowledge in Appendix A.1 that pronoun translation is inherently ambiguous, and exact-match rewards can penalize valid alternatives. Without an external benchmark or human evaluation, it is unclear whether the reported PGL gains reflect genuine pronoun translation improvement or overfitting to the filtering and reference-matching scheme. The authors should supplement their test set with ContraPro, or at least provide a human evaluation of pronoun correctness on a sample.","section":"Sections 4.1 and 3.2"}],"minor_comments":[{"comment":"The text says the context fine-tuning runs for 3 epochs, while Table 1 lists 'Number of Train Epochs' as 10; please reconcile this discrepancy.","section":"Section 4.2 vs. Table 1"},{"comment":"The 'BASELINE' row in Table 3 has exactly the same numeric values as the alpha=1.2, beta=1/#avg-len row in Table 2; the relationship between the baseline and the hyperparameter sweep should be clarified.","section":"Tables 2 and 3"},{"comment":"The 'Loss' column is not described in the text; please explain how the loss is computed and why it is reported alongside the translation metrics.","section":"Table 3"},{"comment":"The paper uses the word 'significant' without any statistical significance testing; given a single seed and a limited test set, please add confidence intervals or at least soften the claim.","section":"Section 5"},{"comment":"There are formatting issues such as 'ONLYPGLREWARD' and 'ONLYQE REWARD' in Table 3, and inconsistent figure references in the appendix (e.g., 'Fig. 5' appears before Figure 5 is introduced).","section":"General"}],"recommendation":"reject","confidential_remarks":"I agree with the reader's report: the central empirical claim is undercut by test-set model selection and by the fact that PGL is both the training reward and the headline pronoun metric. The ONLY-PGL ablation in Table 3 is direct evidence of the failure mode. These issues are not cosmetic; they require re-running the evaluation with a proper validation split and adding an external pronoun benchmark or human evaluation. The paper therefore does not meet the bar for publication in its current form, although the underlying idea may be worth pursuing in a revised study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take on arXiv:2501.03008 (ProNMT). The core idea is reasonable: use a QE model as a reward and add a pronoun-specific likelihood term, then iteratively fine-tune via SFT on the best of k samples. That combination is new, and the paper is clearly written. They also built a filtered Europarl dataset and acknowledge limitations honestly. So there is a real thread to pull.\n\nBut the empirical claims do not hold up under inspection. The biggest issue: the headline pronoun metric, PGL, is exactly the training reward. In Eq. (2), R_PGL is the generation likelihood of the reference pronoun, and Section 4.3.2 reports the average of that same reward on the test set as the evaluation metric. That is circular. The paper's own Table 3 shows the failure mode: the ONLY PGL REWARD row reaches PGL 0.9675 while COMET collapses to 16.79 and BLEU to 0.07. A high PGL can be achieved with incoherent translations. Without an external pronoun benchmark like ContraPro or human judgments, the claim of pronoun-specific gains rests on a metric the model is directly optimized against. The paper mentions ContraPro but only to justify building a new dataset; it never evaluates on ContraPro.\n\nThere is also a test-set selection problem. Table 2 reports four alpha/beta configurations evaluated on the test set, and Table 3 then picks the best one as (α*, β*). That is overfitting to the test set. On top of that, the actual gains over simple context-aware SFT are small: ProNMT gets 81.92 vs 81.19 COMET and 26.96 vs 25.57 BLEU. Those differences are within noise, and no significance tests are provided. No code or data is released, making external validation impossible.\n\nOne more thing: there is a suspicious row in Table 2 where the \"1.2 1/#avg−len\" context-aware entry is identical to the baseline row in Table 3. Either a typo or a copy-paste error, but it does not inspire confidence.\n\nSo the central claim of significant pronoun and overall quality improvements is under-supported. The framework is plausible but not convincingly validated. A cleaner evaluation with held-out reward weights, external pronoun metrics, and significance testing is needed.\n\nThis paper is not ready for publication. I would not cite it, and I would not bring it to our reading group as a substantive contribution, though it could serve as a cautionary example of circular evaluation. If the authors revise and rerun the experiments, it might be worth a second look.\n\nMy recommendation: desk reject, or if you want to be generous, send to referees with specific instructions to check the evaluation protocol. As it stands, treat it as a negative result about the evaluation rather than a positive result about the method.\n\nBest,\n\n[Your name]","headline":"Plausible QE-plus-pronoun-likelihood training idea, but the evaluation rests on a circular metric and test-set tuning.","tokens_in":9701,"tokens_out":3851,"would_cite":false,"duration_ms":111425,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that combining a reference-free quality-estimation reward with a pronoun-generation-likelihood reward in iterative fine-tuning improves both pronoun translation and overall English-to-German translation, reporting a…","keywords":["pronoun translation","neural machine translation","quality estimation","reward-based fine-tuning","document-level context","iterative supervised fine-tuning","pronoun generation likelihood","English-German translation"],"falsifier":"Run the trained context-aware ProNMT checkpoint on the ContraPro contrastive pronoun test: if it does not select the correct pronoun significantly more often than the context-agnostic baseline despite reporting higher PGL, then PGL is not a faithful proxy for pronoun correctness and the central claim is falsified.","tokens_in":8685,"feed_emoji":"🌐","tokens_out":9127,"duration_ms":77352,"temperature":0.7,"pith_summary":"ProNMT is a training framework that iteratively fine-tunes a pre-trained machine translation model using a reward signal built from a reference-free quality-estimation score plus a pronoun-specific bonus. The bonus, called pronoun generation likelihood (PGL), is the model's own probability of emitting the reference pronoun, signed negatively when the candidate pronoun does not match the reference. The authors claim that this feedback, applied to a distilled 600M-parameter NLLB model on a filtered English-to-German Europarl subset, improves both pronoun handling and overall translation quality—best context-aware scores are COMET 81.92 and BLEU 26.95, versus 73.95 and 15.2630 for the context-free baseline. The motivation is to capture part of the benefit of human-feedback training without human annotations.","feed_headline":"COMET jumps from 73.95 to 81.92 on context-aware NMT","feed_subtitle":"ProNMT pairs quality-estimation reward with pronoun likelihood to fine-tune EN-DE translation without human labels.","key_machinery":"The key mechanism is the reward function $r(x,y) = \\alpha R_{\\text{translation}} + \\beta R_{\\text{PGL}}$. $R_{\\text{translation}}$ is the score from a reference-free COMET quality-estimation model, and $R_{\\text{PGL}}$ is the 'pronoun generation likelihood': the conditional probability the model assigns to the reference pronoun token at the pronoun position, taken with a positive sign if the candidate pronoun equals the reference, a negative sign if it does not, and zero if no pronoun appears. This reward is used to choose among $k=10$ sampled candidate translations for each source sentence, and the chosen translation becomes the supervised fine-tuning target for the next iteration. Context is provided by formatting the source as '<context> $x_{i-1}$ </context> $x_i$', and the training data is filtered to English 'it' aligned to German 'er', 'sie', or 'es' to reduce pronoun ambiguity.","core_discovery":"The central discovery the paper argues for is that a linear reward $r(x,y) = \\alpha R_{\\text{translation}} + \\beta R_{\\text{PGL}}$, where $R_{\\text{translation}}$ comes from a reference-free COMET quality-estimation model and $R_{\\text{PGL}}$ is the pronoun generation likelihood (positive if the generated pronoun matches the reference, negative otherwise, zero if no pronoun is generated), can steer iterative supervised fine-tuning to improve pronoun translation and overall translation quality simultaneously. On their filtered Europarl EN→DE test set, the context-aware configuration with $\\alpha=1.2$ and $\\beta=1/\\#\\text{tokens}$ reaches COMET 81.92 and BLEU 26.9574, and the authors conclude that the combination matters because PGL-only training inflates PGL but collapses translation quality, while QE-only training improves translation without the pronoun-specific gains.","pith_inferences":["Because $R_{\\text{PGL}}$ is the model's own probability of generating the reference token, it is a self-referential signal: optimizing it may inflate confidence without improving genuine pronoun disambiguation, and the observed collapse of PGL-only training is consistent with the model learning to output high-probability pronouns rather than correct ones.","A decisive check the paper does not run is the ContraPro contrastive test; if ProNMT's higher PGL does not translate into higher contrastive accuracy on pronouns, the claimed pronoun gains are an artifact of the reward metric.","The framework's design suggests it could extend beyond 'it' to other ambiguous pronouns or morphological agreement phenomena, but the reward definition assumes a unique reference pronoun, which limits it to cases where the target pronoun is unambiguous.","The paper's dismissal of ContraPro as too easy for fine-tuning is itself a claim worth testing: a head-to-head comparison with a model fine-tuned on ContraPro would show whether the filtered Europarl data is the reason for the gains or whether the gains come from the reward mechanism."],"forward_implications":["Context-aware ProNMT exceeds the context-agnostic version on COMET, BLEU, QE, and PGL, so adding the previous sentence as context is part of the reported gains.","The joint QE-plus-PGL reward beats either reward alone; PGL-only training reaches PGL of 0.8998–0.9675 but collapses BLEU, so the two signals are complementary rather than redundant.","Reference-free quality estimation can substitute for human reward models in this pronoun-focused fine-tuning loop, removing the need for human annotations in the training pipeline.","Improvements are not confined to pronoun metrics: the same training run lifts overall COMET and BLEU, indicating the pronoun-focused signal does not trade away general quality."],"supporting_citations":[{"why":"Establishes that quality-estimation models can serve as reward models, the premise for using COMET-QE as the translation reward.","marker":"(He et al., 2024)"},{"why":"Provides the reference-free COMET quality-estimation model used to compute $R_{\\text{translation}}$.","marker":"(Rei et al., 2020)"},{"why":"Shows document-level context helps NMT resolve anaphora, motivating the context-aware setting.","marker":"(Voita et al., 2018)"},{"why":"Defines the pronoun-translation challenge, the ContraPro test set, and the observation that 'it' maps to multiple German pronoun forms used in data filtering.","marker":"(Müller et al., 2018)"},{"why":"Supplies the RLHF/PPO training baseline the framework seeks to avoid on cost and stability grounds.","marker":"(Ramamurthy et al., 2023)"}],"fun_headline_variants":["ProNMT: QE + pronoun reward hits 81.92 COMET","Fine-tune NMT with QE & pronoun likelihood: 81.92 COMET","Pronoun-translation boost: QE-guided fine-tuning reaches 81.92","QE + pronoun reward: ProNMT improves EN-DE translation","No human labels needed: ProNMT lifts COMET to 81.92"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the pronoun generation likelihood—the model's own probability of producing the reference pronoun token—faithfully measures whether the pronoun is translated correctly; the paper never validates PGL against human judgments or the standard ContraPro contrastive test, so if PGL can be inflated without real pronoun improvement, the reported pronoun gains support the central claim only circularly.","fun_headline_variants_meta":{"raw":{"variants":["ProNMT: QE + pronoun reward hits 81.92 COMET","Fine-tune NMT with QE & pronoun likelihood: 81.92 COMET","Pronoun-translation boost: QE-guided fine-tuning reaches 81.92","QE + pronoun reward: ProNMT improves EN-DE translation","No human labels needed: ProNMT lifts COMET to 81.92"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000852,"raw_usage":{"total_tokens":3667,"prompt_tokens":869,"completion_tokens":2798,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":2707}},"tokens_in":485,"tokens_out":2798,"duration_ms":18991,"temperature":1.0,"reasoning_tokens":2707,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:58:38.049258+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained context-aware ProNMT checkpoint on the ContraPro contrastive pronoun test: if it does not select the correct pronoun significantly more often than the context-agnostic baseline despite reporting higher PGL, then PGL is not a faithful proxy for pronoun correctness and the central claim is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the reference-free COMET quality-estimation model used to compute $R_{\\text{translation}}$."}],"review_version":1}