{"id":"699204f6-57eb-4b44-a8a3-0c7fdcb57c74","arxiv_id":"1908.09532","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Fine-tuning a neural translation model on a small subset of sentences selected by test-set similarity can outperform fine-tuning on the full in-domain corpus.","lead":"The authors test three ways to pick a small set of training sentences that look like a document to be translated, then fine-tune a neural machine translation model on just that subset. For two German-to-English test sets, the adapted models often beat both the generic model and a model fine-tuned on the full in-domain data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TF-IDF selection as written maximizes cosine distance, so the paper's best result may come from an unspecified variant.","rationale":"The reader's CONDITIONAL verdict is appropriate, but the most concrete defect is not only the missing random-subset control; it is that the TF-IDF method, the source of the best result in Section 7.3, is mathematically mis-specified. Eq. (3) defines sim as 1-cos, i.e., cosine distance, and Eq. (4) maximizes it, so the written algorithm selects sentences farthest from the test set. That contradicts the stated goal of choosing test-relevant sentences and cannot be dismissed as a mere wording issue because no code is released and the headline result is not replicated. The proposed overlap test would settle whether the printed equations can produce the reported number or whether an unstated correction, such as using cosine similarity or taking a minimum, was actually used. The random-subset control identified by the reader is a separate and real weakness: without it, improvements over full-data fine-tuning could in principle be a small-data effect. Both issues should be addressed before the central claim is accepted. If the authors release the selection code or correct Eqs. (3)-(4) and add a random-subset baseline, the paper would be substantially stronger. I therefore keep the CONDITIONAL verdict, with the added condition that the TF-IDF definition be reconciled with the reported experiments.","tokens_in":12161,"tokens_out":9038,"duration_ms":93492,"concrete_test":"Re-run the TF-IDF selector on the BASE+EMEA pool used in Table 11 under both readings: (A) score(s)=max_r [1-cos(v_s,v_r)] exactly as written, and (B) score(s)=max_r cos(v_s,v_r) (equivalently min cosine distance). Compare the 500K sets selected by A and B; if their overlap is small, they are different algorithms. Then fine-tune BASE12 on each selected set for one epoch on the health test and compute BLEU. If only B reproduces 36.79, Eqs. (3)-(4) as printed cannot have generated the headline result; if A also reproduces it, the paper's theoretical motivation needs rethinking. A cheaper first check is to compute mean cosine similarity to the test set for the top-1000 sentences under A versus B; under A it should be lower, confirming the inversion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest measured result (Section 7.3, Table 11: TFIDF 500K, BLEU 36.79) is produced by the method defined in Eqs. (3)-(4), but those equations do not describe a test-similarity selector. Eq. (3) defines sim(a,b)=1-cos(w_a,w_b), which is cosine distance: identical vectors get sim=0, orthogonal vectors get sim=1. Eq. (4) then scores each candidate by the maximum of this quantity over test sentences, i.e., it chooses the sentence with the largest distance to the test set, the opposite of highest similarity. The surrounding text says the method selects sentences closer to a test set and the method is called transductive, so either Eq. (3) should be the cosine similarity (drop the 1-) or Eq. (4) should take the minimum. The paper does not provide code, and Section 7.3 does not report which variant was run. Since the same sign error affects the definition behind the paper's best number, the paper as written does not specify the algorithm that generated its central empirical claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes three transductive data-selection algorithms — TF-IDF distance, Infrequent n-gram Recovery (INR), and Feature Decay Algorithms (FDA) — that use the source side of a given test set to retrieve a subset of parallel sentences for fine-tuning a pre-trained NMT model. The authors evaluate German-to-English models on news and health test sets, comparing fine-tuning on selected subsets (100K/200K/500K sentences from general-domain, in-domain, or mixed candidate pools) against the base model and against full-data fine-tuning. The main claimed results are that fine-tuning on transductively selected subsets can outperform both the generic model and a domain-adapted model, with the strongest result being a TF-IDF-selected 500K mixture of BASE and EMEA data reaching 36.79 BLEU on the health test, versus 34.69 for full-EMEA fine-tuning (Table 11). Sections 7.1–7.3 also analyze the composition of selected data and the runtime/threshold behavior of INR.","tokens_in":12335,"tokens_out":3833,"duration_ms":39010,"significance":"If the results hold, the paper offers a practical recipe: given a test document at translation time, one can fine-tune an existing NMT model on a small, selected subset of available parallel data and outperform both a generic model and a model fine-tuned on the entire in-domain corpus, without training a full domain model. The work is also useful in framing data selection as a test-time adaptation tool and in comparing three selection criteria under a common experimental protocol. The paper ships a clean setup (WMT15 training data, OpenNMT-py, BPE, significance testing with multeval) and includes a useful analysis of the domain composition of the selected subsets (Table 9). However, the central empirical claim is currently weakened by a missing random-subset control and by an internal inconsistency in the definition of the TF-IDF method that produced the headline number; both issues must be resolved before the contribution is fully supported.","major_comments":[{"comment":"The TF-IDF method as written does not implement the described transductive selection. Eq. (3) defines sim(a,b) = 1 - cos(w_a,w_b), which is cosine distance: identical vectors receive a score of 0 and orthogonal vectors receive a score of 1. Eq. (4) then scores each candidate by the maximum of this quantity over test sentences, i.e., it selects the candidate sentence with the largest distance to the test set, not the highest similarity. The surrounding text and the method name state the opposite. Since the strongest reported result (Table 11, TFIDF 500K, BLEU 36.79) is attributed to this method, the paper as written does not specify the algorithm that generated its central empirical claim. Please either correct Eq. (3) to use cosine similarity (remove the 1-) or change Eq. (4) to take the minimum, and explicitly state which variant was run in Section 7.3.","section":"Section 4, Eqs. (3)–(4)"},{"comment":"All comparisons of selected subsets are made against the full-data fine-tuned model or the base model, never against a random subset of the same size. This is a load-bearing control: without it, the observed improvements cannot be attributed to the selection algorithms, because fine-tuning on a smaller, possibly more homogeneous data set could by itself produce gains. The premise enters in every main experiment table (Tables 5–8, 10–11), where each selected subset at 100K/200K/500K should be accompanied by a random-subset baseline at the same size and from the same candidate pool. Please add these controls or state why they are unnecessary.","section":"Sections 7.1–7.3, Tables 5–11"},{"comment":"The first baseline column is labeled \"BASE13\" but contains the BASE12 values from Tables 1 and 2 (BLEU 26.16 on the news test and 33.29 on the health test, respectively). The text in Section 7.3 says this column \"shows the model built performing 13 epochs,\" which is inconsistent with the values. This labeling error makes the claim that scores are \"better than both baselines\" ambiguous and should be corrected before publication.","section":"Tables 10 and 11"}],"minor_comments":[{"comment":"The text reads \"the component max(0, t - CS(ngr)) is 0\" but CS is not defined; it should presumably be CL(ngr). Also, clarify whether the infrequency threshold t is measured with respect to the selected pool L or the original candidate pool, since the current description makes Eq. (5) depend on an initially empty L.","section":"Section 4, Eq. (5)"},{"comment":"The text states that INR retrieves at most 230K sentences for news and 275K for health, but Tables 5 and 6 show INR rows for 200K and no rows for 500K; please state explicitly that the 500K INR experiments are absent because the retrieved count is below 500K.","section":"Section 7.1, Tables 5 and 6"},{"comment":"The description of the TF-IDF method calls Eq. (3) the \"inverse of the cosine distance,\" which is not an accurate name for 1 - cos(wa,wb); if this wording follows the original work of Hildebrand et al. [14], please clarify the provenance and the intended sign convention.","section":"Section 4, final paragraph"},{"comment":"The paper reports BPE with 89,500 merge operations and then says the vocabulary size is 50,000 words for each language; this could confuse readers about the effective subword vocabulary and should be clarified.","section":"Section 5, model configuration"}],"recommendation":"major_revision","confidential_remarks":"The paper is a revised arXiv version (v3) of what appears to be a workshop paper. The core idea is reasonable, but the TF-IDF sign inconsistency is a serious reproducibility issue that the authors must fix, and the absence of random-subset baselines is a standard control that should have been included. I would not recommend rejection, since the other two selection methods (INR and FDA) and the general experimental design are sound; a focused revision can address these concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thanks for the report. I read the paper with the stress-test note in hand, and the note is correct. Equation (3) defines sim(a,b) = 1 - cos(w_a,w_b), which is cosine distance. Equation (4) takes the max over test sentences, so it picks candidates furthest from the test set. The text says the opposite. Since the authors don't provide code, the paper does not specify the algorithm behind its headline TF-IDF result (health mixture, 500K, BLEU 36.79 vs 34.69 full-EMEA). Maybe the equation is a typo and they used cosine similarity, maybe they used the min. As written, it is wrong.\n\nThe rest of the paper is a reasonably honest empirical study. The transductive framing — selecting sentences with the test set's source side, then fine-tuning for one epoch — is not entirely new, but the systematic comparison across general, in-domain, and mixed candidate pools, with two test domains and significance tests, is a useful contribution. The INR and FDA methods are described consistently enough to reimplement, and their results are modest but credible. I agree with you that the missing random-subset baseline is a real weakness: gains over full-data fine-tuning do not isolate the selection mechanism, so part of the effect could simply be a smaller fine-tuning set. That softens the claim but does not kill it.\n\nAlso confirmed: Tables 10 and 11 label the first baseline column BASE13 but the numbers match BASE12 (26.16 and 33.29). That is a labeling error, not a deep flaw. The abstract overstates consistency; the news-domain results are often not better than BASE13. I have no circularity concerns: test set usage is source-only, and the INR threshold is set by runtime, not test-set performance.\n\nThe paper is worth a serious referee. The main fix is to correct Eq. (3) (or Eq. (4)) and report which variant was actually run, plus add a random-subset control and correct the table labels. If they do that, the central claims become testable. I would not cite the TF-IDF numbers in their current form, but the FDA/INR comparison and the transductive framing are useful. I'd take it to a reading group to talk about the equation issue, if nothing else.\n\nRecommendation: send to peer review with an expectation of revision, and make sure a reviewer checks the TF-IDF definition against the reported experiments.","headline":"Useful empirical comparison of transductive data selection for NMT fine-tuning, but the TF-IDF equations as written select the least similar sentences, so the paper's best result is not reproducible as specified.","tokens_in":12889,"tokens_out":2214,"would_cite":false,"duration_ms":22397,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a translation model on a small subset of sentences chosen for their n-gram overlap with the test set can beat both the generic base model and a model fine-tuned on the full in-domain data.","keywords":["transductive data selection","fine-tuning","neural machine translation","domain adaptation","feature decay algorithm","infrequent n-gram recovery","TF-IDF","test-time adaptation"],"falsifier":"Fine-tune the same base model on a random subset of the same size drawn from the same candidate pool, and compare BLEU on the test set with the transductive subsets; if the random subsets match the selected subsets' scores, the selection criteria add no value beyond reducing data size.","tokens_in":11926,"feed_emoji":"🎯","tokens_out":7231,"duration_ms":57970,"temperature":0.7,"pith_summary":"Translation quality often improves when a model is adapted to the document it will translate. This paper tests a transductive approach: use the test document itself to rank a pool of parallel sentences by source-side n-gram overlap, fine-tune a generic neural machine translation model on the top-ranked subset for one epoch, and translate. Across German-to-English news and health experiments, the selected subsets frequently outperform the base model and often outperform fine-tuning on the complete in-domain corpus. In the strongest result, a health-domain model fine-tuned on 500K sentences selected by TF-IDF from a mixture of general and in-domain data reaches BLEU 36.79, versus 34.69 for full-EMEA fine-tuning and 33.29 for the base model. The paper concludes that test-time adaptation with a small selected subset is a practical alternative to building separate domain models.","feed_headline":"Test-set-selected data beats generic and domain-adapted translation","feed_subtitle":"Pick sentences by test-set overlap, fine-tune for one epoch, and beat the domain-specific model.","key_machinery":"The central mechanism is the transductive scoring function that maps each candidate sentence to a relevance score based on n-gram overlap with the test set. Three variants are used: TF-IDF cosine similarity (the maximum similarity to any test sentence), infrequent n-gram recovery (counting rare n-grams shared with the test set, with counts decayed as sentences are selected), and the feature decay algorithm, whose score for a sentence is the length-normalized sum $\\sum_{ngr\\in F_s} 0.5^{C_L(ngr)}$, where $C_L(ngr)$ is the number of times that n-gram already appears in the selected pool. The decay term encourages diversity, so the selected subset covers many different test-set n-grams rather than repeating the most similar sentences. The selected subset is then used to fine-tune the base model for one additional epoch.","core_discovery":"The paper's central claim is that transductive data selection—ranking candidate parallel sentences by their source-side n-gram similarity to the test set—plus a single epoch of fine-tuning produces a model that translates that test set better than the generic model or a model fine-tuned on the full in-domain data. The claim is supported by experiments using three selection algorithms: TF-IDF cosine distance, infrequent n-gram recovery (INR), and the feature decay algorithm (FDA). All three select 100K–500K sentences from either the general training data, the in-domain corpus, or a mixture of both. In the news domain, the selected subsets recover the performance lost when fine-tuning on the full in-domain news corpus, and in the health domain the mixture experiment with TF-IDF achieves a statistically significant gain of 2.1 BLEU over full-EMEA fine-tuning (36.79 vs 34.69). The authors interpret this as evidence that the selected sentences are closer to the test document than the full domain corpus is.","pith_inferences":["The paper never compares against a random subset of the same size, so part of the observed gain could come from fine-tuning on fewer, more predictable sentences rather than from the n-gram selection itself; a random-subset control would separate these effects.","The selection signals are purely source-side and lexical; incorporating target-side or semantic similarity (e.g., embeddings of test sentences) might further improve ranking, especially for test sets with little n-gram overlap.","Because all experiments use a single language pair and two domains, the generality of the conclusion across languages, domains, and model architectures remains an open question; the same protocol could be applied to multilingual models or low-resource settings."],"forward_implications":["A translation provider that has access to the test document and owns the model can adapt at translation time without training or storing a separate model per domain.","Because only one epoch of fine-tuning is needed, the added latency is that of scoring the candidate pool and one training pass, making the approach feasible for document-level translation workflows.","The improvements are largest when the candidate pool mixes general and in-domain data, suggesting that adding a broad pool of sentences gives the selection algorithm more relevant material to find.","In the news experiments, selecting from the in-domain corpus alone still underperforms the base model, so the choice of candidate pool matters as much as the selection algorithm."],"supporting_citations":[{"why":"Provides the dataset configuration and experimental design the paper bases its setups on.","marker":"[3]"},{"why":"Introduces the feature decay algorithm used as one of the three selection methods.","marker":"[4]"},{"why":"Introduces TF-IDF distance for sentence selection in statistical machine translation, adapted here to neural models.","marker":"[14]"},{"why":"Establishes fine-tuning as the standard adaptation method that the paper applies to the base model.","marker":"[21]"},{"why":"Defines infrequent n-gram recovery, the second selection algorithm used in the experiments.","marker":"[24]"},{"why":"Introduces byte-pair encoding, which the paper argues removes the vocabulary limitation of earlier fine-tuning work.","marker":"[33]"}],"fun_headline_variants":["Test-set-guided data selection beats domain fine-tuning","Transductive data selection outperforms domain-specific fine-tuning","Pick sentences by test set for better NMT than domain adaptation","Use test set to pick fine-tuning data: beats generic and domain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that source-side n-gram overlap with the test set is a sufficient signal for choosing fine-tuning data; because no random-subset control is run, the gains attributed to selection could in principle come from the smaller fine-tuning data size alone.","fun_headline_variants_meta":{"raw":{"variants":["Test-set-guided data selection beats domain fine-tuning","Transductive data selection outperforms domain-specific fine-tuning","Pick sentences by test set for better NMT than domain adaptation","Use test set to pick fine-tuning data: beats generic and domain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000659,"raw_usage":{"total_tokens":2973,"prompt_tokens":866,"completion_tokens":2107,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":2040}},"tokens_in":482,"tokens_out":2107,"duration_ms":17080,"temperature":1.0,"reasoning_tokens":2040,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:07:58.672222+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune the same base model on a random subset of the same size drawn from the same candidate pool, and compare BLEU on the test set with the transductive subsets; if the random subsets match the selected subsets' scores, the selection criteria add no value beyond reducing data size.","supporting_citations":[{"cited_title":"Feature decay algorithms for fast deployment of accurate statistical ma- chine translation systems","cited_arxiv_id":null,"evidence_quote":"Provides the dataset configuration and experimental design the paper bases its setups on."},{"cited_title":"Instance se- lection for machine translation using feature decay algorithms","cited_arxiv_id":null,"evidence_quote":"Introduces the feature decay algorithm used as one of the three selection methods."},{"cited_title":"Adapta- tion of the translation model for statistical machine translation based on information re- trieval","cited_arxiv_id":null,"evidence_quote":"Introduces TF-IDF distance for sentence selection in statistical machine translation, adapted here to neural models."},{"cited_title":"Stanford neural machine translation systems for spoken language domains","cited_arxiv_id":null,"evidence_quote":"Establishes fine-tuning as the standard adaptation method that the paper applies to the base model."},{"cited_title":"Data selection for nmt using infrequent n-gram recovery","cited_arxiv_id":null,"evidence_quote":"Defines infrequent n-gram recovery, the second selection algorithm used in the experiments."},{"cited_title":"Neural machine translation of rare words with subword units","cited_arxiv_id":null,"evidence_quote":"Introduces byte-pair encoding, which the paper argues removes the vocabulary limitation of earlier fine-tuning work."}],"review_version":1}