{"id":"8af9f0b7-a368-4b59-8eb0-69d664732ce9","arxiv_id":"2506.00507","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"DAT generates query-specific in-context translation examples using only an LLM, improving English-to-low-resource translation over zero-shot in most tested languages.","lead":"This paper proposes Demonstration Augmentation for Translation (DAT), a method that makes a large language model generate its own translation examples on the fly when translating into low-resource languages. It reports small but statistically significant gains over zero-shot translation for several languages without needing human-annotated parallel pairs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated target-side quality: DAT's gains depend on LLM-generated translations of synthetic sources, which are only checked with reference-free COMET.","rationale":"The reader's weakest_assumption correctly identifies the quality of the LLM-generated target-side demonstrations as the load-bearing premise. The paper's own Table 2 uses reference-free COMET, which is not a human judgment and may be especially unreliable for low-resource languages. The paper itself (Section 6) admits a fine-tuned NMT model could yield higher-fidelity pairs, indicating the authors recognize the assumption. An additional supporting observation is the Table 2 uniformity contradiction: DAT's selected sources are highly similar to each other, undermining the claimed diversity prior and suggesting the MMR weight λ is either very small or unreported, which further weakens confidence in the mechanism. The central claim is not internally inconsistent, and the reported COMET gains in most languages make the method plausible, but the evidence base does not yet rule out the possibility that the gains stem from relevance alone or from target-side artifacts. A human evaluation of generated pair quality, combined with a comparison against human-translated demonstrations, would directly settle whether the target-generation assumption holds. Thus the reader's CONDITIONAL verdict is appropriate, and no verdict change is needed.","tokens_in":18571,"tokens_out":9048,"duration_ms":88302,"concrete_test":"Sample 100 devtest queries per language, run the full DAT pipeline with Llama-3.1-70B, and retain the MMR-selected source sentences. Have professional translators (a) rate the adequacy and fluency of the LLM-generated target sentences and (b) produce human reference translations for those same source sentences. Then compare final-system COMET when the demonstrations use LLM-generated targets versus human-generated targets. If the human-target COMET exceeds the LLM-target COMET by more than 2 points, the target-generation assumption is load-bearing; if the gap is small, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that DAT improves low-resource translation over zero-shot rests on the assumption in Section 3.3 that the same LLM's zero-shot translations of the generated source sentences are good enough to serve as target-side demonstrations. The only evidence for this is the reference-free COMET 'Quality' column in Table 2, which is itself a model-based estimate rather than a human judgment. If these target sentences are systematically inaccurate, they could mislead the few-shot decoder, making the observed gains fragile or non-generalizable. The paper's Section 6 even acknowledges that a fine-tuned NMT model 'could produce higher-fidelity pairs,' but no experiment quantifies how much the final translation quality depends on the fidelity of the self-generated targets. Notably, Table 2 also shows DAT's Uniformity (24.1) far exceeding the Fixed set's (2.8), contradicting the stated diversity objective and suggesting the MMR weight λ may be near zero or unreported, so the method may effectively be relevance-only. Both issues underscore that the mechanism behind the reported gains is not yet pinned down with externally validated evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Demonstration Augmentation for Translation (DAT), a method for generating in-context examples for LLM-based machine translation without relying on human-annotated parallel data. Given a user query, DAT prompts an LLM to generate several diverse and relevant source-language sentences, filters them via maximal marginal relevance (MMR), translates the selected sentences with the same LLM, and uses the resulting source-target pairs as few-shot demonstrations for translating the query. Experiments on English-to-Nepali, Khmer, Pashto, Zulu, and Swahili using Llama-3.1-8B and Llama-3.1-70B are reported, with COMET and reference-free COMET scores. The authors report that DAT improves over zero-shot translation 'in most' low-resource languages, that fixed human-curated pairs can backfire (notably for Khmer with Llama-3.1-70B), and that an accumulation variant can build a reusable demonstration pool at reduced computational cost. The implementation is publicly available.","tokens_in":18793,"tokens_out":7490,"duration_ms":68424,"significance":"The paper tackles a real and timely problem: enabling in-context learning for machine translation in low-resource languages where human-annotated demonstration pairs are unavailable. The idea of having the LLM generate its own demonstrations is genuinely novel in the context of MT and could be practically useful as a bootstrapping technique. The experimental setup is reasonable for a first exploration, and the authors include some positive elements: publicly released code, significance testing with compare-mt, ablations on the number of generated versus selected demonstrations, and a surface-level qualitative example. However, the central claim is expressed too strongly relative to the data, a key diversity mechanism is contradicted by the paper's own metrics, and a core hyperparameter is unreported, weakening both the internal consistency and reproducibility of the contribution.","major_comments":[{"comment":"The sentence 'In Table 1, DAT without fixed pairs outperforms the few-shot approach in translation quality for all languages except Zulu' is factually incorrect for Llama-3.1-8B. For that model, DAT without fixed pairs is worse than the fixed-pair few-shot baseline on Nepali (74.9 vs 75.9), Khmer (64.4 vs 65.0), Pashto (54.6 vs 57.9), and Zulu (22.3 vs 24.7). The claim holds only for Llama-3.1-70B, and even then Zulu is an exception. This overgeneralization underlies the subsequent research question about high-quality fixed pairs versus query-relevant moderate-quality pairs; it needs to be corrected and the model-size interaction discussed.","section":"§5.1, Table 1"},{"comment":"The reported Uniformity scores conflict with the method's stated diversity objective. DAT's mean pairwise source-side n-gram overlap is 24.1, far higher than the Fixed set's 2.8, indicating that DAT's selected sources are highly similar to one another. The paper acknowledges a relevance/uniformity interdependence but does not reconcile it with the claim that MMR filtering promotes diversity. Moreover, the diversity weight λ in Eq. (3) is never specified anywhere in the paper, so the filtering procedure is not reproducible and the effective trade-off cannot be assessed. Please report λ for each experiment and provide an analysis (e.g., a small sweep) of how λ affects the relevance-diversity balance and downstream COMET.","section":"§5.2, Table 2 and Eq. (3)"},{"comment":"The method's success rests on the assumption that the LLM's zero-shot translations of the generated source sentences are of sufficient accuracy to serve as demonstrations. The only support offered is the reference-free COMET 'Quality' column in Table 2, which is a model-based estimator rather than a human judgment. Because the downstream gains are attributed to these self-generated pairs, the paper would be substantially strengthened by a human evaluation of a sample of generated pairs, or by an experiment that varies the target generator (e.g., a fine-tuned NMT model) to quantify how sensitive DAT's gains are to pair fidelity. The Section 6 discussion acknowledges this possibility but does not test it.","section":"§3.3 and Table 2"}],"minor_comments":[{"comment":"The bullet point 'Experimentes show that DAT boosts the translation quality...' contains a typo; it should read 'Experiments'.","section":"Contributions, §1"},{"comment":"The y-axis of Figure 3 is not labeled; the text mentions 'COMET scores', but the axis itself should carry that label for clarity.","section":"Figure 3"},{"comment":"The definition of Uniformity is vague ('evaluates the same averaged score among the source sides of different in-context examples'); a precise formula, e.g., the average pairwise α(x_i, x_j) over selected sources, would be easier to interpret.","section":"§5.2"},{"comment":"The abstract and conclusion claim that DAT 'achieves superior translation quality' without noting the Zulu exception where DAT is significantly worse than zero-shot (Table 1). A qualifier such as 'in most of the evaluated low-resource languages' would align the claims with the data.","section":"Abstract and §7"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable first exploration of an interesting idea, but the overbroad claim in §5.1 and the unreported λ are the kinds of issues that must be fixed before publication. The diversity contradiction is particularly worrying because it suggests that the MMR component may not be doing what the authors believe; if the method actually reduces to relevance-only generation, the framing should be adjusted accordingly. The target-side quality concern is a matter of external validity; a small human evaluation or a target-generator swap would substantially raise confidence. The paper's positioning relative to prior work on synthetic demonstrations is clear and fair."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper you asked about proposes Demonstration Augmentation for Translation (DAT): for English-to-low-resource translation, the LLM generates query-relevant source sentences, filters them via MMR, translates the survivors with the same LLM, and uses those self-made pairs as few-shot demonstrations. The headline result is real: DAT beats zero-shot on most of their five languages and both Llama sizes, with COMET gains around one to three points. That is a genuinely useful capability for scenarios where no human-annotated pairs exist.\n\nWhat is new is the zero-resource constraint. Prior methods (including El Mekki and Abdul-Mageed 2025) need vocabularies or monolingual target data. This paper needs only the LLM. The building blocks are familiar - self-generated ICL, n-gram relevance, MMR diversity - but the packaging for MT is original and practical. Credit where due: the study uses significance testing, explores why fixed high-quality pairs sometimes backfire, includes an accumulation setting for efficiency, and ships code.\n\nNow the soft spots. The biggest is a factual error in Section 5.1: the text says DAT without fixed pairs outperforms few-shot for all languages except Zulu, but Table 1 shows few-shot is actually better for Nepali, Khmer, and Pashto with Llama-3.1-8B. The abstract's 'superior to baselines' suffers from the same overreach. Second, the MMR weight λ is never reported. Table 2's numbers suggest λ is close to zero: DAT's Uniformity is 24.1, ten times the Fixed set's 2.8, so the diversity term is barely doing anything. The paper mentions the relevance-diversity trade-off but does not give the value. That is a reproducibility gap. Third, the stress-test concern has merit but is not fatal: target-side quality is checked only with reference-free COMET, a model estimate, and the paper itself admits a fine-tuned NMT model could make better pairs. Still, the central comparison to zero-shot is clean - no fitting to the test metric - so this is a limitation, not a circularity.\n\nThis is a paper for people building low-resource MT systems and for anyone working on self-generated ICL. It deserves a serious referee. The core result is reproducible in principle, the method is simple, and the issues are fixable. I would send it to review, with a request to correct the overclaim, report λ, and ideally add a small reference-based or human check on a subset of generated pairs.","headline":"A practical, mostly sound recipe for self-generating MT demonstrations in low-resource directions; the central claim holds up, but the paper has an internal text-table contradiction and hides the MMR weight.","tokens_in":19292,"tokens_out":5019,"would_cite":true,"duration_ms":42320,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A language model can generate its own in-context translation examples without any human-annotated pairs, and those self-made demonstrations improve low-resource translation quality.","keywords":["in-context learning","machine translation","low-resource languages","demonstration generation","maximal marginal relevance","few-shot prompting","zero-shot translation"],"falsifier":"Compare DAT against a version where the k target sentences are replaced by deliberately wrong translations of the same generated source sentences: if the two perform alike, then target-side quality, not source-side relevance, is carrying the result.","tokens_in":18414,"feed_emoji":"🌐","tokens_out":7079,"duration_ms":59179,"temperature":0.7,"pith_summary":"Machine translation into low-resource languages usually needs parallel sentence pairs for in-context examples, but such paired data barely exists. This paper proposes Demonstration Augmentation for Translation (DAT), a method in which the translation model itself writes source sentences that resemble the user's query while remaining different from one another, filters the most relevant and diverse few, and translates those sentences with the same model to build its own demonstration pairs. The claim is that these self-generated, query-tailored examples improve translation quality over zero-shot translation in most of the five tested English-to-low-resource settings, and even outperform few-shot prompting with fixed human-curated pairs. If true, low-resource translation would no longer require a human-annotated demonstration pool, and the same trick could bootstrap reusable data as translations accumulate at test time.","feed_headline":"LLMs can generate their own translation examples, no human pairs needed","feed_subtitle":"The model builds its own query-tailored examples and lifts translation quality in most tested low-resource languages.","key_machinery":"The carrying mechanism is a two-stage demonstration factory. A source generator paraphrases the user query under explicit relevance and diversity instructions, then a Maximal Marginal Relevance (MMR) filter scores each candidate by averaged n-gram overlap with the query ($\\alpha(q,x_i)$) minus a redundancy penalty that averages overlap with already selected sentences. This produces k source sentences; the same LLM then acts as target generator, producing zero-shot translations of those sources. The resulting pairs are query-specific on the source side, which the paper argues matters more than the intrinsic quality of fixed pairs.","core_discovery":"The paper's central discovery is that an LLM can bootstrap its own in-context demonstrations for translation without any external parallel data. Given one source-language query, the LLM is prompted to produce m paraphrastic sentences that are both relevant to the query and mutually diverse; an MMR-style filter selects the k most relevant, least redundant ones; and the same LLM translates each selected sentence into the target language. Those k self-constructed source-target pairs are then placed in a few-shot prompt before the query. Across English-to-Nepali, Khmer, Pashto, Zulu, and Swahili, DAT improves COMET scores relative to zero-shot translation in most cases, with the clearest gains for Nepali (about 2.8 points on Llama-3.1-8B). The paper also reports a counterintuitive result: high-quality fixed human pairs can backfire, such as a 21.6-point drop for English-to-Khmer on Llama-3.1-70B, which it attributes to missing relevance to the query.","pith_inferences":["Beyond the paper, DAT's source-side paraphrasing suggests a cheap way to create domain-adapted demonstrations: because the generated paraphrases inherit terms from the user query, the method may transfer to technical or dialectal text without retraining.","The backfire result implies that a small static human-curated demonstration set can be worse than no demonstrations when the query distribution drifts; a testable extension is to measure DAT against retrieval from a human pool whose topics overlap with the test set to find the crossover point.","One could test the paper's implicit claim that target-side quality is less important than source-side relevance by generating targets with a stronger, weaker, or deliberately corrupted translation model and checking whether DAT's gains persist.","The accumulation experiment points toward a fully unsupervised data flywheel: generated pairs could eventually fine-tune a dedicated translation model, a direction the paper names only in discussion."],"forward_implications":["DAT gives better COMET scores than zero-shot translation in most tested low-resource language pairs, so an LLM can extract translation gains from its own generation without human-annotated examples.","In the fixed-pair setting, DAT outperforms few-shot prompting with human-curated pairs in most languages because DAT's generated sources are more relevant to the query.","Fixed human pairs that are irrelevant to the query can actively hurt translation, as in the Khmer backfire, so relevance should be checked before any static demonstration pool is used.","Accumulating generated pairs into a pool and retrieving them with R-BM25 improves with more seed inputs, though it does not yet match per-query generation.","Using more unfiltered generated demonstrations (10 rather than 4) can lower quality, so the filtering step is doing real work rather than merely saving tokens."],"supporting_citations":[{"why":"Establishes in-context learning and the demonstration-based prompting paradigm that DAT relies on.","marker":"Brown et al., 2020"},{"why":"Supplies maximal marginal relevance, the selection criterion DAT adapts to filter generated sources.","marker":"Carbonell and Goldstein, 1998"},{"why":"Articulates the relevance and diversity criteria for translation memories that DAT turns into generation instructions.","marker":"Cheng et al., 2022"},{"why":"Argues for on-the-fly, query-coherent in-context examples, motivating DAT's source-side paraphrase generation.","marker":"Sia and Duh, 2023"},{"why":"Provides the R-BM25 retrieval method that DAT uses as a baseline and reuses in the accumulation setting.","marker":"Agrawal et al., 2023"},{"why":"Defines the COMET metric used to score translation quality across all experiments.","marker":"Rei et al., 2022a"},{"why":"Defines reference-free COMET, which the paper uses to estimate the quality of generated example pairs in Table 2.","marker":"Rei et al., 2022b"},{"why":"Supplies the Flores benchmark and the English-to-low-resource test data used throughout.","marker":"Goyal et al., 2022"}],"fun_headline_variants":["LLMs craft their own translation examples from a single query","No human pairs? LLMs generate their own few-shot examples","Relevant, diverse, self-made examples improve low-resource LLM translation","LLMs bootstrap translation examples from a single source sentence","Self-generated translation examples aid low-resource LLM translation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the LLM's own translations of the sentences it generates are reliable enough to serve as correct target examples, and the paper checks this only with an automatic quality estimator rather than human judgment.","fun_headline_variants_meta":{"raw":{"variants":["LLMs craft their own translation examples from a single query","No human pairs? LLMs generate their own few-shot examples","Relevant, diverse, self-made examples improve low-resource LLM translation","LLMs bootstrap translation examples from a single source sentence","Self-generated translation examples aid low-resource LLM translation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001301,"raw_usage":{"total_tokens":5317,"prompt_tokens":964,"completion_tokens":4353,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":4268}},"tokens_in":580,"tokens_out":4353,"duration_ms":26746,"temperature":1.0,"reasoning_tokens":4268,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:03:38.510849+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare DAT against a version where the k target sentences are replaced by deliberately wrong translations of the same generated source sentences: if the two perform alike, then target-side quality, not source-side relevance, is carrying the result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Articulates the relevance and diversity criteria for translation memories that DAT turns into generation instructions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Argues for on-the-fly, query-coherent in-context examples, motivating DAT's source-side paraphrase generation."}],"review_version":1}