{"id":"335f51b6-4fec-4b41-bdbb-4fa277a98c81","arxiv_id":"2508.08680","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"TopXGen generates topic-diverse synthetic parallel data by prompting an LLM to write in low-resource languages and backtranslating to English, improving MT in ICL and fine-tuning across ten languages.","lead":"TopXGen is a pipeline that asks a large language model to write new texts directly in ten low-resource languages, then backtranslates them to English to create parallel training data. Tests on FLORES-200 show the synthetic data improves machine translation into those languages, for both fine-tuning and in-context learning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Generator fluency in each LRL is the load-bearing premise, but it is only indirectly checked; no human evaluation of the generated target text is provided.","rationale":"The paper has real strengths: controlled comparisons against Self-Instruct and KNN-Instruct, contamination filtering, paired bootstrap significance testing, ablations over generator, temperature, and back-translator, out-of-domain evaluations on NTREX and TICO-19, and released code and outputs. The residual soft spot is not the execution of the experiments but the unverified premise at the input side: the generator's fluency in ten low-resource languages. The reader's weakest assumption already identifies this, and the reader's conditional verdict appropriately reflects that the central claim is stronger than the evidence for this premise. I considered other possible concerns, such as the ICL comparison confounding pool size with data quality and the absence of a standard backtranslation baseline, but these are secondary: they affect the practical advantage of TOPXGEN rather than the validity of the core fine-tuning result. The proposed native-speaker evaluation would settle the generator-fluency concern directly; until it is run, the conditional verdict is the right one.","tokens_in":37968,"tokens_out":10192,"duration_ms":115725,"concrete_test":"Have two native speakers per language rate a random sample of 100 TOPXGEN target sentences per language (1,000 sentences total) for fluency, grammaticality, and script consistency on a 1-5 scale, and also rate 50 translations produced by the fine-tuned LLaMA-3-8B model per language. If median ratings are at least 4/5 for all ten languages, the generator assumption holds and the central claim is not threatened. If one or more languages score below this threshold while downstream BLEU and MetricX remain high, the reported gains are not attributable to high-quality target text, and the claim should be restricted to languages with native-speaker-verified generation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that TOPXGEN produces 'high quality and topic-diverse' LRL text that, after back-translation, improves MT (Abstract; Section 3). All target-side supervision in fine-tuning and ICL ultimately comes from Gemma-3-27B-It generations, so the assumption that this model writes fluent, grammatical, script-consistent Hausa, Igbo, Xhosa, Kinyarwanda, and the other LRLs is load-bearing. The paper's evidence for this premise is indirect: fastText language identification (Section 3), MetricX-24 QE statistics (Table 14), LLM topic-alignment judgments (Table 15), and the Google-Translate cross-check in Section B.8. LID only checks language labels; QE is a learned estimator trained mainly on high-resource judgments; the topic-alignment test asks whether content matches a topic, not whether the LRL is natural; and the B.8 comparison shows only that two MT systems produce similar English from the generated LRL, which is weak evidence about target-side naturalness. The failure mode is not hypothetical: the temperature ablation (Section 6) shows generation degrades into nonsensical mixed-script output at T > 1.2, so quality is fragile and generator-dependent. If the generator's LRL output is ungrammatical or mixed-script for some of the ten languages, a fine-tuned model can still improve over a no-data baseline by learning alignments from the English side, so the reported MT gains do not cleanly establish the data-quality premise. The authors acknowledge this in the Limitations section. In short, the experimental claims are well controlled, but the 'high quality target-side text' premise that the pipeline is built on is not directly verified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents TopXGen, a pipeline that prompts a multilingual LLM (Gemma-3-27B-It) to generate topic-guided paragraphs directly in ten low-resource languages, then backtranslates them into English with NLLB-200-3.3B to obtain synthetic parallel sentence pairs. The authors evaluate the resulting data by fine-tuning LLaMA-2-7B and LLaMA-3-8B and by performing 5-shot in-context learning with LLaMA-3.1-8B-It, using FLORES-200, NTREX-128, and TICO-19 with BLEU, chrF++, MetricX-24, and XCOMET-XL. They compare against Self-Instruct and KNN-Instruct data under the same training budget and against small professionally translated datasets, and they run ablations on the generator, topic-list size, temperature, back-translator, and iterative self-improvement. The central claim is that TopXGen data improves translation into low-resource languages, with unidirectional fine-tuned models approaching or exceeding the generator's zero-shot quality, and that retrieval from TopXGen gives better MetricX-24 scores than retrieval from the in-domain FLORES dev set.","tokens_in":38136,"tokens_out":10960,"duration_ms":105570,"significance":"If the results hold, the paper makes a practical contribution: a scalable and relatively low-cost recipe for producing synthetic parallel data for low-resource machine translation without relying on existing target-side monolingual corpora. The evaluation is externally grounded in human-translated references, includes contamination filtering and significance testing, reports multiple metrics and benchmarks, and provides valuable ablations. The code and outputs are released, which supports reproducibility. The main caveat is that the quality of the generated LRL text is inferred from automatic and indirect signals rather than from human judgments, and the headline comparison to prior data-generation methods covers only two languages and one base model.","major_comments":[{"comment":"The central premise that Gemma-3-27B-It produces fluent, grammatical, script-consistent text in all ten LRLs is not directly validated. The evidence is indirect: fastText LID checks only the language label; MetricX-24 QE is a learned estimator; the LLM topic-alignment judgments in Table 15 check topic adherence, not target-side naturalness; and the Google Translate cross-check in Table 17 checks semantic consistency with the backtranslated English. The temperature ablation in Section 6 shows that generation degrades sharply above T=1.2, confirming that output quality is fragile and generator-dependent. Because the abstract and introduction claim 'high quality and natural-sounding' target text, I request a human (or native-speaker) evaluation of a sample of generated sentences per language, together with the fastText LID retention rates, so that the data-quality premise is supported directly rather than only by downstream MT gains.","section":"Section 3 and Limitations"},{"comment":"The claim that TopXGen 'consistently outperforms' Self-Instruct and KNN-Instruct is supported only for two languages (Sundanese and Somali) and one base model (LLaMA-2-7B) at 20K sentence pairs. This is a narrow basis for the general statement in the abstract. I recommend either adding at least one additional language/base-model setting for the comparison, or explicitly qualifying the claim to the tested directions.","section":"Section 5.3 (Comparison to Self-Instruct and KNN-Instruct)"},{"comment":"The statement that unidirectional LLaMA-3-8B fine-tuning 'outperforms Gemma-2-27B-It and LLaMA-3.1-70B-It' is not supported by Table 2 for every direction and metric: for example, Basque BLEU is 25.64 for the fine-tuned model versus 26.06 for LLaMA-3.1-70B-It, and Swahili BLEU is 33.49 versus 35.99 for Gemma-2-27B-It, with MetricX also worse in the Swahili case. Please report averages across the ten directions or otherwise qualify the claim by metric and language so that the conclusion reflects the actual pattern of results.","section":"Section 5.1 (Fine-tuning results)"},{"comment":"The paper does not compare TopXGen against standard backtranslation applied to existing target-side monolingual corpora (for example, CC-100, OSCAR, or language-specific news corpora). Since TopXGen is precisely a method for creating target-side text to backtranslate, such a baseline is needed to determine whether the gains come from topic-guided generation and diversity or simply from having additional target-side data. Please add this comparison or clearly justify its absence for the ten selected languages.","section":"Section 3.1 and Related Work"}],"minor_comments":[{"comment":"The phrase 'as they cannot follows instructions' should read 'as they cannot follow instructions'.","section":"Section 4.1"},{"comment":"The text says 'We report the results obtained on NTREX-19 in Table 11,' but the benchmark is NTREX-128; please correct the identifier.","section":"Appendix B.4"},{"comment":"The sentence 'we observe that Gemma-3-27B-It's generations in share lexical overlap' appears to be missing a language specification; please rephrase.","section":"Appendix B.8"},{"comment":"The phrase 'monolingual data scrapped from the internet' should be 'scraped from the internet'.","section":"Limitations"},{"comment":"The tables rely on underline markers to indicate statistical equivalence to the best score, but these markers are not visible in the submitted text; please ensure they are rendered in the final version.","section":"Tables 3, 5, and 6"},{"comment":"The paper alternates between 'SMOLSENT' and 'SMOL'; please unify the notation.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-executed and the main results are credible, but my recommendation hinges on whether the authors can provide more direct evidence of target-side text quality and strengthen the comparison to standard backtranslation. The Self-Instruct/KNN-Instruct comparison is quite narrow, but I would not reject the paper over it; the claims should simply be qualified. No concerns about novelty or scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"TopXGen is worth a serious referee. The core idea is simple and sensible: instead of needing existing LRL monolingual corpora for backtranslation, prompt a multilingual LLM to generate topic-diverse LRL text using seed sentences, then backtranslate to English with NLLB. This extends Marie and Fujita's idea of LLM-synthesized monolingual data, but removes the need for domain fine-tuning and works directly in ten LRLs. That is a practically useful advance for low-resource MT.\n\nWhat the paper does well: the experimental setup is careful. Evaluation is on FLORES, NTREX, and TICO with BLEU and MetricX, paired bootstrap significance tests, contamination filtering toward evaluation sets, and several ablations (generator, backtranslator, temperature, number of topics, self-improvement). The comparison against Self-Instruct and KNN-Instruct under the same budget is clean and consistently favors TopXGen. The ICL result, where retrieval from TopXGen beats retrieval from the in-domain FLORES dev set on MetricX, is a strong point. Code and outputs are released.\n\nThe soft spots are real but not fatal. The load-bearing premise is that Gemma-3-27B-It writes fluent, script-correct Hausa, Igbo, Xhosa, etc. The authors never get human judgments of the generated LRL text. They rely on fastText LID, MetricX QE, LLM topic-alignment, and a Google Translate cross-check. None of these directly verify naturalness. The B.8 cross-check only shows that two MT systems produce similar English from the generated LRL, which is weak evidence about target-side quality. The temperature ablation shows quality collapses above T=1.2, so the pipeline is fragile to generator choice. However, the authors explicitly acknowledge the generator requirement in Limitations, and the downstream MT gains are the real payoff. If the generated text were garbage, you would not expect the fine-tuned LLaMA models to beat strong baselines and approach the generator's own score. So the central comparative claim - TopXGen beats Self-Instruct and KNN-Instruct under identical budgets - holds up.\n\nOne smaller complaint: the abstract and conclusion say the datasets are 'comparable to professionally written text', which overshoots. Figure 3 shows TopXGen does not consistently beat small professional datasets at equal scale; it wins because it can be 100x larger. That is a legitimate advantage, but the claim should be phrased as 'scale compensates for quality differences', not 'comparable quality'.\n\nThe citation pattern is solid. They properly credit Marie and Fujita as the closest prior work, and they are not hiding the relationship.\n\nWho should read it: anyone working on data synthesis for low-resource MT, and anyone evaluating synthetic data methods. It deserves peer review. I would accept it as a serious submission, with a request for either human evaluation of generated target text or a narrowing of the quality claim.","headline":"A well-controlled empirical study showing topic-guided LLM generation plus backtranslation improves low-resource MT, with the main caveat that target-side quality is only indirectly verified.","tokens_in":129,"tokens_out":1992,"would_cite":true,"duration_ms":31846,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM that translates poorly into low-resource languages can still write fluent text in them, and backtranslating that text into English yields parallel data that makes smaller models translate well into those languages.","keywords":["low-resource machine translation","synthetic parallel data","topic-guided generation","backtranslation","in-context learning","fine-tuning","multilingual LLMs","FLORES-200"],"falsifier":"Run TOPXGEN on a language where the generator's target-side writing is known to be weak, or ask native speakers to rate a sample of generated sentences for fluency and correctness, and check whether the fine-tuned student still beats both a zero-shot strong model and a forward-translation baseline; if fluency breaks, the method should collapse toward that baseline.","tokens_in":37616,"feed_emoji":"🌐","tokens_out":7575,"duration_ms":71707,"temperature":0.7,"pith_summary":"The paper tries to establish that the bottleneck for low-resource machine translation is not the translator but the parallel data: a strong multilingual LLM, asked to write paragraphs about arbitrary topics directly in ten low-resource languages, produces target-side text that can be backtranslated into English to form synthetic sentence-level parallel data. Fine-tuning small decoder models on this data moves them from near-random translation to scores that rival or beat far larger instruction-tuned models, and retrieving examples from the synthetic pool for in-context learning beats retrieving from the in-domain human-written development set on the reference-free quality metric. This matters because it suggests that high-quality low-resource MT training data can be manufactured cheaply, without parallel corpora or expert translators, wherever the generator has basic competence in the target language.","feed_headline":"Backtranslated LLM text lets small models rival a 27B generator","feed_subtitle":"Topic-diverse synthetic data for ten low-resource languages lifts 8B fine-tuned models near the performance of a 27B generator.","key_machinery":"The carrier of the argument is a three-part generation prompt: a randomly drawn topic from a 67,573-entry Wikipedia topic list (for content diversity), 240 seed paragraphs from XQuAD in high-resource languages (for expected length and format via cross-lingual in-context learning), and FLORES-dev seed sentences in the target low-resource language (for script, register, and sentence shape). Paragraphs written by Gemma-3-27B-It are sentence-split, run through fastText language identification, filtered for overlap against FLORES, NTREX, TICO-19, and the XQuAD seeds, and then backtranslated into English with NLLB-200-3.3B. Topic diversity and a temperature of 1.0 are the tuning knobs that produce the strongest student models, and the same synthetic pool serves both fine-tuning and retrieval-based few-shot prompting.","core_discovery":"On its own terms, the discovery is that target-side quality is the lever: instead of translating English text into a low-resource language (forward translation) or mining scarce monolingual corpora, TOPXGEN prompts an LLM to produce original paragraphs in the target language, guided by a random topic, by high-resource-language seed paragraphs, and by a handful of target-language seed sentences. These paragraphs are split into sentences, filtered for language, deduplicated, and backtranslated into English with NLLB-200-3.3B. The resulting 1.05M English-to-low-resource-language sentence pairs, about 50k to 150k per language, let a fine-tuned LLaMA-3-8B outperform Gemma-2-27B-It and LLaMA-3.1-70B-It in the tested English-to-LRL directions and come close to the Gemma-3-27B-It generator itself, while a 5-shot in-context learner retrieving from TOPXGEN beats the same retrieval from the FLORES dev set on MetricX-24. The paper also shows the pipeline beats Self-Instruct and KNN-Instruct under identical data budgets, and that data scale plus topic diversity, not just quality, is what drives the gains.","pith_inferences":["An extension the paper does not test is pushing TOPXGEN into languages where the generator is clearly weak, since the stated limitation predicts the method will degrade toward forward translation exactly when target-side fluency fails.","The ablation that swaps back-translators (NLLB vs generator vs student) suggests the English side is not the bottleneck; a testable consequence is that investing in better low-resource-language generation, not better back-translation, should drive further gains.","Because more topics helped even when the topic list was cut from 67,573 to 509, the topic list itself acts as a diversity budget; curating topics per language or per domain could be a cheap way to steer the data.","A head-to-head against human-written monolingual corpora of equal size would isolate whether topic-diverse generation or backtranslation-into-a-high-resource-language is the active ingredient."],"forward_implications":["Fine-tuning an 8B model on TOPXGEN data gives translations into ten low-resource languages that match or beat instruction-tuned models several times its size, with about three hours of training on a single H100.","Retrieving five in-context examples from the synthetic pool outperforms retrieving from the in-domain FLORES dev set on MetricX-24, so the generated data can replace human-written development sets as an example bank.","The generator can be swapped: using GPT-4o-mini to generate the target text yields stronger students than Gemma-3-27B-It, and using either the generator or the fine-tuned student as the backtranslator works almost as well as NLLB.","More topics mean better student models, so the pipeline's value should grow with topic-list coverage rather than with sentence count alone.","Scaling from roughly 900 professionally translated sentences to the roughly 100-times-larger synthetic set outweighs the quality edge of professional data on the tested directions."],"supporting_citations":[{"why":"Defines back-translation, the baseline technique TOPXGEN replaces because it requires existing target-side text.","marker":"Sennrich et al., 2016"},{"why":"Self-Instruct is the instruction-generation baseline that TOPXGEN is compared against under identical data budgets.","marker":"Wang et al., 2023"},{"why":"KNN-Instruct is the second comparison baseline, replacing random selection with nearest-neighbour retrieval.","marker":"Kou et al., 2024"},{"why":"Supplies the 67,573 Wikipedia topics used to guide paragraph generation and enforce diversity.","marker":"Ziadé, 2023"},{"why":"Supplies the 240 XQuAD paragraphs in eleven high-resource languages used as cross-lingual seed examples in the prompt.","marker":"Artetxe et al., 2020"},{"why":"Provides the FLORES dev set: source of target-language seed sentences, the in-domain comparison pool for retrieval, and the evaluation benchmark.","marker":"Goyal et al., 2022"},{"why":"Provides both the FLORES-200 resource and the NLLB-200-3.3B model used as the back-translator.","marker":"Costa-jussà et al., 2022"},{"why":"Gemma-3-27B-It is the generator LLM whose multilinguality the whole pipeline exploits.","marker":"Team et al., 2025"},{"why":"MetricX-24 is the primary reference-based quality metric used to compare translations across all experiments.","marker":"Juraska et al., 2024"}],"fun_headline_variants":["Small MT models rival 27B with topic-diverse synthetic data","LLM-generated target text lifts low-resource translation","Backtranslated LLM paragraphs boost small models near 27B","Topic-diverse synthetic data closes low-resource MT gap","LLM-made target corpora push 8B models past 27B"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes the generator LLM can already write fluent, grammatical, script-correct text in each target low-resource language, and that fluency is verified only indirectly by language identification, quality estimates, and downstream translation scores rather than by human judgement.","fun_headline_variants_meta":{"raw":{"variants":["Small MT models rival 27B with topic-diverse synthetic data","LLM-generated target text lifts low-resource translation","Backtranslated LLM paragraphs boost small models near 27B","Topic-diverse synthetic data closes low-resource MT gap","LLM-made target corpora push 8B models past 27B"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1401,"prompt_tokens":1103,"completion_tokens":298,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":719,"completion_tokens_details":{"reasoning_tokens":210}},"tokens_in":719,"tokens_out":298,"duration_ms":3653,"temperature":1.0,"reasoning_tokens":210,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:33:31.671017+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TOPXGEN on a language where the generator's target-side writing is known to be weak, or ask native speakers to rate a sample of generated sentences for fluency and correctness, and check whether the fine-tuned student still beats both a zero-shot strong model and a forward-translation baseline; if fluency breaks, the method should collapse toward that baseline.","supporting_citations":[],"review_version":2}