{"id":"d73298e9-4add-4672-bc3d-91d1eb2d8c5b","arxiv_id":"2505.19848","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SFT on translated OpenMathInstruct data outperforms directly generated synthetic data for math in African languages, and combining both yields the best AfriMGSM scores.","lead":"This paper tests which data recipe works best for teaching an 8B language model to solve elementary math problems in nine African languages. Translating a large, diverse English math dataset beat generating synthetic problems directly in the target languages, and combining both was best.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Translated-data advantage may be an artifact of GSM8K overlap: OpenMathInstruct's grade-school subset likely contains GSM8K-derived problems, while AfriMGSM is a human translation of GSM8K; the LSH de-duplication in §4.2 is surface-level and cannot exclude cross-lingual contamination.","rationale":"The reader's weakest_assumption is the unvalidated GPT-4o judge, which is a genuine weakness but applies to all rows roughly equally; it would require a judge bias correlated with training-data type to alter the central ranking. The contamination concern is more specific and more damaging: the winning condition (translated OpenMathInstruct) is exactly the one most likely to contain near-duplicates of AfriMGSM, because AfriMGSM is a human-translated GSM8K and OpenMathInstruct is derived from GSM8K-like grade-school problems. The paper's LSH-based de-duplication, described in §4.2, is designed for near-duplicate text in the same language and cannot catch translated or paraphrased overlaps, so it does not rule out this confound. Even with a perfectly accurate judge, the 51.4 vs 39.4 comparison would be invalid if the translated model memorized test-set answers. A second, related confound is that the comparison is not strictly 'translation vs synthesis': the source datasets differ in diversity, difficulty, and provenance, so the causal attribution is already weak. The proposed concrete test—back-translation overlap analysis plus evaluation on a non-GSM8K African-math benchmark—would settle whether the advantage survives intact. Until that check is run, the paper's headline result should remain conditional; the reader's verdict stays CONDITIONAL, but for this more fundamental reason rather than the judge-reliability concern alone.","tokens_in":17889,"tokens_out":6168,"duration_ms":63295,"concrete_test":"Take the exact 30k OpenMathInstruct prompt-response pairs used for Row 11 (or a random sample of 1,000) and translate the prompts back to English with the same GPT-4o translation service. Compare these to the GSM8K training set using exact match and embedding-based nearest-neighbor search (e.g., a multilingual sentence encoder; flag pairs above a calibrated similarity threshold). Then, separately, evaluate the Row 11 and Row 8 models on a new African-language math test set whose source problems are not GSM8K-derived (e.g., translate grade-school problems from a different corpus into the same five languages). If a substantial fraction of training prompts are near-duplicates of GSM8K and the performance gap narrows or reverses on the non-GSM8K test set, the headline claim is an artifact of contamination; if the gap persists on clean data, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central comparison (Table 2, rows 8 vs 11) concludes that translating a diverse English instruction dataset (OpenMathInstruct V2) beats directly generating synthetic data (AfriPersona-Instruct) at a 30k sample budget. Evaluation is on AfriMGSM, which the paper states is a human-translated version of GSM8K (§4.2). OpenMathInstruct V2 is an instruction-tuning math dataset generated by Mixtral, and its grade-school subset is very likely to contain problems derived from GSM8K: the cited reference is to OpenMathInstruct-1, which was built from GSM8K and MATH, and V2 is a follow-up not described in detail. AfriPersona-Instruct, in contrast, is generated from personas and not from GSM8K. The authors report de-duplicating training and test data with LSH (§4.2), but LSH on token shingles detects only surface-level near-duplicates; it cannot identify cross-lingual semantic duplicates, especially after GPT-4o translation changes the surface form. If the translated training data contains paraphrased or translated versions of GSM8K problems, the model trained on it could be memorizing answers rather than learning to reason, inflating the 51.4 vs 39.4 difference and undermining the claim that translated data is superior. A perfect judge would still report these inflated scores. This is therefore more load-bearing than the unvalidated-judge concern: a judge bias would have to be systematic and correlated with training-data type to flip the ranking, whereas contamination is structurally coupled to the winning condition.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper systematically compares strategies for adapting Llama 3.1 8B models to mathematical reasoning in nine African languages, focusing on: (i) translated versus directly generated synthetic training data, (ii) prompt-masking during SFT, (iii) scaling the amount of synthetic data, (iv) monolingual versus multilingual fine-tuning, and (v) continual pretraining. The main empirical claims are that translated OpenMathInstruct V2 data outperforms directly generated AfriPersona-Instruct at the same 30k-sample budget (51.4 vs 39.4 on AfriMGSM), that combining both data types gives the best results (57.7), that prompt masking hurts performance slightly, and that multilingual fine-tuning beats monolingual experts. The authors release AfriPersonaHub and AfriPersona-Instruct datasets. Evaluation uses GPT-4o as an LLM judge on AfriMGSM, a human-translated version of GSM8K.","tokens_in":18225,"tokens_out":6146,"duration_ms":61109,"significance":"If the findings are robust, the paper provides actionable guidance for building multilingual math reasoning models in low-resource African languages: translate diverse English instruction data rather than only generating synthetic data in the target language, and combine both sources for best results. The release of the persona hub and instruction dataset is a concrete contribution to the community. The comparative framing across data types, training stages, and model configurations is valuable and goes beyond typical single-pipeline studies. However, the central claims currently rest on an unvalidated LLM judge and on a training/test overlap risk that are both load-bearing; the paper needs additional validation and decontamination evidence before the conclusions can be accepted.","major_comments":[{"comment":"The evaluation relies entirely on GPT-4o as an LLM judge, and Appendix A states only that the prompt was 'calibrated using a couple of samples from different languages.' No human-validated accuracy, agreement rate, or per-language reliability of the judge is reported. Since every accuracy in Tables 2 and 3 and all cross-condition comparisons depend on this judge, please provide a human-validated sample (e.g., 50–100 responses per language, or at least across the five trained languages) and report judge accuracy per language, or compare against exact-match extraction on a subset. Without this, the headline differences (e.g., 51.4 vs 39.4) could be artifacts of judge bias rather than genuine capability differences.","section":"§4.2, Appendix A, Tables 2–3"},{"comment":"The central comparison between translated OpenMathInstruct and AfriPersona-Instruct is at risk of training/test contamination. OpenMathInstruct V2 is generated by Mixtral from seed problems that, per the cited OpenMathInstruct-1 paper, include GSM8K; AfriMGSM is a human translation of GSM8K. The LSH de-duplication described in §4.2 operates on surface shingles and cannot detect cross-lingual or paraphrased duplicates. If the translated training data contains translated versions of GSM8K problems, the 51.4 vs 39.4 gap may reflect memorization rather than superior reasoning. Please quantify the overlap between translated training prompts and AfriMGSM (e.g., embedding-based retrieval), or retrain on a translated dataset that provably excludes GSM8K-derived items, and show that the ranking persists.","section":"§3.2 and §4.2, Table 2 rows 8 vs 11"},{"comment":"All experiments are reported from a single training run with no variance information. Some conclusions rest on small differences: prompt masking (rows 8 vs 9: 39.4 vs 38.7 overall; 31.8 vs 29.7 African-only) and scaling from 10k to 20k samples (rows 6 vs 7: 34.9 vs 34.7 overall). These gaps are within plausible seed-level noise. Report results over at least three seeds for the key comparisons (translated vs synthetic, masking, scaling) and provide confidence intervals or significance tests.","section":"§4.1, Table 2 rows 6–9"},{"comment":"The continual pretraining ablation compares Llama 3.1 8B Base with Lugha-Llama 8B, a different model with its own pretraining corpus (WURA + OpenWebMath) and training procedure. This design does not isolate the effect of continual pretraining; any performance difference could stem from the data mix, training order, or other hyperparameters. The conclusion that continual pretraining 'may not directly benefit multilingual math reasoning' is therefore not supported. A controlled comparison would require continually pretraining Llama 3.1 Base on the same data, or re-framing the result as a property of the specific Lugha-Llama model rather than a general conclusion about continual pretraining.","section":"§4.3.5, Table 2 rows 14–15"}],"minor_comments":[{"comment":"There are typos: 'todyas' should be 'today's' and 'Our experiments focuses' should be 'Our experiments focus.'","section":"Abstract, §1"},{"comment":"The row references in the text are off by one: 'Row 11' for translated BigMath should be Row 10, and 'Row 12' for translated OpenMathInstruct should be Row 11. Please correct these references.","section":"§5.3"},{"comment":"Please specify the composition of Row 12 'All Data 60,000' — presumably 30k translated OpenMathInstruct plus 30k AfriPersona-Instruct, but the text should state this explicitly.","section":"Table 2"},{"comment":"The monolingual SFT row does not report the number of training samples per language. Please state the sample size and whether it matches the 30k multilingual budget in total or per language.","section":"Table 2, Row 5"},{"comment":"Please describe the LSH de-duplication details for the training/test split: what language(s) the shingles were computed on, the shingle size and similarity threshold, and whether it was applied cross-lingually or only within the same language.","section":"§4.2"},{"comment":"The manual audit of AfriPersona-Instruct is qualitative. Please provide quantitative counts of the identified error types (unnatural phrasing, lexical inaccuracies, ambiguity) and the number of prompts affected.","section":"§5.1"},{"comment":"The paper uses OpenMathInstruct V2 but cites the OpenMathInstruct-1 paper (Toshniwal et al., 2024). Please clarify the relationship between the two and provide a citation or description for V2, especially regarding whether GSM8K was used as a seed source.","section":"§3.2, References"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and practical question, and the released datasets are potentially valuable. The main risk is that the headline comparison (translated vs. synthetic) is threatened by possible GSM8K overlap between OpenMathInstruct and AfriMGSM; this needs to be resolved with explicit decontamination analysis. Also, the entirely LLM-based evaluation without human validation is a concern for a journal audience; I would encourage the editor to require either human-validated judge accuracy or exact-match corroboration before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the paper. The most useful things here are the two released artifacts — AfriPersonaHub and AfriPersona-Instruct — and a set of controlled ablations that a low-resource practitioner can actually use: prompt masking hurt, multilingual fine-tuning beat monolingual experts, 10k→30k scaling helped, and continual pretraining on Lugha-Llama actually made math worse. Those findings don't depend on the paper's most questionable comparison.\n\nThe headline result, that translated OpenMathInstruct beats persona-based synthetic data (51.4 vs 39.4 at 30k samples), is suspect for a concrete reason the paper doesn't address: OpenMathInstruct-1 was built by generating solutions for GSM8K problems, and V2 likely inherits that grade-school subset; AfriMGSM is a human translation of GSM8K. So the translated training data likely overlaps with the test set in the original English source. The LSH deduplication described in §4.2 only catches surface-level near-duplicates; it can't remove a GPT-4o translation of a GSM8K problem that a human translator rendered differently in Yoruba. A model trained on that data can memorize answers, and a perfect judge would still report inflated accuracy. This is more load-bearing than the unvalidated LLM judge, which the reader flagged. Judge bias would have to correlate with training-data type to flip the ranking; contamination is structurally built into the winning condition.\n\nOther soft spots are real but smaller: no error bars and one run per condition; the continual pretraining comparison is confounded because Lugha-Llama differs from Llama Base in pretraining data and procedure; the manual audit in §5.1 found the synthetic data has quality problems, which the authors honestly acknowledge.\n\nNone of this kills the paper. The datasets are reusable, the ablations on masking and multilingual training are still informative, and the limitations section is honest. But the central claim needs another experiment — evaluate on a math benchmark not derived from GSM8K, or filter the training set for GSM8K-derived prompts and re-run the comparison. Without that, I'd be very cautious about citing the 51.4 vs 39.4 result as evidence that translated data is better.\n\nI'd send it to review, but with a note that the contamination issue must be addressed in the revision.","headline":"Useful datasets and ablations, but the headline 'translated data wins' is likely inflated by GSM8K contamination shared with the evaluation benchmark.","tokens_in":18739,"tokens_out":4841,"would_cite":true,"duration_ms":50278,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For mathematical reasoning in African languages, translating a diverse English instruction dataset outperforms directly generating synthetic native-language data at the same budget, and the two together perform best.","keywords":["African languages","mathematical reasoning","low-resource languages","synthetic data generation","machine translation","instruction tuning","supervised fine-tuning","LLM-as-judge evaluation"],"falsifier":"Have fluent human speakers of the target languages score a few hundred model responses from the translated-data and synthetic-data models and compare their verdicts with the GPT-4o judge's verdicts; if agreement is low in any language, or if extracting the final numeric answer exactly reverses the translated-versus-synthetic ordering on the same 30,000-sample budget, then the paper's central claim does not survive.","tokens_in":17707,"feed_emoji":"🧮","tokens_out":13673,"duration_ms":117213,"temperature":0.7,"pith_summary":"This paper asks a practical question: when you extend a modern LLM to African languages, which data strategy actually teaches mathematical reasoning? Fine-tuning Llama 3.1 8B on machine-translated OpenMathInstruct reaches an average of 51.4 on AfriMGSM, a human-translated 16-language math test set, beating the same 30,000-sample budget of directly generated native-language synthetic data (39.4); training on both together does best overall (57.7, and 52.3 on African languages alone). The authors further find that source-dataset breadth decides whether translation pays off, that a single multilingual model beats per-language experts, and that retaining prompt tokens in the loss signal helps rather than hurts. A reader should care because the winning recipes are cheap and concrete: one 8B model, modest data, and no frontier-scale compute.","feed_headline":"Translated math data beats direct generation for African languages","feed_subtitle":"Combining both lifts an 8B Llama model to GPT-4-level math accuracy in several African languages.","key_machinery":"The argument is carried by a controlled experimental setup rather than a single mathematical object. Three data pipelines are placed on equal footing: the persona-based synthetic pipeline that builds AfriPersona-Instruct from text-to-persona and persona-to-persona generation seeded by Wikipedia and WURA articles; GPT-4o translation of two English math datasets, BigMath and OpenMathInstruct V2, into nine target languages; and an LLM-as-judge evaluation in which GPT-4o, an automatic judge, decides whether a generated response matches the golden answer, replacing exact-match scoring that the authors find unreliable in multilingual settings. The load-bearing comparisons hold the model (Llama 3.1 8B instruct or base), the generating model (GPT-4o), and the training recipe (10,000–30,000 samples, two epochs, batch size 128, learning rate $5\\times10^{-5}$) fixed, so observed differences are attributable to data type, training stage, or configuration. The mechanism identified is source-data diversity: the breadth of OpenMathInstruct is what lets translation outperform direct generation, while the narrow BigMath translation fails.","core_discovery":"On the paper's own terms, the core discovery is that the type and diversity of training data, not its origin, governs how well an LLM learns multilingual math reasoning. With the base model, the translation model, the sample budget, and the training schedule all held fixed, a model fine-tuned on 30,000 machine-translated samples of OpenMathInstruct V2 reaches 51.4 average accuracy on AfriMGSM, while the same budget of AfriPersona-Instruct—problems generated directly in nine African languages through a persona-driven GPT-4o pipeline—reaches 39.4. Translated BigMath, a narrower, specialized source, drops to 11.2, indicating that source breadth is what makes translation pay off. Combining translated and synthetic data (60,000 samples) yields the best result, 57.7 overall and 52.3 on African languages, with the 8B model approaching or exceeding GPT-4 on Yorùbá, Igbo, and Swahili. The paper further reports that joint multilingual fine-tuning outperforms monolingual experts, that masking prompt tokens during supervised fine-tuning slightly hurts, and that continual pretraining on general African text does not transfer to math.","pith_inferences":["Beyond the paper: the paper's own qualitative audit (Section 5.1) found frequent lexical and grammatical errors in AfriPersona-Instruct, and its limitations section concedes that the synthetic reasoning steps are not formally verified; if data quality is what drives the translated-data advantage, improving generation fidelity (human correction or grammar feedback) could close the gap more cheaply ","Beyond the paper: because every comparative number flows through a GPT-4o judge calibrated on only a couple of samples, the ordering should be treated as provisional until human speakers confirm the judge's verdicts in each language; re-running the 30,000-sample comparison with exact-match extraction of final numeric answers is a cheap robustness check.","Beyond the paper: the recipe of translating a diverse instruction set and then adding native synthetic data likely transfers to other low-resource language groups and other reasoning domains, but the paper demonstrates it only for mathematics in nine African languages, so that extrapolation is a hypothesis.","Beyond the paper: the negative continual-pretraining result points to one concrete next experiment the authors hint at—pretraining on math-focused African-language text instead of general news and literary text—which would test whether domain alignment rather than language adaptation is what math reasoning needs."],"forward_implications":["Teams building math-capable models for African languages should lead with machine translation of a broad, diverse English instruction dataset; native-language synthetic data is a complement rather than a substitute, and the two together give the largest gains.","Source breadth is the deciding constraint: a translated dataset is only as useful as the diversity of the original, so narrow specialized collections may not justify the translation budget.","A single multilingual model fine-tuned on several African languages outperforms per-language experts and also generalizes to unseen African languages, making joint multilingual training the deployment strategy of choice.","The common practice of masking prompt tokens in the supervised fine-tuning loss should be reconsidered for low-resource multilingual settings, where computing loss over the full input gave higher math accuracy.","With only 30,000 translated samples, an open 8B model approaches or beats GPT-4 on several African languages, showing that frontier-level math reasoning in low-resource languages does not require frontier-scale resources."],"supporting_citations":[{"why":"Supplies the text-to-persona and persona-to-persona generation pipeline that produces the personas behind AfriPersona-Instruct.","marker":"(Ge et al., 2024)"},{"why":"Provides OpenMathInstruct V2, the diverse instruction dataset whose translated 30,000-sample budget yields the best single-data result.","marker":"(Toshniwal et al., 2024)"},{"why":"Provides BigMath, the narrow specialized dataset whose translated version scores only 11.2, establishing source breadth as the deciding factor.","marker":"(Albalak et al., 2025)"},{"why":"Contributes AfriMGSM, the human-translated math benchmark used for every evaluation in the paper.","marker":"(Adelani et al., 2025)"},{"why":"GSM8K is the grade-school math source that AfriMGSM translates into African languages.","marker":"(Cobbe et al., 2021)"},{"why":"Motivates the LLM-as-judge evaluation framework the paper adopts for scoring multilingual math responses.","marker":"(Stephan et al., 2025)"},{"why":"Provides Lugha-Llama, the continually pretrained African-language model used in the continual-pretraining ablation.","marker":"(Buzaaba et al., 2025)"},{"why":"Defines the Llama 3.1 8B base and instruct models that all experiments fine-tune and compare.","marker":"(Grattafiori et al., 2024)"}],"fun_headline_variants":["Translated data beats synthetic for African math LLMs","Combining translated and synthetic data lifts math reasoning","Translated math data wins for African languages in LLMs","Best results from mixing translated and generated math data","Why translated data outperforms generated for African math"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every accuracy number and ranking in the paper is produced by GPT-4o acting as an automatic judge, calibrated on only a couple of samples, with no human validation of judge accuracy across the nine languages and varied response styles; if that judge is biased or unreliable, the reported scores and the ranking of data strategies could change.","fun_headline_variants_meta":{"raw":{"variants":["Translated data beats synthetic for African math LLMs","Combining translated and synthetic data lifts math reasoning","Translated math data wins for African languages in LLMs","Best results from mixing translated and generated math data","Why translated data outperforms generated for African math"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000328,"raw_usage":{"total_tokens":1827,"prompt_tokens":934,"completion_tokens":893,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":819}},"tokens_in":550,"tokens_out":893,"duration_ms":7016,"temperature":1.0,"reasoning_tokens":819,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:04:52.564774+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have fluent human speakers of the target languages score a few hundred model responses from the translated-data and synthetic-data models and compare their verdicts with the GPT-4o judge's verdicts; if agreement is low in any language, or if extracting the final numeric answer exactly reverses the translated-versus-synthetic ordering on the same 30,000-sample budget, then the paper's central claim does not survive.","supporting_citations":[],"review_version":1}