{"id":"4f5dad68-d2b4-4cc5-ad93-d2773571b010","arxiv_id":"2412.03253","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Rewriting Arabic pre-training data with aligned LLM workers improves safety, helpfulness, and Arabic benchmark scores in the released LLaMA3-Tamed models.","lead":"This paper proposes 'native alignment', rewriting web-crawled Arabic pre-training data with LLMs so models are safer and more helpful before instruction tuning or RLHF. The authors release Arabic base models, LLaMA3-Tamed-8B and 70B, and report benchmark gains over vanilla Llama-3.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Native-alignment effect is not isolated: LLaMA3-Tamed-8B/70B received ~100B tokens of continued pretraining including ~38B unaligned ArabicText2022 plus 10B rewritten Arabic tokens, while the Table 2 Llama3-8B/70B baselines received none; no equal-budget unaligned Arabic control is reported.","rationale":"The reader's weakest_assumption is the load-bearing concern. I considered two other candidates: the GPT-4-as-judge loop on BeaverTails, and the Appendix D DPO comparison. The GPT-4 judge issue affects only the harmlessness/helpfulness numbers, and the authors do mitigate position bias; it does not explain the multiple-choice benchmark gains in Table 2. The Appendix D comparison is explicitly labelled unfair by the authors and is not used to support the headline SOTA claim. In contrast, the missing equal-budget control affects every headline result: the Tamed models in Table 2 differ from their baselines not just in data alignment but in about 100B tokens of continued pretraining and a large quantity of additional Arabic data. If a control with unaligned Arabic tokens at the same budget reproduces the gains, the central contribution reduces to 'continued pretraining on more Arabic data,' and the native-alignment mechanism is unestablished. The 12B ablation in Section 4.2 is a partial same-budget check, but it covers only BeaverTails and still changes data source or token count across conditions, so it does not close the gap. The proposed 8B control directly tests the mechanism at the scale where the paper already runs ablations; a positive result would salvage the causal claim, while a null result would require the headline to be reframed. Thus the verdict should remain CONDITIONAL as the reader stated.","tokens_in":16796,"tokens_out":8785,"duration_ms":79509,"concrete_test":"Run a same-budget controlled continued-pretraining experiment at the LLaMA3-8B scale used in Section 4.2. Condition A: 100B tokens with the paper's exact mix, including 10B aligned Arabic tokens. Condition B: identical mix and total token count, but the 10B aligned Arabic tokens replaced by 10B unaligned ArabicText2022 tokens. Condition C: vanilla Llama3-8B. Evaluate A, B, and C on ArabicMMLU, EXAMS, ACVA-clean/all, and AraTrust using the paper's evaluation code, plus BeaverTails harmlessness/helpfulness, with at least two seeds and confidence intervals. If B matches or exceeds A on the Arabic benchmarks, the Table 2/SOTA claim is not attributable to native alignment; if A clearly beats B while both have identical Arabic token budgets, the central causal claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that rewriting Arabic pre-training data during continued pretraining ('native alignment') produces safer, more helpful, higher-scoring base models. The decisive evidence for this claim is Table 2/Section 3.2 and the BeaverTails improvements in Figure 5. In both places the comparison is confounded by compute and data quantity. Per Section 3.1 and Figure 4, LLaMA3-Tamed-8B/70B received continued pretraining on about 100B tokens, of which roughly 38% is unaligned ArabicText2022 plus a further 10B GPT-4-rewritten Arabic tokens; the vanilla Llama3-8B/70B baselines in Table 2 received none of this continued pretraining. The observed gains on ArabicMMLU, EXAMS, ACVA, and AraTrust could therefore be produced by additional Arabic exposure or by continued pretraining alone, without any contribution from the alignment rewrite. Section 4.2's 12B ablation is the only same-budget check, but it evaluates only BeaverTails harmlessness/helpfulness, not the Table 2 benchmarks, and it does not keep both data source and token budget fixed while toggling only the rewrite: 'Only Align-12B' swaps the pretraining data for aligned data, and 'Hybrid' adds alignment tokens on top of the 12B baseline. Thus no experiment cleanly isolates 'alignment rewriting' as the cause of the headline benchmark gains, and the Limitations section itself concedes that no fair alignment benchmark is used. The claim is therefore conditional on a control experiment that has not been run.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces 'native alignment', a data-processing approach that rewrites pre-training data (via a trained alignment worker) to remove harmful, biased, or poorly formatted content before or during continued pre-training. The method is applied to Arabic: the authors construct LLaMA3-Tamed-8B and LLaMA3-Tamed-70B by continued pretraining on a mixed corpus that includes GPT-4-rewritten Arabic tokens, and report state-of-the-art results on ArabicMMLU, EXAMS, ACVA, and AraTrust, together with improvements in harmlessness and helpfulness on the BeaverTails benchmark relative to a 12B-token baseline. The paper also presents ablations on data composition, scaling behavior, seed data selection, and a small English generalization experiment, and releases code, data, and models.","tokens_in":17176,"tokens_out":7177,"duration_ms":59892,"significance":"If the central claim holds, native alignment is a genuinely useful data-centric complement to post-alignment, especially for low-resource and culturally sensitive languages. The paper has clear strengths: it ships open-source models and code, uses external benchmarks for the main knowledge/localization/trust claims, and includes multiple ablation studies including an English transfer experiment in Appendix E. The 12B ablations provide a plausible same-budget signal for harmlessness/helpfulness. However, the headline Table 2 comparison is confounded by continued-pretraining compute and by the quantity of extra Arabic data, and the safety/helpfulness evidence relies on single-run GPT-4 judgments that share a value source with the data-generation process. The central claim is therefore plausible but conditional on a same-budget control that has not been reported.","major_comments":[{"comment":"The headline comparison in Table 2 is confounded by compute and data quantity. LLaMA3-Tamed-8B/70B receive continued pretraining on roughly 100B tokens, of which about 38% is ArabicText2022 and a further 10B are GPT-4-rewritten Arabic tokens (Section 3.1, Figure 4), while the Llama3-8B/70B baselines in Table 2 receive no continued pretraining. The observed gains on ArabicMMLU, EXAMS, ACVA, and AraTrust could therefore be produced by additional Arabic exposure or by continued pretraining itself, without any contribution from the alignment rewrite. The authors should report a same-budget control: continued pretraining with the same 100B token mix but with unaligned ArabicText2022 in place of the rewritten Arabic tokens, evaluated on the Table 2 benchmarks. It would also help to clarify whether the 10B rewritten tokens are a subset of the 38B ArabicText2022 slice or an additional phase after the 100B.","section":"§3.1, Table 2, Figure 4"},{"comment":"The 12B ablation is the only experiment that approaches a matched budget, but it is not sufficient to support the Table 2 claims. First, it evaluates only BeaverTails harmlessness/helpfulness after SFT, not ArabicMMLU, EXAMS, ACVA, or AraTrust. Second, the 'Hybrid' condition (Pre-train-12B + Align-12B + SFT-50K) uses twice the pre-training tokens of the 'Pre-train-12B + SFT-50K' baseline, so its +2.6%/+7.7% improvement is confounded by token quantity. A clean test would compare 12B unaligned vs 12B aligned tokens at equal budget and would also report the Table 2 benchmarks (or a subset) on these small models.","section":"§4.2, Figure 6"},{"comment":"The harmlessness/helpfulness evidence is based on GPT-4 judging model outputs, while GPT-4 also generated the seed alignment data used to train the rewriting worker (Section 3.1). This creates a circularity risk: the rewrite data and the judge share the same value source. In addition, Figure 5 and Section 4.2 report single-run results with no variance, even though Appendix B describes position-bias mitigation by averaging orders. The authors should provide human evaluation or multiple runs with confidence intervals, or at least a detailed analysis of agreement and judge bias, before claiming the 10.4%/4.8% improvements as robust native-alignment effects.","section":"§4.1, §3.1, Appendix B"},{"comment":"The 'state-of-the-art open-source Arabic LLM' claim is not supported by the baseline set in Table 2. AceGPT, which is cited and used as a baseline in Figure 5, is absent from Table 2, as are other Arabic-specific open models. Adding AceGPT-7B/13B (and, if available, AceGPT-70B) to Table 2 is necessary to substantiate the SOTA claim, since the current comparison is mainly against multilingual models and vanilla Llama-3.","section":"§3.2, contributions"}],"minor_comments":[{"comment":"The conclusion contains the typo 'comprmeiyehensive data processing workflow', and the Figure 6 caption says 'various training trategies' instead of 'strategies'.","section":"Conclusion and Figure 6"},{"comment":"The Data Composition description is ambiguous: the percentages in Figure 4 sum to 100B, but the Training Details mention a separate 'native-alignment training with the 10 billion tokens', which could mean the total budget is 110B; please specify the exact token budget and whether the rewritten tokens are in addition to or part of the 100B.","section":"§3.1"},{"comment":"The comparison with DPO is explicitly acknowledged as 'not a fair apples-to-apples comparison'; this caveat should be stated in the main text or the experiment should be removed from the conclusions.","section":"Appendix D"},{"comment":"The Limitations section concedes 'the absence of a suitable and fair benchmark for evaluating alignment', which is in tension with the strong SOTA claim in Section 3.2 and should be reconciled.","section":"Limitations"},{"comment":"Table 2 has minor formatting issues: 'ACV A' contains an unusual space, and model names like 'Llama3-8B' are inconsistent with 'LLaMA3-Tamed-8B' in the same table.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising idea, and the 12B ablations show a positive signal, but the headline SOTA and safety claims currently rest on confounded comparisons and a single GPT-4 judge. I recommend major revision with a same-budget unaligned control, inclusion of Arabic-specific baselines such as AceGPT in Table 2, and human or multi-run evaluation for the safety claims. If the authors cannot run the control at 100B scale, they should at least run it at 12B and report the Table 2 benchmarks there, and temper the SOTA claim accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: native alignment is a real extension of LLM-based data rewriting to value alignment for a low-resource language, and the released models are useful. But the headline comparisons in Table 2 do not isolate the effect they claim, and the causal story rests on a smaller ablation that covers only safety and helpfulness. This should be conditional, not rejected.\n\nWhat is actually new: the framing of alignment at pretraining via rewriting, applied to Arabic, with a worker model distilled from GPT-4 and two released base models. The 12B ablation in Section 4.2 is a genuine same-budget check — Pre-train-12B versus Align-12B toggles rewriting while keeping token count fixed, and it shows gains in harmlessness and helpfulness on BeaverTails. That is real evidence for the mechanism, at least on that benchmark.\n\nWhere the paper is soft: the stress-test concern is accurate. Table 2 compares LLaMA3-Tamed-8B/70B, which received ~100B tokens of continued pretraining including ~10B rewritten Arabic tokens, against vanilla Llama-3 baselines that received none. Any improvement on ArabicMMLU, EXAMS, ACVA, or AraTrust can be explained by extra Arabic exposure alone. The 12B ablation does not include those benchmarks, so the knowledge and localization gains are unisolated. The evaluation loop is also partly circular: GPT-4 writes the seed alignment data and then grades harmlessness and helpfulness in the ablation. That would matter less if the knowledge benchmarks were clean, but they are not. The authors' own Limitations section admits the absence of a fair alignment benchmark, yet the abstract and conclusion still claim state-of-the-art performance. That mismatch needs to be fixed.\n\nWhat would make the paper solid: an equal-budget control with 10B unaligned Arabic tokens, plus a proper same-budget comparison on the Table 2 benchmarks, and ideally multiple seeds or intervals. None of this is exotic; it is a day or two of training and evaluation.\n\nBottom line: the idea is worth engaging, the models are a contribution, and the 12B ablation prevents a rejection. But the central causal claim is not yet isolated. Send it to review with a request for the equal-budget control. I would bring it to a reading group.","headline":"Native alignment is a real extension of LLM-based data rewriting for low-resource languages, but the headline comparisons are confounded by extra continued pretraining and the causal claim is not yet isolated.","tokens_in":17712,"tokens_out":3379,"would_cite":false,"duration_ms":28986,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that rewriting Arabic pre-training data to obey a code of conduct, before the model ever sees it, produces base LLMs that are safer, more helpful, and state-of-the-art on Arabic benchmarks.","keywords":["native alignment","pre-training alignment","Arabic large language models","data-centric alignment","data rewriting","harmlessness and helpfulness","low-resource languages","LLM alignment"],"falsifier":"Train the same 100-billion-token recipe but replace the 10 billion aligned Arabic tokens with 10 billion raw, unaligned ArabicText2022 tokens, then compare on AraTrust and the BeaverTails harmlessness and helpfulness protocol; if the raw-token model matches or beats the aligned-token model, the claim that rewriting is what drives the gains is falsified.","tokens_in":16614,"feed_emoji":"🕌","tokens_out":7207,"duration_ms":61883,"temperature":0.7,"pith_summary":"Most alignment happens after a model is pre-trained: instruction tuning and reinforcement learning teach a finished model to behave. This paper argues that alignment should also happen during pre-training itself, by rewriting the raw training text so that bad content never enters the model in the first place. Concretely, it repairs scarce Arabic web data, fixing formatting, removing hate and ads, and steering away from controversial or religiously sensitive claims, using small alignment worker models trained on expert-rewritten examples. When 10 billion rewritten Arabic tokens are mixed into continued pre-training of Llama-3-8B and Llama-3-70B, the resulting base models beat other open-source Arabic LLMs on knowledge, cultural-localization, and trust benchmarks, and improve harmlessness and helpfulness by 10.4% and 4.8% in the authors' head-to-head ablation. If the results hold, native alignment offers low-resource languages a way to keep scarce data by repairing it, instead of discarding it.","feed_headline":"Rewriting Arabic pre-training data lifts LLM safety and scores","feed_subtitle":"Training Llama-3 on 10 billion rewritten Arabic tokens beats open-source rivals on knowledge, localization, and trust.","key_machinery":"The load-bearing mechanism is the alignment-rewriting pipeline, not a new training objective. Four steps carry the argument: deduplicate the web crawl; have an expert (human or frontier model) rewrite a small seed set under explicit polishing instructions; train a small chat model as an alignment worker on those pairs; and rewrite the full corpus at scale. The rewritten data, about 8.6 billion tokens produced from 10 billion input tokens, is then used as part of continued pre-training. The crucial empirical hinge is that this rewritten data, when mixed with ordinary pre-training data, improves both safety and usefulness of the base model, and that the improvement grows with the volume of aligned data.","core_discovery":"The central claim is that value alignment can be baked into a base model during pre-training, not only bolted on afterwards. The paper introduces native alignment: a data-centric workflow in which raw web text is rewritten to satisfy a code of conduct covering format correctness, fair values, content moderation, and knowledge preservation. A small LLM is trained on expert-produced rewrite pairs and then used to rewrite the corpus at scale; for Arabic, 10 billion tokens of ArabicText2022 are rewritten and mixed with 90 billion tokens of other data for continued pre-training of Llama-3-8B and Llama-3-70B. The paper reports that the resulting LLaMA3-Tamed-70B achieves state-of-the-art results among open-source models on ArabicMMLU, EXAMS, ACVA, and AraTrust, and that in controlled comparisons native alignment raises harmlessness and helpfulness relative to training on the original unaligned data.","pith_inferences":["The headline LLaMA3-Tamed-70B comparison does not include a same-budget control with raw, unaligned Arabic tokens; until that control is run, part of the observed gain could be credited to the extra Arabic data or to continued pre-training itself.","A natural next experiment is to vary the quality of the alignment worker: if a much smaller or weaker model can produce effective rewrite seeds, the front-loaded cost of expert rewriting drops substantially.","One risk the paper does not examine is that fair-values rewriting could flatten legitimate cultural or political diversity in the corpus, reducing the very Arabic-specific knowledge the method aims to preserve.","If native alignment scales, it suggests a data flywheel: each generation of models could clean and align the next generation's web corpus, shifting alignment cost from inference-time tuning to data construction."],"forward_implications":["Base models trained with native alignment can be released directly and still benefit from later SFT or DPO; the paper shows the two stages are complementary rather than competing.","Mixing aligned and ordinary pre-training data beats using either alone, so alignment rewriting should be treated as one component of a data mix, not a full replacement.","Helpfulness rises steadily as the volume of aligned tokens grows up to the tested 12 billion, giving a practical scaling signal for how much rewriting to budget.","The same rewriting recipe transfers to English and outperforms conventional data cleaning in the appendix, so the mechanism is not Arabic-specific.","For low-resource languages, native alignment offers a way to salvage scarce but messy web data instead of throwing it away."],"supporting_citations":[{"why":"Supplies the Llama-3-8B and Llama-3-70B checkpoints that receive continued pre-training, the base for both Tamed models.","marker":"[19]"},{"why":"States the superficial alignment hypothesis that motivates aligning during pre-training instead of only afterward.","marker":"[5]"},{"why":"Supplies the small chat model used as the alignment worker and several comparison baselines.","marker":"[24]"},{"why":"Provides the Arabic evaluation protocol used for the benchmark comparisons.","marker":"[30]"},{"why":"Provides the BeaverTails prompt set used to measure harmlessness and helpfulness improvements.","marker":"[32]"},{"why":"Supplies RefinedWeb, the corpus used in the appendix comparison of native alignment against conventional data cleaning.","marker":"[34]"},{"why":"Motivates the toxicity-reduction pilot study via toxic degeneration from unaligned pretraining data.","marker":"[16]"},{"why":"Supplies the ArabicMMLU knowledge benchmark used in the main results table.","marker":"[21]"},{"why":"Supplies the AraTrust trustworthiness benchmark used in the main results table.","marker":"[23]"}],"fun_headline_variants":["Native alignment: bake safety into LLMs during pre-training","Rewriting pre-training data aligns Arabic LLMs naturally","Arabic LLMs aligned at pre-training, not just fine-tuned","State-of-the-art Arabic models from pre-training alignment","Aligning Arabic LLMs from scratch with rewritten data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on comparing LLaMA3-Tamed-70B with vanilla Llama-3-70B when the only difference is 10 billion rewritten Arabic tokens plus continued pretraining, and no same-budget control with raw Arabic tokens is reported, so the improvement could come partly from the extra Arabic data or the continued pretraining itself.","fun_headline_variants_meta":{"raw":{"variants":["Native alignment: bake safety into LLMs during pre-training","Rewriting pre-training data aligns Arabic LLMs naturally","Arabic LLMs aligned at pre-training, not just fine-tuned","State-of-the-art Arabic models from pre-training alignment","Aligning Arabic LLMs from scratch with rewritten data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000209,"raw_usage":{"total_tokens":1372,"prompt_tokens":877,"completion_tokens":495,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":415}},"tokens_in":493,"tokens_out":495,"duration_ms":5240,"temperature":1.0,"reasoning_tokens":415,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:36:13.978592+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same 100-billion-token recipe but replace the 10 billion aligned Arabic tokens with 10 billion raw, unaligned ArabicText2022 tokens, then compare on AraTrust and the BeaverTails harmlessness and helpfulness protocol; if the raw-token model matches or beats the aligned-token model, the claim that rewriting is what drives the gains is falsified.","supporting_citations":[{"cited_title":"\"arabicmmlu: Assessing massive multitask language understanding in arabic","cited_arxiv_id":null,"evidence_quote":"Supplies the ArabicMMLU knowledge benchmark used in the main results table."}],"review_version":1}