Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Alignment at Pre-training! Towards Native Alignment for Arabic LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that rewriting Arabic pre-training data to obey a code of conduct, before the model ever sees it, produces base LLMs that are safer, more helpful, and state-of-the-art on Arabic benchmarks.

desk verdict Native alignment is a real extension of LLM-based data rewriting for low-resource languages, but the headline comparisons are confounded by extra continued pretraining and the causal claim is not yet isolated. read the letter →

arxiv 2412.03253 v1 pith:24HMRZAR submitted 2024-12-04 cs.CL

Juhao Liang , Zhenyang Cai , Jianqing Zhu , Huang Huang , Kewei Zong , Bang An , Mosen Alharthi , Juncai He
show 4 more authors
Lian Zhang Haizhou Li Benyou Wang Jinchao Xu
This is my paper · ORCID
classification cs.CL
keywords nativealignmentpre-trainingArabiclargelanguagemodelsdata-centricdatarewritingharmlessnessandhelpfulnesslow-resourcelanguagesLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most alignment happens after a model is pre-trained: instruction tuning and reinforcement learning teach a finished model to behave. This paper argues that alignment should also happen during pre-training itself, by rewriting the raw training text so that bad content never enters the model in the first place. Concretely, it repairs scarce Arabic web data, fixing formatting, removing hate and ads, and steering away from controversial or religiously sensitive claims, using small alignment worker models trained on expert-rewritten examples. When 10 billion rewritten Arabic tokens are mixed into continued pre-training of Llama-3-8B and Llama-3-70B, the resulting base models beat other open-source Arabic LLMs on knowledge, cultural-localization, and trust benchmarks, and improve harmlessness and helpfulness by 10.4% and 4.8% in the authors' head-to-head ablation. If the results hold, native alignment offers low-resource languages a way to keep scarce data by repairing it, instead of discarding it.

What carries the argument

The load-bearing mechanism is the alignment-rewriting pipeline, not a new training objective. Four steps carry the argument: deduplicate the web crawl; have an expert (human or frontier model) rewrite a small seed set under explicit polishing instructions; train a small chat model as an alignment worker on those pairs; and rewrite the full corpus at scale. The rewritten data, about 8.6 billion tokens produced from 10 billion input tokens, is then used as part of continued pre-training. The crucial empirical hinge is that this rewritten data, when mixed with ordinary pre-training data, improves both safety and usefulness of the base model, and that the improvement grows with the volume of aligned data.

What would settle it

Train the same 100-billion-token recipe but replace the 10 billion aligned Arabic tokens with 10 billion raw, unaligned ArabicText2022 tokens, then compare on AraTrust and the BeaverTails harmlessness and helpfulness protocol; if the raw-token model matches or beats the aligned-token model, the claim that rewriting is what drives the gains is falsified.

Watch

Extended reading notes

Core claim

The central claim is that value alignment can be baked into a base model during pre-training, not only bolted on afterwards. The paper introduces native alignment: a data-centric workflow in which raw web text is rewritten to satisfy a code of conduct covering format correctness, fair values, content moderation, and knowledge preservation. A small LLM is trained on expert-produced rewrite pairs and then used to rewrite the corpus at scale; for Arabic, 10 billion tokens of ArabicText2022 are rewritten and mixed with 90 billion tokens of other data for continued pre-training of Llama-3-8B and Llama-3-70B. The paper reports that the resulting LLaMA3-Tamed-70B achieves state-of-the-art results among open-source models on ArabicMMLU, EXAMS, ACVA, and AraTrust, and that in controlled comparisons native alignment raises harmlessness and helpfulness relative to training on the original unaligned data.

Load-bearing premise

The claim rests on comparing LLaMA3-Tamed-70B with vanilla Llama-3-70B when the only difference is 10 billion rewritten Arabic tokens plus continued pretraining, and no same-budget control with raw Arabic tokens is reported, so the improvement could come partly from the extra Arabic data or the continued pretraining itself.

Editorial extensions

If this is right

  • Base models trained with native alignment can be released directly and still benefit from later SFT or DPO; the paper shows the two stages are complementary rather than competing.
  • Mixing aligned and ordinary pre-training data beats using either alone, so alignment rewriting should be treated as one component of a data mix, not a full replacement.
  • Helpfulness rises steadily as the volume of aligned tokens grows up to the tested 12 billion, giving a practical scaling signal for how much rewriting to budget.
  • The same rewriting recipe transfers to English and outperforms conventional data cleaning in the appendix, so the mechanism is not Arabic-specific.
  • For low-resource languages, native alignment offers a way to salvage scarce but messy web data instead of throwing it away.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline LLaMA3-Tamed-70B comparison does not include a same-budget control with raw, unaligned Arabic tokens; until that control is run, part of the observed gain could be credited to the extra Arabic data or to continued pre-training itself.
  • A natural next experiment is to vary the quality of the alignment worker: if a much smaller or weaker model can produce effective rewrite seeds, the front-loaded cost of expert rewriting drops substantially.
  • One risk the paper does not examine is that fair-values rewriting could flatten legitimate cultural or political diversity in the corpus, reducing the very Arabic-specific knowledge the method aims to preserve.
  • If native alignment scales, it suggests a data flywheel: each generation of models could clean and align the next generation's web corpus, shifting alignment cost from inference-time tuning to data construction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces 'native alignment', a data-processing approach that rewrites pre-training data (via a trained alignment worker) to remove harmful, biased, or poorly formatted content before or during continued pre-training. The method is applied to Arabic: the authors construct LLaMA3-Tamed-8B and LLaMA3-Tamed-70B by continued pretraining on a mixed corpus that includes GPT-4-rewritten Arabic tokens, and report state-of-the-art results on ArabicMMLU, EXAMS, ACVA, and AraTrust, together with improvements in harmlessness and helpfulness on the BeaverTails benchmark relative to a 12B-token baseline. The paper also presents ablations on data composition, scaling behavior, seed data selection, and a small English generalization experiment, and releases code, data, and models.

Significance. If the central claim holds, native alignment is a genuinely useful data-centric complement to post-alignment, especially for low-resource and culturally sensitive languages. The paper has clear strengths: it ships open-source models and code, uses external benchmarks for the main knowledge/localization/trust claims, and includes multiple ablation studies including an English transfer experiment in Appendix E. The 12B ablations provide a plausible same-budget signal for harmlessness/helpfulness. However, the headline Table 2 comparison is confounded by continued-pretraining compute and by the quantity of extra Arabic data, and the safety/helpfulness evidence relies on single-run GPT-4 judgments that share a value source with the data-generation process. The central claim is therefore plausible but conditional on a same-budget control that has not been reported.

major comments (4)
  1. [§3.1, Table 2, Figure 4] The headline comparison in Table 2 is confounded by compute and data quantity. LLaMA3-Tamed-8B/70B receive continued pretraining on roughly 100B tokens, of which about 38% is ArabicText2022 and a further 10B are GPT-4-rewritten Arabic tokens (Section 3.1, Figure 4), while the Llama3-8B/70B baselines in Table 2 receive no continued pretraining. The observed gains on ArabicMMLU, EXAMS, ACVA, and AraTrust could therefore be produced by additional Arabic exposure or by continued pretraining itself, without any contribution from the alignment rewrite. The authors should report a same-budget control: continued pretraining with the same 100B token mix but with unaligned ArabicText2022 in place of the rewritten Arabic tokens, evaluated on the Table 2 benchmarks. It would also help to clarify whether the 10B rewritten tokens are a subset of the 38B ArabicText2022 slice or an additional phase after the 100B.
  2. [§4.2, Figure 6] The 12B ablation is the only experiment that approaches a matched budget, but it is not sufficient to support the Table 2 claims. First, it evaluates only BeaverTails harmlessness/helpfulness after SFT, not ArabicMMLU, EXAMS, ACVA, or AraTrust. Second, the 'Hybrid' condition (Pre-train-12B + Align-12B + SFT-50K) uses twice the pre-training tokens of the 'Pre-train-12B + SFT-50K' baseline, so its +2.6%/+7.7% improvement is confounded by token quantity. A clean test would compare 12B unaligned vs 12B aligned tokens at equal budget and would also report the Table 2 benchmarks (or a subset) on these small models.
  3. [§4.1, §3.1, Appendix B] The harmlessness/helpfulness evidence is based on GPT-4 judging model outputs, while GPT-4 also generated the seed alignment data used to train the rewriting worker (Section 3.1). This creates a circularity risk: the rewrite data and the judge share the same value source. In addition, Figure 5 and Section 4.2 report single-run results with no variance, even though Appendix B describes position-bias mitigation by averaging orders. The authors should provide human evaluation or multiple runs with confidence intervals, or at least a detailed analysis of agreement and judge bias, before claiming the 10.4%/4.8% improvements as robust native-alignment effects.
  4. [§3.2, contributions] The 'state-of-the-art open-source Arabic LLM' claim is not supported by the baseline set in Table 2. AceGPT, which is cited and used as a baseline in Figure 5, is absent from Table 2, as are other Arabic-specific open models. Adding AceGPT-7B/13B (and, if available, AceGPT-70B) to Table 2 is necessary to substantiate the SOTA claim, since the current comparison is mainly against multilingual models and vanilla Llama-3.
minor comments (5)
  1. [Conclusion and Figure 6] The conclusion contains the typo 'comprmeiyehensive data processing workflow', and the Figure 6 caption says 'various training trategies' instead of 'strategies'.
  2. [§3.1] The Data Composition description is ambiguous: the percentages in Figure 4 sum to 100B, but the Training Details mention a separate 'native-alignment training with the 10 billion tokens', which could mean the total budget is 110B; please specify the exact token budget and whether the rewritten tokens are in addition to or part of the 100B.
  3. [Appendix D] The comparison with DPO is explicitly acknowledged as 'not a fair apples-to-apples comparison'; this caveat should be stated in the main text or the experiment should be removed from the conclusions.
  4. [Limitations] The Limitations section concedes 'the absence of a suitable and fair benchmark for evaluating alignment', which is in tension with the strong SOTA claim in Section 3.2 and should be reconciled.
  5. [Table 2] Table 2 has minor formatting issues: 'ACV A' contains an unusual space, and model names like 'Llama3-8B' are inconsistent with 'LLaMA3-Tamed-8B' in the same table.

Circularity Check

1 steps flagged · score 4.0 of 10

Partial closed loop: GPT-4 both rewrites the native-alignment data and judges the harmlessness/helpfulness gains; benchmark and ACVA/AraTrust results remain externally grounded.

  1. other [Section 3.1 (Data Composition) and Section 4.1 (Benchmark: BeaverTails); Appendix B]
    "Specifically, we utilized GPT-4 as an alignment expert to generate 10k expert alignment data for alignment worker training... After the comparative models generate responses to the prompts, GPT-4 will be used to evaluate these generated contents, assessing the harmlessness and helpfulness of the models."

    The native-alignment intervention is defined by GPT-4's rewriting preferences (GPT-4 created the seed alignment data used to train the Qwen rewrite worker), and the reported harmlessness/helpfulness outcome is also defined by GPT-4's pairwise judgments on BeaverTails. The 10.4%/4.8% improvements therefore measure how well the continued-pretrained model reproduces GPT-4's rewriting policy; the evaluator and the data generator are the same system, so this particular result is a closed loop rather than an independent verification. The external benchmarks (ArabicMMLU, EXAMS, ACVA, AraTrust) are not part of this loop, which is why the circularity is only partial.

full rationale

Most of the paper's evidence is not circular: Table 2 uses external benchmarks (ArabicMMLU, EXAMS, ACVA, AraTrust) and LLaMA3-Tamed-8B/70B are compared against vanilla Llama3, Jais, Qwen, and closed models, so the knowledge/localization/trustworthiness claims have independent grounding. The main circularity concern is confined to the harmlessness/helpfulness evaluation in Section 4.1 and Appendix B, where GPT-4 is both the source of the alignment data and the judge of the outcome; the improvement is at least partially a measure of self-consistency with GPT-4's preferences. The more serious experimental issue—LLaMA3-Tamed received ~100B tokens of continued pretraining (including 38B unaligned ArabicText2022 and 10B aligned tokens) while the vanilla Llama3 baselines received none—is a confound in causal identification, not a circularity by construction, and is noted in the Limitations section. No load-bearing self-citation or imported uniqueness theorem appears; citations to the authors' AceGPT/prior Arabic work provide benchmarks and evaluation code, which are external instruments. The paper honestly concedes the absence of a fair alignment benchmark, which supports treating the closed GPT-4 loop as a partial limitation rather than a complete derivation collapse.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No physical entities are introduced. The scientific load is carried by data-scale choices and evaluation assumptions, listed above.

free parameters (4)
  • Alignment data volume (full models) = 10B tokens
    Chosen for the released models; no ablation at this scale, so the SOTA claim depends on this hand-set budget.
  • Total continued-pretraining budget = 100B tokens
    Mixed-source budget used for LLaMA3-Tamed-8B and 70B; confounds comparison with vanilla Llama-3 baselines.
  • Expert demonstration size = 10k GPT-4 rewrites
    Used to train the Qwen1.5-4B-Chat alignment worker; no sensitivity analysis reported.
  • Alignment worker model = Qwen1.5-4B-Chat
    Selected for speed and quality; no comparison with alternative worker sizes or base models.
assumptions (4)
  • domain assumption LLM rewriting removes toxicity and format errors while preserving knowledge
    Pilot studies cover only 8k Arabic samples (Section 2.2), and hallucination checks cover 90 pairs (Limitations).
  • domain assumption Continued pretraining with 100B tokens including 10B aligned Arabic tokens is comparable to vanilla Llama-3 baselines
    Section 3.1 and Table 2 use vanilla Llama-3 without an equal-budget unaligned Arabic control.
  • domain assumption GPT-4 judgments on translated BeaverTails prompts measure harmlessness and helpfulness
    Section 4.1 and Appendix B; position bias is addressed, but the judge belongs to the same model family that generated the alignment data.
  • domain assumption The superficial alignment hypothesis justifies moving alignment into pretraining
    Introduction relies on LIMA (cited [5]) to argue that alignment is mostly format, motivating native alignment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Alignment at Pre-training! Towards Native Alignment for Arabic LLMs." pith.science (2026). https://pith.science/paper/24HMRZAR

@misc{pith2026241203253,
  author       = {Pith},
  title        = {Pith review of: Alignment at Pre-training! Towards Native Alignment for Arabic LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/24HMRZAR}},
  note         = {Machine review of arXiv:2412.03253}
}
read the original abstract

The alignment of large language models (LLMs) is critical for developing effective and safe language models. Traditional approaches focus on aligning models during the instruction tuning or reinforcement learning stages, referred to in this paper as `post alignment'. We argue that alignment during the pre-training phase, which we term `native alignment', warrants investigation. Native alignment aims to prevent unaligned content from the beginning, rather than relying on post-hoc processing. This approach leverages extensively aligned pre-training data to enhance the effectiveness and usability of pre-trained models. Our study specifically explores the application of native alignment in the context of Arabic LLMs. We conduct comprehensive experiments and ablation studies to evaluate the impact of native alignment on model performance and alignment stability. Additionally, we release open-source Arabic LLMs that demonstrate state-of-the-art performance on various benchmarks, providing significant benefits to the Arabic LLM community.

Figures

Figures reproduced from arXiv: 2412.03253 by the authors.

Figure 1
Figure 1. Comparison of pre-training data quality before and after data alignment rewriting. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Demonstration of pre-training data processing workflow for native alignment. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Perplexity before and after native alignment of Arabic data. Perplexity Analysis Pre-training data pruning [18] demonstrated that simple data pruning using perplex￾ity metrics surpasses other more computationally de￾manding scoring methods. This approach can curate high-quality corpora and enhance model training per￾formance with less data. In accordance with the pa￾per, we calculate the perplexity metric as follows… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: The left side illustrates the datasets utilized during the pre-training phase of the model, [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The ratio of metrics for base models relative [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The left graph illustrates the metric improvements under various training trategies. The right graph demonstrates the performance gains as the volume of alignment data increases. In both graphs, the baseline model, ‘Pre-train-12B + SFT-50K’, is initially trained on 12 …
Figure 7
Figure 7. Figure 7: As we can observe, the prompt emphasizes various aspects such as knowledge preservation, [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 7
Figure 7. Figure 7: The guide for native alignment instructs LLMs to rewrite the pre-training data in accordance [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: The prompt for evaluating the Harmlessness of two responses. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: The prompt for evaluating the Helpfulness of two responses. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: The prompt to evaluate rewriting quality. [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. On The Origin of Cultural Biases in Language Models: From Pre-training Data to Linguistic Phenomena

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Arab cultural entities that double as everyday Arabic words are harder for language models to recognize, especially when tokenized as single tokens.

Reference graph

Works this paper leans on

62 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020

    2020

  2. [2]

    Acegpt, localizing large language models in arabic, 2024

    Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Juncai He, Ziche Liu, Zhiyi Zhang, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. Acegpt, localizing large language models in arabic, 2024. 10

    work page 2024

  3. [3]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022

    2022

  4. [4]

    Finetuned language models are zero-shot learners

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021

    arXiv 2021

  5. [5]

    Lima: Less is more for alignment

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024

    2024

  6. [6]

    Reformatted alignment

    Run-Ze Fan, Xuefeng Li, Haoyang Zou, Junlong Li, Shwai He, Ethan Chern, Jiewen Hu, and Pengfei Liu. Reformatted alignment. arXiv preprint arXiv:2402.12219, 2024

    arXiv 2024

  7. [7]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  8. [8]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

Show all 62 references
  1. [9]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  2. [10]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  3. [11]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 2023

    2023

  4. [12]

    Rlaif: Scaling reinforcement learning from human feedback with ai feedback, 2023

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. Rlaif: Scaling reinforcement learning from human feedback with ai feedback, 2023

    2023

  5. [13]

    Aligning large language models with human: A survey

    Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966, 2023

    2023 arXiv

  6. [14]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  7. [15]

    A study of practical deduplication

    Dutch T Meyer and William J Bolosky. A study of practical deduplication. ACM Transactions on Storage (ToS), 7(4):1–20, 2012

    2012

  8. [16]

    Real- toxicityprompts: Evaluating neural toxic degeneration in language models

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Real- toxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462, 2020

    2009 arXiv

  9. [17]

    Polyglotoxicityprompts: Multilingual evaluation of neural toxic degeneration in large language models

    Devansh Jain, Priyanshu Kumar, Samuel Gehman, Xuhui Zhou, Thomas Hartvigsen, and Maarten Sap. Polyglotoxicityprompts: Multilingual evaluation of neural toxic degeneration in large language models. arXiv preprint arXiv:2405.09373, 2024

    2024 arXiv

  10. [18]

    When less is more: Investigating data pruning for pretraining llms at scale

    Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining llms at scale. arXiv preprint arXiv:2309.04564, 2023

    2023 arXiv

  11. [19]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024

    2024

  12. [20]

    Arabic natural language processing: Challenges and solutions

    Ali Farghaly and Khaled Shaalan. Arabic natural language processing: Challenges and solutions. ACM Transactions on Asian Language Information Processing (TALIP), 8(4):1–22, 2009

    2009

  13. [21]

    "arabicmmlu: Assessing massive multitask language understanding in arabic

    "Fajri Koto, Haonan Li, Sara Shatanawi, Jad Doughman, Abdelrahman Boda Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin". "arabicmmlu: Assessing massive multitask language understanding in...

    2024

  14. [22]

    EXAMS: A multi-subject high school examinations dataset for cross-lingual and multilingual question answering

    Momchil Hardalov, Todor Mihaylov, Dimitrina Zlatkova, Yoan Dinkov, Ivan Koychev, and Preslav Nakov. EXAMS: A multi-subject high school examinations dataset for cross-lingual and multilingual question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natu...

    2020

  15. [23]

    Aratrust: An evaluation of trustworthiness for llms in arabic

    Emad A Alghamdi, Reem I Masoud, Deema Alnuhait, Afnan Y Alomairi, Ahmed Ashraf, and Mohamed Zaytoon. Aratrust: An evaluation of trustworthiness for llms in arabic. arXiv preprint arXiv:2403.09017, 2024

    2024 arXiv

  16. [24]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Si...

    2023 arXiv

  17. [25]

    Jais and jais- chat: Arabic-centric foundation and instruction-tuned open generative large language models, 2023

    Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muham- mad Mujahid, Massa Baali, Alham Fikri Aji, Zhengzhong Liu, Andy Hock, Andrew Feldman, Jonathan Lee, An...

    2023

  18. [26]

    SlimPajama: A 627B token cleaned and deduplicated version of RedPajama

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. https://www.cerebras.net/blog/ slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama , June 2023

    2023

  19. [27]

    Chinese tiny llm: Pretraining a chinese-centric large language model, 2024

    Xinrun Du, Zhouliang Yu, Songyang Gao, Ding Pan, Yuyang Cheng, Ziyang Ma, Ruibin Yuan, Xingwei Qu, Jiaheng Liu, Tianyu Zheng, Xinchen Luo, Guorui Zhou, Binhang Yuan, Wenhu Chen, Jie Fu, and Ge Zhang. Chinese tiny llm: Pretraining a chinese-centric large language model, 2024

    2024

  20. [28]

    Wikimedia downloads

    Wikimedia Foundation. Wikimedia downloads

  21. [29]

    Jiang, Jia Deng, Stella Biderman, and Sean Welleck

    Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics, 2023

    2023

  22. [30]

    Alghamdi, Lian Zhang, Ruoyu Sun, Haizhou Li, Benyou Wang, and Jinchao Xu

    Jianqing Zhu, Huang Huang, Zhihang Lin, Juhao Liang, Zhengyang Tang, Khalid Almubarak, Mosen Alharthi, Bang An, Juncai He, Xiangbo Wu, Fei Yu, Junying Chen, Zhuoheng Ma, Yuhao Du, Yan Hu, He Zhang, Emad A. Alghamdi, Lian Zhang, Ruoyu Sun, Haizhou Li, Benyou Wang, and Jinchao X...

    2024

  23. [31]

    Opencompass: A universal evaluation platform for foundation models

    OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass, 2023

    2023

  24. [32]

    Beavertails: Towards improved safety alignment of llm via a human-preference dataset

    Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Chi Zhang, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. arXiv preprint arXiv:2307.04657, 2023

    2023

  25. [33]

    Large language models are not fair evaluators

    Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926, 2023

    2023 arXiv

  26. [34]

    The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only

    Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv pr...

    2023 arXiv

  27. [35]

    Textbooks are all you need

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023. 12

    2023 arXiv

  28. [36]

    Textbooks are all you need ii: phi-1.5 technical report

    Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463, 2023

    2023 arXiv

  29. [37]

    Phi-2: The surprising power of small language models

    Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 2023

    2023

  30. [38]

    Quality at a glance: An audit of web-crawled multilingual datasets

    Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii- Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, et al. Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for ...

    2022

  31. [39]

    Rephrasing the web: A recipe for compute and data-efficient language modeling

    Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. Rephrasing the web: A recipe for compute and data-efficient language modeling. arXiv preprint arXiv:2401.16380, 2024

    2024 arXiv

  32. [40]

    Preprocessing arabic text on social media

    Mohamed Osman Hegazi, Yasser Al-Dossari, Abdullah Al-Yahy, Abdulaziz Al-Sumari, and Anwer Hilal. Preprocessing arabic text on social media. Heliyon, 7(2), 2021

    2021

  33. [41]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  34. [42]

    Ccnet: Extracting high quality monolingual datasets from web crawl data

    Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359, 2019

    1911 arXiv

  35. [43]

    Constitutional ai: Harmlessness from ai feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022

    2022 arXiv

  36. [44]

    Aligner: Achieving efficient alignment through weak-to-strong correction

    Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, and Yaodong Yang. Aligner: Achieving efficient alignment through weak-to-strong correction. arXiv preprint arXiv:2402.02416, 2024

    2024 arXiv

  37. [45]

    Llamafactory: Unified efficient fine-tuning of 100+ language models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System ...

    2024

  38. [46]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020. 13 A Alignment Data Processing Details The system prompt us...

    2020

  39. [47]

    The data used for native alignment and DPO are not of the same scale

  40. [48]

    Native alignment and DPO are complementary methods that operate at different stages rather than being exclusive. D.1 Experiment Settings We utilized the LLaMA-Factory framework [45], employing LLaMA3-Tamed-8B as the backbone for the experimental group focusing on native alignm...

  41. [49]

    Read the text carefully

  42. [50]

    Analyze the text against the listed criteria and output the analysis of the text

  43. [51]

    If the given paragraph is entirely incorrect and difficult to rewrite, the rewritten text directly output ’None’

  44. [52]

    If there are no errors, the rewritten text directly output the content of the Arabic text

  45. [53]

    Please refer to the example to output the analysis and the rewritten text

  46. [54]

    for example: ### Arabic text data rewriting Arabic text: á ‚j JË éJ ËAÓ XP@ñ Ó © Ôg

    Please do not output any content after the Rewritten text. for example: ### Arabic text data rewriting Arabic text: á ‚j JË éJ ËAÓ XP@ñ Ó © Ôg. ð ñ Ò JË@ é«PA‚ ÖÏ@ éK Qå ”mÌ'@ ‡£A JÖÏ@ ú ¯ àA¾‚Ë@ HAg. AJ Jk@ éJ J.Ê K éJ ªK. AJ J.Ó@ P. HAÓY mÌ'@ YJ ® J K Analysis: Grammar and S...

  47. [55]

    Native alignment can provide higher quality data than traditional data cleaning

  48. [56]

    Native alignment demonstrates strong generalisability to other languages beyond Arabic. F Additional Experiment III: Seed Data Selection To investigate the impact of seed data selection on the performance of the trained alignment model, we conducted an additional experiment. T...

  49. [57]

    Experiment Group 1 (high-ppl): This group consisted of data with a large decrease in text perplexity scores after rewriting, indicating significant changes in the data

  50. [58]

    Experiment Group 2 (low-ppl): This group consisted of data with minimal differences between the original and rewritten texts, according to text perplexity score, indicating no significant changes

  51. [59]

    The variance and average of these experiments are reported as ‘random (x3)’

    Baseline (random): We conducted three random sample seed data experiments to account for randomness, labeled as ‘random-1’, ‘random-2’, and ‘random-3’. The variance and average of these experiments are reported as ‘random (x3)’. All datasets consisted of 1,000 samples of pre-t...

  52. [60]

    Accuracy of information

  53. [61]

    Advertisement removal

  54. [62]

    Level of detail Each aspect receives a score on a scale of 1 to 10, where a higher score indicates better over performance in this aspect. And please return the score by using this format: Formatting: score Accuracy of information: score Content moderation: score Advertisement...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.