REVIEW 4 major objections 7 minor 34 references
Using Scaling Laws for Data Source Utility Estimation in Domain-Specific Pre-Training
T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that data source rankings are not scale-invariant, and that per-source scaling curves fitted from short annealing runs predict which data acquisition strategy will win at larger compute budgets.
desk verdict A practical scaling-law framework for choosing data sources, with a real rank-reversal observation but an extrapolated crossover that needs validation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the per-source utility scaling curve. For each data source $D_i$, the utility $\Delta_i$ is computed as the performance difference between a 10%-upsampled annealing run and a full-replay annealing run at six token budgets, then fit to $\Delta_i(c_i)=a_i+b_i\log(c_i)$ in compute cost $c_i$ (FLOPs, either curation-only or curation plus annealing). The fitted intercept $a_i$ captures small-scale effectiveness and the slope $b_i$ captures long-run returns; the curves are extrapolated beyond the tested range (dotted regions of Fig. 5) to compare sources at the deployment budget. This extends the point-estimate micro-annealing baseline to a scaling-aware evaluation.
What would settle it
Run additional 10%-upsampling anneals at 150B and 300B tokens for the two medical sources with diverging extrapolations, MBF and WRAP, and compare measured Brier-score deltas to full replay against the fitted log-linear lines; deviations larger than seed-to-seed noise would reject the assumed functional form. A cheaper held-out version is to fit the curves using only the 2.1B to 18.9B token runs and check whether they predict the 37.8B and 75.6B observations.
Extended reading notes
Core claim
The paper's central claim is that the utility of a data source in late-stage annealing is itself a quantity that scales, and that its scaling can be estimated cheaply enough to guide acquisition decisions. Utility is defined as the delta between a run that upsamples the candidate source at 10% and a matched full-replay run, $\Delta_i = S_{\mathrm{base}} - S_D$, evaluated with a continuous metric such as Brier Score. From annealing runs at 2.1B, 4.2B, 8.4B, 18.9B, 37.8B, and 75.6B total tokens, the paper computes $\Delta_i$ for each source, fits $\Delta_i(c_i)=a_i+b_i\log(c_i)$ against FLOPs, and extrapolates. The experiments show rank reversals: WRAP beats MBF at small scale in the medical domain, and the order flips as compute grows; TinyGSM beats TinyGSM-MIND at small compute, while MIND scales better in the math domain. The paper concludes that point estimates from micro-annealing can mislead, and that per-source scaling curves enable cost-aware comparison of acquisition methods.
Load-bearing premise
The load-bearing assumption is that each source's utility follows a straight line when plotted against the logarithm of compute, and that the line fitted on runs up to 75 billion tokens continues to hold beyond it; if the true curves bend, the predicted rank reversals and budget splits lose their support.
Editorial extensions
If this is right
- A single micro-annealing point estimate can recommend the wrong source: in the medical experiments WRAP looks better than MBF at low compute, while the scaling curves favor MBF at larger budgets.
- Per-source scaling curves can be obtained with a modest ladder of annealing runs (2.1B to 75B tokens at 10% upsampling) on a 7B model, making scaling-aware acquisition planning feasible in practice.
- Including annealing compute in the cost changes absolute comparisons but preserves the framework; the paper gives both curation-only and curation-plus-annealing analyses.
- Data-source rankings are not portable across evaluation formats: instruction augmentation looks useful in multiple-choice format but does not beat full replay on continuation-format tasks, so utility curves should be fit to the metric that matches the deployment task.
- Since any mixture of sources can be treated as a single source, the same curves can be used to allocate a total budget across sources or to score candidate mixtures before collecting data.
Reading between the lines
- If rank reversals like WRAP versus MBF are common, then single-budget comparisons reported in the data-curation literature should be read as provisional; the annealing ladder could serve as a cheap pre-screen before large synthetic-data commitments.
- The log-linear extrapolation can be stress-tested on the paper's own data by fitting only the small-budget runs and checking whether the remaining 75B-token observations fall on the predicted lines, before trusting longer extrapolations.
- The mixture rule sketched in the appendix, allocating each source a share of the total budget proportional to its fitted slope $b_i$, is a testable consequence: if source utilities interact, measured optimal mixture weights should deviate from $b_i/\sum_j b_j$.
- The fitted curves are tied to the base model and its pretraining corpus, so the same ladder repeated on a smaller or larger base model would reveal how much of the scaling behavior transfers across model families.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a scaling-law framework for estimating the utility of domain-specific data sources during late-stage annealing of a pretrained LLM. Instead of relying on a single point estimate (micro-annealing), the authors run multiple annealing durations with a fixed 10% upsampling of each candidate source, measure the performance delta relative to a full-replay baseline, and fit log-linear utility curves as a function of data curation or total compute. The method is demonstrated with a 7B-parameter model on medical and math domains, comparing model-based filtering (MBF), WRAP-style rephrasing, instruction augmentation, and synthetic math datasets. The main claimed findings are that data-source rankings are not invariant across token scales (e.g., WRAP is better at low compute but MBF overtakes it at scale) and that the fitted scaling curves can guide cost-effective data acquisition decisions.
Significance. If the central claims hold, the paper would make a useful practical contribution: it addresses a real gap in how practitioners choose among data acquisition strategies before committing large budgets, and it provides a concrete empirical procedure (multiple short annealing runs) that is far cheaper than full-scale validation. The paper's cost modeling is unusually detailed, and the observation that rankings can flip with compute is a valuable caution against point-estimate-based decisions. However, several load-bearing aspects of the validation are missing, so the current evidence does not yet support the extrapolated acquisition recommendations.
major comments (4)
- [Section 4.1, Fig. 5, Section 3.1] The central recommendation to prefer MBF over WRAP at large compute rests on a crossover that lies in the extrapolated region for WRAP. Section 3.1 states that the longest WRAP annealing run contains only 3.8B upsampled tokens (18,000 steps), while MBF and Instr.Aug. run to 7.5B tokens (36,000 steps); the dotted portions of Fig. 5 extend WRAP's fitted curve beyond all observed WRAP data. Because R² only measures fit to the same points used to fit the curve, the in-sample R²=0.9 for WRAP does not validate the extrapolated crossover. The claim in Section 1 that scaling curves 'predict the utility of data source D_i at scale' is therefore not supported for the paper's leading example.
- [Section 3.3, Section 4.3] Only the full replay baseline is averaged over two seeds; all data-source runs use a single seed. The paper itself acknowledges in Section 4.3 that methods such as WRAP show greater variability at low scales and that outliers can distort scaling-law coefficients. Without confidence intervals on the fitted slopes, the crossover location in Fig. 5 is not statistically grounded, and the rank non-invariance claim could be an artifact of seed noise at the small-scale points. Multi-seed averaging or at least bootstrap-based error estimates are needed for the curves that drive acquisition decisions.
- [Section 4.1, Fig. 5 (bottom row)] Two of the fitted math-domain curves have R²=0.1 (MBF and TinyGSM, curation-cost panel), i.e., essentially no scaling relationship is present, yet the text states that 'a similar, yet less pronounced effect' is observed on the math domain and uses these fits to draw cost-utility conclusions. The cross-domain claim that scaling curves can be constructed per data source is not supported by the math-domain fits for these methods; the paper should either report the fits that are actually reliable or qualify the conclusion to the medical domain.
- [Section 2.2, Appendix B] The scaling-law form Delta_i(c_i)=a_i+b_i log(c_i) is imposed a priori and is evaluated only in-sample; the dotted extrapolations in Fig. 5 are the same fitted lines continued. The paper should provide a holdout check of the extrapolation (e.g., fit to runs up to 18k steps and predict the 36k-step MBF/Instr.Aug. results, or leave-one-out evaluation) and report uncertainty on extrapolated values. As written, the 'predictions' are restatements of the fitted parameters, which is a circularity concern for the central claim.
minor comments (7)
- [Section 3.3 and Appendix D.1] The 10% upsampling ratio and the MBF threshold of 2.5 are tuned on the medical domain and then applied to the math domain and to other sources; a sensitivity analysis over these choices would strengthen the cross-domain claims, and the main text should state this transfer more prominently.
- [Appendix D.1] The heading reads 'Mode-based filtering details' and should be 'Model-based filtering details'.
- [Section 3] There is a typo in 'open souyrce models' in the base-model paragraph.
- [Section 3.1] The phrase 'it's dialogue augmented version' mixes a possessive with an apostrophe; it should be 'its dialogue-augmented version'.
- [Fig. 14 captions] Both captions in Fig. 14 say 'Medical MMLU CF tasks' and 'Medical MC tasks', but the curves shown are for the math domain (TinyGSM and TinyGSM-MIND) and should be labeled accordingly.
- [Appendix B] There is a typo 'maximum commute budget' (presumably 'compute budget') and 'scaling lows b_i' (presumably 'scaling laws b_i').
- [Section 2.1] The notation in Eq. (1) is confusing: the budget is denoted C while the per-token cost is c_i, and C×c_i^{-1} is easy to misread as a product; a clearer expression such as C/c_i would help.
Circularity Check
No significant circularity: the scaling-law predictions are explicit extrapolations of empirical fits, not derived from the very quantities they predict, and the only self-citation concerns non-load-bearing infrastructure.
full rationale
The paper's derivation chain is an empirical fitting procedure, not a derivation from first principles: U(D_i(n_i)) is measured at six token budgets (Eq. 2), a log-linear model Delta_i(c_i)=a_i+b_i log(c_i) is fitted to those measurements (Section 2.2, Appendix B), and the fitted lines are extended as dotted curves in Fig. 5. The extrapolated 'predictions' are therefore functions of the fitted parameters, but this is standard statistical extrapolation, not circularity: the paper never claims the dotted regions were validated by held-out data, and Section 4.3 explicitly concedes that scaling-law generalization to much larger budgets was not tested. The central empirical observation of rank non-invariance is grounded in the raw annealing measurements shown in Fig. 1. The only self-citation (Fast-LLM, Lamy Poirier et al. 2024) concerns the training framework, not any load-bearing scientific claim, and no uniqueness theorem or ansatz is imported from the authors' prior work. Concerns about the unsupported extrapolation of WRAP's curve beyond its 3.8B-token observed range are correctness/risk issues, not circularity.
Assumptions & free parameters
free parameters (4)
- Upsampling ratio =
0.1 (10%)
- MBF classifier threshold =
2.5
- Scaling law intercept a_i and slope b_i per source =
Not reported numerically; reported R-squared values range from 0.1 to 0.9
- Inference cost coefficient =
2 x |P| FLOPs per token
assumptions (5)
- ad hoc to paper Utility scaling is log-linear in compute: Delta_i(c_i) = a_i + b_i log(c_i).
- domain assumption Additive independence of per-source utility in mixtures: Delta_mixture = sum_i Delta_i.
- domain assumption The 10% upsampling ratio generalizes across sources and scales.
- domain assumption Brier Score on continuation-format MMLU tasks is a robust proxy for domain utility.
- domain assumption The 168B-token annealing checkpoint (50% learning rate decay) is a representative starting point for utility estimates.
Cite this review
Pith. "Pith review of Using Scaling Laws for Data Source Utility Estimation in Domain-Specific Pre-Training." pith.science (2026). https://pith.science/paper/U6DBM6LR
@misc{pith2026250722250,
author = {Pith},
title = {Pith review of: Using Scaling Laws for Data Source Utility Estimation in Domain-Specific Pre-Training},
year = {2026},
howpublished = {\url{https://pith.science/paper/U6DBM6LR}},
note = {Machine review of arXiv:2507.22250}
}
read the original abstract
We introduce a framework for optimizing domain-specific dataset construction in foundation model training. Specifically, we seek a cost-efficient way to estimate the quality of data sources (e.g. synthetically generated or filtered web data, etc.) in order to make optimal decisions about resource allocation for data sourcing from these sources for the stage two pre-training phase, aka annealing, with the goal of specializing a generalist pre-trained model to specific domains. Our approach extends the usual point estimate approaches, aka micro-annealing, to estimating scaling laws by performing multiple annealing runs of varying compute spent on data curation and training. This addresses a key limitation in prior work, where reliance on point estimates for data scaling decisions can be misleading due to the lack of rank invariance across compute scales -- a phenomenon we confirm in our experiments. By systematically analyzing performance gains relative to acquisition costs, we find that scaling curves can be estimated for different data sources. Such scaling laws can inform cost effective resource allocation across different data acquisition methods (e.g. synthetic data), data sources (e.g. user or web data) and available compute resources. We validate our approach through experiments on a pre-trained model with 7 billion parameters. We adapt it to: a domain well-represented in the pre-training data -- the medical domain, and a domain underrepresented in the pretraining corpora -- the math domain. We show that one can efficiently estimate the scaling behaviors of a data source by running multiple annealing runs, which can lead to different conclusions, had one used point estimates using the usual micro-annealing technique instead. This enables data-driven decision-making for selecting and optimizing data sources.
Figures
Figures from the paper (11 more)
Reference graph
Works this paper leans on
-
[1]
Nemotron-4 340b technical report.arXiv preprint arXiv:2406.11704,
Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. Nemotron-4 340b technical report.arXiv preprint arXiv:2406.11704,
-
[4]
Hritik Bansal, Arian Hosseini, Rishabh Agarwal, Vinh Q Tran, and Mehran Kazemi
Retrieved from https://jax-ml.github.io/scaling-book/. Hritik Bansal, Arian Hosseini, Rishabh Agarwal, Vinh Q Tran, and Mehran Kazemi. Smaller, weaker, yet better: Training llm reasoners via compute-optimal sampling.arXiv preprint arXiv:2408.16737,
-
[5]
Does your data spark joy? performance gains from domain upsampling at the end of training
Cody Blakeney, Mansheej Paul, Brett W Larsen, Sean Owen, and Jonathan Frankle. Does your data spark joy? performance gains from domain upsampling at the end of training. arXiv preprint arXiv:2406.03476,
-
[8]
Adapting large language models to domains via reading comprehension.arXiv preprint arXiv:2309.09530,
Daixuan Cheng, Shaohan Huang, and Furu Wei. Adapting large language models to domains via reading comprehension.arXiv preprint arXiv:2309.09530,
-
[10]
Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,
-
[11]
Data filtering networks.arXiv preprint arXiv:2309.17425,
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks.arXiv preprint arXiv:2309.17425,
-
[12]
URL https://zenodo.org/records/12608602. Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge. Zamba: A compact 7b ssm hybrid model.arXiv preprint arXiv:2405.16712,
-
[13]
Nathan Godey, Éric de la Clergerie, and Benoît Sagot. Why do small language models underperform? studying language model saturation via the softmax bottleneck.arXiv preprint arXiv:2404.07647,
Show all 34 references
-
[14]
Xu Guo and Han Yu
URLhttps://arxiv.org/abs/2407.21783. Xu Guo and Han Yu. On the domain adaptation and generalization of pretrained language models: A survey.arXiv preprint arXiv:2211.03154,
-
[15]
Mistral 7b
Albert Q Jiang, A Sablayrolles, A Mensch, C Bamford, D Singh Chaplot, Ddl Casas, F Bressand, G Lengyel, G Lample, L Saulnier, et al. Mistral 7b. arxiv.arXiv preprint arXiv:2310.06825, 10,
-
[16]
Adaptive data optimization: Dynamic sample selection with scaling laws.arXiv preprint arXiv:2410.11820,
Yiding Jiang, Allan Zhou, Zhili Feng, Sadhika Malladi, and J Zico Kolter. Adaptive data optimization: Dynamic sample selection with scaling laws.arXiv preprint arXiv:2410.11820,
-
[17]
Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,
2001 arXiv
-
[18]
Downstream datasets make surprisingly good pretraining corpora.arXiv preprint arXiv:2209.14389,
Kundan Krishna, Saurabh Garg, Jeffrey P Bigham, and Zachary C Lipton. Downstream datasets make surprisingly good pretraining corpora.arXiv preprint arXiv:2209.14389,
-
[19]
com/ServiceNow/Fast-LLM
URL https://github. com/ServiceNow/Fast-LLM. Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language mod...
-
[21]
Regmix: Data mixture as regression for language model pre-training.arXiv preprint arXiv:2407.01492, 2024b
Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training.arXiv preprint arXiv:2407.01492, 2024b. Ilya Loshchilov and Frank Hutter. Decoupled weight decay reg...
-
[22]
Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly
URL https://huggingface.co/datasets/ HuggingFaceFW/fineweb-edu. Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. Rephrasing the web: A recipe for compute and data-efficient language modeling.arXiv preprint arXiv:2401.16380,
-
[23]
Clara Na, Ian Magnusson, Ananya Harsh Jha, Tom Sherborne, Emma Strubell, Jesse Dodge, and Pradeep Dasigi
URL https://arxiv.org/abs/2409.02060. Clara Na, Ian Magnusson, Ananya Harsh Jha, Tom Sherborne, Emma Strubell, Jesse Dodge, and Pradeep Dasigi. Scalable data ablation approximations for language models through modular training and merging.arXiv preprint arXiv:2410.15661,
-
[24]
2 olmo 2 furious.arXiv preprint arXiv:2501.00656,
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious.arXiv preprint arXiv:2501.00656,
-
[25]
Data, data everywhere: A guide for pretraining dataset construction.arXiv preprint arXiv:2407.06380,
Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Bo Liu, Aastha Jhunjhunwala, Zhilin Wang, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Data, data everywhere: A guide for pretraining dataset construction.arXiv preprint arXiv:2407.06380,
-
[26]
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo
URL https://arxiv.org/ abs/1910.02054. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage?Advances in Neural Information Processing Systems, 36:55565– 55581,
1910 arXiv
-
[27]
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,
-
[28]
Pe- ters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Mu...
-
[29]
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115,
-
[30]
Data mixing laws: Optimizing data mixtures by predicting language modeling performance
Jiasheng Ye, Peiju Liu, Tianxiang Sun, Yunhua Zhou, Jun Zhan, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint arXiv:2403.16952,
-
[31]
It is trained with AdamW (Loshchilov & Hutter, 2017), using a sequence length of 8192 tokens and 256 sequences par minibatch, for a total of 2.1M tokens
and uses the same tokenizer. It is trained with AdamW (Loshchilov & Hutter, 2017), using a sequence length of 8192 tokens and 256 sequences par minibatch, for a total of 2.1M tokens. We use β1 = 0.9 and β2 = 0.95 as first and second moments, respectively. The training is done ...
2017
-
[32]
and ZeRO stage 3 (Rajbhandari et al., 2020), and train the model on 64 H100 GPUs with full data parallelization, for a total duration of 32,500 H100-hours, averaging 10,000 tokens/s/GPUs. Default Pretraining Mix:Our pretraining dataset is the concatenation of the Dolma (Sol- d...
2024
-
[33]
We also conducted ablations on classifier training, comparing binary classification with regression and exploring up-sampling vs
6 For the final classifier, we used 500K annotations from Llama3- 70B. We also conducted ablations on classifier training, comparing binary classification with regression and exploring up-sampling vs. down-sampling in the medical domain. Regression performed best in annealing ...
2024
-
[34]
Question:
and TINYGSM- MIND (OLMo et al., 2024), we make the following simplifying assumptions: 17 • As before, we assume the inference cost per token is 2 × |P|, following (Kaplan et al., 2020). • TINYGSM uses GPT-3.5 to generate 12.3M synthetic math problems with Python solutions. Ass...
2015
-
[1950]
Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,
1901
-
[2015]
Scaling- filter: Assessing data quality through inverse utilization of scaling laws.arXiv preprint arXiv:2408.08310, 2024b
Ruihang Li, Yixuan Wei, Miaosen Zhang, Nenghai Yu, Han Hu, and Houwen Peng. Scaling- filter: Assessing data quality through inverse utilization of scaling laws.arXiv preprint arXiv:2408.08310, 2024b. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggan...
-
[2020]
Scaling parameter- constrained language models with quality data.arXiv preprint arXiv:2410.03083,
Ernie Chang, Matteo Paltenghi, Yang Li, Pin-Jie Lin, Changsheng Zhao, Patrick Huber, Zechun Liu, Rastislav Rabatin, Yangyang Shi, and Vikas Chandra. Scaling parameter- constrained language models with quality data.arXiv preprint arXiv:2410.03083,
-
[2023]
Instruction pre-training: Language models are supervised multitask learners.arXiv preprint arXiv:2406.14491,
Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, and Furu Wei. Instruction pre-training: Language models are supervised multitask learners.arXiv preprint arXiv:2406.14491,
-
[2024]
Syeda Nahida Akter, Shrimai Prabhumoye, John Kamalu, Sanjeev Satheesh, Eric Nyberg, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro
URL https://github.com/meta-llama/llama3/ blob/main/MODEL_CARD.md. Syeda Nahida Akter, Shrimai Prabhumoye, John Kamalu, Sanjeev Satheesh, Eric Nyberg, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Mind: Math informed synthetic dialogues for pretraining llms.arXiv pre...
-
[2025]
URL https://arxiv.org/abs/ 2411.04425. AI@Meta. Llama 3 model card.online,
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.