Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

AstroMLab 3: Achieving GPT-4o Level Performance in Astronomy with a Specialized 8B-Parameter Large Language Model

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An 8-billion-parameter model specialized in astronomy matches GPT-4o's 80.4% on the AstroMLab-1 benchmark, scoring 80.9% at a fraction of the inference cost.

desk verdict Real engineering result, but the headline number is tuned to its own test instrument; external evaluation needed before GPT-4o parity is credible. read the letter →

arxiv 2411.09012 v3 pith:MOFFDPMC submitted 2024-11-13 astro-ph.IM

Tijmen de Haan , Yuan-Sen Ting , Tirthankar Ghosal , Tuan Dung Nguyen , Alberto Accomazzi , Azton Wells , Nesar Ramachandra , Rui Pan
show 1 more author
Zechang Sun
This is my paper · ORCID
classification astro-ph.IM
keywords largelanguagemodeldomainspecializationcontinuedpretrainingsupervisedfine-tuningmergingastronomyknowledgerecallAstroMLabbenchmarkscienceAIassistant
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a small, domain-specialized language model can match a flagship general-purpose model on astronomy knowledge recall. The authors build AstroSage-Llama-3.1-8B from Llama-3.1-8B by continued pretraining on roughly 250,000 arXiv preprints, Wikipedia articles, and textbooks, followed by supervised fine-tuning on millions of synthetic question-answer pairs and a targeted model merge. On the 4,425-question AstroMLab-1 benchmark the model scores 80.9%, outperforming every other 8-billion-parameter model and landing at parity with GPT-4o at about one-thousandth of the inference cost. If the result holds, domain specialization becomes a credible path to capable, affordable scientific AI assistants.

What carries the argument

The load-bearing mechanism is a three-stage training pipeline: continued pretraining on a cleaned 3.3-billion-token corpus of astronomy literature (arXiv astro-ph and gr-qc, Wikipedia, textbooks) using perplexity-based filtering; supervised fine-tuning on 8.8 million LLM-vetted synthetic Q&A pairs plus instruction-tuning data; and a DARE-TIES weight merge that combines the specialized model with Meta-Llama-3.1-8B-Instruct at 0.75/0.25 weights. The merge is what recovers general capabilities while preserving the 8-point astronomy gain.

What would settle it

An independent astronomy benchmark constructed from sources published after 2024 (or from journals never included in the training corpus) on which AstroSage-Llama-3.1-8B fails to outperform Meta-Llama-3.1-8B-Instruct, combined with an n-gram overlap analysis showing AstroMLab-1 questions appear in the non-ARAA training data, would refute the claim that the gain comes from domain specialization rather than benchmark leakage.

Watch

Extended reading notes

Core claim

The central claim is that AstroSage-Llama-3.1-8B achieves 80.9% accuracy on the AstroMLab-1 multiple-choice benchmark, an 8-point improvement over its Meta-Llama-3.1-8B baseline and statistically indistinguishable from GPT-4o's 80.4%. The authors argue this is the first verified demonstration that fine-tuning an astronomical LLM can improve over its starting model, and that the gain comes from broad domain pretraining and large-scale SFT rather than from memorizing benchmark sources, since the Annual Review papers used to generate the benchmark were deliberately excluded from training. They further show that merging the specialized model with Meta's instruct model restores general instruction-following and reasoning abilities without sacrificing astronomical performance.

Load-bearing premise

The AstroMLab-1 benchmark, built by the same research group, is a valid and sufficiently leakage-free measure of astronomy knowledge, so that excluding the ARAA source papers from training is enough to guarantee the 80.9% score reflects generalization rather than memorization; no contamination audit is provided for indirect leakage through arXiv, Wikipedia, textbooks, or the synthetic Q&A pipeline.

Editorial extensions

If this is right

  • An 8B model can match flagship proprietary models on astronomy knowledge recall at roughly one-thousandth the inference cost, making high-quality astronomy Q&A practical at scale.
  • The same CPT+SFT+merge recipe should transfer to other scientific domains, where a compact specialist could outperform a generalist flagship on domain benchmarks.
  • The 3.5-point-per-10x-cost trade-off lines in the evaluation suggest that specialized models can shift the cost-performance frontier by 100- to 1000-fold on niche tasks.
  • Scaling the recipe to a 70B-class model would plausibly reach state-of-the-art astronomy-specific performance.
  • The released weights and the control merging experiment give the community a reproducible baseline for future astronomical LLM work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's benchmark is created by the same group and the only contamination control is excluding the ARAA source papers; a third-party benchmark built from independent sources would test whether the 80.9% reflects generalizable astronomy knowledge rather than indirect leakage through arXiv, Wikipedia, or the synthetic Q&A pipeline.
  • Because the synthetic Q&A pairs are generated from the same arXiv corpus used for pretraining, the model's knowledge is effectively bounded by that corpus; pairing the specialist with retrieval from live literature could extend it beyond 2024.
  • The 68% human expert baseline on AstroMLab-1 suggests the benchmark is becoming saturated for frontier models; future specialized benchmarks should emphasize multi-step reasoning and calculation, where the paper admits 8B models still lag.
  • The merging control experiment (fine-tuning on unrelated multiple-choice questions) implies that the merge transfers formatting and instruction ability rather than knowledge; an analogous study varying the merge ratio could map how much general capability is recoverable before astronomy performance erodes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents AstroSage-Llama-3.1-8B, an 8-billion-parameter language model specialized for astronomy. The authors start from Llama-3.1-8B and perform continued pretraining on roughly 250,000 arXiv preprints, about 30,000 Wikipedia articles, and around 800 textbooks, followed by supervised fine-tuning on 8.8 million synthetic Q&A pairs and a final DARE-TIES merge with Llama-3.1-8B-Instruct. The main evaluation is the AstroMLab-1 benchmark, a 4,425-question multiple-choice astronomy test; the model attains 80.9%, compared with 80.4% for GPT-4o and 72.9% for the base Llama-3.1-8B model. The authors report additional results on six general benchmarks and a small blind human preference test, and they release the model weights. The paper's central claim is that a specialized 8B model can match a flagship proprietary model on astronomy knowledge recall at a fraction of the inference cost.

Significance. If the headline result holds, this is a practically significant contribution: an open-weight 8B model matching GPT-4o on astronomy knowledge recall would make high-quality astronomy Q&A substantially more accessible and would demonstrate that domain specialization can be highly cost-effective. The paper is also valuable for its detailed reporting of training hyperparameters, compute budget, dataset composition, and the deliberate exclusion of ARAA source papers from training. The model weights are released, enabling independent testing. However, the central evaluation is currently not fully convincing as a measure of generalization, for two connected reasons: the benchmark was used to guide model development, and no contamination audit or external validation is provided. These issues directly affect whether the 80.9% score and the parity claim can be taken at face value.

major comments (3)
  1. [Section IV and Section V.A] The headline score is not obtained from an evaluation instrument that was untouched during development. Section IV states that the DARE-TIES merge weights (0.75/0.25) were chosen because they improved performance on the AstroMLab-1 multiple-choice benchmark, and Section V.A then reports the final 80.9% on that same benchmark. Because no held-out split, pre-registered protocol, or external astronomy benchmark is presented, the reported score is an optimistically biased estimate of generalization, and the 8-point improvement over the base model is not cleanly attributable to domain knowledge. I request either a development-untouched evaluation set (e.g., a random holdout created before any model selection) or an independently constructed astronomy QA benchmark before the parity claim is accepted.
  2. [Section II.A, Section III.A, and Section V.A] The exclusion of ARAA source papers does not, by itself, rule out indirect leakage. The training corpus contains roughly 250,000 arXiv preprints, about 30,000 Wikipedia articles, about 800 textbooks, and 8.8 million synthetic Q&A pairs generated from those sources. ARAA review articles are frequently summarized, quoted, or paraphrased in such materials, so the model could have memorized content that is close to the benchmark questions without having seen the exact ARAA source file. The paper reports no n-gram overlap analysis, embedding-similarity audit, canary check, or other contamination analysis. Additionally, AstroMLab-1 was constructed by largely the same author team as the present model. A concrete contamination audit or an external benchmark result is needed to support the claim that the 80.9% score reflects generalization rather than memorization or benchmark-specific tuning.
  3. [Section V.A and Figure 3] The comparison protocol is underspecified. The text says that all scores were 'updated using the latest model versions following the methodology from Ting et al.' but it does not specify the prompt format, number of few-shot examples, decoding temperature, or number of runs averaged for AstroSage-Llama-3.1-8B or for the comparison models. Since the central claim is parity with GPT-4o (80.9% versus 80.4%, within the Wilson interval shown), the evaluation harness must be described precisely so that the comparison is apples-to-apples and reproducible. Please report the exact evaluation code, prompts, and uncertainty treatment for all models.
minor comments (6)
  1. [Section III heading] The heading 'SUPER VISED FINE-TUNING' contains a typo; it should read 'SUPERVISED FINE-TUNING'.
  2. [Section V.B] BBH is described as 'binary hypothesis testing,' but BBH refers to Big-Bench Hard; the description should be corrected.
  3. [Section V.C] The blind preference study uses 15 questions and 3 evaluators, with a 73% preference for AstroSage; please report per-question and per-evaluator counts, as well as a confidence interval, since the effective sample size is small.
  4. [Section III.A] The statement that a small number of LLM-generated Q&A quality scores were 'verified and confirmed to be sufficiently accurate' is vague; please specify how many pairs were checked, by whom, and with what agreement.
  5. [Section VI] The code is only available 'upon reasonable request' and the synthetic dataset will be released only after the research trajectory is complete; this limits reproducibility of the core data-creation pipeline. At minimum, a sample of the synthetic Q&A pairs and the cleaning scripts should be released with the paper.
  6. [Figure 2] The learning-rate axis in the caption appears garbled ('2 × 10^5 ... 10^4 1.2 × 10^4'); please correct the formatting and label the units of the learning rate.

Circularity Check

1 steps flagged · score 6.0 of 10

AstroMLab-1 was used to select the final merge weights, so the headline 80.9% is a post-tuning score rather than an independent evaluation; external benchmarks are needed for the GPT-4o parity claim.

  1. fitted input called prediction [Section IV (Model Merging); Section V.A (Evaluation)]
    "The merge was performed at full density, BF16 precision, and with the weight parameters set to 0.75 and 0.25 for AstroSage-Llama-3.1-8B-SFT and Meta-Llama-3.1-8B-Instruct, respectively. The resulting merged model exhibits enhanced instruction-following capabilities and improved performance on the AstroMLab-1 multiple-choice question benchmark in both few-shot and structured output scenarios."

    Section IV presents the final DARE-TIES merge weights (0.75/0.25) and states that the resulting model shows improved performance on the AstroMLab-1 benchmark. Section V.A then reports the headline result on that same benchmark: 80.9%, 'comparable to OpenAI's latest flagship models (GPT-4o: 80.4%).' Because the benchmark score was the selection criterion for the final configuration, the 80.9% is a post-model-selection statistic, not an out-of-sample measurement: it is the value the development loop optimized. Excluding the ARAA papers prevents direct memorization of the benchmark's origin but does not remove selection bias from tuning merge weights on the same 4,425 questions. Comparing against GPT-4o, which was not tuned on AstroMLab-1, is biased in AstroSage's favor by construction.

full rationale

The principal circular step is the dual use of AstroMLab-1 as both the model-selection criterion and the headline evaluation. Section IV states the final DARE-TIES merge weights were adopted in a configuration whose reported justification is improved performance on the AstroMLab-1 multiple-choice benchmark; Section V.A reports 80.9% on that same benchmark as evidence of GPT-4o-level performance. This is test-set contamination through model selection: the reported score is a post-selection estimate, and comparisons against models not tuned on the benchmark are biased in AstroSage's favor by construction. The deliberate exclusion of the ARAA source papers addresses memorization of the benchmark's origin, but not the selection bias, since the benchmark itself guided development choices. The benchmark was also created by the same research group and is described as the only comprehensive astronomy benchmark, which compounds the concern; however, the benchmark is a human-vetted dataset, the model is released, and the paper reports general benchmarks and a blind preference test, so the work retains independent content beyond the fitted score. The paper's own Section VII acknowledges that the field lacks standardized astronomy-specific benchmarks, reinforcing that no external validation of the headline parity claim is provided. Score 6: the central numerical claim is partially circular because it is a fitted value with respect to the evaluation instrument, but the training pipeline and ancillary evaluations keep this from being a complete reduction.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the validity of the AstroMLab-1 benchmark, the sufficiency of the ARAA exclusion to prevent leakage, and the quality of LLM-generated synthetic training data. Six training hyperparameters or filtering thresholds are effectively fitted to data or chosen by hand. No new physical entities are postulated.

free parameters (6)
  • Model merge weights (AstroSage-SFT vs Llama-3.1-8B-Instruct) = 0.75 / 0.25
    Chosen based on observed improvements on AstroMLab-1; no held-out validation described (Section IV).
  • Perplexity cleaning threshold = top 2% of paragraphs removed
    Manually chosen from perplexity histogram and inspection (Section II.C, Fig. 1).
  • Warmup steps = 40 samples
    Determined empirically from an earlier canceled CPT run (Section II.B).
  • Peak learning rate = 1.5e-4 (CPT), 1e-4 (SFT)
    Extrapolated from smaller runs and tuned to avoid exploding gradients (Section II.B, Tables I and II).
  • SFT quality filter threshold = only perfect scores on all four criteria kept
    LLM-as-judge filtering of 11M synthetic QA pairs; only a small number verified (Section III.A).
  • Infinity-Instruct filtering = at least 70% alphanumeric and keyword exclusions
    Ad hoc filter for instruction-following data (Section III.A).
assumptions (5)
  • domain assumption AstroMLab-1 is a valid and transferable measure of astronomical knowledge
    The paper uses it as primary benchmark and generalizes from it to claims about astronomy capability (Section V.A).
  • domain assumption Excluding ARAA papers from training prevents benchmark contamination
    Paper asserts exclusion but provides no contamination audit for indirect leakage through other training sources (Section V.A).
  • domain assumption LLM-generated synthetic QA pairs and LLM-based ratings are sufficiently accurate to serve as training signal
    Only a small number of ratings were manually verified (Section III.A).
  • domain assumption Llama-3.1-8B is a suitable base model and its instruct version contributes general capabilities via merging
    Chosen for license and strong baseline performance (Sections II and IV).
  • domain assumption Nougat OCR faithfully renders PDFs into Markdown for continued pretraining
    No quality metrics for OCR conversion beyond perplexity cleaning (Section II.A).

how reviews work

0 comments
Cite this review

Pith. "Pith review of AstroMLab 3: Achieving GPT-4o Level Performance in Astronomy with a Specialized 8B-Parameter Large Language Model." pith.science (2026). https://pith.science/paper/MOFFDPMC

@misc{pith2026241109012,
  author       = {Pith},
  title        = {Pith review of: AstroMLab 3: Achieving GPT-4o Level Performance in Astronomy with a Specialized 8B-Parameter Large Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MOFFDPMC}},
  note         = {Machine review of arXiv:2411.09012}
}
read the original abstract

AstroSage-Llama-3.1-8B is a domain-specialized natural-language AI assistant tailored for research in astronomy, astrophysics, cosmology, and astronomical instrumentation. Trained on the complete collection of astronomy-related arXiv papers from 2007 to 2024 along with millions of synthetically-generated question-answer pairs and other astronomical literature, AstroSage-Llama-3.1-8B demonstrates remarkable proficiency on a wide range of questions. AstroSage-Llama-3.1-8B scores 80.9% on the AstroMLab-1 benchmark, greatly outperforming all models -- proprietary and open-weight -- in the 8-billion parameter class, and performing on par with GPT-4o. This achievement demonstrates the potential of domain specialization in AI, suggesting that focused training can yield capabilities exceeding those of much larger, general-purpose models. AstroSage-Llama-3.1-8B is freely available, enabling widespread access to advanced AI capabilities for astronomical education and research.

Figures

Figures reproduced from arXiv: 2411.09012 by the authors.

Figure 1
Figure 1. FIG. 1. Histogram of paragraph-level perplexity used [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. FIG. 2. Supervised fine-tuning loss curve. The learning rate schedule, shown as the dashed brick red curve, follows a quadratic [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. FIG. 3. Performance comparison on the AstroMLab 1 benchmark, which contains 4,425 high-quality, human-verified multiple [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: FIG. 4. Performance comparison across general language model benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Spectroscopic Binary Detection as Agent-Callable Tools: Detecting 40,000+ Main-Sequence Binary Candidates from SDSS DR19 APOGEE Spectra

    astro-ph.SR 2026-08 conditional novelty 7.0 of 10

    The authors apply El-Badry's binary decomposition via agent-callable tools to APOGEE DR19, flagging 41,466 SB2 candidates with about 40% estimated contamination, and measuring no eccentricity excess for close twins.

  2. Multi-Agent System for Cosmological Parameter Analysis

    astro-ph.IM 2024-11 conditional novelty 6.0 of 10

    A multi-agent LLM system with retrieval and local code execution reproduced ACT DR6 lensing cosmological parameter constraints without human-written code, and generalized to two research-software tasks.

Reference graph

Works this paper leans on

31 extracted references · 8 canonical work pages · cited by 2 Pith papers

  1. [1]

    Tiny Titans: Can Smaller Large Language Models Punch Above Their Weight in the Real World for Meeting Summarization?

    X.-Y. Fu, M. T. R. Laskar, E. Khasanova, C. Chen and S. B. TN, “Tiny Titans: Can Smaller Large Language Models Punch Above Their Weight in the Real World for Meeting Summarization?” (2024), arXiv:2402.00841

    arXiv 2024

  2. [2]

    Training Compute-Optimal Large Language Models,

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark et al., “Training Compute-Optimal Large Language Models,” (2022), arXiv:2203.15556

    arXiv 2022

  3. [3]

    It’s Not Just Size That Mat- ters: Small Language Models Are Also Few-Shot Learn- ers,

    T. Schick and H. Sch¨ utze, “It’s Not Just Size That Mat- ters: Small Language Models Are Also Few-Shot Learn- ers,” (2021), arXiv:2009.07118

    arXiv 2021

  4. [4]

    Well- Read Students Learn Better: On the Importance of Pre- training Compact Models,

    I. Turc, M.-W. Chang, K. Lee and K. Toutanova, “Well- Read Students Learn Better: On the Importance of Pre- training Compact Models,” (2019), arXiv:1908.08962

    arXiv 2019

  5. [5]

    AstroLLaMA: Towards Specialized Foundation Models in Astronomy,

    T. D. Nguyen, Y.-S. Ting, I. Ciuc˘ a, C. O’Neill, Z.-C. Sun, M. Jab lo´ nska, S. Kruk, E. Perkowski, J. Miller, J. Li et al., “AstroLLaMA: Towards Specialized Foundation Models in Astronomy,” (2023), arXiv:2309.06126v1

    arXiv 2023

  6. [6]

    Perkowski, R

    E. Perkowski, R. Pan, T. D. Nguyen, Y.-S. Ting, S. Kruk, T. Zhang, C. O’Neill, M. Jablonska, Z. Sun, M. J. Smith et al., Research Notes of the AAS 8, 7 (2024), publisher: The American Astronomical Society

    work page 2024

  7. [7]

    R. Pan, T. D. Nguyen, H. Arora, A. Accomazzi, T. Ghosal and Y.-S. Ting, in SC24-W: Workshops of the International Conference for High Performance Comput- ing, Networking, Storage and Analysis (2024) pp. 87–96

    work page 2024

  8. [8]

    de Haan, Astronomy and Computing 51, 100934 (2025)

    T. de Haan, Astronomy and Computing 51, 100934 (2025)

    work page 2025

Show all 31 references
  1. [9]

    Interpreting Multi-band Galaxy Observations with Large Language Model-Based Agents,

    Z. Sun, Y.-S. Ting, Y. Liang, N. Duan, S. Huang and Z. Cai, “Interpreting Multi-band Galaxy Observations with Large Language Model-Based Agents,” (2024), arXiv:2409.14807

    2024 arXiv

  2. [10]

    The Llama 3 Herd of Models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al- Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al., “The Llama 3 Herd of Models,” (2024), arXiv:2407.21783

    2024 arXiv

  3. [11]

    AstroM- Lab 1: Who Wins Astronomy Jeopardy!?

    Y.-S. Ting, T. D. Nguyen, T. Ghosal, R. Pan, H. Arora, Z. Sun, T. de Haan, N. Ramachandra, A. Wells, S. Madireddy and A. Accomazzi, “AstroM- Lab 1: Who Wins Astronomy Jeopardy!?” (2024), arXiv:2407.11194v1

    2024 arXiv

  4. [12]

    Scaling Language Models: Methods, Analysis & Insights from Training Gopher,

    J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoff- mann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young et al., “Scaling Language Models: Methods, Analysis & Insights from Training Gopher,” (2021), arXiv:2112.11446v2

    2021 arXiv

  5. [13]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Ka- plan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., in Advances in Neural Information Pro- cessing Systems , Vol. 33 (Curran Associates, Inc., 2020) pp. 1877–1901

    2020

  6. [14]

    The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only,

    G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru, A. Cappelli, H. Alobeidli, B. Pannier, E. Almazrouei and J. Launay, “The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only,” (2023), arXiv:2306.01116v1

    2023 arXiv

  7. [15]

    ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Laws,

    R. Li, Y. Wei, M. Zhang, N. Yu, H. Hu and H. Peng, “ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Laws,” (2024), arXiv:2408.08310v1

    2024 arXiv

  8. [16]

    Nougat: Neural Optical Understanding for Academic Documents,

    L. Blecher, G. Cucurull, T. Scialom and R. Stojnic, “Nougat: Neural Optical Understanding for Academic Documents,” (2023), arXiv:2308.13418

    2023 arXiv

  9. [17]

    tiktoken,

    “tiktoken,” (2022), https://github.com/openai/tiktoken

    2022

  10. [18]

    Optimizing Dis- tributed Training on Frontier for Large Language Mod- els,

    S. Dash, I. Lyngaas, J. Yin, X. Wang, R. Egele, G. Cong, F. Wang and P. Balaprakash, “Optimizing Dis- tributed Training on Frontier for Large Language Mod- els,” (2023), arXiv:2312.12705v2

    2023 arXiv

  11. [19]

    Megatron-DeepSpeed,

    “Megatron-DeepSpeed,” (2021), https://github.com/deepspeedai/Megatron-DeepSpeed

    2021

  12. [20]

    Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Gen- erative Language Model,

    S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajb- handari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti et al., “Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Gen- erative Language Model,” (2022), arXiv:2201.11990

    2022 arXiv

  13. [21]

    BAAI/Infinity-Instruct · Datasets at Hugging Face,

    Beijing Academy of Artificial Intelligence (BAAI), “BAAI/Infinity-Instruct · Datasets at Hugging Face,” https://huggingface.co/datasets/BAAI/Infinity-Instruct 11 (2024)

    2024

  14. [22]

    What Matters for Model Merging at Scale?

    P. Yadav, T. Vu, J. Lai, A. Chronopoulou, M. Faruqui, M. Bansal and T. Munkhdalai, “What Matters for Model Merging at Scale?” (2024), arXiv:2410.03617

    2024 arXiv

  15. [23]

    Dassanaike-Perera, S

    A. Dassanaike-Perera, S. Waiwitlikhit and K. Gilbai, Cuts and Stitches: Does Model Merging Produce Better Multitask Learners? , Stanford CS224N Default Project (Stanford, 2023)

    2023

  16. [24]

    Arcee’s MergeKit: A Toolkit for Merging Large Language Models,

    C. Goddard, S. Siriwardhana, M. Ehghaghi, L. Meyers, V. Karpukhin, B. Benedict, M. McQuade and J. So- lawetz, “Arcee’s MergeKit: A Toolkit for Merging Large Language Models,” (2024), arXiv:2403.13257

    2024 arXiv

  17. [25]

    Instruction-Following Evaluation for Large Language Models,

    J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou and L. Hou, “Instruction-Following Evaluation for Large Language Models,” (2023), arXiv:2311.07911

    2023 arXiv

  18. [26]

    Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them,

    M. Suzgun, N. Scales, N. Sch¨ arli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou and J. Wei, “Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them,” (2022), arXiv:2210.09261

    2022 arXiv

  19. [27]

    Mea- suring Mathematical Problem Solving With the MATH Dataset,

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song and J. Steinhardt, “Mea- suring Mathematical Problem Solving With the MATH Dataset,” (2021), arXiv:2103.03874

    2021 arXiv

  20. [28]

    GPQA: A Graduate-Level Google-Proof Q&A Benchmark,

    D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael and S. R. Bowman, “GPQA: A Graduate-Level Google-Proof Q&A Benchmark,” (2023), arXiv:2311.12022

    2023 arXiv

  21. [29]

    MuSR: Testing the Limits of Chain- of-thought with Multistep Soft Reasoning,

    Z. Sprague, X. Ye, K. Bostrom, S. Chaudhuri and G. Durrett, “MuSR: Testing the Limits of Chain- of-thought with Multistep Soft Reasoning,” (2023), arXiv:2310.16049v2

    2023 arXiv

  22. [30]

    MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark,

    Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang et al., “MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark,” (2024), arXiv:2406.01574

    2024 arXiv

  23. [31]

    Z. Chen, S. Chen, Y. Ning, Q. Zhang, B. Wang, B. Yu, Y. Li, Z. Liao, C. Wei, Z. Lu et al., (2024), arXiv:2410.05080

    2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.