Pith. sign in

REVIEW 3 major objections 5 minor 5 cited by

Parallel Scaling Law for Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that running a language model in P parallel streams, with the same weights, matches the capacity of scaling parameters by O(log P).

desk verdict A serious empirical scaling-law paper with a load-bearing missing control: the O(log P) equivalence is a fit, not yet a law. read the letter →

arxiv 2505.10475 v1 pith:XEW3BAY3 submitted 2025-05-15 cs.LG cs.CL

Mouxiang Chen , Binyuan Hui , Zeyu Cui , Jiaxi Yang , Dayiheng Liu , Jianling Sun , Junyang Lin , Zhongxin Liu This is my paper · ORCID
classification cs.LGcs.CL
keywords parallelscalinglawlanguagemodelscomputationclassifier-freeguidancemodelensembleinferenceefficiencyprefixtuningtwo-stagepretraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a third scaling axis for language models: instead of adding parameters or adding reasoning tokens, run the same model in $P$ parallel streams, each fed a differently prefixed version of the input, and combine the $P$ outputs with learned weights. It claims that this 'parallel scaling' behaves like parameter scaling: a model with $N$ parameters and $P$ streams approximates a model with $N(k\log P+1)$ parameters, with $k\approx0.39$ on a code corpus and $0.33$ on a general corpus. The fitted law $L=(A/(N(k\log P+1)))^\alpha+E$ matches 24 pretraining runs per corpus with $R^2\approx0.998$. If the claim holds, models can be made more capable without proportional memory growth; at batch size 1, a 1.6B model with $P=8$ costs 22x less memory increase and 6x less latency increase than the parameter-scaled model of equal capacity, which matters for edge deployment. The paper also argues that parameters mostly buy memorization while parallel computation mostly buys reasoning, a distinction that reframes how model capacity is allocated.

What carries the argument

The load-bearing mechanism is parameter-reusing parallel computation: $P$ learnable prefixes (implemented as separate KV caches) create $P$ streams through the same Transformer, and a small MLP on the concatenated stream outputs produces softmax aggregation weights, regularized by label smoothing to keep all streams active. The identity that carries the argument is the substitution $N\mapsto N(k\log P+1)$ inside the standard power-law loss curve, converting 'more parallel passes' into 'equivalent parameters.' The theoretical Proposition 1 performs the same substitution in terms of stream-residual correlation $\rho$, so the empirical fit and the theory share one central object: how much extra effective capacity each stream contributes.

What would settle it

Train a single-stream model that contains exactly the same added prefix and aggregation parameters as the $P=8$ ParScale model, on the same corpus and 42B-token budget, and compare final losses; if the single-stream model matches ParScale's loss, the fitted law is about parameter count, not parallel computation.

Watch

Extended reading notes

Core claim

The central claim is that parallel computation can substitute for parameters at a quantitative rate. On the paper's own terms, the discovery is the parallel scaling law: after training to convergence, the cross-entropy loss of a $P$-stream model fits $L=(A/(N(k\log P+1)))^\alpha+E$, so the effective parameter count is multiplied by $k\log P+1$ with $k=0.39$ on Stack-V2-Python and $k=0.33$ on Pile. The paper also derives a theoretical form $L=(A/(N P^{1/\alpha}\,\mathrm{DIVERSITY}))^\alpha+E$ with $\mathrm{DIVERSITY}=[(P-1)\rho+1]^{-1/\alpha}$, where $\rho$ is the correlation between streams' relative prediction residuals; the empirical logarithmic law is a fitted characterization of how that diversity grows with $P$. Downstream tasks show the substitution is uneven: reasoning-heavy code and math gain more than general-knowledge tasks, which the paper interprets as computation driving reasoning and parameters driving memorization.

Load-bearing premise

The paper attributes the gains to extra parallel computation, but its controls never compare ParScale to a single-stream model that receives the same small set of added trainable parameters, so the interpretation would collapse if those added parameters alone produced the gains.

Editorial extensions

If this is right

  • Every doubling of $P$ adds a constant parameter-equivalent boost: with $k=0.39$, $P=8$ gives roughly a $1.8\times$ larger effective parameter count, and the fitted law says this holds across all tested sizes.
  • Larger models gain more from parallel scaling: because the multiplier multiplies $N$, a 1.6B model at $P=8$ matches a 4.4B model on code tasks while staying far smaller in memory.
  • ParScale is memory-efficient at small batch: the same capacity as parameter scaling with 22x less memory increase and 6x less latency increase at batch size 1, which suits edge deployment.
  • Training cost can be amortized: normal pretraining on 1T tokens followed by 20B ParScale tokens recovers most of the gain, and frozen-backbone variants allow $P$ to be switched at deployment time.
  • Parallel and serial scaling combine: on GSM8K the $P=8$ model improves by 34% relative to the baseline, and the gain persists when chain-of-thought is added.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the effective-parameter substitution is literal, parameter count and parallel compute are partially interchangeable inputs to one scaling law; future fits should include a parallel-compute term alongside $N$ and training data, and compute-optimal allocation would trade one against the other under memory and latency budgets.
  • The logarithmic law implies a saturating ceiling as $P$ grows; Proposition 1 ties the ceiling to stream-residual correlation $\rho$, so measuring how $\rho$ changes at $P=16$ or $32$ would test whether the law extends or bends.
  • The per-token dynamic stream weighting with label smoothing is functionally soft routing, which suggests that combining ParScale with sparse mixture-of-experts architectures could offset each method's weakness, memory-heavy MoE versus compute-heavy ParScale.
  • Because the method is architecture-agnostic, the same $k\log P+1$ law should be testable on vision or multimodal models; a cross-domain comparison of $k$ would reveal whether the computation-versus-parameters trade is universal or domain-dependent.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces ParScale, a method that runs P parallel forward passes of a language model on P learnable input transformations (prefix embeddings) and aggregates the outputs with a dynamic weighted MLP. The central claim is a new scaling law, L = (A / (N (k log P + 1)))^alpha + E, which implies that increasing the number of parallel streams P is equivalent to multiplying the model's effective parameter count by a factor k log P + 1. The authors validate this law with 24 from-scratch pre-training runs per dataset on Stack-V2-Python and Pile (42B tokens), report downstream task improvements, extend the method to a two-stage 1T-token training recipe and to off-the-shelf Qwen-2.5 models, and argue that ParScale gives large inference memory and latency savings relative to parameter scaling at equal model capacity.

Significance. If the equivalence claim holds, the paper identifies a genuinely new scaling axis — parallel computation at roughly constant parameters — and supports it with large-scale experiments, public code and checkpoints, and consistent downstream trends. The 24-run fits per corpus have very high R^2, and the two-stage and PEFT results are practically useful. However, the significance is conditional: the attribution of the gains to parallel computation rather than to the newly introduced adapter parameters is not yet established, and the theoretical proposition does not actually derive the logarithmic form that carries the paper's headline claim. The empirical law is therefore a promising observation rather than a validated scaling law.

major comments (3)
  1. [Section 3.2, Table 6, Figure 6(c)] The loss reduction attributed to parallel computation is not isolated from the parameters introduced by the method. In the scaling-law fits, every P>1 model contains prefix embeddings plus an aggregation MLP (about 0.2% additional parameters per stream), while the P=1 baseline has no added parameters. The pivot experiments in Table 6 vary the transformation type only at fixed P, and Figure 6(c) shows that tuning the introduced parameters on a frozen backbone yields substantial code-generation gains (Pass@1 from 47.4 at P=1 to 53.0 at P=8). A necessary control is a single-stream P=1 model that receives the same number of additional parameters, e.g., the same prefix length and an MLP of matching size. If such a single-stream adapter reproduces the loss curve, the fitted multiplier (k log P + 1) reflects added capacity rather than parallel computation, and the parameter-equivalence interpretation in Section 3.3 loses its basis. The authors should run this control or explicitly restrict the claim to ParScale-with-adapters.
  2. [Section 3.1, Eq. (4); Section 3.2, Eq. (5)] The logarithmic form is assumed, not derived. Proposition 1 expresses the P-dependent factor as P^{1/alpha} * DIVERSITY = [P / ((P-1)rho + 1)]^{1/alpha}. If rho is constant across P, this factor saturates as P grows, so the approximation loss tends to a nonzero limit; that is inconsistent with the unbounded log growth in Eq. (5). The paper defines DIVERSITY by replacing this factor with k log P + 1 "based on the finding of the logarithmic trend" from only the P in {1,2,4,8} points. With 24 runs and four free parameters (A, k, alpha, E), the high R^2 does not discriminate Eq. (5) from saturating or power-law alternatives; indeed, the appendix fit of Eq. (4) with constant rho also achieves R^2 around 0.996. Since the headline O(log P) claim is the empirical form itself, the authors should either measure rho directly, compare several candidate scaling forms and report model selection, or explicitly present the log law as an empirical observation without theoretical support.
  3. [Section 3.3, Figure 4] The 22x and 6x efficiency numbers are derived from the llm-analysis analytical framework (Li, 2023), not from measured inference runs, and no validation against real GPU measurements is reported. The paper should state this clearly and ideally validate the memory and latency model on at least one hardware configuration. As written, the abstract and Section 1 present these ratios as achieved results rather than as analytical projections.
minor comments (5)
  1. [Section 1, Section 3.3] There are typos: "scenerios" in Section 1 and "bottlenect" in Section 3.3 should be "scenarios" and "bottleneck".
  2. [Appendix B, Eq. (9)] The Taylor expansion step drops first-order and higher-order terms with only a heuristic justification and uses "~" loosely; please state the formal assumptions needed for the approximation loss to equal the squared relative residual.
  3. [Section 3.1, Table 6] The theoretical derivation assumes equal aggregation weights 1/P, while the implementation uses a dynamic weighted sum; the statement that the full version is "at least not worse" is not proven. The numerical comparison in Table 6 is reassuring but should be presented as empirical rather than implied by the theory.
  4. [Figure 2, Tables 8-9] The fitted k values (0.393 vs. 0.334) are reported without confidence intervals or seed variance; please include uncertainties or at least a statement about the number of seeds so readers can assess whether the difference is significant.
  5. [Figure 6(c)] Please clarify whether the P=1 condition in the frozen-backbone PEFT experiment includes any introduced parameters (prefix/MLP) or is the unmodified pretrained model; this detail matters for interpreting the control issue.

Circularity Check

2 steps flagged · score 6.0 of 10

The O(logP) parameter-equivalence is a fitted ansatz, not an independent prediction; the theoretical DIVERSITY factor is unmeasured and absorbed into the fitted k.

  1. fitted input called prediction [Section 3.2, Equation (5); Appendix E, Tables 10-11]
    "Thus, we preliminarily try the following form: L=(A/(N·(klogP+1)))^α+E, where we assume that P^{1/α}·DIVERSITY=klogP+1 in Equation (4) based on the finding of the logarithmic trend. (A, k, α, E) are parameters to fit... Recall that Equation (5) implies scaling P equates to increasing parameters by O(NlogP)."

    The headline claim that P parallel streams behave like N(k log P + 1) parameters is not derived from independent data; it is the assumed functional form of Equation (5). The logarithmic trend was observed in the same 24 runs, and k is fitted to those runs (Appendix E: 'based on the 24 runs... we obtain for each dataset... use LBFGS... for curve fitting'). Tables 10-11 then label in-sample fitted values as 'Prediction'. The O(logP) equivalence is therefore a rearrangement of the fitted equation, not an independent prediction.

  2. other [Section 3.1, Proposition 1; Section 6 Discussion]
    "Although we present some theoretical results (Proposition 1), the challenge of directly modeling DIVERSITY limits us to using extensive experiments to fit parallel scaling laws."

    Proposition 1 leaves DIVERSITY (equivalently rho) unmeasured, so Equation (4) cannot by itself predict how loss varies with P. The paper then sets P^{1/α}·DIVERSITY := k log P + 1 'based on the finding of the logarithmic trend', folding all unknown P-dependence into the fitted k. Thus the theoretical formula supplies no independent constraint on the logarithmic form; the O(logP) claim reduces to the empirical ansatz plus the fit.

full rationale

The paper has substantial independent empirical content: ParScale improves downstream performance across many benchmarks, the two-stage 1T-token experiments are genuine out-of-fit validation, and the method is tested on an off-the-shelf Qwen-2.5 model. There is no load-bearing self-citation chain or uniqueness theorem imported from the authors, so this is not a case of score 8-10 circularity. However, the central quantitative claim—that P parallel streams are equivalent to scaling parameters by O(N log P)—is a fitted result rather than a derivation. The theoretical Proposition 1 contains an unmeasured DIVERSITY term that is later absorbed into the fitted constant k via the explicit assumption P^{1/α}·DIVERSITY = k log P + 1. Equation (5) is fit to the same 24 training runs from which the logarithmic trend was read, and Appendix E presents the resulting in-sample residuals as 'Predictions'. Therefore the O(logP) parameter-equivalence is forced by the chosen parametric form and the fitted k, not independently predicted. The efficiency comparisons in Section 3.3 inherit this fitted equivalence. Because the empirical method itself is validated independently, the circularity is partial and localized to the scaling-law claim, warranting a score of 6 rather than higher.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central law rests on four fitted constants plus an assumed functional form; the only theory is a conditional derivation that contains an unmeasured correlation rho. The scaling-law contribution is therefore mostly empirical.

free parameters (7)
  • k (logarithmic gain) = 0.3935 (Stack-V2-Python), 0.3345 (Pile)
    Fitted to 24 pretraining runs per dataset; determines the O(log P) parameter-equivalence, central to the main claim.
  • A (Chinchilla coefficient) = 1.1306e7 (Stack-V2-Python), 1.9735e8 (Pile)
    Fitted scale parameter in Eq. (5) following the Chinchilla fitting procedure.
  • E (irreducible loss) = 0.6912 (Stack-V2-Python), 1.2888 (Pile)
    Fitted entropy or loss floor in Eq. (5).
  • alpha (loss exponent) = 0.1894 (Stack-V2-Python), 0.1963 (Pile)
    Fitted exponent in Eq. (5).
  • rho (stream correlation) = 0.8919 (Stack-V2-Python), 0.8995 (Pile) when fitting Eq. (4)
    Correlation between stream relative residuals; unmeasured in the main text and absorbed into the empirical k log P + 1 term.
  • prefix length = 48 tokens (pivot); 96 tokens tested
    Chosen hyperparameter for input transformation; performance differences were minimal.
  • label smoothing epsilon = 0.1
    Hyperparameter to prevent load imbalance in aggregation weights.
assumptions (5)
  • domain assumption Each parallel stream's loss follows the Chinchilla law L_i = (A/N)^alpha + E with the shared backbone parameter count N (Lemma 3.1).
    Streams share the same backbone and differ only by learned prefixes, so it is not obvious that each stream behaves like an independently trained N-parameter model. This premise grounds Proposition 1.
  • domain assumption Relative residuals Delta p_i are unbiased: E[Delta p_i] = 0.
    Invoked in Appendix B to simplify the MSE derivation; true for a well-calibrated estimator but not proven for shared-parameter streams.
  • standard math Taylor expansion log(1+x) = x - x^2/2 + O(x^3) and omission of higher-order terms.
    Standard approximation used in the proof of Proposition 1; valid for small residuals.
  • ad hoc to paper P^{1/alpha} * DIVERSITY = k log P + 1.
    Assumed in Section 3.2 based on observed logarithmic trend; this is the step that turns the theoretical formula into the practical O(log P) law.
  • domain assumption Models are trained to convergence at 42B tokens; loss at the end of training is the converged Chinchilla loss.
    The paper assumes convergence to apply Chinchilla's converged-loss form; training curves in Appendix F suggest loss is still decreasing at 42B tokens.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Parallel Scaling Law for Language Models." pith.science (2026). https://pith.science/paper/XEW3BAY3

@misc{pith2026250510475,
  author       = {Pith},
  title        = {Pith review of: Parallel Scaling Law for Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XEW3BAY3}},
  note         = {Machine review of arXiv:2505.10475}
}
abstract

It is commonly believed that scaling language models should commit a significant space or time cost, by increasing the parameters (parameter scaling) or output tokens (inference-time scaling). We introduce the third and more inference-efficient scaling paradigm: increasing the model's parallel computation during both training and inference time. We apply $P$ diverse and learnable transformations to the input, execute forward passes of the model in parallel, and dynamically aggregate the $P$ outputs. This method, namely parallel scaling (ParScale), scales parallel computation by reusing existing parameters and can be applied to any model structure, optimization procedure, data, or task. We theoretically propose a new scaling law and validate it through large-scale pre-training, which shows that a model with $P$ parallel streams is similar to scaling the parameters by $O(\log P)$ while showing superior inference efficiency. For example, ParScale can use up to 22$\times$ less memory increase and 6$\times$ less latency increase compared to parameter scaling that achieves the same performance improvement. It can also recycle an off-the-shelf pre-trained model into a parallelly scaled one by post-training on a small amount of tokens, further reducing the training budget. The new scaling law we discovered potentially facilitates the deployment of more powerful models in low-resource scenarios, and provides an alternative perspective for the role of computation in machine learning.

Figures

Figures reproduced from arXiv: 2505.10475 by the authors.

Figure 1
Figure 1. (1) Illustrations of our proposed parallel scaling ( [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Loss of LLMs scaled on parameters and number of parallel streams [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Predicted loss contours for PARSCALE. Each contour line indicates a combination of (parameter, P) with similar performance [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Model capacity (indicated by loss) scales on the inference space-time cost, with three parameters [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Loss for two-stage training, smoothing using an [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: (a)(b) Loss for continual pre-training the Qwen-2.5-3B model on the two datasets. (c) Code [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Loss for training on OpenWebText for repeating several epochs. On the fifth epoch, the [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]
Figure 8
Figure 8. Figure 8: Training loss for the Stack-V2-Python and the Pile, smoothing with 0.98 exponential moving [PITH_FULL_IMAGE:figures/full_fig_p027_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Structured Recurrent Mixers for Massively Parallelized Sequence Generation

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    Structured Recurrent Mixers provide a dual parallel-recurrent representation for sequence models, claiming superior training efficiency, information capacity, and inference throughput over linear complexity alternatives.

  2. ParaThinker: Native Parallel Thinking as a New Paradigm to Scale LLM Test-time Compute

    cs.CL 2025-08 conditional novelty 6.0 of 10

    ParaThinker trains LLMs for native parallel reasoning and reports 7 to 12 percent higher accuracy on math benchmarks over sequential thinking with modest latency overhead.

  3. Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    Grove MoE uses unequal-size adjugate experts with complexity-based activation to run 33B-parameter models at roughly 3.1 to 3.3B active parameters while matching larger open models in benchmarks.

  4. cMoLLM at Scale: Horizontal Scaling Laws for Mixture-of-LLMs

    cs.AI 2026-06 reject novelty 4.0 of 10

    A pipeline-level soft mixture of LLM streams implemented as dynamic convolution improves GPT-2-scale perplexity/GLUE/SQuAD, but only at substantially higher compute and without reported ParaScale/AltUp comparisons.

  5. System Report for CCL25-Eval Task 10: SRAG-MAV for Fine-Grained Chinese Hate Speech Recognition

    cs.CL 2025-07 conditional novelty 4.0 of 10

    A Qwen2.5-7B system with task reformulation, self-retrieval augmentation, and multi-round voting scores 37.505 average on STATE ToxiCN, outperforming cited baselines.

Reference graph

Works this paper leans on

105 extracted references · 8 canonical work pages · cited by 5 Pith papers

  1. [1]

    Smollm - blazingly fast and remarkably powerful

    Loubna Ben Allal, Anton Lozhkov, and Elie Bakouch. Smollm - blazingly fast and remarkably powerful. https://huggingface.co/blog/smollm, 2024

    2024

  2. [2]

    Smollm2: When smol goes big -- data-centric training of a small language model

    Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werr...

    arXiv 2025

  3. [3]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. URL https://arxiv.org/abs/2108.07732

    arXiv 2021

  4. [4]

    Cosmopedia, February 2024

    Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Cosmopedia, February 2024. URL https://huggingface.co/datasets/HuggingFaceTB/cosmopedia

    2024

  5. [5]

    B it F it: Simple parameter-efficient fine-tuning for transformer-based masked language-models

    Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. B it F it: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp.\ 1--9, Dublin, Ireland,...

    doi:10.18653/v1/2022.acl-short.1 2022

  6. [6]

    Lee, Haonan Li, Charles Lovering, Niklas Muennighoff, Ellie Pavlick, Jason Phang, Aviya Skowron, Samson Tan, Xiangru Tang, Kevin A

    Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles Lovering, Niklas Muennighoff, Ellie Pavlick,...

    arXiv 2024

  7. [7]

    PIQA: reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in...

    doi:10.1609/aaai.v34i05.6239 2020

  8. [8]

    Random forests

    Leo Breiman. Random forests. Mach. Learn., 45 0 (1): 0 5–32, October 2001. ISSN 0885-6125. doi:10.1023/A:1010933404324. URL https://doi.org/10.1023/A:1010933404324

    doi:10.1023/a:1010933404324 2001

Show all 105 references
  1. [9]

    Le, Christopher Ré, and Azalia Mirhoseini

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024. URL https://arxiv.org/abs/2407.21787

    2024 arXiv

  2. [10]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffr...

    2005 arXiv

  3. [11]

    Are more LLM calls all you need? towards the scaling properties of compound AI systems

    Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, and James Zou. Are more LLM calls all you need? towards the scaling properties of compound AI systems. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024...

    2024

  4. [12]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021 arXiv

  5. [14]

    Zhijun Chen, Jingzheng Li, Pengpeng Chen, Zhuoran Li, Kai Sun, Yuankai Luo, Qianren Mao, Dingqi Yang, Hailong Sun, and Philip S. Yu. Harnessing multiple large language models: A survey on llm ensemble. arXiv preprint arXiv:2502.18036, 2025. URL https://arxiv.org/abs/2502.18036

    2025 arXiv

  6. [15]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. URL https://arxiv.org/abs/1803.05457

    2018 arXiv

  7. [16]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. U...

    arXiv

  8. [17]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. arXiv preprint arXiv:2205.14135, 2022. URL https://arxiv.org/abs/2205.14135

    2022 arXiv

  9. [18]

    Deepseek llm: Scaling open-source language models with longtermism

    DeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024. URL https://arxiv.org/abs/2401.02954

    2024 arXiv

  10. [19]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. URL https://arxiv.org/abs/2501.12948

    2025 arXiv

  11. [20]

    Diffusion models beat GAN s on image synthesis

    Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GAN s on image synthesis. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=AAWuCvzaVt

    2021

  12. [21]

    Longrope: Extending llm context window beyond 2 million tokens

    Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending llm context window beyond 2 million tokens. arXiv preprint arXiv:2402.13753, 2024. URL https://arxiv.org/abs/2402.13753

    2024 arXiv

  13. [22]

    Switch transformers: scaling to trillion parameter models with simple and efficient sparsity

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23 0 (1), January 2022. ISSN 1532-4435

    2022

  14. [23]

    Scaling laws for sparsely-connected foundation models

    Elias Frantar, Carlos Riquelme Ruiz, Neil Houlsby, Dan Alistarh, and Utku Evci. Scaling laws for sparsely-connected foundation models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=i9K2ZWkYIP

    2024

  15. [24]

    Dropout as a bayesian approximation: Representing model uncertainty in deep learning

    Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Mac...

    2016

  16. [25]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2021. UR...

    2021 arXiv

  17. [26]

    Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein

    Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv:2502.05171, 2025. URL htt...

    2025 arXiv

  18. [27]

    Openwebtext corpus

    Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019

    2019

  19. [28]

    Parameter-efficient fine-tuning for large models: A comprehensive survey

    Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=lIsCS8b6zj

    2024

  20. [29]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021 a . URL https://openreview.net/forum?id=d7KBjmI3GmQ

    2021

  21. [30]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021 b

    2021

  22. [31]

    Scaling laws and interpretability of learning from repeated data

    Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, Scott Johnston, Ben Mann, Chris Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan, and Sam McCandlish. Scaling l...

    2022 arXiv

  23. [32]

    Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou

    Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017. URL https://arxiv.org/abs/1712.00409

    2017 arXiv

  24. [33]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. URL https://arxiv.org/abs/2207.12598

    2022 arXiv

  25. [34]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp.\ 6840--6851. Curran Associates, Inc., 2020. URL http...

    2020

  26. [35]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022 arXiv

  27. [36]

    Lo RA : Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  28. [37]

    Mini CPM : Unveiling the potential of small language models with scalable training strategies

    Shengding Hu, Yuge Tu, Xu Han, Ganqu Cui, Chaoqun He, Weilin Zhao, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Xinrong Zhang, Zhen Leng Thai, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, dahai li, Zhiyuan Liu,...

    2024

  29. [38]

    Data movement is all you need: A case study on optimizing transformers

    Andrei Ivanov, Nikoli Dryden, Tal Ben-Nun, Shigang Li, and Torsten Hoefler. Data movement is all you need: A case study on optimizing transformers. In A. Smola, A. Dimakis, and I. Stoica (eds.), Proceedings of Machine Learning and Systems, volume 3, pp.\ 711--732, 2021. URL ht...

    2021

  30. [39]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. URL https://arxiv.org/abs/2001.08361

    2001 arXiv

  31. [40]

    Kimi k1.5: Scaling reinforcement learning with llms

    Kimi Team . Kimi k1.5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025. URL https://arxiv.org/abs/2501.12599

    2025 arXiv

  32. [41]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.or...

    2015 arXiv

  33. [42]

    RACE : Large-scale R e A ding comprehension dataset from examinations

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE : Large-scale R e A ding comprehension dataset from examinations. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Proc...

    2017 doi

  34. [43]

    Albert: A lite bert for self-supervised learning of language representations

    Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019. URL https://arxiv.org/abs/1909.11942

    1909 arXiv

  35. [44]

    Revisiting prefix-tuning: Statistical benefits of reparameterization among prompts

    Minh Le, Chau Nguyen, Huy Nguyen, Quyen Tran, Trung Le, and Nhat Ho. Revisiting prefix-tuning: Statistical benefits of reparameterization among prompts. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=QjTSaFXg25

    2025

  36. [45]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,...

    2021 doi

  37. [46]

    Solving quantitative reasoning problems with language models

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. ...

    2022 arXiv

  38. [47]

    Llm-analysis: Latency and memory analysis of transformer models for training and inference

    Cheng Li. Llm-analysis: Latency and memory analysis of transformer models for training and inference. https://github.com/cli99/llm-analysis, 2023

    2023

  39. [48]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joi...

    2021 doi

  40. [49]

    Contrastive decoding: Open-ended text generation as optimization

    Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st...

    2023 doi

  41. [50]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024 a

    2024 arXiv

  42. [51]

    Liu and Jorge Nocedal

    Dong C. Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Math. Program., 45 0 (1–3): 0 503–528, August 1989. ISSN 0025-5610

    1989

  43. [52]

    Is your code generated by chat GPT really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chat GPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openrev...

    2023

  44. [53]

    Mobilellm: Optimizing sub-billion parameter language models for on-device use cases

    Zechun Liu, Changsheng Zhao, Forrest Iandola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, Liangzhen Lai, and Vikas Chandra. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. arXi...

    2024 arXiv

  45. [54]

    The llama 3 herd of models

    Llama Team . The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  46. [55]

    On power laws in deep ensembles

    Ekaterina Lobacheva, Nadezhda Chirkova, Maxim Kodryan, and Dmitry Vetrov. On power laws in deep ensembles. arXiv preprint arXiv:2007.08483, 2020 a . URL https://arxiv.org/abs/2007.08483

    2007 arXiv

  47. [56]

    On power laws in deep ensembles

    Ekaterina Lobacheva, Nadezhda Chirkova, Maxim Kodryan, and Dmitry P Vetrov. On power laws in deep ensembles. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp.\ 2375--2385. Curran Associat...

    2020

  48. [57]

    Starcoder 2 and the stack v2: The next generation

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Z...

    2024 arXiv

  49. [58]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterativ...

    2023

  50. [59]

    Coherence boosting: When your pretrained language model is not paying enough attention

    Nikolay Malkin, Zhen Wang, and Nebojsa Jojic. Coherence boosting: When your pretrained language model is not paying enough attention. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational...

    2022 doi

  51. [60]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016. URL https://arxiv.org/abs/1609.07843

    2016 arXiv

  52. [61]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ' ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empiri...

    2018 doi

  53. [62]

    Scaling data-constrained language models

    Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://o...

    2023

  54. [63]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. URL https://arxiv.org/abs/2303.08774

    2023 arXiv

  55. [64]

    New reasoning models: Openai o1-preview and o1-mini

    OpenAI. New reasoning models: Openai o1-preview and o1-mini. https://openai.com/research/o1-pre view-and-o1-mini, 2024

    2024

  56. [65]

    PREADD : Prefix-adaptive decoding for controlled text generation

    Jonathan Pei, Kevin Yang, and Dan Klein. PREADD : Prefix-adaptive decoding for controlled text generation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp.\ 10018--10037, Toronto, Canada, Ju...

    2023 doi

  57. [66]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557, 2024. URL https://arxiv.org/abs/2...

    2024 arXiv

  58. [67]

    O1 replication journey: A strategic progress report -- part 1

    Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, and Pengfei Liu. O1 replication journey: A strategic progress report -- part 1. arXiv preprint arXiv:2410.18982, 2024. URL https://arxiv.org/abs/2410.18982

    arXiv

  59. [68]

    D-cpt law: Domain-specific continual pre-training scaling law for large language models

    Haoran Que, Jiaheng Liu, Ge Zhang, Chenchen Zhang, Xingwei Qu, Yinghao Ma, Feiyu Duan, Zhiqi Bai, Jiakai Wang, Yuanxing Zhang, Xu Tan, Jie Fu, Wenbo Su, Jiamang Wang, Lin Qu, and Bo Zheng. D-cpt law: Domain-specific continual pre-training scaling law for large language models....

    2024 arXiv

  60. [69]

    Qwen2.5 technical report

    Qwen Team . Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024. URL https://arxiv.org/abs/2412.15115

    2024 arXiv

  61. [70]

    Qwq: Reflect deeply on the boundaries of the unknown

    Qwen Team . Qwq: Reflect deeply on the boundaries of the unknown. https://qwenlm.github.io/blog/qwq-32b-preview/, 2025 a

    2025

  62. [71]

    Qwen3 technical report

    Qwen Team . Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025 b . URL https://arxiv.org/abs/2505.09388

    2025 arXiv

  63. [72]

    Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion mod...

    2022 arXiv

  64. [73]

    Winogrande: an adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64 0 (9): 0 99–106, August 2021. ISSN 0001-0782. doi:10.1145/3474381. URL https://doi.org/10.1145/3474381

    2021 doi

  65. [74]

    Stay on topic with classifier-free guidance

    Guillaume V Sanchez, Alexander Spangher, Honglu Fan, Elad Levi, and Stella Biderman. Stay on topic with classifier-free guidance. In Proceedings of the 41st International Conference on Machine Learning, pp.\ 43197--43234, 2024

    2024

  66. [75]

    Beyond chinchilla-optimal: accounting for inference in language model scaling laws

    Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: accounting for inference in language model scaling laws. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024

    2024

  67. [76]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017. URL https://arxiv.org/abs/1701.06538

    2017 arXiv

  68. [77]

    Trusting your evidence: Hallucinate less with context-aware decoding

    Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen-tau Yih. Trusting your evidence: Hallucinate less with context-aware decoding. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chap...

    2024 doi

  69. [78]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. URL https://arxiv.org/abs/1909.08053

    1909 arXiv

  70. [79]

    Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning

    Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?i...

    2025

  71. [80]

    Inference scaling flaws: The limits of llm resampling with imperfect verifiers

    Benedikt Stroebl, Sayash Kapoor, and Arvind Narayanan. Inference scaling flaws: The limits of llm resampling with imperfect verifiers. arXiv preprint arXiv:2411.17501, 2024. URL https://arxiv.org/abs/2411.17501

    2024

  72. [81]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864, 2021. URL https://arxiv.org/abs/2104.09864

    2021 arXiv

  73. [82]

    Stop overthinking: A survey on efficient reasoning for large language models

    Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, and Xia Hu. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503.16419, 2025. URL https://arxi...

    2025 arXiv

  74. [83]

    Rethinking the Inception Architecture for Computer Vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the Inception Architecture for Computer Vision . In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 2818--2826, Los Alamitos, CA, USA, June 2016. IEEE Co...

    2016 doi

  75. [84]

    Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, Neil Band, Tim G

    Dustin Tran, Jeremiah Liu, Michael W. Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, Neil Band, Tim G. J. Rudner, Karan Singhal, Zachary Nado, Joost van Amersfoort, Andreas Kirsch, Rodolphe Jenatton, Nithum Thain, Honglin Yuan, Kelly B...

    2022 arXiv

  76. [85]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Info...

    2017

  77. [86]

    Will we run out of data? limits of llm scaling based on human-generated data

    Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Will we run out of data? limits of llm scaling based on human-generated data. arXiv preprint arXiv:2211.04325, 2022. URL https://arxiv.org/abs/2211.04325

    2022 arXiv

  78. [87]

    Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, St \'e fan J

    Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, St \'e fan J. van der Walt , Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nels...

    2020

  79. [88]

    Lora ensembles for large language model fine-tuning

    Xi Wang, Laurence Aitchison, and Maja Rudolph. Lora ensembles for large language model fine-tuning. arXiv preprint arXiv:2310.00035, 2023 a . URL https://arxiv.org/abs/2310.00035

    2023 arXiv

  80. [89]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023 b . URL https:...

    2023

  81. [90]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. URL https://arxiv.org/abs/2201.11903

    2022 arXiv

  82. [91]

    Liu, and Matt Gardner

    Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Leon Derczynski, Wei Xu, Alan Ritter, and Tim Baldwin (eds.), Proceedings of the 3rd Workshop on Noisy User-generated Text, pp.\ 94--106, Copenhagen, Denmark, September 2017. A...

    2017 doi

  83. [92]

    Batchensemble: An alternative approach to efficient ensemble and lifelong learning

    Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: An alternative approach to efficient ensemble and lifelong learning. arXiv preprint arXiv:2002.06715, 2020. URL https://arxiv.org/abs/2002.06715

    2002 arXiv

  84. [93]

    Sam Wiseman and Alexander M. Rush. Sequence-to-sequence learning as beam-search optimization. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp.\ 1296--1306, Austin, Texas, November 201...

    2016 doi

  85. [94]

    Inference scaling laws: An empirical analysis of compute-optimal inference for LLM problem-solving

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for LLM problem-solving. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/foru...

    2025

  86. [95]

    Speeding up deep model training by sharing weights and then unsharing

    Shuo Yang, Le Hou, Xiaodan Song, Qiang Liu, and Denny Zhou. Speeding up deep model training by sharing weights and then unsharing. arXiv preprint arXiv:2110.03848, 2021. URL https://arxiv.org/abs/2110.03848

    2021 arXiv

  87. [96]

    Data mixing laws: Optimizing data mixtures by predicting language modeling performance

    Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?...

    2025

  88. [97]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. H ella S wag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Llu \'i s M \`a rquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguis...

    2019 doi

  89. [98]

    Scaling vision transformers

    Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. arXiv preprint arXiv:2106.04560, 2021. URL https://arxiv.org/abs/2106.04560

    2021 arXiv

  90. [99]

    When scaling meets LLM finetuning: The effect of data, model and finetuning method

    Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets LLM finetuning: The effect of data, model and finetuning method. In The Twelfth International Conference on Learning Representations, 2024 a . URL https://openreview.net/forum?id=5HCnKDeTws

    2024

  91. [100]

    Generative verifiers: Reward modeling as next-token prediction

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. In The 4th Workshop on Mathematical Reasoning and AI at NeurIPS'24, 2024 b . URL https://openreview.net/forum?id=CxHRoTLmPX

    2024

  92. [101]

    Denny Zhou, Nathanael Sch \"a rli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Le...

    2023

  93. [102]

    Instruction-following evaluation for large language models

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023 b . URL https://arxiv.org/abs/2311.07911

    2023 arXiv

  94. [103]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  95. [104]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  96. [105]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  97. [106]

    https://openai.com/research/o1-pre view-and-o1-mini

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2021 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.