Pith. sign in

REVIEW 3 major objections 5 minor 63 references

Recursive Inference Scaling: A Winning Path to Scalable Inference in Language and Multimodal Systems

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Recursively reusing the first block of a transformer before the final block beats standard training at equal compute and size.

desk verdict A useful empirical paper with a genuinely new finding—ArB recursion wins under a forward-pass compute proxy—but the headline scaling-law and overtraining claims need schedule-controlled and compute-matched rework before they convince. read the letter →

arxiv 2502.07503 v4 pith:T3LNVM54 submitted 2025-02-11 cs.AI cs.LG

Ibrahim Alabdulmohsin , Xiaohua Zhai This is my paper · ORCID
classification cs.AIcs.LG
keywords recursiveinferencescalingparametersharingtest-timecomputelawslanguagemodelingmultimodalpretrainingstochasticdepthfractalself-similarity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces RINS, a recipe that splits a transformer into two equally sized blocks and runs the first block on its own output several times before the second block executes. Under a compute-matched comparison that charges the same training FLOPs and keeps parameters fixed, RINS beats the non-recursive baseline, context-length scaling, and more than fifty-five other parameter-sharing architectures, including repeat-all-over and latent recurrent thinking. The authors show the gain grows with longer training, transfers to contrastive vision-language models, and shows up in fitted data scaling laws as both a better asymptotic loss and a better scaling exponent. A stochastic version with tiny linear adapters makes RINS a no-regret addition: pretraining with recursion still helps even when recursion is switched off at inference, which suggests it could be a standard ingredient in large-model pretraining.

What carries the argument

The organizing device is a two-dimensional taxonomy: a signature records which depthwise blocks share parameters and in what order, so $A^rB$ means block $A$ is repeated $r$ times before block $B$, and a degree records whether the same sharing pattern is nested inside each block. RINS is the signature $A^rB$ with degree 1. The mechanism that carries the argument is recursion of the early block for $r$ rounds during both training and inference, with training steps cut so total forward-pass layer cost matches the baseline; stochastic RINS adds a per-step binomial dropout of recursion rounds, and the no-regret result adds one linear adapter per possible recursion count. The data-scaling-law analysis fits $\varepsilon(x)=\beta x^{-c}+\varepsilon_\infty$ to sweep curves and reads off how $c$ and $\varepsilon_\infty$ move with $r$.

What would settle it

Train a 300M- or 600M-parameter baseline and a RINS $A^3B$ model on the same corpus with identical software-level FLOP profiling that includes backward passes and activation memory traffic, and compare final log-perplexity at equal measured FLOPs; if RINS no longer beats the baseline or beats it only at lower recursion counts than the paper reports, the central compute-matched claim is falsified.

Watch

Extended reading notes

Core claim

RINS is an architecture with degree 1 and signature $A^rB$ for $r>1$: the network is split depthwise into two equal blocks, and block $A$ is applied recursively to its own output $r$ times before block $B$ produces the result. The paper's central claim is that, at fixed parameter count and fixed training compute measured in $\text{layer}\times\text{step}$ FLOPs, a model trained with this recursion consistently reaches lower language-modeling loss than the ordinary non-recursive baseline, than models that scale by lengthening the context, and than the other recursive strategies in its taxonomy. The claim is supported by sweeps over dozens of models at 300M, 600M, and 1B scale, by downstream common-sense tasks where RINS-trained models score highest, and by SigLIP contrastive pretraining where RINS raises zero-shot ImageNet accuracy by about two percentage points. The fitted power law $\varepsilon(x)=\beta x^{-c}+\varepsilon_\infty$ shows that increasing $r$ improves both the exponent $c$ and the asymptotic limit $\varepsilon_\infty$, so overtraining a baseline cannot fully close the gap. The authors also report that stochastic dropout of recursion rounds plus lightweight linear adapters ($<1\%$ of parameters) removes the penalty of not recursing at test time.

Load-bearing premise

The compute-matched comparisons count training cost as layers times steps, counting only forward-pass layer operations; if the true cost including backward passes, memory bandwidth, and data efficiency is not proportional to that count, the claimed advantage of RINS over its baselines could be an artifact of undercounting its training cost.

Editorial extensions

If this is right

  • For a fixed model size and training FLOP budget, a RINS-enabled model should reach lower validation perplexity than the baseline at every sufficiently large budget, and the gap should widen the longer the model is trained.
  • For a fixed quality target, RINS should let the same architecture be trained with less compute or deployed with fewer parameters, which is directly relevant to memory-limited on-device settings.
  • The optimal number of recursion rounds grows with training budget and with the degree of overtraining, so RINS should be scheduled like other hyperparameters as compute scales.
  • Applying RINS to contrastive vision-language pretraining should improve zero-shot classification and cross-modal retrieval without changing model size or training FLOPs.
  • With stochastic RINS and linear adapters, a practitioner can decide at test time whether to spend extra inference compute, because the 1x-inference checkpoint already meets or beats the baseline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper leaves implicit: if self-similarity of language is the cause, RINS gains should vary across domains in proportion to their measured fractal or self-similar structure; a synthetic corpus with controlled self-similarity could confirm or refute the mechanism.
  • Because RINS consumes a different mix of forward versus backward computation than standard training, exact FLOP profiling including backward passes might shift the optimal $r$ downward, even though the qualitative ordering of methods could survive.
  • RINS appears composable with inference-time methods such as chain-of-thought and repeated sampling, and the combined scaling curves could be measured to see whether recursion multiplies or merely adds to their gains.
  • The linear adapters used for the no-regret result suggest a larger design space: adapters that are shared across recursion counts or slightly nonlinear might trade a small parameter increase for a bigger guaranteed gain at 1x inference.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Recursive INference Scaling (RINS), a parameter-sharing architecture in which the first half of a model is applied recursively r times before the second half is applied, and compares it with more than 55 alternative parameter-sharing strategies, including repeat-all-over (RAO) and latent recurrent thinking. The main empirical claims are that, at fixed parameter count and training compute measured in layer-by-step units, RINS improves language-modeling perplexity; that the improvement persists in downstream common-sense tasks and in SigLIP vision-language pretraining; that RINS improves both the scaling exponent and the asymptotic loss limit in data scaling laws; and that a stochastic variant with lightweight linear adapters provides a no-regret strategy when recursion is disabled at inference time. The authors also report a vision control experiment showing that recursion does not help supervised image classification, consistent with their fractal-language motivation.

Significance. If the central claims hold, RINS is a simple and potentially valuable plug-in for compact language and multimodal models, since it improves accuracy without increasing parameter count and offers a no-regret option via stochastic depth and linear adapters. The paper's strengths include the large systematic sweep of 59 models, the downstream validation, the multimodal extension, the explicit attempt at compute-matched comparisons, and the vision control. However, the headline asymptotic claims in Section 6 rest on learning-rate-schedule-contaminated power-law fits, and the multimodal overtraining comparison in Table 3 is not compute-matched. For these reasons the present version does not establish the strongest claims in the abstract, although the fixed-compute language-modeling results remain valuable and mostly well supported.

major comments (3)
  1. [Section 6, Figure 5a and Figure 3b] The data-scaling-law conclusion is contaminated by the learning-rate cooldown. Section 6 states that the fits use the 600M-parameter models of Figure 3a; Section 3 states that all runs use a fixed 5K cooldown; and the Figure 2 caption itself attributes the "sharp drops in perplexity near the end of training" to the cooldown. At matched layer-by-step compute, a RINS model with signature A^rB trains 2/(r+1) as many steps as the baseline, so the fixed 5K cooldown occupies (r+1)/2 times as large a fraction of training and starts from an rsqrt learning rate that is roughly sqrt((r+1)/2) times higher. Larger r therefore receives a larger schedule-induced downward jump at the end of training. Fitting epsilon(x) = beta x^{-c} + epsilon_inf to these curves without excluding cooldown points will bias c upward and epsilon_inf downward as r grows, which is exactly the pattern reported in Figure 5a and in the "Infinite Compute Limit" panels of Figure 3b. The claim that overtraining the baseline cannot close the gap requires a pre-cooldown refit, or an explicit report of fits that exclude cooldown points and a demonstration that the fitted c and epsilon_inf are stable under that exclusion.
  2. [Section 5, Table 3] The multimodal overtraining comparison is not compute-matched. The table caption and the text say that SigLIP-RINS-B/16 uses signature A3B and that both SigLIP-B/16 and SigLIP-RINS-B/16 are trained on 40B examples. Since A3B consumes twice the per-example forward compute of AB, this gives RINS twice the total training compute at the same data scale. Consequently, the +2% ImageNet gain and the sentence claiming a "fundamental advantage in multimodal learning that are not replicated by simply overtraining a non-recursive counterpart" are not established by this experiment; the gain may simply reflect additional training FLOPs. The authors should either compare against a baseline trained to the same total compute (approximately 80B examples) or train RINS on 20B examples so that the compute budgets match.
  3. [Section 2, Definition 2.1 and Figure 2 caption] The paper's central notion of compute matching is defined and operationalized inconsistently. Definition 2.1 calls C(f) the "actual computational cost (in FLOPs)" but restricts it to the forward pass, while Figure 2 and Section 3 use x-axes measured in "Training FLOPs (layers)", i.e. layer-by-step counts. Counting layers as equal-cost units is a monotone proxy for forward FLOPs only when all layers have identical FLOPs, and it ignores backward-pass, optimizer, and data-pipeline costs. Backward FLOPs scale with the same recursion factor, so the ranking is probably conservative, but the abstract's "same training compute FLOPs" wording is not literally supported by the reported accounting. The paper should report a full FLOP budget, or state precisely that the matched quantity is layer-by-step forward cost.
minor comments (5)
  1. [Section 5] The cross-reference is wrong: the text says "As shown in Table 5" for the overtraining results, but the table with those results is Table 3 in the main text; the appendix has yet another Table 5 with the same content.
  2. [Appendix E.1 and Figure 7] The pseudocode contains syntax errors that make the architecture sweep unambiguous only with effort: for example "degree1" in the RAO entries and the extra bracket in "p_skip: Tuple [ float , ...]]".
  3. [Section 3] The main loss curves in Figures 2, 3a, and 4 are single runs with no error bars; given that the central comparison is empirical, at least a small number of seeds or a statement that the curves are representative would strengthen the claim.
  4. [Section 6] The text states that the coefficient beta_r also increases with r, but Figure 5a does not show beta_r; either add a panel for beta_r or omit this statement.
  5. [Section 1] There is a typo in the introduction: "langauge modeling" should be "language modeling".

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RINS is selected and evaluated against external baselines; the scaling-law claims are empirical fits, not predictions forced by construction.

full rationale

The paper's derivation chain is not circular. RINS is defined by a taxonomy of parameter-sharing signatures (Definition 2.2), and its claimed advantage is established empirically by sweeping 59 models against external baselines (RAO, latent recurrent thinking, long-sequence, and the non-recursive baseline) under an explicitly stated forward-pass FLOPs accounting. The loss curves, downstream commonsense benchmarks, SigLIP zero-shot/retrieval evaluations, and the vision control are all external data; no target quantity is used to define RINS or to select the fitted parameters. The Section 6 'data scaling laws' fit the standard power law epsilon(x) = beta*x^{-c} + epsilon_inf to those same loss curves and read off c and epsilon_inf; this is an empirical summary rather than an independent prediction, and the conclusion that overtraining cannot close the gap is an extrapolation of the assumed functional form. It is not circular because the fitted values are determined by the data rather than by the definition of RINS. The fractal-geometry motivation cites prior work by the first author (Alabdulmohsin et al., 2024a), but the paper explicitly labels the mechanism as a hypothesis (Appendix D) and provides an external vision control, so the self-citation is not load-bearing. The fixed 5K-step cooldown is a potential confound for the scaling-law fits, but that is a correctness/schedule-identification concern, not circularity. Therefore no step reduces by construction to its inputs.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

Most of the paper's load is empirical; there are no hidden mathematical axioms beyond standard learning assumptions. Free parameters are architecture and regularization choices. No new physical entities are introduced.

free parameters (4)
  • recursion rounds r = 2 to 4 depending on compute
    RINS performance depends on r; the optimal r is selected from data (Figure 5b).
  • skip probability ps = 0, 0.25, 0.5, 0.8
    Stochastic RINS uses ps as a hand-set hyperparameter; the no-regret claim depends on the chosen ps.
  • power law parameters beta, c, epsilon_inf = fitted per architecture
    Section 6 fits epsilon(x) = beta x^{-c} + epsilon_inf to loss curves; claims about improved asymptotic limit and exponent rest on these fits.
  • learning rate and weight decay = lr=5e-4, wd=5e-5 for language models
    The learning rate was swept for the non-recursive baseline only; recursive models reuse the same schedule.
assumptions (3)
  • domain assumption Language possesses self-similar (fractal) structure that makes recursive depth beneficial
    Motivates RINS; Appendix D admits this remains a hypothesis.
  • ad hoc to paper Training FLOPs can be matched by counting forward-pass layer operations (layer x step)
    Used to compare architectures in Section 3; backward pass and memory costs are ignored.
  • domain assumption Loss follows a power law epsilon(x) = beta x^{-c} + epsilon_inf
    Used to analyze asymptotic limits in Section 6; fitted to finite training runs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Recursive Inference Scaling: A Winning Path to Scalable Inference in Language and Multimodal Systems." pith.science (2026). https://pith.science/paper/T3LNVM54

@misc{pith2026250207503,
  author       = {Pith},
  title        = {Pith review of: Recursive Inference Scaling: A Winning Path to Scalable Inference in Language and Multimodal Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T3LNVM54}},
  note         = {Machine review of arXiv:2502.07503}
}
read the original abstract

Inspired by recent findings on the fractal geometry of language, we introduce Recursive INference Scaling (RINS) as a complementary, plug-in recipe for scaling inference time in language and multimodal systems. RINS is a particular form of recursive depth that significantly outperforms +55 other variants, including the recent "repeat-all-over" (RAO) strategy in Mobile LLM (Liu et al., 2024) and latent recurrent thinking (Geiping et al., 2025). Unlike prior works, we carry out our comparisons on a compute-matched regime, and demonstrate that for a fixed model size and training compute budget, RINS substantially improves language modeling performance. It also generalizes beyond pure language tasks, delivering gains in multimodal systems, including a +2% improvement in 0-shot ImageNet accuracy for SigLIP-B/16. Additionally, by deriving data scaling laws, we show that RINS improves both the asymptotic performance limits and the scaling exponents. More importantly, with light-weight (linear) adapters (comprising <1% of model parameters) and stochastic dropout, RINS offers a no-regret strategy, meaning that RINS-enabled pretraining improves performance in language modeling even when recursive depth is not applied at inference time. This corresponds to improving performance on a training compute-, parameter-, and inference-matched regime, suggesting its potential as a viable component of LLM pretraining!

Figures

Figures reproduced from arXiv: 2502.07503 by the authors.

Figure 1
Figure 1. LEFT: In RINS, the model f : X → Y is split into two parts: the first block fA : X → X is applied iteratively to its own output r times before passing the output to the second block. RIGHT: Illustrative examples of models with different signatures and degrees. From top to bottom: (1) Baseline (Signature: AB, Degree: 1), a feedforward architecture with no recursion. (2) repeat-all-over (RA) (Liu et al., 2024), where … view at source ↗
Figure 2
Figure 2. Language models are trained on 200B tokens. The [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Performance of stochastic RINS (A 3B) with varying inference costs for 1B parameter LMs. The x-axis represents the training compute cost. The legend indicates the inference cost of each stochastic RINS configuration relative to the baseline; e.g. 1.5x denotes 50% increase in inference cost. For ps = 0, RINS@1x is significantly worse, with perplexity scores > 3. As expected, RINS converges in performance to the basel… view at source ↗
Figures from the paper (4 more)
Figure 6
Figure 6. Figure 6: LEFT 2 PLOTS: y-axis corresponds to performance when RINS is enabled during training but disabled at inference time in 600M-parameter LMs. ps = 1 2 in stochastic RINS with linear adapters matches the baseline at 1× the inference cost while ps = 0.8 results in a better …
Figure 7
Figure 7. Figure 7: Numpy-like syntax for models with a fixed signature and degree. When no stochastic depth [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Caption How does depth-wise model recursion relate to self-similarity of language across the temporal dimension? We provide an illustration of this in [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Performance of stochastic RINS (A 3B) with varying inference costs for 1B parameter LMs on C4, similar to [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 43 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  3. [3]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  4. [4]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  5. [5]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    work page

  6. [6]

    Alabdulmohsin, I., Neyshabur, B., and Zhai, X. (2022). Revisiting neural scaling laws in language and vision. In NeurIPS

    work page 2022

  7. [7]

    Q., and Dehghani, M

    Alabdulmohsin, I., Tran, V. Q., and Dehghani, M. (2024a). Fractal patterns may illuminate the success of next-token prediction. In NeurIPS

    work page 2024

  8. [8]

    Alabdulmohsin, I., Zhai, X., Kolesnikov, A., and Beyer, L. (2024b). Getting ViT in shape: Scaling laws for compute-optimal model design

    work page 2024

Show all 63 references
  1. [9]

    D., and Ammanabrolu, P

    Ankner, Z., Paul, M., Cui, B., Chang, J. D., and Ammanabrolu, P. (2024). Critique-out-loud reward models

    2024

  2. [10]

    Bahri, Y., Dyer, E., Kaplan, J., Lee, J., and Sharma, U. (2021). Explaining neural scaling laws. arXiv preprint arXiv:2102.06701

    2021 arXiv

  3. [11]

    Bansal, Y., Ghorbani, B., Garg, A., Zhang, B., Krikun, M., Cherry, C., Neyshabur, B., and Firat, O. (2022). Data scaling laws in NMT : The effect of noise and architecture. arXiv preprint arXiv:2202.01994

    2022 arXiv

  4. [12]

    Beleites, C., Neugebauer, U., Bocklitz, T., Krafft, C., and Popp, J. (2013). Sample size planning for classification models. Analytica chimica acta , 760:25--33

    2013

  5. [13]

    Beyer, L., Zhai, X., and Kolesnikov, A. (2022). Better plain vit baselines for imagenet-1k

    2022

  6. [14]

    L., Gao, J., and Choi, Y

    Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. (2020). Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence

    2020

  7. [15]

    V., Ré, C., and Mirhoseini, A

    Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., and Mirhoseini, A. (2024). Large language monkeys: Scaling inference compute with repeated sampling

    2024

  8. [16]

    Q., Hanin, B., Bailis, P., Stoica, I., Zaharia, M., and Zou, J

    Chen, L., Davis, J. Q., Hanin, B., Bailis, P., Stoica, I., Zaharia, M., and Zou, J. (2024). Are more llm calls all you need? towards the scaling properties of compound ai systems. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  9. [17]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., B...

    2021

  10. [18]

    Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Doll \'a r, P., and Zitnick, C. L. (2015). Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325

    2015 arXiv

  11. [19]

    Cho, J., Lee, K., Shin, E., Choy, G., and Do, S. (2015). How much data is needed to train a medical image deep learning system to achieve necessary high accuracy? arXiv preprint arXiv:1511.06348

    2015 arXiv

  12. [20]

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. (2019). BoolQ : Exploring the surprising difficulty of natural yes/no questions. In NAACL

    2019

  13. [21]

    Universal transformers

    Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., and Łukasz Kaiser (2019). Universal transformers

    2019

  14. [22]

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In CVPR

    2009

  15. [23]

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. (2021). An image is worth 16x16 words: Transformers for image recognition at scale

    2021

  16. [24]

    L., Zeng-Treitler, Q., Kandula, S., and Ngo, L

    Figueroa, R. L., Zeng-Treitler, Q., Kandula, S., and Ngo, L. H. (2012). Predicting sample size required for classification performance. BMC medical informatics and decision making , 12(1):1--10

    2012

  17. [25]

    R., Kailkhura, B., Bhatele, A., and Goldstein, T

    Geiping, J., McLeish, S., Jain, N., Kirchenbauer, J., Singh, S., Bartoldson, B. R., Kailkhura, B., Bhatele, A., and Goldstein, T. (2025). Scaling up test-time compute with latent reasoning: A recurrent depth approach

    2025

  18. [26]

    Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M., Ali, M., Yang, Y., and Zhou, Y. (2017). Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409

    2017 arXiv

  19. [27]

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022). Training compute-optimal large language models. In NeurIPS

    2022

  20. [28]

    Huang, G., Sun, Y., Liu, Z., Sedra, D., and Weinberger, K. (2016). Deep networks with stochastic depth

    2016

  21. [29]

    Hutter, M. (2021). Learning curve theory. arXiv preprint arXiv:2102.04074

    2021 arXiv

  22. [30]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models. arXiv preprint arXiv:2001.08361

    2020 arXiv

  23. [31]

    Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

    2014 arXiv

  24. [32]

    Krizhevsky, A. (2009). Learning multiple layers of features from tiny images. Technical report

    2009

  25. [33]

    Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2020). Albert: A lite bert for self-supervised learning of language representations

    2020

  26. [34]

    A., Larrick, R

    Lawson, M. A., Larrick, R. P., and Soll, J. B. (2020). Comparing fast thinking and slow thinking: The relative benefits of interventions, individual differences, and inferential rules. Judgment and Decision making , 15(5):660--684

    2020

  27. [35]

    J., Sutherland Robson, E., Kohli, P., de Freitas, N., Kavukcuoglu, K., and Vinyals, O

    Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., Hubert, T., Choy, P., de Masson d’Autume, C., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J....

    2022

  28. [36]

    Liu, Z., Zhao, C., Iandola, F., Lai, C., Tian, Y., Fedorov, I., Xiong, Y., Chang, E., Shi, Y., Krishnamoorthi, R., et al. (2024). Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. In ICML

    2024

  29. [37]

    Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2024). Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems , 36

    2024

  30. [38]

    S., Love, J., Tafti, P., Hussenot, L., Sessa, P

    Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., Tafti, P., Hussenot, L., Sessa, P. G., Chowdhery, A., Roberts, A., Barua, A., Botev, A., Castro-Ros, A., Slone, A., Héliou, A., Tacchetti, A., Bulanova, A., Paters...

    2024

  31. [39]

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. (2018). Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789

    2018 arXiv

  32. [40]

    M., Vedaldi, A., Zisserman, A., and Jawahar, C

    Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. V. (2012). Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition

    2012

  33. [41]

    P., Zhai, X., and Alabdulmohsin, I

    Pouget, A., Beyer, L., Bugliarello, E., Wang, X., Steiner, A. P., Zhai, X., and Alabdulmohsin, I. (2024). No filter: Cultural and socioeconomic diversity in contrastive vision-language models. In NeurIPS

    2024

  34. [42]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(140):1--67

    2020

  35. [43]

    V., Lin, S

    Ramaswamy, V. V., Lin, S. Y., Zhao, D., Adcock, A., van der Maaten, L., Ghadiyaram, D., and Russakovsky, O. (2024). Geode: a geographically diverse evaluation dataset for object recognition. Advances in Neural Information Processing Systems , 36

    2024

  36. [44]

    Rojas, W. A. G., Diamos, S., Kini, K. R., Kanter, D., Reddi, V. J., and Coleman, C. (2022). The dollar street dataset: Images representing the geographic and socioeconomic diversity of the world. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and ...

    2022

  37. [45]

    Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y. (2019). Socialiqa: Commonsense reasoning about social interactions

    2019

  38. [46]

    and Kaplan, J

    Sharma, U. and Kaplan, J. (2022). Scaling laws from the data manifold dimension. JMLR , 23(9):1--34

    2022

  39. [47]

    R., Hestness, J., and Dey, N

    Soboleva, D., Al-Khateeb, F., Myers, R., Steeves, J. R., Hestness, J., and Dey, N. (2023). SlimPajama: A 627B token cleaned and deduplicated version of RedPajama

    2023

  40. [48]

    Talmor, A., Herzig, J., Lourie, N., and Berant, J. (2019). C ommonsense QA : A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Te...

    2019

  41. [49]

    W., Fedus, W., Rao, J., Narang, S., Tran, V

    Tay, Y., Dehghani, M., Abnar, S., Chung, H. W., Fedus, W., Rao, J., Narang, S., Tran, V. Q., Yogatama, D., and Metzler, D. (2022a). Scaling laws vs model architectures: How does inductive bias influence scaling? arXiv preprint arXiv:2207.10551

    2022 arXiv

  42. [50]

    Tay, Y., Dehghani, M., Bahri, D., and Metzler, D. (2022b). Efficient transformers: A survey

    2022

  43. [51]

    V., Pont-Tuset, J., Chen, X., and Soricut, R

    Thapliyal, A. V., Pont-Tuset, J., Chen, X., and Soricut, R. (2022). Crossmodal-3600: A massively multilingual multimodal evaluation dataset. arXiv preprint arXiv:2205.12522

    2022 arXiv

  44. [52]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. NeurIPS

    2017

  45. [53]

    Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. (2023). Self-consistency improves chain of thought reasoning in language models

    2023

  46. [54]

    H., Le, Q

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. (2024). Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems , ...

    2024

  47. [55]

    Welleck, S., Bertsch, A., Finlayson, M., Schoelkopf, H., Xie, A., Neubig, G., Kulikov, I., and Harchaoui, Z. (2024). From decoding to meta-generation: Inference-time algorithms for large language models. Transactions on Machine Learning Research . Survey Certification

    2024

  48. [56]

    Weyand, T., Araujo, A., Cao, B., and Sim, J. (2020). Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2575--2584

    2020

  49. [57]

    Wilcoxon, F. (1992). Individual comparisons by ranking methods. In Breakthroughs in statistics: Methodology and distribution , pages 196--202. Springer

    1992

  50. [58]

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2023). React: Synergizing reasoning and acting in language models

    2023

  51. [59]

    Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. (2014). From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics , 2:67--78

    2014

  52. [60]

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. (2019). Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

    2019

  53. [61]

    Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. (2022). Scaling vision transformers. In CVPR

    2022

  54. [62]

    Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. (2023). Sigmoid loss for language image pre-training

    2023

  55. [63]

    N., and Lopez-Paz, D

    Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. (2018). mixup: Beyond empirical risk minimization

    2018

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.