Universal pre-training by iterated random computation

Peter Bloem

REVIEW 3 major objections 7 minor 40 references

Universal pre-training by iterated random computation

T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that random bits passed through a random LSTM approximate the universal distribution, so synthetic computation can pre-train a model for real-world tasks.

desk verdict Honest proof-of-concept, but the headline theoretical claim is not actually proven — the key domination constant vanishes exponentially, and the paper says so itself. read the letter →

arxiv 2506.20057 v1 pith:QL6VT57G submitted 2025-06-24 cs.LG

Peter Bloem This is my paper · ORCID

classification cs.LG MSC 68Q3068T07

keywords universalpre-trainingsyntheticdatadistributionSolomonoffinductionalgorithmiccomplexityin-contextlearningrandomLSTMdata-computetradeoff

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a model can be pre-trained on random noise before any task data is seen, by passing the noise through random computations. The theoretical target is the universal distribution, the Bayesian mixture over all computable processes that an ideal algorithmic predictor would use, and the claim is that feeding random bits iteratively through a randomly initialized recurrent network approximates it in the limit. If that is right, synthetic computation can substitute for real-world training data: one expensive pre-training run can be amortized over many tasks, and data-poor domains can benefit before their own data is collected. The experiments report that a transformer trained on roughly 20 million sampled sequences shows zero-shot in-context learning on six synthetic and five real-world datasets, improves with scale, and finetunes faster with better generalization than training from scratch. The paper itself notes that its practical generator is an approximation to universality, not the ideal, and the whole transfer rests on real data being generated by computational processes.

What carries the argument

The load-bearing mechanism is an iterated random-LSTM source. A single-layer LSTM is initialized with random weights, given an eight-token seed and a conditional sequence drawn from a buffer, and sampled autoregressively; the generated sequence replaces the conditional in the buffer, so later samples are conditioned on earlier generated structure. This is a practical approximation to drawing from the mixture $m^C_{mix}$ over the hierarchy $m^1_C, m^2_C,\ldots$, and the theory shows that each pass adds computational depth while preserving dominance. The linchpin lemma states that an LSTM implementing one step of a Turing machine occupies a ball of non-zero probability under a Gaussian initialization, so the random prior does not make the required computation infinitely unlikely; together with the Turing completeness of recurrent networks this yields Theorem 4.5, that $m_C^{n+2}$ dominates $m^n_{UTM}$.

What would settle it

Compare the zero-shot losses on Wikipedia and Linux for a model pre-trained on the iterated random-LSTM mixture against a model pre-trained on uniformly random bytes; if the two curves are indistinguishable, the iteration is contributing no computational structure and the central claim collapses.

Watch

Extended reading notes

Core claim

The paper's central claim, stated in Section 4, is that sampling random bits and feeding them iteratively through a recurrent neural network approximates the universal distribution in the limit of the number of iterations and the allowed sequence length, even if the network class used to generate the data is bounded in size. For a class $C$ of computable functions with prior $p(c)$, the class-universal distribution is $m_C(x)=\sum_{c\in C,\,r:c(r)=x}p(c)\,2^{-|r|}$; the paper proves that iterating the sampling process yields $m_C^{n+1}$ dominating $m_C^n$, and that with randomly initialized LSTMs the iterated class dominates the $n$-step resource-bounded universal distribution $m^n_{UTM}$ up to a constant. Empirically, the paper claims that a transformer trained on sequences from the resulting mixture $m^C_{mix}$ shows zero-shot in-context learning on synthetic and real-world datasets, outperforms an optimized in-context Markov model on real-world text and code, improves with scale, and that finetuning after this pre-training converges faster and retains more out-of-domain performance than the re-initialized baseline. The authors present this as a complement to, not a replacement for, Solomonoff induction, and are explicit that the practical implementation is an approximation to the universal ideal.

Load-bearing premise

The argument depends on real-world data being generated by a computational process that a randomly initialized LSTM can imitate; Wikipedia, Linux, D3.js and German text must fall inside that class for the synthetic pre-training to dominate them.

Editorial extensions

If this is right

Training data can be manufactured before any task is known, so a single expensive pre-training run can be amortized over many downstream tasks.
The same pre-trained model can be used as a universal base: zero-shot in-context prediction on unseen datasets, with performance that improves as the model grows.
Finetuning from the universal base reaches the same loss about one million instances faster than training from scratch and largely retains performance on other datasets.
Data scarcity becomes a data-compute tradeoff: less real-world data is needed if more random computation is spent on generation.
If real sources are computable, the same transfer should apply to any structured domain, not only byte-level text and code.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Going beyond the paper: because the domination constant shrinks exponentially with each iteration, a practical system will have to balance computational depth against the probability of sampling deep cases; reweighting the buffer toward deeper samples is a testable extension the paper does not run.
Going beyond the paper: if the tradeoff scales as the theory suggests, it weakens the human-generated-data limit on model growth, and centralizing a synthetic pre-training run could make a sensitive-data-free base model freely publishable, an implication the authors sketch in their social-impact section.
Going beyond the paper: the domination argument applies to any Turing-complete generator class, so mixing sources, for example a resource-bounded UTM feeding an LSTM, could combine inductive biases, which the paper leaves as future work.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a useful paper with a real empirical contribution, but the central theoretical claim is over-sold. Theorem 4.5 gives a domination constant p(f)^n which decays exponentially, so the lower bound is vacuous as n grows. The paper concedes this in Section 4.4 — 'we cannot say that in the limit m^∞_C dominates the unbounded universal distribution m(x)' — and again in Section 7.1. The stress-test note is correct on that point.

What is genuinely new: the class-bounded prefix-free framing, the LSTM-iteration source distribution, and the empirical demonstration that a transformer pre-trained on this synthetic data transfers to real-world text and code. The scaling results are suggestive, the finetuning experiment shows faster convergence and retained out-of-domain performance, and the ablation against alternative synthetic sources (UTM, transformer, pointwise) is honest and informative. The paper is also unusually candid about its limitations.

Soft spots, in order of severity. First, the theoretical gap: the limit claim is not established, only resource-bounded domination with a vanishing constant. Second, the empirical protocol: single runs per setting, no error bars, no matched real-data pretraining baseline. The finetuning comparison gives the pretrained model 20M extra synthetic instances; the amortization argument is reasonable, but it is not a like-for-like compute budget. Third, the dominance assumption — that real-world sources fall in the class sampled by a single-layer LSTM with random weights — is asserted, not tested. The model does beat an in-context Markov model on real data, which is real evidence, but not evidence for universality.

None of this is fatal. The core empirical result — random LSTM computation can pre-train a model that transfers to real sequence prediction better than an in-context Markov baseline — holds up as a proof-of-concept. The 'universal' naming is an overstatement, but the authors flag it themselves in Section 7.1.

I would send this to a serious referee. The theory needs reframing or the claims need weakening, and the experiments need error bars and a real-data baseline, but the paper deserves engagement. I would cite it for the experimental setup.

Referee Report

3 major / 7 minor

Summary. The paper proposes universal pre-training: training a transformer on synthetic data generated by iteratively feeding random bits through randomly initialized LSTMs, with the aim of obtaining a model that predicts well zero-shot across a wide range of sequence-prediction tasks and that finetunes efficiently. The theoretical part builds on class-bounded prefix-free algorithmic complexity to show that sampling from a mixture over a model class yields a distribution that dominates any member of the class, that iterating the sampling process creates a hierarchy of richer distributions, and that, with LSTMs as the model class, the iterated mixture dominates any resource-bounded universal Turing machine up to an exponentially vanishing constant. The empirical part trains transformers at several scales on this synthetic data and evaluates zero-shot performance on synthetic and real-world datasets (Wikipedia, Linux, D3.js, German text), showing improvement over chance, scaling with model size, and faster finetuning convergence with some retained generalization.

Significance. If the strong version of the claim were established, the paper would provide a practical data/compute tradeoff that could reduce dependence on real-world training data, an idea of considerable current interest. The empirical contributions are real: the paper provides code, a detailed experimental protocol, multiple ablations, and an honest discussion of where the method fails (e.g., bitsflip and toy2). The class-bounded, prefix-free complexity framework is cleanly presented, and the finite-n domination results (Lemmas 4.1 and 4.3, Theorem 4.5 for each fixed n) are correct as stated. However, the limit claim about approximating the universal distribution is not supported by the theorems, because the domination constant in Theorem 4.5 decays exponentially; the paper itself concedes this at the end of Section 4.4. Thus the theoretical foundation is weaker than the paper's framing suggests, though the empirical findings remain of interest as a proof-of-concept.

major comments (3)

[Section 4, Theorem 4.5] The statement (Section 4, after the three-step argument) that iterating random LSTMs 'will approximate the universal distribution in the limit' is not a consequence of Theorem 4.5. The theorem only gives m^{n+2}_C(x) ≥ p(r)p(f)^n p(s) m^n_UTM(x) for finite resource bounds n, with p(f)^n shrinking exponentially. As n → ∞ the lower bound tends to zero and is compatible with m^∞_C(x) → 0 for every x; it does not imply convergence in total variation, KL divergence, or pointwise domination. The text at the end of Section 4.4 ('we cannot say that in the limit m^∞_C dominates the unbounded universal distribution m(x)') makes this explicit. The paper should either prove a stronger limit statement or rephrase the central claim to match the actual finite-n result, clearly stating that the 'approximation' is a finite-resource domination with a vanishing constant.
[Section 4.4, Lemma 4.4] The proof that a randomly initialized LSTM computes the Turing-machine step function f with non-infinitesimal probability rests on constructing a ball of parameters that all yield the same output. Two issues arise. First, the transformation y' = 3y − 1 is inconsistent with the described parameter change: multiplying all elements of W_o by 3 and subtracting 1 from all elements of b yields y' = 3W_o z + b − 1, not 3W_o z + 3b − 1, so the stated 'y' = 3y − 1' is not realized by the described operation. Second, the backward error argument treats each matrix multiplication as if the inputs were fixed and bounded independent of the parameters, but in a recurrent LSTM the hidden state at every step depends on the same perturbed parameters at previous steps. The proof needs a bound on error accumulation over the k compute tokens (e.g., a Lipschitz/contraction argument on the recurrent dynamics). Without this, the existence of a non-zero-radius ball in which the entire forward pass is exactly preserved is not established. This is load-bearing because p(f) enters Theorem 4.5 with exponent n.
[Sections 5.1 and 7.1] The theoretical development concerns exact iterates m^n_C with independently sampled members of C, whereas the practical algorithm samples from a buffer-mixture m_mix_C. The paper asserts that m_mix_C dominates each m^n_C 'albeit with a constant of domination that vanishes exponentially,' but no formal connection is given between the iterated process of the theory and the buffer-update rule of Algorithm 1. Moreover, the universality claim requires that real-world target distributions are dominated by the specific LSTM mixture, yet no quantitative evidence is provided that Wikipedia, Linux, D3.js, or German text fall in the dominated class; the paper's own results (e.g., the model does not solve bitsflip or toy2, and is often worse than a tuned in-context Markov model) suggest the practical class is far from universal. Section 7.1 partially acknowledges this, but the abstract and Section 3 statements ('we can still make the required claims of universality if the pre-training distribution dominates the source of the data') should be calibrated to the weaker guarantee actually established, and the relationship between the theoretical iterates and the implemented buffer mixture should be made precise.

minor comments (7)

[Section 3] Typo: 'apprixmations' should be 'approximations'.
[Section 6.3] Typo: 'aroudn' should be 'around'.
[Figure 6 caption] Typo: 'LTSM' appears twice in the caption and should be 'LSTM'.
[Section 4.4, Lemma 4.4] The phrase 'compute fi' should be 'compute f' (no subscript is defined for f_i). Also, the reuse of n to denote both the iteration index and the number of LSTM parameters in the lemma statement is confusing; consider using a different symbol for the parameter count.
[Theorem 4.2 proof] The proof uses 'pc' in several lines (e.g., 'pc(x_)' and 'pc(b|x)') where the theorem statement's notation for the dominated distribution is 'p'. This makes the proof hard to follow; the notation should be made consistent.
[References] References [22] and [23] are incomplete placeholders with no titles or publication details; they should be completed before publication.
[Section 4.1] The notation '×>' and '+<' is unconventional and appears with inconsistent spacing; the definitions in the text are clear, but the symbols would benefit from being typeset more carefully (e.g., with explicit constants) to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the theoretical claims rest on external published domination theorems, the only self-citation is a genuine prior proof, and the empirical results use held-out data with no fitted quantities presented as predictions.

full rationale

The derivation chain is self-contained against external results. Section 4.1 invokes [3] for the class-bounded domination inequality mC(x) >× pc(x); [3] is a published theorem by the same first author, but it is a real proof about class-bounded prefix-free complexity whose stated assumptions do not include the iterated-LSTM or universal-pre-training conclusion, so under the review rules it counts as independent evidence rather than circular self-citation. Theorem 4.2 is adapted from Li and Vitanyi [24], an external reference. The iterated-sampling argument (Theorem 4.3) and the LSTM simulation argument (Theorem 4.5) are derived in the paper; the paper does not hide the fact that the domination constant shrinks exponentially: it states explicitly that 'we cannot say that in the limit m∞_C(x) dominates the unbounded universal distribution m(x)' (Section 4.4) and repeats in Section 7 that the practical implementation is 'more of an approximation to this ideal.' No fitted parameter is later renamed as a prediction: the source LSTM is randomly initialized, and the downstream evaluations in Sections 6.2 and 6.3 are performed on held-out data (wp, linux, german, code, and synthetic generators) that were not used to fit any constant in the theory. The paper also explicitly credits [12, 26, 16] for the general idea rather than presenting it as new, so there is no renaming of a known result presented as a derivation. The only weakness is the gap between the vanishing-constant domination theorem and the informal 'approximate the universal distribution in the limit' phrasing, but that is an overstatement of what the proof establishes, not a circular reduction of the conclusion to its inputs.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The paper adds no new physical or mathematical entities. Its central empirical claim rests on a set of hand-chosen source-model hyperparameters and on the domain assumption that target data is dominated by the LSTM-generated class-universal distribution. The theoretical machinery itself is standard algorithmic complexity.

free parameters (6)

LSTM random-multiplier range U(0,1.1) = 0 to 1.1
Hand-chosen in Section 5.1 to vary source models; no theoretical basis or sensitivity analysis.
source sampling temperature = 1e-4
Set in Appendix A.1 to make generated strings nearly deterministic; hand-chosen.
context/sequence length = 512 tokens
Fixed in all experiments for memory predictability; limits longer-range structure and may cause scaling plateau.
buffer size and reset mix = 20x source batch; 20 constant + 20 noise resets
Chosen without ablation over these exact values; buffering itself is ablated, but the constants are not.
base learning rate (MUP) = 3e-4 pretraining; 1e-4 finetuning
Tuned at width 640 on validation data and transferred via MUP; a fitted hyperparameter.
progressive block schedule = enable every 10K instances
Hand-chosen training stabilizer; not ablated.

assumptions (6)

standard math Prefix-free Turing machine enumeration and universal distribution m(x) exist and dominate all computable distributions.
Invoked in Section 2 and used throughout Sections 4.1-4.3; standard algorithmic-complexity background.
standard math Single-layer LSTMs are Turing complete and can simulate one step of a universal Turing machine given sufficient width and precision.
Basis for Theorem 4.5's construction; cited from [7] and [32].
ad hoc to paper A randomly initialized LSTM from the paper's non-degenerate initialization computes a given TM-step function with non-infinitesimal probability.
Lemma 4.4 proves this only under saturated activations and a ball argument; the practical initialization is assumed to satisfy these conditions.
domain assumption Real-world data sources (Wikipedia, Linux, D3.js, German text) are generated by computational processes in the model class C, so a class-universal distribution dominates them.
Stated in Section 4 and in Section 7.1 as the justification for transfer; no empirical verification of dominance.
domain assumption Training an autoregressive transformer with negative log loss on samples from mmix_C produces a model that approximates mmix_C.
Borrowed from PFN theory [26]; the paper relies on it to connect pretraining objective to the prior.
ad hoc to paper Sufficient floating point precision is available so the LSTM exactly reproduces the discrete TM-step function f.
Assumed in Section 4.4 before Lemma 4.4; not guaranteed by the architecture described.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Universal pre-training by iterated random computation." pith.science (2026). https://pith.science/paper/QL6VT57G

@misc{pith2026250620057,
  author       = {Pith},
  title        = {Pith review of: Universal pre-training by iterated random computation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QL6VT57G}},
  note         = {Machine review of arXiv:2506.20057}
}

read the original abstract

We investigate the use of randomly generated data for the sake of pre-training a model. We justify this approach theoretically from the perspective of algorithmic complexity, building on recent research that shows that sequence models can be trained to approximate Solomonoff induction. We derive similar, but complementary theoretical results. We show empirically that synthetically generated data can be used to pre-train a model before the data is seen. We replicate earlier results that models trained this way show zero-shot in-context learning across a variety of datasets, and that this performance improves with scale. We extend earlier results to real-world data, and show that finetuning a model after pre-training offers faster convergence and better generalization.

Figures

Figures reproduced from arXiv: 2506.20057 by the authors.

**Figure 1.** ( ) A string of randomly sampled characters. ( ) The result of passing this string through three randomly initialized neural network models. The latter data is partly predictable, and so has value for pre-training. 1 arXiv:2506.20057v1 [cs.LG] 24 Jun 2025 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

**Figure 2.** The results of the main experiment (Section 6.2). We train a trans [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** bits Each string consists of two random strings of 5 bits, followed by the result of applying the boolean operators xor, ∧, ∨ and = respectively for a sequence length of 30 bits. The first 10 of these are fully random, while the last 20 are fully determined by the first 10. Including the delimiter, a perfect predictor, given sufficient context would get a performance of 10/31 ≈ 0.32 bits. bitsflip Repeatedly: a ran… view at source ↗

Figures from the paper (4 more)

**Figure 3.** Figure 3: The four toy data sources used. Example outputs are shown below [PITH_FULL_IMAGE:figures/full_fig_p021_3.png]

**Figure 4.** Figure 4: The results of the main experiment (Section 6.2) on the validation [PITH_FULL_IMAGE:figures/full_fig_p022_4.png]

**Figure 5.** Figure 5: The results of the finetuning experiment. Colors indicate the dataset [PITH_FULL_IMAGE:figures/full_fig_p023_5.png]

**Figure 6.** Figure 6: The results of the ablation experiment (Section 6.4). Starting with [PITH_FULL_IMAGE:figures/full_fig_p025_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 32 canonical work pages

[1]

Theoretical Computer Science 354(3), 391–404 (2006)

Antunes, L., Fortnow, L., Van Melkebeek, D., Vinodchandran, N.V.: Com- putational depth: concept and applications. Theoretical Computer Science 354(3), 391–404 (2006)

work page 2006
[2]

Computer 49(05), 54–63 (may 2016)

Bal, H., Epema, D., de Laat, C., van Nieuwpoort, R., Romein, J., Sein- stra, F., Snoek, C., Wijshoff, H.: A medium-scale distributed system for computer science research: Infrastructure for the long term. Computer 49(05), 54–63 (may 2016). https://doi.org/10.1109/MC.2016.127

doi:10.1109/mc.2016.127 2016
[3]

In: International conference on algorithmic learning theory

Bloem, P., Mota, F., de Rooij, S., Antunes, L., Adriaans, P.: A safe ap- proximation for kolmogorov complexity . In: International conference on algorithmic learning theory . pp. 336–350. Springer (2014)

work page 2014
[4]

In: Algorithmic Learning Theory: 26th International Conference, ALT 2015, Banff, AB, Canada, October 4-6, 2015, Proceedings 26

Bloem, P., de Rooij, S., Adriaans, P.: Two problems for sophistication. In: Algorithmic Learning Theory: 26th International Conference, ALT 2015, Banff, AB, Canada, October 4-6, 2015, Proceedings 26. pp. 379–

work page 2015
[5]

Bostock, Mike: D3.js, https://d3js.org/

work page
[6]

Ihre geschichtliche Entwicklung und wirtschaftliche Seite

Braun, L.: Die Frauenfrage. Ihre geschichtliche Entwicklung und wirtschaftliche Seite. Hirzel (1901)

work page 1901
[7]

Advances in neural information processing sys- tems 34, 28431–28441 (2021)

Chung, S., Siegelmann, H.: Turing completeness of bounded-precision re- current neural networks. Advances in neural information processing sys- tems 34, 28431–28441 (2021)

work page 2021
[8]

IEEE Access 9, 51416–51431 (2021)

Collins, J., Chand, S., Vanderkop, A., Howard, D.: A review of physics simulators for robotic applications. IEEE Access 9, 51416–51431 (2021)

work page 2021

Show all 40 references

[9]

Community contributors and Torvalds, Linus: The linux kernel, https://github.com/torvalds/linux/tree/ dd83757f6e686a2188997cb58b5975f744bb7786
[10]

John Wiley & Sons (1999)

Cover, T., Thomas, J.: Elements of information theory . John Wiley & Sons (1999)

1999
[11]

arXiv preprint arXiv:2304.05366 (2023)

Goldblum, M., Finzi, M., Rowan, K., Wilson, A.G.: The no free lunch the- orem, kolmogorov complexity , and the role of inductive biases in machine learning. arXiv preprint arXiv:2304.05366 (2023)

2023 arXiv
[12]

ICML (2024) 28

Grau-Moya, J., Genewein, T., Hutter, M., Orseau, L., Delétang, G., Catt, E., Ruoss, A., Wenliang, L.K., Mattern, C., Aitchison, M., et al.: Learning universal predictors. ICML (2024) 28

2024
[13]

MIT press (2007)

Grünwald, P.D.: The minimum description length principle. MIT press (2007)

2007
[14]

Heckendorn, R.: A practical tutorial on context free grammars (2021)

2021
[15]

Neural compu- tation 9(8), 1735–1780 (1997)

Hochreiter, S., Schmidhuber, J.: Long short-term memory . Neural compu- tation 9(8), 1735–1780 (1997)

1997
[16]

Nature 637(8045), 319–326 (2025)

Hollmann, N., Müller, S., Purucker, L., Krishnakumar, A., Körfer, M., Hoo, S.B., Schirrmeister, R.T., Hutter, F.: Accurate predictions on small data with a tabular foundation model. Nature 637(8045), 319–326 (2025)

2025
[17]

Hutter, M., Mahoney , M., Bowery , J.: The hutter prize (2006), http:// prize.hutter1.net/

2006
[18]

Digital Communications and Networks 9(1), 79– 89 (2023)

Kohli, V., Chakravarty , S., Chamola, V., Sangwan, K.S., Zeadally , S.: An analysis of energy consumption and carbon footprints of cryptocurrencies and possible solutions. Digital Communications and Networks 9(1), 79– 89 (2023)

2023
[19]

In: Algorithmic Probability and Friends

Lattimore, T., Hutter, M.: No free lunch versus occam’s razor in supervised learning. In: Algorithmic Probability and Friends. Bayesian Prediction and Artificial Intelligence: Papers from the Ray Solomonoff 85th Memorial Conference, Melbourne, VIC, Australia, November 30–Decem...

2013
[20]

arXiv preprint arXiv:2006.12467 (2020)

Levine, Y., Wies, N., Sharir, O., Bata, H., Shashua, A.: The depth-to-width interplay in self-attention. arXiv preprint arXiv:2006.12467 (2020)

2020 arXiv
[21]

arXiv preprint arXiv:1711.05101 5, 5 (2017)

Loshchilov, I., Hutter, F., et al.: Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101 5, 5 (2017)

2017 arXiv
[24]

In: Algorithms and complexity , pp

Ming, L., Vitányi, P.M.: Kolmogorov complexity and its applications. In: Algorithms and complexity , pp. 187–254. Elsevier (1990)

1990
[25]

The Guardian https://www.theguardian.com/technology/2025/jun/11/ disney-universal-ai-lawsuit

Montgomery , B.: Disney and universal sue ai image creator midjourney , alleging copyright infringement. The Guardian https://www.theguardian.com/technology/2025/jun/11/ disney-universal-ai-lawsuit

2025
[26]

arXiv preprint arXiv:2112.10510 (2021)

Müller, S., Hollmann, N., Arango, S.P., Grabocka, J., Hutter, F.: Transform- ers can do bayesian inference. arXiv preprint arXiv:2112.10510 (2021)

2021 arXiv
[27]

In: In- ternational Conference on Machine Learning

Nagler, T.: Statistical foundations of prior-data fitted networks. In: In- ternational Conference on Machine Learning. pp. 25660–25676. PMLR (2023) 29

2023
[28]

Nakamura, R., Tadokoro, R., Yamada, R., Asano, Y.M., Laina, I., Rup- precht, C., Inoue, N., Yokota, R., Kataoka, H.: Scaling backwards: Mini- mal synthetic pre-training? In: European Conference on Computer Vision. pp. 153–171. Springer (2024)

2024
[29]

The Guardian https://www.theguardian.com/books/2023/sep/20/ authors-lawsuit-openai-george-rr-martin-john-grisham

Press, A.: George rr martin and john grisham among group of authors suing openai. The Guardian https://www.theguardian.com/books/2023/sep/20/ authors-lawsuit-openai-george-rr-martin-john-grisham

2023
[30]

Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al.: Improving language understanding by generative pre-training (2018)

2018
[31]

arXiv preprint arXiv:1608.01745 (2016)

Shafaei, A., Little, J.J., Schmidt, M.: Play and learn: Using video games to train computer vision models. arXiv preprint arXiv:1608.01745 (2016)

2016 arXiv
[32]

In: Proceedings of the fifth annual workshop on Computational learning theory

Siegelmann, H.T., Sontag, E.D.: On the computational power of neu- ral nets. In: Proceedings of the fifth annual workshop on Computational learning theory . pp. 440–449 (1992)

1992
[33]

Solomonoff, R.J.: A formal theory of inductive inference. part i. Informa- tion and control 7(1), 1–22 (1964)

1964
[34]

Solomonoff, R.J.: A formal theory of inductive inference. part ii. Informa- tion and control 7(2), 224–254 (1964)

1964
[35]

Turing, A.M., et al.: On computable numbers, with an application to the entscheidungsproblem. J. of Math 58(345-363), 5 (1936)

1936
[36]

Advances in neural information processing systems 30 (2017)

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)

2017
[37]

In: The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024

Venkataramanan, S., Rizve, M.N., Carreira, J., Asano, Y.M., Avrithis, Y.: Is imagenet worth 1 video? learning strong image encoders from 1 long unlabelled video. In: The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. O...

2024
[38]

ICML (2024)

Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., Hobbhahn, M.: Will we run out of data? limits of llm scaling based on human-generated data. ICML (2024)

2024
[39]

arXiv preprint arXiv:2405.09591 (2024)

Wang, Z., Wang, P., Liu, K., Wang, P., Fu, Y., Lu, C.T., Aggarwal, C.C., Pei, J., Zhou, Y.: A comprehensive survey on data augmentation. arXiv preprint arXiv:2405.09591 (2024)

2024
[40]

IEEE transactions on evolutionary computation 1(1), 67–82 (1997) 30

Wolpert, D.H., Macready , W.G.: No free lunch theorems for optimization. IEEE transactions on evolutionary computation 1(1), 67–82 (1997) 30

1997
[41]

Yang, G., Hu, E., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., Gao, J.: Tuning large neural networks via zero- shot hyperparameter transfer. Advances in Neural Information Processing Systems 34, 17084–17097 (2021) A Appendix A.1 Model and t...

2021
[42]

We clip gradients by scaling every gradient vector larger than 1 to norm 1

with a weight decay of 0.01. We clip gradients by scaling every gradient vector larger than 1 to norm 1. The maximum macrobatch size is 500 in all experiments. We start at a batch size equal to the microbatch size, and warm up linearly to the maximum macrobatch size between 50...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Theoretical Computer Science 354(3), 391–404 (2006)

Antunes, L., Fortnow, L., Van Melkebeek, D., Vinodchandran, N.V.: Com- putational depth: concept and applications. Theoretical Computer Science 354(3), 391–404 (2006)

work page 2006

[2] [2]

Computer 49(05), 54–63 (may 2016)

Bal, H., Epema, D., de Laat, C., van Nieuwpoort, R., Romein, J., Sein- stra, F., Snoek, C., Wijshoff, H.: A medium-scale distributed system for computer science research: Infrastructure for the long term. Computer 49(05), 54–63 (may 2016). https://doi.org/10.1109/MC.2016.127

doi:10.1109/mc.2016.127 2016

[3] [3]

In: International conference on algorithmic learning theory

Bloem, P., Mota, F., de Rooij, S., Antunes, L., Adriaans, P.: A safe ap- proximation for kolmogorov complexity . In: International conference on algorithmic learning theory . pp. 336–350. Springer (2014)

work page 2014

[4] [4]

In: Algorithmic Learning Theory: 26th International Conference, ALT 2015, Banff, AB, Canada, October 4-6, 2015, Proceedings 26

Bloem, P., de Rooij, S., Adriaans, P.: Two problems for sophistication. In: Algorithmic Learning Theory: 26th International Conference, ALT 2015, Banff, AB, Canada, October 4-6, 2015, Proceedings 26. pp. 379–

work page 2015

[5] [5]

Bostock, Mike: D3.js, https://d3js.org/

work page

[6] [6]

Ihre geschichtliche Entwicklung und wirtschaftliche Seite

Braun, L.: Die Frauenfrage. Ihre geschichtliche Entwicklung und wirtschaftliche Seite. Hirzel (1901)

work page 1901

[7] [7]

Advances in neural information processing sys- tems 34, 28431–28441 (2021)

Chung, S., Siegelmann, H.: Turing completeness of bounded-precision re- current neural networks. Advances in neural information processing sys- tems 34, 28431–28441 (2021)

work page 2021

[8] [8]

IEEE Access 9, 51416–51431 (2021)

Collins, J., Chand, S., Vanderkop, A., Howard, D.: A review of physics simulators for robotic applications. IEEE Access 9, 51416–51431 (2021)

work page 2021