AI for the Open-World: the Learning Principles

Jianyu Zhang

REVIEW 4 major objections 6 minor 204 references

AI for the Open-World: the Learning Principles

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The thesis claims that standard i.i.d. training cannot by itself produce representations rich enough for open-world AI, and that open-world AI requires three learning principles: rich features, disentangled representation, and…

desk verdict A serious, experiment-heavy thesis on open-world learning with a load-bearing theoretical claim that outruns its own theorem and its own empirical story. read the letter →

arxiv 2504.14751 v2 pith:IMIIOAJC submitted 2025-04-20 cs.LG cs.AI

Jianyu Zhang This is my paper · ORCID

classification cs.LGcs.AI

keywords open-worldAIrichfeaturespredictivedisentanglementinference-timelearningout-of-distributiongeneralizationmemory-basedmodelsin-contextrepresentation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The thesis sets out to establish that success on one well-specified task does not transfer to open-world competence, where a machine must learn many new tasks quickly from few examples. Its central claim is that ordinary i.i.d. training—optimizing expected loss on a single distribution—has no incentive to build representations rich enough for such open-world competence, and that this is a structural limitation, not a matter of scale. The thesis argues that open-world AI requires three learning principles: rich features (a large toolbox), disentangled representation (an organized toolbox), and inference-time learning (a tool-savvy hand). It supports these principles with a linear-probing information framework, large-scale experiments in transfer learning, invariant learning, few-shot learning, and out-of-distribution fine-tuning, and language-model tests of memory-based architectures. If correct, the thesis says current foundation-model scaling on a single distribution is necessary but not sufficient, and that representation diversity, disentanglement, and memory-based adaptation must be engineered explicitly.

What carries the argument

The central object is the representation $\Phi$, a set of features evaluated by the optimal expected loss $C^*_P(\Phi)$ of a linear classifier under a convex loss. Theorem 2.1.6—any two global optima of the expected loss carry equivalent information in this linear-probing sense—is the load-bearing identity: it implies that optimizing one i.i.d. objective gives no incentive to accumulate features that are redundant in-distribution, regardless of model or data size. The second mechanism is predictive disentanglement, the pressure created by minimizing the sum (area under the curve) of prediction losses across context lengths, which pushes each associative memory unit to produce a good conditional expectation estimate after as few time steps as possible and thereby split the task into independently memorized subtasks. The third mechanism is the associative memory unit implemented by Gaussian kernel smoothing, whose retrieval is a conditional expectation $E(V|K)$; Memory Mosaics v2 layers short-term, long-term, and persistent memories to decouple context-dependent and persistent knowledge, enabling inference-time storage and retrieval.

What would settle it

Train a single network wide enough to match the in-distribution linear-probing error of a concatenation of several separately seeded networks of the same architecture, then compare both by linear probing on a battery of held-out distribution shifts; if the single wide network matches or beats the concatenation on every shift, the thesis's central claim that i.i.d. training cannot build rich features regardless of model size would be empirically contradicted.

Watch

Extended reading notes

Core claim

On its own terms, the thesis claims that a deep network trained by i.i.d. empirical risk minimization has no motivation to retain features that are redundant on the training distribution, even when those features would become useful under a different distribution. The framework defines a representation as a set of features judged by the optimal cost of a linear classifier, and Theorem 2.1.6 shows that any two globally optimal representations carry equivalent linearly exploitable information about the training distribution; hence adding extra features cannot improve in-distribution cost. The thesis then shows experimentally that representations obtained from several independently seeded training episodes, concatenated as Cat or distilled as Distill, transfer substantially better to new tasks and distributions than an equivalently sized single network, and that fine-tuning with very large dropout preserves this richness. In invariant learning, it argues that what matters is the initial rich feature set (Bonsai) rather than the invariance penalties themselves, which are either too strong to optimize or too weak to enforce the desired constraints. For disentanglement, it introduces predictive disentanglement, a training pressure that splits a prediction task into independently memorized subtasks, and grounds it in a reduction of sample complexity from linear to logarithmic dependence on representation dimension. For inference-time learning, it proposes memory-based architectures such as Memory Mosaics v2, which store and retrieve new knowledge at inference time, and reports that they outperform transformers on multi-document question answering, context-length extrapolation, and in-context classification while matching them on standard persistent-knowledge benchmarks.

Load-bearing premise

The load-bearing premise is that a theorem about idealized optimizers—global optima of an expected convex loss probed by a linear classifier—describes what stochastic gradient descent actually does in deep networks, so that features redundant on the training distribution are precisely the features that help under a shifted distribution.

Editorial extensions

If this is right

Simply scaling model and data under one i.i.d. distribution will not yield open-world competence; representation diversity must be explicitly produced, for example by combining independently trained networks or distilling them into one.
Fine-tuning a rich representation with a very high dropout rate (90% or more) becomes a practical tool for out-of-distribution robustness, outperforming ensembles and weight averaging in the reported settings.
In invariant learning, effort should shift from inventing invariance penalties to constructing rich initial features; a frozen rich representation can outperform trained invariance penalties.
Disentanglement lowers the number of examples needed to learn new tasks by reducing sample complexity from linear to logarithmic dependence on representation dimension.
Memory-based architectures that store and retrieve new knowledge at inference time can adapt to new tasks and long contexts without catastrophic forgetting and without position-encoding-dependent extrapolation failures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the thesis, the argument predicts that out-of-distribution robustness will systematically trade off against i.i.d. efficiency whenever regularizers prune redundant features; a benchmark that scores both would test whether that trade is fundamental or an artifact of current optimizers.
The thesis's mechanism suggests a testable extension for standard transformers: if predictive disentanglement is a general training pressure, plotting per-position loss curves on long contexts should reveal whether attention heads specialize by context length, as Memory Mosaics do.
The context-length extrapolation of Memory Mosaics v2 without fine-tuning implies that position encoding, not attention itself, may be the main bottleneck for length generalization; removing or replacing positional information in transformer-like models is a direct next experiment.
Read as a research program, this thesis implies that open-world evaluation should emphasize new-knowledge storage and retrieval and few-shot in-context adaptation under scarce examples, rather than aggregate accuracy on benchmarks that mostly test persistent knowledge.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this thesis. First, it contains real, large-scale experimental work that deserves attention: the Cat ensembles, Bonsai initialization, very-large dropout, and especially Memory Mosaics v2 are concrete contributions with results on external benchmarks like Camelyon17, RegBench, RULER, and standard language tasks. Second, the headline theoretical claim—that i.i.d. training 'lacks the motivation to construct a representation with rich features, regardless of model size and data size'—is not actually established by the paper's own theorem, and one section of the thesis contradicts it outright.

The core idea is worth taking seriously: expected-loss minimization under a single distribution P has no incentive to keep features that are redundant for P but useful under a shifted distribution Q. Theorem 2.1.6 only shows that all global minimizers of the idealized expected loss carry equivalent information under P. That says nothing about whether SGD can or will accumulate P-redundant features, and it says nothing about which minimizers SGD finds. The step to 'regardless of model size and data size' is an unsupported leap. The tension becomes explicit in Section 2.4.3.5, which argues that very-large dropout works precisely because i.i.d.-pretrained models contain rich features 'buried in their hidden layers' and that modern foundation models make this condition 'relatively easy to achieve.' That is a direct counterexample to the strong 'cannot' claim. What the theory plausibly shows is that the P-objective is indifferent to Q-useful features, not that such features are absent.

What is genuinely new and well done: Memory Mosaics v2—adaptive bandwidth, gated time-variant key extraction, three-level memory—plus the large-scale evaluations. The results are plausible: comparable performance on persistent-knowledge benchmarks, clear gains on long-context QA after fine-tuning (12–15% over transformers), and context-length extrapolation without fine-tuning. The authors also transferred transformer hyperparameters verbatim to Memory Mosaics v2, which is a fair baseline choice. The predictive disentanglement chapter is more suggestive than proven—the three-moons toy is nice but not a demonstration of language-scale disentanglement—but it is presented as a principle, not a fait accompli.

The thesis does lean heavily on the author's own prior algorithms to validate its own principles. That is not disqualifying given the external benchmarks, but it does mean the broad framing should carry a heavier burden of independent replication.

Who should read it: people working on OOD generalization, memory-augmented architectures, and in-context learning. It deserves a serious referee, not a desk reject. If it goes to a journal, the right request is not rejection but revision: soften the theoretical overclaim, address the internal contradiction, and separate the empirical findings from the three-principles manifesto.

My recommendation: engage with the work, cite the empirical results, but treat the 'i.i.d. training cannot produce rich features' claim as an open hypothesis rather than a proven theorem.

Referee Report

4 major / 6 minor

Summary. This PhD thesis argues that closed-world i.i.d. training is insufficient for open-world AI and proposes three learning principles: rich features, disentangled representation, and inference-time learning. Chapter 2 develops a linear-probing framework to argue that i.i.d. training lacks the motivation to discover rich features, then supports this with transfer, few-shot, invariant-learning, and fine-tuning experiments using Cat ensembles, Bonsai, and very-large dropout. Chapter 3 introduces predictive disentanglement and the Memory Mosaics architecture, with experiments on a three-moons toy task, BabiStories language modeling, and the RegBench in-context learning benchmark. Chapter 4 presents Memory Mosaics v2, a 9.9B-parameter memory-based architecture, and evaluates it on persistent-knowledge benchmarks, RULER-style multi-document QA, and in-context classification tasks, reporting substantial gains over transformers on new-knowledge storage and retrieval.

Significance. If the thesis' central claims were fully established, the work would be significant: it challenges the prevailing foundation-model scaling paradigm and points to concrete alternatives (Cat ensembles, Bonsai initialization, very-large dropout, Memory Mosaics v2) that are tested on external benchmarks such as Camelyon17, RegBench, RULER, and standard language tasks. The strength of the manuscript is its breadth of empirical evidence and the reproducibility-oriented release of code. The theoretical framework in Section 2.1 is clean, and the observation that multiple training episodes yield features that are P-equivalent but Q-beneficial is a valuable contribution. However, the sharpest advertised conclusion—that i.i.d. training 'cannot' discover rich features—is not supported by the stated theorem, and the manuscript contains an internal tension with its own very-large-dropout analysis.

major comments (4)

[§2.2, Theorem 2.1.6] The claim that Section 2.1 'theoretically proves that this approach lacks the motivation to construct a representation with rich features, regardless of model size and data size' is an overstatement. Theorem 2.1.6 only establishes that all global minimizers of the expected-loss problem (2.2) carry equivalent information under P, where information is defined via optimal linear-probing cost. Because problem (2.2) is unconstrained, a global minimizer can include extra P-redundant features alongside a sparse one; the theorem does not rule out rich representations, and it says nothing about which minimizers SGD finds. The missing premise is an explicit argument that SGD's implicit sparsity bias discards exactly the P-redundant features that would be Q-useful. The literature cited in §2.1.4 suggests a connection but does not establish this premise. The conclusion should be weakened to 'provides no incentive' or 'may not discover' rather than 'cannot'.
[§2.4.3.5] This passage directly contradicts the 'regardless of model size and data size' conclusion. The text states that fine-tuning with very-large dropout works precisely because i.i.d.-pretrained models contain rich features 'buried in their hidden layers' and that modern foundation models make this condition 'relatively easy to achieve.' That reading implies i.i.d. training can and does produce rich features, at least in large models, which undermines the theoretical claim in §2.2. The manuscript needs to reconcile these positions, for example by distinguishing features that are linearly exploitable under P from features that are present but not linearly accessible, and by clarifying that the theory addresses the former while the dropout experiments exploit the latter.
[§3.2.2 and §3.5] The chapter claims that predictive disentanglement drives the learning of disentangled representations, but the evidence does not directly measure disentanglement. The three-moons experiment shows a functional benefit of multiple memory heads, and the RegBench results show strong in-context learning, yet neither quantifies the degree of disentanglement, e.g., through a dimension-factor correspondence metric. Moreover, §3.6 concedes that the analogous effect in standard transformers is only a conjecture. The disentanglement principle would be on firmer ground if the paper either reported a direct measure of disentanglement in the trained Memory Mosaics or explicitly framed predictive disentanglement as a hypothesis supported by indirect evidence.
[§4.2.3] The claim that Memory Mosaics v2 'substantially outperforms transformers (>10%)' is based primarily on the new-knowledge storage tasks (RULER multi-document QA) and the in-context classification tasks. On the 19 persistent-knowledge benchmarks in Table 4.1, Memory Mosaics v2 and transformers perform nearly identically (e.g., 52.2% vs 52.2% for large models). Since the thesis' third principle concerns learning new tasks with fewer examples, the evidence is relevant, but the scope should be stated precisely; the current phrasing implies a broader superiority than the benchmarks support.

minor comments (6)

[Abstract and throughout] There are frequent grammatical and spelling errors, including 'numerous successes of AI has been made,' 'less priori knowledge' (should be 'prior knowledge'), 'innovated techniques' (should be 'innovative'), and 'qick-learning' in the Section 3.2 heading. These should be corrected in a full revision.
[§2.4] The method is referred to inconsistently as 'very large dropout' and 'very-large dropout'; one hyphenated form should be used throughout.
[§4.2.3.1] The sentence 'This is nothing to supervise' appears to be a typo, likely for 'This is nothing to be surprised about' or similar; please rephrase.
[§4.2.3.2] The text refers to 'Appendix Table C.1' without a corresponding in-text description of its contents; please either describe the table or move it into the main text.
[§3.3] The captions read 'Single head network' and 'Three-heads network'; the latter should be 'Three-head network' for grammatical consistency.
[References] Some citations are abbreviated inconsistently (e.g., 'Chen et al. 2019' and 'Chen et al. 2020' for the same few-shot baseline work), and the bibliography should be checked for completeness.

Circularity Check

1 steps flagged · score 6.0 of 10

The 'theory' that i.i.d. training cannot learn rich features restates the paper's own definitions of information and richness; the experimental validation remains external and non-circular.

self definitional [Section 2.2 (first paragraph) reading Theorem 2.1.6 in Section 2.1.1]
"Section 2.1 theoretically proves that this approach lacks the motivation to construct a representation with rich features, regardless of model size and data size."

The claimed impossibility follows from the paper's definitions, not from training facts. Theorem 2.1.6 concerns global minimizers of (2.2), i.e. of C*_P(Phi), the in-distribution linear-probing cost. Definitions 2.1.2-2.1.4 define 'information' via C*_P, and the Introduction defines 'rich features' as features 'beyond the need of i.i.d. generalization.' So 'optimizing (2.2) lacks motivation to build P-redundant features' restates what the objective and 'richness' were defined to mean. The theorem only gives C*_P-equivalence of global optima; it says nothing about SGD or finite data, and 'regardless of model size and data size' is an extrapolation.

full rationale

Most experimental claims are not circular: Cat, Bonsai, very-large dropout, and Memory Mosaics are evaluated against external benchmarks (ImageNet/CIFAR/Inaturalist, DomainBed, Camelyon17, RegBench, RULER) and against strong baselines with hyperparameters transferred from the baseline architectures. No fitted parameter is renamed as a prediction. The thesis does rely heavily on the author's own prior papers (Zhang-Bottou 2023; Zhang et al. 2022; Zhang-Bottou 2024; Zhang et al. 2025), but these are not invoked as unverified authorities; they contain the same experiments and benchmarks, so the self-citation is not load-bearing circularity. The one genuine circular element is the theoretical derivation of the central impossibility claim in Section 2.2: it reduces to the definitions of 'information' (C*_P-based) and 'rich features' (beyond i.i.d. need). The theorem's 'regardless of model size and data size' conclusion is an unjustified extrapolation, and Section 2.4.3.5's 'buried features' passage explicitly contradicts it. Because this tautological step underpins the thesis's headline theoretical claim, the paper is partially circular even though its experimental validation is externally grounded.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on several domain assumptions about the open world and about the behavior of SGD, plus hand-selected hyperparameters for headline methods. No new physical entities are introduced.

free parameters (3)

very-large dropout rate (lambda) = 0.90
Headline fine-tuning results use a 90% penultimate-layer dropout; Table 2.11 shows the OOD performance curve is smooth around 90-95%, so the method is robust, but the central comparison uses a hand-chosen rate.
Bonsai discovery rounds (K) = 2
The Camelyon17 results use a 2-round Bonsai representation; Table 2.7 shows a third round yields weaker features, so K=2 is a selectively chosen hyperparameter.
short-term memory window h and long-term delay m = h=256; m sampled from [64,256] during training, fixed at 64 for inference
Memory Mosaics v2 uses these three-level memory hyperparameters; the text says they were transferred from the transformer baseline without further search, so they are not independently tuned for the proposed architecture.

assumptions (5)

domain assumption The open-world AI goal is defined as performing any task a human could undertake, with few examples and little task-specific prior knowledge, and closed-world competence does not transfer to this goal.
This definition frames the entire thesis and is not derived from data; it is introduced in the abstract and Section 1.
domain assumption The world's generative latent factors change sparsely or locally, such that Sparse Mechanism Shift and Symmetry Transformation make disentangled representations learnable and sample-efficient.
Invoked in Chapter 3's introduction to justify why disentanglement reduces sample complexity; cited to Thomas et al. and Higgins et al. but adopted as a premise.
ad hoc to paper Practical stochastic gradient training discards features that are not incrementally useful under the training distribution, and these weakly relevant features are the right candidates for out-of-distribution usefulness.
Used in Sections 2.1.4 and 2.4 to connect the idealized theorem to Cat, Bonsai, and very-large dropout; it is a heuristic about SGD, not a proven statement.
domain assumption Fine-tuning a pre-trained residual network on a small dataset is approximately a linear process that can exploit existing buried features but cannot create new ones, justifying very large dropout on the penultimate layer.
Section 2.4.2 relies on this near-linearity, citing Raghu et al. and the residual decomposition of Veit et al.; if false, the dropout result would not support the rich-features principle.
domain assumption Combining pieces of knowledge yields exponentially many combinations and knowledge changes over time, so training on everything is neither computationally feasible nor sufficient for open-world AI.
Chapter 4 uses this combinatorial and non-stationary premise to motivate inference-time learning; it is stated, not empirically demonstrated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of AI for the Open-World: the Learning Principles." pith.science (2026). https://pith.science/paper/IMIIOAJC

@misc{pith2026250414751,
  author       = {Pith},
  title        = {Pith review of: AI for the Open-World: the Learning Principles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IMIIOAJC}},
  note         = {Machine review of arXiv:2504.14751}
}

read the original abstract

During the past decades, numerous successes of AI has been made on "specific capabilities", named closed-world, such as artificial environments or specific real-world tasks. This well-defined narrow capability brings two nice benefits, a clear criterion of success and the opportunity to collect a lot of examples. The criteria not only reveal whether a machine has achieved a goal, but reveal how the machine falls short of the goal. As a result, human designers can fix the problems one after the other until the machine is deemed good enough for the task. Furthermore, the large set of collected examples reduces the difficulty of this problem-fixing process (by the central limit theorem). Do the success in closed-world translate into broad open-world, where a machine is required to perform any task that a human could possibly undertake with fewer examples and less priori knowledge from human designers? No. Because competence in a specific task provides little insight in handling other tasks, the valuable criteria for specific tasks become helpless when handling broader unseen tasks. Furthermore, due to the shortage of examples in unseen tasks, central limit theorem does not stand on our side. At the end, human designers lose the oscilloscope to "hack" an AI system for the open-world. Achieving AI for the open-world requires unique learning principles and innovated techniques, which are different from the ones in building AI for the closed-world. This thesis explores necessary learning principles required to construct AI for the open-world, including rich features (analogy a large tool box), disentangled representation (an organized tool box), and inference-time learning (a tool-savvy hand). Driven by the learning principles, this thesis further proposes techniques to use the learning principles, conducts enormous large-scale experiments to verify the learning principles.

Figures

Figures reproduced from arXiv: 2504.14751 by the authors.

**Figure 2.1.** Supervised transfer learning from ImageNet to Inat18, Cifar100, and Cifar10. The top row shows the superior linear probing performance of the Cat𝑛 networks (blue, “cat”). The bottom row shows the performance of fine-tuned Cat𝑛, which is poor with normal fine-tuning (gray, “[init]cat”) and excellent for two-stage fine tuning (blue, “[2ft]cat”). Distill𝑛 (pink, “distill”) representation is obtained by distilling Cat𝑛 … view at source ↗

**Figure 2.2.** Two-stage fine-tuning consists of fine-tuning each network separately, then concatenating their feature extractors, now frozen, and training a final classifier. Fine-tuning is usually achieved by setting up a linear classifier on top of the transferred feature and training it on the target task data while allowing back-propagation to update the transferred features as well. The bottom row of [PITH_FULL_IMAGE:figu… view at source ↗

**Figure 2.3.** Self-supervised transfer learning with swav trained on unlabeled ImageNet(1k) (top row) and with seer on Instagram1B (bottom row). The constructed rich representation, Cat𝑛, yields the best linear probing performance (“cat” and “catsub”) for supervised ImageNet, Inat18, Cifar100, and Cifar10 target tasks. The two-stage fine-tuning (“[2ft]cat”) matches equivalently sized baseline models (“[init]wide” and “[init]wide&… view at source ↗

Figures from the paper (25 more)

**Figure 2.4.** Figure 2.4: Supervised transfer learning from ImageNet21k to ImageNet on vision transformers. 2.2.2 Self-supervised transfer learning In self-supervised transfer learning (SSL), transferable representations are no longer constructed using a supervised auxiliary task, but using a…

**Figure 2.5.** Figure 2.5: Few-shot learning performance on MiniImageNet and Cub. Four common few-shot learning algorithms are shown in red (results from Chen et al. [2019]). Two supervised transfer methods, with either a linear classifier (Baseline) or cosine-based classifier (Baseline++) are…

**Figure 2.6.** Figure 2.6: (Distill𝑛) A multiple head network (red) trained to predict the outputs of the pre-trained networks Φ1, Φ2, · · · (blue) must develop a representation Φ that subsumes those of all the blue networks. The same distillation process is used by the Bonsai algorithm [Zhang…

**Figure 2.7.** Figure 2.7: Test performance of nine penalized invariant-learning methods as a function of the number of epochs used to pre-train the neural network with ERM. The final o.o.d. testing performance is very dependent on choosing the right number of pretraining epochs, illustrating …

**Figure 2.8.** Figure 2.8: o.o.d. performance of invariant-learning methods as a function of training epochs. Top: Six invariant-learning methods are trained from a ‘perfect’ initialization where only the robust feature is well learned. The blue star indicates the initial test accuracy. Bottom…

**Figure 2.9.** Figure 2.9: Left half: o.o.d. testing accuracy as a function of the penalty weight. The six plots correspond to the IRMv1, vREx, and CLOvE algorithms with all other hyper-parameters selected using either the IID or OoD tuning method. Bonsai initialization makes these curves far …

**Figure 2.10.** Figure 2.10: presents the box plot of different hyperparameter combinations (where each choice of training sub-datasets searches 6 hyperparameter combinations). On all four datasets, the bottom of very-large dropout box (25% quartile) outperforms the top of other baseline boxes …

**Figure 2.11.** Figure 2.11: Comparison of dropout rates when training a ResNet50 network from scratch on the vlcs dataset. The optimal dropout rate falls to about zero. Dropout rates greater than 50% negatively impact both the i.i.d.. and the o.o.d. performances. As a reference, a naive linear…

**Figure 3.1.** Figure 3.1: Sample complexity of linear model on disentangled representation and linearly entangled representation. Each point in the figure represents an average of 500 repeating experiments. The results show the huge sample complexity gap between disentangled representation an…

**Figure 3.2.** Figure 3.2: Elementary memory unit. The keys 𝑘𝑇 are computed as a function of past observations (𝑥𝑡)𝑡 ≤𝑇 . The values𝑣𝑇 peek into the future. In this example, the value also depend on the next observation 𝑥𝑇 +1. At time 𝑇 , the associative memory uses the known key 𝑘𝑇 to compute…

**Figure 3.3.** Figure 3.3: The curve plots the prediction losses for all training sequence indices 𝑡 ∈ {1 . . . 𝐷} in the training sequence. Minimizing their sum —the area under the curve— favors memories that produce useful value estimates after fewer time steps. conditional expectations E(𝑉 …

**Figure 3.4.** Figure 3.4: An architecture for the three moons problem. We consider single-layer networks with either 𝑁ℎ = 1 or 𝑁ℎ = 3 memory units whose keys and values belong to either C 3 (𝑁ℎ = 1) or C 1 (𝑁ℎ = 3). Both nets have 3 × 3 × 2 × 3 = 54 trainable real parameters that determine ho…

**Figure 3.5.** Figure 3.5: Single head network prediction error versus context length. The prediction error shows a sharp transition after lcm(𝑝1, 𝑝2, 𝑝3) observations (red vertical line), when the network switches from predicting the future moon position by repeating the last observation to p…

**Figure 3.7.** Figure 3.7: Left: Classic GPT2-small transformer. Right: GPT2-like Memory Mosaic ahead.10 Accordingly, the attention mask excludes the main diagonal to avoid breaking causality. Finally, the feed forward networks (FFNs) of the classic transformers blocks are replaced by a layer …

**Figure 3.8.** Figure 3.8: Training and validation loss of the transformer and Memory Mosaic architectures trained on BabiStories for different model depths. The horizontal axis represents the number of training iterations. All hyper-parameters have been tuned on the transformer architecture …

**Figure 3.9.** Figure 3.9: Prediction performance on the Simple English Wikipedia dataset using models trained on BabiStories. The plot shows the pertoken average loss as a function of the position of the generated token in the 512-token long input window. Memory Mosaics outperform transfo…

**Figure 4.1.** Figure 4.1: The differences between a global parametric representation (i.e. neural network) and a locally weighted learning approach (i.e. local weighted regression) [Atkeson et al. 1997]. A sigmoid neural network (marked as “NN”) and a local (quadratic) weighted regression (ma…

**Figure 4.2.** Figure 4.2: Average attention scores of the last token attending previous tokens. Left: Transformer with RoPE position encoding. Right: Memory Mosaics in Section 3.2.1. The (averaged) attention scores in transformer heavily depends on token positions (curly curves), while the at…

**Figure 4.3.** Figure 4.3: Memory Mosaics v2 architecture. 9A two-layers feed-forward network and a key-value associative memory are interchangeable as shown in Sukhbaatar et al. [2019]. 69 [PITH_FULL_IMAGE:figures/full_fig_p074_4_3.png]

**Figure 4.4.** Figure 4.4: Training data sequence length distributions. For a given maximum sequence length during training (e.g. 4k), longer sequences are truncated to the maximum sequence length. This truncation results in the peaks at the end of distributions. hyperparameters For all Memory…

**Figure 4.5.** Figure 4.5: Randomly overlapped long-term & short-term memory 4.2.3 Three evaluation dimensions The evaluation design provides a means to assess a specific property of a system and contains a clear goal.12 The main focus of this thesis is the ability to learn new tasks with fewe…

**Figure 4.6.** Figure 4.6: Semantic label in-context learning comparison between Memory Mosaics v2 (small/large) and Transformer (small/large). Memory Mosaics v2 significantly outperform Transformer on in-context learning with a large margin (more than 10%). Meanwhile, Memory Mosaics v2 benefi…

**Figure 4.7.** Figure 4.7: Anonymous label in-context learning comparison between Memory Mosaics v2 (small/large) and Transformer (small/large). Memory Mosaics v2 significantly outperform Transformer on in-context learning with a large margin (more than 10%). performs exceptionally well when i…

**Figure 4.8.** Figure 4.8: In-context learning (multiclass classification with semantic labels) comparison between Memory Mosaics v2 Large and Transformer Large. Memory Mosaics v2 is trained on 1t tokens, while three transformers are trained on 200b, 1t, 8t tokens, respectively. Transformer w…

**Figure 4.9.** Figure 4.9: In-context learning (multiclass classification with anonymous labels) comparison between Memory Mosaics v2 Large and Transformer Large. The Transformer trained on 8t (dash red line) still lags behind Memory Mosaics v2 trained on 1t (solid blue line) by a large margin…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

204 extracted references · 19 canonical work pages

[1]

Systematic generalisation with group invariant predictions

Faruk Ahmed, Yoshua Bengio, Harm van Seijen, and Aaron Courville. Systematic generalisation with group invariant predictions. In International Conference on Learning Representations, 2020

2020
[2]

Empirical or invariant risk minimization? a sample complexity perspective

Kartik Ahuja, Jun Wang, Amit Dhurandhar, Karthikeyan Shanmugam, and Kush R Varshney. Empirical or invariant risk minimization? a sample complexity perspective. arXiv preprint arXiv:2010.16412, 2020

arXiv 2010
[3]

In-context language learning: Arhitectures and algorithms

Ekin Aky \"u rek, Bailin Wang, Yoon Kim, and Jacob Andreas. In-context language learning: Arhitectures and algorithms. arXiv preprint arXiv:2401.12973, 2024

arXiv 2024
[4]

Sgd with large step sizes learns sparse features

Maksym Andriushchenko, Aditya Varre, Loucas Pillaud-Vivien, and Nicolas Flammarion. Sgd with large step sizes learns sparse features. arXiv preprint arXiv:2210.05337, 2022

arXiv 2022
[5]

Sgd with large step sizes learns sparse features

Maksym Andriushchenko, Aditya Vardhan Varre, Loucas Pillaud-Vivien, and Nicolas Flammarion. Sgd with large step sizes learns sparse features. In International Conference on Machine Learning, pages 903--925. PMLR, 2023

2023
[6]

Invariant risk minimization

Martin Arjovsky, L \'e on Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019

arXiv 1907
[7]

Invariant risk minimization

Martin Arjovsky, L\'eon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv, 2020

2020
[8]

Ensemble of averages: Improving model selection and boosting performance in domain generalization

Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong. Ensemble of averages: Improving model selection and boosting performance in domain generalization. Advances in Neural Information Processing Systems, 35: 0 8265--8277, 2022

2022

Show all 204 references

[9]

Pitfalls of in-domain uncertainty estimation and ensembling in deep learning

Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. arXiv preprint arXiv:2002.06470, 2020

2002 arXiv
[10]

Locally weighted learning

Christopher G Atkeson, Andrew W Moore, and Stefan Schaal. Locally weighted learning. Lazy learning, pages 11--73, 1997

1997
[11]

Multiple kernel learning, conic duality, and the smo algorithm

Francis R Bach, Gert RG Lanckriet, and Michael I Jordan. Multiple kernel learning, conic duality, and the smo algorithm. In Proceedings of the twenty-first international conference on Machine learning, page 6, 2004

2004
[12]

Meta-learned invariant risk minimization

Jun-Hyun Bae, Inchul Choi, and Minho Lee. Meta-learned invariant risk minimization. arXiv, 2021

2021
[13]

Neural machine translation by jointly learning to align and translate

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conferen...

2015
[14]

From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge

Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al. From detection of individual metastases to classification of lymph node status at the patient level:...

2018
[15]

Predict then interpolate: A simple algorithm to learn stable classifiers, 2021

Yujia Bao, Shiyu Chang, and Regina Barzilay. Predict then interpolate: A simple algorithm to learn stable classifiers, 2021

2021
[16]

Vicreg: Variance-invariance-covariance regularization for self-supervised learning

Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906, 2021

2021 arXiv
[17]

o ppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, G \

Maximilian Beck, Korbinian P \"o ppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, G \"u nter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. Advances in Neural Information Processing Systems, 37: 0 107547-...

2025
[18]

Recognition in terra incognita

Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pages 456--473, 2018

2018
[19]

Longformer: The long-document transformer

Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020

2004 arXiv
[20]

Robust Optimization, volume 28 of Princeton Series in Applied Mathematics

Aharon Ben - Tal, Laurent El Ghaoui, and Arkadi Nemirovski. Robust Optimization, volume 28 of Princeton Series in Applied Mathematics. Princeton University Press, 2009

2009
[21]

Deep learning of representations: Looking forward

Yoshua Bengio. Deep learning of representations: Looking forward. In Statistical Language and Speech Processing: First International Conference, SLSP 2013, Tarragona, volume 7978, page 1. Springer, 2013 a

2013
[22]

Deep learning of representations: Looking forward

Yoshua Bengio. Deep learning of representations: Looking forward. In International conference on statistical language and speech processing, pages 1--37. Springer, 2013 b

2013
[24]

A meta-transfer objective for learning to disentangle causal mechanisms

Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, S \'e bastien Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. A meta-transfer objective for learning to disentangle causal mechanisms. arXiv preprint arXiv:1901.10912, 2019 b

1901 arXiv
[25]

Backgammon computer program beats world champion

Hans J Berliner. Backgammon computer program beats world champion. Artificial Intelligence, 14 0 (2): 0 205--220, 1980

1980
[26]

Birth of a transformer: A memory viewpoint

Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Herve Jegou, and Leon Bottou. Birth of a transformer: A memory viewpoint. Advances in Neural Information Processing Systems, 36, 2024

2024
[27]

Implicit regularization for deep neural networks driven by an ornstein-uhlenbeck like process

Guy Blanc, Neha Gupta, Gregory Valiant, and Paul Valiant. Implicit regularization for deep neural networks driven by an ornstein-uhlenbeck like process. In Conference on learning theory, pages 483--513. PMLR, 2020

2020
[28]

On the opportunities and risks of foundation models

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021

2021 arXiv
[29]

From machine learning to machine reasoning

L\'eon Bottou. From machine learning to machine reasoning. Technical report, arXiv:1102.1808, February 2011

2011 arXiv
[30]

Local learning algorithms

L \'e on Bottou and Vladimir Vapnik. Local learning algorithms. Neural computation, 4 0 (6): 0 888--900, 1992

1992
[31]

Optimization methods for large-scale machine learning

L \'e on Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60 0 (2): 0 223--311, 2018

2018
[32]

AI for open world, https://leon.bottou.org/feuilleton/turing

Bottou 2018. AI for open world, https://leon.bottou.org/feuilleton/turing

2018
[33]

Approximate matching: Definition and terminology

Frank Breitinger, Barbara Guttman, Michael McCarrin, Vassil Roussev, and Douglas White. Approximate matching: Definition and terminology. techreport nist special publication 800-168. national institute of standards and technology, 2014

2014
[34]

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

1901
[35]

Attribute bagging: improving accuracy of classifier ensembles by using random feature subsets

Robert Bryll, Ricardo Gutierrez-Osuna, and Francis Quek. Attribute bagging: improving accuracy of classifier ensembles by using random feature subsets. Pattern recognition, 36 0 (6): 0 1291--1302, 2003

2003
[36]

Unsupervised learning of visual features by contrasting cluster assignments

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33: 0 9912--9924, 2020

2020
[37]

Efficient intent detection with dual sentence encoders

I \ n igo Casanueva, Tadas Tem c inas, Daniela Gerz, Matthew Henderson, and Ivan Vuli \'c . Efficient intent detection with dual sentence encoders. arXiv preprint arXiv:2003.04807, 2020

2003 arXiv
[38]

Transformer flops

Adam Casson. Transformer flops. 2023. URL https://adamcasson.com/posts/transformer-flops

2023
[39]

Swad: Domain generalization by seeking flat minima

Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34: 0 22405--22418, 2021

2021
[40]

Extending context window of large language models via positional interpolation

Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023 a

2023 arXiv
[41]

A simple framework for contrastive learning of visual representations

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597--1607. PMLR, 2020

2020
[42]

A closer look at few-shot classification

Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. arXiv preprint arXiv:1904.04232, 2019

1904 arXiv
[43]

Exploring simple siamese representation learning

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. arXiv preprint arXiv:2011.10566, 2020

2011 arXiv
[44]

Towards understanding feature learning in out-of-distribution generalization

Yongqiang Chen, Wei Huang, Kaiwen Zhou, Yatao Bian, Bo Han, and James Cheng. Towards understanding feature learning in out-of-distribution generalization. arXiv preprint arXiv:2304.11327, 2023 b

2023 arXiv
[45]

Magicpig: Lsh sampling for efficient llm generation

Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, et al. Magicpig: Lsh sampling for efficient llm generation. arXiv preprint arXiv:2410.16179, 2024

arXiv
[46]

Natural language processing (almost) from scratch

Ronan Collobert, Jason Weston, L\'eon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12: 0 2493--2537, Aug 2011

2011
[47]

Independent Component Analysis, a new concept? Signal Processing , 36: 0 287--314, Apr 1994

Pierre Comon. Independent Component Analysis, a new concept? Signal Processing , 36: 0 287--314, Apr 1994

1994
[48]

Environment inference for invariant learning

Elliot Creager, J \"o rn-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In International Conference on Machine Learning, pages 2189--2200. PMLR, 2021

2021
[49]

Goemotions: A dataset of fine-grained emotions

Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. Goemotions: A dataset of fine-grained emotions. arXiv preprint arXiv:2005.00547, 2020

2005 arXiv
[50]

Conditional meta-learning of linear representations

Giulia Denevi, Massimiliano Pontil, and Carlo Ciliberto. Conditional meta-learning of linear representations. Advances in Neural Information Processing Systems, 35: 0 253--266, 2022

2022
[51]

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database . In CVPR09, 2009

2009
[52]

Ensemble methods in machine learning

Thomas G Dietterich. Ensemble methods in machine learning. In International workshop on multiple classifier systems, pages 1--15. Springer, 2000

2000
[53]

An image is worth 16x16 words: Transformers for image recognition at scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...

2010 arXiv
[54]

Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759, 2023

Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759, 2023

2023 arXiv
[55]

ELIZA, https://en.wikipedia.org/wiki/ELIZA1972

Eliza 1972. ELIZA, https://en.wikipedia.org/wiki/ELIZA1972

1972
[56]

Head2toe: Utilizing intermediate representations for better transfer learning

Utku Evci, Vincent Dumoulin, Hugo Larochelle, and Michael C Mozer. Head2toe: Utilizing intermediate representations for better transfer learning. In International Conference on Machine Learning, pages 6009--6033. PMLR, 2022

2022
[57]

Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias

Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In Proceedings of the IEEE International Conference on Computer Vision, pages 1657--1664, 2013

2013
[58]

Model-agnostic meta-learning for fast adaptation of deep networks

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126--1135. PMLR, 2017

2017
[59]

Ensemble deep learning: A review

Mudasir A Ganaie, Minghui Hu, et al. Ensemble deep learning: A review. arXiv preprint arXiv:2104.02395, 2021

2021 arXiv
[60]

Ustinova, Hana Ajakan, Pascal Germain, H

Yaroslav Ganin, E. Ustinova, Hana Ajakan, Pascal Germain, H. Larochelle, François Laviolette, Mario Marchand, and Victor S. Lempitsky. Domain-adversarial training of neural networks. In Journal of machine learning research, 2015. URL https://api.semanticscholar.org/CorpusID:2871880

2015
[61]

Garc\'ia-Portugu\'es

E. Garc\'ia-Portugu\'es. Notes for Nonparametric Statistics. 2024. URL https://bookdown.org/egarpor/NP-UC3M/. Version 6.9.1. ISBN 978-84-09-29537-1

2024
[62]

Better & faster large language models via multi-token prediction

Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozi \`e re, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. arXiv preprint arXiv:2404.19737, 2024

2024 arXiv
[63]

Understanding the difficulty of training deep feedforward neural networks

Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings...

2010
[64]

No one representation to rule them all: Overlapping features of training methods

Raphael Gontijo-Lopes, Yann Dauphin, and Ekin Dogus Cubuk. No one representation to rule them all: Overlapping features of training methods. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=BK-4qbGgIE3

2022
[65]

Accurate, large minibatch sgd: Training imagenet in 1 hour

Priya Goyal, Piotr Doll \'a r, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017

2017 arXiv
[66]

Vision models are more robust and fair when pretrained on uncurated images without supervision

Priya Goyal, Quentin Duval, Isaac Seessel, Mathilde Caron, Mannat Singh, Ishan Misra, Levent Sagun, Armand Joulin, and Piotr Bojanowski. Vision models are more robust and fair when pretrained on uncurated images without supervision. arXiv preprint arXiv:2202.08360, 2022

2022 arXiv
[67]

The fast Gauss transform

Leslie Greengard and John Strain. The fast Gauss transform. SIAM Journal on Scientific and Statistical Computing , 12 0 (1): 0 79--94, 1991

1991
[69]

Mamba: Linear-time sequence modeling with selective state spaces

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023 b

2023 arXiv
[70]

In search of lost domain generalization

Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020

2007 arXiv
[71]

In search of lost domain generalization

Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=lQdXeXDoWtI

2021
[72]

Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

2025 arXiv
[73]

Changing Answer Order Can Decrease MMLU Accuracy , June 2024

Vipul Gupta, David Pantoja, Candace Ross, Adina Williams, and Megan Ung. Changing Answer Order Can Decrease MMLU Accuracy , June 2024. URL http://arxiv.org/abs/2406.19470. arXiv:2406.19470 [cs] version: 1

2024 arXiv
[74]

Structural risk minimization for character recognition

Isabelle Guyon, Vladimir Vapnik, Bernhard Boser, Leon Bottou, and Sara A Solla. Structural risk minimization for character recognition. Advances in neural information processing systems, 4, 1991

1991
[75]

Mathematical Structures of Language

Zellig Harris. Mathematical Structures of Language. John Wiley & Sons, 1968

1968
[76]

The elements of statistical learning, 2009

Trevor Hastie, Robert Tibshirani, Jerome Friedman, et al. The elements of statistical learning, 2009

2009
[77]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770--778, 2016 a

2016
[78]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770--778, 2016 b

2016
[79]

Multicalibration: Calibration for the ( C omputationally-identifiable) masses

Ursula Hebert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Calibration for the ( C omputationally-identifiable) masses. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Pr...

1939
[80]

Towards a definition of disentangled representations

Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.02230, 2018

2018 arXiv
[81]

Distilling the knowledge in a neural network

Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2 0 (7), 2015

2015 arXiv
[82]

Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024

2024 arXiv
[83]

Densely connected convolutional networks

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700--4708, 2017

2017
[84]

Xing, and Dong Huang

Zeyi Huang, Haohan Wang, Eric P. Xing, and Dong Huang. Self-challenging improves cross-domain generalization. arXiv, 2020

2020
[85]

P. Indyk. Nearest neighbours in high-dimensional spaces. In: J.E. Goodman, J. O’Rourke, Eds., Handbook of Discrete and Computational Geometry, 2004

2004
[86]

IsaacZhy. 2023. roberta-large-goemotions: https://huggingface.co/IsaacZhy/roberta-large-goemotions

2023
[87]

Adaptive mixtures of local experts

Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3 0 (1): 0 79--87, 1991

1991
[88]

Neural tangent kernel: Convergence and generalization in neural networks

Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume...

2018
[89]

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

2024
[90]

Understanding dimensional collapse in contrastive self-supervised learning

Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2110.09348, 2021

2021 arXiv
[91]

Inferring algorithmic patterns with stack-augmented recurrent nets

Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015

2015
[92]

Sutherland, and Nathan Srebro

Pritish Kamath, Akilesh Tangella, Danica J. Sutherland, and Nathan Srebro. Does invariant risk minimization capture invariance? AISTATS, 2021

2021
[93]

How to use dropout correctly on residual networks with batch normalization

Bum Jun Kim, Hyeyeon Choi, Hyeonah Jang, Donggeon Lee, and Sang Woo Kim. How to use dropout correctly on residual networks with batch normalization. In Uncertainty in Artificial Intelligence, pages 1058--1067. PMLR, 2023

2023
[94]

Last layer re-training is sufficient for robustness to spurious correlations

Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. In ICML 2022: Workshop on Spurious Correlations, Invariance and Stability, 2022. URL https://openreview.net/forum?id=THOOBy1uWVH

2022
[95]

Overcoming catastrophic forgetting in neural networks

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of scien...

2017
[96]

Earnshaw, Imran S

Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, A...

2021
[97]

Wilds: A benchmark of in-the-wild distribution shifts

Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, p...

2021
[98]

Out-of-distribution generalization with maximal invariant predictor

Masanori Koyama and Shoichiro Yamaguchi. Out-of-distribution generalization with maximal invariant predictor. arXiv, 2020

2020
[99]

Learning multiple layers of features from tiny images

Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. URL https://www.cs.toronto.edu/ kriz/learning-features-2009-TR.pdf

2009
[100]

A new frontier for hopfield networks

Dmitry Krotov. A new frontier for hopfield networks. Nature Reviews Physics, 5 0 (7): 0 366--367, 2023

2023
[101]

Out-of-distribution generalization via risk extrapolation (rex)

David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex). arXiv, 2020

2020
[102]

Out-of-distribution generalization via risk extrapolation (rex)

David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning, pages 5815--5826. PMLR, 2021

2021
[103]

Fine-tuning can distort pretrained features and underperform out-of-distribution

Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202.10054, 2022 a

2022 arXiv
[104]

Fine-tuning can distort pretrained features and underperform out-of-distribution

Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations, 2022 b . URL https://openreview.net/forum?id=UYneFzXSJWh

2022
[105]

Trainable calibration measures for neural networks from kernel mean embeddings

Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine...

2018
[106]

Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks

Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International conference on machine learning, pages 2873--2882. PMLR, 2018

2018
[107]

A path towards autonomous machine intelligence version 0.9

Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62 0 (1): 0 1--62, 2022

2022
[108]

Learning physical intuition of block towers by example

Adam Lerer, Sam Gross, and Rob Fergus. Learning physical intuition of block towers by example. In International conference on machine learning, pages 430--438. PMLR, 2016

2016
[109]

Deeper, broader and artier domain generalization

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pages 5542--5550, 2017

2017
[110]

Learning to generalize: Meta-learning for domain generalization

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-learning for domain generalization. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018 a

2018
[111]

Rethinking the hyperparameters for fine-tuning

Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Rethinking the hyperparameters for fine-tuning. arXiv preprint arXiv:2002.11770, 2020

2002 arXiv
[112]

Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C. Kot. Domain generalization with adversarial feature learning. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5400--5409, 2018 b . doi:10.1109/CVPR.2018.00566

2018
[113]

Long-context llms struggle with long in-context learning

Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-context llms struggle with long in-context learning. arXiv preprint arXiv:2404.02060, 2024

2024 arXiv
[114]

Understanding the disharmony between dropout and batch normalization by variance shift

Xiang Li, Shuo Chen, Xiaolin Hu, and Jian Yang. Understanding the disharmony between dropout and batch normalization by variance shift. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2682--2690, 2019

2019
[115]

Domain generalization via conditional invariant representations

Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao. Domain generalization via conditional invariant representations. In AAAI Conference on Artificial Intelligence, 2018 c . URL https://api.semanticscholar.org/CorpusID:19158057

2018
[116]

Selective in-context data augmentation for intent detection using pointwise v-information

Yen-Ting Lin, Alexandros Papangelis, Seokhwan Kim, Sungjin Lee, Devamanyu Hazarika, Mahdi Namazifar, Di Jin, Yang Liu, and Dilek Hakkani-Tur. Selective in-context data augmentation for intent detection using pointwise v-information. arXiv preprint arXiv:2302.05096, 2023

2023 arXiv
[117]

Just train twice: Improving group robustness without training group information

Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pages 6781--6792. PMLR, 2021

2021
[118]

Decoupled weight decay regularization

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

2017 arXiv
[119]

Moba: Mixture of block attention for long-context llms

Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, et al. Moba: Mixture of block attention for long-context llms. arXiv preprint arXiv:2502.13189, 2025

2025 arXiv
[120]

Multi-digit recognition using a space displacement neural network

Ofer Matan, Christopher J Burges, Yann LeCun, and John Denker. Multi-digit recognition using a space displacement neural network. Advances in neural information processing systems, 4, 1991

1991
[121]

GSM - Symbolic : Understanding the Limitations of Mathematical Reasoning in Large Language Models , October 2024

Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. GSM - Symbolic : Understanding the Limitations of Mathematical Reasoning in Large Language Models , October 2024. URL http://arxiv.org/abs/2410.05229. arXiv:2410.05229

2024 arXiv
[122]

Trivialaugment: Tuning-free yet state-of-the-art data augmentation

Samuel G M \"u ller and Frank Hutter. Trivialaugment: Tuning-free yet state-of-the-art data augmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 774--782, 2021

2021
[123]

Nadaraya

E. Nadaraya. On estimating regression. Theory of Probability and Its Applications, 9: 0 141--142, 1964. URL https://api.semanticscholar.org/CorpusID:120067924

1964
[124]

Learning from failure: Training debiased classifier from biased classifier

Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: Training debiased classifier from biased classifier. arXiv, 2020

2020
[125]

Feature selection, l 1 vs

Andrew Y Ng. Feature selection, l 1 vs. l 2 regularization, and rotational invariance. In Proceedings of the twenty-first international conference on Machine learning, page 78, 2004

2004
[126]

On first-order meta-learning algorithms

Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv, 2018

2018
[127]

In-context learning and induction heads, 2022

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kam...

2022
[128]

Learning and transferring mid-level image representations using convolutional neural networks

Maxime Oquab, L\'eon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid-level image representations using convolutional neural networks. In Proceedings of Computer Vision and Pattern Recognition (CVPR). IEEE, 2014

2014
[129]

Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117 0 (40): 0 24652--24663, 2020. doi:10.1073/pnas.2015509117. URL https://www.pnas.org/doi/abs/10...

2020 doi
[130]

Pedregosa, G

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in P ython. Journal of Machine Learnin...

2011
[132]

Rwkv: Reinventing rnns for the transformer era

Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048, 2023 b

2023 arXiv
[133]

Yarn: Efficient context window extension of large language models

Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023 c

2023 arXiv
[134]

Gradient starvation: A learning proclivity in neural networks

Mohammad Pezeshki, S \' e kou Oumar Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks. arXiv, 2020

2020
[135]

Gradient starvation: A learning proclivity in neural networks

Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks. Advances in Neural Information Processing Systems, 34: 0 1256--1272, 2021

2021
[136]

Train short, test long: Attention with linear biases enables input length extrapolation

Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021

2021 arXiv
[137]

Train short, test long: Attention with linear biases enables input length extrapolation

Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022

2022
[138]

Improving language understanding by generative pre-training, 2018

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018

2018
[139]

Language models are unsupervised multitask learners, 2019

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019

2019
[140]

Rapid learning or feature reuse? towards understanding the effectiveness of maml

Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint arXiv:1909.09157, 2019

1909 arXiv
[141]

Frameworks and results in distributionally robust optimization

Hamed Rahimian and Sanjay Mehrotra. Frameworks and results in distributionally robust optimization. Open Journal of Mathematical Optimization, 3: 0 1--85, 2022

2022
[142]

Squad: 100,000+ questions for machine comprehension of text

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016

2016 arXiv
[143]

Fishr: Invariant gradient variances for out-of-distribution generalization

Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of-distribution generalization. In International Conference on Machine Learning, pages 18347--18377. PMLR, 2022

2022
[144]

Diverse weight averaging for out-of-distribution generalization

Alexandre Ram \'e , Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord. Diverse weight averaging for out-of-distribution generalization. Advances in Neural Information Processing Systems, 35: 0 10821--10836, 2022

2022
[145]

Model ratatouille: Recycling diverse models for out-of-distribution generalization

Alexandre Ram \'e , Kartik Ahuja, Jianyu Zhang, Matthieu Cord, L \'e on Bottou, and David Lopez-Paz. Model ratatouille: Recycling diverse models for out-of-distribution generalization. In International Conference on Machine Learning, pages 28656--28679. PMLR, 2023

2023
[146]

Hopfield networks is all you need

Hubert Ramsauer, Bernhard Sch \"a fl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlovi \'c , Geir Kjetil Sandve, et al. Hopfield networks is all you need. arXiv preprint arXiv:2008.02217, 2020

2008 arXiv
[147]

Rosenblatt

F. Rosenblatt. The perceptron: A perceiving and recognizing automaton. Technical Report 85-460-1, Project PARA, Cornell Aeronautical Lab, 1957

1957
[148]

Domain-adjusted regression or: Erm may already learn features sufficient for out-of-distribution generalization

Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. Domain-adjusted regression or: Erm may already learn features sufficient for out-of-distribution generalization. arXiv preprint arXiv:2202.06856, 2022

2022 arXiv
[149]

Disentanglement of correlated factors via hausdorff factorized support

Karsten Roth, Mark Ibrahim, Zeynep Akata, Pascal Vincent, and Diane Bouchacourt. Disentanglement of correlated factors via hausdorff factorized support. In The Eleventh International Conference on Learning Representations, 2022 a

2022
[150]

Disentanglement of correlated factors via hausdorff factorized support

Karsten Roth, Mark Ibrahim, Zeynep Akata, Pascal Vincent, and Diane Bouchacourt. Disentanglement of correlated factors via hausdorff factorized support. arXiv preprint arXiv:2210.07347, 2022 b

2022 arXiv
[151]

D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning internal representations by error propagation. In Parallel distributed processing: Explorations in the microstructure of cognition, volume I, pages 318--362. Bradford Books, Cambridge, MA, 1986

1986
[152]

Meta-learning with latent embedding optimization

Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05960, 2018

2018 arXiv
[153]

Dynamic routing between capsules

Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. Advances in neural information processing systems, 30, 2017

2017
[154]

Hashimoto, and Percy Liang

Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. ArXiv, abs/1911.08731, 2019 a . URL https://api.semanticscholar.org/CorpusID:208176471

1911 arXiv
[155]

Hashimoto, and Percy Liang

Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. ICLR, 2019 b

2019
[156]

Some studies in machine learning using the game of checkers

Arthur L Samuel. Some studies in machine learning using the game of checkers. IBM Journal of research and development, 3 0 (3): 0 210--229, 1959

1959
[157]

GLU variants improve transformer

Noam Shazeer. GLU variants improve transformer. arXiv preprint arXiv:2002.05202, 2020

2002 arXiv
[158]

Gradient matching for domain generalization

Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv preprint arXiv:2104.09937, 2021

2021 arXiv
[159]

SHRDLU, https://en.wikipedia.org/wiki/SHRDLU

Shrdlu 1972. SHRDLU, https://en.wikipedia.org/wiki/SHRDLU

1972
[160]

Opening the black box of deep neural networks via information

Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810, 2017

2017 arXiv
[161]

Mastering the game of go with deep neural networks and tree search

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529 0 (7587): 0 484-...

2016
[162]

Minimax theorems and their proofs

Stephen Simons. Minimax theorems and their proofs. In Ding-Zhu Du and Panos M. Pardalos, editors, Minimax and Applications, pages 1--23, Boston, MA, 1995. Springer US

1995
[163]

Prototypical networks for few-shot learning

Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, ...

2017
[164]

A new unbiased and efficient class of lsh-based samplers and estimators for partition function computation in log-linear models, 2017

Ryan Spring and Anshumali Shrivastava. A new unbiased and efficient class of lsh-based samplers and estimators for partition function computation in log-linear models, 2017

2017
[165]

Dropout: a simple way to prevent neural networks from overfitting

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15 0 (1): 0 1929--1958, 2014

1929
[166]

How to train your vit? data, augmentation, and regularization in vision transformers

Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106.10270, 2021

2021 arXiv
[167]

Roformer: Enhanced transformer with rotary position embedding

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

2024
[168]

End-to-end memory networks

Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015

2015
[169]

Augmenting self-attention with persistent memory, 2019

Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, and Armand Joulin. Augmenting self-attention with persistent memory, 2019

2019
[170]

Deep coral: Correlation alignment for deep domain adaptation

Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision--ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14, pages 443--450. Springer, 2016

2016
[171]

Learning to compare: Relation network for few-shot learning

Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1199--1208, 2018

2018
[172]

Deepface: Closing the gap to human-level performance in face verification

Yaniv Taigman, Ming Yang, Marc'Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1701--1708, 2014

2014
[173]

Temporal difference learning and td-gammon

Gerald Tesauro et al. Temporal difference learning and td-gammon. Communications of the ACM, 38 0 (3): 0 58--68, 1995

1995
[175]

Disentangling the independently controllable factors of variation by interacting with the world

Valentin Thomas, Emmanuel Bengio, William Fedus, Jules Pondard, Philippe Beaudoin, Hugo Larochelle, Joelle Pineau, Doina Precup, and Yoshua Bengio. Disentangling the independently controllable factors of variation by interacting with the world. arXiv preprint arXiv:1802.09484, 2018 b

2018 arXiv
[176]

Meta-dataset: A dataset of datasets for learning to learn from few examples

Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, et al. Meta-dataset: A dataset of datasets for learning to learn from few examples. arXiv preprint arXiv:1903.03096, 2019

1903 arXiv
[177]

Alan M. Turing. Computing machinery and intelligence. Mind, 49: 0 433--460, 1950

1950
[178]

Ueda and R

N. Ueda and R. Nakano. Generalization error of ensemble estimators. In Proceedings of International Conference on Neural Networks (ICNN'96), volume 1, pages 90--95 vol.1, 1996. doi:10.1109/ICNN.1996.548872

1996
[179]

The inaturalist species classification and detection dataset

Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8...

2018
[180]

V. Vapnik. Principles of risk minimization for learning theory. In J. Moody, S. Hanson, and R.P. Lippmann, editors, Advances in Neural Information Processing Systems, volume 4. Morgan-Kaufmann, 1991 a . URL https://proceedings.neurips.cc/paper_files/paper/1991/file/ff4d5fbbafd...

1991
[181]

Principles of risk minimization for learning theory

Vladimir Vapnik. Principles of risk minimization for learning theory. Advances in neural information processing systems, 4, 1991 b

1991
[182]

Local algorithms for pattern recognition and dependencies estimation

Vladimir Vapnik and L \'e on Bottou. Local algorithms for pattern recognition and dependencies estimation. Neural Computation, 5 0 (6): 0 893--909, 1993

1993
[183]

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural In...

2017
[184]

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017 b

2017
[185]

Residual networks behave like ensembles of relatively shallow networks

Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. Advances in neural information processing systems, 29, 2016

2016
[186]

Deep hashing network for unsupervised domain adaptation

Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5018--5027, 2017

2017
[187]

Matching Networks for One Shot Learning

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray Kavukcuoglu, and Daan Wierstra. Matching Networks for One Shot Learning . In D Lee, M Sugiyama, U Luxburg, I Guyon, and R Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates...

2016
[188]

Legendre memory units: Continuous-time representation in recurrent neural networks

Aaron Voelker, Ivana Kaji \'c , and Chris Eliasmith. Legendre memory units: Continuous-time representation in recurrent neural networks. Advances in neural information processing systems, 32, 2019

2019
[189]

The Caltech-UCSD Birds -200-2011 dataset

Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech-UCSD Birds -200-2011 dataset. Technical report, California Institute of Technology, 2011

2011
[190]

On calibration and out-of-domain generalization

Yoav Wald, Amir Feder, Daniel Greenfeld, and Uri Shalit. On calibration and out-of-domain generalization. arXiv preprint arXiv:2102.10395, 2021

2021 arXiv
[191]

K-adapter: Infusing knowledge into pre-trained models with adapters

Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Guihong Cao, Daxin Jiang, Ming Zhou, et al. K-adapter: Infusing knowledge into pre-trained models with adapters. arXiv preprint arXiv:2002.01808, 2020 a

2002 arXiv
[192]

Structured prediction for conditional meta-learning

Ruohan Wang, Yiannis Demiris, and Carlo Ciliberto. Structured prediction for conditional meta-learning. Advances in Neural Information Processing Systems, 33: 0 2587--2598, 2020 b

2020
[193]

Geoffrey S. Watson. Smooth regression analysis. Sankhy \=a : The Indian Journal of Statistics, Series A , pages 359--372, 1964

1964
[194]

Memory networks

Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. arXiv preprint arXiv:1410.3916, 2014

2014 arXiv
[195]

Procedures as a representation for data in a computer program for understanding natural language

Terry Winograd. Procedures as a representation for data in a computer program for understanding natural language. 1971

1971
[196]

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing infe...

2022
[197]

Robust fine-tuning of zero-shot models

Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision...

2022
[198]

Efficient streaming language models with attention sinks, 2023

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2023

2023
[199]

Hotpotqa: A dataset for diverse, explainable multi-hop question answering

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600, 2018

2018 arXiv
[200]

Native sparse attention: Hardware-aligned and natively trainable sparse attention

Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502.11089, 2025

2025 arXiv
[201]

Cutmix: Regularization strategy to train strong classifiers with localizable features

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023--6032, 2019

2019
[202]

Learning useful representations for shifting tasks and distributions

Jianyu Zhang and L \'e on Bottou. Learning useful representations for shifting tasks and distributions. In International Conference on Machine Learning, pages 40830--40850. PMLR, 2023

2023
[203]

Fine-tuning with very large dropout

Jianyu Zhang and L \'e on Bottou. Fine-tuning with very large dropout. arXiv preprint arXiv:2403.00946, 2024

2024 arXiv
[204]

Towards automatic complex feature engineering

Jianyu Zhang, Fran c oise Fogelman-Souli \'e , and Christine Largeron. Towards automatic complex feature engineering. In Hakim Hacid, Wojciech Cellary, Hua Wang, Hye-Young Paik, and Rui Zhou, editors, Web Information Systems Engineering -- WISE 2018, pages 312--322, Cham, 2018...

2018
[205]

Rich feature construction for the optimization-generalization dilemma

Jianyu Zhang, David Lopez-Paz, and L \'e on Bottou. Rich feature construction for the optimization-generalization dilemma. In International Conference on Machine Learning, pages 26397--26411. PMLR, 2022

2022
[206]

Memory mosaics

Jianyu Zhang, Niklas Nolte, Ranajoy Sadhukhan, Beidi Chen, and L\'eon Bottou. Memory mosaics. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=IiagjrJNwF

2025
[207]

Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017), pages 35--45, 2017. URL h...

2017
[208]

Random erasing data augmentation

Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 34(7), pages 13001--13008, 2020

2020

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Systematic generalisation with group invariant predictions

Faruk Ahmed, Yoshua Bengio, Harm van Seijen, and Aaron Courville. Systematic generalisation with group invariant predictions. In International Conference on Learning Representations, 2020

2020

[2] [2]

Empirical or invariant risk minimization? a sample complexity perspective

Kartik Ahuja, Jun Wang, Amit Dhurandhar, Karthikeyan Shanmugam, and Kush R Varshney. Empirical or invariant risk minimization? a sample complexity perspective. arXiv preprint arXiv:2010.16412, 2020

arXiv 2010

[3] [3]

In-context language learning: Arhitectures and algorithms

Ekin Aky \"u rek, Bailin Wang, Yoon Kim, and Jacob Andreas. In-context language learning: Arhitectures and algorithms. arXiv preprint arXiv:2401.12973, 2024

arXiv 2024

[4] [4]

Sgd with large step sizes learns sparse features

Maksym Andriushchenko, Aditya Varre, Loucas Pillaud-Vivien, and Nicolas Flammarion. Sgd with large step sizes learns sparse features. arXiv preprint arXiv:2210.05337, 2022

arXiv 2022

[5] [5]

Sgd with large step sizes learns sparse features

Maksym Andriushchenko, Aditya Vardhan Varre, Loucas Pillaud-Vivien, and Nicolas Flammarion. Sgd with large step sizes learns sparse features. In International Conference on Machine Learning, pages 903--925. PMLR, 2023

2023

[6] [6]

Invariant risk minimization

Martin Arjovsky, L \'e on Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019

arXiv 1907

[7] [7]

Invariant risk minimization

Martin Arjovsky, L\'eon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv, 2020

2020

[8] [8]

Ensemble of averages: Improving model selection and boosting performance in domain generalization

Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong. Ensemble of averages: Improving model selection and boosting performance in domain generalization. Advances in Neural Information Processing Systems, 35: 0 8265--8277, 2022

2022