Contextures: The Mechanism of Representation Learning

Runtian Zhai

REVIEW 4 major objections 5 minor 171 references

Contextures: The Mechanism of Representation Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Representation learning has one mathematical target: the span of the top singular functions of the conditional expectation operator, and learning it is optimal for every compatible downstream task.

desk verdict A genuinely unifying spectral framework that mostly delivers on its mathematical claims, but the explanation of transfer rests on an unmeasured premise about which tasks are compatible with the context. read the letter →

arxiv 2504.19792 v1 pith:BPPUKMRT submitted 2025-04-28 cs.LG cs.AIstat.ML

Runtian Zhai This is my paper · ORCID

classification cs.LGcs.AIstat.ML MSC 68T0768T0562H25

keywords representationlearningpretrainingconditionalexpectationoperatorpositive-pairkernelself-supervisedspectralmethodsscalinglawstransfer

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Representation learning, the paper argues, has one mathematical target regardless of the objective: the association between the input $X$ and a context variable $A$. That target, the contexture, is the span of the top singular functions of the conditional expectation operator $(T_{P^+}g)(x) = E[g(A)\mid X=x]$. The paper proves that an encoder capturing this span is optimal for every downstream task that is compatible with the context, minimizing worst-case approximation error over the class of $(1-\epsilon)$-compatible tasks, and that a wide family of existing objectives are solved exactly by learning the contexture. If the theory is right, it explains why pretrained representations transfer, why different architectures converge to similar representations, and why scaling the model alone gives diminishing returns once the contexture is expressible. The route to further progress is therefore better contexts, not just larger models.

What carries the argument

The central object is the conditional expectation operator $T_{P^+}: L^2(P_A)\to L^2(P_X)$ defined by $(T_{P^+}g)(x) = E[g(A)\mid X=x]$, together with the positive-pair kernel $k_X^+(x,x') = \int \frac{P^+(a|x)P^+(a|x')}{P_A(a)}\,da$. The integral operator of this kernel is self-adjoint and Hilbert–Schmidt, and its top-$d$ eigenfunctions are the contexture. The argument runs through the spectral decomposition of this operator: the singular value spectrum encodes the strength of the $X$–$A$ association, task compatibility is defined by projection onto the singular functions, and each variational objective is shown to be a spectral objective whose minimizers extract the top eigenspace. The spectrum also drives the paper's quantitative context-usefulness metric and its generalization bounds through the context complexity $\kappa$.

What would settle it

Fix a context $P^+$ and a downstream task $f^*$ with measured compatibility $\rho(f^*,P^+) \ge 1-\epsilon$. Compute the exact top-$d$ left singular functions of $T_{P^+}$ by kernel PCA on a finite approximation and train an independent encoder $\Phi$ that does not span them. If, with a linear probe and enough labeled data to make estimation error negligible, the non-contexture encoder achieves lower error on $f^*$ than the contexture encoder, Theorem 3.4 is contradicted. Alternatively, on any finite space where $T_{P^+}$ is a matrix, one can enumerate all $d$-dimensional subspaces: if any subspace outside the top-$d$ left singular span is closer to a compatible $f^*$ than the contexture span, the claimed uniqueness of the optimum fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that the mechanism of representation learning is spectral: an encoder $\Phi = [\varphi_1,\dots,\varphi_d]$ learns the contexture of $P^+$ exactly when $\operatorname{span}\{\varphi_1,\dots,\varphi_d\} = \operatorname{span}\{\mu_1,\dots,\mu_d\}$, where $\mu_i$ are the top-$d$ left singular functions of $T_{P^+}$ (equivalently, the top eigenfunctions of the positive-pair kernel integral operator). Theorems 2.2–2.14 show that supervised MSE, balanced classification, graph embedding, contrastive and non-contrastive SSL, reconstruction, generative multi-step models, knowledge distillation, and RLHF are optimized if and only if the encoder learns the contexture (possibly with a loss kernel that reweights classes). Theorem 3.4 is the optimality result: among all $d$-dimensional encoders, the contexture uniquely minimizes the worst-case approximation error on the class of tasks with compatibility at least $1-\epsilon$, with error $\frac{s_1^2-(1-\epsilon)^2}{s_1^2-s_{d+1}^2}$, and every non-contexture encoder has a compatible task on which it does no better than this worst case. Section 2.6 draws the scaling-law consequence: once the model is large enough to express the contexture, further scaling is redundant and only a better context helps.

Load-bearing premise

The theory assumes that the tasks that matter can be solved by a linear probe on functions lying in the range of the conditional expectation operator—tasks whose compatibility is defined through the very singular functions that make up the contexture. If real downstream tasks are solved by non-linear fine-tuning or fall outside this function class, the optimality and transfer claims do not apply; the paper itself gives the example of letter counting as a useful-sounding task that is not compatible.

Editorial extensions

If this is right

For any task known to be compatible with the pretraining context, the best possible $d$-dimensional representation is the contexture; no other encoder can beat its worst-case approximation error on that task class.
Because the contexture is independent of architecture, different sufficiently large models trained on the same context should converge to aligned representations, and objectives as different as masked prediction and contrastive learning are interchangeable if they share the same context.
Scaling up model size only helps until the contexture is expressible; after that, performance gains require changing the context or mixing existing ones.
Mixing contexts by convolution, convex combination, or concatenation can convert too-strong or too-weak associations into moderate ones, improving downstream performance; experiments on tabular benchmarks support this.
A context's usefulness can be scored from its singular value spectrum alone, so practitioners can choose pretraining hyperparameters such as mask ratios and augmentation strengths without running full downstream evaluations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: the same spectral argument predicts that representational convergence should occur for any sufficiently expressive function class, not only neural networks, so the convergence phenomenon is a property of the context's operator, not of deep learning itself.
The optimality theorem is proven for linear probes and approximation error; if downstream tasks are solved by non-linear fine-tuning, the contexture may be sufficient but not necessary, and a non-contexture encoder could win on specific tasks. Testing this boundary would require extending the compatibility definition beyond the range of $T_{P^+}$.
A testable engineering consequence the paper leaves implicit: new pretraining methods should be evaluated first by estimating the spectrum of their induced context and only second by end-task accuracy, since the spectrum determines both transfer and sample complexity.
The theory suggests an information-theoretic account of alignment and distillation: RLHF and knowledge distillation are context-mixing operations, so safety-relevant training modifies the target contexture rather than creating a wholly new representation space.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. First, this is a substantial dissertation: it reduces a wide family of representation-learning objectives—supervised, contrastive and non-contrastive SSL, generative models, RLHF—to one spectral object, the top-d left singular functions of the conditional expectation operator. Second, the load-bearing step of the story is still a promise. Theorem 3.4 is a valid minimax statement on the class F_epsilon(P+), but that class is defined through the range of the same operator whose top singular functions are the proposed answer. So the theorem is partly certifying its own definitions, and the paper never measures whether real tasks like sentiment analysis or QA actually fall in that class. This is not a fatal flaw, but it is the main reason the central transfer claim stays conditional.

What deserves credit: the unification itself is valuable, and several pieces are genuinely new—SVME and KISE as general objectives, the context-mixing operations with their near-ring structure, and the statistical bounds built on context complexity. I checked the minimax formula in Theorem 3.4 by direct computation and it holds. The author also credits the spectral machinery that predates the thesis, much of it his own prior work, and is candid about the open problems: high-dimensional kappa making the bounds useless, missing pair-access bounds, the difficulty of enforcing orthonormality, and the gap between global optima and what gradient methods find.

The soft spots are real but proportional. The circularity concern is not a takedown; the optimality result is sharp about a natural class, and the issue is scope. The paper's own letter-counting counterexample concedes that useful-sounding tasks can fall outside the compatible class, which is honest but also highlights the gap. All guarantees are for linear probes, while SFT and RLHF are motivating applications and sit outside the theorem. The empirical support is thinner than the prose: the flagship scaling experiment is one tabular dataset with 4,177 samples, the alignment peaks near 0.9 and then declines, and the decline is attributed to optimization rather than the theory. The context metric is acknowledged as heuristic and has negative correlations on several datasets. The OneDrive code links lack hashes and two appear truncated in the v1 text; that is fixable but not acceptable as release artifacts.

Who should read it: anyone working on the theory of SSL or pretraining. It gives a coherent framework worth arguing with, and the new objectives and mixing operations are useful regardless of whether the grand explanatory framing survives. It deserves a serious referee. My recommendation: engage with it conditionally—ask for either measured compatibility values for a few real tasks or a clear restatement of scope, proper code artifacts, and a tighter separation between the theorem and the transfer narrative.

Referee Report

4 major / 5 minor

Summary. The manuscript (a CMU PhD thesis posted on arXiv) develops the 'contexture theory' of representation learning. A context is a joint distribution P+(X,A); its contexture is the top-d eigenspace of the conditional-expectation operator T_{P+}. The thesis claims: (i) a broad family of objectives—supervised MSE, graph embedding, contrastive/non-contrastive SSL, generative modeling, RLHF—are optimized if and only if the encoder learns the contexture; (ii) learning the contexture minimizes worst-case linear-probe approximation error over the class of (1-epsilon)-compatible tasks, with an explicit error formula; (iii) better contexts, not larger models, are the path to further progress; and (iv) finite-sample bounds follow from a context-complexity assumption. The paper also introduces SVME/KISE objectives, three context-mixing operations, a spectrum-based context-usefulness metric, and a tabular-data evaluation.

Significance. The spectral formalism is clean and genuinely unifying; the minimax formula in Theorem 3.4 is a coherent spectral statement; the paper gives proof sketches for many implications and carefully distinguishes exact eigenfunction extraction from eigenspace extraction. The empirical correlation of the tau metric on 28 OpenML datasets and the tabular-data gains over XGBoost are useful contributions in their own right. The main weakness is that the bridge from the formal class F_epsilon(P+) to tasks practitioners care about is asserted, not measured; all optimality statements are for linear probes; and Theorem 3.2 as stated is internally inconsistent with Definition 3.1. If the compatibility premise can be justified, the framework is a substantial contribution; as it stands, the headline transfer and scaling claims outrun the evidence.

major comments (4)

[Sec. 3.1, Theorem 3.2 and Eq. (3.2)] Theorem 3.2 is false as stated. Since rho(f,P+) = ||T*_{P+} tilde f|| / ||tilde f||, membership in F_epsilon(P+) only forces the majority of L2 mass of tilde f to lie on large singular components; it does not force tilde f to lie in the range of T_{P+}. For a finite-rank context such as deterministic labels (Example 1.12), take f* = a mu_1 + b mu_C with a^2+b^2=1 and a >= 1-epsilon, where mu_C is a zero-singular-value eigenfunction. Then f* is in F_epsilon(P+) but no g* in L2(P_A) satisfies f* = E[g*(A)|X]. The same issue arises for infinite-rank operators because range membership requires sum u_i^2/s_i^2 < infinity, which is strictly stronger than the spectral-energy condition in Eq. (3.2). Consequently, the two 'necessary conditions' of Section 3.1 are not equivalent to Definition 3.1, and the interpretation of F_epsilon as the class of tasks for which the context is helpful is not supported as written. The minimax computation in Theorem 3.4 may still be valid over the stated F_epsilon, but the bridge between compatibility and the range of T_{P+} needs to be repaired (for example, by defining F_epsilon via projection onto the range).
[Sec. 3.1, Definition 3.1/Eq. (3.2) and Fig. 3.1] The central transfer explanation rests on the identification of F_epsilon(P+) with the tasks for which pretrained representations are useful. This identification is not tested. The paper never estimates rho(f,P+) for a real downstream task; it gives intuitive examples and a counterexample (letter counting, Figure 3.1) showing that a useful-sounding task can lie outside F_epsilon. Because F_epsilon is defined by correlation with the same operator T_{P+} whose top singular functions are the contexture, Theorem 3.4 is a spectral minimax statement about a class of functions chosen by the theory, not an explanation of transfer to sentiment analysis, QA, or translation unless those tasks are shown to be in F_epsilon. Please provide direct measurements of rho for representative tasks with standard contexts, or a formal argument (not just examples) that the relevant task families are contained in F_epsilon for practical epsilon. Without this, the paper's headline explanation of why pretraining transfers is an untested empirical premise.
[Sec. 1.2 and Sec. 2.4] All formal guarantees in the paper are for linear probes, as explicitly stated in Section 1.2, yet the motivation and several applications involve nonlinear adaptation: supervised fine-tuning (SFT) and RLHF are discussed in Section 2.4, and the scaling-law discussion in Section 2.6 concerns foundation models used by fine-tuning. Theorem 3.4's worst-case approximation error is defined for a linear predictor W Phi(x) + b on the span of Phi; it does not cover nonlinear fine-tuning. If downstream tasks are solved by nonlinear adaptation, the theoretical optimality of the contexture has not been established for that setting. The manuscript should either restrict its claims to linear probing or extend the analysis to a nonlinear-probe setting.
[Sec. 2.6, Figure 2.4] The empirical basis for the diminishing-returns claim is a single tabular dataset (abalone, 4,177 samples). The alignment metrics peak near 0.9 CCA and are non-monotone in both width and depth; the authors attribute the non-monotonicity to optimization difficulty. That attribution means the experiment does not isolate the proposed mechanism (saturation of expressivity relative to the contexture) from optimization effects. The conclusion that 'increasing model size alone will achieve diminishing returns' is substantially stronger than what this experiment can support. A more direct test would vary model size while controlling optimization effort and measuring alignment to the top-d eigenspace.

minor comments (5)

[Theorem 3.10 proof] The expression Tr(C^{-1}_F B^{-1}_F) appears to be a typo for Tr(C^{-1}_F B_F); the notation in that paragraph should be corrected.
[Eq. (3.5)] The metric tau_d depends on free parameters beta and d_0; the experiments fix beta=1, d_0=512, but no sensitivity analysis is reported despite the paper acknowledging failure cases.
[Sec. 2.6] The description of depth as the number of hidden layers, with 'a 2-layer neural network has depth 1', is confusing; the figure axis labeled 'Depth' should match the definition given in the text.
[Code availability] Several code URLs are truncated (for example, in Section 2.6 and Section 3.3); please provide complete and stable links.
[Sec. 5.1] The 99th-percentile threshold for estimating kappa^2 is a heuristic; the paper should state how sensitive the downstream conclusions are to this threshold.

Circularity Check

1 steps flagged · score 6.0 of 10

Optimality over F_epsilon(P+) reduces to the definition of compatibility: the task class is the spectral subspace of the same operator whose top eigenspace is called the contexture.

self definitional [Definition 3.1 / Eq. (3.2), Definition 1.7, Theorem 3.4]
"ρ(f,P+) = max_{g∈L2(PA),g≠0} ⟨f˜, T_{P+}g⟩_{PX} / (||f˜||_{PX} ||g||_{PA}) ... Fǫ(P+) = { f∈L2(PX) : ρ(f,P+) ≥ 1−ǫ } ... span{φ1,...,φd} = span{µ1,...,µd} ... Φ minimizes err(Φ;Fǫ(P+)) if and only if it learns the contexture of T_{P+}."

The class of tasks on which the contexture is declared optimal is defined by ρ, which equals ||T*_{P+}f˜||/||f˜||, i.e. the fraction of f's centered L2 energy lying in the spectral subspace of the same positive-pair operator whose top-d eigenfunctions are named the contexture in Definition 1.7. Theorem 3.4 is therefore a spectral projection statement: the top-d singular subspace is the best d-dimensional subspace for approximating functions that mostly live in that same top spectral subspace.

full rationale

The paper is internally rigorous: Theorems 2.2-2.14, 4.4, and 5.11-5.12 are proved from the stated definitions, and no load-bearing result is imported solely from the author's prior work; [167,168] are cited for definitions and bounds that the thesis itself re-proves or extends. The alignment experiments (Section 2.6) and tabular experiments (Section 4.4) are genuine empirical checks. The central non-empirical reduction is the bridge from the spectral minimax theorem to the claim that contexture explains representation transfer. Compatibility (Definition 3.1) is defined through the same conditional-expectation operator whose top singular functions constitute the contexture, so Theorem 3.4's optimality over F_epsilon(P+) is true by construction of the task class. The paper acknowledges the resulting limitation in places: it restricts evaluation to linear probes (Section 1.2), states that bounds do not cover all of F_epsilon (Section 5.2), leaves pair-access generalization bounds open, and calls the Section 3.3 metric heuristic. These concessions do not convert the spectral theorem into an empirical prediction; they underscore that the theory's applicability to real tasks rests on an unmeasured compatibility premise. On balance, this is partial definitional circularity in the central transfer claim, not a fully circular derivation: the variational equivalences and generalization bounds retain independent mathematical content, so the score is 6 rather than higher.

Assumptions & free parameters 5 free parameters · 6 assumptions · 1 invented entities

The ledger shows the theory's real input costs. The spectral framework is assumed (Hilbert-Schmidt kernels, compact Hausdorff spaces), and the variational theorems assume exact global optima, which the paper's own experiments show are not reached by gradient methods. The generalization bounds import standard RKHS conditions plus a context complexity kappa that is exponential in dimension for natural masking, which the author states makes the bounds 'not really useful' in high dimensions. The practical evaluations carry hand-set quantities (beta=1, d0=512, STK threshold c=0.1, 99th percentile) and data-dependent mixing weights. No genuinely new physical entities are postulated; the contexture is a repackaging of spectral objects from cited work, with the new work being the theorems and objectives built on top. The most consequential input is definitional: the compatibility class used to certify optimality is generated by the same operator whose eigenspace is the proposed answer.

free parameters (5)

beta (context metric weight) = 1
In Eqn (3.5), tau_d = 1/(1-s^2_{d+1}) + beta * (sum_{i<=d} s^2_i)/(sum_{i<=d0} s^2_i); the paper states 'We choose beta = 1 and d0 = 512'. The metric's tradeoff, and the 'moderate association is best' conclusions shown in Figures 3.2-3.3, depend on this choice.
d0 (max embedding dimension in metric) = 512
Normalizes the second term of tau_d and caps the spectrum sum; the author notes d0 typically ranges from 512 to 8192.
STK threshold c = 0.1
Section 4.4 tabular experiments: the Y-linear kernel is spectrally transformed by setting eigenvalues above c=0.1 to 1 and the rest to 0; the downstream results depend on this hand-set threshold.
99th percentile for context complexity = 99th percentile
Section 5.1 estimates kappa^2 by the 99th percentile of k+_X(x,x) rather than the supremum, because the supremum is statistically impossible to estimate; the generalization bounds in Chapter 5 then hold for a percentile-defined kappa, not the defined sup-kappa.
convex combination weights w in Eqn (4.1) = learned by Hedge (Algorithm 2)
In the tabular experiments the convex-combination weights are the minimax weights computed from the training losses; they are data-dependent and chosen to maximize the weakness of the mixture.

assumptions (6)

domain assumption T_{k+_A} and T_{k+_X} are Hilbert-Schmidt operators, so the spectral theorem gives an ordered eigenbasis with lambda_0 = 1.
Section 1.3: 'Throughout this work, we assume that T_{k+_A} and T_{k+_X} are both Hilbert-Schmidt operators.' All subsequent SVD results (Lemma 1.4, Lemma 1.5, Definition 1.7) rest on this; it excludes the A=X identity context for infinite spaces.
domain assumption The probability spaces of P_X and P_A are compact Hausdorff, and P_X is fixed until Chapter 6.
Section 1.2. Provides the L2(P_X) Hilbert space setup and makes the operator formalism well-posed.
domain assumption The encoder class is expressive enough and the optimizer reaches the global minimizer of each variational objective.
Theorems 2.2-2.14 and 4.4 are 'if and only if' statements about minimizers of population objectives. The paper itself documents that gradient training does not reach these optima (dimension collapse [80], edge of stability [28], non-monotonic alignment in Figure 2.4), so the theorems characterize ideal rather than trained encoders.
domain assumption The positive-pair kernel k+_X (or its centered version) is computable or estimable for the generalization bounds.
Section 5.2: 'assume that the kernel k that we have access to is the centered kernel of k+_X... for simplicity we ignore this difference'. For pair access the paper notes estimating P_A(a)^{-1} precisely is hard (opening of Chapter 2), and deriving bounds for pair access is posed as an open problem.
standard math Moment, eigenvalue-decay, embedding, and source conditions hold for the ridge regression bound.
Eqn (5.3) (moment condition), EVD, EMB, and SRC conditions in Section 5.2 are invoked to apply [41, Theorem 3.1].
domain assumption For T-access contexts, a heuristic inverse Q+ exists and is available; for pair access, P_A is bounded away from zero.
Definition 1.11 introduces Q+; Section 5.2 states that pair and transformation access bounds need P_A(a) bounded away from zero (from [167]).

invented entities (1)

contexture (the top-d eigenspace of the conditional expectation operator) independent evidence
purpose: The hypothesized object that all representation learning recovers; the proposed explanation for transfer and for scaling limits.
This is a mathematical re-formulation rather than a new physical entity, so the 'graviton problem' does not apply. The falsifiable handle is the predicted alignment between trained networks and kernel-PCA eigenfunctions (measured by CCA in Section 2.6, reaching about 0.9 on abalone) and the predicted saturation of scaling returns; independent groups can test these predictions.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Contextures: The Mechanism of Representation Learning." pith.science (2026). https://pith.science/paper/BPPUKMRT

@misc{pith2026250419792,
  author       = {Pith},
  title        = {Pith review of: Contextures: The Mechanism of Representation Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BPPUKMRT}},
  note         = {Machine review of arXiv:2504.19792}
}

read the original abstract

This dissertation establishes the contexture theory to mathematically characterize the mechanism of representation learning, or pretraining. Despite the remarkable empirical success of foundation models, it is not very clear what representations they learn, and why these representations are useful for various downstream tasks. A scientific understanding of representation learning is critical, especially at this point when scaling up the model size is producing diminishing returns, and designing new pretraining methods is imperative for further progress. Prior work treated different representation learning methods quite differently, whereas the contexture theory provides a unified framework for analyzing these methods. The central argument is that a representation is learned from the association between the input X and a context variable A. We prove that if an encoder captures the maximum information of this association, in which case we say that the encoder learns the contexture, then it will be optimal on the class of tasks that are compatible with the context. We also show that a context is the most useful when the association between X and A is neither too strong nor too weak. The important implication of the contexture theory is that increasing the model size alone will achieve diminishing returns, and further advancements require better contexts. We demonstrate that many pretraining objectives can learn the contexture, including supervised learning, self-supervised learning, generative models, etc. Then, we introduce two general objectives -- SVME and KISE, for learning the contexture. We also show how to mix multiple contexts together, an effortless way to create better contexts from existing ones. Then, we prove statistical learning bounds for representation learning. Finally, we discuss the effect of the data distribution shift from pretraining to the downstream task.

Figures

Figures reproduced from arXiv: 2504.19792 by the authors.

**Figure 1.1.** Illustration of the modern ML paradigm driven by [PITH_FULL_IMAGE:figures/full_fig_p006_1_1.png] view at source ↗

**Figure 1.2.** The association between X and A determines the shape of the spectrum. Proof ∀i, P +(x,a) PX (x)PA(a) , νi [PITH_FULL_IMAGE:figures/full_fig_p012_1_2.png] view at source ↗

**Figure 1.3.** Illustration of a transformation graph. T-access is stronger than pair access. For example, in supervised learning, the context has pair access but not T-access, because the label of any x outside the training set is not given, so we cannot sample A ∼ P +(·|x) for such x. As an exercise, the reader can think about what type of access each context in [PITH_FULL_IMAGE:figures/full_fig_p014_1_3.png] view at source ↗

Figures from the paper (23 more)

**Figure 2.1.** Figure 2.1: Two widely used self-supervised learning algor [PITH_FULL_IMAGE:figures/full_fig_p020_2_1.png]

**Figure 2.2.** Figure 2.2: Convolution and convex combination of multiple [PITH_FULL_IMAGE:figures/full_fig_p025_2_2.png]

**Figure 2.3.** Figure 2.3: Estimating the eigenvalues using the post-hoc a [PITH_FULL_IMAGE:figures/full_fig_p029_2_3.png]

**Figure 2.4.** Figure 2.4: Alignment between the learned representation a [PITH_FULL_IMAGE:figures/full_fig_p031_2_4.png]

**Figure 2.** Figure 2: plots the alignment between the two encoders whil [PITH_FULL_IMAGE:figures/full_fig_p031_2.png]

**Figure 3.1.** Figure 3.1: An example where the LLM Claude 3.7 Sonnet (as of A [PITH_FULL_IMAGE:figures/full_fig_p034_3_1.png]

**Figure 3.2.** Figure 3.2: Metric illustration on abalone. Top row: context spectra. Bottom row: solid curves are τd divided by 6; dashed curves are the actual downstream prediction error. We divide τd by 6 to fit it in the same plot. desirable since invertible linear transformations on Φ do n…

**Figure 3.3.** Figure 3.3: Metric illustration on MNIST, similar to [PITH_FULL_IMAGE:figures/full_fig_p043_3_3.png]

**Figure 3.4.** Figure 3.4: Comparison of the downstream task between [PITH_FULL_IMAGE:figures/full_fig_p043_3_4.png]

**Figure 3.5.** Figure 3.5: Scatter plots of τ versus errd∗ . Dashed line: Linear fit. depends on the spectrum of the context. The metric correlates with the actual prediction error on many real datasets. Code and data availability. The code for [PITH_FULL_IMAGE:figures/full_fig_p046_3_5.png]

**Figure 4.1.** Figure 4.1: In this example, |X | = 4. We know that P + induces a joint distribution on 43 [PITH_FULL_IMAGE:figures/full_fig_p047_4_1.png]

**Figure 5.1.** Figure 5.1: Left: Three mask-type data augmentations on the hypercube data model. Right: Their theoretical κ 2/dX with different mask ratio α. This κ was initially introduced as the augmentation complexity by [167] in the context of self-supervised learning. If κ is finite, then…

**Figure 5.2.** Figure 5.2: Histograms of log k + X(x, x) 2 for random masking on wikipedia-simple with mask ratio α. The dashed vertical line in each plot indicates the 99th percentile. Another observation is that all three κ have an exponential dependency on dX . This is a manifestation of th…

**Figure 5.3.** Figure 5.3: Left: Estimated log κ 2 (99th percentile) on wikipedia-simple, which is the average of five runs with different random seeds. Right: Downstream performance on QNLI and SST-2. The solid line is the test accuracy and the dashed line is the train-test gap. The highest t…

**Figure 5.4.** Figure 5.4: (a) A graph example where the kernel is the adjace [PITH_FULL_IMAGE:figures/full_fig_p067_5_4.png]

**Figure 5.5.** Figure 5.5: Illustration of the multiscale smoothness indu [PITH_FULL_IMAGE:figures/full_fig_p068_5_5.png]

**Figure 5.6.** Figure 5.6: Performance of STKR-Prop with s(λ) = λ p and 8 iterations. The best βn with the highest test accuracy is selected. Each experiment is run with 10 random seeds. is similar to Label-Prop, and STKR-Prop with polynomial s is slightly worse. The performance under the indu…

**Figure 6.1.** Figure 6.1: Experiment results of ERM, importance weightin [PITH_FULL_IMAGE:figures/full_fig_p080_6_1.png]

**Figure 6.2.** Figure 6.2: Experiment results of ERM, importance weightin [PITH_FULL_IMAGE:figures/full_fig_p083_6_2.png]

**Figure 6.3.** Figure 6.3: Results of ERM and two DRO methods on the original [PITH_FULL_IMAGE:figures/full_fig_p084_6_3.png]

**Figure 6.4.** Figure 6.4: Average and worst-group test accuracies on: (a) [PITH_FULL_IMAGE:figures/full_fig_p085_6_4.png]

**Figure 6.5.** Figure 6.5: Comparison between DRO and DORO for CVaR. [PITH_FULL_IMAGE:figures/full_fig_p086_6_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

171 extracted references · 64 canonical work pages

[1]

Gpt- 4 technical report

Josh Achiam, Steven Adler, Sandhini Agarwal, et al. Gpt- 4 technical report. Ope- nAI Blog, 2023. 1

2023
[2]

Emergence of inv ariance and disentangle- ment in deep representations

Alessandro Achille and Stefano Soatto. Emergence of inv ariance and disentangle- ment in deep representations. Journal of Machine Learning Research , 19(50):1–34,
[3]

\alpha-req : Assessing representation quality in self-sup ervised learning by mea- suring eigenspectrum decay

Kumar K Agrawal, Arnab Kumar Mondal, Arna Ghosh, and Blak e Richards. \alpha-req : Assessing representation quality in self-sup ervised learning by mea- suring eigenspectrum decay. In S. Koyejo, S. Mohamed, A. Aga rwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, vol- ume 35, pages 17626–17638. Curran Associ...

2022
[4]

S carf: Self-supervised contrastive learning using random feature corruption

Dara Bahri, Heinrich Jiang, Yi Tay , and Donald Metzler. S carf: Self-supervised contrastive learning using random feature corruption. In International Conference on Learning Representations, 2022. (iii)

2022
[5]

Contrastive and non -contrastive self- supervised learning recover global and local spectral embe dding methods

Randall Balestriero and Yann LeCun. Contrastive and non -contrastive self- supervised learning recover global and local spectral embe dding methods. Ad- vances in Neural Information Processing Systems , 35:26671–26685, 2022. 1.5

2022
[6]

VICReg: Varia nce-invariance- covariance regularization for self-supervised learning

Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Varia nce-invariance- covariance regularization for self-supervised learning. In International Conference on Learning Representations, 2022. 2.1, 2.1, 2.2

2022
[7]

Generalized discrimin ant analysis using a ker- nel approach

Gaston Baudat and Fatiha Anouar. Generalized discrimin ant analysis using a ker- nel approach. Neural computation, 12(10):2385–2404, 2000. 3.2

2000
[8]

Laplacian eigenmaps f or dimensionality re- duction and data representation

Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps f or dimensionality re- duction and data representation. Neural computation, 15(6):1373–1396, 2003. 1.2, 1.5

2003

Show all 171 references

[9]

Rep resentation learning: A review and new perspectives

Yoshua Bengio, Aaron Courville, and Pascal Vincent. Rep resentation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013. 1

2013
[10]

Learning eigenfunctions links spectral embedding and kernel pca

Yoshua Bengio, Olivier Delalleau, Nicolas Le Roux, Jea n-François Paiement, Pas- cal Vincent, and Marie Ouimet. Learning eigenfunctions links spectral embedding and kernel pca. Neural computation, 16(10):2197–2219, 2004. 1.5

2004
[11]

Mixmatch: A holistic approach to s emi-supervised learning

David Berthelot, Nicholas Carlini, Ian Goodfellow , Ni colas Papernot, Avital Oliver, and Colin A Raﬀel. Mixmatch: A holistic approach to s emi-supervised learning. Advances in neural information processing systems , 32, 2019. 1.5

2019
[12]

D iscriminative learning for diﬀering training and test distributions

Steﬀen Bickel, Michael Brückner, and T obias Scheﬀer. D iscriminative learning for diﬀering training and test distributions. In Proceedings of the 24th international conference on Machine learning, pages 81–88, 2007. 6 140

2007
[13]

Statistical properties of kernel principal component analysis

Gilles Blanchard, Olivier Bousquet, and Laurent Zwald . Statistical properties of kernel principal component analysis. Machine Learning, 66:259–294, 2007. 5.2

2007
[14]

Demo graphic dialectal vari- ation in social media: A case study of African-American Engl ish

Su Lin Blodgett, Lisa Green, and Brendan O’Connor. Demo graphic dialectal vari- ation in social media: A case study of African-American Engl ish. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1119– 1130, Austin, Texas, Novembe...

2016
[15]

On the opportunities and risks of foundati on models

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altma n, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoin e Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundati on models. arXiv preprint arXiv:2108.07258, 2021. 1, 3

2021 arXiv
[16]

Nuanced metrics for measuring unintended bias with rea l data for text clas- siﬁcation

Daniel Borkan, Lucas Dixon, Jeﬀrey Sorensen, Nithum Th ain, and Lucy Vasser- man. Nuanced metrics for measuring unintended bias with rea l data for text clas- siﬁcation. In Companion Proceedings of The 2019 World Wide Web Conference , pages 491–500, 2019. 6.4

2019
[17]

Richard P . Brent. An algorithm with guaranteed convergence for ﬁnding a zero of a function. The Computer Journal, 14(4):422–425, 1971. 6.4

1971
[18]

Functional analysis, Sobolev spaces and partial diﬀerenti al equations

Haim Brezis. Functional analysis, Sobolev spaces and partial diﬀerenti al equations . Springer, 2011. D.5

2011
[19]

Jonathon Byrd and Zachary Lipton. What is the eﬀect of im portance weighting in deep learning? In Kamalika Chaudhuri and Ruslan Salakhut dinov , editors, Proceedings of the 36th International Conference on Machin e Learning , volume 97 of Proceedings of Machine Learning Resear...

2019
[20]

The SSL interplay: Augmentations, inductive bias, an d generalization

Vivien Cabannes, Bobak Kiani, Randall Balestriero, Ya nn Lecun, and Alberto Bi- etti. The SSL interplay: Augmentations, inductive bias, an d generalization. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Eng elhardt, Sivan Sabato, and Jonathan Scarlett, editors, Pro...

2023
[21]

Lee, Deming Chen, and Tri Dao

Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jas on D. Lee, Deming Chen, and Tri Dao. Medusa: Simple llm inference acceleratio n framework with multiple decoding heads. arXiv preprint arXiv: 2401.10774 , 2024. 4

2024 arXiv
[22]

Xgboost: A scalable tr ee boosting system

Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tr ee boosting system. In Proceedings of the 22nd acm sigkdd international conferenc e on knowledge discovery and data mining, pages 785–794, 2016. 1.1, 2, 4, (ii)

2016
[23]

A sim- ple framework for contrastive learning of visual represent ations

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoﬀ rey Hinton. A sim- ple framework for contrastive learning of visual represent ations. In International conference on machine learning, pages 1597–1607. PMLR, 2020. 1, 1.4, 4.1

2020
[24]

Exploring simple siamese re presentation learning

Xinlei Chen and Kaiming He. Exploring simple siamese re presentation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15750–15758, June 2021. 2.1

2021
[25]

De constructing denois- ing diﬀusion models for self-supervised learning

Xinlei Chen, Zhuang Liu, Saining Xie, and Kaiming He. De constructing denois- ing diﬀusion models for self-supervised learning. In The Thirteenth International Conference on Learning Representations, 2025. 2.4 141

2025
[26]

Spectral graph theory , volume 92

Fan RK Chung. Spectral graph theory , volume 92. American Mathematical Soc.,
[27]

Le, and Christopher D

Kevin Clark, Minh-Thang Luong, Quoc V . Le, and Christopher D. Manning. Elec- tra: Pre-training text encoders as discriminators rather t han generators. In Inter- national Conference on Learning Representations , 2020. 4

2020
[28]

Gra- dient descent on neural networks typically occurs at the edg e of stability

Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gra- dient descent on neural networks typically occurs at the edg e of stability. In Inter- national Conference on Learning Representations , 2021. 2, 7

2021
[29]

Diﬀusion maps

Ronald R Coifman and Stéphane Lafon. Diﬀusion maps. Applied and computational harmonic analysis, 21(1):5–30, 2006. 1.2, 5.3

2006
[30]

Multinomial goodness -of-ﬁt tests

Noel Cressie and Timothy RC Read. Multinomial goodness -of-ﬁt tests. Journal of the Royal Statistical Society Series B: Statistical Methodology, 46(3):440–464, 1984. 6.4

1984
[31]

A kernel theory of modern data augmentation

Tri Dao, Albert Gu, Alexander Ratner, Virginia Smith, C hris De Sa, and Christo- pher Ré. A kernel theory of modern data augmentation. In International conference on machine learning, pages 1528–1537. PMLR, 2019. 1.5

2019
[32]

Fast line ar algebra is stable

James Demmel, Ioana Dumitriu, and Olga Holtz. Fast line ar algebra is stable. Numerische Mathematik, 108(1):59–91, 2007. 2

2007
[33]

Neuralef: Deconst ructing kernels by deep neural networks

Zhijie Deng, Jiaxin Shi, and Jun Zhu. Neuralef: Deconst ructing kernels by deep neural networks. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th Interna- tional Conference on Machine Learning, vol...

2022
[34]

BERT: Pre- training of deep bidirectional transformers for language u nderstanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina T outanova. BERT: Pre- training of deep bidirectional transformers for language u nderstanding. In Pro- ceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Huma...

2019
[35]

Improved regularization of convolutional neural networks with cutout

Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552, 2017. 1.2, 5.1

2017 arXiv
[36]

An image is worth 16x16 words: Transformers for image recognition at scale

Alexey Dosovitskiy , Lucas Beyer, Alexander Kolesniko v , Dirk Weissenborn, Xi- aohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthia s Minderer, Georg Heigold, Sylvain Gelly , Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni...

2021
[37]

The llama 3 herd of models

Abhimanyu Dubey , Abhinav Jauhri, Abhinav Pandey , et al . The llama 3 herd of models. ArXiv, abs/2407.21783, 2024. 1

2024 arXiv
[38]

Learning models with uniform perfor- mance via distributionally robust optimization

John Duchi and Hongseok Namkoong. Learning models with uniform perfor- mance via distributionally robust optimization. arXiv preprint arXiv:1810.08750 ,

arXiv
[39]

T owards revealing the mystery behind chain of thought: A theoretical perspective

Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. T owards revealing the mystery behind chain of thought: A theoretical perspective. In Thirty-seventh Conference on Neural Information Processi ng Systems, 2023. 7

2023
[40]

Matthias Fey and Jan E. Lenssen. Fast graph representat ion learning with PyT orch Geometric. In ICLR Workshop on Representation Learning on Graphs and Mani folds, 142
[41]

Sobolev norm learnin g rates for regularized least-squares algorithms

Simon Fischer and Ingo Steinwart. Sobolev norm learnin g rates for regularized least-squares algorithms. The Journal of Machine Learning Research , 21(1):8464– 8501, 2020. 5.2, 5.2, 5.4, 5.27, D.7

2020
[42]

A decision-theoreti c generalization of on-line learning and an application to boosting

Yoav Freund and Robert E Schapire. A decision-theoreti c generalization of on-line learning and an application to boosting. Journal of computer and system sciences , 55(1):119–139, 1997. 4.2

1997
[43]

Latent functional maps: a spectral framew ork for represen- tation alignment

Marco Fumero, Marco Pegoraro, Valentino Maiorca, Fran cesco Locatello, and Emanuele Rodolà. Latent functional maps: a spectral framew ork for represen- tation alignment. In A. Globerson, L. Mackey , D. Belgrave, A . Fan, U. Paquet, J. T omczak, and C. Zhang, editors, Advances ...
[44]

Scaling laws for reward model overop- timization

Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overop- timization. In International Conference on Machine Learning , pages 10835–10866. PMLR, 2023. 2.4

2023
[45]

Gemini: A Family of Highly Capable Multimo dal Models

Gemini Team. Gemini: A Family of Highly Capable Multimo dal Models. arXiv e-prints, page arXiv:2312.11805, December 2023. 1

2023 arXiv
[46]

Un supervised representa- tion learning by predicting image rotations

Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Un supervised representa- tion learning by predicting image rotations. In International Conference on Learning Representations, 2018. 1.2

2018
[47]

Multiple kernel learn ing algorithms

Mehmet Gönen and Ethem Alpaydin. Multiple kernel learn ing algorithms. Jour- nal of Machine Learning Research , 12(64):2211–2268, 2011. 4.2

2011
[48]

Generat ive adversarial nets

Ian Goodfellow , Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley , Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generat ive adversarial nets. Advances in neural information processing systems , 27, 2014. 2.4

2014
[49]

Revisit- ing deep learning models for tabular data

Yury Gorishniy , Ivan Rubachev , Valentin Khrulkov , andArtem Babenko. Revisit- ing deep learning models for tabular data. In A. Beygelzimer, Y . Dauphin, P . Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Sys- tems, 2021. 4.4

2021
[50]

Bootstrap your own latent - a new approach t o self-supervised learning

Jean-Bastien Grill, Florian Strub, Florent Altché, Co rentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avil a Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuog lu, Remi Munos, and Michal Valko. Bootstrap your own latent - a new...

2020
[51]

In search of lost domain generalization

Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations , 2021. 6.1, 6.4

2021
[52]

Characteriz- ing implicit bias in terms of optimization geometry

Suriya Gunasekar, Jason Lee, Daniel Soudry , and Nathan Srebro. Characteriz- ing implicit bias in terms of optimization geometry. In Jenn ifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Mach...

2018
[53]

Deepsee k-r1: Incen- 143 tivizing reasoning capability in llms via reinforcement le arning

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoy u Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepsee k-r1: Incen- 143 tivizing reasoning capability in llms via reinforcement le arning. arXiv preprint arXiv:2501.12948, 2025. 1.1, 2.3, 7

2025 arXiv
[54]

Statistical intervals: a guide for practitioners , volume 92

Gerald J Hahn and William Q Meeker. Statistical intervals: a guide for practitioners , volume 92. John Wiley & Sons, 2011. 5.1

2011
[55]

Graph representation learning

William L Hamilton. Graph representation learning. Morgan & Claypool Publishers,
[56]

A theoretical study of induc tive biases in con- trastive learning

Jeﬀ Z HaoChen and Tengyu Ma. A theoretical study of induc tive biases in con- trastive learning. arXiv preprint arXiv:2211.14699, 2022. 1.5

2022 arXiv
[57]

P rovable guarantees for self-supervised deep learning with spectral contrastive loss

Jeﬀ Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. P rovable guarantees for self-supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 34:5000–5011, 2021. 1.14, 1.5, 2, 2.1

2021
[58]

HaoChen, Colin Wei, Ananya Kumar, and Tengyu Ma

Jeﬀ Z. HaoChen, Colin Wei, Ananya Kumar, and Tengyu Ma. B eyond separability: Analyzing the linear transferability of contrastive repre sentations to related sub- populations. In Alice H. Oh, Alekh Agarwal, Danielle Belgra ve, and Kyunghyun Cho, editors, Advances in Neural Inf...

2022
[59]

Fairness without demographics in repeated loss minimization

Tatsunori Hashimoto, Megha Srivastava, Hongseok Namk oong, and Percy Liang. Fairness without demographics in repeated loss minimization. In Jennifer Dy and Andreas Krause, editors, International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Rese...

1929
[60]

Masked autoencoders are scalable vision learners

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16000–16009, 2022. 1

2022
[61]

D eep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. D eep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016. 2.6, 6.4

2016
[62]

Hearst, S.T

M.A. Hearst, S.T. Dumais, E. Osuna, J. Platt, and B. Scho lkopf. Support vector machines. IEEE Intelligent Systems and their Applications , 13(4):18–28, 1998. 1.5

1998
[63]

Hinton, Oriol Vinyals, and Jeﬀrey Dean

Geoﬀrey E. Hinton, Oriol Vinyals, and Jeﬀrey Dean. Dist illing the knowledge in a neural network. ArXiv, abs/1503.02531, 2015. 2.3

2015 arXiv
[64]

Denoising di ﬀusion probabilistic mod- els

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising di ﬀusion probabilistic mod- els. Advances in neural information processing systems , 33:6840–6851, 2020. 1, 2.4

2020
[65]

TabPFN: A transformer that solves small tabular classiﬁcat ion problems in a sec- ond

Noah Hollmann, Samuel Müller, Katharina Eggensperger , and Frank Hutter. TabPFN: A transformer that solves small tabular classiﬁcat ion problems in a sec- ond. In The Eleventh International Conference on Learning Represe ntations, 2023. 4.4

2023
[66]

Accurate pre- dictions on small data with a tabular foundation model

Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hu tter. Accurate pre- dictions on small data with a tabular foundation model. Nature, 637(8045):319– 326, 2025. 1

2025
[67]

Tagging performance corr elates with author age

Dirk Hovy and Anders Søgaard. Tagging performance corr elates with author age. In Proceedings of the 53rd annual meeting of the Association fo r Computational Linguis- tics and the 7th international joint conference on natural l anguage processing (volume 2: Short papers), pag...

2015
[68]

Correcting sample selection bias by unlabeled d ata

Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Ber nhard Schölkopf, and Alex Smola. Correcting sample selection bias by unlabeled d ata. Advances in 144 neural information processing systems, 19:601–608, 2006. 6

2006
[69]

A survey on hallucination in large language models: Principles, tax onomy , challenges, and open questions

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhang yin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin , et al. A survey on hallucination in large language models: Principles, tax onomy , challenges, and open questions. ACM T ransactions on Infor...

2025
[70]

Improving trans- former optimization through better initialization

Xiao Shi Huang, Felipe Perez, Jimmy Ba, and Maksims Volk ovs. Improving trans- former optimization through better initialization. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machin e Learning, volume 119 of Proceedings of Mach...

2020
[71]

Robust estimation of a location paramete r

Peter J Huber. Robust estimation of a location paramete r. In Breakthroughs in statis- tics, pages 492–518. Springer, 1992. 6.4

1992
[72]

What m akes imagenet good for transfer learning? arXiv:1608.08614, 2016

Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. What m akes imagenet good for transfer learning? arXiv:1608.08614, 2016. 1, 1.5

2016 arXiv
[73]

Position: The platonic representation hypothesis

Minyoung Huh, Brian Cheung, T ongzhou Wang, and Phillip Isola. Position: The platonic representation hypothesis. In Proc. International Conference on Machine Learning, Vienna, Austria, July 2024. 1, 1.5, 2.6, 2.6, 2.6

2024
[74]

Estimation of non-normalized statist ical models by score match- ing

Aapo Hyvärinen. Estimation of non-normalized statist ical models by score match- ing. Journal of Machine Learning Research , 6(24):695–709, 2005. 1

2005
[75]

T o wards a learning theory of representation alignment

Francesco Insulla, Shuo Huang, and Lorenzo Rosasco. T o wards a learning theory of representation alignment. In The Thirteenth International Conference on Learning Representations, 2025. 1.5

2025
[76]

Adap- tive mixtures of local experts

Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoﬀrey E Hinton. Adap- tive mixtures of local experts. Neural computation, 3(1):79–87, 1991. 4

1991
[77]

Neu ral tangent kernel: Convergence and generalization in neural networks

Arthur Jacot, Franck Gabriel, and Clement Hongler. Neu ral tangent kernel: Convergence and generalization in neural networks. In S. Be ngio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volu...

2018
[78]

Ope- nai o1 system card

Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, A hmed El-Kishky , Aiden Low , Alec Helyar, Aleksander Madry , Alex Beutel, AlexCarney , et al. Ope- nai o1 system card. arXiv preprint arXiv:2412.16720, 2024. 1.1, 7

2024 arXiv
[79]

Schapire, and Matus Telgarsky

Ziwei Ji, Miroslav Dudík, Robert E. Schapire, and Matus Telgarsky. Gradient de- scent follows the regularization path for general losses. I n Jacob Abernethy and Shivani Agarwal, editors, Proceedings of Thirty Third Conference on Learning Theory , volume 125 of Proceedings of ...

2020
[80]

Understanding dimen- sional collapse in contrastive self-supervised learning

Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimen- sional collapse in contrastive self-supervised learning. In International Conference on Learning Representations, 2022. 1.5, 2.6

2022
[81]

Johnson, Ayoub El Hanchi, and Chris J

Daniel D. Johnson, Ayoub El Hanchi, and Chris J. Maddiso n. Contrastive learn- ing can ﬁnd an optimal basis for approximately view-invaria nt functions. In The Eleventh International Conference on Learning Representa tions, 2023. 1.3, 1.5

2023
[82]

Highly accurate protein structure predic tion with alphafold

John Jumper, Richard Evans, Alexander Pritzel, Tim Gre en, Michael Figurnov , Olaf Ronneberger, Kathryn T unyasuvunakool, Russ Bates, Augustin Žídek, Anna 145 Potapenko, et al. Highly accurate protein structure predic tion with alphafold. nature, 596(7873):583–589, 2021. 7

2021
[83]

Thinking, fast and slow

Daniel Kahneman. Thinking, fast and slow . macmillan, 2011. 1.1

2011
[84]

Scaling laws for neural language models

Jared Kaplan, Sam McCandlish, T om Henighan, T om B Brown , Benjamin Chess, Rewon Child, Scott Gray , Alec Radford, Jeﬀrey Wu, and Dario A modei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. 1, 2.6

2001 arXiv
[85]

Supervised contrastive learn- ing

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learn- ing. Advances in neural information processing systems , 33:18661–18673, 2020. 4.1

2020
[86]

Kingma and Jimmy Ba

Diederik P . Kingma and Jimmy Ba. Adam: A method for stoch astic optimization. In International Conference on Learning Representations , 2015. 2, 2.5, 7

2015
[87]

Kingma and Max Welling

Diederik P . Kingma and Max Welling. Auto-Encoding Vari ational Bayes. In Inter- national Conference on Learning Representations , 2014. 2.4

2014
[88]

Wilds: A benchmark of in-the-wild distribution shifts

Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Mich ael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga , Richard Lanas Phillips, Irena Gao, T ony Lee, Etienne David, Ian Stavness,Wei Guo, Berton Earn- shaw , Imran Haque, Sara M Beery , Jure Leskovec,...

2021
[89]

Sim- ilarity of neural network representations revisited

Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Ge oﬀrey Hinton. Sim- ilarity of neural network representations revisited. In Ka malika Chaudhuri and Ruslan Salakhutdinov , editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceed...

2019
[90]

Representational similarity analysis-connecting the branches of systems ne uroscience

Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandet tini. Representational similarity analysis-connecting the branches of systems ne uroscience. Frontiers in systems neuroscience, 2:249, 2008. 1.5

2008
[91]

Learning multi ple layers of features from tiny images, 2009

Alex Krizhevsky , Geoﬀrey Hinton, et al. Learning multi ple layers of features from tiny images, 2009. 1.2

2009
[92]

Imagenet classiﬁcation with deep convolutional neural networks

Alex Krizhevsky , Ilya Sutskever, and Geoﬀrey E Hinton. Imagenet classiﬁcation with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012. 1, 1.5

2012
[93]

Temporal ensembling for sem i-supervised learning

Samuli Laine and Timo Aila. Temporal ensembling for sem i-supervised learning. In International Conference on Learning Representations , 2017. 1.5

2017
[94]

How we analyzed the compas recidivism algorithm

Jeﬀ Larson, Surya Mattu, Lauren Kirchner, and Julia Ang win. How we analyzed the compas recidivism algorithm. ProPublica (5 2016), 9(1):3–3, 2016. 6.3

2016
[95]

Gradient-based learning applied to document recognition

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haﬀ ner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278– 2324, 1998. 3.3

1998
[96]

P redicting what you already know helps: Provable self-supervised learning

Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo. P redicting what you already know helps: Provable self-supervised learning. Advances in Neural Infor- mation Processing Systems, 34:309–323, 2021. 1.5

2021
[97]

Chain of thought empow- 146 ers transformers to solve inherently serial problems

Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. Chain of thought empow- 146 ers transformers to solve inherently serial problems. In The T welfth International Conference on Learning Representations, 2024. 7

2024
[98]

Foundations of multisensory artiﬁcial i ntelligence

Paul Pu Liang. Foundations of multisensory artiﬁcial i ntelligence. arXiv preprint arXiv:2404.18976, 2024. 7

2024 arXiv
[99]

Masked pre- diction tasks: a parameter identiﬁability view

Bingbin Liu, Daniel Hsu, Pradeep Ravikumar, and Andrej Risteski. Masked pre- diction tasks: a parameter identiﬁability view. Advances in Neural Information Pro- cessing Systems, 2022. 1.5

2022
[100]

HaoChen, Adrien Gaidon, and Tengyu Ma

Hong Liu, Jeﬀ Z. HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. In NeurIPS 2021 Workshop on Distri- bution Shifts: Connecting Methods and Applications , 2021. 1.5

2021
[101]

Improving ker nel ﬁsher discrimi- nant analysis for face recognition

Qingshan Liu, Hanqing Lu, and Songde Ma. Improving ker nel ﬁsher discrimi- nant analysis for face recognition. IEEE transactions on circuits and systems for video technology, 14(1):42–49, 2004. 3.2

2004
[102]

Roberta: A robustly optimized bert pretraining approach

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy , Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692, 2019. 1

1907 arXiv
[103]

De ep learning face at- tributes in the wild

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. De ep learning face at- tributes in the wild. In Proceedings of the IEEE international conference on comput er vision, pages 3730–3738, 2015. 6.4

2015
[104]

Decoupled weight de cay regularization

Ilya Loshchilov and Frank Hutter. Decoupled weight de cay regularization. In International Conference on Learning Representations , 2019. 2.5

2019
[105]

When do neural nets outperform boosted trees on tabular data? Advances in Neural Information Process- ing Systems, 36, 2023

Duncan McElfresh, Sujay Khandagale, Jonathan Valver de, Vishak Prasad C, Ganesh Ramakrishnan, Micah Goldblum, and Colin White. When do neural nets outperform boosted trees on tabular data? Advances in Neural Information Process- ing Systems, 36, 2023. 4.2, 4.4

2023
[106]

Fisher discriminant analysis with kernels

Sebastian Mika, Gunnar Ratsch, Jason Weston, Bernhar d Scholkopf, and Klaus- Robert Mullers. Fisher discriminant analysis with kernels . In Neural networks for signal processing IX: Proceedings of the 1999 IEEE signal pr ocessing society workshop (cat. no. 98th8468), pages 41–...

1999
[107]

Virtual ad- versarial training: a regularization method for supervise d and semi-supervised learning

Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual ad- versarial training: a regularization method for supervise d and semi-supervised learning. IEEE transactions on pattern analysis and machine intellig ence, 41(8):1979– 1993, 2018. 1.5

1979
[108]

L earning with group invariant features: A kernel perspective

Youssef Mroueh, Stephen Voinea, and T omaso A Poggio. L earning with group invariant features: A kernel perspective. Advances in neural information processing systems, 28, 2015. 1.5

2015
[109]

Dinov2: Learning robust visual features witho ut supervision

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov , Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El- Nouby , et al. Dinov2: Learning robust visual features witho ut supervision. arXiv preprint arXiv:2304.07193, 2023. 1

2023 arXiv
[110]

Train- ing language models to follow instructions with human feedb ack

Long Ouyang, Jeﬀrey Wu, Xu Jiang, Diogo Almeida, Carro ll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray , et al. Train- ing language models to follow instructions with human feedb ack. In Proc. Ad- vances in Neural Information Processing S...

2022
[111]

A survey on transfer le arning

Sinno Jialin Pan and Qiang Yang. A survey on transfer le arning. IEEE T ransactions on Knowledge and Data Engineering , 22(10):1345–1359, 2010. 6

2010
[112]

Prevalence o f neural collapse during the terminal phase of deep learning training

Vardan Papyan, XY Han, and David L Donoho. Prevalence o f neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24652–24663, 2020. 1, 2.1

2020
[113]

David Pfau, Stig Petersen, Ashish Agarwal, David G. T. Barrett, and Kimberly L. Stachenfeld. Spectral inference networks: Unifying deep and spectral learning. In International Conference on Learning Representations , 2019. 2.2

2019
[114]

Contrasting the land- scape of contrastive and non-contrastive learning

Ashwini Pokle, Jinjin Tian, Yuchen Li, and Andrej Rist eski. Contrasting the land- scape of contrastive and non-contrastive learning. In Gust au Camps-Valls, Fran- cisco J. R. Ruiz, and Isabel Valera, editors, International Conference on Artiﬁcial In- telligence and Statistics...

2022
[115]

Dataset shift in machine learning

Joaquin Quionero-Candela, Masashi Sugiyama, Anton S chwaighofer, and Neil D Lawrence. Dataset shift in machine learning . The MIT Press, 2009. 6

2009
[116]

Learning transferable visual models from natural language supervision

Alec Radford, Jong Wook Kim, Chris Hallacy , Aditya Ramesh, Gabriel Goh, Sand- hini Agarwal, Girish Sastry , Amanda Askell, Pamela Mishkin , Jack Clark, et al. Learning transferable visual models from natural language supervision. In Inter- national Conference on Machine Learn...

2021
[117]

Language models are unsupervised multitask lea rners

Alec Radford, Jeﬀrey Wu, Rewon Child, David Luan, Dari o Amodei, and Ilya Sutskever. Language models are unsupervised multitask lea rners. OpenAI blog,
[118]

Local group invariant representations via orbi t embeddings

Anant Raj, Abhishek Kumar, Youssef Mroueh, T om Fletch er, and Bernhard Schölkopf. Local group invariant representations via orbi t embeddings. In Ar- tiﬁcial Intelligence and Statistics, pages 1225–1235. PMLR, 2017. 1.5

2017
[119]

Lewis Fry Richardson. Ix. the approximate arithmetic al solution by ﬁnite diﬀer- ences of physical problems involving diﬀerential equation s, with an application to the stresses in a masonry dam. Philosophical T ransactions of the Royal Society of London. Series A, containing ...

1911
[120]

Berg, and Li Fei-Fei

Olga Russakovsky , Jia Deng, Hao Su, Jonathan Krause, S anjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy , Aditya Khosla, MichaelBernstein, Alexan- der C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Reco gnition Challenge. International Journal of Computer Vision...

2015
[121]

Hashimoto, a nd Percy Liang

Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, a nd Percy Liang. Dis- tributionally robust neural networks for group shifts: On t he importance of reg- ularization for worst-case generalization. In International Conference on Learning Representations, 2020. 6.2, 6.1, 6.2

2020
[122]

Understandi ng contrastive learning requires incorporating inductive biases

Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Mis ra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy. Understandi ng contrastive learning requires incorporating inductive biases. In International Conference on Ma- chine Learning, pages 19250–19286. PMLR,...

2022
[123]

A theoretical analysis of contrastive unsupervised rep- resentation learning

Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mi khail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised rep- resentation learning. In International Conference on Machine Learning , pages 5628–
[124]

Learning with kernels: support vector machines, regularization, optimization, and beyond

Bernhard Schölkopf and Alexander J Smola. Learning with kernels: support vector machines, regularization, optimization, and beyond . MIT press, 2002. 1.2, 1.5, 2, 5.3, 5.3

2002
[125]

On the eigenspectrum of the gram matrix and the generalizati on error of kernel- pca

John Shawe-Taylor, Christopher KI Williams, Nello Cr istianini, and Jaz Kandola. On the eigenspectrum of the gram matrix and the generalizati on error of kernel- pca. IEEE T ransactions on Information Theory, 51(7):2510–2522, 2005. 2.5, 2.5, 5.29

2005
[126]

Connect, not collapse: Explaining con- trastive learning for unsupervised domain adaptation

Kendrick Shen, Robbie M Jones, Ananya Kumar, Sang Mich ael Xie, Jeﬀ Z HaoChen, Tengyu Ma, and Percy Liang. Connect, not collapse: Explaining con- trastive learning for unsupervised domain adaptation. In International Conference on Machine Learning, pages 19847–19878. PMLR, 2022. 1.5

2022
[127]

Learning with bad trai ning data via iterative trimmed loss minimization

Yanyao Shen and Sujay Sanghavi. Learning with bad trai ning data via iterative trimmed loss minimization. In International Conference on Machine Learning, pages 5739–5748. PMLR, 2019. 6.4

2019
[128]

Improving predictive inferen ce under covariate shift by weighting the log-likelihood function

Hidetoshi Shimodaira. Improving predictive inferen ce under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference , 90(2):227–244, 2000. 2.1, 2.5, 6, 6.1

2000
[129]

An information theory perspective on varian ce-invariance- covariance regularization

Ravid Shwartz-Ziv , Randall Balestriero, Kenji Kawag uchi, Tim GJ Rudner, and Yann LeCun. An information theory perspective on varian ce-invariance- covariance regularization. Advances in Neural Information Processing Systems , 36:33965–33998, 2023. 1.5

2023
[130]

Manning, Andrew Ng, and Christopher Potts

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compo- sitionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empir- ical Methods in Natural Language Pr...

2013
[131]

Fix- match: Simplifying semi-supervised learning with consist ency and conﬁdence

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizh ao Zhang, Han Zhang, Colin A Raﬀel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Li ang Li. Fix- match: Simplifying semi-supervised learning with consist ency and conﬁdence. Advances in neural information processing systems , ...

2020
[132]

Denois ing diﬀusion implicit models

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois ing diﬀusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 1, 2.4

2010 arXiv
[133]

The implicit bias of gradient descent on separable d ata

Daniel Soudry , Elad Hoﬀer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable d ata. The Journal of Machine Learning Research, 19(1):2822–2878, 2018. 6.2

2018
[134]

Test of time award talk: Sequence to sequence learning with neural networks

Ilya Sutskever. Test of time award talk: Sequence to sequence learning with neural networks. Advances in Neural Information Processing Syste ms, 2024. 1, 1.1

2024
[135]

Székely , Maria L

Gábor J. Székely , Maria L. Rizzo, and Nail K. Bakirov. M easuring and testing dependence by correlation of distances. The Annals of Statistics , 35(6):2769–2794,
[136]

Mean teachers are b etter role models: Weight- averaged consistency targets improve semi-supervised deep learning results

Antti Tarvainen and Harri Valpola. Mean teachers are b etter role models: Weight- averaged consistency targets improve semi-supervised deep learning results. Ad- vances in neural information processing systems , 30, 2017. 1.5

2017
[137]

Gender and dialect bias in youtube’s a utomatic captions

Rachael Tatman. Gender and dialect bias in youtube’s a utomatic captions. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, pages 53–59, 2017. 6.1 149

2017
[138]

Imbalance trouble: Revisiting neural-collapse ge ometry

Christos Thrampoulidis, Ganesh Ramachandra Kini, Va la Vakilian, and Tina Behnia. Imbalance trouble: Revisiting neural-collapse ge ometry. Advances in Neu- ral Information Processing Systems, 35:27225–27238, 2022. 2.1

2022
[139]

Deep contrastive learning is provably (almost) principal compo- nent analysis

Yuandong Tian. Deep contrastive learning is provably (almost) principal compo- nent analysis. Advances in Neural Information Processing Systems , 2022. 1.5

2022
[140]

Unders tanding self-supervised learning dynamics without contrastive pairs

Yuandong Tian, Xinlei Chen, and Surya Ganguli. Unders tanding self-supervised learning dynamics without contrastive pairs. In Marina Mei la and T ong Zhang, editors, Proceedings of the 38th International Conference on Machin e Learning, ICML 2021, 18-24 July 2021, Virtual Even...

2021
[141]

Gradient descent

Ryan Tibshirani. Gradient descent. Lecture Notes, 20 19. Available at: https://www.stat.cmu.edu/~ryantibs/convexopt/lectures/grad-descent.pdf. E.1
[142]

Contrastive estima- tion reveals topic posterior information to linear models

Christopher T osh, Akshay Krishnamurthy , and Daniel H su. Contrastive estima- tion reveals topic posterior information to linear models. J. Mach. Learn. Res. , 22:281–1, 2021. 1.5

2021
[143]

Contrastive learning, multi-view redundancy , and linear models

Christopher T osh, Akshay Krishnamurthy , and Daniel Hsu. Contrastive learning, multi-view redundancy , and linear models. In Algorithmic Learning Theory, pages 1179–1206. PMLR, 2021. 1.5

2021
[144]

Position : Why tabular foun- dation models should be a research priority

Boris Van Breugel and Mihaela Van Der Schaar. Position : Why tabular foun- dation models should be a research priority. In Ruslan Salak hutdinov , Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jon athan Scarlett, and Fe- lix Berkenkamp, editors, Proceedings of th...

2024
[145]

A survey on semi- supervised learning

Jesper E Van Engelen and Holger H Hoos. A survey on semi- supervised learning. Machine learning, 109(2):373–440, 2020. 1.5

2020
[146]

van Rijn, Bernd Bischl, and Luis T orgo

Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis T orgo. Openml: Net- worked science in machine learning. SIGKDD Explorations, 15(2):49–60, 2013. 2.5, 4.4

2013
[147]

Introduction to the non-asymptotic analysis of random matri- ces

Roman Vershynin. Introduction to the non-asymptotic analysis of random matri- ces. arXiv preprint arXiv:1011.3027, 2010. E.3

2010 arXiv
[148]

Wainwright

Martin J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint . Cambridge Series in Statistical and Probabilistic Mathema tics. Cambridge Uni- versity Press, 2019. 5.2, D.3

2019
[149]

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hil l, Omer Levy , and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. BLACKBOXNLP@EMNLP, 2018. 5.1

2018
[150]

Is importance weighting incompatible with inte rpolating classiﬁers? In International Conference on Learning Representations , 2022

Ke Alexander Wang, Niladri Shekhar Chatterji, Saminu l Haque, and Tatsunori Hashimoto. Is importance weighting incompatible with inte rpolating classiﬁers? In International Conference on Learning Representations , 2022. 6.2

2022
[151]

Deep visual domain adaptati on: A survey

Mei Wang and Weihong Deng. Deep visual domain adaptati on: A survey. Neuro- computing, 312:135–153, 2018. 6

2018
[152]

A message passing perspective on learning dynamics of contr astive learning

Yifei Wang, Qi Zhang, Tianqi Du, Jiansheng Yang, Zhouchen Lin, and Yisen Wang. A message passing perspective on learning dynamics of contr astive learning. In 150 The Eleventh International Conference on Learning Represe ntations, 2023. 1.5

2023
[153]

Why do pretr ained language mod- els help in downstream tasks? an analysis of head and prompt t uning

Colin Wei, Sang Michael Xie, and Tengyu Ma. Why do pretr ained language mod- els help in downstream tasks? an analysis of head and prompt t uning. Advances in Neural Information Processing Systems, 34:16158–16170, 2021. 1.5

2021
[154]

Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeﬀ D ean, and William Fe- dus

Jason Wei, Yi Tay , Rishi Bommasani, Colin Raﬀel, Barre t Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeﬀ D ean, and William Fe- dus. Emergent abilities of large language models...

2022
[155]

Chain-of-thought prompting eli cits reasoning in large language models

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosm a, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting eli cits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837, 2022. 1.1, 7

2022
[156]

T oward understanding the feature learning process of self-supervised contrastive learning

Zixin Wen and Yuanzhi Li. T oward understanding the feature learning process of self-supervised contrastive learning. In Marina Meila and T ong Zhang, editors, Proceedings of the 38th International Conference on Machin e Learning, ICML 2021, 18- 24 July 2021, Virtual Event, vo...

2021
[157]

The mechanism of prediction h ead in non-contrastive self-supervised learning

Zixin Wen and Yuanzhi Li. The mechanism of prediction h ead in non-contrastive self-supervised learning. Advances in Neural Information Processing Systems , 2022. 1.5

2022
[158]

Should you mask 15% in masked language modeling? In Proceedings of the 17th Conference of the European Chapter of the Association for Computational L inguistics: Main V olume,

Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. Should you mask 15% in masked language modeling? In Proceedings of the 17th Conference of the European Chapter of the Association for Computational L inguistics: Main V olume,
[159]

Denoising diﬀusion autoencoders are uniﬁed self-supervised learners

Weilai Xiang, Hongyu Yang, Di Huang, and Yunhong Wang. Denoising diﬀusion autoencoders are uniﬁed self-supervised learners. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15802–15812, 2023. 2.4

2023
[160]

Self-training with noisy student improves imagenet classiﬁcation

Qizhe Xie, Minh-Thang Luong, Eduard Hovy , and Quoc V Le . Self-training with noisy student improves imagenet classiﬁcation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10687–10698, 2020. 1.5

2020
[161]

On la yer normaliza- tion in the transformer architecture

Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Z heng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On la yer normaliza- tion in the transformer architecture. InInternational Conference on Machine Learning, pages 10524–10533. PMLR, 2020. 1.5

2020
[162]

Understanding the role of importance weighting for deep learning

Da Xu, Yuting Ye, and Chuanwei Ruan. Understanding the role of importance weighting for deep learning. In International Conference on Learning Representations,
[163]

Barlow twins: Self-supervised learning via redundancy reduction

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pages 12310–12320. PMLR, 2021. 2.1

2021
[164]

Un- derstanding why generalized reweighting does not improve o ver ERM

Runtian Zhai, Chen Dan, J Zico Kolter, and Pradeep Kuma r Ravikumar. Un- derstanding why generalized reweighting does not improve o ver ERM. In The Eleventh International Conference on Learning Representa tions, 2023. 6.1

2023
[165]

Doro: Distribu- 151 tional and outlier robust optimization

Runtian Zhai, Chen Dan, Zico Kolter, and Pradeep Ravik umar. Doro: Distribu- 151 tional and outlier robust optimization. In Marina Meila and T ong Zhang, editors, Proceedings of the 38th International Conference on Machin e Learning, volume 139 of Proceedings of Machine Learni...

2021
[166]

Boosted CVar classiﬁcation

Runtian Zhai, Chen Dan, Arun Suggala, J Zico Kolter, and Pradeep Kumar Raviku- mar. Boosted CVar classiﬁcation. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. 6.1

2021
[167]

Understanding augmentation-based self-supervised repre sentation learning via rkhs approximation and regression

Runtian Zhai, Bingbin Liu, Andrej Risteski, Zico Kolt er, and Pradeep Ravikumar. Understanding augmentation-based self-supervised repre sentation learning via rkhs approximation and regression. In International Conference on Learning Repre- sentations, 2024. 1.5, 5.1, 5.2

2024
[168]

Spectrally transformed kernel regression

Runtian Zhai, Rattana Pukdee, Roger Jin, Maria Florin a Balcan, and Pradeep Ku- mar Ravikumar. Spectrally transformed kernel regression. In The T welfth Interna- tional Conference on Learning Representations , 2024. 1.5, 5.3

2024
[169]

Dauphin, and Da vid Lopez-Paz

Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and Da vid Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018. 1.2, 1.5

2018
[170]

Learning with local and global consistency

Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason We ston, and Bernhard Schölkopf. Learning with local and global consistency. Advances in neural informa- tion processing systems, 16, 2003. 5.3, 5.5

2003
[171]

Learning from labe led and unlabeled data with label propagation

Xiaojin Zhu and Zoubin Ghahramani. Learning from labe led and unlabeled data with label propagation. In CMU CALD tech report CMU-CALD-02-107 , 2002. 5.3 152

2002

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Gpt- 4 technical report

Josh Achiam, Steven Adler, Sandhini Agarwal, et al. Gpt- 4 technical report. Ope- nAI Blog, 2023. 1

2023

[2] [2]

Emergence of inv ariance and disentangle- ment in deep representations

Alessandro Achille and Stefano Soatto. Emergence of inv ariance and disentangle- ment in deep representations. Journal of Machine Learning Research , 19(50):1–34,

[3] [3]

\alpha-req : Assessing representation quality in self-sup ervised learning by mea- suring eigenspectrum decay

Kumar K Agrawal, Arnab Kumar Mondal, Arna Ghosh, and Blak e Richards. \alpha-req : Assessing representation quality in self-sup ervised learning by mea- suring eigenspectrum decay. In S. Koyejo, S. Mohamed, A. Aga rwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, vol- ume 35, pages 17626–17638. Curran Associ...

2022

[4] [4]

S carf: Self-supervised contrastive learning using random feature corruption

Dara Bahri, Heinrich Jiang, Yi Tay , and Donald Metzler. S carf: Self-supervised contrastive learning using random feature corruption. In International Conference on Learning Representations, 2022. (iii)

2022

[5] [5]

Contrastive and non -contrastive self- supervised learning recover global and local spectral embe dding methods

Randall Balestriero and Yann LeCun. Contrastive and non -contrastive self- supervised learning recover global and local spectral embe dding methods. Ad- vances in Neural Information Processing Systems , 35:26671–26685, 2022. 1.5

2022

[6] [6]

VICReg: Varia nce-invariance- covariance regularization for self-supervised learning

Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Varia nce-invariance- covariance regularization for self-supervised learning. In International Conference on Learning Representations, 2022. 2.1, 2.1, 2.2

2022

[7] [7]

Generalized discrimin ant analysis using a ker- nel approach

Gaston Baudat and Fatiha Anouar. Generalized discrimin ant analysis using a ker- nel approach. Neural computation, 12(10):2385–2404, 2000. 3.2

2000

[8] [8]

Laplacian eigenmaps f or dimensionality re- duction and data representation

Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps f or dimensionality re- duction and data representation. Neural computation, 15(6):1373–1396, 2003. 1.2, 1.5

2003