Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws

Fanjiang Ye; Fengguang Song; Liangliang Cao; Ming Lin; My T. Thai; Tianbao Yang; Xiyuan Wei

REVIEW 3 major objections 5 minor 37 references

Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Model steering with a reference model provably tightens generalization bounds and cuts sample complexity.

desk verdict Strong empirical case for reference-model steering, but the main theorem is missing a factor of two and the load-bearing variance assumption is not proven. read the letter →

arxiv 2505.06699 v3 pith:EWX7F7ZX submitted 2025-05-10 cs.LG cs.AIcs.CVstat.ML

Xiyuan Wei , Ming Lin , Fanjiang Ye , Fengguang Song , Liangliang Cao , My T. Thai , Tianbao Yang This is my paper · ORCID

classification cs.LGcs.AIcs.CVstat.ML

keywords modelsteeringRHOlossdistributionallyrobustoptimizationgeneralizationboundssamplecomplexitycontrastivelearningCLIPscalinglaws

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Model steering uses a trained reference model to guide a target model's training by replacing the ordinary loss $\ell(\theta,z)$ with the shifted loss $\ell(\theta,z)-\ell(\theta_{\mathrm{ref}},z)$, so samples the reference already handles well receive less weight. The paper formalizes this as DRRho risk minimization, a distributionally robust objective over the shifted loss, and proves a generalization bound whose variance term is $\mathrm{Var}(\ell(\theta^*,\cdot)-\ell(\theta_{\mathrm{ref}},\cdot))$ instead of $\mathrm{Var}(\ell(\theta^*,\cdot))$. Because a well-trained reference makes the shifted loss close to constant, this variance can be much smaller, yielding tighter excess-risk bounds and, in Corollary 4.3, an $O(\sqrt{m})$ sample complexity for matching a reference model trained on $m$ samples. The paper then instantiates the framework as DRRho-CLIP for contrastive language-image pretraining, reporting that it matches standard CLIP training with half the data and exhibits a better scaling law.

What carries the argument

The DRRho risk $F(\theta)=\sup_{p\in\Delta,\,D_{\phi}(p,1/n)\le \rho/n}\sum_i p_i(\ell(\theta,z_i)-\ell(\theta_{\mathrm{ref}},z_i))$, built on the shifted RHO loss $\ell(\theta,\cdot)-\ell(\theta_{\mathrm{ref}},\cdot)$, is the central object. It imports the variance-based generalization guarantees of distributionally robust optimization into the function class $\mathcal{F}_r=\{\ell(\theta,\cdot)-\ell(\theta_{\mathrm{ref}},\cdot)\}$, so the bound's variance term is computed on the difference rather than the raw loss; different divergences recover top-$k$ selection (CVaR) and exponential reweighting (KL).

What would settle it

Measure $\mathrm{Var}(\ell(\theta^*,\cdot)-\ell(\theta_{\mathrm{ref}},\cdot))$ on held-out data for a reference model trained on a shifted distribution; if it is not below $\mathrm{Var}(\ell(\theta^*,\cdot))$, Theorem 4.1's bound is no tighter than DRO's and the argument loses its quantitative force.

Watch

Extended reading notes

Core claim

The paper's central claim is that minimizing the DRRho risk, $F(\theta)=\sup_{p: D_{\phi}(p,1/n)\le \rho/n}\sum_i p_i(\ell(\theta,z_i)-\ell(\theta_{\mathrm{ref}},z_i))$, gives a risk bound $R(\tilde{\theta}^*)\le \inf_{\theta}\left[R(\theta)+\sqrt{2\rho/n\,\mathrm{Var}(\ell(\theta,\cdot)-\ell(\theta_{\mathrm{ref}},\cdot))}\right]+C_2/n$. This replaces the DRO variance term $\mathrm{Var}(\ell(\theta^*,\cdot))$ with the variance of the shifted loss at the optimum. Under the assumption that a sufficiently trained reference model has losses distributed like those of the target optimum, the shifted-loss variance is much smaller than the original variance, so DRRho risk minimization achieves a lower excess-risk bound than DRO or ERM. Corollary 4.3 makes the data-efficiency claim quantitative: if the reference model is trained by ERM on $m$ samples, DRRho needs only $n=O(\sqrt{m})$ samples to reach the same excess-risk level, compared with $O(m)$ for ERM alone.

Load-bearing premise

The load-bearing premise is that a sufficiently trained reference model has losses $\ell(\theta_{\mathrm{ref}},\cdot)$ distributed like the target optimum's $\ell(\theta^*,\cdot)$, so the variance of their difference is much smaller than the variance of $\ell(\theta^*,\cdot)$ alone.

Editorial extensions

If this is right

DRRho risk minimization supplies the first generalization theory for model steering, identifying the benefit of a reference model as a reduction in the variance term of the DRO bound.
Corollary 4.3 implies that DRRho needs only $O(\sqrt{m})$ samples to match a reference model trained on $m$ samples, so an existing public model acts as a data-efficiency multiplier.
The framework turns existing RHO-loss heuristics into principled special cases: CVaR divergence gives whole-dataset top-$k$ selection, and KL divergence gives exponential reweighting, both as the DRRho objective.
Applied to CLIP, DRRho-CLIP matches baseline CLIP training with 50% less data, outperforms existing heuristic data-selection methods on ImageNet-1K accuracy, and combines with knowledge distillation to exceed both.
Measured scaling exponents are smaller for DRRho-CLIP (roughly $\beta \approx 0.116$) than for standard CLIP training (roughly $\beta \approx 0.127$), so the advantage is not a fixed offset but grows as compute increases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the variance-reduction assumption holds across domains, the same steering recipe should transfer to any pretraining regime with an accessible reference model, such as autoregressive language models, where the shifted loss would be the per-token cross-entropy difference.
The bound suggests viewing the reference model as a control variate; this raises the testable possibility that an ensemble of references, or a reference trained on the same distribution as the target, would shrink the shifted-loss variance further than any single reference.
The $O(\sqrt{m})$ sample-complexity result implies an exchange rate between data and an existing model: a reference trained on $m$ samples could substitute for roughly $m-\sqrt{m}$ additional target samples, a prediction that a controlled data-ablation study could quantify.
The paper's empirical check of the variance assumption is limited to one 200K-subset measurement, so whether the improvement holds for weaker or out-of-distribution reference models is an open question that a broader variance audit would settle.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read the paper. The thing to know: it is worth engaging, but the main theorem overclaims as written. The contribution is real—first explicit DRO-based generalization bound for RHO-loss steering, plus DRRho-CLIP with released code and extensive experiments. The empirical work is the strongest part: on DFN-192M, DRRho-CLIP beats OpenCLIP and FastCLIP, matches a 400M-data reference while using 50% of the data, and outperforms JEST; the distillation combination also looks solid. These are practical, reproducible claims and they deserve a serious referee.

Now the soft spots. Theorem 4.1 states R(theta_tilde*) ≤ inf_theta [R(theta) + sqrt(2ρ/n Var(ℓ−ℓ_ref))] + C2/n. The proof in Appendix A applies Theorem A.3 with range 2M, but Theorem A.3's square-root term is 2·sqrt(2ρ/n Var(f)), not sqrt(...). The factor of two disappears without explanation. That is not cosmetic: with the correct constant, the variance reduction must beat a doubled C2/n term, and a measured ratio around 0.6 no longer guarantees an improvement over plain DRO.

The load-bearing premise is also under-verified. The remark after Corollary 4.2 assumes Var(ℓ(θ*,·)−ℓ(θ_ref,·)) is much smaller than Var(ℓ(θ*,·)). The direct evidence is one in-sample measurement on a 200K subset, comparing a FastCLIP-trained model with a DRRho-CLIP-trained model—so the two variances are not evaluated at the same θ*. Table 2 is more worrying: with a weak ViT-B/32 reference and fixed temperature, DRRho-CLIP underperforms FastCLIP (37.65 vs 43.49 ImageNet top-1). So the advertised improvement is conditional on a premise that can fail.

Also, the theory is for χ²-divergence while DRRho-CLIP optimizes a KL-regularized objective with fixed τ; the connection is asserted, not proved. Corollary 4.3's n=O(√m) sample-complexity claim also glosses over the C2/n term and log factors. These are less serious than the constant issue, but should be cleaned up.

Who this is for: anyone working on CLIP-style pretraining, data selection, or DRO-based learning. The experimental contribution will be valuable even after the theory is corrected. The citation pattern and related-work placement look appropriate—the paper builds honestly on Duchi–Namkoong, RHO, JEST, and SogCLR.

Recommendation: send to peer review, but expect major revision. The authors need to fix the theorem, re-derive the condition under which steering beats DRO, and provide better variance evidence at the same θ* and out of sample.

Referee Report

3 major / 5 minor

Summary. The paper formalizes "model steering," in which a pretrained reference model is used to guide training of a target model through the RHO loss ℓ(θ,z) − ℓ(θ_ref,z). It proposes minimizing a distributionally robust version of this loss, termed DRRho risk, and derives generalization bounds (Theorem 4.1) with corollaries that claim improved excess risk relative to DRO/ERM when the variance of the RHO loss is smaller than that of the original loss, and reduced sample complexity n = O(√m) to match a reference model trained on m samples. The paper then instantiates the framework as DRRho-CLIP, a contrastive pretraining method with a reference model, and reports experiments on CC12M, DFN-12M, and DFN-192M showing improved data efficiency, gains over heuristic baselines such as JEST, and a better scaling law than OpenCLIP.

Significance. If the results hold, this is a valuable first theoretical treatment of model steering with RHO losses, connecting them to DRO and giving a principled justification for existing heuristics. The paper also contributes a practical method, DRRho-CLIP, with substantial empirical results, released code, and scaling-law evidence. The theoretical framework is genuinely novel and the proofs are systematically written, but the central claim of improvement over DRO rests on an explicit variance-reduction assumption (Remark after Corollary 4.2) that is neither proved nor convincingly measured, and there is a coefficient inconsistency in Theorem 4.1 compared to the cited DRO bound. The empirical results are strong enough that the paper remains valuable even with a more cautious theoretical framing, but the theory as stated overclaims.

major comments (3)

[Theorem 4.1 and Appendix A] The bound in Theorem 4.1 has the variance term √(2ρ/n · Var(ℓ(θ,·)−ℓ(θ_ref,·))), but applying the second part of Theorem A.3 with f = ℓ(θ,·)−ℓ(θ_ref,·) and range 2M yields 2√(2ρ/n · Var(ℓ(θ,·)−ℓ(θ_ref,·))) plus the doubled constant (50ρ/3+4)M/n. The factor 2 is dropped without comment. This is load-bearing because Corollary 4.2 and the comparison in the Remark after it use the stated coefficient; with the factor 2 restored, a variance ratio of about 0.6 reduces the variance term by only about 22% while the constant term roughly doubles from (25ρ/3+2)M to (50ρ/3+4)M, so the claimed improvement over DRO is much less clear. Please correct the theorem or justify the smaller coefficient explicitly.
[Remark after Corollary 4.2 and Section 6.1] The central premise that Var(ℓ(θ*,·)−ℓ(θ_ref,·)) is 'much smaller' than Var(ℓ(θ*,·)) is stated as a reasonable assumption but is not proved. The empirical support in Section 6.1 is confounded: the two variances are computed on a 200K in-sample subset, and the original loss uses a FastCLIP-trained model while the RHO loss uses a DRRho-CLIP-trained model, so the comparison is not at the same θ*. The measured ratio (4.49 vs 7.26 for image loss, i.e., about 0.62) is not 'much smaller' once the coefficient issue in Theorem 4.1 is accounted for. Moreover, Table 2 shows a case with a weak reference (ViT-B/32 (DFN-12M)) where fixed-temperature DRRho-CLIP underperforms FastCLIP (37.65 vs 43.49 ImageNet top-1), contradicting the unconditional claim that model steering improves generalization. Please state the variance-reduction condition as an explicit assumption, verify it at comparable model parameters and on a held-out set, and temper the claims accordingly.
[Corollary 4.3 and Section 6.1] Corollary 4.3 requires θ_ref ∈ Θ and the remark then uses R(θ_ref) − R(θ*) = O(1/√m) for a reference trained on m samples from the same distribution. The experiments, however, frequently use reference models with different architectures and different pretraining data (e.g., ViT-B/32 (WIT-400M) as reference for a ViT-B/16 target in Figure 3b, or ViT-L/14 (DFN-2B) in Table 1). In these settings the condition θ_ref ∈ Θ is not satisfied, and the claimed n = O(√m) data-efficiency result does not theoretically apply. The empirical data-efficiency findings remain interesting, but the theoretical statement should be restricted to the same-architecture/same-distribution setting, or an appropriate extension should be provided.

minor comments (5)

[Section 1, Contributions] There is a typo: 'th RHO loss' should be 'the RHO loss'.
[Appendix A, Theorem A.3] In Theorem A.3, 'VC-dimension of Θ' should read 'VC-dimension of the function class F'; Θ is a parameter space, not the function class.
[Appendix A, Proof of Theorem 4.1] The notation 'Θo' appears in the infimum in the proof of Theorem 4.1 and Corollary 4.3; it should be 'Θ'.
[Section 6.3] The scaling-law exponents in Figure 2 (0.116 vs 0.127) are close; please report uncertainty estimates or the number of fitted points to support the claim that DRRho-CLIP has a better scaling law.
[Section 5 and Appendix C] Equation (16) uses \hat{τ} without defining it; please state that it is the reference model's temperature.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the DRRho bound is an external DRO theorem applied to a shifted loss; the variance-reduction premise is an assumption, not a definitional equivalence.

full rationale

The central derivation is self-contained rather than circular. Theorem 4.1 is proved in Appendix A by substituting f(·) := ℓ(θ,·)−ℓ(θ_ref,·) into Theorem A.3, which is the paper's restatement of the external Duchi and Namkoong DRO bound. This is an application of an existing, parameter-free mathematical result to a new loss; it is not an input that is renamed as a prediction. The improvement claim over DRO is explicitly conditional: the remark after Corollary 4.2 says 'It is reasonable to assume that the reference model θ_ref is sufficiently trained such that ℓ(θ_ref,·) has a similar distribution to ℓ(θ*,·); hence we expect that Var(ℓ(θ*,·)−ℓ(θ_ref,·)) would be much smaller than Var(ℓ(θ*,·)).' That is an empirical premise, not a theorem, and an unverified or confounded premise is a correctness or empirical-validity concern rather than circularity. Corollary 4.3 does use the construction θ = θ_ref to make Var(ℓ(θ_ref,·)−ℓ(θ_ref,·)) vanish, but this is a legitimate feasible-point argument showing the reference model is a candidate solution; the resulting bound R(θ_tilde*)−R(θ*) ≤ R(θ_ref)−R(θ*) + C2/n and the remark n = O(√m) follow by arithmetic, and the corollary only promises to match the reference model, not to beat it. Self-citations to Qiu et al. (2023), Yuan et al. (2022), and Wei et al. (2024) supply the contrastive-DRO duality and optimization machinery; these are published, independently checkable results, not uniqueness theorems or unverified author-specific premises. The empirical variance measurement in Section 6.1 compares models trained by different methods, so it is an imperfect test of the variance-reduction premise, but that affects evidential weight, not the logical direction of the derivation. One internal inconsistency noted: Theorem 4.1 drops the factor 2 appearing in Theorem A.3's variance term, which would affect the strength of the claimed bound; this is a correctness risk, not a circular step.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on Duchi and Namkoong's DRO theorem (external), bounded-loss and i.i.d. assumptions, finite VC dimension of the shifted function class, and one ad hoc assumption: the RHO loss has lower variance than the original loss. The algorithm also relies on SogCLR's convergence guarantee from prior work by the same group. No new entities are introduced.

free parameters (3)

rho (DRO radius) = not fitted; hyperparameter constrained by rho >= 9t in theory
Appears in the DRRho risk (Eq. 2) and in the generalization bound (Theorem 4.1); the bound's variance term and C2 grow with rho, so the claimed improvement depends on the choice of rho.
tau (temperature) = 0.01 in most experiments; learnable in some (Appendix C.3)
In the KL-regularized DRRho objective (Eq. 6) and DRRho-CLIP losses (Eqs. 8, 9), tau controls the sharpness of the softmax weights; it is a tuned or learned hyperparameter.
lambda (distillation weight) = 0.0, 0.25, 0.4, or 0.75 depending on setting
Used in MobileCLIP-style distillation combinations (Eq. 12); the paper sets it per dataset and reference model strength.

assumptions (5)

domain assumption Loss bounded in [M0, M1]
Required by Theorem 3.1 and Theorem 4.1 to apply Duchi and Namkoong's DRO bound; standard but nontrivial for deep learning losses.
domain assumption Training data drawn i.i.d. from P
Standard generalization setup; the reference model's training distribution is ignored in the theory, but theta_ref is treated as a fixed function.
domain assumption Finite VC dimension of the shifted function class F_r = {ell(theta,.)-ell(theta_ref,.)}
Needed for the covering-number bound (Corollary A.2); the paper abuses the notation d_v = VC(F_r) without proof.
domain assumption Reference model belongs to the same hypothesis class Theta (Corollary 4.3)
Needed to plug theta=theta_ref into the inf and zero the variance term; not true in general when the reference is a different architecture.
ad hoc to paper Variance reduction: Var(ell(theta*,.)-ell(theta_ref,.)) is smaller than Var(ell(theta*,.))
The entire improved generalization claim in the Remark after Corollary 4.2 rests on this. It is stated as reasonable to assume, not proven, and verified in only one experiment (Section 6.1).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws." pith.science (2026). https://pith.science/paper/EWX7F7ZX

@misc{pith2026250506699,
  author       = {Pith},
  title        = {Pith review of: Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EWX7F7ZX}},
  note         = {Machine review of arXiv:2505.06699}
}

abstract

This paper formalizes an emerging learning paradigm that uses a trained model as a reference to guide and enhance the training of a target model through strategic data selection or weighting, named $\textbf{model steering}$. While ad-hoc methods have been used in various contexts, including the training of large foundation models, its underlying principles remain insufficiently understood, leading to sub-optimal performance. In this work, we propose a theory-driven framework for model steering called $\textbf{DRRho risk minimization}$, which is rooted in Distributionally Robust Optimization (DRO). Through a generalization analysis, we provide theoretical insights into why this approach improves generalization and data efficiency compared to training without a reference model. To the best of our knowledge, this is the first time such theoretical insights are provided for the new learning paradigm, which significantly enhance our understanding and practice of model steering. Building on these insights and the connection between contrastive learning and DRO, we introduce a novel method for Contrastive Language-Image Pretraining (CLIP) with a reference model, termed DRRho-CLIP. Extensive experiments validate the theoretical insights, reveal a superior scaling law compared to CLIP without a reference model, and demonstrate its strength over existing heuristic approaches.

Figures

Figures reproduced from arXiv: 2505.06699 by the authors.

**Figure 2.** Scaling performance of OpenCLIP (Cherti et al., 2023) and the proposed DRRho-CLIP, which uses the OpenAI CLIP model (Radford et al., 2021) as the reference model. We conduct experiments of the two methods under different settings to fit scaling laws, as shown in the bottom left corner (c.f. Section 6 for more detail). Our framework not only provides a theoretical foundation for existing heuristic approaches to data… view at source ↗

**Figure 3.** Performance curves of FastCLIP and DRRho-CLIP with different target and reference models (with [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Zero-shot Top 1 Accuracy on ImageNet-1K of [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5: ImageNet Top 1 accuracy curves of DRRho-CLIP with fixed and learnable temperature on different target models [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]

**Figure 6.** Figure 6: Zero-shot Top 1 Accuracy on ImageNet-1K of different models. Reference model is a ViT-B/16 (left figure) or [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 25 canonical work pages

[1]

L., and Paul, M

Ankner, Z., Blakeney, C., Sreenivasan, K., Marion, M., Leavitt, M. L., and Paul, M. Perplexed by perplexity: Perplexity-based data pruning with small reference models. arXiv preprint arXiv:2405.20541, 2024

arXiv 2024
[2]

Theory of classification: A survey of some recent advances

Boucheron, S., Bousquet, O., and Lugosi, G. Theory of classification: A survey of some recent advances. ESAIM: probability and statistics, 9: 0 323--375, 2005

work page 2005
[3]

Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

Changpinyo, S., Sharma, P., Ding, N., and Soricut, R. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 3558--3568, June 2021

work page 2021
[4]

Net2net: Accelerating learning via knowledge transfer

Chen, T., Goodfellow, I., and Shlens, J. Net2net: Accelerating learning via knowledge transfer. In The Fourth International Conference on Learning Representations, 2016

work page 2016
[5]

Reproducible scaling laws for contrastive language-image learning

Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, J. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 2818--2829, June 2023

work page 2023
[6]

Decaf: A deep convolutional activation feature for generic visual recognition

Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., and Darrell, T. Decaf: A deep convolutional activation feature for generic visual recognition. CoRR, abs/1310.1531, 2013. URL http://arxiv.org/abs/1310.1531

arXiv 2013
[7]

Stacking your transformers: A closer look at model growth for efficient LLM pre-training

Du, W., Luo, T., Qiu, Z., Huang, Z., Shen, Y., Cheng, R., Guo, Y., and Fu, J. Stacking your transformers: A closer look at model growth for efficient LLM pre-training. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=FXJDcriMYH

work page 2024
[8]

The llama 3 herd of models

Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

arXiv 2024

Show all 37 references

[9]

and Namkoong, H

Duchi, J. and Namkoong, H. Variance-based regularization with convex objectives. arXiv preprint arXiv:1610.02581, 2016

2016 arXiv
[10]

C., Glynn, P

Duchi, J. C., Glynn, P. W., and Namkoong, H. Statistics of robust optimization: A generalized empirical likelihood approach. Mathematics of Operations Research, 46 0 (3): 0 946--969, 2021. doi:10.1287/moor.2020.1085. URL https://doi.org/10.1287/moor.2020.1085

2021
[11]

Evans, T., Parthasarathy, N., Merzic, H., and Henaff, O. J. Data curation via joint example selection further accelerates multimodal learning. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024 a . URL https://openreview...

2024
[12]

Evans, T., Pathak, S., Merzic, H., Schwarz, J., Tanno, R., and H\' e naff, O. J. Bad students make great teachers: Active learning accelerates large-scale visual understanding. In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024...

2024 doi
[13]

Evans, T., Pathak, S., Merzic, H., Schwarz, J., Tanno, R., and H \'e naff, O. J. Bad students make great teachers: Active learning accelerates large-scale visual understanding. In Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., and Varol, G. (eds.), Computer ...

2024
[14]

M., Jain, A., Schmidt, L., Toshev, A

Fang, A., Jose, A. M., Jain, A., Schmidt, L., Toshev, A. T., and Shankar, V. Data filtering networks. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=KAk6ngZ09F

2024
[15]

Gadre, S. Y., Ilharco, G., Fang, A., Hayase, J., Smyrnis, G., Nguyen, T., Marten, R., Wortsman, M., Ghosh, D., Zhang, J., Orgad, E., Entezari, R., Daras, G., Pratt, S., Ramanujan, V., Bitton, Y., Marathe, K., Mussmann, S., Vencu, R., Cherti, M., Krishna, R., Koh, P. W. W., Sau...

2023
[16]

Distilling the knowledge in a neural network

Hinton, G. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

2015 arXiv
[17]

Non-smooth weakly-convex finite-sum coupled compositional optimization

Hu, Q., Zhu, D., and Yang, T. Non-smooth weakly-convex finite-sum coupled compositional optimization. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informat...

2023
[18]

Information theory and statistics

Kullback, S. Information theory and statistics. Courier Corporation, 1997

1997
[19]

C., and Sidford, A

Levy, D., Carmon, Y., Duchi, J. C., and Sidford, A. Large-scale methods for distributionally robust optimization. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.\ 8847--8860. Curran ...

2020
[20]

Not all tokens are what you need for pretraining

Lin, Z., Gou, Z., Gong, Y., Liu, X., yelong shen, Xu, R., Lin, C., Yang, Y., Jiao, J., Duan, N., and Chen, W. Not all tokens are what you need for pretraining. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/for...

2024
[21]

When less is more: Investigating data pruning for pretraining llms at scale

Marion, M., \"U st \"u n, A., Pozzobon, L., Wang, A., Fadaee, M., and Hooker, S. When less is more: Investigating data pruning for pretraining llms at scale. arXiv preprint arXiv:2309.04564, 2023

2023 arXiv
[22]

M., Razzak, M

Mindermann, S., Brauner, J. M., Razzak, M. T., Sharma, M., Kirsch, A., Xu, W., H \"o ltgen, B., Gomez, A. N., Morisot, A., Farquhar, S., and Gal, Y. Prioritized training on points that are learnable, worth learning, and not yet learnt. In Chaudhuri, K., Jegelka, S., Song, L., ...

2022
[23]

Stochastic constrained DRO with a complexity independent of sample size

Qi, Q., Lyu, J., Chan, K., Bai, E., and Yang, T. Stochastic constrained DRO with a complexity independent of sample size. Trans. Mach. Learn. Res., 2023, 2023 a . URL https://openreview.net/forum?id=VpaXrBFYZ9

2023
[24]

Attentional-biased stochastic gradient descent

Qi, Q., Xu, Y., Yin, W., Jin, R., and Yang, T. Attentional-biased stochastic gradient descent. Trans. Mach. Learn. Res., 2023, 2023 b . URL https://openreview.net/forum?id=B0WYWvVA2r

2023
[25]

Knowledge inheritance for pre-trained language models

Qin, Y., Lin, Y., Yi, J., Zhang, J., Han, X., Zhang, Z., Su, Y., Liu, Z., Li, P., Sun, M., and Zhou, J. Knowledge inheritance for pre-trained language models. In Carpuat, M., de Marneffe, M.-C., and Meza Ruiz, I. V. (eds.), Proceedings of the 2022 Conference of the North Ameri...

2022 doi
[26]

Not all semantics are created equal: Contrastive self-supervised learning with automatic temperature individualization

Qiu, Z.-H., Hu, Q., Yuan, Z., Zhou, D., Zhang, L., and Yang, T. Not all semantics are created equal: Contrastive self-supervised learning with automatic temperature individualization. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), P...

2023
[27]

W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In Meila, M. and Zhang, T. (eds.), Proceedings of the 3...

2021
[28]

Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021

2021 arXiv
[29]

Laion-5b: An open large-scale dataset for training next generation image-text models

Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., Schramowski, P., Kundurthy, S., Crowson, K., Schmidt, L., Kaczmarczyk, R., and Jitsev, J. Laion-5b: An open large-scale dataset for training next gen...

2022
[30]

van der Vaart, A. W. and Wellner, J. A. Weak Convergence, pp.\ 16--28. Springer New York, New York, NY, 1996. ISBN 978-1-4757-2545-2. doi:10.1007/978-1-4757-2545-2_3. URL https://doi.org/10.1007/978-1-4757-2545-2_3

1996 doi
[31]

Vasu, P. K. A., Pouransari, H., Faghri, F., Vemulapalli, R., and Tuzel, O. Mobileclip: Fast image-text models through multi-modal reinforced training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 15963--15974, June 2024

2024
[32]

and Yang, T

Wang, B. and Yang, T. ALEXR: optimal single-loop algorithms for convex finite-sum coupled compositional stochastic optimization. CoRR, abs/2312.02277, 2023. doi:10.48550/ARXIV.2312.02277. URL https://doi.org/10.48550/arXiv.2312.02277

2023 doi arXiv
[33]

T., Greengard, P., Karlinsky, L., Feris, R., Cox, D

Wang, P., Panda, R., Hennigen, L. T., Greengard, P., Karlinsky, L., Feris, R., Cox, D. D., Wang, Z., and Kim, Y. Learning to grow pretrained models for efficient transformer training. In The Eleventh International Conference on Learning Representations, 2023. URL https://openr...

2023
[34]

Fastclip: A suite of optimization techniques to accelerate clip training with limited resources

Wei, X., Ye, F., Yonay, O., Chen, X., Sun, B., Tao, D., and Yang, T. Fastclip: A suite of optimization techniques to accelerate clip training with limited resources. arXiv preprint arXiv:2407.01445, 2024

2024 arXiv
[35]

M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y., Liang, P

Xie, S. M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y., Liang, P. S., Le, Q. V., Ma, T., and Yu, A. W. Doremi: Optimizing data mixtures speeds up language model pretraining. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neura...

2023
[36]

Provable stochastic optimization for global contrastive learning: Small batch does not harm performance

Yuan, Z., Wu, Y., Qiu, Z.-H., Du, X., Zhang, L., Zhou, D., and Yang, T. Provable stochastic optimization for global contrastive learning: Small batch does not harm performance. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings...

2022
[37]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

L., and Paul, M

Ankner, Z., Blakeney, C., Sreenivasan, K., Marion, M., Leavitt, M. L., and Paul, M. Perplexed by perplexity: Perplexity-based data pruning with small reference models. arXiv preprint arXiv:2405.20541, 2024

arXiv 2024

[2] [2]

Theory of classification: A survey of some recent advances

Boucheron, S., Bousquet, O., and Lugosi, G. Theory of classification: A survey of some recent advances. ESAIM: probability and statistics, 9: 0 323--375, 2005

work page 2005

[3] [3]

Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

Changpinyo, S., Sharma, P., Ding, N., and Soricut, R. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 3558--3568, June 2021

work page 2021

[4] [4]

Net2net: Accelerating learning via knowledge transfer

Chen, T., Goodfellow, I., and Shlens, J. Net2net: Accelerating learning via knowledge transfer. In The Fourth International Conference on Learning Representations, 2016

work page 2016

[5] [5]

Reproducible scaling laws for contrastive language-image learning

Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, J. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 2818--2829, June 2023

work page 2023

[6] [6]

Decaf: A deep convolutional activation feature for generic visual recognition

Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., and Darrell, T. Decaf: A deep convolutional activation feature for generic visual recognition. CoRR, abs/1310.1531, 2013. URL http://arxiv.org/abs/1310.1531

arXiv 2013

[7] [7]

Stacking your transformers: A closer look at model growth for efficient LLM pre-training

Du, W., Luo, T., Qiu, Z., Huang, Z., Shen, Y., Cheng, R., Guo, Y., and Fu, J. Stacking your transformers: A closer look at model growth for efficient LLM pre-training. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=FXJDcriMYH

work page 2024

[8] [8]

The llama 3 herd of models

Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

arXiv 2024