{"id":"3ebc93b3-d029-418a-8145-bf94af0219a0","arxiv_id":"2505.06699","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using a pretrained reference model inside a distributionally robust risk objective can improve generalization bounds and yields a CLIP variant that matches baseline performance with half the data.","lead":"This paper shows that training a new model while using an existing model as a reference, through a mathematically grounded reweighting of data, can improve accuracy and reduce the amount of data needed. The authors derive generalization bounds for this 'model steering' approach and demonstrate a faster scaling law on image-text pretraining.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The improvement over DRO rests on an unverified variance-reduction premise; the Section 6.1 measurement is confounded and, under the correct Duchi–Namkoong constant, a ~0.6 variance ratio may not beat the doubled C2 term.","rationale":"I agree with the reader that the variance-reduction assumption is the weakest link in the central argument. Theorem 4.1 is a valid-looking DRO-style bound, but its advertised advantage over standard DRO depends entirely on the unproven claim that Var(ℓ(θ*,·)−ℓ(θref,·)) is much smaller than Var(ℓ(θ*,·)). The Section 6.1 verification is confounded because the original loss is measured with a FastCLIP-trained model and the RHO loss with a DRRho-CLIP-trained model, so the comparison does not instantiate the theorem's quantity at a common θ*. The reported ratio of about 0.6 is also less convincing once the Duchi–Namkoong constant is applied correctly: the paper's own Theorem A.1 has 2√(2ρ/n Var(f)), while the displayed Theorem 4.1 drops the leading 2. Re-deriving Theorem 4.1 from Theorem A.1 would put 2√(2ρ/n Var(diff)) in the bound and double the additive C2/n term, making the observed variance ratio a much weaker basis for a 'much smaller variance' claim. I also note the KL-versus-χ² gap between the CLIP algorithm and the theorem, and the scaling-law fitting that selects the lowest error per compute budget; these are real secondary concerns, but the variance premise is the mathematical hinge. Since the reader already reached CONDITIONAL based on this same assumption, my read does not move the verdict.","tokens_in":29255,"tokens_out":16204,"duration_ms":152003,"concrete_test":"Run a controlled variance test: train ViT-B/16 on DFN-12M with DRRho-CLIP using (a) OpenAI ViT-B/32 (WIT-400M), (b) a weak ViT-B/32 trained on DFN-12M, and (c) a random-initialized reference. On a held-out set, evaluate for the same resulting θ~* both Var(ℓ(θ~*,z)) and Var(ℓ(θ~*,z)−ℓ(θref,z)) at the end of training and at intermediate checkpoints, across 3 seeds. If the ratio is not consistently below the threshold implied by the corrected Theorem 4.1 (small enough to overcome the doubled C2/n term), then the claimed improvement over DRO is unsupported in the regime where model steering is expected to help.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is the remark after Corollary 4.2: Var(ℓ(θ*,·)−ℓ(θref,·)) must be much smaller than Var(ℓ(θ*,·)). Without this, Theorem 4.1 gives no improvement over DRO, and Corollary 4.3 only promises to match the reference model, not to beat it. The paper's direct evidence in Section 6.1 is not decisive: the two variances are computed on a 200K training subset, and the original loss comes from a FastCLIP-trained model while the RHO loss comes from a DRRho-CLIP-trained model, so the quantities are not evaluated at the same θ* and are in-sample. Moreover, the measured ratio (~0.6) is not 'much smaller' once Theorem 4.1 is re-derived with the constant in the paper's own Theorem A.1, which uses 2√(2ρ/n Var(f)); applying that to f=ℓ−ℓref would give 2√(2ρ/n Var(diff)) with a doubled C2/n term. A 0.6 variance ratio then shrinks the variance term by only about 22%, and the larger C2 can offset that gain. Table 2 in Appendix C.3 also shows the premise can fail: with a weak ViT-B/32 (DFN-12M) reference, fixed-temperature DRRho-CLIP underperforms FastCLIP (37.65 vs 43.49 ImageNet top-1). Thus the advertised improvement is conditional on a premise that is neither proved nor convincingly measured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper formalizes \"model steering,\" in which a pretrained reference model is used to guide training of a target model through the RHO loss ℓ(θ,z) − ℓ(θ_ref,z). It proposes minimizing a distributionally robust version of this loss, termed DRRho risk, and derives generalization bounds (Theorem 4.1) with corollaries that claim improved excess risk relative to DRO/ERM when the variance of the RHO loss is smaller than that of the original loss, and reduced sample complexity n = O(√m) to match a reference model trained on m samples. The paper then instantiates the framework as DRRho-CLIP, a contrastive pretraining method with a reference model, and reports experiments on CC12M, DFN-12M, and DFN-192M showing improved data efficiency, gains over heuristic baselines such as JEST, and a better scaling law than OpenCLIP.","tokens_in":29637,"tokens_out":8739,"duration_ms":83471,"significance":"If the results hold, this is a valuable first theoretical treatment of model steering with RHO losses, connecting them to DRO and giving a principled justification for existing heuristics. The paper also contributes a practical method, DRRho-CLIP, with substantial empirical results, released code, and scaling-law evidence. The theoretical framework is genuinely novel and the proofs are systematically written, but the central claim of improvement over DRO rests on an explicit variance-reduction assumption (Remark after Corollary 4.2) that is neither proved nor convincingly measured, and there is a coefficient inconsistency in Theorem 4.1 compared to the cited DRO bound. The empirical results are strong enough that the paper remains valuable even with a more cautious theoretical framing, but the theory as stated overclaims.","major_comments":[{"comment":"The bound in Theorem 4.1 has the variance term √(2ρ/n · Var(ℓ(θ,·)−ℓ(θ_ref,·))), but applying the second part of Theorem A.3 with f = ℓ(θ,·)−ℓ(θ_ref,·) and range 2M yields 2√(2ρ/n · Var(ℓ(θ,·)−ℓ(θ_ref,·))) plus the doubled constant (50ρ/3+4)M/n. The factor 2 is dropped without comment. This is load-bearing because Corollary 4.2 and the comparison in the Remark after it use the stated coefficient; with the factor 2 restored, a variance ratio of about 0.6 reduces the variance term by only about 22% while the constant term roughly doubles from (25ρ/3+2)M to (50ρ/3+4)M, so the claimed improvement over DRO is much less clear. Please correct the theorem or justify the smaller coefficient explicitly.","section":"Theorem 4.1 and Appendix A"},{"comment":"The central premise that Var(ℓ(θ*,·)−ℓ(θ_ref,·)) is 'much smaller' than Var(ℓ(θ*,·)) is stated as a reasonable assumption but is not proved. The empirical support in Section 6.1 is confounded: the two variances are computed on a 200K in-sample subset, and the original loss uses a FastCLIP-trained model while the RHO loss uses a DRRho-CLIP-trained model, so the comparison is not at the same θ*. The measured ratio (4.49 vs 7.26 for image loss, i.e., about 0.62) is not 'much smaller' once the coefficient issue in Theorem 4.1 is accounted for. Moreover, Table 2 shows a case with a weak reference (ViT-B/32 (DFN-12M)) where fixed-temperature DRRho-CLIP underperforms FastCLIP (37.65 vs 43.49 ImageNet top-1), contradicting the unconditional claim that model steering improves generalization. Please state the variance-reduction condition as an explicit assumption, verify it at comparable model parameters and on a held-out set, and temper the claims accordingly.","section":"Remark after Corollary 4.2 and Section 6.1"},{"comment":"Corollary 4.3 requires θ_ref ∈ Θ and the remark then uses R(θ_ref) − R(θ*) = O(1/√m) for a reference trained on m samples from the same distribution. The experiments, however, frequently use reference models with different architectures and different pretraining data (e.g., ViT-B/32 (WIT-400M) as reference for a ViT-B/16 target in Figure 3b, or ViT-L/14 (DFN-2B) in Table 1). In these settings the condition θ_ref ∈ Θ is not satisfied, and the claimed n = O(√m) data-efficiency result does not theoretically apply. The empirical data-efficiency findings remain interesting, but the theoretical statement should be restricted to the same-architecture/same-distribution setting, or an appropriate extension should be provided.","section":"Corollary 4.3 and Section 6.1"}],"minor_comments":[{"comment":"There is a typo: 'th RHO loss' should be 'the RHO loss'.","section":"Section 1, Contributions"},{"comment":"In Theorem A.3, 'VC-dimension of Θ' should read 'VC-dimension of the function class F'; Θ is a parameter space, not the function class.","section":"Appendix A, Theorem A.3"},{"comment":"The notation 'Θo' appears in the infimum in the proof of Theorem 4.1 and Corollary 4.3; it should be 'Θ'.","section":"Appendix A, Proof of Theorem 4.1"},{"comment":"The scaling-law exponents in Figure 2 (0.116 vs 0.127) are close; please report uncertainty estimates or the number of fitted points to support the claim that DRRho-CLIP has a better scaling law.","section":"Section 6.3"},{"comment":"Equation (16) uses \\hat{τ} without defining it; please state that it is the reference model's temperature.","section":"Section 5 and Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and addresses a timely topic. The empirical work is substantial and likely to be of interest even if the theoretical claims are weakened. The main risks are the factor-2 inconsistency in Theorem 4.1 relative to the cited DRO bound and the unproven variance-reduction premise; both are fixable but require careful revision. I would not recommend rejection, as the core idea is sound and the experiments are extensive, but the current claims overstate what the theory establishes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the paper. The thing to know: it is worth engaging, but the main theorem overclaims as written. The contribution is real—first explicit DRO-based generalization bound for RHO-loss steering, plus DRRho-CLIP with released code and extensive experiments. The empirical work is the strongest part: on DFN-192M, DRRho-CLIP beats OpenCLIP and FastCLIP, matches a 400M-data reference while using 50% of the data, and outperforms JEST; the distillation combination also looks solid. These are practical, reproducible claims and they deserve a serious referee.\n\nNow the soft spots. Theorem 4.1 states R(theta_tilde*) ≤ inf_theta [R(theta) + sqrt(2ρ/n Var(ℓ−ℓ_ref))] + C2/n. The proof in Appendix A applies Theorem A.3 with range 2M, but Theorem A.3's square-root term is 2·sqrt(2ρ/n Var(f)), not sqrt(...). The factor of two disappears without explanation. That is not cosmetic: with the correct constant, the variance reduction must beat a doubled C2/n term, and a measured ratio around 0.6 no longer guarantees an improvement over plain DRO.\n\nThe load-bearing premise is also under-verified. The remark after Corollary 4.2 assumes Var(ℓ(θ*,·)−ℓ(θ_ref,·)) is much smaller than Var(ℓ(θ*,·)). The direct evidence is one in-sample measurement on a 200K subset, comparing a FastCLIP-trained model with a DRRho-CLIP-trained model—so the two variances are not evaluated at the same θ*. Table 2 is more worrying: with a weak ViT-B/32 reference and fixed temperature, DRRho-CLIP underperforms FastCLIP (37.65 vs 43.49 ImageNet top-1). So the advertised improvement is conditional on a premise that can fail.\n\nAlso, the theory is for χ²-divergence while DRRho-CLIP optimizes a KL-regularized objective with fixed τ; the connection is asserted, not proved. Corollary 4.3's n=O(√m) sample-complexity claim also glosses over the C2/n term and log factors. These are less serious than the constant issue, but should be cleaned up.\n\nWho this is for: anyone working on CLIP-style pretraining, data selection, or DRO-based learning. The experimental contribution will be valuable even after the theory is corrected. The citation pattern and related-work placement look appropriate—the paper builds honestly on Duchi–Namkoong, RHO, JEST, and SogCLR.\n\nRecommendation: send to peer review, but expect major revision. The authors need to fix the theorem, re-derive the condition under which steering beats DRO, and provide better variance evidence at the same θ* and out of sample.","headline":"Strong empirical case for reference-model steering, but the main theorem is missing a factor of two and the load-bearing variance assumption is not proven.","tokens_in":30149,"tokens_out":5271,"would_cite":false,"duration_ms":57017,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Model steering with a reference model provably tightens generalization bounds and cuts sample complexity.","keywords":["model steering","RHO loss","distributionally robust optimization","generalization bounds","sample complexity","contrastive learning","CLIP","scaling laws"],"falsifier":"Measure $\\mathrm{Var}(\\ell(\\theta^*,\\cdot)-\\ell(\\theta_{\\mathrm{ref}},\\cdot))$ on held-out data for a reference model trained on a shifted distribution; if it is not below $\\mathrm{Var}(\\ell(\\theta^*,\\cdot))$, Theorem 4.1's bound is no tighter than DRO's and the argument loses its quantitative force.","tokens_in":29081,"feed_emoji":"⚡","tokens_out":14470,"duration_ms":112862,"temperature":0.7,"pith_summary":"Model steering uses a trained reference model to guide a target model's training by replacing the ordinary loss $\\ell(\\theta,z)$ with the shifted loss $\\ell(\\theta,z)-\\ell(\\theta_{\\mathrm{ref}},z)$, so samples the reference already handles well receive less weight. The paper formalizes this as DRRho risk minimization, a distributionally robust objective over the shifted loss, and proves a generalization bound whose variance term is $\\mathrm{Var}(\\ell(\\theta^*,\\cdot)-\\ell(\\theta_{\\mathrm{ref}},\\cdot))$ instead of $\\mathrm{Var}(\\ell(\\theta^*,\\cdot))$. Because a well-trained reference makes the shifted loss close to constant, this variance can be much smaller, yielding tighter excess-risk bounds and, in Corollary 4.3, an $O(\\sqrt{m})$ sample complexity for matching a reference model trained on $m$ samples. The paper then instantiates the framework as DRRho-CLIP for contrastive language-image pretraining, reporting that it matches standard CLIP training with half the data and exhibits a better scaling law.","feed_headline":"One trick cuts CLIP data in half: subtract the reference model's loss","feed_subtitle":"New bounds show the shifted loss lowers variance, so half the data matches full-data CLIP training.","key_machinery":"The DRRho risk $F(\\theta)=\\sup_{p\\in\\Delta,\\,D_{\\phi}(p,1/n)\\le \\rho/n}\\sum_i p_i(\\ell(\\theta,z_i)-\\ell(\\theta_{\\mathrm{ref}},z_i))$, built on the shifted RHO loss $\\ell(\\theta,\\cdot)-\\ell(\\theta_{\\mathrm{ref}},\\cdot)$, is the central object. It imports the variance-based generalization guarantees of distributionally robust optimization into the function class $\\mathcal{F}_r=\\{\\ell(\\theta,\\cdot)-\\ell(\\theta_{\\mathrm{ref}},\\cdot)\\}$, so the bound's variance term is computed on the difference rather than the raw loss; different divergences recover top-$k$ selection (CVaR) and exponential reweighting (KL).","core_discovery":"The paper's central claim is that minimizing the DRRho risk, $F(\\theta)=\\sup_{p: D_{\\phi}(p,1/n)\\le \\rho/n}\\sum_i p_i(\\ell(\\theta,z_i)-\\ell(\\theta_{\\mathrm{ref}},z_i))$, gives a risk bound $R(\\tilde{\\theta}^*)\\le \\inf_{\\theta}\\left[R(\\theta)+\\sqrt{2\\rho/n\\,\\mathrm{Var}(\\ell(\\theta,\\cdot)-\\ell(\\theta_{\\mathrm{ref}},\\cdot))}\\right]+C_2/n$. This replaces the DRO variance term $\\mathrm{Var}(\\ell(\\theta^*,\\cdot))$ with the variance of the shifted loss at the optimum. Under the assumption that a sufficiently trained reference model has losses distributed like those of the target optimum, the shifted-loss variance is much smaller than the original variance, so DRRho risk minimization achieves a lower excess-risk bound than DRO or ERM. Corollary 4.3 makes the data-efficiency claim quantitative: if the reference model is trained by ERM on $m$ samples, DRRho needs only $n=O(\\sqrt{m})$ samples to reach the same excess-risk level, compared with $O(m)$ for ERM alone.","pith_inferences":["If the variance-reduction assumption holds across domains, the same steering recipe should transfer to any pretraining regime with an accessible reference model, such as autoregressive language models, where the shifted loss would be the per-token cross-entropy difference.","The bound suggests viewing the reference model as a control variate; this raises the testable possibility that an ensemble of references, or a reference trained on the same distribution as the target, would shrink the shifted-loss variance further than any single reference.","The $O(\\sqrt{m})$ sample-complexity result implies an exchange rate between data and an existing model: a reference trained on $m$ samples could substitute for roughly $m-\\sqrt{m}$ additional target samples, a prediction that a controlled data-ablation study could quantify.","The paper's empirical check of the variance assumption is limited to one 200K-subset measurement, so whether the improvement holds for weaker or out-of-distribution reference models is an open question that a broader variance audit would settle."],"forward_implications":["DRRho risk minimization supplies the first generalization theory for model steering, identifying the benefit of a reference model as a reduction in the variance term of the DRO bound.","Corollary 4.3 implies that DRRho needs only $O(\\sqrt{m})$ samples to match a reference model trained on $m$ samples, so an existing public model acts as a data-efficiency multiplier.","The framework turns existing RHO-loss heuristics into principled special cases: CVaR divergence gives whole-dataset top-$k$ selection, and KL divergence gives exponential reweighting, both as the DRRho objective.","Applied to CLIP, DRRho-CLIP matches baseline CLIP training with 50% less data, outperforms existing heuristic data-selection methods on ImageNet-1K accuracy, and combines with knowledge distillation to exceed both.","Measured scaling exponents are smaller for DRRho-CLIP (roughly $\\beta \\approx 0.116$) than for standard CLIP training (roughly $\\beta \\approx 0.127$), so the advantage is not a fixed offset but grows as compute increases."],"supporting_citations":[{"why":"It supplies the variance-based DRO generalization bound that the paper adapts to the shifted loss.","marker":"(Duchi & Namkoong, 2016)"},{"why":"It extends DRO generalization theory to generalized empirical likelihood divergences, used for the KL and other variants.","marker":"(Duchi et al., 2021)"},{"why":"It introduced the RHO loss $\\ell(\\theta,z)-\\ell(\\theta_{\\mathrm{ref}},z)$ for data selection, the heuristic that DRRho formalizes and improves.","marker":"(Mindermann et al., 2022)"},{"why":"It establishes the connection between contrastive losses and DRO that underlies the DRRho-CLIP loss construction.","marker":"(Qiu et al., 2023)"},{"why":"It defines CLIP and provides the pretrained reference models used as steering references in the experiments.","marker":"(Radford et al., 2021)"},{"why":"It provides the optimization algorithm used to train the global contrastive loss without large mini-batches.","marker":"(Yuan et al., 2022)"},{"why":"It provides the standard CLIP scaling-law baseline against which DRRho-CLIP's scaling exponents are compared.","marker":"(Cherti et al., 2023)"},{"why":"It supplies the DFN training datasets and the strong pretrained reference models used in the main experiments.","marker":"(Fang et al., 2024)"}],"fun_headline_variants":["Reference models slash data needs: theory proves variance drop","DRRho: learning with a reference model, half the data","Steer training with a reference: tighter bounds, less data","Model steering: reference loss cuts CLIP data in half"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a sufficiently trained reference model has losses $\\ell(\\theta_{\\mathrm{ref}},\\cdot)$ distributed like the target optimum's $\\ell(\\theta^*,\\cdot)$, so the variance of their difference is much smaller than the variance of $\\ell(\\theta^*,\\cdot)$ alone.","fun_headline_variants_meta":{"raw":{"variants":["Reference models slash data needs: theory proves variance drop","DRRho: learning with a reference model, half the data","Steer training with a reference: tighter bounds, less data","Model steering: reference loss cuts CLIP data in half"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000309,"raw_usage":{"total_tokens":1801,"prompt_tokens":1021,"completion_tokens":780,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":711}},"tokens_in":637,"tokens_out":780,"duration_ms":7195,"temperature":1.0,"reasoning_tokens":711,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:36:20.483464+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure $\\mathrm{Var}(\\ell(\\theta^*,\\cdot)-\\ell(\\theta_{\\mathrm{ref}},\\cdot))$ on held-out data for a reference model trained on a shifted distribution; if it is not below $\\mathrm{Var}(\\ell(\\theta^*,\\cdot))$, Theorem 4.1's bound is no tighter than DRO's and the argument loses its quantitative force.","supporting_citations":[{"cited_title":"M., Razzak, M","cited_arxiv_id":null,"evidence_quote":"It introduced the RHO loss $\\ell(\\theta,z)-\\ell(\\theta_{\\mathrm{ref}},z)$ for data selection, the heuristic that DRRho formalizes and improves."},{"cited_title":"Not all semantics are created equal: Contrastive self-supervised learning with automatic temperature individualization","cited_arxiv_id":null,"evidence_quote":"It establishes the connection between contrastive losses and DRO that underlies the DRRho-CLIP loss construction."},{"cited_title":"Provable stochastic optimization for global contrastive learning: Small batch does not harm performance","cited_arxiv_id":null,"evidence_quote":"It provides the optimization algorithm used to train the global contrastive loss without large mini-batches."},{"cited_title":"Reproducible scaling laws for contrastive language-image learning","cited_arxiv_id":null,"evidence_quote":"It provides the standard CLIP scaling-law baseline against which DRRho-CLIP's scaling exponents are compared."},{"cited_title":"M., Jain, A., Schmidt, L., Toshev, A","cited_arxiv_id":null,"evidence_quote":"It supplies the DFN training datasets and the strong pretrained reference models used in the main experiments."}],"review_version":1}