Pith. sign in

REVIEW 4 major objections 6 minor 50 references

Scaling Laws for Robust Comparison of Open Foundation Language-Vision Models and Datasets

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding a captioning loss to contrastive training (MaMMUT) gives a steeper error-versus-compute curve than CLIP, with the ranking flipping above a stable crossover near $10^{10}$–$10^{11}$ GFLOPs.

desk verdict Dense, reproducible scaling-law comparison that makes a plausible case for MaMMUT over CLIP at scale; the crossover location is fit-dependent, but the qualitative trend survives scrutiny. read the letter →

arxiv 2506.04598 v1 pith:2BLVNSH3 submitted 2025-06-05 cs.LG cs.AIcs.CV

Marianna Nezhurina , Tomer Porian , Giovanni Pucceti , Tommie Kerssies , Romain Beaumont , Mehdi Cherti , Jenia Jitsev This is my paper
classification cs.LGcs.AIcs.CV MSC 68T07
keywords scalinglawsCLIPMaMMUTvision-languagepretrainingmodelcomparisondatasetcompute-optimaltrainingzero-shotevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that scaling-law curves can be used not only to predict performance at larger compute budgets but to decide which pre-training procedure and which dataset deserve the budget. Fitting the error curve $L(C)=A_c(C+B_c)^{-\alpha_c}+E_c$ to dense measurements of open CLIP and MaMMUT models trained on three open datasets of 1.4 billion image-text pairs, the authors find that the contrastive-plus-captioning objective of MaMMUT scales better than the contrastive-only objective of CLIP: at small compute CLIP has the lower error, and above a threshold consistently located between $10^{10}$ and $10^{11}$ GFLOPs the ordering reverses in MaMMUT's favor. The reversal is stable across zero-shot classification, retrieval, and fine-tuned segmentation, across DataComp-1.4B, DFN-1.4B, and Re-LAION-1.4B, and across cosine and constant learning-rate schedules. If this holds, model and dataset choices made from a few small-scale benchmark points are systematically misleading, and a cheap comparison method becomes available: the constant-schedule variant costs about 98% less compute and reproduces the same trends. The paper validates its fits on held-out points and releases all intermediate checkpoints, including an open MaMMUT-L/14 that reaches 80.3% ImageNet-1k zero-shot accuracy.

What carries the argument

The mechanism that carries the argument is a parametric scaling-law fit to the Pareto front of measured error. For each compute budget the authors keep the training run with the lowest downstream error, binning GFLOPs logarithmically as in prior work, and fit $L(C)=A_c(C+B_c)^{-\alpha_c}+E_c$: a power law with a saturation term $B_c$ capturing chance-level behavior at tiny compute and a floor term $E_c$ capturing an irreducible error, chosen to keep the curve positive, monotone decreasing, and bounded below. Fit quality is checked by fitting only points below a threshold $C_{\rm threshold}$ and computing the mean squared error on the held-out higher-compute points, and uncertainty is propagated through the Jacobian of the fit into 95% confidence intervals. The same functional form is applied to samples seen $D$, producing data-efficiency curves and compute-optimal dataset sizes $D_{\rm opt}(C)=D_0 C^a$. The comparison statistic is the compute location where the CLIP and MaMMUT curves cross, and its stability across datasets, downstream tasks, and learning-rate schedules is the paper's evidence that the ranking is genuine.

What would settle it

Train CLIP and MaMMUT at a single large budget near $5\times10^{11}$ to $10^{12}$ GFLOPs on the same data with less than about 3x repetition and compare zero-shot ImageNet-1k error and MS-COCO R@5: the paper's own fits predict MaMMUT error clearly below CLIP at these scales with reported confidence intervals, so measured MaMMUT error equal to or above CLIP's would falsify the crossover claim. A cheaper probe is the paper's Table 1 prediction of 79.6% versus 81.6% at $2.14\times10^{12}$ GFLOPs, which requires a dataset larger than 1.4B unique pairs to test without repetition confounds.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that adding a captioning (text-generative) loss to the contrastive loss, as MaMMUT does, changes the scaling law qualitatively: MaMMUT's error curve is steeper, so although standard CLIP achieves lower error at equal compute on small budgets, the two curves cross and MaMMUT stays below CLIP for every compute budget above roughly $10^{10}$--$10^{11}$ GFLOPs. The crossover is found consistently on DataComp-1.4B, DFN-1.4B, and Re-LAION-1.4B, for zero-shot ImageNet-1k classification, MS-COCO retrieval, and fine-tuned ADE20K segmentation, and with both cosine and constant learning-rate schedules, which the authors take as evidence that the ranking is a property of the training procedures rather than of a particular scale or benchmark. The same fit machinery is then turned on the datasets: DFN-1.4B beats DataComp-1.4B and Re-LAION-1.4B on both classification and retrieval for both architectures, while DataComp-1.4B beats Re-LAION-1.4B on classification but matches it on retrieval. On data efficiency, the fits predict that MaMMUT needs fewer samples to be compute-optimal, and extrapolation to $2.14\times10^{12}$ GFLOPs predicts MaMMUT at 81.6% versus 79.6% for CLIP on ImageNet-1k zero-shot, with the measured 79.2% of the original DataComp CLIP run falling inside the CLIP confidence interval.

Load-bearing premise

The comparison is only as solid as the assumed error curve $L(C)=A_c(C+B_c)^{-\alpha_c}+E_c$ fitted through the best run per compute budget: if that curve does not keep describing error outside the measured range, or if the choice of which runs sit on the best-per-compute front favors one model, then the crossover point and the final ranking could be products of the fitting procedure rather than true differences between the two training methods.

Editorial extensions

If this is right

  • Comparisons at a single small scale are misleading: below the crossover CLIP looks better and above it MaMMUT does, so claims about which procedure is 'better' need to state a compute budget.
  • Applying the comparison at scale produced openMaMMUT-L/14, which reaches 80.3% zero-shot ImageNet-1k accuracy on 12.8B DataComp-1.4B samples, above the 79.2% of openCLIP at the same budget but below the 82% the fit predicts for low-repetition training.
  • Dataset rankings are task-dependent: DFN-1.4B outperforms the other two datasets on both classification and retrieval for both architectures, while DataComp-1.4B beats Re-LAION-1.4B on classification but only ties on retrieval.
  • Scaling-law comparison can be run with a constant learning-rate schedule, cutting the compute cost of the derivation by roughly 98% while preserving the same trends and crossover location.
  • Under controlled data, SigLIP shows no scaling advantage over CLIP on ImageNet-1k zero-shot accuracy, and decoder-only MaMMUT overtakes encoder-decoder CoCa at equal compute, with captioning-only training (Cap) lagging both.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the crossover threshold is stable rather than a fit artifact, the paper's curves imply a budget-dependent training recipe: spend a small budget on contrastive-only training and move to the contrastive-plus-captioning objective only above roughly $10^{10}$-$10^{11}$ GFLOPs; the paper reports the crossover but does not frame it as an objective-mixture rule.
  • The same Pareto-front fitting could rank axes the paper leaves untested (masked or diffusion losses, text-tower depth, patch size, input resolution) one axis at a time, since each ranking only needs enough runs to define that axis's front rather than a full factorial grid.
  • A direct stress test of the extrapolation: the paper's Table 1 predicts 79.6% versus 81.6% ImageNet-1k accuracy at $2.14\times10^{12}$ GFLOPs, so training both models on a dataset with more than about 1.4B unique pairs would check the fit outside its measured range, something current open datasets cannot do without heavy repetition.
  • The gap between the measured 80.3% and the predicted 82% at 12.8B samples quantifies a repetition penalty of roughly 1.7 accuracy points; adding a repetition-correction term to the fit would let labs with small open datasets predict their unique-data ceiling, which the paper leaves as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper derives compute-scaling laws for two open vision-language training procedures, CLIP and MaMMUT, on three open datasets (DataComp-1.4B, Re-LAION-1.4B, DFN-1.4B) and uses the fitted curves to compare their scalability. Measurements span model sizes from ViT-S to ViT-H, patch sizes, and samples seen from 1.28M to 3B, with downstream evaluation on zero-shot classification, retrieval, and fine-tuned segmentation. The central claim is that MaMMUT, which adds a captioning loss to the contrastive loss, has a steeper error-versus-compute scaling curve than CLIP and overtakes CLIP at a compute threshold consistently found between 1e10 and 1e11 GFLOPs across datasets, tasks, and learning-rate schedules. The paper also compares datasets (DFN favored over DataComp and Re-LAION), derives data-efficiency and compute-optimal-dataset-size laws, tests a constant-LR schedule that reduces compute cost, and releases training code, intermediate checkpoints, and the openMaMMUT-L/14 model (80.3% zero-shot ImageNet-1k accuracy).

Significance. If the central claim holds, this is a significant and reproducible contribution: it is the first dense scaling-law comparison of open CLIP and MaMMUT training, it explicitly controls for pre-training compute across a wide span, and it shows consistent trends across three datasets, three task types, and two learning-rate schedules. The release of all intermediate checkpoints, raw experiment data, and code is exemplary and makes the analysis independently verifiable. The held-out validation (e.g., Table 8) and the external CLIP match (79.2% within the predicted CI in Table 1) are concrete strengths. However, the headline crossover is a property of a specific four-parameter fitted form, and the paper does not yet demonstrate that the ranking and crossover are stable to the choice of functional form or to the Pareto-front envelope construction; the validation at 12.8B samples is one-sided and repetition-confounded. With the additional robustness analyses suggested below, the work would provide a strong and trustworthy basis for model and dataset comparison via scaling laws.

major comments (4)
  1. [Sec. 2.3, Eq. (1); App. C, Tabs. 8-9] The central crossover claim is a property of the four-parameter double-saturation form L(C) = A_c(C+B_c)^(-alpha_c)+E_c. The paper selects this form over the three-parameter form of Eq. (3) using held-out RMSE alone, but it never checks whether the ordering of CLIP and MaMMUT, or the crossover location, survives under Eq. (3) or under other comparably good fits. Because E_c is a free asymptotic error, a small change in its fitted value can change which curve is lower at the scales where the claim is made. Please report the predicted ordering and the crossover for Eq. (3), and for at least one alternative form (e.g., a power law with no offset or a form with an additional log term), together with confidence intervals on the crossover; if the ranking flips under any of these forms, the conclusion should be weakened accordingly.
  2. [Sec. 2.3; Figs. 15-16] The curves being compared are fits to separately constructed Pareto-front envelopes, not to a common set of (N,D) configurations. For each architecture, the envelope is built by taking the minimal-error model per compute bin, so the identity and number of envelope points can differ substantially between CLIP and MaMMUT, and the fitted slope of each envelope can be influenced by which model sizes happen to appear on that architecture's frontier. The paper should report, for each compute bin, which and how many configurations contribute to the CLIP versus MaMMUT envelope, and should show that the fitted crossover and ranking survive (i) fitting all configurations rather than only the envelope, and (ii) restricting the comparison to matched (N,D) grid points. Without such sensitivity analysis, the crossover could reflect a difference in envelope composition rather than a true property of the training procedures.
  3. [Sec. 3.1, Tab. 1; Sec. 3.7] The validation evidence for the central ranking is incomplete. The external 12.8B CLIP point matches the prediction, but it is repetition-confounded (approximately 9x, as the paper states) and validates only the CLIP curve; the corresponding openMaMMUT point (80.3% versus the no-repetition prediction of 82.0%) is below the prediction and therefore does not test the MaMMUT curve or the crossover. The held-out 3B points in Table 8 do validate each model's extrapolation within confidence intervals, but they are not presented as a matched-compute comparison of the two fitted curves, and the table does not provide a confidence interval for the crossover or for the probability that MaMMUT outperforms CLIP at a target compute budget. Please add such uncertainty quantification for the comparison, or explicitly restrict the claim to a prediction whose dependence on the repetition correction is quantified.
  4. [Sec. 3.1, Fig. 3; Sec. 2.1] The DFN-1.4B scaling laws are derived from measurements up to only 300M samples seen and model sizes up to L/14, a much narrower span than DataComp-1.4B or Re-LAION-1.4B, and the fitted crossover for DFN appears near or beyond the upper edge of the measured compute range. Since the paper's "consistent across datasets" claim includes DFN, the DFN result is substantially more dependent on the functional form than the other two datasets. The DFN comparison should either be presented with an explicit uncertainty caveat or be supported by additional measurements beyond 300M samples seen.
minor comments (6)
  1. [Sec. 2.3] The sentence "In general, the relationship between compute and performance follows a power law: L = aC b, where C is compute in FLOPs [10, 37] and C = arg min L(C)" is garbled: the exponent is missing in the typeset formula and the optimal-compute notation C = arg min L(C) is undefined. Please rewrite it.
  2. [Captions of Figs. 1-3] The figure captions state that MaMMUT takes over CLIP at "larger compute scale > 10^11 GFLOPs" (DataComp, Re-LAION) or "close to 10^11 GFLOPs" (DFN), while Sections 3.1 and 3.6 state the crossover is "between 10^10 and 10^11 GFLOPs". These numbers should be reconciled.
  3. [Fig. 1, Fig. 2, and related legends] The fitted equations in the figure legends, e.g., "57.86 * (x + exp(18.39)) 0.227 + 0.11", appear to omit the minus sign on the fitted exponent; readers may read 0.227 as a positive exponent. Please use an explicit "-0.227" in all legends.
  4. [Tab. 2] The reported Dopt confidence interval for the first compute budget, "2.30e+10 (2.75e+10, 1.91e+10)", is not a valid interval because the lower and upper bounds are in reverse order. Please check the reported values.
  5. [App. C, Tabs. 8-9] In Table 8, the MaMMUT RMSE is reported as 7.57e-03 for both Cthreshold values, while the text says that adding more points reduces RMSE; this suspicious coincidence deserves an explicit explanation or a correction.
  6. [Sec. 3.3, Fig. 8(b)] The compute-optimal dataset size exponents for CLIP and MaMMUT are both reported as 0.74; given the claimed data-efficiency advantage, the paper should state whether the exponents were fit independently and provide confidence intervals for them.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the fitted scaling-law comparison is validated on held-out points and against an external CLIP result; self-citations are methodological, not load-bearing.

full rationale

The paper's central claim is an empirical curve-fitting claim, not a definitional one. Scaling laws of the form L(C) = Ac(C+Bc)^(-alpha_c)+Ec are fitted to measured error rates on a Pareto front, and the MaMMUT-vs-CLIP comparison is read off from the fitted curves. This is not circular: the fitted curves are summaries of the measurements, not quantities defined in terms of the conclusion. The paper partially guards against fit artifacts by fitting only up to a compute threshold and evaluating held-out points (Tab. 8, Fig. 17), and by checking an external CLIP ViT-L-14 12.8B result (79.2%, within the predicted 95% CI of 0.788-0.804). The crossover around 1e10-1e11 GFLOPs is a derived crossing of two fitted curves, and its consistency across datasets, downstream tasks, and LR schedules is independent supporting evidence rather than an input. The choice of the double-saturation functional form over the no-irreducible-error form is justified by RMSE on held-out points (Tab. 8 vs Tab. 9), which is a legitimate model-selection criterion, not circularity. The acknowledged failure of the 12.8B MaMMUT point to reach the prediction (80.3% vs 82%) is an honest limitation due to sample repetition, not a circular step. Self-citations exist -- e.g., [10] for openCLIP methodology and [44] for constant-LR scaling-law derivation -- but they are background/method references; the dense new measurements and open release are independent of those prior results. Fragility of the functional form or of the Pareto-front selection would be a correctness or robustness concern, not a circularity, and the paper does not claim to derive the crossover from first principles. Overall circularity is low and confined to minor, non-load-bearing self-citation.

Assumptions & free parameters 9 free parameters · 6 assumptions · 0 invented entities

The paper's central claims rest on fitting parameters (Ac, Bc, alpha_c, Ec) of Eq. 1 to its own training runs, plus domain assumptions about the functional form, Pareto-front selection, and compute accounting. No new physical or theoretical entities are postulated; openMaMMUT is an open-source model implementation. The main free parameters are the scaling law amplitudes, offsets, exponents, and error floors for each architecture, dataset, and task; the exponents are the direct evidence for stronger scaling.

free parameters (9)
  • Ac, Bc, alpha_c, Ec for openCLIP on DataComp-1.4B (IN1k) = 57.862, exp(18.391), -0.227, 0.111
    Fitted in Eq. 1 from Pareto-front measurements; determines the CLIP scaling curve.
  • Ac, Bc, alpha_c, Ec for openMaMMUT on DataComp-1.4B (IN1k) = 79.970, exp(19.111), -0.233, 0.076
    Fitted in Eq. 1; the larger alpha_c magnitude is the paper's main evidence for stronger MaMMUT scaling.
  • Ac, Bc, alpha_c, Ec for CLIP and MaMMUT on DataComp-1.4B (MS-COCO retrieval) = CLIP: 53.913, exp(18.413), -0.230, 0.216; MaMMUT: 119.751, exp(19.122), -0.263, 0.212
    Fitted in Eq. 1 for the retrieval task; used for the crossover comparison in Fig. 1b.
  • Ac, Bc, alpha_c, Ec for CLIP and MaMMUT on Re-LAION-1.4B (IN1k) = CLIP: 21.54, exp(18.21), -0.173, 0.09; MaMMUT: 25.41, exp(19.18), -0.169, 0.00
    Fitted in Eq. 1; supports the robustness of the MaMMUT-over-CLIP trend on a second dataset.
  • Ac, Bc, alpha_c, Ec for CLIP and MaMMUT on DFN-1.4B (IN1k) = CLIP: 9473.08, exp(19.07), -0.498, 0.28; MaMMUT: 3496.51, exp(19.65), -0.428, 0.22
    Fitted in Eq. 1 from measurements up to 300M samples seen; used to rank DFN against other datasets.
  • Ac, Bc, alpha_c, Ec for constant-LR fits (DataComp-1.4B, IN1k) = CLIP: 14.769, exp(16.725), -0.168, 0.121; MaMMUT: 1850.286, exp(20.521), -0.379, 0.198
    Fitted in Eq. 1 using a constant learning rate schedule; parameters differ substantially from cosine fits but are claimed to yield the same comparison trend.
  • D0 and exponent a in Dopt(C) power law = CLIP: D0=16.88, a=0.74; MaMMUT: D0=8.66, a=0.74
    Fitted in Sec. 3.3 to estimate compute-optimal dataset size; the identical exponent 0.74 for both models is notable.
  • lambda (captioning loss weight) = 1
    Set in Sec. 2.1 as L = L_contrastive + lambda*L_cap; not tuned, but affects the relative behavior of MaMMUT.
  • Cthreshold (fit vs. held-out split) = 2.5e11 or 5e11 GFLOPs
    Chosen by the authors to divide fitting points from held-out points; varied in Appendix C to study fit quality.
assumptions (6)
  • domain assumption Scaling law functional form L(C)=Ac(C+Bc)^(-alpha_c)+Ec with monotonic decrease and positive irreducible error Ec (Eq. 1).
    Assumed to hold for extrapolation; the paper cites [37,38] but the form is chosen, not derived. If the true curve is not of this form, the crossover and ranking could change.
  • domain assumption Pareto-front selection: for each compute budget, the model configuration with minimal error rate represents the scaling law.
    The fit is performed on the lower envelope of the measurements (Sec. 2.3); this standard practice assumes the envelope is the right object to model, but differing envelope densities between architectures could bias the comparison.
  • domain assumption Compute GFLOPs is a sufficient and comparable measure of training cost across architectures.
    Models are compared at equal total compute (Sec. 2.3); MaMMUT has more parameters and different samples-seen at the same compute, so the comparison encodes a specific notion of efficiency.
  • domain assumption Data repetition up to about 3x (3B samples seen on 1.4B unique samples) does not materially distort the scaling law.
    The paper limits fits to <=3B samples seen (Sec. 3.1) and cites prior work on repetition effects; this assumption is load-bearing for the extrapolation to 12.8B.
  • domain assumption Constant learning rate schedules yield scaling laws comparable to cosine schedules for comparison purposes.
    The constant-LR derivation (Sec. 3.6) follows [44] and omits warmup points; the resulting fits have very different parameters, yet the comparison trend is asserted to hold.
  • domain assumption DFN-1.4B scaling laws, measured only up to 300M samples and L/14, extrapolate reliably enough for dataset comparison.
    The paper acknowledges higher uncertainty for DFN (Sec. 3.2); the conclusion that DFN is the best dataset depends on this extrapolation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling Laws for Robust Comparison of Open Foundation Language-Vision Models and Datasets." pith.science (2026). https://pith.science/paper/2BLVNSH3

@misc{pith2026250604598,
  author       = {Pith},
  title        = {Pith review of: Scaling Laws for Robust Comparison of Open Foundation Language-Vision Models and Datasets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2BLVNSH3}},
  note         = {Machine review of arXiv:2506.04598}
}
abstract

In studies of transferable learning, scaling laws are obtained for various important foundation models to predict their properties and performance at larger scales. We show here how scaling law derivation can also be used for model and dataset comparison, allowing to decide which procedure is to be preferred for pre-training. For the first time, full scaling laws based on dense measurements across a wide span of model and samples seen scales are derived for two important language-vision learning procedures, CLIP and MaMMUT, that use either contrastive only or contrastive and captioning text generative loss. Ensuring sufficient prediction accuracy for held out points, we use derived scaling laws to compare both models, obtaining evidence for MaMMUT's stronger improvement with scale and better sample efficiency than standard CLIP. To strengthen validity of the comparison, we show scaling laws for various downstream tasks, classification, retrieval, and segmentation, and for different open datasets, DataComp, DFN and Re-LAION, observing consistently the same trends. We show that comparison can also be performed when deriving scaling laws with a constant learning rate schedule, reducing compute cost. Accurate derivation of scaling laws provides thus means to perform model and dataset comparison across scale spans, avoiding misleading conclusions based on measurements from single reference scales only, paving the road for systematic comparison and improvement of open foundation models and datasets for their creation. We release all the pre-trained models with their intermediate checkpoints, including openMaMMUT-L/14, which achieves $80.3\%$ zero-shot ImageNet-1k accuracy, trained on 12.8B samples from DataComp-1.4B. Code for reproducing experiments in the paper and raw experiments data can be found at https://github.com/LAION-AI/scaling-laws-for-comparison.

Figures

Figures reproduced from arXiv: 2506.04598 by the authors.

Figure 1
Figure 1. Scaling on DataComp-1.4B. Comparison of CLIP and MaMMUT via scaling laws on DataComp-1.4B. Error rate on downstream tasks is plotted against compute. MaMMUT outperforms CLIP in terms of scalability, indicated by crossing scaling law fit lines, where MaMMUT takes over CLIP in performance from larger compute scale > 1011 GFLOPS on. 10 7 10 8 10 9 10 10 10 11 10 12 Compute C [GFLOPs] 10 0 3 × 10 1 4 × 10 1 6 × 10 1 Ima… view at source ↗
Figure 2
Figure 2. Scaling on Re-LAION-1.4B. Comparison of CLIP and MaMMUT via scaling laws on Re-LAION-1.4B. Error rate on downstream tasks is plotted against compute. MaMMUT outperforms CLIP in terms of scalability, indicated by crossing scaling law fit lines, where MaMMUT takes over CLIP in performance from larger compute scale > 1011 GFLOPS on, showing similar trends as on DataComp-1.4B. scalability and performance on MS-COCO retr… view at source ↗
Figure 3
Figure 3. Scaling on DFN-1.4B. Comparison of CLIP and MaMMUT via scaling laws on DFN-1.4B. Error rate on downstream tasks is plotted against compute. MaMMUT outperforms CLIP in terms of scalability, indicated by crossing scaling law fit lines, where MaMMUT takes over CLIP in performance from larger compute close to 1011 GFLOPS on, again showing similar trend as observed on DataComp and Re-LAION. 10 7 10 8 10 9 10 10 10 11 10 … view at source ↗
Figures from the paper (19 more)
Figure 4
Figure 4. Figure 4: Scaling laws for IN1k 0-shot performance of openCLIP (left) and openMaMMUT (right), [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Scaling laws for MS-COCO image retrieval performance (1- Recall@5) of openCLIP (left) [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Scaling laws for IN1k 0-shot performance of openCLIP (left) and openMaMMUT (right), [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Scaling laws for MS-COCO image retrieval performance (1- Recall@5) of openCLIP (left) [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Comparison of data efficiency and optimal dataset size for CLIP and MaMMUT via scaling [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Scaling laws for ImageNet-1k 0-shot classification, comparing SigLIP ( [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Scaling law fit for ImageNet-1k 0-shot classification, comparing MaMMUT, CLIP and [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: Scaling law on DataComp evaluation suite (average over 35 tasks, 0-shot classification), [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: Scaling laws for ImageNet-v2 (left) and ImageNet robustness set (right, averaged perfor [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]
Figure 13
Figure 13. Figure 13: Scaling law for semantic segmentation. Downstream error rate (1 – mIoU) of openCLIP and openMaMMUT pre-trained on DataComp-1.4B and fine-tuned on ADE20K. MaMMUT shows higher performance than CLIP for segmentation at higher scales. Crossing point appears earlier around…
Figure 14
Figure 14. Figure 14: Scaling law fits using constant learning rate scheduler. Comparison of CLIP and MaMMUT via scaling laws on DataComp-1.4B. Error rate on downstream tasks is plotted against compute. Using constant learning rate scheduler for scaling law derivation reveals the same tren…
Figure 15
Figure 15. Figure 15: Detailed version of the scaling law fit for ImageNet 0-shot classification error rate [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]
Figure 16
Figure 16. Figure 16: Detailed version of the scaling law fit for ImageNet 0-shot classification error rate [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]
Figure 17
Figure 17. Figure 17: Comparison of the fit quality for ImageNet-1k 0-shot classification error rate for open [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]
Figure 18
Figure 18. Figure 18: Comparison of the fit quality for MS-COCO image retrieval error rate for openMaMMUT [PITH_FULL_IMAGE:figures/full_fig_p023_18.png]
Figure 19
Figure 19. Figure 19: Comparison of the fit quality for ImageNet-1k 0-shot classification error rate for open [PITH_FULL_IMAGE:figures/full_fig_p024_19.png]
Figure 20
Figure 20. Figure 20: Comparison of the fit quality for MS-COCO image retrieval error rate for openMaMMUT [PITH_FULL_IMAGE:figures/full_fig_p024_20.png]
Figure 21
Figure 21. Figure 21: Detailed scaling law for downstream semantic segmentation performance of openCLIP [PITH_FULL_IMAGE:figures/full_fig_p027_21.png]
Figure 22
Figure 22. Figure 22: Detailed scaling law for downstream semantic segmentation performance of openMaM [PITH_FULL_IMAGE:figures/full_fig_p029_22.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 24 canonical work pages

  1. [1]

    On the opportunities and risks of foundation models

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021

    arXiv 2021

  2. [2]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

  3. [3]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 2https://discord.gg/BZqhreFazY 3https://discord.gg/GsKh4mBVcv 15

    work page 2020

  4. [4]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877–1901, 2020

    1901

  5. [5]

    Scaling language-free visual representa- tion learning

    David Fan, Shengbang Tong, Jiachen Zhu, Koustuv Sinha, Zhuang Liu, Xinlei Chen, Michael Rabbat, Nicolas Ballas, Yann LeCun, Amir Bar, et al. Scaling language-free visual representa- tion learning. arXiv preprint arXiv:2504.01017, 2025

    arXiv 2025

  6. [6]

    Clap learning audio concepts from natural language supervision

    Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1–5. IEEE, 2023

    2023

  7. [7]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748–8763. PMLR, 2021

    work page 2021

  8. [8]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    arXiv 2001

Show all 50 references
  1. [9]

    An empirical analysis of compute-optimal large language model training

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon ...

    2022

  2. [10]

    Reproducible scaling laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF conference on computer vision...

    2023

  3. [11]

    (mis)fitting scaling laws: A survey of scaling law fitting techniques in deep learning

    Margaret Li, Sneha Kudugunta, and Luke Zettlemoyer. (mis)fitting scaling laws: A survey of scaling law fitting techniques in deep learning. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  4. [12]

    Visual instruction tuning.Advances in neural information processing systems , 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems , 36:34892–34916, 2023

    2023

  5. [13]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vis...

    2024

  6. [14]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alab- dulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, an...

    2025 arXiv

  7. [15]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023

    2023 arXiv

  8. [16]

    Coca: Contrastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022

    2022 arXiv

  9. [17]

    Dai, Zhifeng Chen, Claire Cui, and Anelia Angelova

    Weicheng Kuo, AJ Piergiovanni, Dahun Kim, xiyang luo, Benjamin Caine, Wei Li, Abhijit Ogale, Luowei Zhou, Andrew M. Dai, Zhifeng Chen, Claire Cui, and Anelia Angelova. MaM- MUT: A simple architecture for joint learning for multimodal tasks. Transactions on Machine Learning Res...

    2023

  10. [18]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023

    2023

  11. [19]

    Datacomp: In search of the next generation of multimodal datasets

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Syste...

    2023

  12. [20]

    Data filtering networks

    Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. In The Twelfth International Conference on Learning Representations, 2024

    2024

  13. [21]

    Releasing re-laion 5b: transparent iteration on laion-5b with additional safety fixes

    LAION. Releasing re-laion 5b: transparent iteration on laion-5b with additional safety fixes. https://laion.ai/blog/relaion-5b/, 2024. Accessed: 30 aug, 2024

    2024

  14. [22]

    Ilharco, M

    G. Ilharco, M. Wortsman, N. Carlini, R. Taori, A. Dave, V . Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip. 2021

    2021

  15. [23]

    Representation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  16. [24]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  17. [25]

    How do we know how smart ai systems are? Science, 381(6654):eadj5957, 2023

    Melanie Mitchell. How do we know how smart ai systems are? Science, 381(6654):eadj5957, 2023

    2023

  18. [26]

    J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pages 248–255, June 2009

    2009

  19. [27]

    Do imagenet classifiers generalize to imagenet? In International conference on machine learning , pages 5389–5400

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning , pages 5389–5400. PMLR, 2019

    2019

  20. [28]

    The many faces of robustness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. Inter...

    2021 arXiv

  21. [29]

    Natural adversarial examples

    Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. Conference on Computer Vision and Pattern Recognition (CVPR) , 2021. https://arxiv.org/abs/1907.07174

    2021 arXiv

  22. [30]

    Learning robust global represen- tations by penalizing local predictive power

    Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global represen- tations by penalizing local predictive power. In Advances in Neural Information Processing Systems (NeurIPS), 2019. https://arxiv.org/abs/1905.13549

    2019 arXiv

  23. [31]

    Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models

    Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems (N...

    2019

  24. [32]

    Clip benchmark

    Romain Beaumont Mehdi Cherti et al. Clip benchmark. https://github.com/LAION-AI/ CLIP_benchmark, 2023

    2023

  25. [33]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014

    2014

  26. [34]

    Scene Parsing through ADE20K Dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene Parsing through ADE20K Dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5122–5130, 2017. 17

    2017

  27. [35]

    Tommie Kerssies, Daan De Geus, and Gijs Dubbelman. How to Benchmark Vision Foundation Models for Semantic Segmentation? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 1162–1171, 2024

    2024

  28. [36]

    Your ViT is Secretly an Image Segmen- tation Model

    Tommie Kerssies, Niccolò Cavagnero, Alexander Hermans, Narges Norouzi, Giuseppe Averta, Bastian Leibe, Gijs Dubbelman, and Daan de Geus. Your ViT is Secretly an Image Segmen- tation Model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CV...

    2025

  29. [37]

    Scaling vision transform- ers

    Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12104–12113, 2022

    2022

  30. [38]

    Scaling laws for autoregressive generative modeling

    Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020

    2010 arXiv

  31. [39]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  32. [40]

    The skyline operator

    Stephan Borzsony, Donald Kossmann, and Konrad Stocker. The skyline operator. InProceedings 17th international conference on data engineering , pages 421–430. IEEE, 2001

    2001

  33. [41]

    Scaling data-constrained language models

    Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems , 36:50358–50376, 2023

    2023

  34. [42]

    LAION-5B: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmar- czyk, and Jenia Jitsev....

    2022

  35. [43]

    Image captioners are scalable vision learners too

    Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer. Image captioners are scalable vision learners too. Advances in Neural Information Processing Systems, 36:46830–46855, 2023

    2023

  36. [44]

    Resolving discrepancies in compute-optimal scaling of language models

    Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Resolving discrepancies in compute-optimal scaling of language models. Advances in Neural Information Processing Systems, 37:100535–100570, 2024

    2024

  37. [45]

    Demystifying clip data

    Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang- Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. arXiv preprint arXiv:2309.16671, 2023

    2023 arXiv

  38. [46]

    Eva-clip: Improved training techniques for clip at scale

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023

    2023 arXiv

  39. [47]

    Tulip: Towards unified language-image pretraining

    Zineng Tang, Long Lian, Seun Eisape, XuDong Wang, Roei Herzig, Adam Yala, Alane Suhr, Trevor Darrell, and David M Chan. Tulip: Towards unified language-image pretraining. arXiv preprint arXiv:2503.15485, 2025

    2025 arXiv

  40. [48]

    Openvision: A fully-open, cost-effective family of advanced vision encoders for multimodal learning

    Xianhang Li, Yanqing Liu, Haoqin Tu, Hongru Zhu, and Cihang Xie. Openvision: A fully-open, cost-effective family of advanced vision encoders for multimodal learning. arXiv preprint arXiv:2505.04601, 2025

    2025 arXiv

  41. [49]

    Scaling laws vs model architectures: How does inductive bias influence scaling? arXiv preprint arXiv:2207.10551, 2022

    Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. Scaling laws vs model architectures: How does inductive bias influence scaling? arXiv preprint arXiv:2207.10551, 2022. 18

    2022 arXiv

  42. [50]

    Scaling laws for data filtering–data curation cannot be compute agnostic

    Sachin Goyal, Pratyush Maini, Zachary C Lipton, Aditi Raghunathan, and J Zico Kolter. Scaling laws for data filtering–data curation cannot be compute agnostic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22702–22711, 2024. 19 Ap...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.