Pith. sign in

REVIEW 2 major objections 1 minor 1 cited by

Instruction-heavy data mixtures scale better than caption-heavy ones for vision-language model training.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-07-02 21:08 UTC pith:HLF232VG

load-bearing objection DCVLM gives a controlled benchmark and 6T-token corpus for VLM data curation, with evidence that instruction-heavy mixing beats filtering at scale, but the 33-task suite may over-weight the very capabilities that mixing targets. the 2 major comments →

arxiv 2606.28551 v2 pith:HLF232VG submitted 2026-06-26 cs.CV cs.CLcs.LG

DataComp-VLM: Improved Open Datasets for Vision-Language Models

Matteo Farina , Vishaal Udandarao , Thao Nguyen , Selim Kuzucu , Maximilian B\"other , Andreas Hochlehnert , Adhiraj Ghosh , Marianna Nezhurina
show 28 more authors
Karsten Roth Joschka Struber Yuhui Zhang Sebastian Dziadzio Elaine Sui Soumya Jahagirdar Dhruba Ghosh Hasan Hammoud Thomas De Min Simone Caldarella Jehanzeb Mirza Sedrick Keh Mehdi Cherti Hilde Kuehne Bernt Schiele Serena Yeung-Levy Muhammad Ferjad Naeem Federico Tombari Ana Klimovic Elisa Ricci Matthias Bethge Sewoong Oh Ameya Prabhu Alessio Tonioni Jenia Jitsev Massimiliano Mancini Ludwig Schmidt Nikhil Parthasarathy
This is my paper
classification cs.CV cs.CLcs.LG
keywords vision-language modelsdataset curationdata mixinginstruction tuningbenchmarkmultimodal trainingopen datasets
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces DataComp for VLMs, a benchmark that collects 160 datasets totaling 6T tokens across image captions, interleaved documents, text, and instructions. It runs controlled experiments varying filtering, mixing, formatting, and sampling on models from 1B to 8B parameters and token budgets up to 200B. Results show that the choice of mixture composition matters more than aggressive filtering, with instruction-heavy blends producing larger gains as scale increases. The resulting DCVLM-Baseline dataset trains an 8B VLM to 63.6 percent accuracy on a 33-task core evaluation suite, exceeding the prior open dataset FineVision by 5.4 points.

Core claim

Data mixing, not filtering, is the dominant factor in building high-quality VLM training sets; instruction-heavy mixtures outperform caption-heavy ones, with the performance gap widening at larger model and data scales. The DCVLM-Baseline mixture derived from these experiments enables an 8B-parameter VLM trained on 200B tokens to reach 63.6 percent average accuracy across the 33-task core suite, a 5.4-point improvement over the previous state-of-the-art open VLM dataset FineVision.

What carries the argument

The DCVLM benchmark, which standardizes curation operations (filtering, mixing, formatting, sampling) across fixed model sizes and token budgets and evaluates on a fixed suite of up to 52 downstream tasks in nine domains.

Load-bearing premise

The selected downstream benchmarks adequately represent the full range of general VLM capabilities.

What would settle it

A caption-heavy mixture or a purely filtering-based curation strategy that achieves higher average accuracy than DCVLM-Baseline on the same 33-task core suite when trained at the 8B scale with 200B tokens.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Instruction-heavy mixtures deliver increasing returns as model size and token count grow.
  • Filtering alone yields smaller gains than careful composition of data types.
  • The DCVLM-Baseline dataset can be used directly to train stronger open VLMs without proprietary data.
  • Curation effort should prioritize mixing ratios over removal of individual low-quality examples.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Future work could test whether the same mixing preference holds when the evaluation suite is expanded to include more reasoning-heavy or long-context tasks.
  • The benchmark design makes it straightforward to measure whether new data sources improve performance mainly by changing the overall mixture balance.
  • Practitioners building custom VLM datasets may achieve comparable gains by reweighting existing public collections toward instruction data rather than collecting new filtered captions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper introduces DataComp-VLM (DCVLM), a benchmark and corpus of 6T multimodal tokens from 160 datasets across four data types. It enables controlled experiments on curation operations (filtering, mixing, formatting, sampling) for 1B-8B VLMs trained on 6.25B-200B tokens, evaluated on up to 52 downstream benchmarks across 9 domains. The central empirical finding is that data mixing—not filtering—is the dominant factor, with instruction-heavy mixtures outperforming caption-heavy ones (gains widening at larger scales); the resulting DCVLM-Baseline yields an 8B VLM at 63.6% on the 33-task core suite (+5.4pp over FineVision).

Significance. If the results hold, the work supplies the first large-scale, controlled benchmark for VLM data curation and demonstrates that mixing strategies can be systematically optimized, with public release of the corpus, baseline, and evaluation suite as a concrete community resource. The scale of the experiments (multiple model sizes and token budgets) and the quantitative improvement over an existing SOTA open dataset are notable strengths.

major comments (2)
  1. [Evaluation] Evaluation section (and abstract): the claim that 'instruction-heavy mixtures scale better than caption-heavy ones, with gains widening at larger scales' is supported only by relative performance on the 33-task core suite. The manuscript provides no evidence that this suite was constructed with held-out domains, that deltas were tested for sensitivity to task re-weighting, or that performance was measured on underrepresented categories such as pure captioning, OCR, or retrieval; this is load-bearing for the generalization that mixing is the key curation operation.
  2. [Experiments] Methods / Experiments: the abstract and results report specific accuracy gains (e.g., 63.6% and +5.4pp) without mention of error bars, multiple random seeds, or data-exclusion criteria for the downstream benchmarks. Because the central mixing-vs-filtering conclusion rests on these measured deltas, the absence of statistical characterization weakens verification of the reported improvements.
minor comments (1)
  1. [Abstract] The abstract states 'up to 52 downstream benchmarks' while the core suite is described as 33 tasks; clarify the exact overlap and selection criteria in the main text.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback. We address the two major comments point by point below, indicating where revisions will be made.

read point-by-point responses

  1. Referee: [Evaluation] Evaluation section (and abstract): the claim that 'instruction-heavy mixtures scale better than caption-heavy ones, with gains widening at larger scales' is supported only by relative performance on the 33-task core suite. The manuscript provides no evidence that this suite was constructed with held-out domains, that deltas were tested for sensitivity to task re-weighting, or that performance was measured on underrepresented categories such as pure captioning, OCR, or retrieval; this is load-bearing for the generalization that mixing is the key curation operation.

    Authors: The 33-task core suite was explicitly selected to ensure coverage across all 9 evaluation domains (including captioning, OCR, retrieval, and VQA), with the full 52-task results provided in the appendix showing consistent trends. We will add an explicit subsection on task selection methodology, domain balance, and sensitivity analysis to re-weighting in the revised manuscript to further support the generalization. revision: partial

  2. Referee: [Experiments] Methods / Experiments: the abstract and results report specific accuracy gains (e.g., 63.6% and +5.4pp) without mention of error bars, multiple random seeds, or data-exclusion criteria for the downstream benchmarks. Because the central mixing-vs-filtering conclusion rests on these measured deltas, the absence of statistical characterization weakens verification of the reported improvements.

    Authors: We agree that additional statistical detail would strengthen the presentation. All experiments use fixed seeds for reproducibility; the scale of the 8B/200B-token runs made multiple independent seeds computationally prohibitive. We will add a limitations paragraph on this point, report error bars for all smaller-scale ablations, and expand the existing description of benchmark data-exclusion criteria in Section 4. revision: partial

Circularity Check

0 steps flagged

No circularity: empirical results on external benchmarks

full rationale

The paper's central claims rest on empirical measurements: models trained on curated mixtures are evaluated on a held-out suite of up to 52 downstream benchmarks across 9 domains. No equations, fitted parameters, or self-referential definitions appear in the derivation; the superiority of instruction-heavy mixing is reported as observed performance deltas (e.g., +5.4pp over FineVision), not as a quantity forced by the curation process itself. Self-citations, if present, are not load-bearing for the mixing-vs-filtering conclusion. The evaluation distribution is external to the training data construction, satisfying the condition for a self-contained empirical result.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Empirical benchmark paper with no mathematical derivations. No free parameters are fitted to the reported accuracy numbers. Relies only on standard machine-learning training assumptions.

axioms (1)
  • standard math Standard i.i.d. sampling and gradient-based optimization assumptions used in large-scale model training.
    Implicit background for all reported training runs.

pith-pipeline@v0.9.1-grok · 5969 in / 1293 out tokens · 32271 ms · 2026-07-02T21:08:12.704374+00:00 · methodology

0 comments
read the original abstract

Building performant Vision-Language Models (VLMs) requires carefully curating large-scale training datasets, yet the community lacks systematic benchmarks for evaluating such curation strategies. We introduce DataComp for VLMs (DCVLM), a benchmark for controlled data-centric experiments to improve VLM training. As part of DCVLM, we collect 160 datasets spanning four data types -- image-caption pairs, multimodal interleaved documents, text-only, and instruction-tuning data -- into a corpus of 6T multimodal tokens. DCVLM allows participants to test curation strategies (filtering, mixing, formatting, sampling) across 1B-8B models and 6.25B-200B token budgets. Models are then evaluated on a carefully selected suite of up to 52 downstream benchmarks across 9 domains. We conduct extensive experiments on DCVLM and find that data mixing, not filtering, is key to a high-quality training dataset: instruction-heavy mixtures scale better than caption-heavy ones, with gains widening at larger scales. The resulting dataset, DCVLM-Baseline, enables training an 8B VLM to 63.6% accuracy on our 33-task core suite with 200B training tokens. Compared to FineVision, the state-of-the-art open VLM training dataset, this represents an improvement of +5.4pp. DCVLM and all accompanying artifacts will be made publicly available at https://www.datacomp.ai/dcvlm/.

Figures

Figures reproduced from arXiv: 2606.28551 by Adhiraj Ghosh, Alessio Tonioni, Ameya Prabhu, Ana Klimovic, Andreas Hochlehnert, Bernt Schiele, Dhruba Ghosh, Elaine Sui, Elisa Ricci, Federico Tombari, Hasan Hammoud, Hilde Kuehne, Jehanzeb Mirza, Jenia Jitsev, Joschka Struber, Karsten Roth, Ludwig Schmidt, Marianna Nezhurina, Massimiliano Mancini, Matteo Farina, Matthias Bethge, Maximilian B\"other, Mehdi Cherti, Muhammad Ferjad Naeem, Nikhil Parthasarathy, Sebastian Dziadzio, Sedrick Keh, Selim Kuzucu, Serena Yeung-Levy, Sewoong Oh, Simone Caldarella, Soumya Jahagirdar, Thao Nguyen, Thomas De Min, Vishaal Udandarao, Yuhui Zhang.

Figure 1
Figure 1. Figure 1: DCVLM-BASELINE outperforms open VLM training datasets. DCVLM-BASELINE (left) combines 160 sources as 10% image-caption pairs, 5% multimodal documents, 15% text-only, and 70% multimodal instruction-tuning data. On our 33-evaluation Core set (right), it outperforms existing datasets [12, 58, 310] across all scales. Notably, a 4B model trained on DCVLM-BASELINE for 100B tokens beats an 8B model trained on FIN… view at source ↗
Figure 2
Figure 2. Figure 2: DCVLM allows researchers to construct effective multimodal datasets. Participants can choose one of four scales (small, medium, large, and x-large) according to their compute availability. We provide tools to format, filter, and mix the data pool so that participants can create their own datasets. The resulting datasets are then used to train an autoregressive VLM using a fixed training recipe. Models are … view at source ↗
Figure 3
Figure 3. Figure 3: Filtering rarely helps, but changing the data composition does move performance substantially. Established data filtering techniques do not significantly outperform a no-filter baseline. This observation holds consistently at both the small ( [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Upstream filtering leads to diminishing returns from additional (i.e., “downstream”) filtering. This failure is surprising, especially in light of strong results from prior works. We hypothesize this is because there is no significant noise to remove from our base pool: unlike raw Common Crawl (used in DCLM [158]) or raw web-crawled image-text pairs (used in DataComp￾CLIP [73]), existing VLM training sets … view at source ↗
Figure 5
Figure 5. Figure 5: Instruction-heavy mixtures scale better with compute. For the 1B model (left), the Instruction-heavy mix (red) starts as the worst mixture with 6.25B training tokens, but recovers quickly up to becoming the second-best with 25B training tokens. For the 2B model (middle), all mixtures have comparable performance with 6.25B tokens, but performance gains consistently grow in favor of the Instruction-heavy mix… view at source ↗
Figure 6
Figure 6. Figure 6: Control experiments. (Left) Pretraining performance predicts post-SFT performance with near-perfect fidelity. (Right) Data mixture rankings are preserved when switching the LM backbone from Qwen2.5-Base to Qwen2.5-Instruct, verifying robustness of our results to choice of backbone. Repeatability of instruction-tuning data. Given our previous finding that Instruction-heavy mixes scale better, a natural conc… view at source ↗
Figure 7
Figure 7. Figure 7: DCVLM pool composition by data type. Share of total samples (top) vs. share of total multimodal tokens (bottom) for each of the four data types. The pool is dominated by image-caption pairs on both axes (83% tokens vs 74% samples). Text-only data exhibits the opposite asymmetry, with 19% of samples but only 5% of tokens. Instruction-tuning data and Multimodal documents are token-dense, i.e, their overall t… view at source ↗
Figure 7
Figure 7. Figure 7: DCVLM pool composition by data type. Share of total samples (top) vs. share of total multimodal tokens (bottom) for each of the four data types. The pool is dominated by image-caption pairs on both axes (83% tokens vs 74% samples). Text-only data exhibits the opposite asymmetry, with 19% of samples but only 5% of tokens. Instruction-tuning data and Multimodal documents are token-dense, i.e, their overall t… view at source ↗
Figure 8
Figure 8. Figure 8: Samples vs. multimodal tokens (log–log). Each marker is one of 160 datasets in our pool, colored by data type. Diagonal reference lines mark constant tokens-per-sample regimes (100, 1K, 10K). Image-caption datasets cluster tightly along the 1–2K tok/sample diagonal driven by the visual￾token contribution per image; multimodal documents sit one decade higher (multi-image samples); text-only datasets occupy … view at source ↗
Figure 8
Figure 8. Figure 8: Samples vs. multimodal tokens (log–log). Each marker is one of 160 datasets in our pool, colored by data type. Diagonal reference lines mark constant tokens-per-sample regimes (100, 1K, 10K). Image-caption datasets cluster tightly along the 1–2K tok/sample diagonal driven by the visual￾token contribution per image; multimodal documents sit one decade higher (multi-image samples); text-only datasets occupy … view at source ↗
Figure 9
Figure 9. Figure 9: Language distribution of DCVLM pool. Top-15 languages by per-sample top-1 prediction from Lingua (left, blue) and NLLB (right, red), shown on a log-scaled x-axis. The pool is dominated by English (91.1% Lingua / 92.8% NLLB), followed by Chinese (3.9% / 3.5%); the remaining ∼5% is spread across 73 languages (Lingua) or 114 languages (NLLB), with French, German, Japanese, Korean, and Russian forming the next… view at source ↗
Figure 9
Figure 9. Figure 9: Language distribution of DCVLM pool. Top-15 languages by per-sample top-1 prediction from Lingua (left, blue) and NLLB (right, red), shown on a log-scaled x-axis. The pool is dominated by English (91.1% Lingua / 92.8% NLLB), followed by Chinese (3.9% / 3.5%); the remaining ∼5% is spread across 73 languages (Lingua) or 114 languages (NLLB), with French, German, Japanese, Korean, and Russian forming the next… view at source ↗
Figure 10
Figure 10. Figure 10: Not all benchmarks monotonically increase with scale. We report the performance of 65 benchmarks (averaged over three runs at both the small and the medium scales) with a focus on their scaling behaviour. Average performance at the small scale is arranged on the x-axis; the y-axis displays average performance at the medium scale. Benchmarks falling below the identity line (highlighted in red in the figure… view at source ↗
Figure 10
Figure 10. Figure 10: Not all benchmarks monotonically increase with scale. We report the performance of 65 benchmarks (averaged over three runs at both the small and the medium scales) with a focus on their scaling behaviour. Average performance at the small scale is arranged on the x-axis; the y-axis displays average performance at the medium scale. Benchmarks falling below the identity line (highlighted in red in the figure… view at source ↗
Figure 11
Figure 11. Figure 11: Seed variance of the initial 65-benchmark pool. We conduct runs at small and medium scales, to measure the seed variance of benchmarks in our initial 65 candidates. We remove POPE [164], as it leads to a standard deviation of up to 16% over three runs at the small scale. monotonically improve from small to medium scale, as such benchmarks would risk obscuring the signal attributable to data curation choic… view at source ↗
Figure 11
Figure 11. Figure 11: Seed variance of the initial 65-benchmark pool. We conduct runs at small and medium scales, to measure the seed variance of benchmarks in our initial 65 candidates. We remove POPE [164], as it leads to a standard deviation of up to 16% over three runs at the small scale. monotonically improve from small to medium scale, as such benchmarks would risk obscuring the signal attributable to data curation choic… view at source ↗
Figure 12
Figure 12. Figure 12: Image-decontamination removal rate vs. SSCD similarity threshold. Fraction of pool images removed as a function of the similarity threshold for four sources. The shaded band between our threshold (0.75, green) and FineVision’s (0.95, red) is detected train/test overlap that FineVision’s protocol leaves in the training pool. GSM8K, MATH, HumanEval, MBPP, MBPP-CN, TheoremQA). We use a two-stage MinHash + ex… view at source ↗
Figure 12
Figure 12. Figure 12: Image-decontamination removal rate vs. SSCD similarity threshold. Fraction of pool images removed as a function of the similarity threshold for four sources. The shaded band between our threshold (0.75, green) and FineVision’s (0.95, red) is detected train/test overlap that FineVision’s protocol leaves in the training pool. GSM8K, MATH, HumanEval, MBPP, MBPP-CN, TheoremQA). We use a two-stage MinHash + ex… view at source ↗
Figure 13
Figure 13. Figure 13: Qualitative train/test image matches by SSCD similarity band. Each pair shows a training-pool image (left) and its top-1 match in the DCVLM-Extended evaluation suite (right). Rows are grouped by the maximum SSCD cosine similarity of the pair, and labels above each pair give the pool source. Only near-pixel-identical copies score ≥ 0.95 (top row, removed by both protocols). The 0.75–0.95 band (orange) cons… view at source ↗
Figure 13
Figure 13. Figure 13: Qualitative train/test image matches by SSCD similarity band. Each pair shows a training-pool image (left) and its top-1 match in the DCVLM-Extended evaluation suite (right). Rows are grouped by the maximum SSCD cosine similarity of the pair, and labels above each pair give the pool source. Only near-pixel-identical copies score ≥ 0.95 (top row, removed by both protocols). The 0.75–0.95 band (orange) cons… view at source ↗
Figure 14
Figure 14. Figure 14: Text-decontamination hyperparameter sweep. Per-dataset match rate (% of training queries flagged) across the three MinHash design choices. Signature size is immaterial (a), so we use 128 permutations. Longer n-grams miss short evaluation prompts and detect almost no overlap (c), so we use 5-grams (b). Match rates are small in absolute terms and fall steeply with the threshold [PITH_FULL_IMAGE:figures/ful… view at source ↗
Figure 14
Figure 14. Figure 14: Text-decontamination hyperparameter sweep. Per-dataset match rate (% of training queries flagged) across the three MinHash design choices. Signature size is immaterial (a), so we use 128 permutations. Longer n-grams miss short evaluation prompts and detect almost no overlap (c), so we use 5-grams (b). Match rates are small in absolute terms and fall steeply with the threshold [PITH_FULL_IMAGE:figures/ful… view at source ↗
Figure 15
Figure 15. Figure 15: Threshold selection via human annotation. True-positive (TPR, green) and false￾positive (FPR, red) rates of MinHash matches at each similarity bin, aggregated over seven annotators. Below ∼0.55 most matches are spurious (high FPR). The TPR climbs steeply just above it and the curves cross around 0.55–0.6, motivating our first-stage threshold of 0.55. TPR rises steeply just above it and the two curves cros… view at source ↗
Figure 15
Figure 15. Figure 15: Threshold selection via human annotation. True-positive (TPR, green) and false￾positive (FPR, red) rates of MinHash matches at each similarity bin, aggregated over seven annotators. Below ∼0.55 most matches are spurious (high FPR). The TPR climbs steeply just above it and the curves cross around 0.55–0.6, motivating our first-stage threshold of 0.55. TPR rises steeply just above it and the two curves cros… view at source ↗
Figure 16
Figure 16. Figure 16: Per-dataset removal rates of the full decontamination protocol. The 30 pool sources with the highest fraction of samples removed by the combined image-based (SSCD s ≥ 0.75) and text-based (MinHash Jaccard ≥ 0.55 + substring check) decontamination against the DCVLM￾Extended evaluation suite. Removal concentrates in sources whose training and evaluation splits share an underlying distribution (InfoVQA, Scie… view at source ↗
Figure 16
Figure 16. Figure 16: Per-dataset removal rates of the full decontamination protocol. The 30 pool sources with the highest fraction of samples removed by the combined image-based (SSCD s ≥ 0.75) and text-based (MinHash Jaccard ≥ 0.55 + substring check) decontamination against the DCVLM￾Extended evaluation suite. Removal concentrates in sources whose training and evaluation splits share an underlying distribution (InfoVQA, Scie… view at source ↗
Figure 17
Figure 17. Figure 17: Filtering rarely helps, but changing the data composition does move performance substantially (cont). Complementary experiments at the small scale of our benchmark confirm the outcome of Sec. 4.1: established quality filtering rarely provides significant gains over a no-filter baseline, yet variations between global and local filtering remain frequent. H Additional Experiments and Details H.1 Filtering ra… view at source ↗
Figure 17
Figure 17. Figure 17: Filtering rarely helps, but changing the data composition does move performance substantially (cont). Complementary experiments at the small scale of our benchmark confirm the outcome of Sec. 4.1: established quality filtering rarely provides significant gains over a no-filter baseline, yet variations between global and local filtering remain frequent. H Additional Experiments and Details H.1 Filtering ra… view at source ↗
Figure 18
Figure 18. Figure 18: Length-proportional sampling (T = 1) is near-optimal. Validation average as a function of sampling temperature T in p(d) ∝ len(d) 1/T . Both sharpening (T < 1) and flattening (T > 1) degrade performance relative, with the near-uniform T = 4 setting losing 4.1pp. We study this finer, within-type sampling in a principled manner via temperature-scaled sampling, p(d) ∝ len(d) 1/T , (1) where T = 1 recovers le… view at source ↗
Figure 18
Figure 18. Figure 18: Length-proportional sampling (T = 1) is near-optimal. Validation average as a function of sampling temperature T in p(d) ∝ len(d) 1/T . Both sharpening (T < 1) and flattening (T > 1) degrade performance relative, with the near-uniform T = 4 setting losing 4.1pp. We study this finer, within-type sampling in a principled manner via temperature-scaled sampling, p(d) ∝ len(d) 1/T , (1) where T = 1 recovers le… view at source ↗
Figure 19
Figure 19. Figure 19: Online filtering adds negligible overhead to training. Left: Runtime for 50M tokens (2 nodes, 8 GPUs, 1 filter) across rejection percentiles—total spread is 3.5%, within normal run￾to-run variance. Center: Same measurement at 500M tokens, where the spread shrinks to 0.3%, confirming that any per-sample overhead is amortized over longer runs. Right: Scaling the number of simultaneous filters at 80% rejecti… view at source ↗
Figure 19
Figure 19. Figure 19: Online filtering adds negligible overhead to training. Left: Runtime for 50M tokens (2 nodes, 8 GPUs, 1 filter) across rejection percentiles—total spread is 3.5%, within normal run￾to-run variance. Center: Same measurement at 500M tokens, where the spread shrinks to 0.3%, confirming that any per-sample overhead is amortized over longer runs. Right: Scaling the number of simultaneous filters at 80% rejecti… view at source ↗
Figure 20
Figure 20. Figure 20: The optimal pretraining data mixture is scale-dependent. At the small scale, performance is flat or slightly degrades as the instruction-tuning share grows; at the medium scale, the trend reverses, with performance improving monotonically and peaking at 65% instruction-tuning data. This highlights the need for scale-aware data curation. We find that the optimal data mixture at the small scale is the 65% c… view at source ↗
Figure 20
Figure 20. Figure 20: The optimal pretraining data mixture is scale-dependent. At the small scale, performance is flat or slightly degrades as the instruction-tuning share grows; at the medium scale, the trend reverses, with performance improving monotonically and peaking at 65% instruction-tuning data. This highlights the need for scale-aware data curation. We find that the optimal data mixture at the small scale is the 65% c… view at source ↗
Figure 21
Figure 21. Figure 21: Streaming best-fit sequence packing. (a) Incoming samples are assigned online to a pool of open buffers kept sorted from fullest to emptiest. A sample Si is placed in the first (hence fullest) buffer whose token budget L and image-tile budget M both still accommodate it. Here b1 would overflow L, so Si lands in b2. A buffer is emitted once it hits either budget exactly (or when the pool overflows its size… view at source ↗
Figure 21
Figure 21. Figure 21: Streaming best-fit sequence packing. (a) Incoming samples are assigned online to a pool of open buffers kept sorted from fullest to emptiest. A sample Si is placed in the first (hence fullest) buffer whose token budget L and image-tile budget M both still accommodate it. Here b1 would overflow L, so Si lands in b2. A buffer is emitted once it hits either budget exactly (or when the pool overflows its size… view at source ↗
Figure 22
Figure 22. Figure 22: replicates the analysis of Section 4.3 using Mammoth-VL-12M [89] as the SFT dataset in place of LLaVA-665K [151]. Results are fully consistent: pretraining and post-SFT scores remain near-perfectly correlated (Pearson r=0.99; Spearman ρ=0.99), and the pretraining ranking is preserved across all 27 checkpoints. These results confirm that the conclusion is robust to the choice of SFT dataset: a stronger pre… view at source ↗
Figure 22
Figure 22. Figure 22: replicates the analysis of Section 4.3 using Mammoth-VL-12M [89] as the SFT dataset in place of LLaVA-665K [151]. Results are fully consistent: pretraining and post-SFT scores remain near-perfectly correlated (Pearson r=0.99; Spearman ρ=0.99), and the pretraining ranking is preserved across all 27 checkpoints. These results confirm that the conclusion is robust to the choice of SFT dataset: a stronger pre… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. CAPEval: A Decoupled Caption Evaluation across Understanding and Generation

    cs.CV 2026-08 conditional novelty 5.0

    Caption quality can be split into Coverage and Precision; in controlled training runs, Coverage predicts VLM understanding while Precision predicts T2I generation.

Reference graph

Works this paper leans on

299 extracted references · 299 canonical work pages · cited by 1 Pith paper · 58 internal anchors

  1. [1]

    SemDeDup: Data-efficient learning at web-scale through semantic deduplication

    A. Abbas, K. Tirumala, D. Simig, S. Ganguli, and A. S. Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication.arXiv preprint arXiv:2303.09540, 2023. Cited on page 45

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  2. [2]

    Abbas, E

    A. Abbas, E. Rusak, K. Tirumala, W. Brendel, K. Chaudhuri, and A. S. Morcos. Effective pruning of web-scale datasets based on complexity of concept clusters.arXiv preprint arXiv:2401.04578, 2024. Cited on page 45

    work page arXiv 2024

  3. [3]

    Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs

    A. Abouelenin, A. Ashfaq, A. Atkinson, H. Awadalla, N. Bach, J. Bao, A. Benhaim, M. Cai, V . Chaudhary, C. Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras.arXiv preprint arXiv:2503.01743, 2025. Cited on page 2

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  4. [4]

    Acharya, K

    M. Acharya, K. Kafle, and C. Kanan. TallyQA: Answering complex counting questions. In AAAI Conference on Artificial Intelligence (AAAI), 2019. Cited on pages 53 and 54

    work page 2019

  5. [5]

    Agnolucci, L

    L. Agnolucci, L. Galteri, M. Bertini, and A. Del Bimbo. Arniqa: Learning distortion manifold for image quality assessment. InIEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 189–198, 2024. Cited on pages 45 and 75

    work page 2024

  6. [6]

    Ainslie, J

    J. Ainslie, J. Lee-Thorp, M. De Jong, Y . Zemlyanskiy, F. Lebrón, and S. Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. InConference on Empirical Methods in Natural Language Processing (EMNLP), pages 4895–4901, 2023. Cited on page 48

    work page 2023

  7. [7]

    S. N. Akter, S. Prabhumoye, E. Nyberg, M. Patwary, M. Shoeybi, Y . Choi, and B. Catanzaro. Front-loading reasoning: The synergy between pretraining and post-training data.arXiv preprint arXiv:2510.03264, 2025. Cited on page 8

    work page arXiv 2025

  8. [8]

    Alayrac, J

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems (NeurIPS), 35:23716–23736, 2022. Cited on pages 2 and 45

    work page 2022

  9. [9]

    L. B. Allal, A. Lozhkov, E. Bakouch, G. M. Blázquez, G. Penedo, L. Tunstall, A. Marafioti, H. Kydlíˇcek, A. P. Lajarín, V . Srivastav, et al. Smollm2: When smol goes big–data-centric training of a small language model.arXiv preprint arXiv:2502.02737, 2025. Cited on pages 6 and 46

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  10. [10]

    Allen-Zhu and Y

    Z. Allen-Zhu and Y . Li. Physics of language models: Part 3.1, knowledge storage and extraction.arXiv preprint arXiv:2309.14316, 2023. Cited on page 8

    work page arXiv 2023

  11. [11]

    M ath QA : Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms

    A. Amini, S. Gabriel, S. Lin, R. Koncel-Kedziorski, Y . Choi, and H. Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In J. Burstein, C. Doran, and T. Solorio, editors,Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol...

    work page doi:10.18653/v1/n19-1245 2019

  12. [12]

    X. An, Y . Xie, K. Yang, W. Zhang, X. Zhao, Z. Cheng, Y . Wang, S. Xu, C. Chen, D. Zhu, et al. Llava-onevision-1.5: Fully open framework for democratized multimodal training.arXiv preprint arXiv:2509.23661, 2025. Cited on pages 2, 3, 9, and 45. 12

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  13. [13]

    Ankner, C

    Z. Ankner, C. Blakeney, K. Sreenivasan, M. Marion, M. L. Leavitt, and M. Paul. Perplexed by perplexity: Perplexity-based data pruning with small reference models.arXiv preprint arXiv:2405.20541, 2024. Cited on page 5

    work page arXiv 2024

  14. [14]

    Awadalla, L

    A. Awadalla, L. Xue, O. Lo, M. Shu, H. Lee, E. Guha, M. Jordan, S. Shen, M. Awadalla, S. Savarese, et al. Mint-1t: Scaling open-source multimodal data by 10x: A multimodal dataset with one trillion tokens.Advances in Neural Information Processing Systems (NeurIPS), 37: 36805–36828, 2024. Cited on pages 4, 53, 54, 55, and 76

    work page 2024

  15. [15]

    C. Baek, R. P. Monti, D. Schwab, A. Abbas, R. Adiga, C. Blakeney, M. Böther, P. Burstein, A. G. Carranza, A. Deng, et al. The finetuner’s fallacy: When to pretrain with your finetuning data.arXiv preprint arXiv:2603.16177, 2026. Cited on page 8

    work page arXiv 2026

  16. [16]

    S. Bai, Y . Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025. Cited on pages 2, 3, and 4

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  17. [17]

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y . Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y . Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin. Qwen2.5-vl technical report, 2025. URLhttps://arxiv.org/abs/2502.13923. Cited on pages 2, 3, and 62

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  18. [18]

    Berasi, M

    D. Berasi, M. Farina, M. Mancini, and E. Ricci. Linear model merging unlocks simple and scalable multimodal data mixture optimization.arXiv preprint arXiv:2602.04937, 2026. Cited on pages 3 and 45

    work page arXiv 2026

  19. [19]

    Bevli, S

    A. Bevli, S. Chaybouti, Y . Dahou, H. Hacid, N. D. Huynh, P. H. L. Khac, S. Narayan, W. R. Para, and A. Singh. Falcon perception.arXiv preprint arXiv:2603.27365, 2026. Cited on page 2

    work page arXiv 2026

  20. [20]

    L. Beyer. On the speed of ViTs and CNNs.http://lb.eyer.be/a/vit-cnn-speed.html, 2024. Cited on page 68

    work page 2024

  21. [21]

    PaliGemma: A versatile 3B VLM for transfer

    L. Beyer, A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang, D. Salz, M. Neumann, I. Alabdulmohsin, M. Tschannen, E. Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer.arXiv preprint arXiv:2407.07726, 2024. Cited on pages 2, 45, and 46

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  22. [22]

    A. F. Biten, R. Tito, A. Mafla, L. Gomez, M. Rusinol, E. Valveny, C. Jawahar, and D. Karatzas. Scene text visual question answering. InIEEE/CVF International Conference on Computer Vision (ICCV), pages 4291–4301, 2019. Cited on pages 53 and 54

    work page 2019

  23. [23]

    Bordt, S

    S. Bordt, S. Srinivas, V . Boreiko, and U. V on Luxburg. How much can we forget about data contamination?arXiv preprint arXiv:2410.03249, 2024. Cited on page 46

    work page arXiv 2024

  24. [24]

    Breuel and WebDataset Contributors

    T. Breuel and WebDataset Contributors. WebDataset: A high-performance Python-based I/O system for large (and small) deep learning problems, with strong support for PyTorch. https://github.com/webdataset/webdataset, 2020. Cited on page 85

    work page 2020

  25. [25]

    A. Z. Broder. On the resemblance and containment of documents. InProceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pages 21–29. IEEE, 1997. Cited on pages 4, 46, and 69. 13

    work page 1997

  26. [26]

    Cahyawijaya, H

    S. Cahyawijaya, H. Lovenia, J. R. A. Moniz, T. H. Wong, M. R. Farhansyah, T. T. Maung, F. Hudi, D. Anugraha, M. R. S. Habibi, M. R. Qorib, et al. Crowdsource, crawl, or generate? creating sea-vl, a multicultural vision-language dataset for southeast asia. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: ...

    work page 2025

  27. [27]

    Cao and J

    J. Cao and J. Xiao. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. InInternational Conference on Computational Linguistics (COLING), 2022. Cited on pages 53 and 54

    work page 2022

  28. [28]

    Carlini, D

    N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang. Quantifying memorization across neural language models. InInternational Conference on Learning Representations (ICLR), 2022. Cited on page 8

    work page 2022

  29. [29]

    J. Carter. TextOCR-GPT4V: A re-captioning of TextOCR with GPT-4V, 2024. Hugging Face dataset card,https://huggingface.co/datasets/jimmycarter/textocr-gpt4v. Cited on pages 53 and 54

    work page 2024

  30. [30]

    Chang, D

    S. Chang, D. Palzer, J. Li, E. Fosler-Lussier, and N. Xiao. MapQA: A dataset for question answering on choropleth maps.arXiv preprint arXiv:2211.08545, 2022. Cited on pages 53 and 54

    work page arXiv 2022

  31. [31]

    Changpinyo, P

    S. Changpinyo, P. Sharma, N. Ding, and R. Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3558–3568, 2021. Cited on page 45

    work page 2021

  32. [32]

    G. H. Chen, S. Chen, R. Zhang, J. Chen, X. Wu, Z. Zhang, Z. Chen, J. Li, X. Wan, and B. Wang. ALLaV A: Harnessing GPT4V-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.11684, 2024. Cited on pages 53 and 54

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  33. [33]

    J. Chen, T. Li, J. Qin, P. Lu, L. Lin, C. Chen, and X. Liang. UniGeo: Unifying geometry logical reasoning via reformulating mathematical expression. InConference on Empirical Methods in Natural Language Processing (EMNLP), 2022. Cited on pages 53 and 54

    work page 2022

  34. [34]

    L. Chen, J. Li, X. Dong, P. Zhang, C. He, J. Wang, F. Zhao, and D. Lin. ShareGPT4V: Improving large multi-modal models with better captions. InEuropean Conference on Computer Vision (ECCV), pages 370–387. Springer, 2024. Cited on pages 53, 54, and 55

    work page 2024

  35. [35]

    L. Chen, J. Li, X. Dong, P. Zhang, Y . Zang, Z. Chen, H. Duan, J. Wang, Y . Qiao, D. Lin, et al. Are we on the right way for evaluating large vision-language models? InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. Cited on page 66

    work page 2024

  36. [36]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021. Cited on page 66

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  37. [37]

    M. F. Chen, T. Murray, D. Heineman, M. Jordan, H. Hajishirzi, C. Ré, L. Soldaini, and K. Lo. Olmix: A framework for data mixing throughout lm development.arXiv preprint arXiv:2602.12237, 2026. Cited on pages 3, 45, and 93

    work page arXiv 2026

  38. [38]

    W. Chen, M. Yin, M. Ku, P. Lu, Y . Wan, X. Ma, J. Xu, X. Wang, and T. Xia. Theoremqa: A theorem-driven question answering dataset. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7889–7901, 2023. Cited on page 66. 14

    work page 2023

  39. [39]

    X. Chen, H. Fang, T.-Y . Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick. Microsoft COCO captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325, 2015. Cited on pages 53, 54, and 66

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  40. [40]

    Y . Chen, S. Qian, H. Tang, X. Lai, Z. Liu, S. Han, and J. Jia. LongloRA: Efficient fine-tuning of long-context large language models. InInternational Conference on Learning Representations (ICLR), 2024. URLhttps://openreview.net/forum?id=6PmJoRfdaK. Cited on pages 53 and 54

    work page 2024

  41. [41]

    Z. Chen, W. Wang, Y . Cao, Y . Liu, Z. Gao, E. Cui, J. Zhu, S. Ye, H. Tian, Z. Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024. Cited on pages 1, 4, 5, 47, 50, 52, 54, 62, and 86

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  42. [42]

    Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. Cited on pages 7, 45, 47, 53, and 54

    work page 2024

  43. [43]

    C. K. Chng, Y . Liu, Y . Sun, C. C. Ng, C. Luo, Z. Ni, C. Fang, S. Zhang, J. Han, E. Ding, et al. ICDAR2019 robust reading challenge on arbitrary-shaped text - RRC-ArT. InInternational Conference on Document Analysis and Recognition (ICDAR), 2019. Cited on pages 53 and 54

    work page 2019

  44. [44]

    J. H. Cho, A. Madotto, E. Mavroudi, T. Afouras, T. Nagarajan, M. Maaz, Y . Song, T. Ma, S. Hu, S. Jain, et al. Perceptionlm: Open-access data and models for detailed visual understanding. arXiv preprint arXiv:2504.13180, 2025. Cited on pages 2 and 86

    work page arXiv 2025

  45. [45]

    Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding

    C. Clark, J. Zhang, Z. Ma, J. S. Park, M. Salehi, R. Tripathi, S. Lee, Z. Ren, C. D. Kim, Y . Yang, et al. Molmo2: Open weights and data for vision-language models with video understanding and grounding.arXiv preprint arXiv:2601.10611, 2026. Cited on pages 3, 45, and 77

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  46. [47]

    Training Verifiers to Solve Math Word Problems

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021. Cited on page 66

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  47. [48]

    Conover, M

    M. Conover, M. Hayes, A. Mathur, J. Xie, J. Wan, S. Shah, A. Ghodsi, P. Wendell, M. Zaharia, and R. Xin. Free Dolly: Introducing the world’s first truly open instruction- tuned LLM, 2023. Databricks Blog https://www.databricks.com/blog/2023/04/12/ dolly-first-open-commercially-viable-instruction-tuned-llm. Cited on pages 4, 53, and 54

    work page 2023

  48. [49]

    Contributors

    O. Contributors. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass, 2023. Cited on page 62

    work page 2023

  49. [50]

    M. R. Costa-Jussà, J. Cross, O. Çelebi, M. Elbayad, K. Heafield, K. Heffernan, E. Kalbassi, J. Lam, D. Licht, J. Maillard, et al. No language left behind: Scaling human-centered machine translation.arXiv preprint arXiv:2207.04672, 2022. Cited on page 59

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  50. [51]

    E. Cui, Y . He, Z. Ma, Z. Chen, H. Tian, W. Wang, K. Li, Y . Wang, W. Wang, X. Zhu, L. Lu, T. Lu, Y . Wang, L. Wang, Y . Qiao, and J. Dai. Sharegpt-4o: Comprehensive multimodal annotations with gpt-4o, 2024. URLhttps://sharegpt4o.github.io/. Cited on page 3. 15

    work page 2024

  51. [52]

    G. Cui, L. Yuan, N. Ding, G. Yao, B. He, W. Zhu, Y . Ni, G. Xie, R. Xie, Y . Lin, Z. Liu, and M. Sun. UltraFeedback: Boosting language models with scaled AI feedback.International Conference on Machine Learning (ICML), 2024. Cited on pages 53 and 54

    work page 2024

  52. [53]

    D. Dai, Y . Li, Y . Liu, M. Jia, Z. YuanHui, and G. Wang. 15M multimodal facial image-text dataset.arXiv preprint arXiv:2407.08515, 2024. Cited on pages 53 and 54

    work page arXiv 2024

  53. [54]

    T. Dao. Flashattention-2: Faster attention with better parallelism and work partitioning.arXiv preprint arXiv:2307.08691, 2023. Cited on page 47

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  54. [55]

    A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. M. Moura, D. Parikh, and D. Batra. Visual dialog. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017. Cited on pages 53 and 54

    work page 2017

  55. [56]

    J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, et al. Large scale distributed deep networks.Advances in Neural Information Processing Systems (NeurIPS), 25, 2012. Cited on page 50

    work page 2012

  56. [57]

    Deitke, C

    M. Deitke, C. Clark, S. Lee, R. Tripathi, Y . Yang, J. S. Park, M. Salehi, N. Muennighoff, K. Lo, L. Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art vision- language models. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 91–104, 2025. Cited on pages 2, 3, 45, 53, 54, 55, 77, and 86

    work page 2025

  57. [58]

    A. S. Deshmukh, K. Chumachenko, T. Rintamaki, M. Le, T. Poon, D. M. Taheri, I. Karmanov, G. Liu, J. Seppanen, G. Chen, et al. Nvidia nemotron nano v2 vl.arXiv preprint arXiv:2511.03929, 2025. Cited on pages 2, 3, 9, 45, and 86

    work page arXiv 2025

  58. [59]

    S. Diao, Y . Yang, Y . Fu, X. Dong, D. Su, M. Kliegl, Z. Chen, P. Belcak, Y . Suhara, H. Yin, et al. Nemotron-climb: Clustering-based iterative data mixture bootstrapping for language model pre-training.arXiv preprint arXiv:2504.13161, 2025. Cited on pages 3, 45, and 77

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  59. [60]

    N. Ding, Y . Chen, B. Xu, Y . Qin, S. Hu, Z. Liu, M. Sun, and B. Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In H. Bouamor, J. Pino, and K. Bali, editors,Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3029–3051, Singapore, Dec. 2023. Association for Computational Linguistics. doi...

    work page 2023

  60. [61]

    Dodge, M

    J. Dodge, M. Sap, A. Marasovi ´c, W. Agnew, G. Ilharco, D. Groeneveld, M. Mitchell, and M. Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. InProceedings of the 2021 conference on empirical methods in natural language processing, pages 1286–1305, 2021. Cited on page 94

    work page 2021

  61. [62]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020. Cited on page 47

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  62. [63]

    Douze, A

    M. Douze, A. Guzhva, C. Deng, J. Johnson, G. Szilvasy, P.-E. Mazaré, M. Lomeli, L. Hosseini, and H. Jégou. The faiss library.IEEE Transactions on Big Data, 2025. Cited on page 68

    work page 2025

  63. [64]

    H. Duan, J. Yang, Y . Qiao, X. Fang, L. Chen, Y . Liu, X. Dong, Y . Zang, P. Zhang, J. Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. InACM International Conference on Multimedia, pages 11198–11201, 2024. Cited on pages 2 and 62. 16

    work page 2024

  64. [65]

    Evans, N

    T. Evans, N. Parthasarathy, H. Merzi ´c, and O. J. Henaff. Data curation via joint example selection further accelerates multimodal learning.Advances in Neural Information Processing Systems (NeurIPS), 37:141240–141260, 2024. Cited on pages 5 and 6

    work page 2024

  65. [66]

    L. Fan, D. Krishnan, P. Isola, D. Katabi, and Y . Tian. Improving clip training with language rewrites.Advances in Neural Information Processing Systems (NeurIPS), 36:35544–35575, 2023. Cited on page 45

    work page 2023

  66. [67]

    A. Fang, G. Ilharco, M. Wortsman, Y . Wan, V . Shankar, A. Dave, and L. Schmidt. Data determines distributional robustness in contrastive language image pre-training (clip). In International Conference on Machine Learning (ICML), pages 6216–6234. PMLR, 2022. Cited on page 1

    work page 2022

  67. [68]

    A. Fang, A. M. Jose, A. Jain, L. Schmidt, A. Toshev, and V . Shankar. Data filtering networks. arXiv preprint arXiv:2309.17425, 2023. Cited on pages 5 and 45

    work page arXiv 2023

  68. [69]

    A. Fang, H. Pouransari, M. Jordan, A. Toshev, V . Shankar, L. Schmidt, and T. Gunter. Datasets, documents, and repetitions: The practicalities of unequal data quality.arXiv preprint arXiv:2503.07879, 2025. Cited on page 8

    work page arXiv 2025

  69. [70]

    L. Feng, G. R. Ghosal, J. M. Springer, Z. Zhong, and A. Raghunathan. Early data exposure improves robustness to subsequent fine-tuning.arXiv preprint arXiv:2605.12705, 2026. Cited on page 8

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  70. [71]

    C. Fu, P. Chen, Y . Shen, Y . Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023. Cited on pages 65 and 66

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  71. [72]

    X. Fu, Y . Hu, B. Li, Y . Feng, H. Wang, X. Lin, D. Roth, N. A. Smith, W.-C. Ma, and R. Krishna. Blink: Multimodal large language models can see but not perceive. InEuropean Conference on Computer Vision, pages 148–166. Springer, 2024. Cited on page 66

    work page 2024

  72. [73]

    S. Y . Gadre, G. Ilharco, A. Fang, J. Hayase, G. Smyrnis, T. Nguyen, R. Marten, M. Wortsman, D. Ghosh, J. Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems (NeurIPS), 36:27092–27112, 2023. Cited on pages 1, 3, 5, 6, 45, 53, 54, 55, 59, 78, 79, and 89

    work page 2023

  73. [74]

    L. Gao. An empirical exploration in quality filtering of text data.arXiv preprint arXiv:2109.00698, 2021. Cited on page 7

    work page arXiv 2021

  74. [75]

    L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027, 2020. Cited on page 46

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  75. [76]

    Gervais, A

    P. Gervais, A. Fadeeva, and A. Maksai. Mathwriting: A dataset for handwritten mathematical expression recognition. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V .2, KDD ’25, page 5459–5469, New York, NY , USA, 2025. Association for Computing Machinery. ISBN 9798400714542. doi: 10.1145/3711896.3737436. URLhttps://...

    work page doi:10.1145/3711896.3737436 2025

  76. [77]

    arXiv preprint arXiv:2403.03864 , year=

    D. Ghosal, V . T. Y . Han, C. Y . Ken, and S. Poria. Are language models puzzle prodigies? Algorithmic puzzles unveil serious challenges in multimodal reasoning.arXiv preprint arXiv:2403.03864, 2024. Cited on pages 53 and 54. 17

    work page arXiv 2024

  77. [78]

    Ghosh, S

    A. Ghosh, S. Dziadzio, A. Prabhu, V . Udandarao, S. Albanie, and M. Bethge. Onebench to test them all: Sample-level benchmarking over open-ended capabilities. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32445–32481, 2025. Cited on page 2

    work page 2025

  78. [79]

    Ghosh, V

    A. Ghosh, V . Udandarao, T. Nguyen, M. Farina, M. Cherti, J. Jitsev, S. Oh, E. Ricci, L. Schmidt, and M. Bethge. Concept-aware batch sampling improves language-image pretraining.arXiv preprint arXiv:2511.20643, 2025. Cited on pages 1, 45, 78, and 79

    work page arXiv 2025

  79. [80]

    Glaive-Code-Assistant, 2023

    Glaive AI. Glaive-Code-Assistant, 2023. https://huggingface.co/datasets/ glaiveai/glaive-code-assistant. Cited on pages 53 and 54

    work page 2023

  80. [81]

    Goyal, P

    S. Goyal, P. Maini, Z. C. Lipton, A. Raghunathan, and J. Z. Kolter. Scaling laws for data filtering–data curation cannot be compute agnostic. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22702–22711, 2024. Cited on pages 4, 7, 46, and 82

    work page 2024

Showing first 80 references.