{"id":"d610a580-1c6b-41ca-b852-a92124266557","arxiv_id":"2607.22043","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"In models trained from scratch on text plus images, the text-objective scaling law is data-mix-invariant while the image-conditioned objective shifts toward many more tokens relative to parameters as the multimodal share grows.","lead":"This paper measures how to split a fixed computing budget between model size and training data when a vision–language model is trained from scratch on a mix of text and image–text data. It finds that the best split for the text-only part of the loss does not depend on the data mix, while the best split for the image-dependent part does — yielding a practical recipe for model size, token counts, and data-mix ratio.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed language/multimodal asymmetry is not supported by the paper's own envelope numbers: text and multimodal allocation exponents decline by nearly identical amounts (~0.03), and the text IsoFLOP decline (0.697→0.663) is dismissed as noise while a comparable multimodal decline (0.709→0.643) i","rationale":"The reader's weakest assumption focuses on the decoupled compute split Ctext=6NDtext, Cmm=6NDmm. That is a real concern, but it is less decisive than the internal inconsistency in how the exponent declines are interpreted: multiplicative rescaling of the x-axis by (1+r) shifts intercepts, not slopes, so the decoupling alone would not change the exponent-vs-r pattern. The more load-bearing issue is that the paper's own envelope estimator shows essentially equal declines for the two objectives, and the text IsoFLOP decline is also positive, yet only the multimodal one is treated as a law. Since the central claim is the asymmetry, and the paper provides no uncertainty quantification and no released code/data to check it, the current evidence is insufficient. The verdict should move from CONDITIONAL to REJECT unless the authors can show, via the proposed bootstrap/differential analysis, that text exponents are statistically flat while multimodal exponents decline. This is a good-faith reading: the methodology is standard and the downstream results are of interest, but the headline contribution is not established by the reported numbers.","tokens_in":28643,"tokens_out":7081,"duration_ms":79864,"concrete_test":"Reanalyze the raw training curves/checkpoints behind Figures 2 and 5. For each objective and each r, estimate a(r) by the envelope procedure, then bootstrap 95% confidence intervals by resampling model runs (or by residual resampling around the fitted envelope). Fit a(r) = a0 + α·r separately for text and multimodal exponents and test the null α_text = α_mm, using both the envelope and IsoFLOP estimators. If the confidence intervals for α_text and α_mm overlap substantially, or if |α_text| is not significantly smaller than |α_mm|, the claimed asymmetry is an artifact of selective interpretation rather than a scaling law.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that language allocation is composition-invariant while multimodal allocation is composition-sensitive. The evidence for this asymmetry rests entirely on the fitted exponents a(r). From the paper's own figures: text envelope a-values are 0.671, 0.643, 0.644, 0.639 (decline ≈0.032); multimodal envelope a-values are 0.665, 0.641, 0.634 (decline ≈0.031). These declines are virtually identical in magnitude. The text IsoFLOP a-values decline from 0.697 to 0.663 (≈0.034), while multimodal IsoFLOP declines from 0.709 to 0.643 (≈0.066). Thus the text envelope shows a decline as large as the multimodal envelope decline, and both text estimators actually decline. Section 3.1 nevertheless labels this 'minor numerical drift' and says the envelope shows 'no monotonic downward trend'; Section 3.2 labels a similar-magnitude envelope decline a 'sharp downward trajectory' and a 'robust scaling law.' No confidence intervals or bootstrap uncertainties are given anywhere, and the multimodal fit uses only three r values (0.1, 0.2, 0.3) while the text fit includes r=0. This is an internal inconsistency: the paper applies different evidentiary standards to the two objectives. The joint Pareto analysis (§3.3) then hard-codes the text-invariance it is claiming to discover, so it cannot validate the asymmetry. If the text and multimodal exponent declines are statistically indistinguishable, the paper's headline asymmetry disappears.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies compute-optimal scaling for native multimodal pre-training from scratch. It decouples the training budget by modality (Ctext=6NDtext, Cmm=6NDmm, with Dtext=D/(1+r), Dmm=rD/(1+r)) and uses IsoFLOP profiles and training-curve envelopes to estimate allocation exponents a and b in Nopt∝C^a, Dopt∝C^b for each objective. The central claim is that the language allocation law is largely invariant to the multimodal ratio r, while the multimodal allocation law is highly composition-sensitive; the paper then derives a joint Pareto frontier and reports downstream text preservation, cross-modal transfer to spatial reasoning, and multimodal in-context learning. The empirical core is the asymmetry between the two objectives' scaling exponents as r varies over {0,0.1,0.2,0.3}.","tokens_in":28989,"tokens_out":6337,"duration_ms":62165,"significance":"If the claimed language/multimodal asymmetry were established, the paper would provide practically valuable guidance for allocating parameters and tokens in native multimodal pre-training, and the two-estimator design plus the broad downstream benchmark suite are genuine strengths. However, the present manuscript does not establish the asymmetry: the same estimator that allegedly shows a sharp multimodal decline shows a nearly identical envelope decline for text, no uncertainty intervals are reported, and the joint Pareto model hard-codes the invariance it claims to discover. The decoupling assumption and the undefined multimodal loss further undermine the central conclusion. With substantial revision, the study could become a useful contribution, but the current evidence is not sufficient for acceptance.","major_comments":[{"comment":"The central asymmetry is not supported by the paper's own envelope estimates. Text envelope a-values are 0.671 (r=0), 0.643 (r=0.1), 0.644 (r=0.2), 0.639 (r=0.3), a decline of ≈0.032; multimodal envelope a-values are 0.665 (r=0.1), 0.641 (r=0.2), 0.634 (r=0.3), a decline of ≈0.031. These declines are virtually identical in magnitude. Section 3.1 dismisses the text decrease as 'minor numerical drift' with 'no monotonic downward trend', while Section 3.2 calls the same-size multimodal decrease a 'sharp downward trajectory' and a 'robust scaling law.' No bootstrap intervals or standard errors are provided anywhere, and the multimodal fits cover only three r values (0.1, 0.2, 0.3) whereas the text fits include r=0. The paper therefore applies different evidentiary standards to the two objectives; as written, the headline invariance/sensitivity dichotomy is an unsupported interpretation of th","section":"§3.1, §3.2, Figs. 2 and 5"},{"comment":"The joint Pareto analysis assumes the asymmetry rather than testing it. The text explicitly says the framework 'pair[s] a composition-invariant language objective with a composition-variant multimodal objective' and that forcing the text exponent to fluctuate would introduce 'overfitting and localized optimization noise.' Figures 7(b) and 7(c) then display text as r-free and multimodal as r-dependent, but these outputs are consequences of the modeling assumption, not independent evidence. To validate the central claim, the authors need to fit a model that allows both exponents to depend on r and compare it to the constrained model, e.g., with a likelihood-ratio or information criterion. As it stands, §3.3 cannot corroborate the language-invariance result.","section":"§3.3"},{"comment":"The decoupling premise is load-bearing and is not tested. Equation (1) and the surrounding text define Ctext and Cmm by splitting the token budget according to r, as if each objective were trained under a separate compute budget despite fully shared parameters. All invariance/variance conclusions live inside this coordinate system. Moreover, Appendix A states that 'the training loss is computed only on text tokens, with vision tokens masked out from loss computation,' yet the paper analyzes a multimodal objective Lmm in Figures 4–6. The operational definition of Lmm is never given, and if vision tokens do not contribute to the loss, it is unclear what objective the multimodal scaling law describes. A concrete test of the decoupling assumption would fit Ltext as a function of Ctext and r (or Dmm) and test whether r enters significantly; the same should be done for Lmm. Without such a test","section":"§2.1, Appendix A"},{"comment":"No uncertainty quantification is provided for any fitted quantity—exponents a,b, the frontier parameters E, Cc, β, or the projected Pareto frontier. The paper extrapolates from compute budgets around 10^19–10^21 to 10^25 in Figure 7(a), but these 'predictions' are simply the fitted power laws of Eqs. (2)–(3) evaluated at larger C; they are not validated on held-out budgets. The authors should report standard errors or bootstrap confidence intervals for the exponent differences (especially the text-vs-multimodal declines) and perform an out-of-sample check, such as fitting on lower compute budgets and predicting a held-out higher budget. Without this, the sharpness of the claimed asymmetry and the reliability of the efficiency frontier cannot be assessed.","section":"§2.2, §3.3, Figs. 2–7"}],"minor_comments":[{"comment":"Equation (3) is called a 'power law,' but the additive irreducible loss E makes it a shifted power law. Please adjust the terminology for precision.","section":"Eq. (3)"},{"comment":"The panel labels '(flat)' and '(vary with r)' in Figures 3(c) and 6(c) are interpretive conclusions, not neutral descriptions. Consider relabeling them descriptively (e.g., 'exponent estimates by r').","section":"Figs. 3 and 6"},{"comment":"The benchmark name 'MMBenchen' appears to be a typo for 'MMBench-en.' Please correct it in Tables 6–8.","section":"Appendix B"},{"comment":"Figure 8(b) labels the horizontal axis 'Text tokens Ntext (B)', but tokens are usually denoted D in this paper; please use consistent notation.","section":"Fig. 8"},{"comment":"The cross-modal transfer claim in Section 4.2 is based on two SpatialEval subtasks, and no significance testing or variance across seeds is reported. This is a supporting result, but the 'widening gap' statement should be softened or supplemented with error bars.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope and the experimental effort is substantial, but the central asymmetry claim is not supported by the reported numbers. The authors need to either provide uncertainty-aware reanalysis showing a statistically distinguishable difference, or substantially reframe the conclusions. The undefined multimodal loss in Appendix A is a particularly serious gap that should be resolved before the paper can be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a real attempt to bring Chinchilla-style scaling-law estimation to native multimodal pre-training, and the per-objective decomposition is a genuine new angle. But the paper's central claim — that the language allocation law is composition-invariant while the multimodal law is composition-variant — is not supported by the numbers in the paper itself.\n\nThe envelope exponents for text are 0.671, 0.643, 0.644, 0.639 across r=0 to 0.3; for multimodal they are 0.665, 0.641, 0.634 across r=0.1 to 0.3. Both decline by about 0.03. Section 3.1 calls the text decline 'minor numerical drift' and says the envelope shows no monotonic trend; Section 3.2 calls a nearly identical decline a 'sharp downward trajectory' and a 'robust scaling law.' No confidence intervals or bootstrap uncertainties are given anywhere. With four points for text and three for multimodal, those exponent differences are exactly the kind of thing that could be fitting noise. The IsoFLOP declines are larger for multimodal (0.066 vs 0.034), and that is the best evidence for the asymmetry, but it still needs error bars before I believe it.\n\nThe joint Pareto analysis in Section 3.3 doesn't rescue the claim — it builds text-invariance into the model in advance. So that section confirms its own assumption, not the data.\n\nTwo more soft spots. First, the decoupling premise: C_text = 6ND/(1+r) and C_mm = 6NDr/(1+r) treats the two objectives as if trained on separate budgets, but parameters are shared. The paper should state explicitly that this is a modeling choice, not a fact about optimization. Also, Appendix A says loss is computed only on text tokens; the 'multimodal objective' is therefore presumably text loss on image-conditioned tokens. That definition belongs in the main text, because it changes how to read the entire asymmetry.\n\nSecond, the cross-modal transfer result in Section 4.2 is confounded by total tokens: r=0.3 models train on 325B tokens, r=0 on 250B. An equal-total-token text-only control would make the spatial-reasoning claim solid.\n\nCredit where it's due: the scaling-law pipeline is faithful to Hoffmann et al., the checkpoint-wise tables are extensive, the limitation section is honest, and the in-context-learning emergence with scale and data is a clean observational finding.\n\nVerdict: worth a serious referee. I would not desk-reject it, but I would ask for uncertainty quantification on the exponents, an equal-token control for the transfer claim, and a joint model that does not hard-code the asymmetry. Without those, the headline should be downgraded to 'not conclusively distinguished.'","headline":"A competent per-objective scaling study whose headline asymmetry is not supported by its own envelope numbers — text and multimodal exponents decline by similar amounts, and the joint model assumes the conclusion.","tokens_in":29598,"tokens_out":3993,"would_cite":false,"duration_ms":37554,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper establishes that in native multimodal pre-training, the compute-optimal allocation for language is invariant to data composition, while the multimodal allocation shifts sharply toward more tokens as the image-text ratio rises.","keywords":["scaling laws","native multimodal pre-training","compute-optimal allocation","vision-language models","data composition","cross-modal transfer","in-context learning","IsoFLOP"],"falsifier":"Train two models at the same effective text compute C_text and model size N, one with r=0.1 and one with r=0.3, and compare the final text loss Ltext at identical (N, D_text); a systematic difference beyond noise would falsify the composition-invariant language law.","tokens_in":28387,"feed_emoji":"📈","tokens_out":9421,"duration_ms":79807,"temperature":0.7,"pith_summary":"This paper attempts to establish that compute-optimal scaling of native multimodal pre-training—training a vision-language model from scratch on mixed text and image data—is governed by two distinct laws rather than one. For the language objective, the optimal split of a fixed compute budget between model size and token count is nearly unchanged as the multimodal data ratio r rises from 0 to 0.3. For the multimodal objective, the same split is strongly composition-dependent: the optimal model-size exponent falls from about 0.71 at r=0.1 to about 0.64 at r=0.3, pushing compute-optimal training toward more tokens and fewer added parameters. Modeling this dependence yields an efficiency frontier that specifies, for any compute budget and data mixture, the optimal model size plus text and image token counts. If correct, this turns the design of native multimodal models from guesswork into a predictable engineering recipe, and it implies that the two modalities should be scaled under different rules.","feed_headline":"Adding image tokens changes the scaling law for vision, not for text","feed_subtitle":"A new efficiency frontier tells you how to split any compute budget between model size, text tokens, and image tokens.","key_machinery":"The central machinery is the decoupled compute-optimal allocation: the joint objective is split into language loss Ltext and multimodal loss Lmm, each assigned its own effective compute C_text=6ND_text and C_mm=6ND_mm, with D_text=D/(1+r) and D_mm=rD/(1+r)—as if each objective trained under its own budget despite shared parameters. Two independent estimators—IsoFLOP profiles (parabola fit to loss vs log N at fixed compute) and training-curve envelopes (pooled lower envelope over all runs)—determine Nopt(C)∝C^a and Dopt(C)∝C^b. The load-bearing identity is a+b=1; its variation with r is the whole story. The joint Pareto analysis imposes an asymmetric model, pairing the invariant language law","core_discovery":"IsoFLOP profiles and training-curve envelopes show each objective's minimal loss follows a predictable compute law, with compute-optimal model size and token counts obeying Nopt(C)∝C^a and Dopt(C)∝C^b (a+b=1). The central discovery is an asymmetry: language allocation exponents stay nearly constant as the multimodal ratio r varies over {0,0.1,0.2,0.3}, while the multimodal exponent a declines from ≈0.71 (r=0.1) to ≈0.64 (r=0.3), flattening parameter scaling and pushing compute toward tokens. Pairing the invariant language law with the variant multimodal law traces a Pareto efficiency frontier: for any total compute, each point specifies optimal model size, text token count, and multimodal to","pith_inferences":["The invariance of the language law suggests the data ratio r could itself be a tunable knob, making the efficiency frontier imply an optimal r(C) strategy—at very large budgets, text-heavy mixtures may be preferred because parameter scaling is more compute-efficient; the paper does not derive this explicitly.","Because the text loss is masked to text tokens only, image tokens may act as context rather than direct training signal; a natural extension is to test whether the composition invariance survives changes in sequence-level interleaving at fixed r.","The concentration of in-context gains on spatial/relational tasks predicts that visual pretraining transfers to other abstract relational text tasks (geometry, graph reasoning, navigation); this is a testable downstream prediction beyond the paper's benchmarks.","The two-law picture could be an artifact of the decoupled budget split; comparing against a model trained with a single aggregate loss under the same compute would show whether the asymmetry is real or coordinate-induced."],"forward_implications":["The efficiency frontier gives a concrete recipe: for any total compute budget and any target multimodal ratio r, it specifies the model size, text token count, and multimodal token count that minimize combined loss.","The language allocation law is stable across r up to 0.3, so adding multimodal data does not force a re-optimization of text scaling or sacrifice text benchmark accuracy.","As r increases, compute-optimal training shifts from parameter scaling to token scaling; for image-heavy mixtures, extra tokens reduce loss more than extra parameters.","Native multimodal pre-training produces positive cross-modal transfer: abstract spatial reasoning in pure text improves with visual pretraining, and the gain widens with model scale.","Multimodal in-context learning emerges with scale and data: few-shot gains over zero-shot rise from near zero at 71M parameters to +2.43 points at 3B, concentrated on spatial and relational tasks."],"fun_headline_variants":["Vision scales differently: image tokens shift the law","Image tokens reshape scaling, text law stays put","Multimodal scaling asymmetry: text invariant, vision not","New efficiency frontier for native multimodal training","Compute split: model vs tokens vs image ratio"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper assumes the language loss depends only on the effective text compute C_text=6ND_text and the multimodal loss only on C_mm=6ND_mm, treating the shared parameters as if each objective had its own budget; if text learning actually depends on the full token mix, the claimed invariance and variance are artifacts of that split.","fun_headline_variants_meta":{"raw":{"variants":["Vision scales differently: image tokens shift the law","Image tokens reshape scaling, text law stays put","Multimodal scaling asymmetry: text invariant, vision not","New efficiency frontier for native multimodal training","Compute split: model vs tokens vs image ratio"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000468,"raw_usage":{"total_tokens":2208,"prompt_tokens":820,"completion_tokens":1388,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":1317}},"tokens_in":564,"tokens_out":1388,"duration_ms":10255,"temperature":1.0,"reasoning_tokens":1317,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T05:59:40.057126+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train two models at the same effective text compute C_text and model size N, one with r=0.1 and one with r=0.3, and compare the final text loss Ltext at identical (N, D_text); a systematic difference beyond noise would falsify the composition-invariant language law.","supporting_citations":[],"review_version":1}