Pith. sign in

REVIEW 4 major objections 5 minor 20 references

The paper establishes that in native multimodal pre-training, the compute-optimal allocation for language is invariant to data composition, while the multimodal allocation shifts sharply toward more tokens as the image-text ratio rises.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 05:59 UTC pith:Z7RM57C2

load-bearing objection A competent per-objective scaling study whose headline asymmetry is not supported by its own envelope numbers — text and multimodal exponents decline by similar amounts, and the joint model assumes the conclusion. the 4 major comments →

arxiv 2607.22043 v1 pith:Z7RM57C2 submitted 2026-07-24 cs.CL cs.CV

Scaling Native Multimodal Pre-Training From Scratch

Haoyuan Wu , Aoqi Wu , Hai Wang , Jiajia Wu , Jinxiang Ou , Bei Yu This is my paper
classification cs.CL cs.CV
keywords scaling lawsnative multimodal pre-trainingcompute-optimal allocationvision-language modelsdata compositioncross-modal transferin-context learningIsoFLOP
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper attempts to establish that compute-optimal scaling of native multimodal pre-training—training a vision-language model from scratch on mixed text and image data—is governed by two distinct laws rather than one. For the language objective, the optimal split of a fixed compute budget between model size and token count is nearly unchanged as the multimodal data ratio r rises from 0 to 0.3. For the multimodal objective, the same split is strongly composition-dependent: the optimal model-size exponent falls from about 0.71 at r=0.1 to about 0.64 at r=0.3, pushing compute-optimal training toward more tokens and fewer added parameters. Modeling this dependence yields an efficiency frontier that specifies, for any compute budget and data mixture, the optimal model size plus text and image token counts. If correct, this turns the design of native multimodal models from guesswork into a predictable engineering recipe, and it implies that the two modalities should be scaled under different rules.

Core claim

IsoFLOP profiles and training-curve envelopes show each objective's minimal loss follows a predictable compute law, with compute-optimal model size and token counts obeying Nopt(C)∝C^a and Dopt(C)∝C^b (a+b=1). The central discovery is an asymmetry: language allocation exponents stay nearly constant as the multimodal ratio r varies over {0,0.1,0.2,0.3}, while the multimodal exponent a declines from ≈0.71 (r=0.1) to ≈0.64 (r=0.3), flattening parameter scaling and pushing compute toward tokens. Pairing the invariant language law with the variant multimodal law traces a Pareto efficiency frontier: for any total compute, each point specifies optimal model size, text token count, and multimodal to

What carries the argument

The central machinery is the decoupled compute-optimal allocation: the joint objective is split into language loss Ltext and multimodal loss Lmm, each assigned its own effective compute C_text=6ND_text and C_mm=6ND_mm, with D_text=D/(1+r) and D_mm=rD/(1+r)—as if each objective trained under its own budget despite shared parameters. Two independent estimators—IsoFLOP profiles (parabola fit to loss vs log N at fixed compute) and training-curve envelopes (pooled lower envelope over all runs)—determine Nopt(C)∝C^a and Dopt(C)∝C^b. The load-bearing identity is a+b=1; its variation with r is the whole story. The joint Pareto analysis imposes an asymmetric model, pairing the invariant language law

Load-bearing premise

The paper assumes the language loss depends only on the effective text compute C_text=6ND_text and the multimodal loss only on C_mm=6ND_mm, treating the shared parameters as if each objective had its own budget; if text learning actually depends on the full token mix, the claimed invariance and variance are artifacts of that split.

What would settle it

Train two models at the same effective text compute C_text and model size N, one with r=0.1 and one with r=0.3, and compare the final text loss Ltext at identical (N, D_text); a systematic difference beyond noise would falsify the composition-invariant language law.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • The efficiency frontier gives a concrete recipe: for any total compute budget and any target multimodal ratio r, it specifies the model size, text token count, and multimodal token count that minimize combined loss.
  • The language allocation law is stable across r up to 0.3, so adding multimodal data does not force a re-optimization of text scaling or sacrifice text benchmark accuracy.
  • As r increases, compute-optimal training shifts from parameter scaling to token scaling; for image-heavy mixtures, extra tokens reduce loss more than extra parameters.
  • Native multimodal pre-training produces positive cross-modal transfer: abstract spatial reasoning in pure text improves with visual pretraining, and the gain widens with model scale.
  • Multimodal in-context learning emerges with scale and data: few-shot gains over zero-shot rise from near zero at 71M parameters to +2.43 points at 3B, concentrated on spatial and relational tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The invariance of the language law suggests the data ratio r could itself be a tunable knob, making the efficiency frontier imply an optimal r(C) strategy—at very large budgets, text-heavy mixtures may be preferred because parameter scaling is more compute-efficient; the paper does not derive this explicitly.
  • Because the text loss is masked to text tokens only, image tokens may act as context rather than direct training signal; a natural extension is to test whether the composition invariance survives changes in sequence-level interleaving at fixed r.
  • The concentration of in-context gains on spatial/relational tasks predicts that visual pretraining transfers to other abstract relational text tasks (geometry, graph reasoning, navigation); this is a testable downstream prediction beyond the paper's benchmarks.
  • The two-law picture could be an artifact of the decoupled budget split; comparing against a model trained with a single aggregate loss under the same compute would show whether the asymmetry is real or coordinate-induced.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies compute-optimal scaling for native multimodal pre-training from scratch. It decouples the training budget by modality (Ctext=6NDtext, Cmm=6NDmm, with Dtext=D/(1+r), Dmm=rD/(1+r)) and uses IsoFLOP profiles and training-curve envelopes to estimate allocation exponents a and b in Nopt∝C^a, Dopt∝C^b for each objective. The central claim is that the language allocation law is largely invariant to the multimodal ratio r, while the multimodal allocation law is highly composition-sensitive; the paper then derives a joint Pareto frontier and reports downstream text preservation, cross-modal transfer to spatial reasoning, and multimodal in-context learning. The empirical core is the asymmetry between the two objectives' scaling exponents as r varies over {0,0.1,0.2,0.3}.

Significance. If the claimed language/multimodal asymmetry were established, the paper would provide practically valuable guidance for allocating parameters and tokens in native multimodal pre-training, and the two-estimator design plus the broad downstream benchmark suite are genuine strengths. However, the present manuscript does not establish the asymmetry: the same estimator that allegedly shows a sharp multimodal decline shows a nearly identical envelope decline for text, no uncertainty intervals are reported, and the joint Pareto model hard-codes the invariance it claims to discover. The decoupling assumption and the undefined multimodal loss further undermine the central conclusion. With substantial revision, the study could become a useful contribution, but the current evidence is not sufficient for acceptance.

major comments (4)
  1. [§3.1, §3.2, Figs. 2 and 5] The central asymmetry is not supported by the paper's own envelope estimates. Text envelope a-values are 0.671 (r=0), 0.643 (r=0.1), 0.644 (r=0.2), 0.639 (r=0.3), a decline of ≈0.032; multimodal envelope a-values are 0.665 (r=0.1), 0.641 (r=0.2), 0.634 (r=0.3), a decline of ≈0.031. These declines are virtually identical in magnitude. Section 3.1 dismisses the text decrease as 'minor numerical drift' with 'no monotonic downward trend', while Section 3.2 calls the same-size multimodal decrease a 'sharp downward trajectory' and a 'robust scaling law.' No bootstrap intervals or standard errors are provided anywhere, and the multimodal fits cover only three r values (0.1, 0.2, 0.3) whereas the text fits include r=0. The paper therefore applies different evidentiary standards to the two objectives; as written, the headline invariance/sensitivity dichotomy is an unsupported interpretation of th
  2. [§3.3] The joint Pareto analysis assumes the asymmetry rather than testing it. The text explicitly says the framework 'pair[s] a composition-invariant language objective with a composition-variant multimodal objective' and that forcing the text exponent to fluctuate would introduce 'overfitting and localized optimization noise.' Figures 7(b) and 7(c) then display text as r-free and multimodal as r-dependent, but these outputs are consequences of the modeling assumption, not independent evidence. To validate the central claim, the authors need to fit a model that allows both exponents to depend on r and compare it to the constrained model, e.g., with a likelihood-ratio or information criterion. As it stands, §3.3 cannot corroborate the language-invariance result.
  3. [§2.1, Appendix A] The decoupling premise is load-bearing and is not tested. Equation (1) and the surrounding text define Ctext and Cmm by splitting the token budget according to r, as if each objective were trained under a separate compute budget despite fully shared parameters. All invariance/variance conclusions live inside this coordinate system. Moreover, Appendix A states that 'the training loss is computed only on text tokens, with vision tokens masked out from loss computation,' yet the paper analyzes a multimodal objective Lmm in Figures 4–6. The operational definition of Lmm is never given, and if vision tokens do not contribute to the loss, it is unclear what objective the multimodal scaling law describes. A concrete test of the decoupling assumption would fit Ltext as a function of Ctext and r (or Dmm) and test whether r enters significantly; the same should be done for Lmm. Without such a test
  4. [§2.2, §3.3, Figs. 2–7] No uncertainty quantification is provided for any fitted quantity—exponents a,b, the frontier parameters E, Cc, β, or the projected Pareto frontier. The paper extrapolates from compute budgets around 10^19–10^21 to 10^25 in Figure 7(a), but these 'predictions' are simply the fitted power laws of Eqs. (2)–(3) evaluated at larger C; they are not validated on held-out budgets. The authors should report standard errors or bootstrap confidence intervals for the exponent differences (especially the text-vs-multimodal declines) and perform an out-of-sample check, such as fitting on lower compute budgets and predicting a held-out higher budget. Without this, the sharpness of the claimed asymmetry and the reliability of the efficiency frontier cannot be assessed.
minor comments (5)
  1. [Eq. (3)] Equation (3) is called a 'power law,' but the additive irreducible loss E makes it a shifted power law. Please adjust the terminology for precision.
  2. [Figs. 3 and 6] The panel labels '(flat)' and '(vary with r)' in Figures 3(c) and 6(c) are interpretive conclusions, not neutral descriptions. Consider relabeling them descriptively (e.g., 'exponent estimates by r').
  3. [Appendix B] The benchmark name 'MMBenchen' appears to be a typo for 'MMBench-en.' Please correct it in Tables 6–8.
  4. [Fig. 8] Figure 8(b) labels the horizontal axis 'Text tokens Ntext (B)', but tokens are usually denoted D in this paper; please use consistent notation.
  5. [§4.2] The cross-modal transfer claim in Section 4.2 is based on two SpatialEval subtasks, and no significance testing or variance across seeds is reported. This is a supporting result, but the 'widening gap' statement should be softened or supplemented with error bars.

Circularity Check

2 steps flagged

The joint Pareto frontier hard-codes the claimed text-invariant / multimodal-variant asymmetry, and the projected efficiency frontier is the fitted power-law surface re-expressed as a prediction.

specific steps
  1. self definitional [Section 3.3, Joint Pareto Frontier (page 5)]
    "To rigorously define this global architectural trade-off, we employ an asymmetric modeling framework based on our primary IsoFLOP estimator. Specifically, we pair a composition-invariant language objective with a composition-variant multimodal objective."

    The Pareto model is built by assuming, not testing, the paper's central asymmetry. The subsequent frontier and the 'text (r-free)' versus multimodal exponents in Figure 7 are consequences of this assumed pairing, so the joint analysis cannot serve as evidence for the claimed language-invariance / multimodal-sensitivity duality. 'By capturing this empirical duality' presents the construction input as an empirical output.

  2. fitted input called prediction [Section 3.3 / Figure 7 (pages 5-6)]
    "Projecting the globally optimal allocation exponents, a(r) and b(r), reveals that under a low multimodal ratio (r=0.1 ), optimal parameter scaling follows Nopt ∝C 0.69 total. However, increasing the multimodal allocation to r=0.3 imposes the substantial data requirements of dense multimodal inputs onto the entire system. This reduces the parameter scaling proportionality to C0.66 total, concurrently necessitating a more aggressive scaling of the system-wide token allocation (∝C 0.34 total)."

    The numbers quoted are the fitted exponents a(r) from Equations (2)-(3) and Figures 1-6, re-expressed in the total-compute coordinate. The 'efficiency frontier' and the claim that text-heavy mixtures become compute-efficient only at larger scales are therefore evaluations of the fitted power law rather than independent predictions. The predictive content is forced by the fitted constants by construction.

full rationale

The paper's core decoupled fits are not themselves circular: models are actually trained at different multimodal ratios r, the language and multimodal losses are measured, and allocation exponents are fitted directly to those runs. Downstream evaluations on external benchmarks are also independent evidence for the transfer and in-context-learning claims. There is no load-bearing self-citation chain, and the decoupled-budget assumption (Ltext as a function only of Ctext, Lmm only of Cmm) is a strong modeling assumption rather than a definitional identity hiding the result. However, the joint Pareto section is circular with respect to its own headline asymmetry: it constructs the model by pairing a composition-invariant language objective with a composition-variant multimodal objective, then presents the resulting frontier as a derivation. Moreover, the projected large-compute frontier is just the fitted power-law surface evaluated at larger C; no withheld data or external check distinguishes it from the fit. Hence the central 'efficiency frontier' reduces by construction to the fitted exponents, while the decoupled empirical fits retain independent content. This warrants a partial circularity score rather than a full 8-10.

Axiom & Free-Parameter Ledger

3 free parameters · 8 axioms · 0 invented entities

The paper's quantitative output is almost entirely empirical fitting: three frontier constants and two allocation exponents per (objective, r), plus an r-composition model with 3-4 support points. The two load-bearing modeling choices are the per-objective compute decoupling (§2.1) and the text-only loss mask (Appendix A), which together define what 'the multimodal objective' means. No new entities (particles, forces, dimensions) are introduced.

free parameters (3)
  • Frontier parameters E, Cc, β (per objective, per r) = text r=0: E=1.503, Cc=9.7e19, β=0.130; mm r=0.3: E=0.001, Cc=4.5e26, β=0.070
    Eq. (3): three fitted constants per curve; the multimodal r=0.3 fit has E collapsing to 0.001, indicating an unidentifiable floor.
  • Allocation exponents a(r), b(r)=1-a(r) = text a: 0.697→0.663 (IsoFLOP), 0.671→0.639 (envelope) across r=0→0.3; mm a: 0.709→0.643 (IsoFLOP), 0.665→0.634 (envelope
    Eq. (2) and Figs. 1-6: the central exponents are least-squares slopes with no reported uncertainty; between-estimator spread is 0.01-0.04, comparable to the claimed r-trend.
  • Composition model (r-dependence of exponents and frontier) = r ∈ {0, 0.1, 0.2, 0.3} (4 text, 3 multimodal support points)
    §3.3 and Fig. 7: the Pareto frontier and a(r), b(r) curves are built from 3-4 mixture ratios and extrapolated to C up to 1e25.
axioms (8)
  • standard math FLOPs(C) = 6ND with N the number of activated non-embedding parameters
    Eq. (1), §2.1: standard Hoffmann/Kaplan approximation, applied to an MoE by counting only the 8 activated experts (Table 1: 128 total, 8 active).
  • domain assumption The smoothed training loss proxies the objective loss under single-pass training
    §2.1: 'we rely solely on the smoothed training loss as a proxy for the test loss' because each token is seen roughly once and no multimodal validation metric exists.
  • domain assumption Functional forms L(C)=E+(Cc/C)^β and Nopt∝C^a, Dopt∝C^b
    Eqs. (2)-(3), §2.2: power-law/irreducible-floor forms are assumed a priori and fitted by least squares; the log-sum-exp parameterization enforces positivity.
  • ad hoc to paper Decoupling: Ltext depends only on Ctext=6NDtext and Lmm only on Cmm=6NDmm, with Dtext=D/(1+r), Dmm=rD/(1+r)
    §2.1: this is the paper's coordinate system; it assumes each objective's loss depends only on its own effective compute despite fully shared parameters, and the joint model (§3.3) then imposes text-invariance by construction.
  • domain assumption Loss masking: gradients flow only from text tokens; vision tokens contribute compute but no loss
    Appendix A ('The training loss is computed only on text tokens'); consequence: Lmm is text-token loss in image-conditioned segments, not an independent visual objective.
  • domain assumption The optimal loss vs log N at fixed compute is a parabola
    §2.2 IsoFLOP method; the parabolic shape is the identification assumption for Nopt minima.
  • domain assumption MoE scaling tracks dense scaling as a function of activated parameters
    Throughout §3: the exponents are interpreted as if for a dense model with N active parameters (Kaplan/Hoffmann framework); no dense baseline is provided for comparison.
  • standard math Dopt = C/(6Nopt), so a+b=1
    Eq. (2), §2.1: b is not independently fitted; it is algebraic once Nopt(C) is fitted.

pith-pipeline@v1.3.0-alltime-deepseek · 28038 in / 32838 out tokens · 317220 ms · 2026-08-01T05:59:40.057126+00:00 · methodology

0 comments
read the original abstract

Although large language models (LLMs) exhibit remarkable reasoning capabilities, their reliance on text-only pre-training restricts the perception of the multimodal physical world. Native multimodal pre-training avoids this limitation by training models from scratch on multimodal inputs, thereby achieving deep cross-modal integration and mitigating optimization asymmetries inherent to traditional late-fusion architectures. Despite these advantages, the scaling properties of this paradigm remain systematically uncharacterized. To address this gap, we investigate the optimal model size and token count for training a transformer-based vision-language model under a fixed computational budget. We demonstrate that minimal objective loss adheres to a predictable compute law, whereas compute-optimal model sizes and token counts scale as power laws. Notably, language and multimodal objectives manifest distinct scaling behaviors. The language allocation law is largely invariant to the composition of the data, indicating stable language learning regardless of the multimodal data ratio. Conversely, the multimodal allocation law is highly sensitive to this composition. Specifically, text-heavy mixtures become compute-efficient only at larger model scales, shifting the optimal resource allocation toward greater model capacity. Additionally, by modeling the influence of data composition on compute laws and allocation exponents, we derive an efficiency frontier specifying precise configurations of model size, token count, and data mixture. Downstream evaluations further reveal that native multimodal pre-training induces positive cross-modal transfer, thereby enhancing pure-text spatial reasoning and enabling robust multimodal in-context learning. In summary, this empirical research establishes the essential groundwork for predictably scaling multimodal foundation models.

Figures

Figures reproduced from arXiv: 2607.22043 by Aoqi Wu, Bei Yu, Hai Wang, Haoyuan Wu, Jiajia Wu, Jinxiang Ou.

Figure 1
Figure 1. Figure 1: IsoFLOP curves (language objective). For a range of model sizes, we adjust the number of training tokens to maintain a constant final FLOPs, setting the cosine cycle length to match this target compute budget. The distinct valley in the loss curve demonstrates that an optimal model size exists for any given FLOP budget. Based on the locations of these minima, we project the compute-optimal model size and t… view at source ↗
Figure 2
Figure 2. Figure 2: Training curve envelope (language objective). Training curves are shown for all runs across varying values of r, encompassing various model sizes. By extracting the envelope of minimal loss per FLOP from these curves, we estimate the optimal model size and training token allocation for a given compute budget. Training Curve Envelope. For a fixed model size N, increasing the token count D during training pr… view at source ↗
Figure 3
Figure 3. Figure 3: Compute-optimal allocation (language objective). (a) Optimal model size Nopt and (b) optimal token count Dopt as a function of the compute budget Ctext for each r. Solid lines and stars represent the IsoFLOP fits and their respective minima, while thin dashed lines and scattered points provide a cross-check using the training envelope. The two estimation methods align closely, and the slopes exhibit minima… view at source ↗
Figure 4
Figure 4. Figure 4: IsoFLOP curves (multimodal objective). For a range of model sizes, we adjust the number of training tokens to maintain a constant final FLOPs, setting the cosine cycle length to match this target compute budget. The distinct valley in the loss curve demonstrates that an optimal model size exists for any given FLOP budget. Based on the locations of these minima, we project the compute-optimal model size and… view at source ↗
Figure 5
Figure 5. Figure 5: Training curve envelope (multimodal objective). Training curves are shown for all runs across varying values of r, encompassing various model sizes. By extracting the envelope of minimal loss per FLOP from these curves, we estimate the optimal model size and training token allocation for a given compute budget. 10 18 10 19 10 20 10 21 Cmm = 6NDmm 100M 1B 10B Nopt (a) Nopt / C a r = 0:1 r = 0:2 r = 0:3 10 1… view at source ↗
Figure 6
Figure 6. Figure 6: Compute-optimal allocation (multimodal objective). (a) Optimal model size Nopt and (b) optimal token count Dopt as a function of the compute budget Cmm for each r. Solid lines and stars represent the IsoFLOP fits and their respective minima, while thin dashed lines and scattered points provide a cross-check using the training envelope. The two estimation methods align closely, and the slopes vary significa… view at source ↗
Figure 7
Figure 7. Figure 7: Joint Pareto frontier. (a) Pareto frontier demonstrating the trade-off between multimodal loss Lmm and language loss Ltext by sweeping the mixture ratio r. The curves are bounded by fixed total compute budgets Ctotal, plotting actual data points alongside extrapolated frontiers for larger compute scales. (b) Optimal model size Nopt and (c) optimal token count Dopt as a function of compute Ctotal. The disti… view at source ↗
Figure 8
Figure 8. Figure 8: Text capabilities are preserved under native multimodal pre-training. Average accuracy across 16 text benchmarks with varying multimodal data ratios r (fixed 250B text token budget). (a) Text performance across model sizes N. (b) A3B text performance over training tokens Dtext. Consistently overlapping curves indicate multimodal data integration does not degrade core language abilities. b(r), reveals that … view at source ↗
Figure 9
Figure 9. Figure 9: Multimodal pre-training enhances pure-text spatial reasoning. Accuracy on SpatialEval’s text-only abstract spatial-reasoning sub-tasks (Wang et al., 2024a), comparing baseline (r = 0) and multimodal (r = 0.3) settings. (a) Accuracy across model sizes N. (b) A3B accuracy over training tokens Dtext. Multimodal models consistently outperform text-only baselines, with the performance gap widening at larger sca… view at source ↗
Figure 10
Figure 10. Figure 10: Multimodal in-context learning emerges with model scaling. (a) Average multimodal accuracy for 0-, 1-, and 3-shot settings across model sizes N. (b) Relative performance gains of k-shot settings over the 0-shot baseline. The increasing gains demonstrate that multimodal in-context learning emerges with model scaling. 4.2 Text Performance under Multimodal Pre-Training A central concern in native multimodal … view at source ↗
Figure 11
Figure 11. Figure 11: Few-shot accuracy under data scaling. Average benchmark accuracy of each model under 0-, 1-, and 3-shot across training tokens. Although the performance trajectories remain overlapping throughout training for smaller models, the few-shot settings progressively outperform the 0-shot baseline in larger models. A71M A128M A340M A590M A874M A3B 30 35 40 Accuracy (%) Aggregate 0-shot 1-shot 3-shot A71M A128M A… view at source ↗
Figure 12
Figure 12. Figure 12: Few-shot trends by task category across model scale. Each panel aggregates the benchmarks of one category and reports 0-, 1-, and 3-shot accuracy across model size N. The few-shot margin is largest and most consistent on spatial reasoning, modest on aggregate and vision-knowledge suites, and absent or negative on OCR/recognition benchmarks. points for the A874M model, and ultimately reaching +2.43 points … view at source ↗
Figure 13
Figure 13. Figure 13: Few-shot gain trends upward with training. Few-shot gain of the A3B model over the 0-shot baseline for the 1-shot and 3-shot settings, shown for r ∈ {0.1, 0.2, 0.3} against total training tokens D. Despite run-to-run fluctuation, the gains show an overall positive trend with D across all r. particular r. Together with [PITH_FULL_IMAGE:figures/full_fig_p009_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 16 linked inside Pith

  1. [1]

    B Experiment Results This section details the comprehensive per-benchmark results supporting the analyses presented in Section 4, organized into three primary categories

    A71M A128M A340M A590M A874M A3B Architecture Configurations layers 11 15 19 23 27 48 hidden size 640 768 1152 1280 1536 2048 FFN hidden size 2048 2048 3072 4096 4096 6912 attention heads 8 12 16 20 24 32 query groups 4 kv channels 128 total experts 128 activated experts 8 expert FFN hidden size 256 256 384 512 512 768 shared-expert hidden size 256 256 38...

    2048

  2. [3]

    SuperGPQA: Scaling LLM Evaluation Across 285 Graduate Disciplines.arXiv preprint arXiv:2502.14739,

    Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, et al. SuperGPQA: Scaling LLM Evaluation Across 285 Graduate Disciplines.arXiv preprint arXiv:2502.14739,

    Pith/arXiv arXiv

  3. [7]

    TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension.arXiv preprint arXiv:1705.03551,

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension.arXiv preprint arXiv:1705.03551,

    Pith/arXiv arXiv

  4. [11]

    SocialiQA: Commonsense Reasoning about Social Interactions.arXiv preprint arXiv:1904.09728,

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. SocialiQA: Commonsense Reasoning about Social Interactions.arXiv preprint arXiv:1904.09728,

    Pith/arXiv arXiv 1904

  5. [13]

    CountQA: How Well Do MLLMs Count in the Wild?arXiv preprint arXiv:2508.06585,

    11 Jayant Sravan Tamarapalli, Rynaa Grover, Nilay Pande, and Sahiti Yerramilli. CountQA: How Well Do MLLMs Count in the Wild?arXiv preprint arXiv:2508.06585,

    Pith/arXiv arXiv

  6. [14]

    Beyond Language Modeling: An Exploration of Multimodal Pretraining

    Shengbang Tong, David Fan, John Nguyen, Ellis Brown, Gaoyue Zhou, Shengyi Qian, Boyang Zheng, Th´eophane Vallaeys, Junlin Han, Rob Fergus, et al. Beyond Language Modeling: An Exploration of Multimodal Pretraining. arXiv preprint arXiv:2603.03276,

    arXiv

  7. [15]

    SigLIP 2: Multilingual Vision- Language Encoders with Improved Semantic Understanding, Localization, and Dense Features.arXiv preprint arXiv:2502.14786,

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. SigLIP 2: Multilingual Vision- Language Encoders with Improved Semantic Understanding, Localization, and Dense Features.arXiv preprint arXiv:2502.14786,

    Pith/arXiv arXiv

  8. [16]

    Logicvista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts.arXiv preprint arXiv:2407.04973,

    Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logicvista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts.arXiv preprint arXiv:2407.04973,

    Pith/arXiv arXiv

  9. [17]

    Hellaswag: Can A Machine Really Finish Your Sentence?arXiv preprint arXiv:1905.07830,

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can A Machine Really Finish Your Sentence?arXiv preprint arXiv:1905.07830,

    Pith/arXiv arXiv 1905

  10. [18]

    AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models.arXiv preprint arXiv:2304.06364,

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models.arXiv preprint arXiv:2304.06364,

    Pith/arXiv arXiv

  11. [2016]

    Kimi K2.5: Visual Agentic Intelligence.arXiv preprint arXiv:2602.02276,

    Kimi, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi K2.5: Visual Agentic Intelligence.arXiv preprint arXiv:2602.02276,

    Pith/arXiv arXiv

  12. [2017]

    Scaling Laws for Neural Language Models.arXiv preprint arXiv:2001.08361,

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models.arXiv preprint arXiv:2001.08361,

    Pith/arXiv arXiv 2001

  13. [2019]

    Challenging Big-Bench Tasks and Whether Chain-of- Thought Can Solve Them.arXiv preprint arXiv:2210.09261,

    Mirac Suzgun, Nathan Scales, Nathanael Sch¨arli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging Big-Bench Tasks and Whether Chain-of- Thought Can Solve Them.arXiv preprint arXiv:2210.09261,

    Pith/arXiv arXiv

  14. [2020]

    Training Compute-Optimal Large Language Models.arXiv preprint arXiv:2203.15556,

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, DDL Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training Compute-Optimal Large Language Models.arXiv preprint arXiv:2203.15556,

    Pith/arXiv arXiv

  15. [2021]

    Emu3.5: Native Multimodal Models are World Learners.arXiv preprint arXiv:2510.26583,

    Yufeng Cui, Honghao Chen, Haoge Deng, Xu Huang, Xinghang Li, Jirong Liu, Yang Liu, Zhuoyan Luo, Jin- sheng Wang, Wenxuan Wang, et al. Emu3.5: Native Multimodal Models are World Learners.arXiv preprint arXiv:2510.26583,

    Pith/arXiv arXiv

  16. [2022]

    OmniS- patial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models.arXiv preprint arXiv:2506.03135,

    Mengdi Jia, Zekun Qi, Shaochen Zhang, Wenyao Zhang, Xinqiang Yu, Jiawei He, He Wang, and Li Yi. OmniS- patial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models.arXiv preprint arXiv:2506.03135,

    arXiv

  17. [2024]

    Measuring Massive Multitask Language Understanding.arXiv preprint arXiv:2009.03300,

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding.arXiv preprint arXiv:2009.03300,

    Pith/arXiv arXiv 2009

  18. [2025]

    Training Verifiers to Solve Math Word Problems.arXiv preprint arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training Verifiers to Solve Math Word Problems.arXiv preprint arXiv:2110.14168,

    Pith/arXiv arXiv

  19. [2026]

    DeepSeek-V3 Technical Report.arXiv preprint arXiv:2412.19437, 2024a

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. DeepSeek-V3 Technical Report.arXiv preprint arXiv:2412.19437, 2024a. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. InAnnual Conference on Neural Information Processing Systems (NIPS), 202...

    Pith/arXiv arXiv

  20. [4096]

    We detail the specific architecture configurations and training settings in Table

    The training loss is computed only on text tokens, with vision tokens masked out from loss computation (Kimi et al., 2026). We detail the specific architecture configurations and training settings in Table

    2026