{"id":"8f1312b7-9410-4a10-9286-d694de129608","arxiv_id":"2608.11167","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Replacing an object's name in a caption with the image tokens of that object during pretraining gives explicit object-entity grounding, making MLLM alignment several times more data-efficient and boosting grounding and perception scores.","lead":"The authors train vision-language models by inserting picture regions of objects directly into sentences, replacing the words that name them. This explicit object-level supervision makes pretraining far more data-efficient and improves grounding, so a model trained on 50,000 samples matches one trained on 600,000.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Grounding and data-efficiency gains are evaluated on images that overlap the pretraining/SFT corpus (COCO/GQA), with no disjoint-image test reported; the claimed transfer of object-level alignment to unseen images is therefore not established.","rationale":"Reader's weakest assumption was the accuracy of automated object-entity synthesis and the interleaved-to-global transfer. The synthesis quality concern is substantially mitigated by the perturbation experiment (Section D.2): 10–30% label corruption preserves most gains, so small annotation errors are not the binding constraint. The transfer concern, however, is more serious than the reader stated: the empirical evidence for transfer is contaminated by train/test image overlap. RefCOCO is built on COCO; the pretraining pool contains COCO (Appendix A.1); GQA and TextVQA appear both in pretraining/SFT and evaluation. This makes the grounding and data-efficiency results consistent with memory of the exact images rather than a generalizable object-level alignment. The paper provides no disjoint-image evaluation or overlap analysis, so the central mechanism and the strongest quantitative claims are not yet established on unseen images. The correct check is to retrain on a deduplicated corpus and re-measure; until then the verdict should remain conditional, with this as the binding condition.","tokens_in":24111,"tokens_out":7809,"duration_ms":71840,"concrete_test":"Retrain the Qwen2.5-3B MMCS and Caption baselines with the same pipeline and SFT recipe, but exclude all COCO, GQA, and Flickr30k images from the pretraining pool (or deduplicate against evaluation splits), then re-evaluate RefCOCO/+/g, GQA, and TextVQA. If the 50K-vs-600K crossover and the +7.9% grounding gain persist on disjoint images, the transfer claim survives; if they shrink or vanish, the reported gains are substantially a memorization artifact.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4 evaluates visual grounding on RefCOCO/RefCOCO+/RefCOCOg and general VQA on GQA and TextVQA. The pretraining corpus (Appendix A.1) includes MS COCO and GQA images, and the SFT set (Section A.2) includes GQA and TextVQA. Thus the exact images used for evaluation appear in training. RefCOCO is built on COCO images, so the grounding benchmark is directly contaminated. The paper reports no overlap statistics and no evaluation on a disjoint image set. Consequently, the headline claims — 50K MMCS surpassing 600K image-level pretraining (Figure 4/Table 11) and +7.9% grounding (Table 1) — may reflect memorized object layouts from pretraining rather than a transferable object-level alignment mechanism. The controlled comparison versus the caption baseline is internally valid, but it does not establish generalization to unseen images.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MultiModal Code-Switching (MMCS), a pretraining paradigm for multimodal LLMs that replaces textual entity tokens in captions with the corresponding visual object tokens, training with a language modeling loss and an entity reconstruction loss. A pipeline using Qwen-based captioning/entity extraction, Grounding DINO, and SAM-2.1 generates a 773K-sample pretraining dataset. Experiments report data efficiency (50K samples matching 600K image-level pretraining) and gains on visual grounding (+7.9%) and perception benchmarks, with mechanistic analyses via CKA/CKNNA/Mutual k-NN and attention maps.","tokens_in":24201,"tokens_out":6474,"duration_ms":53868,"significance":"If the gains persist on disjoint images, this is a meaningful improvement over image-level alignment, providing explicit object-level supervision during pretraining. The paper's controlled setup (identical image-caption pairs for MMCS and the image-level baseline) and robustness ablations (caption-source swap, correspondence perturbation, resolution sweep) are strengths. The release of code and dataset supports reproducibility.","major_comments":[{"comment":"The evaluation benchmarks overlap with the pretraining and SFT corpora. RefCOCO is built on COCO images, and COCO and GQA images are included in the pretraining sources (Appendix A.1); GQA is also in the SFT set (Section A.2, which explicitly lists GQA). The paper reports no overlap statistics and no evaluation on a disjoint image set. Consequently, the headline claims of data efficiency (Figure 4/Table 11) and transferable grounding gains (Table 1) could be inflated by memorized object layouts rather than a generalizable object-level alignment mechanism. Please either evaluate on benchmarks whose images are provably disjoint from all training data, or report results on the non-overlapping subsets of the current benchmarks, and discuss any performance difference.","section":"Section 4.1 / Appendix A.1 / A.3"},{"comment":"The quality of the synthesized correspondences is validated on only 200 objects per source by a VLM judge and 20 objects per source by human annotators. This is a thin basis for the claim of 'accurate object-entity correspondences' at 773K scale, particularly because the filtering thresholds (box threshold 0.4, text threshold 0.3, mask-area fraction 0.2) are hand-chosen. Although the perturbation analysis (D.2) shows robustness to random box noise, it does not address systematic errors in entity extraction or captioning. Please provide a larger human-validated sample or a per-category error analysis, and report the sensitivity of downstream performance to the filtering thresholds.","section":"Appendix A.1, Table 5; Section 3.3"},{"comment":"The pretraining objective operates on interleaved sequences where textual entities are replaced by object tokens, yet SFT and inference revert to global image representations. The mechanistic evidence of improved alignment (Figure 5) and sharper attention (Figure 6) is presented for pretraining-only models, so it is not directly shown that these improvements survive the format mismatch after SFT, when the downstream numbers are measured. Please either report the alignment metrics and attention analyses after SFT or demonstrate via ablations (e.g., SFT with interleaved inputs for both methods) that the transfer path is consistent.","section":"Section 4.1; Section 5"}],"minor_comments":[{"comment":"The dataset name is mistyped as 'LLaV A-NeXT' twice; it should be 'LLaVA-NeXT'.","section":"Section 4.1 and A.2"},{"comment":"The abbreviation 'VQAT' is used without definition; it should be 'TextVQA'.","section":"Table 2 and Table 14"},{"comment":"Specify whether the attention maps are from pretraining-only checkpoints or after SFT, to avoid ambiguity.","section":"Section 5.2"},{"comment":"State the method for computing the 41.4% reduction explicitly (e.g., average token count across 3,000 samples per source).","section":"Figure 7 caption"},{"comment":"The paired t-test is only reported for general VQA; consider reporting per-benchmark significance or confidence intervals for the grounding and perception gains.","section":"Section D.5"}],"recommendation":"major_revision","confidential_remarks":"The main concern is evaluation contamination; the authors should be required to provide disjoint-image results. The thin data-quality validation is a secondary issue. The paper is otherwise well-executed with strong controlled comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nHere's my read on MMCS. The central idea—replacing textual entities with their visual object tokens during modality alignment, plus a dedicated entity-reconstruction loss—is genuinely distinct from the patch-level alignment and coordinate-text grounding work cited. The paper also does something right that many don't: the caption baseline trains on the exact same image-caption pairs, so the comparison isolates the interleaving mechanism. The robustness probes (caption source swap, 10/30% correspondence perturbation, resolution sweep, scaling to 1M) are thoughtful, and the multi-seed significance test on general VQA, while limited to that bucket, is honest.\n\nThe soft spot is the evaluation-image overlap. The pretraining corpus draws from COCO, GQA, and Open Images, and RefCOCO is built on COCO images; GQA is both in pretraining and evaluation. The paper reports no overlap statistics and no test on a disjoint image set. That means the headline numbers—50K beating 600K, +7.9% grounding—could reflect memorized object layouts rather than a transferable object-level alignment. The internal comparison against the caption baseline is still valid, but it only shows that MMCS is better at exploiting the training images, not that it generalizes to unseen images. For a paper whose whole premise is better alignment, that's a substantial gap.\n\nSmaller concerns: the headline data-efficiency and grounding gains are single-run; the 773K dataset quality rests on 120 human checks and a VLM judge; and the pretrain-interleaved/SFT-global mismatch is an empirical result without a mechanistic guarantee. None of these are deal-breakers on their own.\n\nAll in, this paper has a good idea, a clean design for the controlled comparison, and an honest limitations section. The missing disjoint-image evaluation is the main thing standing between \"conditional\" and \"accept\" in my mind. I'd send it to a serious referee, but the referee should push for a disjoint-image evaluation (e.g., an unseen split of the same benchmarks, or at minimum overlap analysis and an evaluation on a held-out source) and multi-seed results for the grounding claims.\n\nI'd bring it to a reading group if someone is working on alignment pretraining, but I wouldn't cite the current numbers as evidence of generalization yet.\n\nBest,\n[Name]","headline":"A genuinely novel alignment mechanism with a well-controlled study, but the headline generalization claims are not yet supported because the evaluation images overlap the training corpus.","tokens_in":24893,"tokens_out":3325,"would_cite":false,"duration_ms":29993,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Interleaving visual object tokens into caption text during alignment pretraining gives a 12-fold data-efficiency gain over image-caption pretraining, matching or beating 600K pairs with 50K samples.","keywords":["multimodal large language models","modality alignment pretraining","object-level supervision","referential ambiguity","visual grounding","data efficiency","code-switching","data synthesis"],"falsifier":"Take the 50K MMCS pretraining set, randomly replace 10% of the object bounding boxes with unrelated same-size boxes from the same image, and train the identical Qwen2.5-3B setup; if that noised run no longer beats the 600K image-caption baseline on RefCOCO, RefCOCO+, and RefCOCOg, the data-efficiency claim depends on near-perfect synthetic correspondences rather than on the code-switching format itself.","tokens_in":23796,"feed_emoji":"🎯","tokens_out":10277,"duration_ms":86444,"temperature":0.7,"pith_summary":"The paper argues that image-level alignment pretraining—encoding a whole image into one global representation and training the model to predict a long caption—is fundamentally data-hungry because of referential ambiguity: the model must guess which visual region corresponds to each textual phrase. MMCS removes that guessing by replacing each textual entity in the caption with the image tokens of its corresponding visual object, so the model is forced to ground local language in local vision. The authors build a 773K-sample synthetic dataset of such code-switched captions and show that, even with only 50K samples, MMCS matches or surpasses a model pretrained on 600K standard image-text pairs. Across three LLM backbones and two vision encoders, MMCS raises referring-expression grounding by 7.9% on average and perception-centric benchmarks by 2.1%, with CKA, CKNNA, and Mutual k-NN scores confirming tighter cross-modal alignment. If true, this points to explicit object-level correspondence as the missing ingredient in current modality-alignment recipes.","feed_headline":"Object-level alignment cuts pretraining data 12x","feed_subtitle":"Replacing text entities with visual tokens grounds each object, lifting grounding by 7.9% and perception by 2.1%.","key_machinery":"The central object is the code-switched sequence $X_{\\mathrm{MMCS}}=\\operatorname{Concat}(X_{<i}, v_{\\mathrm{object}}, X_{>i+m})$: a caption in which each textual entity $e=[x_i,\\dots,x_{i+m}]$ is replaced by the image tokens $v_{\\mathrm{object}}$ whose spatial regions intersect the object's bounding box. The projector is trained with a language-modeling loss over the remaining text tokens plus an entity-reconstruction loss $\\mathcal{L}_{\\mathrm{entity}}=-\\log p_\\theta(e\\mid X_{<i}^{\\mathrm{MMCS}}, v_{\\mathrm{object}})$, which forces visual object tokens to carry identity-bearing information capable of regenerating the original phrase. The data-synthesis pipeline (captioner, entity extractor, Grounding DINO localizer, SAM-2.1 segmenter, box/text thresholds 0.4/0.3, and the 20% mask-area rule) supplies the one-to-one correspondences that make the substitution meaningful.","core_discovery":"The paper claims that the standard practice of aligning a global image representation with a long caption leaves referential ambiguity—the model must guess which visual region corresponds to which textual phrase—and that this ambiguity is the root cause of data inefficiency and weak grounding in multimodal LLMs. MMCS removes the guesswork by mechanically interleaving: for each noun-phrase entity in the caption, the textual tokens are replaced by the image tokens of the corresponding object (Eq. 1), so next-token prediction and an explicit entity-reconstruction loss (Eq. 3) are conditioned directly on local visual features. Trained on 773K such synthetic code-switched samples, a Qwen2.5-3B-based model with only 50K samples matches or exceeds an image-caption baseline trained on 600K samples, and the full setup improves referring-expression grounding by an average of 7.9% and perception-centric benchmarks by 2.1% across three LLM backbones and two vision encoders. The paper supports the mechanism with layer-wise CKA/CKNNA/Mutual k-NN alignment scores and attention maps.","pith_inferences":["A direct extension not explored in the paper: apply MMCS to domains where text regions are the objects—charts, documents, scene text—since the pipeline's measured implicit OCR supervision and the paper's own limitation statement suggest the same substitution mechanism would work there.","The reported reduction of roughly 41% in image tokens during pretraining implies MMCS could lower alignment-stage compute cost as well as data cost; the paper reports the token count but does not model total FLOPs or energy.","If the observed trend continues past 1M samples, object-level code-switching could combine naturally with bootstrapping loops in which the model proposes its own entity-grounding pairs, a path the paper lists as future work but does not test.","The part-of-speech negative-log-likelihood results suggest the benefit is not confined to nouns: verbs and prepositions also become more predictable, hinting that object-level grounding could scaffold learning of relations and actions without explicit relation labels."],"forward_implications":["A 50K-sample MMCS pretraining run outperforms a 600K-sample image-caption run on the same downstream SFT, so the data-efficiency gain is a factor of roughly 12, not an incremental improvement.","Grounding gains hold across model scales: 3B and 8B LLMs with different vision encoders all improve on RefCOCO, RefCOCO+, and RefCOCOg, averaging +7.9%.","The two loss terms have separable roles: dropping $\\mathcal{L}_{\\mathrm{entity}}$ costs 4.0 points on perception, while dropping the language-modeling loss costs 4.9 points on grounding; both are needed.","The alignment learned during interleaved pretraining transfers to standard global-image SFT and inference, requiring no coordinate tokens or grounding modules at inference time.","Representation-level evidence (CKA, CKNNA, Mutual k-NN) and sharper attention maps are consistent with the claim that explicit object-entity supervision, not merely more data, improves cross-modal alignment."],"supporting_citations":[{"why":"Grounding DINO localizes each extracted textual entity to a bounding box, supplying the object-entity correspondences that MMCS substitutes into the caption.","marker":"(Liu et al., 2024c)"},{"why":"SAM-2.1 generates segmentation masks used to filter out heavily occluded or unreliable object regions before substitution.","marker":"(Ravi et al., 2025)"},{"why":"Qwen3-VL-32B-Instruct produces the detailed captions that are the textual backbone of the synthesized pretraining set.","marker":"(Bai et al., 2025a)"},{"why":"Qwen2.5-72B-Instruct extracts the noun-phrase entities with attribute descriptions that make one-to-one grounding possible.","marker":"(Yang et al., 2024)"},{"why":"Patch Aligned Training is the fine-grained patch-level baseline that MMCS must surpass in the data-efficiency and alignment comparisons.","marker":"(Jiang et al., 2025)"},{"why":"The LLaVA-NeXT instruction-following dataset is the fixed SFT stage applied after every pretraining variant, isolating the effect of the alignment method.","marker":"(Liu et al., 2024b)"},{"why":"The CKNNA and Mutual k-NN metrics, along with the representational-alignment hypothesis, provide the framework the paper uses to argue that better alignment explains downstream gains.","marker":"(Huh et al., 2024)"}],"fun_headline_variants":["MMCS: interleaving visual objects into text cuts data need 12x","Object-level alignment from code-switching beats image-caption pairs","Swap text entities for visual tokens to ground MLLMs with 12x less data","Explicit object grounding via code-switching makes MLLM data 12x efficient","Forget global captions: interleave objects for explicit MLLM alignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automated pipeline produces one-to-one object-entity correspondences accurate enough at scale to teach rather than mislead, and that the object-level grounding learned in interleaved pretraining still transfers when SFT and inference revert to global image representations.","fun_headline_variants_meta":{"raw":{"variants":["MMCS: interleaving visual objects into text cuts data need 12x","Object-level alignment from code-switching beats image-caption pairs","Swap text entities for visual tokens to ground MLLMs with 12x less data","Explicit object grounding via code-switching makes MLLM data 12x efficient","Forget global captions: interleave objects for explicit MLLM alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000236,"raw_usage":{"total_tokens":1513,"prompt_tokens":963,"completion_tokens":550,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":447}},"tokens_in":579,"tokens_out":550,"duration_ms":5370,"temperature":1.0,"reasoning_tokens":447,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:52:57.571183+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 50K MMCS pretraining set, randomly replace 10% of the object bounding boxes with unrelated same-size boxes from the same image, and train the identical Qwen2.5-3B setup; if that noised run no longer beats the 600K image-caption baseline on RefCOCO, RefCOCO+, and RefCOCOg, the data-efficiency claim depends on near-perfect synthetic correspondences rather than on the code-switching format itself.","supporting_citations":[{"cited_title":"Forty-first International Conference on Machine Learning,","cited_arxiv_id":null,"evidence_quote":"The CKNNA and Mutual k-NN metrics, along with the representational-alignment hypothesis, provide the framework the paper uses to argue that better alignment explains downstream gains."}],"review_version":1}