Pith. sign in

REVIEW 3 major objections 6 minor 15 references

GRR-CoCa: Leveraging LLM Mechanisms in Multimodal Model Architectures

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that modernizing CoCa's vision transformer with GEGLU, RMSNorm, and RoPE improves vision-language pretraining and fine-tuning across all reported metrics with nearly unchanged model size.

desk verdict Clean controlled ablation of GEGLU/RMSNorm/RoPE in CoCa's vision encoder, but single-run results make 'significantly' unsupported and the paper overclaims SOTA. read the letter →

arxiv 2507.18009 v1 pith:ZYQSOVQU submitted 2025-07-24 cs.CV cs.AIcs.CL

Jake R. Patock , Nicole Catherine Lewis , Kevin McCoy , Christina Gomez , Canling Chen , Lorenzo Luzi This is my paper
classification cs.CVcs.AIcs.CL
keywords vision-languagemodelCoCavisiontransformerGEGLURMSNormrotarypositionembeddingimagecaptioningcontrastivelearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes that a vision-language model's visual encoder can be modernized by transplanting three ideas from large language models: gated feedforward networks with GELU activation (GEGLU), root-mean-square normalization (RMSNorm) in place of LayerNorm, and rotary position embeddings (RoPE) in place of absolute position encodings. The authors build GRR-CoCa by applying these changes to the vision transformer of the Contrastive Captioner (CoCa), while a control model, Baseline CoCa, keeps the original ViT but shares the same upgraded text decoders. On the CC12M pretraining set, GRR-CoCa reports a 27.25% lower contrastive loss, 3.71% lower perplexity, and 7.15% lower CoCa loss than the control; averaged over three fine-tuning datasets (MSCOCO, ROCO, Flickr30K), it reports 13.66%, 5.18%, and 5.55% improvements on the same metrics. The paper's central claim is that these gains come from the encoder architecture itself, since parameter count grows by only 0.17% and the control already contains the decoder changes. Why this matters is that if the claim holds, any ViT-based multimodal model can adopt the same LLM building blocks without a redesign to get better alignment and generation for roughly the same computational cost.

What carries the argument

The mechanism is the upgraded vision transformer block. It replaces the standard feedforward network with a Gated Linear Unit using GELU activation, where the input goes through two parallel linear projections and one is gated by the other via a Hadamard product; to keep parameter count nearly constant, the hidden dimension is scaled by 2.7 instead of 4. It replaces LayerNorm pre-normalization with RMSNorm, which drops the mean-centering term and keeps the per-dimension scaling, simplifying the layer and, per the paper, reducing noise modeling. It replaces fixed absolute position encodings with rotary position embeddings, which rotate query and key vectors by position-dependent angles and are reapplied inside every block rather than added once to the input. These three changes sit inside CoCa's dual-task training loop, where one contrastive loss aligns image and text latents and one autoregressive captioning loss measures per-token cross-entropy, so the same modified encoder has to serve both alignment and generation.

What would settle it

Train both models five or more times with different random seeds, and compute 95% confidence intervals for perplexity and contrastive loss on the same validation splits; if the intervals overlap for any dataset—particularly ROCO, where the reported gain is only 2.40%—the architectural effect is not established. Reporting BLEU or CIDEr on the fine-tuned models would also settle whether the loss improvements translate to visibly better captions.

Watch

Extended reading notes

Core claim

On its own terms, the discovery is that the visual encoder, not just the text side, is a bottleneck that modern LLM components can relieve. The paper's controlled comparison isolates exactly this: Baseline CoCa has the modified text decoders, so the reported differences in Table 2 are attributed entirely to the ViT exchange of feedforward, LayerNorm, and absolute position encoding for GEGLU, RMSNorm, and RoPE. The modified model wins on every reported metric on every dataset—pretraining and all three fine-tuning sets—with the largest pretraining gain, 27.25% in contrastive loss, and the smallest, 2.40% in ROCO perplexity. The authors interpret this as the encoder producing more feature-rich image latents that simultaneously improve image-text alignment and caption generation.

Load-bearing premise

The paper assumes that the difference between one training run of GRR-CoCa and one training run of Baseline CoCa is caused by the architecture, not by run-to-run randomness; no seeds, error bars, or statistical tests are reported.

Editorial extensions

If this is right

  • Any ViT-based vision-language model can swap in GEGLU, RMSNorm, and RoPE with negligible parameter change and likely see similar loss reductions on its own validation sets.
  • The largest gains appear in contrastive loss, so the encoder modifications mainly improve image-text alignment; generative perplexity improves less, implying caption quality gains may be secondary.
  • GRR-CoCa improves on ROCO, a medical image-captioning domain far from general web data, which the paper takes as evidence that the architectural changes improve transfer to new distributions.
  • Because the authors report only validation loss and perplexity, the claims apply to training objectives; the paper's next step, scaling to larger datasets and benchmarking on the original CoCa tasks, will test whether the gains persist.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A component-wise ablation with RoPE alone, RMSNorm alone, and GEGLU alone is the natural unrun experiment; without it, the paper cannot say which of the three mechanisms carries the improvement, leaving the gating and positional-persistence explanations untested.
  • If the transfer claim is right, the same three modifications should improve other contrastive image-text encoders that still use absolute positions and LayerNorm, such as older CLIP-style encoders; that is a direct testable prediction the paper does not make.
  • The paper reports no standard captioning metrics like BLEU or CIDEr and no comparison with published CoCa results, so a fair reader should treat the practical, end-user benefit as an open question rather than an established quantity.
  • The paper's explanation for RoPE—that absolute position information attenuates in deeper layers while per-block rotation preserves it—can be tested directly with attention-map or positional-probe experiments, which the paper does not include.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. GRR-CoCa is a modification of the Contrastive Captioner (CoCa) in which GEGLU feedforward blocks, RMSNorm, and rotary positional embedding (RoPE) are inserted into both the unimodal/multimodal text decoders and the ViT encoder. The authors train two models: GRR-CoCa and Baseline CoCa, where Baseline CoCa has the same modified text decoders but retains the original ViT feedforward, LayerNorm, and absolute position encoding. Models are pretrained on CC12M and fine-tuned on MSCOCO, ROCO, and Flickr30K. Table 2 reports validation CoCa loss, perplexity, and contrastive loss; GRR-CoCa shows lower (better) values on all 12 reported metrics, with percentage improvements from 2.40% to 27.25%. The authors interpret this as evidence that LLM-style ViT modifications improve performance and generalization at near-constant parameter count.

Significance. The controlled design is a real strength: because the two models share modified text decoders and differ only in the ViT, any measured difference can be attributed to the ViT changes, and the parameter counts are nearly identical. The direction of the effect is consistent across all twelve metrics, and the components have independent prior support in the LLM literature. If supported by appropriate uncertainty quantification and held-out evaluation, the result would be a useful, low-cost recipe for updating ViT-based multimodal encoders. At present, however, the empirical evidence is not sufficient to support the abstract's 'significantly outperformed' and 'improves performance and generalization' claims, because each comparison rests on one training run and on validation metrics that were used for early stopping.

major comments (3)
  1. [Section 4 / Table 2 / Abstract] Every reported improvement comes from a single pretraining run and single fine-tuning run per model; no seeds, standard deviations, confidence intervals, or statistical tests are provided. The word 'significantly' in the abstract is therefore not supported. The smallest gains (ROCO perplexity -2.40%, ROCO CoCa loss -2.69%) are of a magnitude that can arise from run-to-run stochasticity in large-scale training, so the causal claim that the ViT modifications improve performance is not yet established. At minimum, the authors should train both models with multiple seeds and report means and intervals or an appropriate inferential comparison.
  2. [Section 3.6 / Table 2] The reported fine-tuning numbers are validation metrics obtained after early stopping with a soft-reset mechanism that returns to the best validation state. Because the same validation split is used for model selection and for reporting, the numbers in Table 2 are optimistically selected and are not unbiased estimates of generalization on a held-out test set. The authors should report metrics on a separate test split, or clearly state that these are validation-selected values and avoid the term 'generalization'.
  3. [Section 3.5 / Table 2 / Section 6.2] Evaluation is limited to contrastive loss and caption perplexity/CoCa loss; no standard captioning metrics (e.g., BLEU, CIDEr, SPICE) and no comparison to published CoCa results are reported. Section 6.2 concedes that benchmarking on the original CoCa tasks is future work. Consequently, the abstract's claim that the architecture 'improves performance and generalization across vision-language domains' is broader than the evidence: the paper demonstrates lower training-objective losses on validation splits of four datasets, not improved task performance relative to external baselines. The title's 'improved SOTA CoCa' is also not supported because the control is an author-built baseline, not the original CoCa.
minor comments (6)
  1. [Section 3.4] The table caption contains the typo 'Talbe 1'; it should read 'Table 1'.
  2. [Table 2] The dataset name is misspelled as 'Flicker30K' in Table 2; it should be 'Flickr30K'.
  3. [Section 2.1.2] The section title 'Root Mean Squared Error Normalization' should be 'Root Mean Square Layer Normalization', matching the cited work by Zhang and Sennrich (2019).
  4. [Section 3.5, Eq. (1) and Eq. (4)] In Eq. (1), the bias symbols W b, V b, and Ob are confusing because they resemble products of matrices and vectors; please use distinct bias notation and specify tensor shapes. In Eq. (4), y_t is used both as the label vector and as its element, which conflicts with the notation in Eq. (3); please clarify.
  5. [Section 5 / Section 6.1] The Discussion attributes the gains separately to GEGLU, RMSNorm, and RoPE, but no ablations are reported to isolate these components; the authors should either add ablations or soften the component-level attributions.
  6. [General] No code or checkpoint release is mentioned, which prevents independent verification of the reported numbers; a data/code availability statement would strengthen the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the architecture comparison is a controlled empirical benchmark with externally supported components.

full rationale

The paper's central claim is an empirical comparison between GRR-CoCa and Baseline CoCa, which are identical except for the ViT encoder: 'Baseline CoCa' used a ViT submodel that followed the original architecture from Alexey [2020]. GRR-CoCa used a modified ViT that exchanged the feedforward network for GEGLUs, absolute positional encoding for RoPe, and LayerNorms for RMSNorms. This control structure means the reported losses and perplexities are measured outcomes of two independently trained models, not quantities implied by the definitions of the modifications themselves. GEGLU, RMSNorm, and RoPE are established components supported by external prior work (Shazeer 2020, Zhang and Sennrich 2019, Su et al. 2024), and no load-bearing step relies on a self-citation or an author-imported uniqueness theorem. No equation is shown to equal another by construction, no fitted parameter is renamed as a prediction, and no known result is merely relabeled. The main scientific weakness, that all reported gains come from a single training run per model without seeds, error bars, or statistical tests, is a question of inferential robustness rather than circularity. Since the comparison is self-contained against a purpose-built baseline and the architectural components carry independent external evidence, the circularity score is 0.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The paper contributes an empirical comparison, not a derivation, so the ledger lists the hand-chosen training hyperparameters the comparison depends on and the domain assumptions about evaluation. No invented entities are introduced; GRR-CoCa is a configuration of existing components.

free parameters (5)
  • GEGLU hidden layer scaler = 2.7
    Hand-chosen in Section 3.4 to keep GRR-CoCa's parameter count close to Baseline CoCa (0.17% larger); this value controls the capacity of the modified feedforward and therefore the fairness of the comparison.
  • Loss weights lambda_Con, lambda_Cap = 2 and 1 during pretraining; 1 and 2 during fine-tuning
    Hand-chosen in Section 3.5; the pretraining weights make contrastive loss improvements dominate the reported CoCa loss during pretraining.
  • Contrastive temperature sigma = not reported
    Appears in Eq. 2 but is never assigned a value in the paper; it scales all contrastive logits and is required to reproduce the reported losses.
  • Dropout rates = 0.15 pretraining, 0.1 fine-tuning
    Universal dropout values in Section 3.4; chosen by hand and applied identically to both models.
  • Optimizer and schedule hyperparameters = AdamW lr 1e-4/1e-6 pretrain, 1e-5/1e-7 fine-tune; CAWR one cycle per epoch; soft-reset factor 100
    Hand-chosen training settings in Section 3.6; the soft-reset early stopper selects the best validation state and influences the reported numbers.
assumptions (3)
  • domain assumption Validation loss and perplexity are treated as proxies for captioning quality and generalization.
    The paper's generalization claims rest on validation loss and perplexity only; no BLEU, CIDEr, or human evaluation is reported (Section 4).
  • domain assumption The downloaded splits of CC12M, MSCOCO, ROCO, and Flickr30K are clean and representative enough for the comparison to be informative.
    The paper uses web-sourced image-caption data with noisy captions (CC12M) as a stand-in for JFT (Sections 3.1 to 3.2).
  • domain assumption Attribution of the gains to the ViT modifications assumes the shared training protocol favors neither model.
    Both models share all hyperparameters except the architectural changes and the GEGLU scale; no per-model tuning or repeated seeds is reported (Sections 3.4 to 3.6).

how reviews work

0 comments
Cite this review

Pith. "Pith review of GRR-CoCa: Leveraging LLM Mechanisms in Multimodal Model Architectures." pith.science (2026). https://pith.science/paper/ZYQSOVQU

@misc{pith2026250718009,
  author       = {Pith},
  title        = {Pith review of: GRR-CoCa: Leveraging LLM Mechanisms in Multimodal Model Architectures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZYQSOVQU}},
  note         = {Machine review of arXiv:2507.18009}
}
read the original abstract

State-of-the-art (SOTA) image and text generation models are multimodal models that have many similarities to large language models (LLMs). Despite achieving strong performances, leading foundational multimodal model architectures frequently lag behind the architectural sophistication of contemporary LLMs. We propose GRR-CoCa, an improved SOTA Contrastive Captioner (CoCa) model that incorporates Gaussian error gated linear units, root mean squared normalization, and rotary positional embedding into the textual decoders and the vision transformer (ViT) encoder. Each architectural modification has been shown to improve model performance in LLMs, but has yet to be adopted in CoCa. We benchmarked GRR-CoCa against Baseline CoCa, a model with the same modified textual decoders but with CoCa's original ViT encoder. We used standard pretraining and fine-tuning workflows to benchmark the models on contrastive and generative tasks. Our GRR-CoCa significantly outperformed Baseline CoCa on the pretraining dataset and three diverse fine-tuning datasets. Pretraining improvements were 27.25% in contrastive loss, 3.71% in perplexity, and 7.15% in CoCa loss. The average fine-tuning improvements were 13.66% in contrastive loss, 5.18% in perplexity, and 5.55% in CoCa loss. We show that GRR-CoCa's modified architecture improves performance and generalization across vision-language domains.

Figures

Figures reproduced from arXiv: 2507.18009 by the authors.

Figure 1
Figure 1. Overview of GRR-CoCa vs. Baseline-CoCa ViT-based visual encoder architecture. The GRR-CoCa model [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Architectural overview of GRR-CoCa and Baseline CoCa’s attention poolers, unimodal textual decoder, and [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 2 canonical work pages

  1. [3]

    GPT-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [5]

    CoCa: Contrastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. CoCa: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917,

    arXiv

  3. [7]

    GLU variants improve transformer

    Noam Shazeer. GLU variants improve transformer. arXiv preprint arXiv:2002.05202,

    arXiv 2002

  4. [8]

    Gemma 2: Improving open language models at a practical size

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118,

    arXiv

  5. [9]

    Nemotron-4 340B technical report

    Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. Nemotron-4 340B technical report. arXiv preprint arXiv:2406.11704,

    arXiv

  6. [10]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671,

    arXiv

  7. [11]

    URL https://huggingface.co/datasets/laion/ conceptual-captions-12m-webdataset . 11 GRR-CoCa: Leveraging LLM Mechanisms in Multimodal Model Architectures A PREPRINT Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In Computer Vision–ECCV 20...

    work page 2014

  8. [12]

    PubMedClip: How much does CLIP benefit visual question answering in the medical domain? In Findings of the Association for Computational Linguistics: EACL 2023, pages 1181–1193,

    Sedigheh Eslami, Christoph Meinel, and Gerard De Melo. PubMedClip: How much does CLIP benefit visual question answering in the medical domain? In Findings of the Association for Computational Linguistics: EACL 2023, pages 1181–1193,

    work page 2023

Show all 15 references
  1. [14]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101,

    arXiv

  2. [2017]

    SGDR: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983,

    arXiv

  3. [2019]

    The Llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  4. [2020]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee,

    2009

  5. [2022]

    Visionllama: A unified LLaMA interface for vision tasks

    Xiangxiang Chu, Jianlin Su, Bo Zhang, and Chunhua Shen. Visionllama: A unified LLaMA interface for vision tasks. arXiv preprint arXiv:2403.00522,

    arXiv

  6. [2023]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy Alexey. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 2010.11929,

    2010 arXiv

  7. [2024]

    DeepSeek-V3 technical report

    10 GRR-CoCa: Leveraging LLM Mechanisms in Multimodal Model Architectures A PREPRINT Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.