Pith. sign in

REVIEW 5 major objections 5 minor 50 references

Pre-training a vision encoder with hierarchical cross-attention to a language model yields richer visual representations and better vision-language performance than feeding flattened image embeddings, while cutting training cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A three-stage pre-training scheme that feeds multi-layer vision features into an LLM reports marginal benchmark gains, but lacks data, code, and ablations needed to support the claim.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection HIVE's core idea is plausible but the evidence stops before the point of transfer; the missing ablation of retaining cross-attention at fine-tuning makes the headline claims unsupported. the 5 major comments →

arxiv 2604.00086 v2 pith:UI7PZNFY submitted 2026-03-31 cs.CV cs.AIcs.CLcs.LG

Hierarchical Pre-Training of Vision Encoders with Large Language Model

Eugene Lee , Ting-Yu Chang , Jui-Huang Tsai , Jiajie Diao , Chen-Yi Lee This is my paper
classification cs.CV cs.AIcs.CLcs.LG
keywords hierarchical cross-attentionvision encoder pre-trainingvision-language alignmentthree-stage traininggradient flowfine-grained classificationmultimodal reasoningtraining efficiency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a vision encoder produces stronger features when, during pre-training, its intermediate layers are directly connected to matching layers of a large language model through cross-attention, rather than only feeding the final-layer output. This is claimed to improve gradient flow into early encoder layers, preserving fine-grained visual details, and to boost both image classification and vision-language benchmarks. The authors argue this is a new way to pre-train the vision encoder itself, not just to train an adapter for a frozen encoder. If correct, it offers a path to better multimodal models without changing how they are deployed: the hierarchical connections are used only during pre-training, and the improved encoder is later used with standard, cheaper fine-tuning.

Core claim

The paper claims that HIVE, which inserts cross-attention between a selected 25% of vision-encoder layers and corresponding LLM layers during pre-training, produces a vision encoder whose final-layer representations are better for downstream tasks than those from the same encoder trained with self-attention-based integration. The claim is supported by classification results on eight datasets and vision-language results on four benchmarks, where HIVE outperforms the self-attention baseline. The paper also claims substantial training efficiency gains, reporting a roughly 3x per-epoch speedup and about 55-59% lower peak memory. The hierarchical connections are deliberately omitted during fine-t

What carries the argument

The central mechanism is hierarchical cross-attention: at a subset of uniformly sampled vision-encoder depths, the projected features are used as keys and values in cross-attention layers inserted into the LLM, while queries come from the LLM's own hidden states. This physically routes intermediate visual features into the language model, so the language-modeling loss directly supervises all sampled encoder levels, not just the final layer. A three-stage training schedule (projector only; then projector plus LLM; then full end-to-end) is used to stabilize this integration.

Load-bearing premise

The central claim depends on the untested assumption that the benefits of multi-layer cross-attention during pre-training survive the switch to a standard single-layer downstream architecture, since the paper provides no experiment that removes this architecture shift while keeping the three-stage schedule fixed.

What would settle it

Train the same vision encoder with the same three-stage schedule but replace hierarchical cross-attention with a single final-layer projection into the LLM, keeping downstream fine-tuning identical; if the vision-language and classification scores do not change, the hierarchical mechanism is not the source of the reported gains.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If correct, standard late-fusion vision-language models can be improved without changing their inference architecture: pre-train with hierarchical cross-attention, then deploy using only the final-layer tokens.
  • The claimed efficiency gains mean larger vision encoders or datasets could be pre-trained within a fixed compute budget, potentially improving scalability of multimodal models.
  • The reported gains on fine-grained classification benchmarks (Pets, Caltech-256, Cars) and on VQA benchmarks (OK-VQA, ScienceQA) suggest that hierarchical pre-training improves both low-level feature retention and high-level semantic alignment.
  • The framework is positioned as a vision-encoder pre-training method, so it could be combined with any downstream encoder consumer, including linear probes and various LLM adapters.
  • The paper's gradient and attention visualizations, while qualitative, indicate that the cross-attention objective changes where gradients concentrate in early layers, offering a potential diagnostic tool for representation quality.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's own regularization hypothesis implies that the pre-trained encoder could be plugged into any downstream token-consumer architecture, not just the standard connector used in fine-tuning; this is testable by evaluating the pre-trained encoder with different adapters.
  • The 25% layer-selection density and its placement (uniform for one encoder, later for another) are not ablated; the reported gains might be sensitive to which layers are connected, so sweeping connection density and location is a direct next experiment.
  • Because the three-stage schedule is asserted without an ablation, a controlled comparison that removes hierarchical cross-attention while keeping the schedule fixed would separate the contribution of the schedule from the contribution of the hierarchy itself.
  • The pre-training experiments use only a small language model; whether the benefits transfer to larger LLMs remains open, and testing at 1B+ scale would clarify the practical value of the method.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes HIVE, a pre-training framework that connects multiple layers of a vision encoder (CLIP or SigLIP) to an LLM via hierarchical cross-attention, using a three-stage training recipe (projector-only, projector+LLM, end-to-end). After pre-training, the cross-attention mechanism is discarded: classification is evaluated with a linear probe on the final vision-encoder layer, and VLM fine-tuning uses standard LLaVA-style token concatenation. The authors claim that this hierarchical pre-training produces superior vision encoders, reporting accuracy gains on eight classification benchmarks and improvements on MME, GQA, OK-VQA, and ScienceQA over CLIP/SigLIP baselines and over a self-attention (SA) variant trained with the same three-stage recipe. They also claim substantial training efficiency gains.

Significance. If the central claim were established, the paper would make a useful contribution: using hierarchical cross-attention with an LLM as a pre-training regularizer to improve final-layer visual features, without paying the cross-attention cost at inference, is a plausible and interesting idea. However, the manuscript as written does not establish that claim. The novel mechanism is removed in all downstream evaluations, the pre-training data is undisclosed, the VLM results have no variance estimates, and key design choices (25% density, layer placement) are selected from unreported 'preliminary observations.' These are load-bearing gaps. The paper does make a good-faith effort to compare HIVE against a self-attention baseline trained with the same recipe, and it reports means/SD for classification, but the evidence is currently too weak to support the abstract's strong claims. The idea is worth pursuing; the present version needs substantial additional experimentation and disclosure.

major comments (5)
  1. [Section 3.6; Tables 1-2] The central claim is untested at the point of transfer. HIVE's only novel component is hierarchical cross-attention during pre-training, but all downstream evaluations remove it: a linear probe on the final layer for classification, and LLaVA-style token concatenation for VLM tasks. Section 3.6 hypothesizes that cross-attention acts as a 'pre-training regularization objective' that leaves final-layer representations richer, but no experiment compares HIVE pre-training evaluated with versus without the cross-attention layers retained, nor SA pre-training under the retained-architecture condition. Without this control, the gains in Tables 1 and 2 could be caused by the three-stage schedule or by optimization differences rather than by hierarchical cross-attention. This is the load-bearing link and requires a direct ablation.
  2. [Section 4.1; Table 2] VLM results are reported as single runs with no variance or significance testing. For example, MME 1298 vs 1263, GQA 58.05 vs 57.69, OK-VQA 51.01 vs 46.19, ScienceQA 63.12 vs 59.56. With one seed it is impossible to assess whether these differences are reliable. In Table 1, many HIVE-vs-SA differences are within one standard deviation (e.g., ImageNet-1K CLIP: 84.08±0.00 vs 84.14±0.02; SigLIP CIFAR-10: 98.45±0.04 vs 98.42±0.04). Please provide multi-seed results and statistical tests, or substantially soften the performance claims.
  3. [Section 4.1] The pre-training dataset is never disclosed. The paper lists evaluation datasets only. Since HIVE is a pre-training method, the identity, size, and filtering of the pre-training corpus are essential for reproducibility and for assessing contamination with evaluation benchmarks such as ImageNet-1K, CIFAR, and the VLM benchmarks. Please specify the pre-training dataset(s), any deduplication or overlap-removal procedure, and whether the evaluation sets or their near-duplicates were excluded.
  4. [Section 4.5] The 25% connection density and the per-backbone layer-placement strategy (uniform for CLIP, later layers for SigLIP) are chosen from 'preliminary observations' that are not reported. If those observations used the reported test benchmarks, then part of the improvement may be selection on the test set. Please report the preliminary experiments, the selection criterion, and ideally an ablation over density (e.g., 12.5%, 25%, 50%) and over placement schemes. This is especially important because the paper claims the 25% setup 'balances performance and efficiency' without showing the trade-off.
  5. [Section 3.7; Eq. (5); Tables 6-9] The complexity comparison is not well-defined. L_s is defined as the number of selected vision-encoder layers, but each selected layer still contributes N_v patch tokens; cross-attention complexity should scale with L_s * N_v * d, not L_s * d^2. As written, Eq. (5) omits N_v entirely, making the claimed savings appear much larger than they are. There are also numerical inconsistencies: Section 4.6 and Table 9 report ~55-59% memory reduction, while Table 6 reports memory overhead of 0.8x (i.e., 20% reduction), and Section 4.6 says '3x' speedup while Table 8 reports 3.43x. Please correct the derivation and reconcile the reported numbers.
minor comments (5)
  1. [Section 3.1] The text says the overall architecture is shown in 'Figure 4,' but Figure 4 in the supplementary is a conceptual comparison of hierarchical vs cascaded pre-training. The architecture overview should refer to Figure 1.
  2. [References] Reference [25] is incomplete: 'I Loshchilov. Decoupled weight decay regularization' should be Loshchilov and Hutter, AdamW, ICLR 2019.
  3. [Section 4.2] The text says 'we adopt the training procedure outlined in LLaVA [41]' but reference [41] is LLaVA-o1; the standard LLaVA reference is [23]. Please fix the citation.
  4. [Section 4.1] Pre-training uses MobileLLM-350M as the LLM, while VLM fine-tuning uses Llama-3.2-1B-Instruct. This is a cross-LLM transfer that is not discussed. Please clarify whether this is intentional and whether the SA baseline underwent the same cross-LLM transfer.
  5. [Tables 6 and 9] The memory numbers are inconsistent: Table 6 reports 'Memory Overhead 0.8x' (a 20% reduction), Table 9 reports a 59.3% reduction, and Section 4.6 reports 55%. Align these values and specify exactly what is being measured.

Circularity Check

0 steps flagged

No circularity: the core claim is an empirical transfer result; the unablated downstream shift is a reproducibility gap, not a by-construction equivalence.

full rationale

The paper's derivation chain is empirical rather than definitional. The pre-training objective (Eq. 1) is generic next-token prediction; the hierarchical cross-attention mechanism is specified by explicit formulas (Section 3.3); and the downstream evaluation removes the cross-attention layers, so the reported gains are a transfer hypothesis ('hierarchical cross-attention acts as a rigorous pre-training regularization objective', Section 3.6). That hypothesis is testable and is tested by comparing HIVE against an SA baseline that follows the same three-stage schedule, so the comparison is not forced by construction. The paper openly states that it does not ablate the projection warm-up stages (Section 3.5) and describes the downstream architecture shift as a 'deliberate design choice' (Section 3.6); these are missing controls and reproducibility concerns, not circular reductions. The 25% connection density is described as 'selected from preliminary observations' (Section 4.5), but the paper does not disclose what those observations were or whether they used the reported benchmarks; without that evidence, this is an under-specified hyperparameter choice, not a demonstrated fit renamed as prediction. Citations to LLaVA and DeepSeek-VL for the three-stage schedule are external prior work, not self-citations, and they are not the load-bearing justification for the claimed cross-attention benefit. No equation in the paper is shown to equal its own input, and no fitted parameter is re-presented as a prediction. Therefore no significant circularity is present.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

The paper introduces no new theoretical entities. Its load-bearing elements are design choices (density, layer placement) selected without disclosed criteria, plus domain assumptions about staged training and data purity. The lack of a named pre-training dataset is the most serious gap.

free parameters (2)
  • connection density (25%) = 25% of vision encoder layers
    Chosen from 'preliminary observations' without reporting the selection procedure; directly controls the hierarchical cross-attention design (Section 4.5).
  • layer selection strategy = uniform for CLIP, later-heavy for SigLIP
    Hand-set per backbone to balance performance and efficiency; described after seeing results (Section 4.5).
axioms (3)
  • domain assumption Three-stage progressive unlocking is required for convergence; removing warm-up stages would fail
    Stated in Section 3.5, but the paper explicitly says 'we do not ablate the removal'; cited to prior work [21, 26] rather than demonstrated for HIVE.
  • domain assumption Next-token prediction on image-text data pre-trains a vision encoder to produce representations useful for classification and VQA
    The entire method rests on this objective shaping visual features; no direct validation that final-layer features improve due to this loss beyond the reported benchmarks.
  • domain assumption The pre-training dataset and evaluation sets are disjoint / no contamination
    The pre-training dataset is never named; if it overlaps with CIFAR, ImageNet, GQA, or OK-VQA, the results are invalid. This is unstated and unverified.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Hierarchical Pre-Training of Vision Encoders with Large Language Model." pith.science (2026). https://pith.science/paper/UI7PZNFY

@misc{pith2026260400086,
  author       = {Pith},
  title        = {Pith review of: Hierarchical Pre-Training of Vision Encoders with Large Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UI7PZNFY}},
  note         = {Machine review of arXiv:2604.00086}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The field of computer vision has experienced significant advancements through scalable vision encoders and multimodal pre-training frameworks. However, existing approaches often treat vision encoders and large language models (LLMs) as independent modules, limiting the integration of hierarchical visual features. In this work, we propose HIVE (Hierarchical Pre-Training of Vision Encoders), a novel framework that enhances vision-language alignment by introducing hierarchical cross-attention between the vision encoder and LLM. Unlike conventional methods that flatten image embeddings, HIVE enables structured feature fusion across multiple layers, improving gradient flow and representation learning. To optimize this interaction, we introduce a three-stage training strategy that progressively aligns the vision encoder with the LLM, ensuring stable optimization and effective multimodal fusion. Empirical evaluations demonstrate that HIVE achieves superior performance not only in image classification but also on various vision-language tasks, outperforming self-attention-based methods in benchmarks such as MME, GQA, OK-VQA, and ScienceQA. Our results highlight the benefits of hierarchical feature integration, paving the way for more efficient and expressive vision-language models.

Figures

Figures reproduced from arXiv: 2604.00086 by Chen-Yi Lee, Eugene Lee, Jiajie Diao, Jui-Huang Tsai, Ting-Yu Chang.

Figure 1
Figure 1. Figure 1: Overview of the proposed Hierarchical Pre-Training of [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Gradient map visualization from the first to the final [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of hierarchical and cascaded pre-training approaches for vision encoders. Hierarchical pre-training (left) establishes [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Gradient map visualization for Sample 2: ”Colorful [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 5
Figure 5. Figure 5: Gradient map visualization for Sample 1: ”Investigators [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Gradient map visualization for Sample 3: ”Person makes [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 10
Figure 10. Figure 10: Gradient map visualization for Sample 6. HIVE con [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Gradient map visualization for Sample 7. Stable gra [PITH_FULL_IMAGE:figures/full_fig_p016_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Attention map visualization for Sample 1: ”Investigators and journalists gather around the car of person after an attack on [PITH_FULL_IMAGE:figures/full_fig_p017_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Attention map visualization for Sample 2: ”Colorful plastic and aluminum chairs leaning against tables at a cafe outdoor dining [PITH_FULL_IMAGE:figures/full_fig_p017_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Attention map visualization for Sample 3: ”Person makes a move on defenders during the spring game.” HIVE effectively [PITH_FULL_IMAGE:figures/full_fig_p017_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 13 linked inside Pith

  1. [1]

    Clip the bias: how useful is balancing data in multimodal learning? arXiv preprint arXiv:2403.04547, 2024

    Ibrahim Alabdulmohsin, Xiao Wang, Andreas Steiner, Priya Goyal, Alexander D’Amour, and Xiaohua Zhai. Clip the bias: how useful is balancing data in multimodal learning? arXiv preprint arXiv:2403.04547, 2024. 3

    Pith/arXiv arXiv 2024

  2. [2]

    Getting vit in shape: Scaling laws for compute-optimal model design.Advances in Neural Information Processing Systems, 36:16406–16425, 2023

    Ibrahim M Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting vit in shape: Scaling laws for compute-optimal model design.Advances in Neural Information Processing Systems, 36:16406–16425, 2023. 2

    2023

  3. [3]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736,

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736,

  4. [4]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization.Text Reading, and Beyond, 2,

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization.Text Reading, and Beyond, 2,

  5. [5]

    Effective conditioned and composed im- age retrieval combining clip-based features

    Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Al- berto Del Bimbo. Effective conditioned and composed im- age retrieval combining clip-based features. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 21466–21474, 2022. 2

    2022

  6. [6]

    Text2live: Text-driven layered image and video editing

    Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kas- ten, and Tali Dekel. Text2live: Text-driven layered image and video editing. InEuropean conference on computer vi- sion, pages 707–723. Springer, 2022. 3

    2022

  7. [7]

    Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models.Ad- vances in neural information processing systems, 32, 2019

    Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models.Ad- vances in neural information processing systems, 32, 2019. 2

    2019

  8. [8]

    Are we done with imagenet?arXiv preprint arXiv:2006.07159, 2020

    Lucas Beyer, Olivier J H ´enaff, Alexander Kolesnikov, Xi- aohua Zhai, and A ¨aron van den Oord. Are we done with imagenet?arXiv preprint arXiv:2006.07159, 2020. 3

    Pith/arXiv arXiv 2006

  9. [9]

    Flexivit: One model for all patch sizes

    Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Kornblith, Xiaohua Zhai, Matthias Minderer, Michael Tschannen, Ibrahim Alabdulmohsin, and Filip Pavetic. Flexivit: One model for all patch sizes. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14496–14506, 2023. 3

    2023

  10. [10]

    Paligemma: A versatile 3b vlm for trans- fer.arXiv preprint arXiv:2407.07726, 2024

    Lucas Beyer, Andreas Steiner, Andr ´e Susano Pinto, Alexan- der Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for trans- fer.arXiv preprint arXiv:2407.07726, 2024. 3

    Pith/arXiv arXiv 2024

  11. [11]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. InInternational conference on ma- chine learning, pages 1597–1607. PMLR, 2020. 2

    2020

  12. [12]

    Vqgan-clip: Open domain image generation and editing with natural language guidance

    Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Ed- ward Raff. Vqgan-clip: Open domain image generation and editing with natural language guidance. InEuropean Con- ference on Computer Vision, pages 88–105. Springer, 2022. 3

    2022

  13. [13]

    Patch n’pack: Navit, a vision transformer for any aspect ratio and resolution.Advances in Neural Infor- mation Processing Systems, 36:2252–2274, 2023

    Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdul- mohsin, et al. Patch n’pack: Navit, a vision transformer for any aspect ratio and resolution.Advances in Neural Infor- mation Processing Systems, 36:2252–2274, 2023. 3

    2023

  14. [14]

    Data curation via joint example selection further accelerates multimodal learning.Advances in Neural Information Processing Systems, 37:141240–141260, 2024

    Talfan Evans, Nikhil Parthasarathy, Hamza Merzic, and Olivier Henaff. Data curation via joint example selection further accelerates multimodal learning.Advances in Neural Information Processing Systems, 37:141240–141260, 2024. 3

    2024

  15. [15]

    Improving clip training with language rewrites.Advances in Neural Information Processing Sys- tems, 36:35544–35575, 2023

    Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites.Advances in Neural Information Processing Sys- tems, 36:35544–35575, 2023. 3

    2023

  16. [16]

    Mul- timodal autoregressive pre-training of large vision encoders

    Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, Sai Aitharaju, Victor Guil- herme Turrisi da Costa, Louis B´ethune, Zhe Gan, et al. Mul- timodal autoregressive pre-training of large vision encoders. arXiv preprint arXiv:2411.14402, 2024. 1, 2, 3

    Pith/arXiv arXiv 2024

  17. [17]

    Clipdraw: Exploring text-to-drawing synthesis through language-image encoders.Advances in Neural Information Processing Sys- tems, 35:5207–5218, 2022

    Kevin Frans, Lisa Soros, and Olaf Witkowski. Clipdraw: Exploring text-to-drawing synthesis through language-image encoders.Advances in Neural Information Processing Sys- tems, 35:5207–5218, 2022. 3

    2022

  18. [18]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000– 16009, 2022. 2

    2022

  19. [19]

    Knowledge-aware prompt tun- ing for generalizable vision-language models

    Baoshuo Kan, Teng Wang, Wenpeng Lu, Xiantong Zhen, Weili Guan, and Feng Zheng. Knowledge-aware prompt tun- ing for generalizable vision-language models. InProceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 15670–15680, 2023. 3

    2023

  20. [20]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 2, 4, 5

    Pith/arXiv arXiv 2024

  21. [21]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models.arXiv preprint arXiv:2407.07895, 2024

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models.arXiv preprint arXiv:2407.07895, 2024. 4, 5

    Pith/arXiv arXiv 2024

  22. [22]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InIn- ternational conference on machine learning, pages 19730– 19742. PMLR, 2023. 1, 3

    2023

  23. [23]

    Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024. 6

    2024

  24. [24]

    Mo- bilellm: Optimizing sub-billion parameter language models for on-device use cases

    Zechun Liu, Changsheng Zhao, Forrest Iandola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, et al. Mo- bilellm: Optimizing sub-billion parameter language models for on-device use cases. InForty-first International Confer- ence on Machine Learning, 2024. 6

    2024

  25. [25]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    I Loshchilov. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017. 6, 2, 3

    Pith/arXiv arXiv 2017

  26. [26]

    Deepseek-vl: towards real-world vision- language understanding.arXiv preprint arXiv:2403.05525,

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision- language understanding.arXiv preprint arXiv:2403.05525,

    Pith/arXiv arXiv

  27. [27]

    Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26439–26455, 2024. 2

    2024

  28. [28]

    Lexlip: Lexicon-bottlenecked language-image pre-training for large-scale image-text sparse retrieval

    Ziyang Luo, Pu Zhao, Can Xu, Xiubo Geng, Tao Shen, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Lexlip: Lexicon-bottlenecked language-image pre-training for large-scale image-text sparse retrieval. InProceedings of the IEEE/CVF international conference on computer vision, pages 11206–11217, 2023. 2

    2023

  29. [29]

    Pro-tuning: Unified prompt tuning for vision tasks.IEEE Transactions on Cir- cuits and Systems for Video Technology, 2023

    Xing Nie, Bolin Ni, Jianlong Chang, Gaofeng Meng, Chun- lei Huo, Shiming Xiang, and Qi Tian. Pro-tuning: Unified prompt tuning for vision tasks.IEEE Transactions on Cir- cuits and Systems for Video Technology, 2023. 1, 3

    2023

  30. [30]

    Tokenize anything via prompting

    Ting Pan, Lulu Tang, Xinlong Wang, and Shiguang Shan. Tokenize anything via prompting. InEuropean Conference on Computer Vision, pages 330–348. Springer, 2025. 2, 3

    2025

  31. [31]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, pages 8748–8763. PMLR, 2021. 2

    2021

  32. [32]

    Hierarchical text-conditional image gener- ation with clip latents.arXiv preprint arXiv:2204.06125, 1 (2):3, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gener- ation with clip latents.arXiv preprint arXiv:2204.06125, 1 (2):3, 2022. 2

    Pith/arXiv arXiv 2022

  33. [33]

    Learning vi- sual prompts for guiding the attention of vision transformers

    Razieh Rezaei, Masoud Jalili Sabet, Jindong Gu, Daniel Rueckert, Philip Torr, and Ashkan Khakzar. Learning vi- sual prompts for guiding the attention of vision transformers. arXiv preprint arXiv:2406.03303, 2024. 2

    Pith/arXiv arXiv 2024

  34. [34]

    Clip for all things zero-shot sketch-based image retrieval, fine- grained or not

    Aneeshan Sain, Ayan Kumar Bhunia, Pinaki Nath Chowd- hury, Subhadeep Koley, Tao Xiang, and Yi-Zhe Song. Clip for all things zero-shot sketch-based image retrieval, fine- grained or not. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2765– 2775, 2023. 2

    2023

  35. [35]

    Umg-clip: a unified multi-granularity vision generalist for open-world understanding

    Bowen Shi, Peisen Zhao, Zichen Wang, Yuhang Zhang, Yaoming Wang, Jin Li, Wenrui Dai, Junni Zou, Hongkai Xiong, Qi Tian, et al. Umg-clip: a unified multi-granularity vision generalist for open-world understanding. InEuropean Conference on Computer Vision, pages 259–277. Springer,

  36. [36]

    Generative multimodal mod- els are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiy- ing Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal mod- els are in-context learners. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14398–14409, 2024. 2

    2024

  37. [37]

    Alpha- clip: A clip model focusing on wherever you want

    Zeyi Sun, Ye Fang, Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Alpha- clip: A clip model focusing on wherever you want. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13019–13029, 2024. 2

    2024

  38. [38]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024. 3

    2024

  39. [39]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 2, 3

    Pith/arXiv arXiv 2024

  40. [40]

    Diffusion feedback helps clip see better.CoRR, 2024

    Wenxuan Wang, Quan Sun, Fan Zhang, Yepeng Tang, Jing Liu, and Xinlong Wang. Diffusion feedback helps clip see better.CoRR, 2024. 3

    2024

  41. [41]

    Llava-o1: Let vision language models reason step- by-step.arXiv preprint arXiv:2411.10440, 2024

    Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-o1: Let vision language models reason step- by-step.arXiv preprint arXiv:2411.10440, 2024. 2, 6, 1

    Pith/arXiv arXiv 2024

  42. [42]

    Tcp: Textual- based class-aware prompt tuning for visual-language model

    Hantao Yao, Rui Zhang, and Changsheng Xu. Tcp: Textual- based class-aware prompt tuning for visual-language model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23438–23448, 2024. 3

    2024

  43. [43]

    Coca: Contrastive captioners are image-text foundation models.arXiv preprint arXiv:2205.01917, 2022

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mo- jtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models.arXiv preprint arXiv:2205.01917, 2022. 2

    Pith/arXiv arXiv 2022

  44. [44]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 2, 1

    2023

  45. [45]

    Learning to prompt for vision-language models.In- ternational Journal of Computer Vision, 130(9):2337–2348,

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models.In- ternational Journal of Computer Vision, 130(9):2337–2348,

  46. [47]

    Hierarchical vs. Cascaded Pre-Training In this section, we compare hierarchical and cascaded pre- training approaches for vision encoders, highlighting their impact on gradient propagation and feature integration. Hierarchical Pre-TrainingHierarchical pre-training es- tablishes direct cross-attention between multiple layers of the vision encoder and the L...

  47. [48]

    For tokenization, we adopt the tokenizer used by SigLIP [44] and truncate any text longer than 77 tokens

    Hyperparameters Pre-TrainingWe outline the optimization hyperparame- ters and data augmentations used during HIVE pre-training in Table 3. For tokenization, we adopt the tokenizer used by SigLIP [44] and truncate any text longer than 77 tokens. Classifier Fine-TuningThe optimization hyperparame- ters used during classifier fine-tuning are detailed in Ta- ...

  48. [49]

    Table 6 presents the measured training cost and memory overhead during model pretraining

    Computational Efficiency We evaluate the computational efficiency of hierarchical cross-attention compared to self-attention methods. Table 6 presents the measured training cost and memory overhead during model pretraining. Our method achieves improved efficiency by applying cross-attention to only25%of the vision encoder layers, significantly reducing th...

  49. [50]

    Gradient and Attention Map Visualizations In this section, we present qualitative visualizations of gra- dient flow and attention maps to illustrate the impact of hier- archical cross-attention on feature extraction and alignment. 9.1. Gradient Flow Analysis Figures 5 to 11 present gradient map visualizations for var- ious sample images. Each figure shows...

  50. [2022]

    1, 3 Hierarchical Pre-Training of Vision Encoders with Large Language Models Supplementary Material

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.