HIRE: Lightweight High-Resolution Image Feature Enrichment for Multimodal LLMs

Aradhya Neeraj Mathur; Mausoom Sarkar; Nikitha SR; Rishabh Jain; Tarun Ram Menta

REVIEW 4 major objections 4 minor 13 references

HIRE: Lightweight High-Resolution Image Feature Enrichment for Multimodal LLMs

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 250k-parameter feature enricher generates high-resolution vision features from a single low-resolution ViT forward pass, cutting FLOPs by 35% while staying competitive on fine-grained visual QA.

desk verdict HIRE's efficiency numbers are solid, but the competitive claim rests on an unmatched baseline; worth reviewing for the method, not the conclusion. read the letter →

arxiv 2506.17608 v1 pith:3CN76X7S submitted 2025-06-21 cs.CV

Nikitha SR , Aradhya Neeraj Mathur , Tarun Ram Menta , Rishabh Jain , Mausoom Sarkar This is my paper · ORCID

classification cs.CV

keywords featureenrichmenthigh-resolutionimagesmultimodallargelanguagemodelsjointbilateralupsamplingUNetvisiontransformervisualquestionansweringefficientinference

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

High-resolution inputs improve multimodal LLMs' fine-grained perception, but current methods pay for this by running the vision transformer many times on image crops. This paper argues that the expensive part is unnecessary: instead of computing multiple high-resolution feature maps, you can enrich a single low-resolution ViT feature map with details pulled from the high-resolution image by a tiny 250k-parameter module. The module, HIRE, combines a small UNet that extracts multi-scale semantic guidance with a stack of joint bilateral upsampling layers that progressively double the feature resolution. The result is claimed to match or beat the multi-crop baseline S2 on several visual question answering and fine-grained localization benchmarks at 672px, while cutting first-token FLOPs from 6.66 to 4.32 TFLOPs and roughly halving inference time on a VQA test set. If right, it makes high-resolution MLLM training and inference substantially cheaper without giving up fine-grained understanding.

What carries the argument

The load-bearing mechanism is the joint bilateral upsampling (JBU) layer, a parameterized module that produces a locality-aware bilateral kernel from guidance features and applies it to upsample the low-resolution ViT feature map by a factor of two. A five-module UNet acts as the semantic detail retriever: it processes the full high-resolution image and provides guidance features at successive scales ($1\times$, $1/2\times$, $1/4\times$, $1/8\times$, $1/16\times$), and a stack of five JBU layers uses these to double the feature resolution in steps, ending at the original image resolution. The enriched high-resolution features are then average-pooled and concatenated with the original low-resolution features, so the LLM context length is unchanged while the feature quality carries high-frequency detail. The whole enricher is about 250k parameters, which is what keeps the added FLOPs negligible compared with extra ViT calls.

What would settle it

Feed the UNet a blurred or downsampled version of the high-resolution image as guidance (for example, downsample to 336px and bilinearly upsample back to 672px) and re-measure V*Spa and MMT-Bench keypoint detection; if the score does not drop, HIRE is not actually using high-frequency image detail, and the enrichment mechanism claim fails.

Watch

Extended reading notes

Core claim

HIRE's central claim is that a shallow feature enricher—a five-stage UNet plus a parameterized stack of joint bilateral upsampling layers, totalling about 250k parameters—can synthesize high-resolution visual features directly, so that a high-resolution MLLM needs only one forward pass through its vision encoder instead of the many passes required by cropping methods. Given a high-resolution image I and the encoder's output $E(i)$ for the downsampled image $i$, the UNet extracts features at scales from $1\times$ to $1/16\times$; each JBU layer uses the UNet's next-finer scale as guidance to create a locality-aware kernel and doubles the resolution of $E(i)$. The resulting enriched features are then average-pooled back to the original $24\times 24$ scale and concatenated with $E(i)$ before the projector. In experiments on LLaVA-1.5 with CLIP-336, HIRE at 672px outperforms vanilla LLaVA and remains competitive with the S2 method at 1008px on several VQA benchmarks, and beats S2 on fine-grained MMT-Bench subtasks like pixel localization and keypoint detection; the authors also report a 35% FLOP reduction and more than $2\times$ faster inference than S2 on the VQAT test set. The paper notes that HIRE falls behind on text-heavy benchmarks (VQAT and InfographicVQA), attributing this to difficulty enriching text-filled images.

Load-bearing premise

The comparison against the S2 baseline assumes that the S2 scores quoted in Table 3 were obtained under fairly comparable training and evaluation conditions, even though only one of those scores was re-run by the authors under their own protocol.

Editorial extensions

If this is right

High-resolution MLLM pipelines can keep the same number of visual tokens as low-resolution ones, because HIRE pools enriched features back to the encoder's native scale before concatenation, so the LLM context length does not change.
The compute cost of adding high-resolution information no longer scales with the number of crops or with ViT size: HIRE's overhead stays roughly constant when moving from ViT-L to ViT-H/G, while the multi-crop S2 approach adds more FLOPs.
At 672px, HIRE roughly matches S2 at 1008px on GQA and SEED (within about 0.3 points), exceeds vanilla LLaVA on VQA v2, GQA, and SEED, and beats S2 by about 10 points on MMT-Bench pixel localization and dense captioning and about 15 points on keypoint detection.
Inference time on the VQAT test set drops to 23.2 minutes versus 43 minutes for S2, a more than $2\times$ saving.
HIRE is not a win on text-heavy images: VQAT and InfographicVQA scores fall below both S2 and vanilla LLaVA, which the paper leaves as future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the enriched-then-pooled features are what carry the gain, then the benefit comes from improved feature quality rather than longer context; this predicts that an even simpler guided-interpolation enricher without a UNet would recover part of the gain, while a larger UNet would buy more.
The fixed 250k-parameter enricher is resolution-agnostic in spirit, so it may transfer to resolutions or aspect ratios never seen in training; a cheap test is to train at 672px and evaluate at 1008px without retraining.
The text-heavy failure suggests the JBU guidance cannot synthesize legible glyphs; adding an OCR-aware loss or a separate text-region enrichment stream could extend HIRE to document VQA, where multi-crop methods still dominate.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a real efficiency contribution with a measurement problem. The core idea—replace multiple ViT forward passes in high-res MLLMs with a 250k-parameter UNet+JBU enricher—is sensible and clearly motivated. The FLOPs analysis (Table 1, Fig. 3) is the paper's strongest asset: it is self-contained, reproducible in principle, and shows HIRE adding negligible overhead across ViT scales. That part holds up.

What's genuinely new is the application of feature upsampling (FeatUp/LiFT line) inside an MLLM to avoid multi-crop encoding, plus the concrete 250k design. The authors cite FeatUp and LiFT, so they're not overselling novelty. The MMT-Bench fine-grained gains are interesting if they hold.

The soft spots are experimental, not conceptual. The 'competitive' and 'surpasses' wording in the conclusion is not backed by Table 3: HIRE wins only a handful of benchmarks and clearly trails S2 on DocVQA, InfoQA, and V*Spa. The stress-test note is right that only V*Att is re-run under the authors' protocol; the rest of the S2 scores are quoted from the original paper, which used full fine-tuning at 1008 resolution vs. HIRE's LoRA at 672. If those quoted scores are protocol-inflated, the gaps may shrink—but the authors can't claim superiority on the current evidence. Single runs without error bars add to the uncertainty. The paper acknowledges the text-heavy weakness and leaves it to future work, which is fair.

I'd also want code/data and a matched re-run of S2 under the same LoRA/resolution scheme before trusting the performance comparison. The efficiency story is solid enough that the paper deserves a serious referee, but it needs major revision on the evaluation protocol, not on the method itself.

Who benefits: people working on efficient MLLMs or high-res VQA will use the FLOPs numbers and the enricher design. I'd bring it to a reading group for the efficiency angle, and I'd cite the FLOPs analysis if I wrote about MLLM efficiency.

Recommendation: send to peer review with the demand for matched baselines and error bars; desk rejection would lose a useful efficiency result.

Referee Report

4 major / 4 minor

Summary. The paper proposes HIRE, a lightweight module (approximately 250k parameters) combining a small UNet with a stack of joint bilateral upsampling (JBU) layers to enrich low-resolution CLIP ViT features with high-resolution image detail. The goal is to replace the multiple high-resolution ViT forward passes used by S2-style tiling in multimodal LLMs. HIRE is inserted into LLaVA-1.5, trained in a two-stage procedure with LoRA on the LLM, and evaluated on VQA benchmarks, V*, hallucination tasks, and MMT-Bench. The authors report large FLOPs savings relative to S2, competitive or better scores on several benchmarks, and scaling experiments showing that the overhead is nearly flat as the ViT backbone grows.

Significance. The core idea is simple and practically valuable: if a shallow feature enricher can substitute for multiple ViT passes, high-resolution MLLMs become substantially cheaper. The paper provides a direct FLOPs accounting with fvcore (Table 1), transparently marks one re-run of an S2 checkpoint (V*Att in Table 3), includes ablations of the UNet and learning rates (Table 4), and explicitly acknowledges that text-heavy VQA tasks remain weak. These are genuine strengths. However, the empirical case is weakened by unmatched S2 baselines, single runs without variance, and a headline claim that couples 1008-resolution FLOPs with 672-resolution accuracy. The idea is worth publishing if the comparison protocol and claims are corrected.

major comments (4)

[Table 3 / Sec. 3]
[Sec. 1 / Tables 1 and 3]
[Sec. 4 (Conclusion)]
[Sec. 3 / Tables 2-3]

minor comments (4)

[Abstract / Sec. 1]
[Table 4]
[Sec. 3]
[Fig. 3]

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: HIRE is an empirical system trained on standard instruction data and evaluated on external benchmarks, with no equation-level reduction of predictions to inputs.

full rationale

The paper's central claim is that a 250k-parameter UNet plus JBU stack can generate enriched high-resolution features, replacing multiple ViT forward passes in high-resolution MLLMs. The method is defined explicitly as Eenr(I) = Concat(E(i), pool(x_en)) with x_en = J(U(I;phi), E(i);theta), and the enricher is trained in a two-stage LLaVA-style instruction-tuning loop with LoRA. No benchmark score is used to define the architecture or its loss; performance is measured on external tasks such as V*, DocVQA, MMT-Bench, and GQA. Consequently, there is no fitted parameter that is later renamed as a prediction, and no target quantity is present in the training objective by construction. The ablations in Table 4 select the UNet component and learning rate using V*, but this is standard hyperparameter and component selection, not circular derivation. The method builds on externally published feature upsampling ideas (FeatUp [4], LIFT [10]) and compares against an external baseline S2 [9]; there are no load-bearing self-citations, uniqueness theorems, or ansatz smuggled in via the authors' prior work. Concerns about the S2 baseline numbers not being reproduced under a matched training protocol are legitimate evaluation-comparability risks, but they do not constitute circularity: the quoted S2 scores are external data, not outputs of the HIRE pipeline. The FLOPs comparison in Table 1 is computed with Fvcore for both models and is an arithmetic efficiency accounting, not a derivation that presupposes the claimed conclusion. Overall, the paper does not exhibit any circular step in which a prediction is equivalent to an input by definition.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities. The enricher is a learned neural network, not a postulated physical or theoretical object. The central efficiency claim relies on standard empirical assumptions about ViT features, JBU behavior, training data sufficiency, and FLOPs measurement.

assumptions (4)

domain assumption CLIP ViT features provide a useful semantic representation for MLLMs.
The method builds on LLaVA-1.5 which uses CLIP ViT-L/14-336; no independent evidence in this paper for this premise.
domain assumption Joint bilateral upsampling (JBU) can faithfully transfer high-resolution guidance from the UNet into ViT features.
Adopted from FeatUp [4]; the paper does not validate this mechanism independently beyond its own results.
domain assumption Training on LLaVA's pretraining and finetuning data is sufficient to teach the enricher without dedicated high-resolution supervision.
The enricher is trained with the standard LLaVA pipeline; no auxiliary loss is used.
domain assumption FLOPs measured by Fvcore reflect representative inference cost.
FLOPs for first token generation are reported for a single image and single text token; real conversations may differ.

how reviews work

0 comments

Cite this review

Pith. "Pith review of HIRE: Lightweight High-Resolution Image Feature Enrichment for Multimodal LLMs." pith.science (2026). https://pith.science/paper/3CN76X7S

@misc{pith2026250617608,
  author       = {Pith},
  title        = {Pith review of: HIRE: Lightweight High-Resolution Image Feature Enrichment for Multimodal LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3CN76X7S}},
  note         = {Machine review of arXiv:2506.17608}
}

read the original abstract

The integration of high-resolution image features in modern multimodal large language models has demonstrated significant improvements in fine-grained visual understanding tasks, achieving high performance across multiple benchmarks. Since these features are obtained from large image encoders like ViT, they come with a significant increase in computational costs due to multiple calls to these encoders. In this work, we first develop an intuition for feature upsampling as a natural extension of high-resolution feature generation. Through extensive experiments and ablations, we demonstrate how a shallow feature enricher can achieve competitive results with tremendous reductions in training and inference time as well as computational cost, with upto 1.5x saving in FLOPs.

Figures

Figures reproduced from arXiv: 2506.17608 by the authors.

**Figure 2.** We show the original LLava-based methods (left), mod [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Required FLOPs for first token generation for LLaVA1.5, LLaVA1.5-S [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 5 canonical work pages

[1]

Flamingo: a visual language model for few-shot learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,
[2]

Dragonfly: Multi-resolution zoom supercharges large visual-language model

Kezhen Chen, Rahul Thapa, Rahul Chalamala, Ben Athi- waratkun, Shuaiwen Leon Song, and James Zou. Dragonfly: Multi-resolution zoom supercharges large visual-language model. arXiv preprint arXiv:2406.00977, 2024. 1

arXiv 2024
[3]

An image is worth 16x16 words: Transformers for image recognition at scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representa- tions, 2021. 1

2021
[4]

Featup: A model- agnostic framework for features at any resolution

Stephanie Fu, Mark Hamilton, Laura Brandt, Axel Feldman, Zhoutong Zhang, and William T Freeman. Featup: A model- agnostic framework for features at any resolution. arXiv preprint arXiv:2403.10516, 2024. 1, 2

arXiv 2024
[5]

Lora: Low-rank adaptation of large language models

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 3

arXiv 2021
[6]

Improved baselines with visual instruction tuning, 2024

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2024. 1, 3

work page 2024
[7]

Dinov2: Learning robust visual features with- out supervision, 2024

Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mah- moud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herv ´e Je- gou, Julien Mairal, ...

2024
[8]

Learning transferable visual models from natural language supervi- sion

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748–8763. PMLR, 2021. 1

2021

Show all 13 references

[9]

When do we not need larger vision models?, 2024

Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang, and Trevor Darrell. When do we not need larger vision models?, 2024. 1, 2, 3

2024
[10]

Lift: A surprisingly simple lightweight feature transform for dense vit descriptors

Saksham Suri, Matthew Walmer, Kamal Gupta, and Abhinav Shrivastava. Lift: A surprisingly simple lightweight feature transform for dense vit descriptors. In European Conference on Computer Vision, pages 110–128. Springer, 2025. 1

2025
[11]

Dragonfly: Multi-resolution zoom-in encoding enhances vision-language models, 2024

Rahul Thapa, Kezhen Chen, Ian Covert, Rahul Chalamala, Ben Athiwaratkun, Shuaiwen Leon Song, and James Zou. Dragonfly: Multi-resolution zoom-in encoding enhances vision-language models, 2024. 1, 2

2024
[12]

Cogvlm: Visual expert for pretrained language models, 2024

Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models, 2024. 1

2024
[13]

Sigmoid loss for language image pre-training

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 1

2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Flamingo: a visual language model for few-shot learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

[2] [2]

Dragonfly: Multi-resolution zoom supercharges large visual-language model

Kezhen Chen, Rahul Thapa, Rahul Chalamala, Ben Athi- waratkun, Shuaiwen Leon Song, and James Zou. Dragonfly: Multi-resolution zoom supercharges large visual-language model. arXiv preprint arXiv:2406.00977, 2024. 1

arXiv 2024

[3] [3]

An image is worth 16x16 words: Transformers for image recognition at scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representa- tions, 2021. 1

2021

[4] [4]

Featup: A model- agnostic framework for features at any resolution

Stephanie Fu, Mark Hamilton, Laura Brandt, Axel Feldman, Zhoutong Zhang, and William T Freeman. Featup: A model- agnostic framework for features at any resolution. arXiv preprint arXiv:2403.10516, 2024. 1, 2

arXiv 2024

[5] [5]

Lora: Low-rank adaptation of large language models

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 3

arXiv 2021

[6] [6]

Improved baselines with visual instruction tuning, 2024

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2024. 1, 3

work page 2024

[7] [7]

Dinov2: Learning robust visual features with- out supervision, 2024

Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mah- moud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herv ´e Je- gou, Julien Mairal, ...

2024

[8] [8]

Learning transferable visual models from natural language supervi- sion

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748–8763. PMLR, 2021. 1

2021