{"id":"0bd9c4d9-6c64-4c87-bdac-79bdd0e82938","arxiv_id":"2506.17608","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A lightweight UNet plus joint bilateral upsampling produces high-resolution image features for MLLMs, replacing multiple ViT forward passes and saving up to 1.5x FLOPs with competitive VQA performance.","lead":"HIRE replaces expensive multi-crop vision encoding in high-resolution multimodal LLMs with a single 250,000-parameter feature enricher, cutting first-token FLOPs by up to 35 percent at competitive benchmark scores. A shallow UNet and joint bilateral upsampling module synthesize high-resolution features from one low-resolution ViT pass plus the original image.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HIRE's central efficiency-performance claim rests on S2 baseline numbers that were not re-run under a matched training protocol; if those quoted scores are protocol-inflated, the 'competitive' conclusion may not hold.","rationale":"The reader's CONDITIONAL verdict is appropriate. The paper proposes a genuinely lightweight feature enricher: a 250k-parameter UNet plus JBU stack that replaces multiple ViT forward passes, and Table 1 shows the extra FLOPs are small (4.296 vs 4.278 TFLOPs over vanilla). HIRE improves over vanilla LLaVA-1.5 on many benchmarks, and the VQAT inference-time comparison (23.2 min vs 43 min) supports the practical efficiency claim. Independent support, however, is limited: no code or checkpoints are released, the abstract claims training-time reductions that are never measured, and all benchmark numbers are single runs. The dominant issue is the uncontrolled S2 baseline. Because the S2 scores in Table 3 are mostly quoted from [9] rather than re-run under HIRE's training recipe, the observed performance differences may reflect protocol mismatch rather than method capability. This is not a conceptual flaw in the architecture; it is a measurement-control flaw. A matched-protocol re-run of S2 would settle whether the 'competitive' claim is substantive. The conclusion's statement that HIRE 'surpasses' S2 is also an overstatement of the reported numbers, where HIRE wins 4 of 12 benchmarks and loses 7, with the largest losses on text-heavy VQA tasks that the authors explicitly acknowledge. These issues are addressable with more careful experimentation and reporting, so a conditional acceptance is the right call.","tokens_in":5913,"tokens_out":13862,"duration_ms":124702,"concrete_test":"Retrain S2 under the exact HIRE training protocol described in §3—same LLaVA-1.5 backbone, same LoRA configuration, same 2-stage pretraining and finetuning data, same evaluation scripts—and compare the full Table 3 benchmark set, including DocVQA and InfoQA, to HIRE. If S2's re-run scores on DocVQA or InfoQA drop by more than ~2 points relative to the quoted values, the baseline was protocol-inflated and HIRE's 'competitive' claim is substantiated; if they remain near the quoted levels, HIRE's text-heavy-VQA deficit is real and the claim must be qualified. Report 3-seed means and standard deviations to bound noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is that the central claim—'HIRE achieves a 35% reduction in FLOPs ... while remaining competitive in performance'—is evaluated against S2 numbers in Table 3 that are not controlled for training protocol. The table's footnote marks only V*Att as re-run (superscript T); the remaining S2 scores are quoted from [9]. HIRE is trained with LoRA on the LLM, a 2-stage pretrain/finetune, and 672 resolution, whereas the quoted S2 results come from the original paper's recipe at 1008 resolution. Single runs without error bars amplify this mismatch. If the quoted S2 scores are inflated by its original training setup (e.g., different data mixture, learning rates, LoRA vs full fine-tuning), then the apparent performance gaps—DocVQA 20.7 vs 26.0, InfoQA 18.93 vs 21.9, V*Spa 60.5 vs 61.8—could shrink or reverse under a matched protocol. The efficiency numbers (Table 1) are plausible and the VQAT inference-time comparison is useful, but the 'competitive' and 'surpasses' wording in the conclusion is premature: on the reported numbers HIRE wins only 4 of 12 benchmarks (plus a tie), with the largest losses on text-heavy VQA, which the authors acknowledge. The value proposition depends on the tradeoff being favorable; without a matched baseline, the tradeoff is not reliably quantified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HIRE, a lightweight module (approximately 250k parameters) combining a small UNet with a stack of joint bilateral upsampling (JBU) layers to enrich low-resolution CLIP ViT features with high-resolution image detail. The goal is to replace the multiple high-resolution ViT forward passes used by S2-style tiling in multimodal LLMs. HIRE is inserted into LLaVA-1.5, trained in a two-stage procedure with LoRA on the LLM, and evaluated on VQA benchmarks, V*, hallucination tasks, and MMT-Bench. The authors report large FLOPs savings relative to S2, competitive or better scores on several benchmarks, and scaling experiments showing that the overhead is nearly flat as the ViT backbone grows.","tokens_in":6146,"tokens_out":7110,"duration_ms":69053,"significance":"The core idea is simple and practically valuable: if a shallow feature enricher can substitute for multiple ViT passes, high-resolution MLLMs become substantially cheaper. The paper provides a direct FLOPs accounting with fvcore (Table 1), transparently marks one re-run of an S2 checkpoint (V*Att in Table 3), includes ablations of the UNet and learning rates (Table 4), and explicitly acknowledges that text-heavy VQA tasks remain weak. These are genuine strengths. However, the empirical case is weakened by unmatched S2 baselines, single runs without variance, and a headline claim that couples 1008-resolution FLOPs with 672-resolution accuracy. The idea is worth publishing if the comparison protocol and claims are corrected.","major_comments":[{"comment":"","section":"Table 3 / Sec. 3"},{"comment":"","section":"Sec. 1 / Tables 1 and 3"},{"comment":"","section":"Sec. 4 (Conclusion)"},{"comment":"","section":"Sec. 3 / Tables 2-3"}],"minor_comments":[{"comment":"","section":"Abstract / Sec. 1"},{"comment":"","section":"Table 4"},{"comment":"","section":"Sec. 3"},{"comment":"","section":"Fig. 3"}],"recommendation":"major_revision","confidential_remarks":"The main risks are the unmatched S2 baseline and the resolution conflation in the headline efficiency claim; both are fixable within the scope of a revision. The paper would also benefit from releasing code and checkpoints so that the re-run of S2 under a matched protocol is feasible. The scope is appropriate for a vision-and-language venue, and I see no grounds for rejection if the comparison issues are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real efficiency contribution with a measurement problem. The core idea—replace multiple ViT forward passes in high-res MLLMs with a 250k-parameter UNet+JBU enricher—is sensible and clearly motivated. The FLOPs analysis (Table 1, Fig. 3) is the paper's strongest asset: it is self-contained, reproducible in principle, and shows HIRE adding negligible overhead across ViT scales. That part holds up.\n\nWhat's genuinely new is the application of feature upsampling (FeatUp/LiFT line) inside an MLLM to avoid multi-crop encoding, plus the concrete 250k design. The authors cite FeatUp and LiFT, so they're not overselling novelty. The MMT-Bench fine-grained gains are interesting if they hold.\n\nThe soft spots are experimental, not conceptual. The 'competitive' and 'surpasses' wording in the conclusion is not backed by Table 3: HIRE wins only a handful of benchmarks and clearly trails S2 on DocVQA, InfoQA, and V*Spa. The stress-test note is right that only V*Att is re-run under the authors' protocol; the rest of the S2 scores are quoted from the original paper, which used full fine-tuning at 1008 resolution vs. HIRE's LoRA at 672. If those quoted scores are protocol-inflated, the gaps may shrink—but the authors can't claim superiority on the current evidence. Single runs without error bars add to the uncertainty. The paper acknowledges the text-heavy weakness and leaves it to future work, which is fair.\n\nI'd also want code/data and a matched re-run of S2 under the same LoRA/resolution scheme before trusting the performance comparison. The efficiency story is solid enough that the paper deserves a serious referee, but it needs major revision on the evaluation protocol, not on the method itself.\n\nWho benefits: people working on efficient MLLMs or high-res VQA will use the FLOPs numbers and the enricher design. I'd bring it to a reading group for the efficiency angle, and I'd cite the FLOPs analysis if I wrote about MLLM efficiency.\n\nRecommendation: send to peer review with the demand for matched baselines and error bars; desk rejection would lose a useful efficiency result.","headline":"HIRE's efficiency numbers are solid, but the competitive claim rests on an unmatched baseline; worth reviewing for the method, not the conclusion.","tokens_in":6746,"tokens_out":2086,"would_cite":true,"duration_ms":19613,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 250k-parameter feature enricher generates high-resolution vision features from a single low-resolution ViT forward pass, cutting FLOPs by 35% while staying competitive on fine-grained visual QA.","keywords":["feature enrichment","high-resolution images","multimodal large language models","joint bilateral upsampling","UNet","vision transformer","visual question answering","efficient inference"],"falsifier":"Feed the UNet a blurred or downsampled version of the high-resolution image as guidance (for example, downsample to 336px and bilinearly upsample back to 672px) and re-measure V*Spa and MMT-Bench keypoint detection; if the score does not drop, HIRE is not actually using high-frequency image detail, and the enrichment mechanism claim fails.","tokens_in":5686,"feed_emoji":"⚡","tokens_out":10593,"duration_ms":92025,"temperature":0.7,"pith_summary":"High-resolution inputs improve multimodal LLMs' fine-grained perception, but current methods pay for this by running the vision transformer many times on image crops. This paper argues that the expensive part is unnecessary: instead of computing multiple high-resolution feature maps, you can enrich a single low-resolution ViT feature map with details pulled from the high-resolution image by a tiny 250k-parameter module. The module, HIRE, combines a small UNet that extracts multi-scale semantic guidance with a stack of joint bilateral upsampling layers that progressively double the feature resolution. The result is claimed to match or beat the multi-crop baseline S2 on several visual question answering and fine-grained localization benchmarks at 672px, while cutting first-token FLOPs from 6.66 to 4.32 TFLOPs and roughly halving inference time on a VQA test set. If right, it makes high-resolution MLLM training and inference substantially cheaper without giving up fine-grained understanding.","feed_headline":"A 250k-parameter enricher cuts high-res MLLM compute by 35%","feed_subtitle":"Shallow UNet with joint bilateral upsampling replaces crop-wise ViT passes, staying competitive on visual QA.","key_machinery":"The load-bearing mechanism is the joint bilateral upsampling (JBU) layer, a parameterized module that produces a locality-aware bilateral kernel from guidance features and applies it to upsample the low-resolution ViT feature map by a factor of two. A five-module UNet acts as the semantic detail retriever: it processes the full high-resolution image and provides guidance features at successive scales ($1\\times$, $1/2\\times$, $1/4\\times$, $1/8\\times$, $1/16\\times$), and a stack of five JBU layers uses these to double the feature resolution in steps, ending at the original image resolution. The enriched high-resolution features are then average-pooled and concatenated with the original low-resolution features, so the LLM context length is unchanged while the feature quality carries high-frequency detail. The whole enricher is about 250k parameters, which is what keeps the added FLOPs negligible compared with extra ViT calls.","core_discovery":"HIRE's central claim is that a shallow feature enricher—a five-stage UNet plus a parameterized stack of joint bilateral upsampling layers, totalling about 250k parameters—can synthesize high-resolution visual features directly, so that a high-resolution MLLM needs only one forward pass through its vision encoder instead of the many passes required by cropping methods. Given a high-resolution image I and the encoder's output $E(i)$ for the downsampled image $i$, the UNet extracts features at scales from $1\\times$ to $1/16\\times$; each JBU layer uses the UNet's next-finer scale as guidance to create a locality-aware kernel and doubles the resolution of $E(i)$. The resulting enriched features are then average-pooled back to the original $24\\times 24$ scale and concatenated with $E(i)$ before the projector. In experiments on LLaVA-1.5 with CLIP-336, HIRE at 672px outperforms vanilla LLaVA and remains competitive with the S2 method at 1008px on several VQA benchmarks, and beats S2 on fine-grained MMT-Bench subtasks like pixel localization and keypoint detection; the authors also report a 35% FLOP reduction and more than $2\\times$ faster inference than S2 on the VQAT test set. The paper notes that HIRE falls behind on text-heavy benchmarks (VQAT and InfographicVQA), attributing this to difficulty enriching text-filled images.","pith_inferences":["If the enriched-then-pooled features are what carry the gain, then the benefit comes from improved feature quality rather than longer context; this predicts that an even simpler guided-interpolation enricher without a UNet would recover part of the gain, while a larger UNet would buy more.","The fixed 250k-parameter enricher is resolution-agnostic in spirit, so it may transfer to resolutions or aspect ratios never seen in training; a cheap test is to train at 672px and evaluate at 1008px without retraining.","The text-heavy failure suggests the JBU guidance cannot synthesize legible glyphs; adding an OCR-aware loss or a separate text-region enrichment stream could extend HIRE to document VQA, where multi-crop methods still dominate."],"forward_implications":["High-resolution MLLM pipelines can keep the same number of visual tokens as low-resolution ones, because HIRE pools enriched features back to the encoder's native scale before concatenation, so the LLM context length does not change.","The compute cost of adding high-resolution information no longer scales with the number of crops or with ViT size: HIRE's overhead stays roughly constant when moving from ViT-L to ViT-H/G, while the multi-crop S2 approach adds more FLOPs.","At 672px, HIRE roughly matches S2 at 1008px on GQA and SEED (within about 0.3 points), exceeds vanilla LLaVA on VQA v2, GQA, and SEED, and beats S2 by about 10 points on MMT-Bench pixel localization and dense captioning and about 15 points on keypoint detection.","Inference time on the VQAT test set drops to 23.2 minutes versus 43 minutes for S2, a more than $2\\times$ saving.","HIRE is not a win on text-heavy images: VQAT and InfographicVQA scores fall below both S2 and vanilla LLaVA, which the paper leaves as future work."],"supporting_citations":[{"why":"Supplies the LLaVA-1.5 base architecture and training pipeline into which HIRE is inserted, plus the vanilla baseline scores.","marker":"[6]"},{"why":"The main high-resolution baseline S2 that HIRE is compared against; provides the comparison scores and the FLOPs figures HIRE claims to beat.","marker":"[9]"},{"why":"Provides the joint bilateral upsampling idea that HIRE's enrichment stack is built on.","marker":"[4]"},{"why":"A lightweight feature transform for dense ViT descriptors that motivates the shallow-enricher approach.","marker":"[10]"},{"why":"An example high-resolution MLLM method requiring many ViT forward passes; motivates the efficiency problem HIRE addresses.","marker":"[11]"},{"why":"LoRA, the parameter-efficient finetuning method used to train the LLM layers during HIRE training.","marker":"[5]"},{"why":"CLIP, the vision encoder whose features HIRE enriches in the experiments.","marker":"[8]"},{"why":"Introduces the ViT architecture that underlies the vision encoders used throughout the paper.","marker":"[3]"}],"fun_headline_variants":["One ViT pass: 250k-param enricher saves 35% FLOPs","250k params replace multi-crop ViT, cutting MLLM FLOPs 35%","HIRE: 250k-param enricher yields high-res features sans extra ViT","Replace multi-crop ViT with a 250k-param enricher, save 35% FLOPs","High-res MLLM features without extra ViT calls: HIRE's enricher"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison against the S2 baseline assumes that the S2 scores quoted in Table 3 were obtained under fairly comparable training and evaluation conditions, even though only one of those scores was re-run by the authors under their own protocol.","fun_headline_variants_meta":{"raw":{"variants":["One ViT pass: 250k-param enricher saves 35% FLOPs","250k params replace multi-crop ViT, cutting MLLM FLOPs 35%","HIRE: 250k-param enricher yields high-res features sans extra ViT","Replace multi-crop ViT with a 250k-param enricher, save 35% FLOPs","High-res MLLM features without extra ViT calls: HIRE's enricher"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001509,"raw_usage":{"total_tokens":6065,"prompt_tokens":973,"completion_tokens":5092,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":4965}},"tokens_in":589,"tokens_out":5092,"duration_ms":34312,"temperature":1.0,"reasoning_tokens":4965,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:05:12.348637+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Feed the UNet a blurred or downsampled version of the high-resolution image as guidance (for example, downsample to 336px and bilinearly upsample back to 672px) and re-measure V*Spa and MMT-Bench keypoint detection; if the score does not drop, HIRE is not actually using high-frequency image detail, and the enrichment mechanism claim fails.","supporting_citations":[{"cited_title":"Improved baselines with visual instruction tuning, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the LLaVA-1.5 base architecture and training pipeline into which HIRE is inserted, plus the vanilla baseline scores."},{"cited_title":"When do we not need larger vision models?, 2024","cited_arxiv_id":null,"evidence_quote":"The main high-resolution baseline S2 that HIRE is compared against; provides the comparison scores and the FLOPs figures HIRE claims to beat."},{"cited_title":"Lift: A surprisingly simple lightweight feature transform for dense vit descriptors","cited_arxiv_id":null,"evidence_quote":"A lightweight feature transform for dense ViT descriptors that motivates the shallow-enricher approach."},{"cited_title":"Dragonfly: Multi-resolution zoom-in encoding enhances vision-language models, 2024","cited_arxiv_id":null,"evidence_quote":"An example high-resolution MLLM method requiring many ViT forward passes; motivates the efficiency problem HIRE addresses."}],"review_version":1}