{"id":"90a832e0-5078-4958-97f7-1ee9e3603bee","arxiv_id":"2505.06840","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"CoRoI injects a chain of language-guided image regions into LLM hidden layers and reports improved MLLM benchmark scores at 7B-34B scale.","lead":"This paper proposes CoRoI, a method that helps multimodal AI models process high-resolution images efficiently by selecting question-relevant regions instead of the whole image. The authors report consistent benchmark gains over similar open models, but the selector's training is unclear and the 'high-resolution' images are just upsampled low-resolution ones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The hard argmax in §3.1 gives the selection network f_θv zero gradient from the LM loss in §3.3, so CoRoI's central claim of learning informative regions is unsupported and gains may come from arbitrary crops.","rationale":"The decisive issue is not whether the empirical numbers are high—they may be—but whether the method's stated mechanism can exist as described. A hard argmax has zero gradient almost everywhere, so the score network f_θv cannot be trained by the only loss in §3.3. The paper explicitly lists θv among trainable parameters and claims differentiability, making this an internal contradiction. Without a learned selector, CoRoI reduces to 'inject any crops,' and the headline claim about prioritizing informative regions is unfalsified. This is the weakest assumption because all downstream evidence (benchmark gains, qualitative region chains) is consistent with an untrained or random selector. The paper's own limitation about bilinear upsampling is secondary: even interpolated crops can help by magnifying subregions, so the differentiability gap is the more fundamental problem. The missing code and single-run numbers increase the risk but are not the core logical flaw. I therefore keep the reader's REJECT verdict.","tokens_in":15365,"tokens_out":4836,"duration_ms":52258,"concrete_test":"Compute the gradient of the instruction-tuning loss with respect to θv through the full pipeline, ideally in the authors' code. If ‖∂ℓ/∂θv‖ is zero, the selector is untrained. Then run a controlled ablation: replace the learned selector with (a) a fixed center-crop chain and (b) random crops, keeping n, m, λ, and the cross-attention injection identical, and compare TextVQA, MMBench, and MME. If fixed/random crops reproduce CoRoI's gains, the 'informative-region' mechanism is not the cause; if they do not, the claim needs a differentiable training signal or a surrogate selection loss to be verified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 selects each RoI by taking the argmax over a sliding-window average-pooling score map produced by f_θv(VLow, U), then cropping integer pixel coordinates from the HR image. Section 3.3 trains θv by the causal LM loss ℓ = −Σ log pθ(yi|...), where θ includes θv. Because the argmax and the crop are non-differentiable, ∂ℓ/∂θv = 0 almost everywhere; no Gumbel/softmax relaxation, straight-through estimator, REINFORCE baseline, or auxiliary selection objective is described anywhere in the paper. Thus the network that supposedly implements 'language-guided' RoI selection receives no training signal, contradicting the Section 2 claim that 'the entire process is differentiable.' The central mechanism—that the model learns to prioritize the most informative regions—is therefore unsupported. The consistent benchmark gains over LLaVA-NeXT may be real but could come from injecting n×m fixed/random crops into the LLM via cross-attention, not from learned selection. The qualitative examples in Figure 3 cannot establish learning; they may be selected to fit the narrative. This is an internal inconsistency, not merely a departure from community practice.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CoRoI (Chain of Region-of-Interest), a visual instruction tuning method for high-resolution multimodal LLMs. CoRoI first uses low-resolution visual features and question embeddings to iteratively select a chain of high-resolution image regions through a sliding-window average-pooling argmax, then injects the resulting region tokens into intermediate LLM hidden states via cross-attention residual connections. The authors claim the entire pipeline is differentiable and trainable with the standard causal language modeling loss. Experiments are conducted on 11 benchmarks with Mistral-7B and Hermes-2-Yi-34B backbones, comparing with LLaVA-NeXT and proprietary models, and reporting consistent gains across most benchmarks.","tokens_in":15618,"tokens_out":5457,"duration_ms":56356,"significance":"If the claimed learning mechanism worked as described, CoRoI would be a useful efficiency/accuracy contribution to high-resolution MLLM design, and the experimental setup has notable strengths: the backbones match LLaVA-NeXT, the benchmark coverage is broad, and the ablation study is extensive. However, the central claim that the model learns to select informative regions is not supported by the described method, because the hard-argmax selector receives no gradient from the language modeling loss. This is a load-bearing issue, not a presentation issue. The empirical gains may be genuine, but they cannot be attributed to learned RoI selection as stated.","major_comments":[{"comment":"The selection of each RoI uses a hard argmax over a sliding-window average-pooling score map produced by f_theta_v, followed by a non-differentiable integer crop from the high-resolution image. Section 3.3 trains theta_v jointly with the LLM using the causal language modeling loss l = -sum log p_theta(y_i | ...). Because the argmax has zero gradient almost everywhere, and no relaxation (Gumbel-softmax, straight-through estimator, REINFORCE baseline, or auxiliary selection objective) is described anywhere in the paper, the gradient of the LM loss with respect to theta_v is zero almost everywhere. This directly contradicts the Section 2 claim that 'the entire process is differentiable.' Consequently, the paper's central conclusion that CoRoI learns to identify and prioritize the most informative regions is unsupported. The authors must either describe a concrete training signal that reaches the selector, or explicitly state that the selector is fixed; in the latter case, the benchmark gains can only be attributed to injecting fixed or random crops, not to learned RoI selection.","section":"Section 3.1 and Section 3.3"},{"comment":"The 'high-resolution' images used throughout the method and experiments are generated by bilinearly upsampling 336x336 images to 672x672, 1008x1008, and 1344x1344. Bilinear interpolation does not introduce new high-frequency information, so the model never actually observes genuine high-resolution detail. This is inconsistent with the stated motivation that high-resolution input is essential for small objects and fine-grained recognition, and with the Limitations paragraph, which only says that information loss 'may' occur. The experiments therefore evaluate a multi-scale cropped-view model rather than a model that processes true high-resolution images. The authors should either provide experiments with native high-resolution images or substantially revise the claims to reflect that the inputs are interpolated versions of low-resolution images.","section":"Section 4.1 and Section 5"},{"comment":"The hyperparameters n (number of RoIs), m and lambda (multi-scale factors), and the injection layer positions k are selected based on ablations on TextVQA, MMBench, and MME, and these same benchmarks appear in the final comparison tables (Tables 1 and 2). This selection-on-evaluation procedure introduces optimistic bias into the reported gains, because the model is effectively tuned to the evaluation set. The authors should either perform hyperparameter selection on a held-out validation split or disclose and justify the selection protocol; otherwise the claim of consistent gains across benchmarks is weakened.","section":"Tables 3 and 4"}],"minor_comments":[{"comment":"The abstract states that the 34B model surpasses Gemini Pro 1.0 on 'six benchmarks,' but Table 1 shows only five of six comprehensive benchmarks where CoRoI-v2-34B outperforms Gemini Pro (SEED-I, MMB, MME, MMMU, MathVista, but not MM-Vet), and Gemini Pro is not listed in Table 2. Please reconcile the count or specify the exact benchmarks.","section":"Abstract and Table 1"},{"comment":"The text refers to a 'Titanic example' in Figure 3, but the second example in Figure 3 is a radar chart about MM-Vet, not a Titanic-related image. Please correct the description or replace the figure.","section":"Section 4.3"},{"comment":"Table 3 is difficult to read: the separation into an 'upper part' and 'lower part' is not visually clear, and the delta values (e.g., +5.1, -0.7) are not explicitly defined relative to which baseline. Please restructure the table or clearly state the reference model for each delta.","section":"Table 3"},{"comment":"The loss notation p_theta(y_i | hat{y}_{1:i-1}, q) uses a hat on the history, which is confusing under teacher forcing; the history should be ground-truth tokens y_{1:i-1} unless a generation-based training objective is intended. Please clarify.","section":"Section 3.3"},{"comment":"The layer selection rule described in the appendix says the default layer set is obtained by 'doubling the layers within the section close to the middle of LLM,' but the example for 32 layers starts with {7, 15, 23} and ends with {7, 11, 15, 19, 23, 27}; this is not a simple doubling of a contiguous section, so the selection rule is unclear. Please provide a precise algorithm.","section":"Appendix"}],"recommendation":"reject","confidential_remarks":"The experimental breadth is real, and the comparisons with LLaVA-NeXT are mostly well controlled. However, the non-differentiability of the hard-argmax selector is not a minor omission: it invalidates the core learning claim as written. If the authors can demonstrate an actual training signal for the selector or reframe the contribution as a fixed-crop injection method with correspondingly weaker claims, a revised submission could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things before reading further. The headline claim—that CoRoI learns to select the most informative regions—is not supported by the text. The selection uses a hard argmax over a score map, and the paper says the whole process is differentiable, but an argmax has zero gradient almost everywhere. No relaxation, straight-through estimator, or auxiliary loss is described. So the network f_θv that should produce the regions receives no training signal from the language-model loss. The other thing is that the 'high-resolution' images are bilinear upsamplings of 336×336 images. That means the model never actually sees more detail than the low-res version, so the high-resolution claim isn't really tested.\n\nThe paper does have a genuinely new idea. The chain-of-region-of-interest mechanism, where each region is selected using language and the previously selected region, and then injected into LLM hidden layers via cross-attention, is something I haven't seen in the cited LLaVA or InternVL lines. The experimental side is also solid in many ways: 11 benchmarks, three model sizes, controlled comparisons against LLaVA-NeXT with the same backbones, and ablations over design choices and training data. The consistent gains over LLaVA-NeXT are worth taking seriously, even if the mechanism behind them is unclear.\n\nWhere the paper falls apart is the central mechanism. The differentiability claim is load-bearing; if the selector isn't trained, the method reduces to injecting arbitrary crops into the LLM. The gains might then come simply from the multi-crop cross-attention, not from language-guided selection, and the paper provides no ablation that separates those. The bilinear upsampling issue compounds this: even if the selector worked, the crops would carry no information beyond what the low-res image already has. The qualitative examples in Figure 3 don't establish learning—they could easily be cherry-picked. Missing code and single-run numbers are minor in this subfield, but they don't help. The citation gaps to prior zoom-in/region-selection methods are real but minor.\n\nI'd send this to a serious referee, but I'd expect it to come back with major revision. The authors need to describe or add a differentiable relaxation for the selection, or show that the selector is trained in some other way, and they need to test on genuinely high-resolution images. As it stands, the central claim is unsupported.","headline":"CoRoI's central claim of learned region selection is unsupported by a non-differentiable argmax, and the high-res experiments use upsampled images; the benchmark gains are real but the mechanism is not established.","tokens_in":16145,"tokens_out":4077,"would_cite":false,"duration_ms":41350,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CoRoI lets multimodal language models read high-resolution images by walking a question-guided chain of crops, injecting only those crops into the model's hidden layers.","keywords":["visual instruction tuning","high-resolution images","multimodal large language models","region of interest","chain of RoI","cross-attention injection","LLaVA-NeXT"],"falsifier":"Run CoRoI under the paper's protocol and compare it with a control that replaces the selected regions with random crops of the same size from the same multi-scale upsampled images. If the random-crop control matches CoRoI's benchmark scores, the learned chain-of-relevance mechanism is doing none of the work. Separately, measure the gradient norm of the selector network $f_{\\theta_v}$ during instruction fine-tuning: if it is identically zero, the hard argmax blocks the learning signal the paper's differentiable claim requires.","tokens_in":15123,"feed_emoji":"🔍","tokens_out":9297,"duration_ms":85580,"temperature":0.7,"pith_summary":"CoRoI is a visual-instruction-tuning method that lets multimodal large language models (MLLMs) exploit high-resolution images without feeding the entire high-resolution image into the language model. It combines a low-resolution global view with the text question, then iteratively selects a chain of regions of interest from upsampled high-resolution images, choosing the most informative crops one at a time. Those crops are injected into the LLM's hidden layers through cross-attention residual connections, so the model can zoom into small text and fine details while the token count stays close to the low-resolution baseline. The paper reports consistent gains over LLaVA-NeXT across 11 benchmarks at 7B-34B scales, with the largest variant matching or surpassing proprietary closed models on several general benchmarks.","feed_headline":"High-res images, low-res cost: CoRoI beats LLaVA-NeXT","feed_subtitle":"A chain of question-guided crops lets 7B-34B models match or beat proprietary rivals on several benchmarks.","key_machinery":"The chain-of-region-of-interest module is the load-bearing mechanism. A score network $f_{\\theta_v}$ takes the low-resolution visual features and the question-token embeddings, runs them through self-attention and cross-attention, and outputs a one-channel score map. A sliding-window average-pooling scan over that map selects the highest-scoring patch; the corresponding high-resolution crop is encoded with the frozen ViT and concatenated with the question to select the next patch, building a chain of $n$ regions. The default configuration uses $m=3$ upsampled scales ($\\lambda = 2,3,4$) and a chain of $n=4$ regions per scale. Each region's features are then injected into selected LLM layers by a cross-attention module in which the low-resolution hidden states serve as queries and the region features as keys and values, with the output added as a residual. The paper states that this entire pipeline is differentiable.","core_discovery":"This paper claims that for any given question, the useful content of a high-resolution image is concentrated in a short chain of crops, and a model can find those crops by using the low-resolution image plus the question as a guide. It argues that injecting the selected crops' visual features into the LLM's hidden layers via cross-attention lets the model reason about fine-grained detail, most visibly in OCR-heavy tasks such as TextVQA, while avoiding the quadratic token-cost growth of long high-resolution sequences. The evidence is benchmark performance: CoRoI-v1 and CoRoI-v2, built on Mistral-7B and Hermes-2-Yi-34B, improve on LLaVA-NeXT at comparable sizes, and the 34B variant outperforms Gemini Pro 1.0 on five of six comprehensive benchmarks and GPT-4V on MMB, SEED-I, and MME.","pith_inferences":["Because the chain mechanism is trained end-to-end and query-guided, a natural extension is multi-turn or agentic look-where-uncertain behavior, where the LLM decides the next region from its own current answer state.","Since high-resolution images are produced by bilinear upsampling, the method's ceiling is set by information already present at the base resolution; replacing upsampling with true high-resolution captures or learned super-resolution is a direct next step.","The design suggests a general recipe: keep the language model's token budget fixed and spend it on query-relevant crops, which could transfer to videos or gigapixel medical and satellite images with the same cross-attention injection."],"forward_implications":["If CoRoI is right, a multimodal LLM can answer questions about high-resolution images while feeding the language model only a low-resolution token sequence, because the informative crops enter through cross-attention residuals rather than as extra input tokens.","The consistent gains over LLaVA-NeXT across 7B, 13B, and 34B backbones imply that selective region zooming is a transferable recipe, not a quirk of one model size.","On text-heavy benchmarks such as TextVQA, the chain mechanism can move to signs and text blocks that carry the answer, giving the largest relative improvements.","A 34B CoRoI model can match or surpass closed-source models on several general multimodal benchmarks, suggesting that efficient input selection can close part of the gap that brute-force resolution scaling leaves open."],"supporting_citations":[{"why":"Supplies the LLaVA-NeXT baseline that CoRoI is compared against at 7B, 13B, and 34B scales.","marker":"[36]"},{"why":"Supplies the LLaVA-1.5 base model and the 336-by-336 low-resolution visual-instruction-tuning recipe that CoRoI extends.","marker":"[35]"},{"why":"Provides the frozen CLIP-pretrained ViT-L that extracts both low-resolution and high-resolution visual features.","marker":"[49]"},{"why":"Provides the projector-pretraining stage and the LLaVA-Instruct-158K data used in instruction fine-tuning.","marker":"[37]"},{"why":"Adds ShareGPT4V high-quality captions in the data ablations that improve visual alignment.","marker":"[13]"},{"why":"Contributes ALLaVA GPT-4V-synthesized caption and instruction data used when building the training mixture.","marker":"[11]"},{"why":"Serves as a proprietary comparison model on the comprehensive benchmark suite.","marker":"[56]"},{"why":"Serves as a proprietary comparison model on MMB, SEED-I, and MME.","marker":"[1]"}],"fun_headline_variants":["CoRoI: ask the image where to look","Chain of crops cuts HR token cost, beats LLaVA-NeXT","CoRoI: efficient HR vision, tops LLaVA-NeXT and more","CoRoI 34B surpasses Gemini Pro and GPT-4V on several tests","Question-guided crops: CoRoI trims tokens, boosts benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The weaker link is the region-selection step: it picks the single highest-scoring window by a hard argmax, which has zero gradient almost everywhere, so the score network may receive no learning signal from the language-model loss; if that is true, the selected regions are not actually learned and the method becomes the injection of fixed crops into the LLM.","fun_headline_variants_meta":{"raw":{"variants":["CoRoI: ask the image where to look","Chain of crops cuts HR token cost, beats LLaVA-NeXT","CoRoI: efficient HR vision, tops LLaVA-NeXT and more","CoRoI 34B surpasses Gemini Pro and GPT-4V on several tests","Question-guided crops: CoRoI trims tokens, boosts benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000345,"raw_usage":{"total_tokens":1895,"prompt_tokens":946,"completion_tokens":949,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":850}},"tokens_in":562,"tokens_out":949,"duration_ms":9014,"temperature":1.0,"reasoning_tokens":850,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:32:00.309788+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CoRoI under the paper's protocol and compare it with a control that replaces the selected regions with random crops of the same size from the same multi-scale upsampled images. If the random-crop control matches CoRoI's benchmark scores, the learned chain-of-relevance mechanism is doing none of the work. Separately, measure the gradient norm of the selector network $f_{\\theta_v}$ during instruction fine-tuning: if it is identically zero, the hard argmax blocks the learning signal the paper's differentiable claim requires.","supporting_citations":[{"cited_title":"Llava-next: Improved reasoning, ocr, and world knowledge, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the LLaVA-NeXT baseline that CoRoI is compared against at 7B, 13B, and 34B scales."},{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Provides the frozen CLIP-pretrained ViT-L that extracts both low-resolution and high-resolution visual features."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Provides the projector-pretraining stage and the LLaVA-Instruct-158K data used in instruction fine-tuning."},{"cited_title":"https://cdn.openai.com/papers/GPTV_System_Card.pdf, 2023","cited_arxiv_id":null,"evidence_quote":"Serves as a proprietary comparison model on MMB, SEED-I, and MME."}],"review_version":1}