{"id":"dcc9a98f-770d-42e5-b87f-025738c126dc","arxiv_id":"2506.12776","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new resolution-focused benchmark and an open-source native-resolution training framework show that preserving original image resolution improves VLM performance on fine-grained visual tasks.","lead":"Vision-language models often lose fine detail when images are resized to a fixed low resolution. This paper introduces RC-Bench, a benchmark for testing models under extreme resolutions and aspect ratios, and NativeRes-LLaVA, an open-source framework that keeps images at native resolution, showing gains on detail-heavy tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RC-Bench's answer-type mix and normalization rules are unspecified; the benchmark is unreleased, so the central RC-Bench comparison cannot be independently verified or audited for resolution sensitivity.","rationale":"The reader's verdict is CONDITIONAL with moderate confidence, and I agree with that verdict. The paper has genuine strengths: the controlled ViT swap (Tables 5 and 8) comparing Qwen2-VL-ViT under Crop versus Native strategies is the right experimental design, the external benchmarks (OCRBench, DocVQA, ChartQA, InfographicVQA) provide independent support for the claim that native-resolution encoding helps on detail-heavy tasks, and the code release for the framework is a real contribution. However, the central benchmark RC-Bench is the measuring instrument for the paper's most distinctive claims: the RC/SC classification, the ACV/RCV robustness conclusions, and the Table 8 head-to-head that isolates the resolution strategy. That instrument is not released, and the construction details provided (Section 3.3, Appendix A.1, A.4, A.5) are insufficient to rule out confounds. The most load-bearing gap is not merely that the benchmark is unreleased, but that the scoring protocol itself is under-specified in ways that could interact with resolution: answer types are assigned by an undisclosed hybrid of keyword rules and LLMs, the distribution of answer types across the 35 cells is unreported, and the normalization rules are elaborate enough that two independent implementations could produce different scores. The absence of error bars further weakens the quantitative precision. These are addressable issues, not fundamental flaws: the paper's controlled ablation and external benchmark results suggest the direction of the finding is likely correct. Therefore the appropriate verdict is CONDITIONAL: accept the framework contribution and the qualitative direction, but require release of RC-Bench (or a representative subset) plus evaluation code and either variance estimates or multiple seeds before the central RC-Bench quantitative claims are treated as settled. This aligns with the reader's verdict; a change to ACCEPT or REJECT is not warranted because the direction of the claim has independent support and the concerns are about verifiability, not demonstrated error.","tokens_in":21924,"tokens_out":2485,"duration_ms":22234,"concrete_test":"Release RC-Bench (or a statistically representative subset, e.g., 350 stratified images with full QA pairs and meta-data) along with the exact evaluation and normalization code. Then independently recompute Table 8 with the released subset: if the Native-vs-Crop gap at 728x728 (53.6 vs 47.9) shrinks below the per-cell noise floor or flips sign when answer-type mix is controlled, the central RC-Bench claim fails. Also report the answer-type distribution per area-ratio cell and recompute ACV/RCV on answer-type-balanced subsets to check whether robustness metrics reflect resolution sensitivity rather than answer-type confounding.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim is that Native Resolution Encoding significantly improves VLM performance on RC-Bench, with the strongest controlled comparison being Table 8: LLaVA-NeXT-QwenViT (Crop, 728x728) at 47.9 vs NativeRes-LLaVA (Native, 728x728) at 53.6 on RC-Bench. This comparison is the crux of the paper's central claim, and its validity hinges entirely on RC-Bench as a measuring instrument. However, RC-Bench is not released, and its construction (Section 3.3) involves undisclosed 'proprietary data resources'; QA pairs are generated by GPT-4o and human-verified, but no inter-annotator agreement, per-type accuracy, or adversarial filtering statistics are reported. The evaluation protocol (Appendix A.4) uses heuristic keyword-based answer-type classification plus LLM-assisted categorization, then EM for short answers and ANLS for long answers. This introduces two load-bearing confounds. First, the answer-type split (dates, numbers, identifiers, phrases, addresses, sentences) determines which metric is applied, and the paper does not report the distribution of answer types across the 35 area-ratio cells. If answer type is correlated with resolution or aspect ratio, then the reported cell-level accuracy and the ACV/RCV robustness metrics partially measure the mix of answer types, not resolution sensitivity. Second, the normalization rules in Appendix A.4 (e.g., accepting '193', '193 $', and '$ 193' interchangeably) are human-defined and could systematically advantage models that produce certain output formats; without releasing the exact normalization code and ground-truth reference sets, the comparison cannot be replicated. There is also no error bar or multiple-seed variance anywhere in Tables 6-8, so the 5.7-point gap at 728x728 could in principle be within run-to-run noise, although a gap of that size is likely robust if the benchmark is reliable. The reader identified the same core concern: the validity of RC-Bench as a measuring instrument.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses what it calls the 'Resolution Dilemma' in vision-language models (VLMs): most models rely on fixed low-resolution visual encoders, and existing benchmarks do not systematically vary image resolution and aspect ratio. The authors introduce two artifacts: RC-Bench, a new benchmark with a balanced distribution over seven area levels and five aspect-ratio categories, and NativeRes-LLaVA, an open-source training framework that encodes images at native resolution using a 2D-RoPE ViT with sequence packing. Experiments compare native-resolution encoding against cropping, hybrid, and fixed-resolution strategies, reporting that native encoding substantially improves performance on 'Resolution-Centric' benchmarks (e.g., TextVQA, DocVQA, OCRBench, RC-Bench) while roughly preserving performance on 'Semantic-Centric' benchmarks. A controlled ablation in Table 5 swaps the same Qwen2-VL ViT into both the cropping and native pipelines, and the native strategy still wins on most resolution-centric tasks. The paper includes code and detailed training hyperparameters, and it candidly discusses limitations in Section 6.","tokens_in":22172,"tokens_out":4414,"duration_ms":50106,"significance":"If the claims hold, the paper makes two useful contributions. First, NativeRes-LLaVA is a genuinely open and modular framework for native-resolution visual encoding, which fills a gap in the open-source ecosystem relative to closed models like Qwen2-VL and Kimi-VL; the code release and detailed training recipes are commendable. Second, the controlled comparison in Table 5—using the same Qwen2-VL ViT for both the cropping and native strategies—is a strong experimental design that isolates the visual-encoding strategy from the backbone choice, and it provides credible evidence that native-resolution encoding yields real gains. The paper also honestly discusses training-data mismatches and computational bottlenecks. However, the central empirical evidence on RC-Bench is weakened by the benchmark being unreleased and by the undisclosed details of its construction and evaluation pipeline; moreover, the Semantic-Centric versus Resolution-Centric taxonomy is derived from an undescribed manual sensitivity analysis. These issues currently prevent the benchmark-based claims from being fully audited.","major_comments":[{"comment":"RC-Bench's validity as a measuring instrument is not adequately established. The benchmark is not released, its construction uses undisclosed 'proprietary data resources,' and the paper does not report the distribution of answer types (numbers, dates, phrases, addresses, sentences, etc.) across the 35 area/ratio cells. Because short answers are scored with Exact Match and long answers with ANLS, and because Appendix A.4 applies additional human-defined normalization rules, any correlation between answer type and area or ratio would directly contaminate the per-cell accuracies and the ACV/RCV statistics in Tables 6-8. The authors should release the benchmark or, at minimum, provide the full per-cell answer-type breakdown, per-cell image counts, inter-annotator agreement statistics for the human verification step, and a description of how the answer-type classification was validated.","section":"Section 3.3 and Appendix A.4"},{"comment":"The manual sensitivity analysis that classifies benchmarks into Semantic-Centric and Resolution-Centric types is not described. The paper states that 'we systematically adjusted the components of VLM and resolution processing strategies' to analyze sensitivity, but it does not report the exact procedure: which model variants, which resolution ranges, which thresholds defined 'sensitivity,' and how manual case analysis entered the classification. This taxonomy is load-bearing because it selects the evaluation benchmarks and supports the claim that native resolution helps only Resolution-Centric tasks (Tables 3-5 and Figure 3). Without a reproducible procedure, the reader cannot verify that the benchmark categorization is objective rather than shaped by the authors' expectations.","section":"Section 3.1"},{"comment":"The controlled comparison between Cropping and Native on the same Qwen2-VL ViT is not fully specified. The text says the ViT in LLaVA-NeXT was replaced with Qwen2-VL's ViT and its input resolution fixed to match SigLip-384, but it does not explain how the cropping/tiling mechanism was adapted to the new ViT: specifically, how the global thumbnail and local tiles are encoded, whether the 4x patch-merging compression module is applied to both pipelines, and how positional embeddings are handled for non-native fixed resolutions. Without this detail, the observed gains of NativeRes-LLaVA over LLaVA-NeXT-QwenViT could partly reflect an improperly tuned cropping pipeline rather than the native-resolution strategy itself. Please provide precise architecture specifications and token-budget equivalence for both compared pipelines.","section":"Section 5.4.1 and Table 5"},{"comment":"The headline comparison of 47.9 (Crop) versus 53.6 (Native) on RC-Bench rests entirely on the benchmark's soundness, which is currently not verifiable (see the first major comment). Even setting that aside, the paper does not report per-cell sample sizes or confidence intervals. With 1,750 images spread over 35 cells, the average cell has only 50 images, and some cells could be much smaller; the reported accuracy differences (e.g., -0.29 at (NM,B) and +0.24 at (AH,A)) may not be statistically reliable. The authors should provide per-cell counts and bootstrapped confidence intervals for the accuracy differences in Figure 6(c) and for the ACV/RCV values in Tables 6-8.","section":"Section 5.4.2 and Table 8"}],"minor_comments":[{"comment":"The evaluation protocol list has a duplicated numbering: item (5) appears twice, once for 'SC-type tasks' and once for 'RC-Bench.' This should be renumbered.","section":"Section 5.2"},{"comment":"The ACV and RCV values are described as 'multiplied by 10^2' only in the table captions; the main text in Section 5.4.2 should also state this scaling to avoid misinterpretation of the metric magnitudes.","section":"Tables 6-8"},{"comment":"The paper says RC-Bench images are 'uniformly distributed across seven predefined resolution levels and five aspect ratio categories,' but Figure 9(a) shows unequal color intensities across cells. Please clarify whether the target distribution is exactly uniform or approximately balanced, and report the actual counts per cell.","section":"Section 3.3 and Appendix A.1"},{"comment":"The NaViT reference is malformed: the author list reads 'Basil Mustafa Mostafa Dehghani, Jonathan Heek Josip Djolonga, et al.' and the title is incorrectly rendered. Please fix the citation to include the full author list and correct title.","section":"Reference [38]"},{"comment":"The heatmap color legend in Figure 6 is incomplete: the text mentions green, red, and yellow cells, but the figure does not show a colorbar or threshold values. A clear legend would help readers interpret the magnitude of differences.","section":"Section 5.4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong open-source contribution and a well-controlled ablation on external benchmarks, but the RC-Bench instrument itself is not release and its construction/evaluation pipeline contains undisclosed components. This is a fairness and reproducibility concern for a benchmark paper. I would encourage the editor to require the benchmark and full evaluation details as a condition of acceptance. The use of 'proprietary data resources' without any description of their origin or license may also raise ethical/legal issues that the authors should be asked to clarify."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper gives the most controlled comparison yet of native-resolution vs cropping visual encoding for VLMs, and native wins on most resolution-centric benchmarks. Table 8 swaps in the same Qwen2-VL ViT for both strategies at the same 728x728 token cap, so the 5.7-point gap on RC-Bench is not just a better-backbone effect. That is real evidence and it's the main contribution.\n\nWhat's new is less the architecture—the pieces are Qwen2-VL's ViT, NaViT-style packing, and LLaVA-style training—and more the packaging: a modular open-source framework and a new benchmark, RC-Bench, that deliberately balances image area and aspect ratio. The framework is genuinely open. The benchmark fills a real gap, though calling it the first is a bit strong given HR-Bench. The ablations on MaxRes and LLM scale are useful, and the limitations section is candid about not reaching SOTA and about training-data resolution limits.\n\nSoft spots are mostly around the measuring stick. RC-Bench is not released; its construction uses undisclosed proprietary data and GPT-4o-generated QAs with no reported quality stats. The evaluation protocol uses a heuristic answer-type classifier and human-defined normalization rules. Without the benchmark and normalization code, you cannot audit whether cell-level scores reflect resolution sensitivity or an artifact of answer-type mix across cells. The SC/RC taxonomy comes from an unspecified sensitivity experiment that needs to be described. No error bars or seeds are reported anywhere, so the 5.7-point gap sits on a single run, though a gap of that size is likely robust if the benchmark is sound.\n\nOverall the central claim is plausible and independently supported by gains on external benchmarks like OCRBench and DocVQA. The benchmark issues are significant but fixable. The paper is for VLM developers choosing an encoding strategy and for benchmark builders. I'd send it to a serious referee: it deserves careful review, and the framework release is useful to the community. My own verdict would be conditional, with a clear request to release RC-Bench, report answer-type distribution and normalization code, and add variance estimates.","headline":"Controlled evidence that native-resolution encoding beats cropping for fine-grained VLM tasks, but the central benchmark is unreleased and under-specified, so the claim is solid-but-conditional.","tokens_in":22906,"tokens_out":3244,"would_cite":true,"duration_ms":34223,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that letting a vision-language model encode images at their native resolution and aspect ratio, rather than resizing or cropping, produces large gains on fine-grained visual tasks and can be cleanly tested with a new…","keywords":["vision-language models","native resolution visual encoding","resolution-centric benchmark","aspect ratio robustness","dynamic resolution","visual instruction tuning","OCR and document understanding","sequence packing"],"falsifier":"Release RC-Bench and re-score every model after downsampling each image to a fixed low resolution while keeping the same questions; if accuracy does not drop sharply for models with high native-resolution scores, the benchmark is not measuring resolution sensitivity and the central claim would be undercut.","tokens_in":21642,"feed_emoji":"🖼️","tokens_out":8818,"duration_ms":93174,"temperature":0.7,"pith_summary":"Vision-language models usually resize every image to a fixed low resolution before encoding, which destroys detail in small text, dense charts, and extreme panoramas. The paper claims that this \"resolution dilemma\" has two parts: most training frameworks cannot natively encode arbitrary resolutions, and most benchmarks do not measure resolution sensitivity. It introduces RC-Bench, a benchmark of 1,750 images deliberately balanced across seven area levels and five aspect-ratio categories, with human-verified question-answer pairs, and NativeRes-LLaVA, an open training framework whose vision encoder works directly at an image's original resolution and shape. Across resolution-centric benchmarks the native strategy substantially outperforms fixed and cropping-based strategies, while leaving semantic-centric benchmarks largely unchanged.","feed_headline":"Native resolution beats cropping for vision-language models","feed_subtitle":"A 7B model trained on 1.34M samples tops OCR and document benchmarks by keeping original image size and shape.","key_machinery":"The load-bearing mechanism is a native-resolution vision encoder: a ViT with 2D rotary position embeddings that accepts arbitrary image sizes and produces a variable number of visual tokens via 2x2 average pooling, a 4x patch merger, and an MLP that projects features into the language model. Because native resolution yields variable-length patch sequences, training uses Patch n' Pack style packing so multiple images' patches are concatenated into one sequence and attention is isolated per image with variable-length flash attention. On the evaluation side, RC-Bench supplies a 7x5 grid of area levels and aspect-ratio categories, and its coefficient-of-variation metrics (ACV and RCV) turn \"robustness to resolution variation\" into a directly measurable quantity.","core_discovery":"The central claim is that native-resolution visual encoding, meaning processing each image at its original pixel dimensions and aspect ratio with a variable-token vision transformer instead of resizing or tiling it, is the decisive factor in fine-grained visual understanding. On a matched 1.34M-sample training setup, the proposed model reaches 74.0 on TextVQA, 705 on OCRBench, 89.7 on DocVQA, 79.0 on ChartQA, 61.0 on InfographicVQA, 61.3 on HR-Bench, and 60.1 on RC-Bench, beating cropping-based and hybrid-encoder models that train on much larger data. The advantage is largest on extreme aspect ratios and extreme areas; in the per-cell comparison at matched 728x728 resolution, the native strategy wins or ties in nearly every cell and loses clearly in only one \"sweet spot\" configuration that matches the pretraining distribution. The paper frames this as resolving a dilemma: native encoding is needed for resolution-centric tasks, while semantic-centric tasks are resolution-insensitive, so benchmarks and models must separate the two.","pith_inferences":["If RC-Bench's proprietary images were replaced by an independently built public version with the same balancing protocol, the reported native-versus-crop gaps could be checked for benchmark-specific bias; this is the most direct way to test whether the conclusion generalizes beyond the paper's selected 1,750 images.","The single cell where cropping wins suggests a concrete hybrid design: route images whose size and shape match the pretraining distribution through a fixed-resolution path, and send every other image through the native path; this could be tested while holding all other training conditions fixed.","Because the vision encoder is initialized from a strong pretrained native-resolution model rather than trained from scratch, part of the gain may reflect the pretrained encoder's quality; a from-scratch native ViT trained at scale is the natural test of the framework's own contribution.","A resolution-aware benchmark like RC-Bench could also serve as a diagnostic for deployed models: plotting accuracy as a function of downsampling factor would reveal at what resolution each model's fine-grained understanding breaks."],"forward_implications":["For fine-grained OCR, document, chart, and infographic tasks, preserving original resolution and aspect ratio should be expected to beat cropping-based pipelines even when the latter see more training data.","Benchmark reporting should include accuracy breakdowns by area and aspect ratio, plus coefficients of variation, because average accuracy hides which models fail on extreme inputs.","Raising the maximum supported input resolution in a native-resolution model improves resolution-centric scores while leaving semantic-centric scores roughly flat, so high MaxRes is a targeted design choice rather than a universal upgrade.","Smaller language models using native resolution can outperform larger ones using cropping: a 2B-scale native model already beats a 7B cropping-based baseline on several resolution-sensitive benchmarks.","The open modular training stack makes it possible to swap vision encoders, pack multimodal sequences, and run elastic native-resolution training, which the paper argues the open-source community previously lacked."],"supporting_citations":[{"why":"Supplies the native-resolution encoding design and the pretrained vision-encoder weights the framework initializes from.","marker":"[48]"},{"why":"Supplies the Patch n' Pack sequence-packing mechanism used to train on variable-length image patch sequences.","marker":"[38]"},{"why":"Provides the visual-instruction-tuning architecture, the 558K pretrain corpus, and the 665K instruction mix used in training.","marker":"[28]"},{"why":"Provides the cropping-based baseline, the 779K fine-tuning dataset, and the comparison setup behind the main results.","marker":"[27]"},{"why":"Defines OCRBench, a resolution-centric benchmark used to measure the claimed gains.","marker":"[30]"},{"why":"Defines ChartQA, another resolution-centric benchmark in the main evaluation set.","marker":"[35]"},{"why":"Defines DocVQA, used to show document-understanding gains from native resolution.","marker":"[37]"},{"why":"Defines TextVQA, the OCR-centric benchmark where the native model posts its largest visible lead under limited data.","marker":"[42]"},{"why":"Provides the hybrid-encoder baseline against which the native-resolution strategy is compared in ablations.","marker":"[46]"}],"fun_headline_variants":["Native resolution wins over cropping in vision-language models","New benchmark exposes resolution gap in VLMs","NativeRes-LLaVA: native-resolution VLM tops OCR benchmarks","Resolution dilemma resolved: keep image size, beat baselines","Why native resolution beats tiling for vision-language AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that RC-Bench's 1,750 images and their human-verified question-answer pairs truly isolate resolution and aspect ratio as the factors that matter; if those images or questions are biased, noisy, or not actually resolution-sensitive, the central comparison loses its evidentiary force.","fun_headline_variants_meta":{"raw":{"variants":["Native resolution wins over cropping in vision-language models","New benchmark exposes resolution gap in VLMs","NativeRes-LLaVA: native-resolution VLM tops OCR benchmarks","Resolution dilemma resolved: keep image size, beat baselines","Why native resolution beats tiling for vision-language AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1288,"prompt_tokens":987,"completion_tokens":301,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":223}},"tokens_in":603,"tokens_out":301,"duration_ms":3933,"temperature":1.0,"reasoning_tokens":223,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:43:01.246765+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Release RC-Bench and re-score every model after downsampling each image to a fixed low resolution while keeping the same questions; if accuracy does not drop sharply for models with high native-resolution scores, the benchmark is not measuring resolution sensitivity and the central claim would be undercut.","supporting_citations":[{"cited_title":"Patch n’ pack: Navit, a vision transformer for any aspect ratio and resolution, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the Patch n' Pack sequence-packing mechanism used to train on variable-length image patch sequences."},{"cited_title":"LLaV A-NeXT: Improved reasoning, ocr, and world knowledge.https://llava-vl.github.io/blog/ 2024-01-30-llava-next/","cited_arxiv_id":null,"evidence_quote":"Provides the cropping-based baseline, the 779K fine-tuning dataset, and the comparison setup behind the main results."},{"cited_title":"Docvqa: A dataset for vqa on document images","cited_arxiv_id":null,"evidence_quote":"Defines DocVQA, used to show document-understanding gains from native resolution."}],"review_version":1}