{"id":"b79522b5-a081-405f-af33-b4f8f5eee19b","arxiv_id":"2507.13348","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A vision-language model learns via reinforcement learning when to upscale a low-resolution image, cutting visual tokens roughly in half while preserving accuracy on most benchmarks.","lead":"VisionThink trains a vision-language model to first read a low-resolution image and, when needed, ask for the full-resolution version, saving compute on easy questions. It uses an LLM judge to give rewards during reinforcement learning, and the resulting model cuts visual tokens by about half while staying close to full-resolution accuracy on most benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim hinges on an LLM-as-Judge that also scores the custom ChartQA† evaluation; a judge bias would inflate both the RL training signal and the reported 103.7% relative performance, and the only reliability evidence is a 1,000-case self-report.","rationale":"The central claim is that VisionThink learns a per-sample resolution decision, preserving accuracy while cutting visual tokens. That claim rests on two coupled measurements: the model's performance and its upscale-call ratios. The upscale-call ratios are trained to maximize judge-rated correctness, and the Table 2 ChartQA numbers are evaluated with the same style of judge (ChartQA†). If the judge is systematically biased, both measurements move together, so the conclusion that the model is 'smart' could be an artifact of the reward/evaluation loop. The 1,000-case user study is not sufficient evidence: with zero errors it only bounds error rate to roughly 0–0.3% under strong sampling assumptions, it is self-reported, and it does not test whether the judge treats vanilla and RL-trained outputs equally. Replacing the admittedly flawed official ChartQA metric is justified in principle, but a replacement oracle must be validated on the exact evaluation set with blind human labels. The concern is not about author integrity; it is the standard risk of using a learned judge for both training and evaluation. The paper has real strengths: a clean GRPO extension to multi-turn resolution decisions, a careful penalty design with ablations (Sec. 3.4, C.3, C.4), a useful SFT-vs-RL comparison (C.1), and plausible per-benchmark resize ratios (Fig. 5). These make the method credible but do not remove the need for an independent audit of the judge. I therefore agree with the reader's weakest-assumption analysis and keep the verdict conditional: the paper should be accepted only if the judge audit and released evaluation logs confirm the reported numbers under a metric that is not trained to be satisfied by the model.","tokens_in":967,"tokens_out":1172,"duration_ms":112477,"concrete_test":"Using the promised code and checkpoints, rerun Vanilla and VisionThink on ChartQA test with the original lmms-eval metric, and have two independent annotators, blind to model identity and to whether the image was upscaled, label 500 sampled predictions from each model. Then compute (a) judge-human agreement separately by model and (b) whether VisionThink's ChartQA score relative to Vanilla changes under the official metric. If judge agreement differs between models, or if the relative gain vanishes under the official metric, the shared judge is the source of the reported advantage; if not, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The load-bearing step is the assumption that the LLM-as-Judge yields correct labels for general VQA, because the same judge is simultaneously (i) the GRPO accuracy reward that teaches the model when to upscale (§3.2, Eq. 4–5) and (ii) the oracle for the custom ChartQA† metric in Table 2. A systematic judge bias—for example, leniency toward the RL policy's verbose, well-formatted outputs—would inflate both the training signal and the headline 103.7% average relative performance, making the apparent 'smartness' partly an artifact of the reward/evaluation loop. The paper's only direct reliability evidence is a 1,000-case user study with no misclassifications (Appendix B.1.1), which lacks confidence intervals, sampling details, and independent audit. Appendix B.4 also shows that the official lmms-eval ChartQA metric is flawed for percent-style answers and replaces it with GPT-4o-Judge plus human verification; if the Vanilla baseline in Table 2 was not re-scored under that same protocol, the comparison is apples-to-oranges. The per-benchmark resize ratios in Fig. 5 are plausible and well-motivated, but they were optimized against the same judge, so the smartness claim inherits this risk.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"VisionThink proposes a sample-adaptive visual token compression paradigm for vision-language models. Instead of applying a fixed pruning ratio, the model first processes a low-resolution image and may emit a tool-call token to request the original resolution; training uses multi-turn GRPO with an LLM-as-Judge accuracy reward, a format reward, and a penalty mechanism (Eq. 5) that penalizes direct answers or resizing depending on the model's own per-resolution correct-answer counts. On nine benchmarks, the paper reports an average retained visual-token ratio of about 51.3% and an average relative performance of 103.7% of Qwen2.5-VL-7B-Instruct, with per-benchmark resize ratios ranging from about 7% (DocVQA) to 79% (ChartQA). The authors also show that the LLM-as-Judge reward improves general VQA performance when trained on 130K samples (VisionThink♠/‡).","tokens_in":28386,"tokens_out":10596,"duration_ms":110759,"significance":"If the reported results are robust, the work makes a useful contribution to efficient VLM inference by shifting from fixed-ratio token pruning to per-sample decisions about input resolution. The multi-turn GRPO formulation with an LLM-as-Judge reward is a transferable recipe for applying RL to open-ended VQA, and the ablations on penalty threshold (Fig. 8), prompt choice (Fig. 7), SFT vs RL (Table 8), and cold-start (Table 9) are helpful. The paper also provides code and model links. The significance is conditional on resolving two load-bearing issues: the reliability and consistency of the judge-based ChartQA evaluation, and the precise accounting of the reported 51.3% token retention. If those are addressed, the adaptive-resolution idea is a credible alternative to existing efficient-VLM methods.","major_comments":[{"comment":"The manuscript replaces the official lmms-eval ChartQA metric with GPT-4o-Judge plus human verification (ChartQA†), and the same judge provides the accuracy reward in Eq. (4). Because the judge labels both the RL training signal and the ChartQA† column of Table 2, a systematic judge bias would inflate the reported 103.7% average relative performance and the 'smartness' claim simultaneously. The only evidence of judge reliability is the 1,000-case user study in Appendix B.1.1, which reports no misclassifications but provides no sampling protocol, confidence intervals, or independent audit. Please report judge-human agreement (e.g., Cohen's kappa) on a stratified sample, including open-ended vs numeric and direct vs resize answers, and state explicitly whether all rows of Table 2, including Vanilla and the efficient-VLM baselines, were scored under the same ChartQA† protocol; if not, the comparison is not well-defined.","section":"Appendix B.4, Table 2"},{"comment":"The headline 'Retain Approximately 51.3% Visual Tokens' is not reproducible because the paper does not define how this number is computed. It could be an unweighted average of per-benchmark retained ratios, a sample-weighted average, or a global token count ratio; for samples that request resizing, the ratio must combine the initial low-resolution tokens and the additional high-resolution tokens. Please provide the exact formula and the per-benchmark token counts (or FLOPs), and clarify the relation to the per-benchmark resize ratios in Fig. 5.","section":"Section 4.3, Table 2"},{"comment":"VisionThink underperforms the vanilla model on five of the nine benchmarks in Table 2 (OCRBench, DocVQA, RealWorldQA, POPE, MathVista), with absolute drops of roughly 0.5-1.5 points, while the overall 103.7% average is driven by large gains on MME and MMVet. The abstract and Sec. 4.2 claim 'superiority' or 'comparable or even superior' without statistical support. Please report standard errors or confidence intervals for the point estimates, or at least temper the claims to a per-benchmark accuracy-efficiency tradeoff analysis.","section":"Section 4.2, Tables 1 and 2"},{"comment":"Equation (5) defines the penalty using r = C_direct/(C_direct+C_high), but the text never specifies how C_direct and C_high are obtained during training (e.g., over the 16 GRPO rollouts, over a sliding window of training steps, or over the whole dataset) or how the case C_direct=C_high=0 is handled. Since these counts are produced by the judge and by the policy being trained, the penalty is non-stationary and can steer the model toward the judge's notion of when to upscale; the data-selection rule in Sec. 3.5 has the same structure. Please make the computation precise and include an analysis of the sensitivity of the final resize ratios to the rollout count and to the threshold θ beyond the ablation in Fig. 8.","section":"Section 3.4, Eq. (5)"},{"comment":"The comparison to training-free efficient-VLM baselines (FastV, SparseVLM, VisionZip) is confounded: VisionThink is trained with RL on 20K (and in variants, 130K) samples, while the baselines are used off-the-shelf. The reported average relative improvement may therefore reflect the RL procedure rather than the adaptive-resolution decision. Please add a control that isolates the contribution of the RL training under matched token budgets, for example by evaluating the full-resolution RL model (VisionThink‡) at 100% token retention and at the same average retention, or by fine-tuning a token-pruning baseline on the same data.","section":"Section 4.3, Table 2"}],"minor_comments":[{"comment":"The sentence 'achieving up to 100% for DocVQA' is ambiguous; Fig. 4 shows a large speedup on DocVQA, so the text should say 'up to 100% speedup' or a similar concrete claim.","section":"Section 1"},{"comment":"The caption says Down-Sample uses an image 'resolution reduced by half,' while Sec. 1 says 'reducing the image resolution by a factor of four' and Fig. 1 says '1/4 Resolution Image.' Please define whether 1/4 refers to linear dimensions (token count roughly 1/16) or to pixel area (token count roughly 1/4), and use consistent terminology throughout.","section":"Table 2 caption"},{"comment":"The MME entries in Table 9 are given as '693/1707' and '444/1354,' which appear to be perception/cognition subtotals, but Tables 1 and 2 report a single MME total (e.g., 2400). Please clarify which MME score is used for the comparisons.","section":"Appendix C.2, Table 9"},{"comment":"The reference list contains several duplicated entries (e.g., [16]/[17], [19]/[20], [78]/[79], [83]/[84], [89]/[90]); please consolidate them.","section":"References"},{"comment":"There are minor typos: 'computataion' in Sec. 3.1, 'Mutli-Turn' in Sec. 3.3, and 'Instrcut' in the Table 2 caption.","section":"Various"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a computer vision or multimodal ML venue. The central risk is the reliance on the LLM-judge for both training and ChartQA evaluation; if the authors provide judge-human agreement and re-score baselines consistently, the contribution is solid. The duplicated references and ambiguous table formatting suggest the manuscript needs a careful revision pass. I would not recommend rejection because the adaptive-resolution mechanism is plausible and the ablations are supportive, but the evaluation and reporting gaps need to be closed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real contribution, but the paper oversells it. The idea—train a VLM with RL to decide per sample whether to request a higher-resolution image, using an LLM judge as the reward—is new and useful. The empirical evidence shows the model actually learns sensible behavior: roughly 79% resize calls on ChartQA, about 7% on DocVQA. That is the kind of result that makes a reader take notice.\n\nThe design is thoughtful. Using the Qwen agent prompt to enable zero-shot special-token calls, avoiding a cold-start, and the penalty-threshold ablation are all signs of care. The data selection via base-model rollouts to label resolution-requiring samples is reasonable. The code release promise counts.\n\nNow the soft spots. First, the headline \"103.7% average relative performance\" hides the fact that VisionThink is below the vanilla baseline on five of the nine benchmarks (OCRBench, DocVQA, RealWorldQA, POPE, MathVista). The gains are concentrated on MMVet and MME. That is not fatal—efficiency is the point—but the abstract and intro should not claim unqualified superiority.\n\nSecond, the ChartQA† custom evaluation is a real concern. The authors argue the official metric is broken and replace it with GPT-4o-Judge plus human verification. If the Vanilla baseline was re-scored under the same protocol, the comparison is fair; the paper says \"all ChartQA evaluations\" were done that way, but it never explicitly says the baselines were re-scored. Make that explicit. Also, because the same LLM judge supplies the RL reward and the evaluation metric, a systematic judge bias could inflate both. The 1,000-case user study with zero misclassifications is encouraging but thin; I would like confidence intervals and details on sampling.\n\nThird, no error bars and no multiple seeds. The differences on several benchmarks are small enough that noise could change the ranking. Fourth, the 51.3% token-retention figure is presented without a computation formula; it should be defined precisely.\n\nThe math and the method hold up. The judge-bias concern is real but not demonstrated to be fatal; it is a standard RLHF-style issue, and the human verification bolsters the setup. This is a solid paper, not a breakthrough. It needs a revision for honest reporting and evaluation transparency, but the core idea is worth peer review.\n\nI would bring it to a reading group and would consider citing it for the dynamic-upscaling idea. Send it to reviewers.","headline":"Solid, useful dynamic-resolution RL method whose per-benchmark results are more mixed than the abstract claims; deserves review after honest reporting.","tokens_in":28908,"tokens_out":4842,"would_cite":true,"duration_ms":52325,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A vision-language model can be trained to answer most questions from a quarter-resolution image and request full resolution only for detail-hungry samples.","keywords":["vision-language model","visual token compression","adaptive resolution","reinforcement learning","GRPO","LLM-as-Judge","efficient inference","OCR"],"falsifier":"Take a random sample of 1,000 QA pairs from the reward-training distribution, including OCR-heavy cases, and compare the LLM judge's binary scores with human scores; if disagreements are substantially more frequent than the paper's reported zero-misclassification user study, or if they cluster in the fine-grained OCR cases that drive VisionThink's upscale decisions, the RL signal and the ChartQA† numbers would rest on a biased reward.","tokens_in":27907,"feed_emoji":"🔍","tokens_out":6667,"duration_ms":72759,"temperature":0.7,"pith_summary":"VisionThink proposes that a vision-language model can decide for itself, per question, whether a downsampled image is enough to answer or whether it must request the original resolution. Starting from a 1/4-resolution image, the model answers directly in most cases and emits a special tool call only when fine detail is needed. The authors train this behavior with multi-turn GRPO, using an LLM-as-Judge to give correct/incorrect rewards because general VQA answers cannot be verified by fixed rules. Across nine benchmarks the trained model retains about 51.3% of visual tokens while averaging 103.7% of the base model's accuracy; on OCR-heavy benchmarks it upscales far more often (roughly 79% on ChartQA, 62% on OCRBench) than on general ones (31% on MME, 7% on DocVQA). If correct, this means token-compression efficiency does not have to come from a fixed pruning ratio applied to every image.","feed_headline":"A vision model that asks for a sharper image only when needed","feed_subtitle":"VisionThink keeps 51.3% of visual tokens while averaging 103.7% of the base model's accuracy across nine benchmarks.","key_machinery":"The load-bearing mechanism is a learned two-round image-access policy. The model sees a downsampled image and either answers immediately or emits a special upscale token, framed as a tool call, that doubles the image resolution; a multi-turn extension of GRPO optimizes the whole trajectory. A binary LLM-as-Judge reward decides correctness, and a thresholded penalty term $P_{\\text{control}}=0.1[\\mathbb{1}_{\\text{direct}}I(r<\\theta)+\\mathbb{1}_{\\text{high}}I(r\\geq\\theta)]$ with $r=C_{\\text{direct}}/(C_{\\text{direct}}+C_{\\text{high}})$ prevents the model from collapsing into always upscaling or always guessing at low resolution.","core_discovery":"The central claim is that a VLM can learn to route each sample to either a cheap direct answer from a 1/4-resolution image or an expensive high-resolution pass, and that this routing can be learned entirely by reinforcement learning with no cold-start stage. The paper's evidence is that VisionThink, built on Qwen2.5-VL-7B-Instruct, reaches 103.7% of the base model's average score over nine general-VQA benchmarks while keeping 51.3% of visual tokens, and that its upscale-request rate tracks the OCR content of the benchmark (high on ChartQA and OCRBench, low on MME and DocVQA). The same LLM-as-Judge training also improves quality when run at full resolution, so the reward design is claimed to work for general VQA rather than only math.","pith_inferences":["The two-round policy is an instance of a more general idea: any VLM can be trained to treat its own uncertainty about image detail as a gating signal; a testable extension would replace resizing with cropping or patch selection and see whether the same RL training learns which regions to request.","If the LLM judge systematically favors answers that only a high-resolution image can support, the same bias would inflate both the RL reward and the reported ChartQA† score; measuring judge accuracy separately on OCR-heavy and general questions would separate the model's routing skill from the reward's preference.","The observed call-ratio pattern (79% ChartQA, 7% DocVQA) suggests a cheap classifier could predict upscale need from image statistics and question tokens, letting the efficiency transfer to smaller models that were not RL-trained."],"forward_implications":["Deploying a VLM with VisionThink's policy spends about half the visual tokens on average, with the exact saving determined per sample rather than by a fixed ratio.","OCR-heavy workloads still pay for full resolution, so accuracy on charts, documents, and text-heavy images is preserved instead of being traded away for speed.","The LLM-as-Judge reward makes general VQA, not just visual math, trainable by RL, so open-ended answer correctness can be optimized without hand-written verifiers.","Because the upscale decision is a token output, the mechanism stays compatible with other token-compression methods; integrating them should lower the direct-answer cost further."],"supporting_citations":[{"why":"Supplies the base VLM and the official agent prompt that makes the special upscale token available in zero-shot rollouts.","marker":"[5]"},{"why":"Supplies GRPO, the policy-gradient algorithm that VisionThink extends to the multi-turn setting.","marker":"[52]"},{"why":"Supplies the initial penalty idea for tool calls that VisionThink modifies into its ratio-based threshold penalty.","marker":"[23]"},{"why":"Provides the agent-prompt format used to enable the image-resize tool call.","marker":"[6]"},{"why":"Provides the evaluation harness whose ChartQA float-comparison behavior motivates the paper's ChartQA† re-evaluation.","marker":"[86]"},{"why":"ChartQA is the strongly OCR-related benchmark whose judge-based re-scoring is central to the reported gains.","marker":"[43]"},{"why":"OCRBench is one of the strongly OCR-related benchmarks used to show that VisionThink requests high resolution when needed.","marker":"[37]"},{"why":"DocVQA is used as a weakly OCR-related benchmark where most samples are answered directly at low resolution.","marker":"[44]"}],"fun_headline_variants":["VisionThink: RL decides when to request sharper images","Vision model saves tokens by zooming in only when needed","Reinforcement learning lets VLMs choose resolution per sample","Dynamic resolution routing cuts visual tokens by half"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the external language model that says 'right' or 'wrong' about an answer agrees with human judgment on general visual questions; the only direct check reported is a 1,000-case user study.","fun_headline_variants_meta":{"raw":{"variants":["VisionThink: RL decides when to request sharper images","Vision model saves tokens by zooming in only when needed","Reinforcement learning lets VLMs choose resolution per sample","Dynamic resolution routing cuts visual tokens by half"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000429,"raw_usage":{"total_tokens":2204,"prompt_tokens":970,"completion_tokens":1234,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":1171}},"tokens_in":586,"tokens_out":1234,"duration_ms":13753,"temperature":1.0,"reasoning_tokens":1171,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:24:58.254152+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of 1,000 QA pairs from the reward-training distribution, including OCR-heavy cases, and compare the LLM judge's binary scores with human scores; if disagreements are substantially more frequent than the paper's reported zero-misclassification user study, or if they cluster in the fine-grained OCR cases that drive VisionThink's upscale decisions, the RL signal and the ChartQA† numbers would rest on a biased reward.","supporting_citations":[],"review_version":1}