{"id":"1bc51acd-20ce-4be7-9298-e94ba63e8e8c","arxiv_id":"2506.00806","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"FOCUS improves VQA accuracy by routing easy questions through fast zero-shot answering and hard questions through question-conditioned image segmentation before the final answer.","lead":"This paper introduces FOCUS, a plug-and-play method that lets multimodal AI models judge whether a visual question is easy or hard, then overlays segmented, question-specific image highlights only for hard questions. On four benchmarks it reports accuracy gains over a prior visual-prompt method, with about 44% lower inference time than that method.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The router that decides fast vs. deliberate thinking is validated only on 100 VizWiz samples through a GPT-4V proxy, and its per-sample decisions are never checked against the actual benefit of deliberate thinking on the full benchmarks.","rationale":"The reader's weakest assumption identifies the router calibration as the critical risk, and my reading agrees. The paper's own ablation shows that adaptive selection helps, but it does not show that the particular 'three Answerable samples' rule is well-calibrated on the full benchmarks. The 100-sample, single-benchmark validation with a GPT-4V proxy is too thin to support the cross-benchmark claims of both accuracy gain and efficiency gain. My proposed test directly checks whether the router's decisions align with the per-sample benefit of deliberate thinking, which is the load-bearing condition. Since the reader already issued a CONDITIONAL verdict, this concern does not change the verdict; it sharpens the specific requirement for acceptance.","tokens_in":11744,"tokens_out":4877,"duration_ms":50208,"concrete_test":"On a stratified sample of at least 300 questions per benchmark from ScienceQA, TextVQA, VizWiz, and MME, run three evaluations: (1) fast intuition alone, (2) deliberate thinking alone, and (3) the FOCUS router. For each question, label the oracle-optimal strategy as the one that produces the correct answer (or the better answer for MME). Then compute (i) the router's agreement with the oracle, (ii) the accuracy of FOCUS with the actual router versus FOCUS with an oracle router, and (iii) the time savings when the router is used versus always deliberate thinking. If the router's agreement is at or near chance, or if the gap between oracle-routed and actual-routed accuracy is comparable to the reported gain over the 'Only FI' baseline, the adaptive mechanism is not doing the attributed work.","verdict_should_be":"UNCHANGED","load_bearing_attack":"FOCUS's central mechanism is the question-complexity router in Section 2.2: a question is 'simple' only if three high-temperature 'Answerable' responses are unanimous, otherwise it is sent to the deliberate-thinking pipeline. This rule is supported solely by Appendix B.1, where GPT-4V labels 100 VizWiz images as requiring fine-grained perception, and the router agrees 81% of the time. That is a proxy validation: a model saying 'Unanswerable' is not the same as the model being wrong under fast intuition, nor does it guarantee that the deliberate-thinking pipeline would fix the error. No evidence is presented that, on the actual benchmarks, the questions routed to deliberate thinking are those where deliberate thinking beats fast intuition. Table 4 shows FOCUS outperforming both 'Only FI' and 'Only DT' on aggregate accuracy, but aggregate accuracy does not demonstrate that the router's individual decisions are correct; a router that sends nearly all questions to DT would also look adaptive in aggregate. The efficiency claim of nearly 44% faster inference than SoM likewise depends entirely on the router's false-positive rate: if the router is too conservative and routes too many simple questions to the expensive deliberate pipeline, the time savings shrink or disappear. Because the validation is 100 samples from a single benchmark, the cross-benchmark accuracy gains and efficiency gains are not yet securely traced to the routing mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FOCUS, a plug-and-play inference-time method for visual question answering that routes each question to either a fast zero-shot response (fast intuition) or a slower 'conceptualizing before observation' pipeline (deliberate thinking) that extracts key phrases with GPT-3.5 and highlights corresponding regions with Grounded-SAM. The router is a self-consistency check over three high-temperature 'Answerable/Unanswerable' responses from the MLLM. Experiments on ScienceQA, TextVQA, VizWiz, and MME with three open-source and two black-box MLLMs report consistent accuracy gains, state-of-the-art results with LLaVA-1.5-13B, and a roughly 44% inference-time reduction versus Set-of-Mark prompting.","tokens_in":11937,"tokens_out":3928,"duration_ms":36615,"significance":"If the routing mechanism is reliable, FOCUS is a practical, model-agnostic contribution: it is training-free, improves both accuracy and inference efficiency, and is evaluated across diverse MLLMs and benchmarks. The paper includes ablations, attention visualizations, analysis of failure modes, and a cost comparison, and the authors promise code release. However, the central empirical claims currently rest on the router, whose validation is thin, and on single-run accuracy numbers without error bars; the significance of the result therefore depends on whether the requested additional analyses confirm that the routing policy, rather than the overall pipeline, drives the observed gains.","major_comments":[{"comment":"The question-complexity router is the core mechanism of FOCUS, yet it is validated only on 100 VizWiz samples against a GPT-4V proxy, with agreement 81% for N=3. This does not establish that the questions routed to deliberate thinking are actually those for which deliberate thinking improves accuracy on the four benchmarks. The aggregate ablation in Table 4 cannot distinguish an adaptive router from a trivial policy that sends most questions to one branch. Please provide a per-benchmark analysis that partitions questions by router decision and reports accuracy under FOCUS, Only FI, and Only DT in each partition, together with router precision/recall relative to an oracle that knows which branch yields the correct answer.","section":"§2.2, Appendix B.1"},{"comment":"No error bars, confidence intervals, or significance tests are reported anywhere. Several headline gains are small (for example, InstructBLIP +1.8 on ScienceQA in Table 2, and the +2.1 TextVQA gain in Table 1), and the black-box results in Table 3 are based on only 250 samples. Without repeated runs or bootstrap intervals, the claim that FOCUS 'consistently improves' all models across all benchmarks is not formally supported. At minimum, report standard errors or bootstrap confidence intervals for the main comparisons, especially those versus SoM.","section":"Tables 1–5"},{"comment":"The authors' closely related method JVTP (Jiang et al., 2024b) is cited in Related Work as a prompt-optimization approach but is absent from all experimental tables. Because JVTP is also an object-centric visual and text prompting method and is part of the same line of work, omitting it from comparisons undermines the claim of state-of-the-art performance. Please add JVTP (or an explicit and justified statement of why it is not comparable) to Table 1 and the open-source model comparisons.","section":"Appendix C, Tables 1–3"},{"comment":"The efficiency claim of 'nearly 44%' faster inference than SoM is load-bearing for the paper's contribution, but the manuscript provides neither the measurement protocol nor the breakdown of time spent in each component (router, GPT-3.5 keyword extraction, Grounded-SAM segmentation, and final MLLM inference). The percentage depends directly on the router's false-positive rate, which is validated only on 100 VizWiz samples. Please specify the hardware, the number of queries, how the 44% figure is computed, and report the time breakdown separately for the FI branch and the DT branch.","section":"Figure 4, §3.2 'Cost Analysis'"},{"comment":"Hyperparameters N=3 and the grounded-SAM confidence threshold 0.7 are selected using 100 VizWiz samples and then applied to all four benchmarks. If those 100 samples are part of the VizWiz test set, the method's hyperparameters have been tuned on a test set, which would inflate the reported gains. Please clarify whether these samples are a held-out validation set, and report sensitivity of the main results to N and the threshold across benchmarks.","section":"§2.2, Appendices B.1, B.2"}],"minor_comments":[{"comment":"The section title 'Experiement' contains a typo; it should read 'Experiment'.","section":"Section 3 heading"},{"comment":"The text mentions 'LLaV A-1.6' and 'LLaV A-1.5' with inconsistent spacing; presumably both refer to LLaVA-1.5. Please standardize the model name.","section":"Section 4, 'Analysis of Answerable and Unanswerable Proportions'"},{"comment":"The caption states 'we used LLaV A-1.5 as the MLLM' but does not specify the 7B or 13B variant. The reported numbers are identical to the FOCUS + LLaVA-1.5-13B row in Table 1; please clarify which variant was used.","section":"Table 4 caption"},{"comment":"The bars are labeled with values such as 100%, 53%, 47%, 68%, and 57% without definition in the caption, and the caption contains the typo 'comparasion'. Please explain what these percentages represent and how relative inference time was measured.","section":"Figure 4"},{"comment":"The column headers 'DT (Count)', 'FI (Count)', and 'Corrected (DT/FI)' are not defined in the caption. Please explain the experimental setup for the 100 challenging questions and what the counts refer to.","section":"Table 6"},{"comment":"Equation (3) defines the processed image as a sum over segmented regions, but Appendix B.3 states that when keyword extraction fails, the original image is preserved. Please clarify how a failed extraction (K=0) is handled in the notation of Eq. (3).","section":"Section 2.3, Eq. (3)"},{"comment":"The benchmark name 'TextVQA' is inconsistently abbreviated as 'TextQA' or 'TVQA' in Tables and text. Please use a single consistent abbreviation.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a VQA / multimodal LLM venue and the idea is timely, but the missing JVTP comparison and the weak router validation are the two issues that most affect the paper's credibility. If the authors can supply the per-decision routing analysis and tighten the statistical and efficiency evidence, the work would likely be acceptable. No concerns about attribution or novelty beyond the need to situate against JVTP."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: FOCUS is a plug-and-play wrapper that routes each VQA question either to a fast zero-shot answer or to a deliberate-thinking pipeline that segments the image around key elements extracted from the question. The headline numbers are real in the tables: consistent gains over baseline MLLMs and over Set-of-Mark across ScienceQA, TextVQA, VizWiz, and MME, for open-source and black-box models, with a claimed ~44% inference-time saving over SoM.\n\nWhat is genuinely new is the routing signal: instead of annotating all detected objects for every question (SoM), FOCUS uses three high-temperature 'Answerable/Unanswerable' self-reports, and only if all three say 'Answerable' does it skip the expensive segmentation. The components are known—self-consistency, GPT keyword extraction, Grounded-SAM—but the adaptive combination is a reasonable, incremental step. The ablation showing FOCUS beats both 'Only FI' and 'Only DT' supports the dual-strategy framing.\n\nThe soft spots are real but not fatal. The router's validity rests on Appendix B.1: 100 VizWiz images labeled by GPT-4V as needing fine-grained perception, with 81% agreement. That is a proxy for 'the model would get this wrong without refinement,' not the real thing. A model can say 'Answerable' and still answer incorrectly; it can say 'Unanswerable' and then answer correctly after segmentation. The aggregate tables don't show that the routed questions are the ones where deliberate thinking helps—a router that sends most questions to DT would also beat both extremes in aggregate if DT helps on average. The paper's own unanswerable rates (LLaVA-1.5 ~42%) suggest the router is not very selective, so the time-savings figure deserves scrutiny.\n\nAlso missing: error bars, significance tests, code, a comparison to the authors' closely related JVTP method, and a 'SoTA' claim against a comparison set that stops around early 2024. Black-box results are on 250 samples. None of these sink the paper; they are reasons for revision. The limitations section is honest about the external-tool dependency and the fact that hallucinations need architectural fixes.\n\nBottom line: a solid, practical contribution with a clear idea and directionally consistent evidence, but the load-bearing router is under-validated. It deserves a serious referee and a request for per-sample routing analysis, calibration, error bars, and code. I would not cite it in its current form.","headline":"FOCUS is a sensible plug-and-play VQA wrapper with consistent gains, but its answerability router is validated on only 100 VizWiz samples and needs per-sample evidence before the accuracy and efficiency claims can be fully trusted.","tokens_in":12561,"tokens_out":3190,"would_cite":false,"duration_ms":28132,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FOCUS, a training-free method that routes each VQA question to fast zero-shot answering or to deliberate visual-marking, reports consistent gains across four benchmarks and roughly 44% faster inference than the all-object-marking baseline.","keywords":["FOCUS","visual question answering","multimodal large language models","visual prompting","question complexity routing","dual process theory","self-consistency","Grounded-SAM"],"falsifier":"Human-annotate a few thousand questions from the four benchmarks as needing fine-grained visual perception or not, compare FOCUS's router decisions to these labels, and then run FOCUS with an oracle router: if the oracle yields substantially larger gains than the self-consistency router, the confidence-check premise is the weak link.","tokens_in":11431,"feed_emoji":"🎯","tokens_out":13228,"duration_ms":104814,"temperature":0.7,"pith_summary":"FOCUS is a training-free wrapper for multimodal large language models (MLLMs) that treats visual prompting as a scarce resource to be spent only when needed. It first asks the model to judge, at high temperature and three times, whether a given visual question is answerable from the raw image; if all three answers are 'Answerable', the model answers immediately, while otherwise FOCUS extracts key objects mentioned in the question, segments them with Grounded-SAM, and feeds the highlighted image back to the model. The paper's central claim is that this complexity-adaptive strategy beats indiscriminately annotating all objects on ScienceQA, TextVQA, VizWiz, and MME, achieving state-of-the-art scores with LLaVA-1.5-13B (74.4, 63.6, 58.5, 1551.0) and cutting inference time by nearly 44% compared with Set-of-Mark prompting. If true, it would mean that the bottleneck in MLLM visual reasoning is not the visual encoder alone but attention misallocation, and that routing questions by self-reported confidence is a viable way to fix it.","feed_headline":"FOCUS saves 44% inference time and beats all-object visual prompting","feed_subtitle":"It routes easy questions straight to the answer and marks only key objects for hard ones.","key_machinery":"The load-bearing mechanism is the question-complexity router built on self-consistency: the MLLM is prompted at high temperature to answer 'Answerable' or 'Unanswerable', three samples are drawn, and the question is treated as simple only if all three agree on 'Answerable'. Complex questions then enter the conceptualizing-before-observation pipeline: GPT-3.5 extracts K key elements from the question text, Grounded-SAM produces bounding-box-conditioned segmentations for those elements, and the resulting cropped image replaces the original input. The design choice that carries the argument is the contrast with Set-of-Mark prompting, which segments all objects for every question; FOCUS's gains are attributed to spending visual markers only where the model's own uncertainty indicates attention is needed.","core_discovery":"The paper's central discovery is that the dual-process distinction from cognitive psychology transfers to VQA: some questions are answered correctly by the model's fast zero-shot response, while others need a deliberate second pass in which the model's attention is pointed at the right image regions. FOCUS operationalizes this by routing every query through a lightweight confidence check and, only for low-confidence queries, substituting a 'conceptualizing before observation' pass—the language model names the key objects from the question, a segmentation model crops exactly those regions, and the MLLM re-answers on the refined image. The paper reports that this consistently improves five MLLMs and achieves SoTA on four benchmarks with LLaVA-1.5-13B, while reducing inference time by nearly 44% versus Set-of-Mark, because simple questions skip visual marking entirely and complex questions mark only a few key elements rather than the whole image.","pith_inferences":["Editorial inference: the confidence router could be strengthened by replacing the all-or-nothing majority vote with a semantic-entropy threshold; the paper's own Table 6 shows entropy-based variants correct more challenging cases than the original strategy.","Editorial inference: the reported gains on GPT-4V and Gemini Pro come from only 250 sampled ScienceQA questions, so larger black-box evaluations are needed before the plug-and-play claim for black-box models is settled.","Editorial inference: because keyword extraction and cropping depend on GPT-3.5 and Grounded-SAM, the method inherits their failure modes; a single end-to-end module that learns which visual regions matter could be more robust, though it would sacrifice the training-free property.","Editorial inference: the same complexity-routing idea could be transferred to other multimodal tasks such as referring expression comprehension or OCR-heavy document QA, where attention misallocation is also a known bottleneck."],"forward_implications":["Visual-prompting methods for VQA should route by question difficulty rather than annotate all objects for every question.","FOCUS with a 7-billion-parameter LLaVA reaches accuracy close to vanilla 13-billion-parameter LLaVA, suggesting selective routing can substitute for part of the benefit of larger models.","The near-44% inference-time reduction over all-object marking makes selective prompting practical for latency-sensitive VQA applications.","The method is plug-and-play and improved three open-source and two black-box MLLMs, so the routing principle transfers across architectures and scales.","Attention visualizations indicate the mechanism operates by shifting the model's attention to question-relevant image regions."],"supporting_citations":[{"why":"Supplies the dual-process theory that motivates routing questions between fast intuition and deliberate thinking.","marker":"(Kahneman, 2011)"},{"why":"Provides the empirical basis that LLMs can express their confidence in answers, the premise of the complexity router.","marker":"(Xiong et al., 2023)"},{"why":"Supplies the self-consistency technique used to stabilize confidence by requiring three agreeing 'Answerable' responses.","marker":"(Wang et al., 2022)"},{"why":"Defines Set-of-Mark prompting, the all-object visual-marking baseline that FOCUS outperforms and compares against.","marker":"(Yang et al., 2023)"},{"why":"Provides Grounded-SAM, the open-set segmentation model that crops the key elements named in the question.","marker":"(Ren et al., 2024)"},{"why":"Contributes the VizWiz benchmark and the Answerable/Unanswerable prompt format adopted by the router.","marker":"(Gurari et al., 2018)"},{"why":"Provides the ScienceQA benchmark, one of the four SoTA targets.","marker":"(Lu et al., 2022)"},{"why":"Provides the MME benchmark, one of the four SoTA targets.","marker":"(Fu et al., 2023)"},{"why":"Provides LLaVA-1.5, the MLLM on which the reported SoTA numbers are obtained.","marker":"(Liu et al., 2023a)"}],"fun_headline_variants":["FOCUS: Route easy VQA questions fast, hard ones with key objects","FOCUS cuts marking to key objects, saves 44% inference time","Dual-process VQA: fast intuition or deliberate marking, not both","FOCUS: Question-aware visual prompting beats all-object marking","Smarter VQA: Only mark what matters, cut inference by 44%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The effectiveness of FOCUS rests on the assumption that three high-temperature 'Answerable' responses from the MLLM reliably distinguish questions that need no visual refinement from those that do; if that confidence proxy is miscalibrated, FOCUS either skips deliberate thinking on hard questions or wastes computation on easy ones.","fun_headline_variants_meta":{"raw":{"variants":["FOCUS: Route easy VQA questions fast, hard ones with key objects","FOCUS cuts marking to key objects, saves 44% inference time","Dual-process VQA: fast intuition or deliberate marking, not both","FOCUS: Question-aware visual prompting beats all-object marking","Smarter VQA: Only mark what matters, cut inference by 44%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000581,"raw_usage":{"total_tokens":2746,"prompt_tokens":968,"completion_tokens":1778,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":1679}},"tokens_in":584,"tokens_out":1778,"duration_ms":10759,"temperature":1.0,"reasoning_tokens":1679,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:57:50.970709+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Human-annotate a few thousand questions from the four benchmarks as needing fine-grained visual perception or not, compare FOCUS's router decisions to these labels, and then run FOCUS with an oracle router: if the oracle yields substantially larger gains than the self-consistency router, the confidence-check premise is the weak link.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the dual-process theory that motivates routing questions between fast intuition and deliberate thinking."}],"review_version":1}