{"id":"a30693dc-511a-4dfb-a8f3-ac5c9894633e","arxiv_id":"2501.05452","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Visual editing of input images as a chain of thought improves multimodal LLM accuracy on structured image tasks by 3 to 12 points.","lead":"ReFocus lets GPT-4o edit its own input image with Python code, drawing boxes, highlighting sections, or masking parts of tables and charts before answering. On table and chart question-answering benchmarks it reports gains of 3 to 12 accuracy points, and the visual chain-of-thought data it generates outperforms plain QA pairs as training signal.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gains may reflect the textual header/bbox hints in ReFocus prompts rather than the visual edits themselves; the paper lacks a no-editing control with identical text cues.","rationale":"The reader's weakest_assumption focuses on OpenCV coordinate acquisition, which is a real implementation risk but not the most load-bearing one: if coordinates were frequently wrong, the edits would likely hurt accuracy or produce inconsistent results, whereas the observed gains are consistent and large. The more serious threat is the input confound: the ReFocus prompt contains textual column/row headers and bounding-box coordinates that the vanilla GPT-4o baseline does not receive. This directly threatens the paper's central attribution of gains to visual editing, because the model could be answering better simply because it is given OCR-derived text cues, tool names, and coordinate information, independent of any actual image modification. The reader's rationale does mention the 'no additional information' overstatement and the need for a textual-hint baseline, so we partially agree, but we elevate this to the primary stress point. The SFT comparison in Section 5 is also weakened by filtering on correct predictions and by hyperparameter selection, as the reader notes, but the prompting claim is the foundational result. A no-editing control with identical textual cues is a single, decisive experiment: if it closes most of the gap, the core claim collapses to 'text hints help'; if it does not, the visual-editing mechanism is substantially supported. Because this check is missing, the current CONDITIONAL verdict is appropriate, and no verdict change is recommended.","tokens_in":22699,"tokens_out":4243,"duration_ms":42229,"concrete_test":"Run a no-editing control on the exact evaluation subsets of Table 1: prompt GPT-4o with the original image plus the identical text block that ReFocus injects ('USER Bounding Box Info' with column/row header names and coordinates for tables; x/y value names and coordinates for ChartQA; subplot contour list for CharXiv), but remove all editing tools and instruct the model to answer directly in one step. Compare accuracy. If this control reproduces most of the ReFocus gain (within about 2-3 points), the central claim that image editing drives the improvement fails; if the gap remains large, the concern is refuted. Also report per-dataset gains and the fraction of examples where ReFocus performed no edit, since those examples isolate the text-hint effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is a confound in the central comparison (Table 1). The ReFocus prompt (Section 3.3; Appendix C) injects structured text that the vanilla GPT-4o baseline never sees: for tables, 'USER Bounding Box Info' lists column headers and row headers with bounding boxes; for ChartQA, x/y value names are listed; for CharXiv, the top-k contour coordinates are provided. Thus the reported +11.0/+6.8 gains are attributed to visual editing, but they could instead come from the model receiving OCR-style header/coordinate hints and tool descriptions that improve attention or reduce OCR errors. The 'no additional information' argument (Section 4.3) treats these cues as non-informative because they are derived from the image, but they are additional to the model's input and are absent from the baseline. Table 3's Text+Figure baseline supplies full gold CSV text, not the same minimal header/bbox hint set, so it does not isolate the editing effect. Without a control that gives GPT-4o the same textual hints but forbids editing, the causal claim of Section 3 is underdetermined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ReFocus, a framework that lets a multimodal LLM (GPT-4o) generate Python code to edit the input image—masking, highlighting, or drawing boxes around rows, columns, bars, or subplots—as intermediate 'visual thoughts' before answering. The authors report consistent accuracy gains over vanilla GPT-4o on table (VWTQ, VWTQ_syn, VTabFact) and chart (CharXiv, ChartQA horizontal/vertical bar) benchmarks, and they collect a 14k training set that, when used for SFT of Phi-3.5-vision, outperforms QA-pair and textual-CoT supervision. The paper includes detailed prompts, tool descriptions, and analyses of which editing operations help.","tokens_in":22908,"tokens_out":4225,"duration_ms":39206,"significance":"If the causal attribution holds, the idea of using executable visual edits as a chain of thought is a useful and transferable contribution to structured image understanding. The paper is clearly written, provides full prompts and tool code, and the released SFT data is a potentially valuable resource. The cross-checkpoint consistency (two GPT-4o snapshots) and the open-source model transfer experiment (Table 2) strengthen the empirical case. However, the main experimental comparison is confounded by additional textual information injected only into the ReFocus prompt, which undermines the central claim that visual editing itself drives the gains; a control experiment is needed to isolate the effect.","major_comments":[{"comment":"The central comparison in Table 1 is confounded. In the ReFocus system prompt (Appendix C), the model receives 'USER Bounding Box Info' listing column headers, row headers, and their bounding-box coordinates for tables, and x/y value names for charts. The vanilla GPT-4o baseline does not receive this textual information. Thus the reported gains (+11.0 table, +6.8 chart) could arise from the OCR-style header/coordinate hints that improve attention or reduce OCR errors, rather than from the visual edits themselves. The claim in §4.3 that 'REFOCUS does not introduce any additional information' is contradicted by the prompt design. The paper needs a control condition that gives GPT-4o the same textual hints but forbids editing (or, conversely, applies the edits without the textual hints). Without such a control, the causal claim of §3 is underdetermined.","section":"§3.3, Appendix C, Table 1"},{"comment":"The SFT comparison between 'SFT w/ REFOCUS VCoT' and 'SFT w/ REFOCUS CoT' is not cleanly controlled. Table 7 reports 'include edited image in input' as 'No' for the VCoT model and 'Yes' for the CoT model, while the text says the only difference between the two data types is the presence of refocus bounding-box coordinates. If CoT training includes the edited image and VCoT training does not, then the 2.6% average gain attributed to bounding boxes could instead reflect the difference in visual input, or the comparison could be accidentally favorable in a different way. Please clarify the exact input formats and, ideally, run a matched control where the only varying factor is the bounding-box coordinates.","section":"§5, Table 7"},{"comment":"The coordinate-acquisition heuristics (longest vertical/horizontal contours for tables; top-k contours with k=10 for CharXiv subplots) are not evaluated for accuracy. If the detected bounding boxes are wrong, the mask/draw/highlight edits could hide or emphasize the wrong regions. The paper reports no detection-quality metrics or failure cases, so the robustness of the pipeline across the diverse table/chart styles is unclear. Reporting the fraction of cases where the acquired boxes match the intended row/column/bar regions would substantially increase confidence in the mechanism.","section":"§3.1, §4.3"}],"minor_comments":[{"comment":"The training-set size is given as 14k in the abstract and §5, but §1 (contribution 3) states 'we curate a 21k training set.' Please align these numbers.","section":"Abstract vs. §1 vs. §5"},{"comment":"The CharXiv row for 'Text input' is shown as '\\' (missing). Since this is a key comparison, a value or an explicit statement that CharXiv text input is unavailable would improve the table.","section":"Table 3"},{"comment":"The caption says the model 'correctly recognize the characters as \"Partenvia\"' but the correct spelling appears to be 'Partenavia' (as in the original and the model's first attempt). This seems like a typo and should be fixed.","section":"Figure 5 caption"},{"comment":"The analysis of editing tools (mask vs. draw vs. highlight) compares only edited conditions; adding a no-edit baseline with the same prompt structure would make the tool comparison more interpretable.","section":"§4.3, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The confound in the headline comparison is the main barrier. If the authors can run a text-only control (same headers/bbox hints, no editing tools) and show that the gains persist or are substantially reduced, the paper will be much stronger. The SFT comparison also needs clarification/control for the edited-image input difference. The contribution and dataset are valuable; I recommend a major revision rather than rejection, as the required experiments are within the scope of the current benchmark setup."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper shows a neat prompting trick, but the headline gains are undercut by a confound. In Table 1, the ReFocus prompt supplies column/row headers and bounding boxes as text (Appendix C), while the vanilla GPT-4o baseline only sees the image. So the +10.4 on VWTQ, +7.2 on horizontal bars, etc., could come from the model receiving a structured text list of names and coordinates rather than from the visual edits. The paper's claim that it works 'without introducing additional information' does not survive reading the prompt. They need a control that gives GPT-4o the same header/bbox text but forbids editing. Table 3's Text+Figure baseline uses full gold text, so it does not isolate the editing effect.\n\nWhat is genuinely new: applying code-based visual editing (mask, draw, highlight) to structured images—tables, bar charts, multi-subplot scientific figures—is a sensible extension of Visual Sketchpad, and the coordinate acquisition via OpenCV is clearly described. The SFT experiment is the more convincing part: adding focus-area bounding boxes to CoT data yields a small, plausible gain over plain CoT on Phi-3.5-vision (+2.6%). That comparison is less confounded because both arms share the same filtered training data.\n\nSoft spots, in proportion: the missing no-editing control is the load-bearing issue. Also, Table 1 shows a negative result (CharXiv with gpt-4o-2024-08-06: -2.7) that contradicts the 'consistent gains' phrasing. The abstract says 14k training set, the introduction says 21k; Section 5 says 14,344. That discrepancy needs fixing. The SFT data is filtered to examples where ReFocus+GPT-4o succeeded, so the 'better than standard VQA data' claim is really 'better than QA on this filtered selection.' No code or data are released, and there is no variance or multi-seed reporting.\n\nWho this is for: researchers in document/chart VQA and visual prompting. The framework is useful and the SFT direction is worth pursuing. It deserves peer review, but authors should be pushed to run the same-text-cues-no-editing control and to release artifacts. As-is, I would not lean on the main table for a causal claim.","headline":"A useful visual-editing prompting framework for tables and charts, but the headline gains are likely inflated because the ReFocus prompt also feeds the model OCR header/coordinate text the baseline never sees.","tokens_in":23459,"tokens_out":2669,"would_cite":true,"duration_ms":28263,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ReFocus claims that letting a multimodal LLM edit the input image between reasoning steps—masking, boxing, or highlighting regions—improves its answers on tables and charts, and that the resulting visual traces are better training data…","keywords":["visual chain-of-thought","multimodal large language models","visual question answering","image editing","selective attention","tables","charts","structured image understanding"],"falsifier":"Run ReFocus on a table or chart benchmark with the automatic OpenCV or dataset coordinates replaced by ground-truth boxes, and separately with boxes deliberately shifted by 5 to 10 percent of image width. If the shifted-box version matches the correct-coordinate version, the gains are not actually caused by isolating the intended region; if the ground-truth version beats the automatic version, the coordinate detector is the bottleneck.","tokens_in":22472,"feed_emoji":"📊","tokens_out":9364,"duration_ms":79378,"temperature":0.7,"pith_summary":"The paper argues that a multimodal language model can do better multi-step visual reasoning on tables and charts if, between reasoning steps, it edits the image itself: masking out irrelevant columns, drawing boxes around relevant rows, or highlighting bars and subplots. The authors build ReFocus, which gives GPT-4o a small set of Python image-editing tools and prompts it to generate code that refocuses the image before answering. On six benchmarks the edited-image loop outperforms plain GPT-4o by an average of 11.0 points on table tasks and 6.8 points on chart tasks, and it does so without adding external information. The same loop also produces a 14k training set whose visual chain-of-thought records—including the bounding boxes the model chose to focus on—teach a smaller model (Phi-3.5-vision) more effectively than the same question-answer pairs or text-only CoT. If this holds, intermediate visual edits, not just text reasoning, are a viable way to steer large multimodal models on structured images.","feed_headline":"Masking and boxing images lifts GPT-4o table/chart scores by 11 points","feed_subtitle":"ReFocus lets the model edit the input image with Python code; the same traces also train smaller models 8 points better.","key_machinery":"The load-bearing mechanism is the ReFocus loop. The multimodal LLM is prompted to alternate THOUGHT and ACTION, where an ACTION is Python code calling one of a fixed set of image-editing tools—highlight, mask-out, or draw-box applied to columns, rows, bars, or subplots. The code runs, the edited image replaces the model's visual input, and the loop terminates when the model outputs an answer. Coordinates come from OpenCV contour detection for table rows and columns and chart subplots, plus dataset-provided axis coordinates for bar charts. The physical edit is the point: it removes or marks the distracting regions before the next perceptual step, so the model re-reads a simplified image instead of re-attending to the original one.","core_discovery":"The paper's central claim is that selective attention on structured images can be operationalized as executable visual edits: when a multimodal LLM is allowed to iteratively call code that masks columns, draws red boxes over rows, or highlights bars and subplots, its accuracy on visual QA over tables and charts rises consistently. ReFocus does not add outside knowledge; it only returns a modified version of the input image. The authors attribute the gains to improved visual grounding, OCR, and counting, and they show the edited images also transfer: open-source models that never saw such edits improve when given ReFocus-edited images. When the edited-image traces are distilled into about 14k supervised fine-tuning examples, a Phi-3.5-vision model trained with the visual chain-of-thought data beats the same model trained on QA pairs by 8.0 points on average and on CoT text by 2.6 points.","pith_inferences":["A direct extension the paper does not test: the same edit loop should transfer to other dense visual layouts such as forms, maps, or diagrams, since the mechanism is not table- or chart-specific.","The SFT result suggests focus-area bounding boxes are a cheap supervision signal: any correct GPT-4o run can be recorded as visual chain-of-thought data, so the data-generation pipeline could scale to more benchmarks without human annotation.","The bounding-box dependency points to a testable improvement: replacing OpenCV coordinate detection with a model that predicts layout, or with OCR-derived boxes, could remove the main source of brittle edits and improve gains on unusual table and chart styles.","One could also make ReFocus iterative at training time, letting the model decide when to stop editing and compare answers across multiple edited views; the paper's loop is fixed, so the error cases where it stops too early or over-masks are unexplored."],"forward_implications":["The largest reported gains are on VWTQ and horizontal bar charts, with near-zero gain on VTabFact, so the benefit is strongest where the question forces the model to pick among many visually similar rows, columns, or bars.","ReFocus with figure-only input beats GPT-4o given both gold text and the figure on most tasks, meaning visual editing can substitute for perfectly transcribed table or chart content.","The edited images transfer: LLaVA-NeXT and Phi-3-vision, which were not trained on visual prompts, mostly improve when fed ReFocus-edited images.","The visual chain-of-thought data is a better SFT supervision signal than QA pairs by 8.0 points on average and than text-only CoT by 2.6 points, so the focus-area information, not just the reasoning text, carries training signal.","Mask-out, draw-box, and highlight produce similar results, indicating the benefit comes from refocusing itself rather than from one specific edit style."],"supporting_citations":[{"why":"Supplies the TableVQA-bench test sets (VWTQ, VWTQ_syn, VTabFact) and the rendered table images that ReFocus edits.","marker":"[16]"},{"why":"Source of the ChartQA horizontal and vertical bar subsets and of the 15,059 training questions used to build the ReFocus SFT data.","marker":"[24]"},{"why":"Source of the 143 multi-subplot CharXiv reasoning questions used for the scientific-chart evaluation.","marker":"[34]"},{"why":"Prior visual chain-of-thought approach whose vision tools handle natural images; ReFocus contrasts itself by adding no external information.","marker":"[13]"},{"why":"Visual programming baseline, re-implemented with GPT-4o for a fair comparison in the main results table.","marker":"[10]"},{"why":"The GPT-4o multimodal backbone that ReFocus augments and that generates the editing traces for the SFT set.","marker":"[26]"},{"why":"Phi-3 and Phi-3.5-vision, the open-source model family used for SFT experiments and for testing whether edited images transfer to models not trained on visual prompts.","marker":"[1]"},{"why":"Defines chain-of-thought prompting, the text-based reasoning paradigm that ReFocus extends to the visual domain and the CoT baseline in SFT.","marker":"[35]"},{"why":"Set-of-Mark visual prompting work showing that segmentation boosts GPT-4V grounding; used to contrast object-centered visual prompts with structured-image editing.","marker":"[37]"}],"fun_headline_variants":["Visual edits as chain of thought boost image QA by 11 points","ReFocus: coding image edits sharpens structured visual QA","Image editing via Python code unlocks better table/chart reasoning","Editable images improve LLM reasoning on charts and tables"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline rests on the coordinate detector drawing accurate boxes around rows, columns, bars, and subplots; if those boxes are wrong, the edits hide or emphasize the wrong region and the claimed gains should shrink or reverse.","fun_headline_variants_meta":{"raw":{"variants":["Visual edits as chain of thought boost image QA by 11 points","ReFocus: coding image edits sharpens structured visual QA","Image editing via Python code unlocks better table/chart reasoning","Editable images improve LLM reasoning on charts and tables"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00124,"raw_usage":{"total_tokens":5115,"prompt_tokens":997,"completion_tokens":4118,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":4049}},"tokens_in":613,"tokens_out":4118,"duration_ms":27109,"temperature":1.0,"reasoning_tokens":4049,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:12:22.762181+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ReFocus on a table or chart benchmark with the automatic OpenCV or dataset coordinates replaced by ground-truth boxes, and separately with boxes deliberately shifted by 5 to 10 percent of image width. If the shifted-box version matches the correct-coordinate version, the gains are not actually caused by isolating the intended region; if the ground-truth version beats the automatic version, the coordinate detector is the bottleneck.","supporting_citations":[{"cited_title":"Visual program- ming: Compositional visual reasoning without training","cited_arxiv_id":null,"evidence_quote":"Visual programming baseline, re-implemented with GPT-4o for a fair comparison in the main results table."},{"cited_title":"Chain-of- thought prompting elicits reasoning in large language models","cited_arxiv_id":null,"evidence_quote":"Defines chain-of-thought prompting, the text-based reasoning paradigm that ReFocus extends to the visual domain and the CoT baseline in SFT."}],"review_version":1}