{"id":"8e00e33a-3f30-4df3-8785-42bae3216f0a","arxiv_id":"2607.03530","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MLLMs that generate and render SVG sketches as multi-turn intermediate reasoning steps reach 55.1% on VSIBench and 76.0% on MindCube, far above the Qwen2.5-VL-7B backbone.","lead":"MentalThink trains multimodal language models to reason by writing, rendering, and revising SVG code as an intermediate 'mental sketch'. The approach lifts spatial-reasoning accuracy by large margins on standard benchmarks and supplies a verifiable geometric workspace inside the model loop.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"RL data drawn from VSIBench/MindCube training splits risks teaching benchmark-specific SVG patterns rather than transferable geometric self-correction.","rationale":"The Reader already flags the closed teacher and the use of training-split prompts as the weakest assumption; that is exactly the load-bearing soft spot. The paper cleanly isolates SVG versus text/bbox under matched teacher supervision (Table 3) and shows complementary gains from SVG data + general perception data + RL (Table 2), so the contribution of the intermediate representation itself is real. What remains unproven is that the multi-turn RL loop installs genuine geometric self-correction rather than high-fidelity imitation of Gemini-3 traces on the evaluation families. Because the reward contains no intermediate geometric signal and the RL prompts come from the same benchmarks, a simple contamination check is required before the strongest claim can be accepted at face value. The existing CONDITIONAL verdict already captures this; no stronger rejection is warranted, but the concrete hold-out test should be required for full acceptance.","tokens_in":22094,"tokens_out":579,"duration_ms":5748,"concrete_test":"Hold out a completely disjoint spatial suite never seen in SFT or RL (e.g., a fresh subset of OmniSpatial or SpatialViz items with no prompt overlap) and re-evaluate the final RL checkpoint versus the SFT-only checkpoint under identical decoding. If the RL-minus-SFT delta collapses to near zero while remaining large on the original VSIBench/MindCube test splits, the self-correction claim is benchmark-contaminated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (executable SVG as a verifiable workspace enabling genuine multi-turn geometric self-correction) rests on Stage-2 GRPO (Eq. 10, λ_fmt=0.25 / λ_ans=0.75, T_max=5) producing iterative inspection/revision. Section 4.1 and A.1 state that the RL corpus uses prompts from the training splits of the same two benchmarks that dominate the headline numbers (VSIBench 55.1%, MindCube 76.0%). The reward is only format-validity plus final-answer correctness; there is no geometric consistency, multi-view consistency, or intermediate-SVG quality term. Consequently the observed gains (Table 1, Table 2 last row) and the qualitative patterns in Figure 3 can be explained by the policy learning to emit teacher-style SVG sketches that co-occur with correct answers on these particular prompt distributions, rather than by acquiring a general, inspect-and-revise geometric capability. The controlled paradigm comparison (Table 3) and the SFT ablations do not isolate this contamination because they also operate inside the same data families.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"MentalThink proposes a think-with-SVG pipeline that lets multimodal LLMs externalize spatial hypotheses as executable SVG code, render them deterministically, and condition subsequent reasoning on the rendered images. The method is realized by a two-stage recipe: SFT for syntactic alignment and SVG-mediated thought externalization, followed by multi-turn GRPO (Eq. 10) that rewards format validity and final-answer correctness. On a Qwen2.5-VL-7B backbone the approach reports large gains on spatial benchmarks (55.1% VSIBench, 76.0% MindCube-Tiny; Table 1), with ablations isolating SVG data, general spatial data, and RL (Table 2), a controlled paradigm comparison against text- and bbox-based intermediates (Table 3), and qualitative traces of perspective taking, visual refinement, and scene construction (Figure 3).","tokens_in":22446,"tokens_out":1186,"duration_ms":9298,"significance":"If the gains reflect a transferable geometric workspace rather than benchmark-specific pattern matching, the work supplies a clean, tool-free intermediate representation that is both linguistically native to LLMs and deterministically visualizable. The formalization of SVG-augmented states (Definitions 3.4–3.6), the closed render–inspect loop, and the controlled paradigm comparison (Table 3) are concrete contributions that other multimodal reasoning systems can build on. The adaptive System-1/System-2 statistics (Table 4) further suggest that the model can allocate visual computation according to task difficulty, which is a useful empirical finding for test-time scaling of spatial reasoning.","major_comments":[{"comment":"Section 4.1 and Appendix A.1 state that the RL corpus is drawn from the training splits of VSIBench and MindCube—the same two benchmarks that dominate the headline numbers in Table 1. The reward (Eq. 10) contains only format validity and final-answer match; there is no geometric-consistency, multi-view-consistency, or intermediate-SVG quality term. Consequently the large RL increments (especially MindCube 36%→76%) and the multi-turn revision patterns in Figure 3 remain compatible with the policy learning to emit teacher-style SVG sketches that co-occur with correct answers on these particular prompt distributions, rather than acquiring a general inspect-and-revise geometric capability. A held-out spatial suite never seen in RL (or an ablation that freezes the SVG generator after SFT and only optimizes answer selection) is needed to separate these explanations.","section":null},{"comment":"Table 3’s paradigm comparison (Text / BBox / SVG) is the cleanest evidence that the intermediate representation itself matters, yet it is reported only for a single teacher-generated SFT setting and without error bars or multiple seeds. Because the subsequent RL stage re-uses the same benchmark families, it is unclear whether the SVG advantage survives once the policy is optimized under the format+answer reward. Reporting the same three-way comparison after RL (or at least variance across seeds) would make the central claim load-bearing rather than suggestive.","section":null},{"comment":"The qualitative claim of “iterative inspection, revision, and refinement” (Abstract, §3.3, Figure 3) is supported only by selected traces. Table 4 shows that average turns rise only from 1.19 to 1.64 and that SVG is invoked on 63.7% of compositional items; there is no quantitative measure of how often an intermediate SVG is actually revised after rendering (as opposed to a single successful sketch followed by an answer). Without such a revision-rate or self-correction statistic, the multi-turn RL narrative remains under-supported relative to the performance claims.","section":null}],"minor_comments":[{"comment":"Table 1 reports MindCube* (MindCube-Tiny) without stating the exact subset size or selection criterion; a one-sentence clarification would aid reproducibility.","section":null},{"comment":"The reward coefficients λ_fmt=0.25 / λ_ans=0.75 and T_max=5 are given without sensitivity analysis; a short appendix note on robustness would strengthen confidence in the free parameters.","section":null},{"comment":"Figure 2’s example trajectory is helpful, but the rendered <image1–3> thumbnails are too small to verify that the SVG actually encodes the claimed egocentric axes; larger insets or an appendix figure would help.","section":null},{"comment":"Several related-work citations (e.g., Visual Sketchpad, cognitive-map methods) are discussed at a high level; a short paragraph contrasting the executable, code-native nature of SVG against non-executable sketch or map tokens would sharpen the novelty claim.","section":null},{"comment":"Typographical inconsistencies appear in model names (LLaV A vs LLaVA) and in the arXiv header date; a light copy-edit pass is warranted.","section":null}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the controlled SFT paradigm comparison is a genuine strength, but the RL data contamination risk is load-bearing for the central claim and is not currently isolated. I would be comfortable moving to minor revision or accept once the authors either (a) evaluate on a spatial suite never used in RL or (b) provide a clear ablation that freezes SVG generation after SFT. Without that, the 40-point MindCube jump is hard to interpret as evidence of transferable geometric self-correction."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The new piece is treating SVG as the multi-turn mental medium itself—generate code, render it deterministically, re-ingest the image, revise—rather than as a final drawing target or a sparse box/map. That is distinct from text CoT, bbox layouts, cognitive maps, and external tool calls, and the paper shows it works.\n\nWhat they do well: the two-stage recipe is clear (SFT for syntactic alignment + multi-turn GRPO with a simple format+answer reward). Table 2 cleanly separates SVG-reasoning data, general spatial data, and RL. Table 3 holds teacher and supervision fixed across text / bbox / SVG and still favors SVG. The qualitative traces (perspective taking, left-right correction after orientation mismatch, multi-view composition) match the claimed patterns. Gains on MindCube (+40) and VSIBench (+24) over the 7B backbone are large and not just scaling noise. Formal definitions of the SVG-augmented state and the render loop are light but coherent; no load-bearing circularity.\n\nSoft spots, in proportion. The stress-test note is real but not fatal: RL prompts come from the training splits of the two headline benchmarks, and the reward has no geometric-consistency term. So some of the final jump can be “emit teacher-style SVG that co-occurs with correct answers on these distributions.” That weakens the strongest reading of “genuine geometric self-correction,” but it does not erase the SFT ablations or the controlled paradigm comparison, both of which already move the needle before RL. Other limits are ordinary for the genre: closed teacher (Gemini 3), no released code/data, no error bars, free hyperparameters (λ, T_max, data mix). Reproducibility is limited; the central idea is still testable.\n\nWho it is for: people working on spatial MLLMs, intermediate representations, and test-time visual scratchpads. A serious referee should see it. I would bring it to reading group, cite the paradigm and the numbers when discussing intermediate visual media, and expect revision mainly on open artifacts and a cleaner out-of-family RL set. Worth engaging.","headline":"SVG as a revisable intermediate medium is a clean, useful idea with large gains; the main soft spot is RL on the same benchmark families, not a collapse of the claim.","tokens_in":23067,"tokens_out":532,"would_cite":true,"duration_ms":5224,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Multimodal models reason better about space when they draw and re-render their own intermediate SVG sketches.","keywords":["MentalThink","think-with-SVG","multimodal reasoning","scalable vector graphics","spatial reasoning","multi-turn RL","mental imagery","visual-symbolic"],"falsifier":"Train an identical backbone with matched multi-turn text or bounding-box trajectories (same teacher, same tasks, same RL budget) and check whether the SVG version still shows the reported 40-point MindCube and 24-point VSIBench gains and the same adaptive multi-turn revision patterns on held-out spatial problems.","tokens_in":23016,"feed_emoji":"✏️","tokens_out":634,"duration_ms":5201,"temperature":0.7,"pith_summary":"MentalThink argues that multimodal language models can do stronger spatial reasoning if they treat scalable vector graphics (SVG) code as an executable mental scratchpad. Instead of describing a scene only in words or sparse boxes, the model writes SVG, has it rendered into an image, looks at the result, and revises the sketch over multiple turns. A two-stage recipe first teaches the model to produce valid SVG (supervised fine-tuning) and then uses multi-turn reinforcement learning to reward iterative inspection and self-correction. On spatial benchmarks the method lifts a 7B backbone to 55.1% on VSIBench and 76.0% on MindCube, and qualitative traces show the model spontaneously taking new perspectives, fixing orientation errors, and composing multi-view scenes. The claim is that a deterministic, compositional visual language gives models a verifiable workspace that pure text lacks, so spatial hypotheses can be checked rather than merely narrated.","feed_headline":"Models that sketch in SVG solve spatial puzzles far better","feed_subtitle":"Drawing and re-rendering intermediate vector code lifts MindCube to 76% and VSIBench to 55%","key_machinery":"The think-with-SVG pipeline: an SVG-augmented reasoning state that interleaves linguistic steps with optional SVG code, a deterministic renderer that turns code into an image observation, and a multi-turn RL loop (format reward plus answer reward) that rewards the model for generating, inspecting, and revising those mental sketches.","core_discovery":"Executable SVG code functions as a structured intermediate representation that lets multimodal models externalize spatial hypotheses, render them deterministically, and refine them multi-turn, producing large gains on geometry-heavy spatial benchmarks and eliciting dynamic perspective taking, visual reflection, and compositional scene construction that language-only or bounding-box intermediates do not match.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["SVG sketches let models externalize spatial hypotheses","Executable SVG code enables multi-turn spatial refinement","Models draw and revise SVG for stronger spatial reasoning","Renderable vector code gives MLLMs a verifiable visual workspace","Think-with-SVG turns intermediate sketches into spatial insights"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That teacher-synthesized multi-turn SVG traces plus a simple format-and-answer reward are enough to produce genuine geometric self-correction rather than style-matching or benchmark-prompt memorization.","fun_headline_variants_meta":{"raw":{"variants":["SVG sketches let models externalize spatial hypotheses","Executable SVG code enables multi-turn spatial refinement","Models draw and revise SVG for stronger spatial reasoning","Renderable vector code gives MLLMs a verifiable visual workspace","Think-with-SVG turns intermediate sketches into spatial insights"]},"model":"grok-4.5","effort":"low","cost_usd":0.006754,"raw_usage":{"total_tokens":1681,"prompt_tokens":737,"num_sources_used":0,"completion_tokens":62,"cost_in_usd_ticks":67540000,"prompt_tokens_details":{"text_tokens":737,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":882,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":737,"tokens_out":62,"duration_ms":7191,"temperature":1.0,"reasoning_tokens":882,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T01:50:11.613697+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train an identical backbone with matched multi-turn text or bounding-box trajectories (same teacher, same tasks, same RL budget) and check whether the SVG version still shows the reported 40-point MindCube and 24-point VSIBench gains and the same adaptive multi-turn revision patterns on held-out spatial problems.","supporting_citations":[],"review_version":1}