{"id":"e68b4131-3297-4465-b97a-7264a549ba84","arxiv_id":"2505.23977","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"LLM-derived rules, genetic expansion, and program-drawn images produce 660K visual logic puzzles; GRPO training on them lifts a VLM's accuracy on those puzzles and modestly on MathVista.","lead":"The authors built VisualSphinx, a dataset of over 660,000 synthetic visual logic puzzles with machine-generated rules and answers, and trained a vision-language model on them with reinforcement learning. The resulting model improved sharply on similar puzzles and gained about 5 points on a standard visual-math benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run RL with no control condition cannot support the causal claim that VisualSphinx's logical coherence and readability drive the reported gains; the MathVista transfer may reflect a generic GRPO or format effect.","rationale":"The dataset pipeline is cheap, the release is valuable, and the 930 manually checked test set plus external MathVista evaluation are real evidence. The reader's weakest assumption about label correctness is legitimate, but even with perfect labels the central claim needs a control for the RL intervention itself: the abstract asserts that logical coherence and readability are what drive improvement, and the experiments do not isolate those properties from the generic effects of GRPO, longer chain-of-thought, or format changes. My concrete test would settle that attribution. The reader already issued a CONDITIONAL verdict, and this concern does not move the verdict beyond that condition; it sharpens the condition by naming the missing control and seed variance.","tokens_in":20665,"tokens_out":5219,"duration_ms":57613,"concrete_test":"Run the identical GRPO recipe (Section 5.1) on: (a) the 10k VisualSphinx puzzles as-is; (b) the same puzzles with answer labels randomly permuted; (c) a matched set of synthetic images of the same visual style that follow no coherent rule, with the same 4/10-option layout. Use 3 random seeds per condition and report mean ± std on VisualSphinx-TEST and MathVista-testmini. If (b) or (c) reproduces the +26.64/+4.6 gains, the causal claim fails; if only (a) improves, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.2's central claim is causal: GRPO on VisualSphinx improves logical reasoning and transfers to algebra, arithmetic, and geometry. The evidence rests on one Qwen2.5-VL-7B run compared only with its untrained checkpoint. Two load-bearing conditions are unverified. First, label correctness: Section 3 Step 3 has an LLM write correct.py and incorrect.py, and only the 930-puzzle VisualSphinx-TEST is manually checked; the 10k RL examples are filtered by VLM scores and pass rates. Noisy labels would let RL exploit generator artifacts. Second, and more decisive, there is no control condition: no GRPO run on the same puzzles with shuffled answers, on matched incoherent visual data, or on seed-only data. Without such a control, the +26.64 point gain on VisualSphinx-TEST (Table 1) can be explained as in-distribution overfitting, and the +4.6 point MathVista gain (Figure 8) can be explained as a generic GRPO effect on answer format, chain-of-thought length, or calibration. No error bars or multiple seeds are reported, so even the raw transfer number is fragile. The causal attribution to 'logical coherence and readability' therefore lacks a direct test.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VisualSphinx, a synthetic dataset of over 660K visual logic puzzles spanning inductive, deductive, spatial, and structural reasoning categories. A four-stage pipeline—seed-question collection and rule abstraction, a rule-level genetic algorithm for rule expansion, LLM-generated Python scripts that render five rule-compliant and three rule-violating images per rule, and three puzzle assembly strategies—produces the dataset at a total cost under $1000. The authors then train Qwen2.5-VL-7B with GRPO on a 10K-example subset and report accuracy gains on a manually checked 930-puzzle test set (29.30% to 55.94%) and on MathVista-testmini (59.4% to 64.0% overall), with subcategory improvements in algebraic, arithmetic, geometric, and logical reasoning.","tokens_in":20932,"tokens_out":4287,"duration_ms":39584,"significance":"The dataset construction is a genuine engineering contribution: the pipeline is fully automated, low-cost, and yields a large, diverse set of visual logic puzzles with a manually verified test set and an accompanying human quality study (78% of 1,200 sampled puzzles rated comparable or better than seed puzzles). If the empirical claims are confirmed with proper controls, VisualSphinx would be a valuable, reproducible resource for VLM post-training. However, the current evidence for the causal role of 'logical coherence and readability' is incomplete: the experiments lack control conditions, multiple seeds, and uncertainty quantification, and the training-label verification is limited to the small test set.","major_comments":[{"comment":"The central causal claim that GRPO on VisualSphinx improves logical reasoning and transfers to algebra, arithmetic, and geometry is not isolated by the reported experiments: the only comparison is between one post-RL run of Qwen2.5-VL-7B and its untrained checkpoint. There is no control run with GRPO on matched data that lacks VisualSphinx's logical structure (e.g., the same puzzles with shuffled answers, incoherent image sequences, or seed-only questions), no second seed, and no error bars. Under these conditions the +26.64-point gain on VisualSphinx-TEST is consistent with in-distribution overfitting or a generic GRPO/format effect, and the +4.6-point MathVista gain (Figure 8) could arise from longer chain-of-thought or better answer formatting rather than from 'logical coherence and readability' of the dataset. Please add at least one control GRPO condition and report multiple seeds or bootstrap confidence intervals.","section":"Section 5.2, Table 1, Figure 7"},{"comment":"Label correctness is load-bearing and is verified only on the 930-puzzle VisualSphinx-TEST. The 660K training puzzles (and the 10K RL subset selected in Section 5.1) are generated by LLM-written Python scripts and filtered by pHash/SSIM and VLM-based readability/coherence scores; a script that renders images not actually satisfying the extracted rule would teach the model to match generator artifacts. The paper should report a human audit of a random sample of the 10K RL examples (or an automated check that the rendered images provably satisfy the rule constraints), with per-category agreement, and should state the distribution of VLM coherence scores for the RL subset.","section":"Section 3, Step 3; Section 4"},{"comment":"The difficulty filter used to select the RL training set is partly circular: pass rates come from a Qwen2.5-VL-7B model trained on the same seed-question distribution, and the readability/coherence filter uses GPT-4.1-mini, a VLM similar in capability to the model that generated the images. Selecting examples with pass rate 0.375-0.875 and coherence >= 8 could preferentially retain puzzles where the generator's artifacts are recognized by the scorer rather than puzzles with genuinely sound logic. Please show that the main results are robust to the selection thresholds, for example by also training on a random or differently filtered subset of the 110K groups.","section":"Section 4.1, Pass Rate; Section 5.1"},{"comment":"The transfer claim rests on aggregate and subcategory accuracy on MathVista-testmini with no measures of uncertainty or per-category sample sizes; some subcategories (e.g., LOG, NUM) are small, so the reported gains may be within noise. In addition, without a control RL run the transfer gain could be a general effect of RL training on any visually grounded multiple-choice data. Please provide per-category counts, confidence intervals, and a matched control.","section":"Figure 8"}],"minor_comments":[{"comment":"The text says training uses a batch size of 128, whereas Table 4 reports Rollout Batch Size 64 and PPO Mini Batch Size 64; please reconcile these numbers.","section":"Section 5.1 and Table 4"},{"comment":"Please clarify whether the six additional distractors in the 10-option variant are chosen to be rule-violating with respect to the target rule, or merely drawn from other image groups; the current wording is ambiguous.","section":"Section 3, Step 4"},{"comment":"The text does not specify which LLM produced the readability and coherence scores in Figure 5; please state the model and, ideally, provide a small human agreement study on a subsample.","section":"Section 4, Figure 5"},{"comment":"The human quality study reports that 935/1200 puzzles were comparable or superior to seed puzzles, but no inter-annotator agreement or detailed annotation instructions are provided; please include them or cite an annotation protocol.","section":"Appendix D"},{"comment":"The statement that VisualSphinx-TEST is 'fully synthetic, ensuring no overlap with existing benchmarks' does not by itself rule out overlap with the training subset; please report the exact deduplication method (e.g., hash-based) used to separate the 930 test puzzles from the 10K RL examples.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a computer vision / multimodal reasoning venue and the dataset release is useful. My main concern is that the headline causal claim is not yet supported by the experimental design; the requested control conditions and uncertainty quantification are standard and should be feasible. I would not reject on the current evidence, but I cannot accept without the controls and a clearer verification of the training labels."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know about VisualSphinx is that the resource is real and the experiment is not the reason. The four-stage pipeline—abstracting rules from civil-service exam puzzles, expanding them with a rule-level genetic algorithm, rendering them via LLM-generated Python scripts, and assembling 660K puzzles—is a genuine new combination, and at a remarkably low cost. They've released the dataset, which makes it an immediate candidate for anyone doing VLM post-training or synthetic reasoning benchmarks. The in-domain gain (Qwen2.5-VL-7B from 29.3 to 55.9 on their test set) is large and monotonic, and the MathVista improvement (59.4 to 64.0) is modest but in the right direction. The manual check of the 930-item test set and the human quality annotation on 1,200 puzzles are also points in their favor.\n\nThe soft spots are real, but they are mostly about the abstract's causal language rather than the dataset's utility. There is no control RL run—no GRPO on shuffled labels, matched incoherent images, or seed-only data—and no error bars or multiple seeds. So the sentence claiming the model benefits from the dataset's 'logical coherence and readability' is not supported by any direct test. The MathVista transfer could be a generic RL or format effect. Also, the training labels are not human-verified; the automated filters (pHash, VLM coherence scores) help, but they don't guarantee the designated answer is uniquely correct. And the difficulty measure comes from a model trained on the same distribution, which can bias the difficulty split. That said, the paper is explicit about the mechanism being underexplored in its limitations, which I appreciate.\n\nFor a dataset paper, this is a solid contribution. The pipeline is novel, the release is public, and the cost is absurdly low. The right fix is to add a control condition or two, run multiple seeds, and soften the causal claims in the abstract. The resource deserves referee time.\n\nI'd send it to review, with the expectation of a revision that addresses the control issue. This is the kind of work that will be cited and used regardless; the peer review should make the empirical claims match the evidence.","headline":"A genuinely useful synthetic dataset for visual reasoning; treat the RL experiment as a demo, not evidence for the causal claim.","tokens_in":21485,"tokens_out":2734,"would_cite":true,"duration_ms":26067,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VisualSphinx shows that RL training on over 660,000 automatically generated visual logic puzzles, produced for under $1,000, sharply improves a vision-language model's logical reasoning and transfers to algebraic, arithmetic, and geometry…","keywords":["visual reasoning","synthetic dataset","vision-language models","reinforcement learning","GRPO","visual logic puzzles","rule-to-image synthesis","MathVista"],"falsifier":"Sample a few hundred puzzles from the 660K training set and have independent human solvers check that the designated answer is uniquely consistent with the stated rule; if a nontrivial share (say, more than a few percent) are ambiguous or wrong, the RL signal would be teaching artifact-matching rather than logic.","tokens_in":20477,"feed_emoji":"🧩","tokens_out":5505,"duration_ms":47221,"temperature":0.7,"pith_summary":"This paper tries to establish that large-scale synthetic visual logic puzzles are a viable and cheap way to train vision-language models to reason. The authors build VisualSphinx, a dataset of over 660K visual puzzles, each grounded in an explicit rule and assembled from program-generated images, and show that a 7-billion-parameter VLM trained with GRPO on a 10K-puzzle sample improves from 29.3% to 55.9% on a manually checked test set, beating several much larger closed models. It also reports transfer gains on MathVista, raising average accuracy from 59.4% to 64.0%, with improvements in algebraic, arithmetic, geometry, and logical reasoning. If this is right, it would make high-quality reasoning data for multimodal RL accessible at negligible cost and would suggest that structured synthetic puzzles can sharpen general visual reasoning.","feed_headline":"660K synthetic puzzles lift VLM logic accuracy by 27 points","feed_subtitle":"Low-cost generated training data improves visual reasoning and transfers to math tasks.","key_machinery":"The load-bearing mechanism is the rule-to-image synthesis pipeline: each puzzle is generated from an explicit five-bullet visual rule, a rule-level genetic algorithm diversifies the rules, and an LLM writes Python scripts (Matplotlib, PIL, or NumPy) that render five images obeying the rule and three images breaking it. The fifth compliant image is the designated answer and the three violators are distractors, so every puzzle carries a verifiable ground truth without human labeling. GRPO then turns that ground truth into a binary reward for RL training.","core_discovery":"The central discovery is that rule-grounded synthetic puzzles, not human-annotated data, can supply the verified reward signal that RL needs for multimodal reasoning. VisualSphinx starts from 2.4K seed rules abstracted from Chinese civil-service exam questions, expands them to 40K rules with a rule-level genetic algorithm, renders each rule into five compliant and three violating images via LLM-written Python scripts, and assembles the images into 660K four-option and ten-option puzzles. Training Qwen2.5-VL-7B with GRPO on 10K of these puzzles raises its accuracy on the VisualSphinx test set from 29.3% to 55.9% and produces transferable gains on MathVista, including logical, algebraic, arithmetic, and geometric reasoning. The authors interpret this as evidence that logical coherence and readability of the synthetic data are what drive the improvement.","pith_inferences":["Editorial inference: if the gains come from rule-grounded logical coherence rather than style, then adding harder rule types such as temporal sequences or interactive puzzles should extend the same transfer; this is a testable prediction.","Editorial inference: a direct way to check whether the model learned general logic or generator artifacts is to hold out entire rendering styles or genetic-rule lineages; if accuracy drops sharply on held-out styles, part of the gain is artifact-specific.","Editorial inference: the same rule-to-image pipeline could be adapted to other domains, such as physics diagrams, circuit diagrams, or spatial navigation, wherever executable code can render rule-compliant and rule-violating instances, providing unlimited reward-grounded training data."],"forward_implications":["A 7B open VLM trained on VisualSphinx can outperform much larger closed models on visual logic puzzles, suggesting model scale is not the only route to reasoning ability.","The improvements transfer to MathVista, where the trained model gains on logical, algebraic, arithmetic, and geometric reasoning, implying visual logic training can strengthen broader multimodal math reasoning.","Since the dataset costs under $1,000 to generate, this recipe makes multimodal RL training data affordable for groups without large annotation budgets.","The three rendering styles and answer-shuffling variants reduce position and style overfitting, so the model must attend to relational patterns rather than visual surface features.","The pipeline can scale to more rules and styles without additional human effort, so the 660K puzzle count is not an upper bound on the dataset."],"supporting_citations":[{"why":"Supplies the GRPO reinforcement learning algorithm used for training.","marker":"[33]"},{"why":"Supplies Qwen2.5-VL-7B, the base VLM that is fine-tuned and evaluated.","marker":"[36]"},{"why":"Supplies MathVista-testmini, the benchmark used to measure transfer to algebraic, arithmetic, geometry, and logical reasoning.","marker":"[21]"},{"why":"Motivates reinforcement learning for reasoning in LLMs, the training paradigm the paper applies to VLMs.","marker":"[7]"},{"why":"Provides the training framework used to run the GRPO experiments.","marker":"[34]"}],"fun_headline_variants":["Synthetic logic puzzles boost VLM reasoning by 27 points","Rule-to-image pipeline creates 660K puzzles for VLM training","VisualSphinx: RL for VLMs gains from synthetic visual logic","From 2.4K rules to 660K puzzles: a VLM reasoning leap","Generated puzzles lift VLM scores on visual and math tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The generated labels are correct: the LLM-written Python scripts really render five images that follow the intended rule and three that break it, even though only the 930-puzzle test set was checked by hand.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic logic puzzles boost VLM reasoning by 27 points","Rule-to-image pipeline creates 660K puzzles for VLM training","VisualSphinx: RL for VLMs gains from synthetic visual logic","From 2.4K rules to 660K puzzles: a VLM reasoning leap","Generated puzzles lift VLM scores on visual and math tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000227,"raw_usage":{"total_tokens":1432,"prompt_tokens":864,"completion_tokens":568,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":480,"completion_tokens_details":{"reasoning_tokens":475}},"tokens_in":480,"tokens_out":568,"duration_ms":5942,"temperature":1.0,"reasoning_tokens":475,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:38:17.598030+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sample a few hundred puzzles from the 660K training set and have independent human solvers check that the designated answer is uniquely consistent with the stated rule; if a nontrivial share (say, more than a few percent) are ambiguous or wrong, the RL signal would be teaching artifact-matching rather than logic.","supporting_citations":[],"review_version":1}