{"id":"212b3b50-432c-4da2-b40a-d4ea8d28d1ce","arxiv_id":"2411.14832","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new graph-image benchmark generator shows that six large vision-language models are sensitive to layout, labeling, and visual defects across seven graph tasks.","lead":"This paper introduces VisGraphVar, a generator that creates graph images for testing how well vision-language AI models understand graphs, and uses it to score six large models on seven tasks. The results suggest that visual choices like layout, labels, and overlapping nodes change model accuracy, so graph-based AI evaluations need to look beyond pure reasoning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Style effects are confounded with topology: §3.3 generates independent random graphs per style condition, so layout/label differences (e.g., Figure 15's spectral deficit) may be artifacts rather than true visual-style effects.","rationale":"The most load-bearing concern is internal validity: the paper asserts that visual style changes performance, but its experimental design does not isolate style from graph topology. Section 3.3 describes generating 10 images per style variation, with edges sampled by probability, and never states that the underlying graph is reused across style conditions. Hence every comparison between layouts, label conditions, or overlap conditions is potentially comparing different graph instances. The Figure 15 spectral-layout result is the clearest example: with n=10 and no paired controls, the 20-30 point gap could be random topology variation. This is not a disagreement with external consensus; it is a threat to the logical inference from data to the central claim. If paired controls were added and differences disappeared, the paper's headline finding would collapse. Other limitations—small sample size, no human baseline, no released code—are real but secondary; they would not invalidate a properly controlled finding. The reader's weakest-assumption analysis identifies exactly this confounding, so no verdict adjustment is needed beyond the existing CONDITIONAL decision. The concrete paired-image test would settle the concern definitively.","tokens_in":24068,"tokens_out":3238,"duration_ms":36773,"concrete_test":"Regenerate the Task 1 (and Task 6/7) datasets with a fixed set of base graphs: for each base graph, render every style variant (all seven layouts, labels on/off, directed/undirected, color schemes, and overlap levels) and evaluate at least Claude-3.5-Sonnet, GPT-4o, and Gemini-Pro-1.5 on the paired set. Compute per-condition mean scores with paired bootstrap confidence intervals. If the spectral-layout deficit in Table 2 persists under identical topology and exceeds noise, the style claim stands; if it shrinks or reverses, the original conclusion is an artifact of graph sampling.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is that visual attributes such as layout, labels, and overlap significantly affect LVLM performance. For that claim to be supported, comparisons across style conditions must hold graph topology fixed or otherwise control for it. Section 3.3 says Task 1 graphs all contain 10 nodes with edges 'determined by the probability percentage (2%)', and Tasks 6 and 7 similarly sample edges probabilistically, but it never states that the same graph instance is rendered under different styles. Each style condition therefore appears to be a fresh random graph sample. With only 10 images per condition and no error bars or significance tests, random differences in edge count, density, or path structure can produce the reported gaps. The striking spectral-layout deficit in Figure 15 and Table 2 (48.82% vs 67-79% for other layouts) may be caused by the particular graphs assigned to the spectral condition rather than by the layout itself. No code, dataset, or random seeds are provided, so the confounding cannot be checked post hoc. Unless the authors generate paired images with identical topology across style variants, the conclusion that 'variations in visual attributes significantly affect model performance' is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VisGraphVar, a Python-based generator for producing graph images with configurable visual styles (layout, labels, colors, arrows, overlaps) across seven graph-analysis tasks: detection, classification, segmentation, pattern recognition, link prediction, reasoning, and matching. Using a dataset of 990 generated images, the authors evaluate six LVLMs under zero-shot and chain-of-thought prompting, reporting average scores per model, task, and style condition. The central claim is that variations in visual attributes such as node labeling, layout, and visual imperfections like node overlap significantly affect LVLM performance, and that current LVLMs are not robust to representational variability in graph analysis. The paper also argues for evaluating LVLMs on a broader range of graph tasks beyond pure reasoning.","tokens_in":24244,"tokens_out":7024,"duration_ms":69707,"significance":"If the central claim is supported, VisGraphVar would be a useful benchmark generator for stress-testing LVLMs on visual graph interpretation, and the finding that style variations affect performance would have practical implications for evaluation methodology. The paper has strengths: it covers seven diverse tasks, evaluates six leading LVLMs, compares two prompting strategies, and the generator is designed to be customizable. The authors also provide a public website for supplementary results. However, the empirical evidence for the style-effects claim is currently weakened by a confound between visual style and graph topology, and by the absence of any statistical inference. The benchmark generator itself is a constructive contribution that could be valuable after the experiments are made rigorous.","major_comments":[{"comment":"The edge-generation probabilities are inconsistent with the displayed examples and with the reported image content. Section 3.3 states that Task 1 graphs have 10 nodes with edges determined by a 2% probability per pair, Task 6 uses a 3% probability, and Task 7 uses 4%. For 10 nodes, 2% yields an expected 0.9 edges, but Figure 16 shows a graph with 10 nodes and 20 edges, and Figure 19 shows graphs with 10 nodes and 16 edges; Appendix B.6 shows a multi-edge weighted graph for a 5–7-node Task 6 instance. The probability of observing these edge counts under the stated Bernoulli model is vanishingly small. Either the percentages are misreported or the generator uses a different procedure. This is load-bearing because the dataset's difficulty and the reproducibility of the generator depend on the actual edge densities, and the paper's parameters cannot be used to regenerate the images.","section":"§3.3, Figs. 16 and 19, App. B.6"},{"comment":"The comparisons across style conditions confound visual style with graph topology. Section 3.3 specifies that for each style variation 10 images are generated, and that Task 1 edge sets are obtained probabilistically for graphs with 10 nodes; it does not state that the same underlying graph instance is rendered under different layouts, label conditions, or arrow types. Therefore, the score differences attributed to style—for example, the spectral-layout deficit of 48.82% versus 67–79% in Figure 15 or the label-vs-unlabel gap in Figure 18—could be caused by random differences in edge count, density, or path structure among the graph samples assigned to each condition, rather than by the visual style itself. To support the claim that visual attributes significantly affect performance, the authors should generate paired images with identical topology across style variants, or otherwise statistically control for graph complexity (e.g., edge count, number of overlaps) in the analysis. Providing the generator code and random seeds would allow this confound to be checked.","section":"§3.3, Figs. 15 and 18, Table 2"},{"comment":"The paper reports point estimates over 10 images per style condition with no confidence intervals, significance tests, or effect sizes, yet the abstract and conclusions assert that visual variations 'significantly affect' model performance. For example, Figure 18 shows a roughly 9-point difference for GPT-4o between labeled and unlabeled graphs, and Figure 15 shows an 18-point spectral deficit for Claude-3.5-Sonnet, but with n=10 per condition these magnitudes may be within sampling noise, and no measure of variance is reported. The authors should report per-condition variability (e.g., standard deviation or confidence interval) and ideally a permutation or bootstrap test, or a human baseline, to support the causal language used in the central claim.","section":"§4.2, Figs. 11–18"}],"minor_comments":[{"comment":"The text introduces the metric as 'MEA' but the equation and subsequent text use 'MAE'; please make the abbreviation consistent.","section":"§3.4"},{"comment":"The list of Task 2 graph types is inconsistent: §3.2.2 lists 'acyclic, cyclic, bipartite, complete, meshs, planar, tree', whereas §3.3 says 'tree, planar, mesh, cyclic, complete bipartite, and acyclic' (six items, with 'complete bipartite' rather than separate 'complete' and 'bipartite'). Please reconcile the list and the category names.","section":"§3.2.2 and §3.3"},{"comment":"The discussion of overlapping nodes implies that partial credit should reward counting 9 of 10 nodes when 3 overlap, but the MAE metric does not implement such partial-credit semantics; it simply takes the absolute difference between the true and predicted counts. Please clarify how overlap is handled in scoring.","section":"§3.2.1"},{"comment":"The qualitative explanation for the spectral-layout deficit is based on a single example and a speculative link to the model's pattern-recognition strength; this should be framed as a hypothesis, not an explanation, especially given the topology confound noted above.","section":"§4.3.1"},{"comment":"There are several placeholders where 'LLVM' appears instead of 'LVLM' (e.g., 'if an LLVM fails to perform consistently'); please correct these typos.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The benchmark generator is a potentially useful contribution, and the multi-task, multi-model evaluation is substantial. However, the empirical conclusions are currently undermined by the topology-style confound and the incorrect edge-density parameters. I recommend requiring the authors to either rerun the experiments with paired graph instances across style conditions or provide a reanalysis controlling for graph complexity, and to correct the parameter reporting. If code and seeds are not made available, the confound cannot be resolved post hoc."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: VisGraphVar is a genuinely useful benchmark generator, but the paper's central empirical claim—that visual style significantly affects LVLM performance—is not backed by the experiments as designed. The generator itself deserves attention; the specific numbers should be taken as preliminary.\n\nWhat's new and good: seven task categories (detection through matching) with configurable layout, labels, colors, and deliberate imperfections like node overlap. Prior benchmarks (VisionGraph, GITA, VNA) focus on reasoning and use cleaner renderings. The task design is sensible, the prompts are documented, and the six-model evaluation is a solid amount of work. The qualitative examples are instructive, e.g., Pixtral failing matching for the wrong reason. The tool is the contribution, and it is reusable.\n\nThe soft spot is the confounding the stress-test notes. Section 3.3 says Task 1 graphs all have 10 nodes with edges sampled at 2% probability per pair—that means every image is an independent random graph. The same holds for Tasks 6 and 7. So when you compare layouts, labels, or overlap conditions, you are not comparing the same graph rendered differently; you are comparing different random graphs that happen to be in different style buckets. With only 10 images per condition and no error bars or significance tests, the reported gaps—like the spectral layout deficit in Figure 15—could easily be noise or topology differences. The paper's language (\"significantly affect\") overstates what the data can support.\n\nA second, smaller issue: no code, dataset, or seeds are provided. The website offers supplementary materials, but the generator itself is not shipped. For a benchmark paper, that's a gatekeeping problem; the \"reusable\" claim only holds if the code is actually available and usable.\n\nThe math and metrics are fine; MAE, accuracy, and Jaccard are reasonable choices. The citation pattern is fair. The Discussion honestly lists open questions, which is good.\n\nBottom line: This is a useful benchmark paper in need of a stronger empirical section. The authors should generate paired images that hold topology fixed across style variations, add error bars and a simple test (e.g., bootstrap or Wilcoxon), and tone down the causal language. If they do that, it becomes a solid contribution to the LVLM-evaluation literature.\n\nI'd send it to peer review with that expectation, but not desk-reject it. The generator fills a real gap. I'd cite it if I were working on LVLM graph benchmarks, but it's not something I'd use to draw conclusions about style effects today.","headline":"Useful benchmark generator, but the headline claim about style effects is confounded by independent random graphs per condition and tiny samples.","tokens_in":24797,"tokens_out":2642,"would_cite":false,"duration_ms":26538,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that changing a graph's visual appearance—node labels, layout, node overlap—shifts large vision-language model accuracy across seven graph-analysis tasks, so benchmarks must cover representational variability, not…","keywords":["large vision-language models","visual graph analysis","benchmark generator","graph layout variability","node overlap","chain-of-thought prompting","graph reasoning","representation robustness"],"falsifier":"Generate the same set of graph topologies, render each in multiple layouts, label settings, and overlap levels, and rerun the seven tasks; if performance gaps shrink or vanish, the reported style effects were artifacts of uncontrolled topology.","tokens_in":23844,"feed_emoji":"📊","tokens_out":5351,"duration_ms":53798,"temperature":0.7,"pith_summary":"This paper claims that how a graph is drawn changes what large vision-language models are able to see in it, and that this representational variability matters as much as the underlying reasoning task. To show this, the authors build VisGraphVar, a configurable generator that renders graph images across seven tasks—detection, classification, segmentation, pattern recognition, link prediction, reasoning, and matching—while varying node labels, colors, arrows, layouts, and deliberate imperfections such as overlapping nodes. Applying it to 990 images and six state-of-the-art models under zero-shot and chain-of-thought prompting, they find that stylistic variations and visual imperfections significantly shift accuracy. The intended upshot is that LVLM evaluation on graphs must cover multiple tasks and multiple visual styles rather than reasoning alone.","feed_headline":"Layout and labels alone shift AI graph-reading accuracy","feed_subtitle":"A new 990-image benchmark shows visual style, not just reasoning, decides how well vision-language models read graphs.","key_machinery":"The central object is VisGraphVar itself, a Python-based benchmark generator that parameterizes graph rendering along several visual axes—layout, node color, node labels, edge arrows, and node-edge overlap—and emits images for seven graph-analysis tasks. The generator makes style variation an independent variable: 560 detection images test style axes directly, while smaller task blocks for classification, segmentation, pattern recognition, link prediction, reasoning, and matching test whether style sensitivity persists once the task requires global structure or algorithmic reasoning. Scores are normalized per task with mean absolute error, accuracy with partial credit, and Jaccard similarity for shortest paths.","core_discovery":"VisGraphVar's central discovery is that visual style is not a neutral carrier of graph content for LVLMs. On the generated dataset, every tested model is sensitive to how nodes are labeled and arranged; spectral layouts depress detection performance for the strongest model, labeled graphs yield higher detection scores than unlabeled ones for all models, and overlapping nodes degrade counting and shortest-path answers. The paper argues that no model is robust across all seven tasks, that proprietary models currently lead open-weight ones on average, and that chain-of-thought prompting does not consistently beat zero-shot prompting. The authors' conclusion is that graph benchmarks must be judged by how well models cope with representational variability, not by reasoning accuracy alone.","pith_inferences":["An untested but direct consequence: reporting only mean accuracy over a dataset hides style fragility; minimum or variance across layouts would be a sharper robustness metric.","The labeled-versus-unlabeled gap points to a concrete experiment: fine-tune a model on unlabeled graph images and re-run detection; if the gap closes, the deficit is training-distribution bias rather than vision.","Because the detection graphs are independently generated per style condition, a paired regeneration holding topology fixed would separate style effects from difficulty effects; this is a testable extension of the benchmark design.","A task-specific model-routing scheme—choosing the best model per task instead of one global model—would likely lift average performance immediately, given the large per-task reversals reported."],"forward_implications":["LVLM evaluations that use only one graph rendering style overstate a model's ability to analyze graphs, because layout and labeling alone can shift scores.","A capable model for real graph images must handle overlapping nodes and imperfect drawings, not just clean synthetic layouts.","Reasoning-heavy evaluation is insufficient: foundational tasks such as node and edge detection and matching show large gaps and should be tested first.","Prompt strategy is not a reliable lever: zero-shot and chain-of-thought produce similar results overall, so gains must come from model or data changes.","No single tested model dominates all seven tasks, so deploying LVLMs for graph analysis should be task-specific rather than based on aggregate ranking."],"supporting_citations":[{"why":"Supplies the seven-dimension task taxonomy that shapes VisGraphVar's task design.","marker":"[6]"},{"why":"Prior visual graph reasoning benchmark whose single-domain focus VisGraphVar extends with broader tasks and style variation.","marker":"[27]"},{"why":"Existing end-to-end visual graph reasoning framework used as the comparison point for why reasoning-only benchmarks are insufficient.","marker":"[53]"},{"why":"Benchmark showing multimodal models struggle with basic visual network analysis, motivating the style-variability approach.","marker":"[54]"},{"why":"Chain-of-thought prompting strategy that forms one of the two prompt conditions compared in the evaluation.","marker":"[52]"},{"why":"Source of the accuracy metric and the dynamic visual benchmark idea that VisGraphVar adapts to graph tasks.","marker":"[64]"}],"fun_headline_variants":["Visual style, not reasoning, decides AI graph-reading success","New benchmark: layout, labels, and overlaps derail AI graph analysis","990 graph images show vision-language models are style-sensitive","Visual variations, not reasoning depth, decide LVLM graph accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that images from different style conditions are comparable: graph topology is not held fixed across layouts, labels, or overlap levels, so score differences could reflect independent random graph difficulty instead of visual style.","fun_headline_variants_meta":{"raw":{"variants":["Visual style, not reasoning, decides AI graph-reading success","New benchmark: layout, labels, and overlaps derail AI graph analysis","990 graph images show vision-language models are style-sensitive","Visual variations, not reasoning depth, decide LVLM graph accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000595,"raw_usage":{"total_tokens":2778,"prompt_tokens":929,"completion_tokens":1849,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":1779}},"tokens_in":545,"tokens_out":1849,"duration_ms":13255,"temperature":1.0,"reasoning_tokens":1779,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:49:24.417361+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate the same set of graph topologies, render each in multiple layouts, label settings, and overlap levels, and rerun the seven tasks; if performance gaps shrink or vanish, the reported style effects were artifacts of uncontrolled topology.","supporting_citations":[{"cited_title":"A First Course in Graph Theory","cited_arxiv_id":null,"evidence_quote":"Supplies the seven-dimension task taxonomy that shapes VisGraphVar's task design."},{"cited_title":"Multimodal LLMs struggle with basic visual network analysis: A VNA benchmark","cited_arxiv_id":null,"evidence_quote":"Benchmark showing multimodal models struggle with basic visual network analysis, motivating the style-variability approach."},{"cited_title":"DynaMath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models.arXiv [cs.CV], October 2024","cited_arxiv_id":null,"evidence_quote":"Source of the accuracy metric and the dynamic visual benchmark idea that VisGraphVar adapts to graph tasks."}],"review_version":1}