{"id":"f1456a80-0976-4664-b57e-f31f30979b8e","arxiv_id":"2505.03678","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Graph layout and drawing quality measurably affect how well vision-capable LLMs answer graph queries, with orthogonal drawings helping path tasks and straight-line drawings helping global tasks.","lead":"This paper tests whether the visual layout of a graph, such as straight-line versus orthogonal drawings, changes how well large language models answer graph questions. It finds that layout choice, prompt style, and drawing readability all affect accuracy, so how a graph is drawn matters for AI assistants that read images.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Experiment 3's 'improved' drawings are manually adjusted with no before/after readability metrics, so the R3 claim that human readability drives LLM accuracy is not established.","rationale":"The paper's contribution rests on three empirical claims; the strongest headline claim in the abstract is that readability optimization 'can significantly improve performance.' The R1 layout-paradigm claim and R2 prompting claim are at least operationalized by systematic algorithm choices and prompt templates. R3, by contrast, uses a subjective manual redrawing as the sole intervention, with no measurement of the readability constructs it is supposed to isolate. This is exactly the gap the Reader flagged as weakest. I agree with that identification rather than elevating the absence of statistical tests or the sigma-notation ambiguity in the accuracy formulas: those are important but secondary, and the sigma issue is likely a typo (the stated 0.5 examples are consistent with sigma as a count rather than a fraction). The proposed check—quantifying readability metrics before/after and holding confounds fixed—would settle whether R3 actually demonstrates what it claims. Until that check is done, the paper should not be treated as establishing R3; the conditional verdict is appropriate, and no verdict change is needed relative to the Reader.","tokens_in":15165,"tokens_out":7385,"duration_ms":72087,"concrete_test":"Make the 28 SlV/I-SlV image pairs available and compute for both versions per graph the standard readability metrics named in the paper: number of edge crossings, crossing-angle distribution, symmetry (reflectional and rotational), minimum vertex/edge resolution, and label-overlap/legibility. Report before/after values and correlate per-graph metric deltas with per-graph accuracy deltas in Table 9. Then run a controlled ablation on a subset: e.g., apply an automated planarization or crossing-minimization that keeps node positions and label style unchanged, and compare accuracy with the manual I-SlV set. If metric deltas do not consistently favor I-SlV, or if accuracy gains vanish when label legibility and scale are held fixed, the R3 claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise for R3 (Section 3.3.2: 'improving the readability of a graph drawing based on human readability metrics increases the LLM's ability') is that the I-SlV drawings actually are better on those metrics. Section 3.3.1 says only that they come from 'manually adjusting' force-directed drawings 'based on human experience and well-accepted metrics such as symmetry and number of edge crossings.' No crossing count, symmetry score, angular-resolution value, or label-legibility measure is reported for either SlV or I-SlV for the 28 Bench-4 graphs. The headline numbers (I-SlV 0.58 vs SlV 0.52; I-SlM 0.85 vs SlM 0.82) could therefore be driven by incidental changes to label placement, font size, node spacing, whitespace, aspect ratio, or simply redrawing luck. The effect is also small and not consistently positive within Table 9: GPT-4o I-SlM Std-Zero is 0.82 vs 0.84 for SlM, and Claude I-SlM Std-Few is 0.61 vs 0.51 for SlM. Section 4.2 itself defers a 'more systematic study of what readability metrics have a greater impact,' acknowledging that no metric was isolated. Without a quantified before/after comparison and a controlled manipulation, the causal claim in R3 is untestable from the data as reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports an empirical study of two multimodal LLMs (GPT-4o and Claude-3.7-Sonnet) on graph reasoning tasks where the input is an image of a graph drawing. It compares text, visual, and mixed modalities; straight-line versus orthogonal layout paradigms; standard, chain-of-thought, few-shot, and a newly proposed 'Spell-out Adjacency List' (SoAL) prompting technique; and force-directed versus manually 'improved' drawings. The paper's central claims are that the layout paradigm affects LLM accuracy (R1), that prompting technique choice matters with no universally best option (R2), and that improving the human readability of a drawing improves LLM accuracy (R3). The main experimental results are reported through task-specific accuracy metrics designed to be robust to hallucinated edges or vertices.","tokens_in":15442,"tokens_out":6410,"duration_ms":61284,"significance":"The research questions are timely and directly relevant to both the graph drawing and the LLM evaluation communities. The design of hallucination-aware accuracy metrics is a genuine contribution, and the use of two frontier LLMs across a substantial matrix of conditions is a strength. If the claims about layout paradigm and drawing readability are established, the results would provide practical guidance for building visual graph assistants. However, at present the evidence is weakened by an internal inconsistency in the accuracy metric definitions, by the lack of quantified readability controls in the drawing 'improvement' experiment, and by the absence of statistical inference for the small observed differences. The paper also promises public data and code but the footnotes defer their release, which should be resolved.","major_comments":[{"comment":"The formulas for the hallucination-aware accuracy metrics are inconsistent with the prose definitions of σ. For ShPa, σ is defined as 'the fraction of existing edges in the path outputted by the model,' but the second factor is min{σ/Δ, Δ/σ}; for an output of correct length Δ with half of the edges existing (σ = 0.5), the formula gives min{0.5/Δ, 2Δ}, not 0.5 unless Δ = 1. For MaxC, the factor 2σ/(Δ(Δ−1)) equals 1 when σ = 1 only for Δ = 2, contradicting the claim that a correct clique yields αMaxC = 1. For MinVC, the formula uses (1 − σ/m), which is not 1 − σ if σ is a fraction. If σ is instead intended to be a count of existing edges or uncovered edges, then the prose definitions are wrong. Since every reported accuracy in Tables 1–9 is computed with these formulas, the definitions must be corrected and the tables checked against the intended metric. The examples given in the text only match the count interpretation, so this is a fixable but load-bearing inconsistency.","section":"Section 3.3"},{"comment":"The key finding for R3, that improving human readability metrics increases LLM accuracy, is not established by the data as reported. Section 3.3.1 states only that I-SlV drawings are obtained by 'manually adjusting' force-directed drawings 'based on human experience and well-accepted metrics such as symmetry and number of edge crossings.' No before-and-after values are reported for symmetry, edge crossings, angular resolution, label legibility, or any other readability metric for the 28 Bench-4 graphs. The observed aggregate improvement (I-SlV 0.58 vs SlV 0.52; I-SlM 0.85 vs SlM 0.82) could therefore be driven by incidental changes in scale, whitespace, label placement, or drawing-specific luck. This concern is amplified by Table 9, where some cells move in the opposite direction, e.g., GPT-4o I-SlM Std-Zero is 0.82 vs 0.84 for SlM. Section 4.2 itself defers a 'more systematic study of what readability metrics have a greater impact,' acknowledging that no metric was isolated. Without a quantified readability comparison or a controlled manipulation of individual metrics, the causal claim in R3 is untestable from the current data.","section":"Section 3.3"},{"comment":"The R1 claim that orthogonal drawings are superior for local/path tasks is based on small aggregate differences without confidence intervals or significance tests. The averaged differences are minute (OrM 0.88 vs SlM 0.87; OrV 0.69 vs SlV 0.67), and several cells in the underlying tables contradict the aggregate direction; for example, GPT-4o on CoNe with Std-Zero gives OrV 0.54 versus SlV 0.60, and Claude-3.7-Sonnet on CoNe with Std-Few gives OrV 0.35 versus SlV 0.35 as a tie. The paper says the pattern is 'confirmed' and 'consistent,' but with 20 graphs per benchmark, no error bars, and no per-condition statistical testing, small differences cannot be distinguished from noise. The authors should report per-graph variability, confidence intervals, or an appropriate statistical test, and should temper the strength of the R1 conclusion accordingly.","section":"Section 3.1.2"}],"minor_comments":[{"comment":"The text states that 'All experimental data (including benchmarks, drawings, code, and full prompts) are publicly available,' but footnotes 1 and 6 say 'To be provided after publication or under request.' These statements are contradictory and should be reconciled before publication.","section":"Throughout"},{"comment":"There are several typos in the modality definitions: 'Straght-line Visual' and 'Orhtogonal' should be 'Straight-line' and 'Orthogonal', and in Section 3.1.1 the phrase 'This benchmark hae been used' should read 'has been used.'","section":"Section 3.1.1"},{"comment":"Reference [9] contains the stray text 'Cited by: 19,' which appears to be an artifact of a citation manager and should be removed.","section":"References"},{"comment":"In the discussion of Experiment 2, the phrase 'follow by SoAL' should be 'followed by SoAL' for grammatical correctness.","section":"Section 3.2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important and timely question, and the overall experimental design is reasonable. However, the accuracy metric formulas contain an internal inconsistency that affects the interpretation of every quantitative result, and the R3 experiment lacks the quantified readability measures needed to support its causal claim. I believe both issues are fixable within the scope of a revision, so I recommend major revision rather than rejection. Please also ensure that the promised data and code are actually made available, as the current footnotes contradict the paper's availability statement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful paper for anyone building or evaluating multimodal graph assistants. It does something prior work didn't: pit straight-line against orthogonal drawings head-to-head for LLM graph queries, on four structural tasks, with two frontier models, and it introduces SoAL, a prompting trick that has the model spell out the adjacency list it thinks it sees. The hallucination-aware accuracy measures (Jaccard for common neighbors, edge-fraction weightings for paths, cliques, covers) are a real methodological upgrade over vanilla accuracy, and the token/cost data is a nice practical add. The R1 finding — orthogonal helps on local/path tasks, straight-line slightly better on global tasks — is plausible and mostly consistent across models, though the effect sizes are small and there are no error bars or significance tests.\n\nThe soft spots are real, but none of them kills the paper at the review stage. First, the public data/code/prompts promised in Section 3 are not actually public — footnotes say 'to be provided after publication or under request.' That is not acceptable for an empirical paper making new measurements; reviewers should demand the artifacts. Second, the benchmarks are tiny: 20 graphs per bench, 28 in Bench-4. Third, and most importantly, the R3 claim about human readability metrics is underdetermined. Experiment 3 manually redraws the SlV drawings 'based on human experience and well-accepted metrics,' but no before/after crossing counts, symmetry scores, or angular resolutions are reported. The stress-test note is right: the accuracy gain (0.58 vs 0.52; 0.85 vs 0.82) could come from label legibility, whitespace, or redrawing luck. Worse, Table 9 contains reversals (GPT-4o I-SlM Std-Zero 0.82 vs 0.84 for SlM; Claude I-SlM Std-Few 0.61 vs 0.51 for SlM), so the R3 headline is fragile. The authors themselves defer a 'more systematic study' in Section 4.2, which is honest but confirms the point.\n\nWho is this for? Researchers in graph drawing, visualization, and LLM reasoning. It deserves a serious referee — conditional acceptance with requests for significance testing, quantified readability deltas, and the missing artifacts. My own verdict would be skeptical-but-constructive: R1 likely survives tightening; R3 needs more work.","headline":"Solid empirical start on layout-as-lever for LLM graph reasoning, but the headline R3 claim rests on an unquantified manual redrawing step, and the promised artifacts aren't public yet.","tokens_in":15952,"tokens_out":2436,"would_cite":true,"duration_ms":19993,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The layout paradigm used to draw a graph measurably changes how accurately a multimodal LLM can answer graph questions from the image, with orthogonal drawings best for path-following tasks and human-readable redrawings boosting accuracy.","keywords":["graph drawing","large language models","visual graph reasoning","orthogonal layouts","prompt engineering","graph readability metrics","hallucination-aware accuracy","visual modality"],"falsifier":"Re-run Experiment 3 with automated layout optimization, measuring crossing number and symmetry before and after each change; if accuracy does not rise when the metrics improve, or if gains appear on drawings with unchanged metrics, the R3 claim is falsified.","tokens_in":14956,"feed_emoji":"📊","tokens_out":5832,"duration_ms":56040,"temperature":0.7,"pith_summary":"This paper investigates whether the way a graph is drawn, not just the graph data itself, changes how well multimodal LLMs can answer questions about it. Across two frontier models and four structural tasks, the authors compare text-only input, images of straight-line versus orthogonal drawings, and mixed text-plus-image input. They find that layout paradigm matters: orthogonal drawings give better accuracy on tasks that require tracing local connections or paths, while force-directed straight-line drawings do better on more global, complex tasks. They also find that improving the drawing according to human readability criteria such as symmetry and fewer edge crossings improves accuracy, and that no single prompting technique dominates. The work matters because general-purpose AI assistants often see graphs only as images, so layout and prompting become practical levers for their reliability.","feed_headline":"How a graph is drawn changes LLM accuracy","feed_subtitle":"Orthogonal layouts win on path-following tasks, and cleaner drawings lift accuracy on both tested models.","key_machinery":"The carrying object is an experimental framework pairing three input modalities, textual adjacency list, visual drawing, and mixed, with four graph tasks (common neighbor, shortest path, maximum clique, minimum vertex cover), each scored by a hallucination-aware accuracy formula rather than by raw correctness. The visual modality splits into straight-line drawings from the FMMM force-directed algorithm and orthogonal drawings whose edges are chains of horizontal and vertical segments. The prompting techniques are standard, chain-of-thought, and the new SoAL, each with zero-shot and few-shot variants. In a third experiment, straight-line drawings are manually redrawn to improve symmetry and reduce crossings, producing an 'improved' visual modality. These components work together to isolate the effect of layout, prompt, and drawing quality on model accuracy.","core_discovery":"The central empirical claim is that layout paradigm changes an LLM's ability to answer graph queries from an image. The authors state this directly: orthogonal drawings appear superior on tasks where following local connections or paths matters, while straight-line drawings produced by force-directed algorithms support global tasks such as maximum clique or minimum vertex cover. A second load-bearing claim is that improving a drawing's readability by human metrics, namely symmetry and number of edge crossings, raises accuracy on shortest-path questions with both tested models. A third finding is that no prompting technique wins outright: chain-of-thought is generally accurate but costly, and the newly introduced Spell-out Adjacency List (SoAL) prompt, which asks the model to extract the adjacency list from the image before answering, is promising but model-dependent. The paper also introduces hallucination-aware accuracy metrics that weigh whether the nodes, edges, or paths the model names actually exist in the graph.","pith_inferences":["If human readability metrics transfer to machine readers, then graph layout algorithms could be tuned directly against an LLM's accuracy, turning the model into a proxy for readability experiments that currently require human subjects.","The local-versus-global task split suggests a general design principle: the right visual encoding depends on the query type, and other encodings such as edge colors, shapes, bundled edges, and directed arrows could be mapped onto the same split.","A testable extension would automate the manual redrawing in Experiment 3 by optimizing an explicit crossing and symmetry objective; if accuracy rises monotonically with the measured metric, the causal claim in R3 would be much stronger.","The visual-only gap over text will likely shrink as vision encoders improve, but the layout effect may persist and could even grow on larger graphs where text input becomes impractical."],"forward_implications":["An AI assistant that receives a graph image should have the layout chosen by task: orthogonal layouts for path-tracing and local queries, straight-line layouts for global structural questions.","Feeding the model an adjacency list alongside the image generally outperforms either modality alone, and visual-only input is the weakest of the three.","Drawings optimized for human readability also read better to current LLMs, with no additional token cost, since the accuracy gain in Experiment 3 comes with equal latency.","Prompt selection is task- and model-specific: chain-of-thought is a strong default, while SoAL matches it on some models, especially on complex tasks, and costs only slightly more tokens.","Hallucination-aware scoring changes the ranking of answers by penalizing nonexistent edges, so future benchmarks on visual graph reasoning should report structural validity, not just numeric agreement."],"supporting_citations":[{"why":"Establishes that image input can be a strong modality for graph tasks and that performance correlates with human readability; this is the baseline the paper extends.","marker":"[7]"},{"why":"Compares node-link and matrix representations, motivating task-based comparison of layout paradigms.","marker":"[23]"},{"why":"Earlier user study contrasting node-link and matrix readability, grounding the task-driven evaluation of drawings.","marker":"[14]"},{"why":"Describes the FMMM force-directed algorithm used to compute the straight-line drawings in the visual modality.","marker":"[16, 17]"},{"why":"Defines chain-of-thought prompting, one of the prompting techniques the paper benchmarks.","marker":"[32]"},{"why":"Supplies the human readability metrics, symmetry and edge crossings, used to justify the manual drawing improvements in Experiment 3.","marker":"[24, 25]"},{"why":"House of Graphs is used to generate benchmarks with controlled clique and vertex-cover sizes.","marker":"[6]"},{"why":"Source of the four graph tasks and the accuracy-style evaluation used in the experiments.","marker":"[29]"},{"why":"Standard references for orthogonal drawing algorithms, defining the layout paradigm contrasted with straight-line drawings.","marker":"[2, 11, 21]"}],"fun_headline_variants":["Graph layout decides LLM accuracy on visual queries","Cleaner drawings improve LLM graph reasoning","Orthogonal layouts win for LLM path-following","LLM graph IQ depends on how you draw the graph"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise behind the readability result is that the manually redrawn images in Experiment 3 are genuinely better on symmetry and crossing counts; no before-and-after metric values are reported, so the accuracy gain is not cleanly tied to those factors.","fun_headline_variants_meta":{"raw":{"variants":["Graph layout decides LLM accuracy on visual queries","Cleaner drawings improve LLM graph reasoning","Orthogonal layouts win for LLM path-following","LLM graph IQ depends on how you draw the graph"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000348,"raw_usage":{"total_tokens":1848,"prompt_tokens":837,"completion_tokens":1011,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":453,"completion_tokens_details":{"reasoning_tokens":949}},"tokens_in":453,"tokens_out":1011,"duration_ms":9863,"temperature":1.0,"reasoning_tokens":949,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:44:51.576634+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run Experiment 3 with automated layout optimization, measuring crossing number and symmetry before and after each change; if accuracy does not rise when the metrics improve, or if gains appear on drawings with unchanged metrics, the R3 claim is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that image input can be a strong modality for graph tasks and that performance correlates with human readability; this is the baseline the paper extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Compares node-link and matrix representations, motivating task-based comparison of layout paradigms."},{"cited_title":"Ghoniem, J","cited_arxiv_id":null,"evidence_quote":"Earlier user study contrasting node-link and matrix readability, grounding the task-driven evaluation of drawings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines chain-of-thought prompting, one of the prompting techniques the paper benchmarks."},{"cited_title":"Coolsaet, S","cited_arxiv_id":null,"evidence_quote":"House of Graphs is used to generate benchmarks with controlled clique and vertex-cover sizes."}],"review_version":1}