{"id":"06f1782e-1233-4f49-b874-83d8516b5a1c","arxiv_id":"2412.11088","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"On a newly generated benchmark, multimodal models solve up to 87.6% of visual tree problems and 56.2% of visual graph problems, undercutting the idea that diagrams make exam questions AI-proof.","lead":"Eight large multimodal AI models were tested on a new benchmark of 9,072 image-based graph and tree problems, with GPT-4o reaching 87.6% accuracy on tree tasks and Gemini 1.5 Flash reaching 56.2% on graph tasks. The work quantifies how quickly visual data structure problems, often proposed as AI-resistant exam formats, are becoming solvable by off-the-shelf AI.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Accuracy claims rest on an unaudited regex/ground-truth pipeline; a small parse or label mismatch could shift the Table 1 point estimates.","rationale":"The paper's central contribution is an empirical measurement: with a simple zero-shot prompt, GPT-4o reaches 87.6% pass@3 on trees and Gemini 1.5 Flash reaches 56.2% pass@3 on graphs. The reader's conditional verdict is appropriate because the measurement pipeline is plausible but unvalidated. I agree with the reader's weakest-assumption analysis and make it more specific: the exact pass@k values depend on two unverified links. First, each ground-truth string must be entailed by the rendered image; for tree traversals this requires a definite root and left/right child relation, which the generator's networkx edge list does not itself encode. Second, the regex-based scorer must neither discard correct answers nor accept incorrect substrings. The paper reports neither parse-failure rates nor a human spot-check, so these links are asserted rather than demonstrated. The open-source generator and 72,576 collected responses are real evidence in favor of reproducibility, and the Limitations section appropriately notes coverage and prompting constraints, but it does not address scorer validation. The proposed audit is cheap and would settle the concern: if independent re-derivation of ground truth and parse outcomes show high agreement, the conditional verdict can be upgraded; if they reveal even a few percentage points of mismatch, the exact headline numbers and model ordering would need revision.","tokens_in":19545,"tokens_out":6468,"duration_ms":65143,"concrete_test":"Use the open-source generator to recreate a random subset of at least 200 samples (50 per structure). For each rendered PNG, independently reconstruct the structure from the image: detect node positions and labels, infer left/right child relations from drawing geometry for trees and arrowhead directions for directed graphs, then compute the traversal or adjacency list a human would accept. Compare to the stored A_true. Separately, take a random sample of model responses that did not match the regex and have two annotators judge whether the response contains the correct answer; report the parse-failure rate and recompute Table 1 with parse failures counted both as incorrect and as human-parsed. If both comparisons show agreement above 95%, the Table 1 accuracies are stable; if not, corrected numbers should be reported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline accuracies in Table 1 are produced by a binary comparison of a regex-extracted model answer to a ground-truth string, but the paper never validates that either side of that comparison is correct. Section 3.4 says 'cleaning and parsing were done' without reporting how many responses failed to parse or what cleaning was applied. Section 3.1.3 defines A_true as 'the accurate solution to Q_i with respect to I_i', but gives no independent check that the rendered PNG entails the stored answer. For tree traversals this is nontrivial: an in/pre/post-order is only defined once a root and left/right child relation are fixed, and the paper does not show that the matplotlib layout used to draw the image and the generator's internal tree structure agree on that relation. If the drawing is ambiguous or the stored answer mismatches the visible orientation for even a few percent of samples, the 87.6% tree number is not a clean measurement of visual graph solving. The same uncertainty applies to arrowhead perception on directed graphs and to regex extraction from verbose responses: a correct-but-unparseable answer is scored wrong, while a parseable substring can be scored right. The absence of error bars also weakens small model-to-model gaps, though the broad qualitative conclusion is less sensitive to this concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a benchmark of 9,072 programmatically generated graph and tree problems, evaluates eight large multimodal models (GPT-4o, GPT-4V, Gemini 1.5 Pro, Gemini 1.5 Flash, Gemini 1.0 Pro Vision, and Claude 3 Opus/Sonnet/Haiku) under zero-shot prompting, and reports pass@1 and pass@3 accuracies. The headline results are GPT-4o at 87.6% pass@3 on tree problems and Gemini 1.5 Flash at 56.2% pass@3 on graph problems. The authors also fit logistic regression models to engineered structural, aesthetic, and image features to answer questions about which variations influence accuracy, and they release the generator and dataset under an MIT license. The paper's central claim is that off-the-shelf LMMs can solve a substantial share of visually presented graph and tree exam questions, which has implications for assessment design in computing education.","tokens_in":19747,"tokens_out":5414,"duration_ms":45013,"significance":"If the reported measurements are valid, this is a useful and timely contribution to computing-education research. The open-source generator and benchmark address data-leakage concerns by creating novel items, and the systematic variation of node counts, edge widths, colors, layouts, and task types enables a more fine-grained analysis than prior visual Parsons-problem work. The paper explicitly positions image-based graph and tree questions as an assessment strategy and provides evidence that this strategy is already fragile. The inclusion of multiple model families and explicit zero-shot prompting is a practical strength. However, the significance of the headline accuracy numbers and the feature-importance findings depends on the validity of the rendering-to-ground-truth alignment, the answer extraction pipeline, and the statistical treatment of the point estimates, all of which are currently under-specified.","major_comments":[{"comment":"The paper reports pass@1/pass@3 accuracies without reporting how many model responses failed the regular-expression extraction or what cleaning steps were applied. A correct answer that is not parseable is scored as wrong, while a parseable substring of an incorrect answer can be scored right, so the Table 1 point estimates are not a clean measurement of visual problem-solving until the parse-failure rate and a robustness check (e.g., alternative extraction or manual review of a sample) are reported per model.","section":"Section 3.4, Table 1"},{"comment":"The ground-truth answers A_true are generated by the same program that renders the images, but the paper provides no independent check that the stored answer is entailed by the visible rendering. For tree traversals, in/pre/post-order depend on the root and the left/right child relation, which are properties of the internal tree, not necessarily of the matplotlib layout used to draw the image. Without a human spot-check of rendered images against A_true, or an audit of layout consistency, the headline tree accuracies (e.g., GPT-4o at 87.6% pass@3) may be biased by ambiguous or mismatched renderings.","section":"Section 3.1.3, Figure 2"},{"comment":"The definition of pass@3 is not specified: the paper does not state whether three independent samples are drawn per item at temperature 1.0, how per-item aggregation is computed, or how ties are broken. Furthermore, no confidence intervals or significance tests are reported, so interpretations of small differences in Table 1 are unsupported. A cluster-bootstrap confidence interval over items, or an equivalent, is needed for the comparative claims in Section 4.1.","section":"Section 3.4, pass@3 definition"},{"comment":"The graph density formulas are printed incorrectly: directed density should be E/(V(V-1)) and undirected density should be 2E/(V(V-1)), but the manuscript writes edges in the denominator, making the ratio depend only on node count. If the implemented feature used these formulas, the density feature in the logistic regression is not density, and the RQ2 finding in Section 4.2.4 ('positive influence at lower and negative at higher densities') is not about density as defined. The formulas must be corrected and the feature analysis re-run or shown to be unaffected.","section":"Appendix C.1.6"},{"comment":"The discussion claims that models achieve performance 'sufficient to pass traditional assessments,' but no human or student baseline and no passing threshold is provided. The 87.6% tree accuracy is hard to calibrate without knowing expected human performance on the same generated items; a small human study or a comparison to instructor-produced solutions is needed to support this pedagogical conclusion.","section":"Section 5.1"}],"minor_comments":[{"comment":"The sentence 'The dataset is divided equally between graph (|D_G| = 4536) and tree (|D_G| = 4536)' uses the same symbol twice; the second should be |D_T| = 4536.","section":"Section 3.1.3"},{"comment":"The statement that 'Claude 3 Opus outperformed other models on directed graphs' contradicts Table 1, where Gemini 1.5 Flash and Gemini 1.5 Pro achieve much higher directed-graph accuracies; the sentence likely meant that Claude 3 Opus performed better on directed graphs than on undirected graphs.","section":"Section 4.4"},{"comment":"The sentence 'Preemptive sample evaluations on Gemini 1.0 Pro Vision show minimal differences in accuracy compared to the six prompts tested' does not specify what the six prompts were, the sample size used, or the magnitude of the differences; please include these details for reproducibility.","section":"Section 3.2"},{"comment":"Please provide exact model API identifiers and snapshot dates (e.g., gpt-4o-2024-05-13) in addition to the evaluation dates, since model versions can affect results.","section":"Section 3.3"},{"comment":"The ACM reference format line has placeholder text 'In ,.' and '14 pages'; this should be completed with the actual proceedings name and location before the camera-ready version.","section":"ACM reference block"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and practically important question, and the open-source resource is valuable. My main concern is not with the overall direction but with the measurement pipeline: the parse-failure rate, the rendering-to-ground-truth consistency, and the absence of error bars all bear directly on the headline numbers. These are fixable with additional analysis, so I do not see a need for rejection, but the authors should be asked to provide the missing validation before the results are taken as established. The density formula typo also needs correction and a re-check of the feature analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful benchmark paper with a real result — current LMMs can solve a large share of image-based graph and tree questions, especially trees — but the measurement pipeline has undocumented spots, so treat the exact percentages as approximations rather than precise measurements. I largely agree with the reader's conditional verdict.\n\nWhat's new: a 9,072-sample benchmark of programmatically generated visual data structure tasks, plus an open-source generator. Both are concrete, reusable contributions. Table 1 is a straightforward head-to-head and the main finding — GPT-4o at 87.6% pass@3 on trees, graph performance much lower across all models — is plausible and worth knowing. The feature importance analysis is descriptive, and they don't oversell it.\n\nSoft spots, in order of impact:\n\n1. The parsing/ground-truth pipeline is unaudited. The paper says cleaning and parsing were done but doesn't report how many responses failed to parse, and there's no independent check that the rendered PNG matches the stored answer. For tree traversals, left/right orientation matters, and the layout is generated by matplotlib without a documented guarantee of internal tree consistency. A few percent of ambiguous cases could nudge the headline numbers. This doesn't overturn the main qualitative claim, but it matters for precise model ranking.\n\n2. No error bars or significance tests. Some close calls, like GPT-4o vs GPT-4V on tree pass@1 at 77.8 vs 77.8, are reported without uncertainty, which makes small model-to-model differences untrustworthy.\n\n3. Minor: the density formulas in Appendix C.1.6 are wrong — they use 'edges' in the denominator instead of the maximum possible edges. Presumably a typo, but should be fixed. A human baseline is absent, but that's acceptable for a capability measurement; the pedagogical implication would be stronger with one.\n\nThe reader's circularity burden of 1.0 is right: ground truth is generated independently of model outputs, and the logistic regressions are post-hoc descriptive, so there's no circular argument here. The stress-test note correctly points out the missing parse-failure analysis, but the broad conclusion is not sensitive to it.\n\nWho this is for: computing-education researchers, assessment designers, and anyone tracking multimodal models' spatial reasoning. It deserves a serious referee — the benchmark and generator are worth engaging even if the top-line accuracies shift slightly after the pipeline is audited. Recommend sending to review with a request for parse-failure statistics, error bars, and a check on tree layout orientation.","headline":"Useful benchmark and a credible 'LMMs already solve many visual tree problems' result, but exact accuracies need a parsing audit and error bars.","tokens_in":20334,"tokens_out":3050,"would_cite":true,"duration_ms":25247,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that off-the-shelf multimodal models, given only a rendered image and a simple zero-shot prompt, correctly solve most tree and many graph data-structure exam problems, led by GPT-4o at 87.6% on trees and Gemini 1.5 Flash…","keywords":["large multimodal models","visual graph problems","tree traversal","benchmark dataset","zero-shot prompting","computing education","academic integrity","assessment integrity"],"falsifier":"Human annotators should re-derive the expected traversal or adjacency list for a random sample of the benchmark images; if even a small share of images produces disagreement with the generator's ground truth, or if the regex extraction discards answers a human would call correct, the measured accuracies are not a clean measure of visual problem-solving.","tokens_in":19322,"feed_emoji":"🌳","tokens_out":7074,"duration_ms":62061,"temperature":0.7,"pith_summary":"Educators have leaned on image-based graph and tree problems as a way to keep AI assistants out of take-home and online exams. This paper argues that the defense has already failed: current large multimodal models, prompted with one short instruction and no examples, recover the correct traversal or adjacency list from a rendered image most of the time on trees and more than half the time on graphs. To make the measurement leakage-resistant, the authors generated 9,072 unique problems spanning binary trees, binary search trees, undirected graphs, and directed graphs, with controlled variation in node count, edge width, node color, and task type. GPT-4o reached 87.6% pass@3 on trees and Gemini 1.5 Flash 56.2% pass@3 on graphs. The paper concludes that visual presentation cannot be treated as a reliable assessment safeguard and that pedagogy must move toward forms of assessment that do not depend on rendering a problem in an image.","feed_headline":"Multimodal models solve most visual tree exam problems","feed_subtitle":"Zero-shot GPT-4o hits 87.6% on image-only tree traversals; Gemini 1.5 Flash leads graphs at 56.2%.","key_machinery":"The engine of the study is a programmatic benchmark generator. It produces 9,072 image-text-answer triples across four structure classes — binary tree, binary search tree, undirected graph, directed graph — with node counts from 3 to 9, two edge widths, two node colors, several node-value sets, and six operation prompts, with each expected answer computed from the structure used to render the image. This generator serves three roles: it removes the risk of training-data leakage, it creates controlled structural and aesthetic variation for the feature analysis, and it yields a reusable open-source tool for re-testing future LMMs. The evaluation machinery is zero-shot prompting plus pass@k scoring with regular-expression answer extraction, and a logistic-regression model over engineered graph and image features identifies which variations drive accuracy.","core_discovery":"In the paper's own terms, the central finding is empirical: with a zero-shot user message that names the structure and asks for a Python-typed answer, the best tested LMMs can read a rendered diagram and produce the correct pre-order, in-order, post-order, breadth-first, or depth-first traversal, or the adjacency list, for a substantial fraction of programmatically generated, never-before-seen problems. GPT-4o achieves 77.8% pass@1 and 87.6% pass@3 on trees, and Gemini 1.5 Flash achieves 52.0% pass@1 and 56.2% pass@3 on graphs. Performance falls as the number of edges grows and is better on binary search trees than on directed graphs, which the authors attribute partly to the top-left-to-bottom-right patch reading of vision transformers. Because the prompting is deliberately minimal, the authors read these accuracies as a lower bound on what a student could obtain by iterating on prompts, making the result a statement about assessment integrity rather than only about model capability.","pith_inferences":["Beyond the paper: because no human verification of the rendered images or parse-failure rate is reported, the measurement's cleanliness is asserted rather than demonstrated; if valid model outputs are discarded by the extraction step, the true visual-reading rate would be higher than reported.","An untested implication is that the same problems posed as adjacency-matrix text rather than images would be nearly trivial for the same models; comparing those two conditions would isolate the visual-reading component of the errors.","The patch-order explanation suggests a concrete extension: rotating or mirroring the rendered trees and graphs should change accuracy in a predictable way if vision transformers parse images top-left to bottom-right, and the dataset could test that directly.","The authors stop at solving operations on given structures; an immediate next step is generating code from a diagram, which they flag as future work and which would widen the assessment threat to algorithm design questions."],"forward_implications":["Visual graph and tree problems should no longer be treated as inherently AI-resistant assessment items; a bare prompt suffices to solve most tree questions at pass@3.","Because the reported numbers come from zero-shot prompting, they are a floor: students who iterate or use few-shot prompt engineering would plausibly do better, so the practical integrity risk exceeds the headline accuracies.","Accuracy is systematically sensitive to structural load: performance degrades as edge count, density, and degree histograms rise, so any future image-based assessment that wants to survive must push problems into high-complexity regimes.","The released generator and 9,072-problem benchmark give instructors and researchers a standardized way to re-measure new models as they appear, so the result is a moving baseline rather than a one-time snapshot."],"supporting_citations":[{"why":"Prior demonstration that multimodal models solve image-based Parsons problems; the study's launching point and benchmark predecessor.","marker":"[33]"},{"why":"Supplies the vision-language task design standards (open-endedness, minimal prompt detail) that the benchmark follows.","marker":"[1]"},{"why":"CS2013 curriculum guidelines ground the selection of BST and graph traversal tasks as core curricular topics.","marker":"[65]"},{"why":"The proposal that visual problems mitigate AI cheating, which this paper's results directly challenge.","marker":"[22]"},{"why":"Earlier argument that diagram- and video-based OOP exercises counter LLM over-reliance; the position this study extends to graphs and trees.","marker":"[15]"},{"why":"Prior evaluation of code-generation models on Parsons problems with small prompt variations; informs the zero-shot evaluation design.","marker":"[56]"},{"why":"Establishes a scalable multi-language approach to benchmarking code generation that the generated dataset adapts to visual tasks.","marker":"[12]"},{"why":"Describes how vision transformers read images in top-left-to-bottom-right patches, used to explain trees being easier than graphs.","marker":"[66]"}],"fun_headline_variants":["GPT-4o aces 87.6% of visual tree problems","LMMs read diagrams: GPT-4o tops trees, Gemini leads graphs","Zero-shot LMMs solve 87.6% tree diagrams, 56.2% graphs","Trees? GPT-4o nails 87.6%. Graphs? Gemini leads"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole measurement assumes every rendered diagram is an unambiguous picture of the graph or tree whose traversal is the ground truth, and that a model answer that says the right thing in slightly different words is still counted as correct; if any rendering is ambiguous or any correct answer fails the extraction step, the reported percentages are biased.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4o aces 87.6% of visual tree problems","LMMs read diagrams: GPT-4o tops trees, Gemini leads graphs","Zero-shot LMMs solve 87.6% tree diagrams, 56.2% graphs","Trees? GPT-4o nails 87.6%. Graphs? Gemini leads"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001484,"raw_usage":{"total_tokens":6006,"prompt_tokens":1035,"completion_tokens":4971,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":651,"completion_tokens_details":{"reasoning_tokens":4881}},"tokens_in":651,"tokens_out":4971,"duration_ms":33769,"temperature":1.0,"reasoning_tokens":4881,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:18:06.783621+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Human annotators should re-derive the expected traversal or adjacency list for a random sample of the benchmark images; if even a small share of images produces disagreement with the generator's ground truth, or if the regex extraction discards answers a human would call correct, the measured accuracies are not a clean measure of visual problem-solving.","supporting_citations":[{"cited_title":"Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses?","cited_arxiv_id":"2303.09325","evidence_quote":"CS2013 curriculum guidelines ground the selection of BST and graph traversal tasks as core curricular topics."},{"cited_title":"A Picture Is Worth a Thousand Words: Exploring Diagram and Video-Based OOP Exercises to Counter LLM Over-Reliance","cited_arxiv_id":"2403.08396","evidence_quote":"Earlier argument that diagram- and video-based OOP exercises counter LLM over-reliance; the position this study extends to graphs and trees."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes how vision transformers read images in top-left-to-bottom-right patches, used to explain trees being easier than graphs."}],"review_version":1}