{"id":"3bc7cfd0-e132-4ac1-bebe-75bec5d33fae","arxiv_id":"2607.02853","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.5,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"Reversing only the UML relation arrow while keeping class names and layout fixed cuts open-source VLM relation accuracy by about 33%, revealing prior-over-vision bias.","lead":"Vision-language models often answer UML class-diagram questions from name-based priors rather than arrow direction. A controlled reverse-arrow benchmark shows large accuracy drops, especially for open-source models and harder multi-class diagrams.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the reader's already-flagged prior-construction residual; the matched-pair design and controls support the headline claim.","rationale":"The reader's strongest claim matches the paper's headline numbers and multi-control structure (matched reverse, prior-free, scale, relation type, multi-class, family scaling). The weakest_assumption correctly identifies the only material residual: whether LLM-assisted generation plus vocabulary constraints truly yield unambiguous canonical priors. That residual is already reflected in the CONDITIONAL verdict and does not rise to a new correctness risk that would move the verdict. Independent human prior-label validation is the cleanest single check; if it passes, confidence in the diagnostic can only increase. Artifact anonymity and closed-model APIs limit perfect re-run guarantees but do not undermine the reported open-source pattern given the transparent per-cell appendix. No equation-level inconsistency, metric redefinition issue, or unaddressed confound is more load-bearing than the already-noted prior-construction premise.","tokens_in":21327,"tokens_out":525,"duration_ms":5452,"concrete_test":"On a stratified sample of ~200 prior-conform/conflict pairs (balanced across four relations), collect independent human majority votes for the expected 'canonical' direction from class names alone (no image). If human agreement with the paper's prior labels is ≥95% and re-scoring Δ after dropping low-agreement pairs changes open-source mean Δ by <5 absolute points, the isolation claim holds; larger shifts would weaken it.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that open-source VLMs substitute class-name priors for arrow direction on UML diagrams, quantified by a large RelAcc drop when only the arrow is reversed (open-source mean Δ=33.48% on two-class; larger on three-class). For that Δ to isolate knowledge conflict rather than residual ambiguity or generation artifacts, the controlled vocabulary and four validation constraints in §3.1 must make each ordered pair's 'canonical' direction unambiguous. The paper already addresses the main alternatives: prior-free separates pure visual failure from bias; scale invariance rules out a pure perceptual bottleneck; relation-type slices show larger gaps where priors are stronger (Aggregation/Inheritance); and prerequisite EM/RelAcc_prior-free filters uninformative InternVL variants. No stronger internal inconsistency is evident in the reported design or appendix cells. The residual risk is exactly the reader's weakest_assumption—not a new load-bearing hole that overturns the diagnostic.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper argues that current VLMs often answer UML class-diagram relation questions from pretrained class-name priors rather than from the depicted arrow direction. It introduces a controlled benchmark in which each prior-conforming diagram is paired with a prior-conflicting counterpart that keeps class names and layout fixed while reversing only the relation arrow, plus a prior-free control that replaces names with opaque labels. Using separate class-name exact-match (EM) and relation-direction accuracy (RelAcc) metrics, the authors evaluate InternVL3.5 and Qwen3 size variants plus GPT-5.4 / GPT-5.4 Mini. They report a large open-source conflict gap (mean Δ ≈ 33.48% on two-class reverse; larger on three-class variants), scale-invariance of the gap, stronger effects on Aggregation/Inheritance, family-dependent scaling, and a reduced but non-zero gap for frontier models on harder structural conditions.","tokens_in":21626,"tokens_out":1263,"duration_ms":20677,"significance":"If the diagnostic holds, the work is a clear and useful contribution to multimodal evaluation for software-engineering artifacts: it shows that aggregate UML VQA accuracy can mask a post-reading prior override, and it supplies a falsifiable matched-pair test rather than only naturalistic counterfactuals. Strengths include the paired prior-conform/prior-conflict design, the prior-free disentangling control, separate perception vs. inference metrics, multi-relation and multi-scale slices, the prerequisite-skill analysis that correctly qualifies InternVL scaling claims, the full per-cell appendix table, and a public artifact. These make the central claim actionable for both VLM robustness research and SE tooling that relies on diagram-grounded reasoning.","major_comments":[{"comment":"Section 3.1 (Generation/Validation) and the definition of Δ: the causal claim that Δ isolates knowledge-prior override assumes that each ordered class pair has an unambiguous, model-shared 'canonical' direction. The four structural validation constraints are necessary but not sufficient; the manuscript does not report an independent prior-strength check (e.g., text-only or image-free probing of the same pairs, or human prior ratings). Without that measurement, residual name ambiguity or generation artifacts remain a load-bearing alternative explanation for part of the gap, even though prior-free and scale controls already rule out pure unreadability and pure resolution bottlenecks.","section":null},{"comment":"Section 4.1 / Fig. 5: the further RelAcc drop from 2-reverse to 3-reverse/3-mixed is interpreted as 'adding more visual context exacerbates prior-over-vision.' That interpretation is plausible but not fully isolated. The paper needs a clearer three-class prior-free (or prior-conform three-class) control reported on the same footing, or an analysis that holds visual complexity fixed while varying only prior alignment of the auxiliary edge. Otherwise complexity, attention dilution, or multi-edge parsing difficulty can co-explain the ~12-point additional drop and the open-source 45.28% three-class figure.","section":null},{"comment":"Section 3.2 scoring protocol: counting Unknown as incorrect is a defensible conservative choice, but it interacts with the main claim. If models systematically abstain more under conflict than under conform (or under three-class than two-class), part of Δ could be refusal/uncertainty rather than confident prior substitution. Please report Unknown rates by condition (at least for the family means) so readers can separate forced prior answers from abstention-driven accuracy loss.","section":null}],"minor_comments":[{"comment":"Abstract and §1: spacing/punctuation glitches around percentages (e.g., '33.48%on', '10%gap', '45.28%for') should be cleaned for camera-ready readability.","section":null},{"comment":"Fig. 1 caption and body: the Mammal/Elephant example is clear, but the figure panel labels (a/b/c) and the 'Yes! ❌' annotation are a bit dense; a single sentence stating the gold label under each panel would help skimmers.","section":null},{"comment":"Table 1 / Appendix D: prior-free is much smaller after abbreviation deduplication; state explicitly in the main text that prior-free is a diagnostic control, not size-matched to the semantic conditions, so readers do not over-interpret absolute prior-free RelAcc as comparable sample power.","section":null},{"comment":"§4.3 / Fig. 9: the EM≥95% and RelAcc_prior-free≥50% thresholds are reasonable analysis filters; briefly justify the 50% baseline (always-True / chance) in the main text rather than only in the figure discussion.","section":null},{"comment":"Related Work §2.1: the UML extraction literature is well covered; a short note on how the present conflict design differs from ordinary UML VQA datasets (Shehzadi et al., Naboichenko & Peinl) would sharpen novelty for SE readers.","section":null},{"comment":"Prompts in Appendix F are a strength; consider releasing the exact PlantUML templates and the controlled vocabulary list in the artifact README so the 'unambiguous prior' assumption can be audited externally.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The matched-pair design is stronger than many VLM prior-bias papers, and the appendix table is unusually complete. My main residual concern is prior-strength validation rather than any internal contradiction. Scope fits a CV/ML evaluation venue well; for a pure SE journal the authors may need one more applied case (e.g., student UML grading or reverse-engineering pipeline impact). GPT-5.4 naming will age with the literature; ensure model identifiers match the public API names at camera-ready time."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is a useful diagnostic paper, not a theory rewrite. The new piece is a controlled UML knowledge-conflict benchmark: matched prior-conform / prior-conflict diagrams that keep class names and layout fixed and reverse only the relation arrow, plus prior-free letter controls and three-class variants. On that setup the headline result is clear: open-source InternVL3.5 and Qwen3 lose about 33.5 points of relation-direction accuracy when the arrow flips, with larger drops on three-class diagrams; GPT-5.4 nearly closes the two-class gap but still slips on harder structure.\n\nWhat they do well is the experimental hygiene. Separate class-name EM from RelAcc, prior-free disentangling, scale and relation slices, and the prerequisite-skill filter for the InternVL scaling story all make the claim harder to dismiss as pure OCR failure or image-size artifact. The full per-cell appendix and PlantUML rendering pipeline are the right kind of transparency. Related UML-VLM and natural-image prior-bias work is cited honestly; the novelty is the formal-diagram instantiation, not inventing prior bias from scratch.\n\nSoft spots are real but secondary. The load-bearing assumption is that the controlled vocabulary and LLM-assisted generation make each ordered pair’s “canonical” direction unambiguous. They put four validation constraints on that, and prior-free / scale / relation-type checks reduce the main alternatives, but residual generation ambiguity is still the main risk, not a hidden math flaw. Scope is intentionally narrow (one target relation query per diagram), closed-model APIs and the anonymous artifact host limit perfect re-run guarantees, and “GPT-5.4” naming will age oddly. None of that overturns the diagnostic as stated.\n\nThis is for people building or auditing multimodal SE tools and for VLM reliability folks who care about formal diagrams rather than natural photos. I would bring it to reading group, cite the benchmark and the open-source gap numbers, and send it to peer review. It deserves a serious referee, not a desk reject.","headline":"Clean matched reverse-arrow UML benchmark shows open-source VLMs follow class-name priors over notation; design is solid and the residual risk is mainly prior-construction, not a broken claim.","tokens_in":22219,"tokens_out":525,"would_cite":true,"duration_ms":6135,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"VLMs often answer UML relation questions from class-name priors, not from the arrow they are shown.","keywords":["vision-language models","UML class diagrams","prior bias","knowledge conflict","relation direction","software engineering diagrams","model scaling"],"falsifier":"Find a model and evaluation setting in which relation-direction accuracy on the prior-conflicting diagrams stays within a few points of the prior-conforming diagrams across inheritance and aggregation, including the three-class conditions, while class-name recovery and prior-free arrow reading remain high.","tokens_in":22203,"feed_emoji":"📐","tokens_out":564,"duration_ms":5626,"temperature":0.7,"pith_summary":"Vision-language models are being asked to read software diagrams whose meaning is fixed by visual notation alone. This paper asks whether those models actually use the arrow on a UML class diagram or instead fall back on what they already know about how the named classes usually relate. The authors build matched diagram pairs that keep the same class names and layout and reverse only the relation arrow, then measure how much relation-direction accuracy collapses under that single change. On eight open-source models the average drop is about one-third; three-class diagrams make the failure worse, and even frontier models that nearly close the simple two-class gap still lose accuracy once structure is added. Scaling helps only in a family-dependent way and does not remove the bias. The result matters because software-engineering pipelines that treat diagrams as ground truth can silently inherit pretrained design stereotypes instead of the diagram that is actually drawn.","feed_headline":"Reversing one UML arrow cuts VLM accuracy by a third","feed_subtitle":"Models follow class-name priors over the diagram; three-class cases and even frontier models still fail.","key_machinery":"A matched prior-conforming / prior-conflicting UML benchmark: each diagram pair shares class names, layout, and query, differs only by arrow direction, and is supplemented by a prior-free control that replaces names with opaque labels so perception failures can be separated from prior-driven inference.","core_discovery":"When a UML class diagram is rewritten so that only the relation arrow is reversed while class names and layout stay fixed, open-source vision-language models lose roughly 33 percent relation-direction accuracy on average; the gap is larger still on three-class variants and is only partially closed by frontier models. The models therefore often answer from the knowledge prior implied by the class names rather than from the visual notation.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["UML arrow reverse cuts VLM relation accuracy by a third","VLMs favor class-name priors over reversed UML arrows","Open-source VLMs drop 33% when only the UML arrow flips","Three-class UML arrow flips cut VLM accuracy by 45%","Frontier VLMs still show 10–18% gaps after arrow reverse"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The controlled class-name vocabulary and generation checks really produce unambiguous canonical priors for every ordered pair, so that the accuracy drop can be attributed to knowledge conflict rather than residual name ambiguity or diagram artifacts.","fun_headline_variants_meta":{"raw":{"variants":["UML arrow reverse cuts VLM relation accuracy by a third","VLMs favor class-name priors over reversed UML arrows","Open-source VLMs drop 33% when only the UML arrow flips","Three-class UML arrow flips cut VLM accuracy by 45%","Frontier VLMs still show 10–18% gaps after arrow reverse"]},"model":"grok-4.5","effort":"low","cost_usd":0.006824,"raw_usage":{"total_tokens":1668,"prompt_tokens":796,"num_sources_used":0,"completion_tokens":92,"cost_in_usd_ticks":68240000,"prompt_tokens_details":{"text_tokens":796,"audio_tokens":0,"image_tokens":0,"cached_tokens":0},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":780,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":796,"tokens_out":92,"duration_ms":5951,"temperature":1.0,"reasoning_tokens":780,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T06:35:16.670794+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Find a model and evaluation setting in which relation-direction accuracy on the prior-conflicting diagrams stays within a few points of the prior-conforming diagrams across inheritance and aggregation, including the three-class conditions, while class-name recovery and prior-free arrow reading remain high.","supporting_citations":[],"review_version":1}