{"id":"d336fb81-3f91-4165-86bd-0a0f6e13c46b","arxiv_id":"2506.05440","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A Blender-based diagnostic toolkit that tests VLMs on fine-grained visual skills by varying one visual attribute at a time, exposing failure modes that coarse benchmarks miss.","lead":"The authors built a system that automatically renders synthetic images with strictly controlled visual properties, such as object count, blur, or camera distance, and then uses them to test vision-language models on specific low-level skills. In chess and poker scenes, models showed systematic mistakes that standard aggregate benchmarks would hide, such as overcounting blurred objects and underestimating overlapping cards.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The real-world transfer claim rests on a single task and an aggregate correlation computed over just 8 count levels; per-sample correlations in the same appendix are much lower (0.23–0.48 for Gemma3 variants), so the >0.99 \"strong proxy\" claim is not yet established.","rationale":"Good-faith reading: the paper's core contribution is a procedural-generation framework for controlled VLM diagnostics, and the internal results (counting, localization, blur, overlap) are extensive and largely convincing. The central claim about aggregate benchmarks masking brittle behaviors is supported by the demonstration that models with strong overall performance fail systematically under specific controlled conditions; that portion does not require real-world transfer. The load-bearing weakness is the step from \"synthetic stress exposes failure modes\" to \"these failures are indicative of real-life scenarios\" (Introduction, second-to-last paragraph; Appendix E). Appendix E's headline correlation (>0.99) is computed after averaging over samples, prompts, and then over only 8 count levels. Since accuracy on counting decreases monotonically with object count for nearly all models on both synthetic and real images, the level-aggregated correlation is high by construction and is not evidence of per-image transfer. The authors themselves provide per-sample correlations (Figure 52) that are considerably lower, especially for smaller models, undermining the \"strong proxy\" language. A single concrete check—per-scene, per-prompt synthetic-vs-real correlation—would show whether the aggregate statistic is doing the work. If it fails, the central practical claim should be softened to \"the framework reveals synthetic-condition failure modes that may or may not transfer to natural imagery,\" which is already hinted at in the Limitations. This does not change the overall conditional acceptance; it sharpens the condition.","tokens_in":52180,"tokens_out":9826,"duration_ms":120309,"concrete_test":"Recompute the Appendix E synthetic-to-real alignment at the per-scene level: for each of the 80 scene pairs and each model, take the per-prompt predictions (not averaged over count levels) and compute Pearson and Spearman correlations between synthetic-image predictions and real-image predictions. Report the minimum correlation across the six prompt conditions. If the minimum per-scene correlation for a headline model (e.g., GPT-4.1) is below 0.9, the >0.99 headline is an aggregation artifact and the \"strong proxy\" wording should be withdrawn; if it remains above 0.9, the concern is lower but the matched-scene, one-task scope still warrants a qualifier.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix E claims a \"strong proxy\" for real-world counting from 80 matched chess scenes, citing Pearson/Spearman >0.99 for most models after averaging over 10 samples, 6 prompt variants, and then correlating over the 8 object-count levels. Because counting accuracy declines monotonically with item count for essentially every model on both synthetic and real images, a correlation over 8 level means is near 1 regardless of per-image transfer. The authors also report per-sample correlations (Figure 52) that are substantially lower—e.g., Gemma3-4B at 0.48–0.57 and Gemma3-12B at 0.23–0.36—so the headline statistic is an artifact of aggregation. This matters because the central practical claim—that failures revealed by synthetic stress tests are indicative of real-world perception failures—is otherwise supported only by (i) one task (counting), (ii) physically matched scenes, and (iii) no transfer check for blur, overlap, or localization, which are the most striking brittle behaviors (GPT-4.1 underestimating overlapped cards, LLaMA-4-Scout extrapolating an 8×8 board). The paper's own Limitations (Section 5) concedes that more realistic, industrial-scenario images are future work, effectively acknowledging the gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BYO-Eval, a framework for procedurally generating synthetic image datasets with controlled visual attributes (object count, blur, camera distance, overlap) to create fine-grained diagnostic tasks—counting, localization, and identification—for vision-language models (VLMs). Using Blender, the authors generate chess and poker scenes where one visual variable is varied while others are held constant, and they evaluate eight VLMs (GPT-4.1, GPT-4.1-mini, LLaMA-4-Scout/Maverick, Gemma3-4B/12B, LLaMA3.2-Vision, Mistral-3.1) on this suite. The paper reports several brittle behaviors, such as GPT-4.1 overestimating chess piece counts and underestimating overlapping poker cards, and LLaMA-4-Scout exhibiting an 8x8 board bias in 4x4 localization. It also includes a real-world validation study (Appendix E) with 80 physically recreated chess scenes, claiming strong correlations (Spearman > 0.99 for most models) between synthetic and real counting performance after averaging over samples and prompt variants. The central claim is that general-purpose benchmarks mask these brittle behaviors and that controlled diagnostic stress reveals them.","tokens_in":52425,"tokens_out":4884,"duration_ms":58366,"significance":"If validated, the methodology offers a useful complement to aggregate benchmarks: it is open-source, reproducible, and provides a systematic one-variable-at-a-time control over visual parameters without fitting any model parameters. The diagnostic approach, inspired by ophthalmologic stress tests, is a genuinely valuable idea for isolating perception failures from reasoning or textual biases, and the authors demonstrate a broad range of tasks across eight VLMs. However, the external-validity evidence is currently weak: the only real-world test is a single counting task on 80 scenes that are physically recreated from the same synthetic legends, and the headline correlation is computed over aggregated level means rather than per-image predictions. The internal synthetic diagnostics are the paper's core strength, but the claim that synthetic performance is a strong proxy for real-world counting is not established by the reported statistics.","major_comments":[{"comment":"The headline correlation of >0.99 between synthetic and real counting is computed by first averaging over 10 samples and 6 preprompt–instruction pairs, then correlating only the 8 target-count-level means. Because counting accuracy declines monotonically with object count for essentially every model on both image sets, a correlation over level means is near 1 regardless of per-image transfer. The per-sample correlations in Figure 52 are much lower—e.g., 0.48–0.57 for Gemma3-4B and 0.23–0.36 for Gemma3-12B—so the statement in Appendix E that synthetic performance is \"a strong proxy for real-world counting ability\" is not supported. Please report per-level or per-image correlations, or use an agreement metric (e.g., Bland–Altman limits, prediction-error agreement), and temper the \"strong proxy\" claim accordingly.","section":"Appendix E, Figures 51–53; Section 4.2"},{"comment":"The 80 real scenes are physical recreations of the synthetic chess legends, so high correlation partly reflects that the real stimuli replicate the synthetic content by construction rather than demonstrating transfer to natural, unmatched imagery. The paper's own Limitations section (Section 5) concedes that more realistic, industrial-scenario images are future work. Since no transfer check is provided for blur, overlap, or localization—the very behaviors highlighted as brittle in the paper—the practical real-world relevance of those diagnostic findings is not established. I recommend either adding at least one unmatched real-image condition (e.g., an existing counting benchmark) or explicitly reframing the real-synthetic comparison as a sanity check of physical plausibility, not evidence of generalization.","section":"Appendix E, Figure 50; Section 5"},{"comment":"Per-cell sample sizes are small (k=5 or 10 images per condition) and the paper reports only means and standard deviations, with no confidence intervals or significance tests. For example, the blur-dependent overestimation of GPT-4.1 in Figure 14 is based on 5 samples per blur level, and Figure 36a shows cell-level NMAE differences (e.g., 0.00 vs. 0.40) without any measure of uncertainty. These small samples make specific quantitative claims about model biases fragile; at minimum, per-cell error bars or bootstrap intervals should be reported, and the conclusions should be limited to the observed qualitative directions rather than the precise magnitudes.","section":"Sections 4.1, B.4, B.9; Tables 2–3; Figure 36"}],"minor_comments":[{"comment":"The word \"opthalmologic\" should be spelled \"ophthalmologic\".","section":"Abstract"},{"comment":"The phrase \"precised, tailored evaluation\" should read \"precise, tailored evaluation\".","section":"Section 2, paragraph 1"},{"comment":"Table 2 says \"Mean and standard deviation over 50 images for each task,\" while Table 3 says \"40 samples per level\"; please clarify the per-cell sample sizes and how they differ across tasks.","section":"Tables 2 and 3 captions"},{"comment":"The phrase \"blur decrease (higher values meaning less blur)\" is confusing given the x-axis is labeled \"blur\" with values 1–5; specify which direction corresponds to stronger blur.","section":"Figures 14 and 15 captions"},{"comment":"The sentence \"All experiments are executed using a unified evaluation pipeline...\" appears nearly verbatim twice; please remove the duplicate.","section":"Appendix C, first paragraph"}],"recommendation":"major_revision","confidential_remarks":"The methodology contribution is solid and the paper is likely to interest the VLM evaluation community. The main gap is the overclaimed real-world transfer evidence; the revision should either add stronger validation or substantially soften the proxy claim. I would also encourage the authors to provide per-sample correlations and uncertainty estimates throughout, as the small per-cell sample sizes otherwise make the specific quantitative findings fragile."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honestly, this is a solid, well-engineered evaluation tool paper. What's new: a configurable Blender-based pipeline that generates synthetic VLM benchmark images with one-variable-at-a-time control (object count, blur, overlap, camera distance) while holding other visual parameters fixed, plus a clean set of diagnostic metrics and a multi-model comparison. The \"ophthalmologic\" framing is appropriate. It delivers what it promises: aggregate benchmarks do mask brittle behaviors. GPT-4.1 overcounting blurred objects, undercounting overlapped poker cards, and LLaMA-4-Scout predicting out-of-bounds rows on a 4x4 board (a likely 8x8 prior) are concrete, useful findings. The prompt-sensitivity results (declarative outputs beating fill-in-the-blank, CoT destabilizing under blur) are a bonus.\n\nThe core methodology has no fitted parameters, so the circularity burden is low. I credit the matched real-synthetic correlation study for good intent, but here's the main soft spot: the \"strong proxy for real-world counting\" claim (Appendix E) is built on an aggregate correlation over just 8 count levels, after averaging over samples and prompt variants. For any model that roughly tracks item count, that correlation is near 1. The per-sample correlations, which the authors also report, are far lower (Gemma3-12B at 0.23-0.36, Gemma3-4B at 0.48-0.57). The summary statement \"strong proxy\" overstates what the evidence supports; the paper's own limitations section, which calls more realistic, unmatched industrial imagery future work, is the honest framing.\n\nOther soft spots are minor: per-cell sample sizes of 5 or 10 limit the precision of some quantitative claims, and the code is promised but not verifiable from the preprint (which matters, since the pipeline is the contribution). The paper is honest, engages properly with related work, and the central argument that controlled diagnostics reveal benchmark-masked failures holds up.\n\nWho is this for: researchers or teams needing cheap, interpretable, stress-test-style VLM evaluation of specific visual skills. It deserves a serious referee. I'd recommend accept-with-revisions: qualify the real-synthetic transfer claim, present the per-sample correlations in the main text, and ideally ship the code before camera-ready.","headline":"A genuinely useful VLM stress-test toolkit; the real-synthetic 'strong proxy' claim is overstated, but the core diagnostic methodology holds up and deserves peer review.","tokens_in":52951,"tokens_out":2413,"would_cite":true,"duration_ms":29648,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that aggregate accuracy scores on general benchmarks hide predictable, model-specific perception failures in vision-language models, and that these failures surface only when a single visual attribute is varied under…","keywords":["vision language models","multimodal evaluation","synthetic data","procedural generation","object counting","localization","visual perception diagnostics","benchmark design"],"falsifier":"Render the same diagnostic tasks on natural, unmatched photographs of chessboards and poker tables and check whether the documented bias patterns still appear, such as GPT-4.1 overcounting blurred scenes and undercounting overlapped cards, or LLaMA-4-Scout emitting out-of-bounds coordinates on 4×4 boards; if those patterns vanish, the synthetic-to-real proxy claim fails. A cheaper version is to repeat the 80-scene physical reproduction check on the poker counting task, which the paper itself shows to be far harder for models.","tokens_in":51968,"feed_emoji":"👁️","tokens_out":6219,"duration_ms":69723,"temperature":0.7,"pith_summary":"The paper argues that aggregate accuracy scores on general benchmarks hide predictable, model-specific perception failures in vision-language models (VLMs). It proposes a diagnostic framework that renders synthetic chess and poker scenes in Blender, varying exactly one visual attribute such as object count, blur, overlap, or camera distance while holding everything else fixed. Running eight VLMs through these graded task series reveals systematic biases, for instance GPT-4.1 overcounting blurred objects but undercounting overlapping poker cards, and LLaMA-4-Scout anchoring its localization answers to a standard 8×8 chessboard when the scene uses a 4×4 board. If the framework is right, VLM evaluation can shift from opaque leaderboards to interpretable skill profiles, and model choice for concrete applications can be grounded in tested visual abilities.","feed_headline":"Controlled synthetic scenes expose hidden VLM failure modes","feed_subtitle":"Varying one visual attribute at a time turns benchmark scoring into a model-by-model skills exam.","key_machinery":"The framework's organizing mechanism is the Cartesian product over controlled variables: for variables v1,...,vn the dataset is D = V1 × ... × Vn, so every combination of tested levels is rendered while unrelated scene parameters stay constant. Scenes are produced procedurally in Blender through its Python API, each image carries an automatic legend with ground-truth object states, and each task type (counting, localization, identification) has its own scoring metric, such as Mean Absolute Error for counts and L1 grid distance for localization. This design isolates the effect of the single stressed variable, turning a VLM evaluation into an interpretable diagnostic comparable to an ophthalmologic field test.","core_discovery":"The paper's central claim is that general-purpose benchmarks mask brittle behaviors that only emerge under controlled diagnostic stress. The authors establish this by generating synthetic chess and poker images with precise, per-attribute control and measuring how each VLM's accuracy degrades as one variable is stressed. The discovered patterns are systematic and model-dependent: GPT-4.1 stays accurate up to five chess pieces yet overcounts beyond that, undercounts poker cards increasingly with horizontal overlap, and overcounts as blur rises; LLaMA-4-Scout predicts out-of-bounds row and column coordinates on 4×4 boards, apparently inferring a standard 8×8 chessboard, and underestimates relative distances between pieces on 8×8 boards. The paper further supports the real-world relevance of these synthetic diagnostics with matched physical reproductions of 80 chess scenes, reporting Spearman and Pearson correlations above 0.99 for most tested models when synthetic and real counting accuracy are compared.","pith_inferences":["If this diagnostic logic transfers to industrial scenes, the same one-attribute-at-a-time rendering could stress-test models for specific production tasks, such as counting items on a conveyor or locating defects, before deployment.","The paper's validation covered only counting on 80 chess scenes physically reproduced from synthetic legends; an obvious extension is to test whether the documented bias patterns (blur overcount, overlap undercount, 8×8 anchoring) replicate on unconstrained real photographs.","The strong synthetic-real agreement for chess counting may partly reflect chess being well represented in VLM training data; the poker scenes, which the paper shows are much harder, could exhibit weaker transfer and would provide a sharper test of the proxy claim.","Following the paper's own sketch, the same framework could be extended to brightness, color, and measurement skills, producing a full perceptual skill card for each model instead of a single score."],"forward_implications":["Evaluation practice can move from static, aggregate benchmarks to task-specific synthetic diagnostics that separate perception failures from reasoning failures.","Model selection for an application can be based on verified skill profiles, such as which model remains robust under blur, occlusion, or distant camera viewpoints.","Prompt phrasing is part of the measured capability: declarative instructions paired with helpful preprompts consistently raise accuracy, so perception scores are only meaningful when reported with the full prompt protocol.","Strong synthetic-to-real correlation for most models implies that targeted synthetic tests can forecast counting accuracy on matched real scenes, at least in structured chess-like layouts.","Failure modes are model-specific rather than universal, so a model that leads on an aggregate leaderboard may still be the wrong choice in a particular perceptual setting."],"supporting_citations":[{"why":"Supplies the closest prior evidence that controlled synthetic image variations expose VLM detection failures as distractors increase.","marker":"[2]"},{"why":"Documents seven elementary visual tasks where VLMs fail, anchoring the claim that basic perception remains unreliable.","marker":"[3]"},{"why":"Shows VLMs can answer many multiple-choice benchmark questions without seeing the image, motivating the need for controlled diagnostics.","marker":"[10]"},{"why":"Benchmarks spatial reasoning in VLMs and finds significant shortcomings, supporting the localization task design.","marker":"[16]"},{"why":"Demonstrates GPT-4V is good at captioning but bad at counting and localization on earth observation data, a key low-level failure precedent.","marker":"[17]"},{"why":"Closest related framework, building synthetic datasets from scene graphs and reporting a real-synthetic performance correlation that the paper treats as precedent for its proxy validation.","marker":"[46]"},{"why":"Survey of synthetic data methods whose Section 5 is cited to justify using computer-graphics rendering for evaluation.","marker":"[50]"}],"fun_headline_variants":["Synthetic scenes expose VLM counting failures","One-attribute stress tests reveal VLM brittleness","Diagnose VLMs with synthetic chess and poker scenes","Pinpoint VLM weaknesses by stressing one visual attribute"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that synthetic chess and poker scenes, plus physical reproductions matched to synthetic legends for counting one to eight pieces, are a valid proxy for real-world visual skills rests on those matched scenes standing in for natural, unmatched imagery.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic scenes expose VLM counting failures","One-attribute stress tests reveal VLM brittleness","Diagnose VLMs with synthetic chess and poker scenes","Pinpoint VLM weaknesses by stressing one visual attribute"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000691,"raw_usage":{"total_tokens":3124,"prompt_tokens":938,"completion_tokens":2186,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":2125}},"tokens_in":554,"tokens_out":2186,"duration_ms":20222,"temperature":1.0,"reasoning_tokens":2125,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:29:12.476539+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Render the same diagnostic tasks on natural, unmatched photographs of chessboards and poker tables and check whether the documented bias patterns still appear, such as GPT-4.1 overcounting blurred scenes and undercounting overlapped cards, or LLaMA-4-Scout emitting out-of-bounds coordinates on 4×4 boards; if those patterns vanish, the synthetic-to-real proxy claim fails. A cheaper version is to repeat the 80-scene physical reproduction check on the poker counting task, which the paper itself shows to be far harder for models.","supporting_citations":[{"cited_title":"Understanding the limits of vision language models through the lens of the binding problem,","cited_arxiv_id":null,"evidence_quote":"Supplies the closest prior evidence that controlled synthetic image variations expose VLM detection failures as distractors increase."},{"cited_title":"Vision language models are blind,","cited_arxiv_id":null,"evidence_quote":"Documents seven elementary visual tasks where VLMs fail, anchoring the claim that basic perception remains unreliable."},{"cited_title":"Is a picture worth a thousand words? delving into spatial reasoning for vision language models,","cited_arxiv_id":null,"evidence_quote":"Benchmarks spatial reasoning in VLMs and finds significant shortcomings, supporting the localization task design."},{"cited_title":"Task Me Anything","cited_arxiv_id":"2406.11775","evidence_quote":"Closest related framework, building synthetic datasets from scene graphs and reporting a real-synthetic performance correlation that the paper treats as precedent for its proxy validation."},{"cited_title":"A survey of synthetic data augmentation methods in machine vision,","cited_arxiv_id":null,"evidence_quote":"Survey of synthetic data methods whose Section 5 is cited to justify using computer-graphics rendering for evaluation."}],"review_version":1}