{"id":"894bed8f-9eeb-42f0-8cb6-8a11e805170a","arxiv_id":"2507.04741","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"On a new benchmark of odd-one-out images, state-of-the-art vision-language models, including GPT-4o, often fail to identify which low-level feature makes the odd object stand out.","lead":"This paper introduces SalBench, a benchmark that tests whether vision-language models can spot the single odd object in images, such as one red circle among many blue ones. The authors find that even GPT-4o scores below fifty percent on natural images, suggesting models miss simple visual oddities that people notice instantly.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim that targets are 'obvious' lacks a human baseline; the paper's own supplementary admits some attributes are hard for humans, so model scores near 50% F1 may not yet demonstrate a large human-model gap.","rationale":"The reader's weakest assumption identifies the missing human baseline, and I agree that is the most load-bearing point. In a multi-label exact-match evaluation, gold labels are multi-attribute sets (e.g., 'Color, Focus, Size, Pattern' in Figure A.16). A model can be penalized for omitting a subtle attribute that a human might not spontaneously name; without human data, low model scores conflate 'cannot perceive the odd-one-out' with 'did not output the full label set.' The paper's own supplementary text admits the focus attribute is hard for humans (Section C), and Figure A.16's caption describes the focus difference as subtle, making the assumed human ceiling an empirical question rather than a design fact. The abstract's '47.6% accuracy' is actually an F1 score; the matching accuracy in Table A.1 is 45.2% for GPT-4o on natural detection. This is a real reporting error, but it does not by itself overturn the qualitative result. A human study would settle whether the benchmark's difficulty and the 'obvious' framing are warranted. The paper would also benefit from releasing code/data, but that is secondary to the scientific claim. Thus I keep the reader's CONDITIONAL verdict: the concern is real but addressable by data the authors can collect.","tokens_in":33597,"tokens_out":3045,"duration_ms":32626,"concrete_test":"Recruit naive participants (e.g., 20) to label a stratified sample of SalBench: 100 natural O3 images (balanced by attribute and distractor count) and 100 synthetic P3 images. Use the same prompts and the same seven/three feature categories; ask each participant to list all features in which one object differs from the others. Compute exact-match multi-label accuracy and per-category F1, and also majority-vote agreement. If human exact-match accuracy on the natural subset is near GPT-4o's 45-48%, the 'obvious' premise is unsupported; if it is above 85%, the central claim is empirically grounded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that LVLMs fail on 'obviously salient' odd-one-out targets, with GPT-4o at 47.6% (abstract; actually F1, not accuracy). This claim depends on targets being at or near ceiling for human observers. No human subjects were run on SalBench. The assumption is empirically load-bearing because scoring is multi-label exact-match: a target annotated with {Color, Focus, Size, Pattern} is counted wrong unless the model outputs exactly that set. Human annotators may disagree on which attributes are 'notably different', especially for attributes like focus; the supplementary (Section C) explicitly says the focus attribute is 'particularly challenging not only for models but also for humans.' If human-human exact-match agreement on the natural split is closer to the model scores than to 100%, the headline claim 'can't see the obvious' would be overstated, and the benchmark's difficulty calibration would be open. The F1/accuracy conflation in the abstract is a separate reporting flaw but not the core of the argument.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SalBench, a benchmark built from the P3 and O3 datasets, with three tasks (Odd-One-Out Detection, Referring Odd-One-Out, and Visual Referring Odd-One-Out) that ask LVLMs to identify low-level visual attributes (color, orientation, size, focus, etc.) in which a singleton differs from distractors. The authors evaluate a wide range of open and closed LVLMs under zero-shot and few-shot settings, report F1 and matching-accuracy results, analyze difficulty levels and distractor counts, test vision-backbone retrieval, and train custom LVLMs on synthetic saliency data. The central claim is that current LVLMs, including GPT-4o, fail at detecting visually obvious odd-one-out targets, with GPT-4o achieving only 47.6 F1 on the natural image split of the Detection task.","tokens_in":33855,"tokens_out":3684,"duration_ms":41262,"significance":"If the central claim holds, SalBench fills a real gap in vision-centric LVLM evaluation: existing benchmarks emphasize high-level reasoning, whereas this benchmark targets low-level perceptual features. The paper's strengths are its broad model coverage, the three-task design that separates detection from grounding, the per-attribute and per-difficulty breakdowns, and the honest training experiment showing that in-distribution synthetic saliency data does not produce high scores. The benchmark itself is likely to be a useful resource for future work. However, the headline claim that the targets are 'obvious' to humans is currently an assumption rather than a measured fact, and the difference between F1 and matching accuracy is conflated in the abstract and conclusion. These issues affect the interpretation of the results, not the raw measurements themselves.","major_comments":[{"comment":"The paper's headline number is mislabeled. The abstract says GPT-4o achieves '47.6% accuracy', and the conclusion repeats '46% accuracy', but Table 1 explicitly reports F1 scores, and the matching accuracy for GPT-4o on the natural Detection split at zero-shot is 45.2% (Supplementary Table A.1). Since the abstract and conclusion make the paper's central quantitative claim using the wrong metric, the text should be corrected to state F1, and matching accuracy should be reported in the abstract if the authors intend accuracy as the headline metric.","section":"Abstract; Section 7; Table 1 caption; Supplementary Table A.1"},{"comment":"The premise that SalBench targets are 'readily apparent' and 'visually obvious' to humans is load-bearing but unmeasured. No human participants were tested on SalBench, and the supplementary explicitly states that the focus attribute is 'particularly challenging not only for models but also for humans' (Supplementary Section C). Because the evaluation uses exact-match multi-label scoring, a model is marked wrong unless it outputs exactly the annotated attribute set; if human annotators disagree on which attributes are salient (especially for focus), the observed model scores may be closer to human-level agreement than the 'can't see the obvious' framing suggests. The authors should run a human study on a representative sample, report human-human exact-match agreement and per-attribute agreement, and compare model scores against that baseline.","section":"Section 1; Section 3; Supplementary Section C"},{"comment":"The natural-split F1 scores are not calibrated against chance or majority baselines. The natural split is heavily dominated by the color attribute (over 80% of images), so a model that always predicts color would obtain high color F1 and a nontrivial overall F1; the reported GPT-4o overall F1 of 47.6% needs to be interpreted against such a baseline, especially because the paper's central comparison is between model performance and human performance. Adding random-predictor and majority-predictor baselines, computed with the same exact-match and F1 protocols, would make the headline numbers much easier to assess.","section":"Section 4.1; Table 4"},{"comment":"There is a direct inconsistency between the prose and Table 1 in the same section. The text states that GPT-4o obtains 'an F1 score of 70.9% in the Detection task at 5-shot on synthetic split' and '47.6% F1 in the same task and shot setting' on natural images, but Table 1 lists 86.0% and 41.9% for GPT-4o at 5-shot, and 47.6% is the zero-shot natural Detection value. The narrative numbers should match the table.","section":"Section 4.1, 'Better Performance on Synthetic Data'"}],"minor_comments":[{"comment":"The phrase 'F1 accuracy' is used in several places (e.g., Section 4.1); F1 and accuracy are distinct metrics and should not be combined into one term.","section":"Throughout"},{"comment":"There are multiple typos in model and attribute names: 'orinetation' (Section 2), 'V´ısion' (Supplementary Table A.2 caption), 'Idefic2' and 'Idefics3' (Table 1), and 'LLama3.1' (Section 5). These should be corrected.","section":"Section 2; Section 4; Supplementary tables"},{"comment":"The supplementary tables are extremely dense and difficult to read; splitting them by task or using a per-attribute figure would improve usability.","section":"Table A.1 and A.2"},{"comment":"The training experiment is a useful negative result, but the text says the models 'struggle to capture the salient features' without reporting the number of training steps or final validation loss; adding these details would make the experiment more reproducible.","section":"Section 5"},{"comment":"Reference [24] is cited for both P3 and O3, but the paper also cites [6] and [52] for the 7x7 grid design; please verify that the dataset source is unambiguous and that all dataset attributions are consistent.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The empirical resource is valuable and the core observation that LVLMs score around 50% F1 on natural odd-one-out detection is likely reproducible. The main risk for the journal is overstatement: the headline metric is F1 rather than accuracy, and the 'obvious to humans' premise needs a human baseline before the paper's central claim is fully supported. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection. The paper is within the journal's scope, and no circularity issue arises from the benchmark measurements themselves."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read on SalBench. The empirical core is real: across a wide range of LVLMs, detection of odd-one-out in natural images lands around 50% F1 at best, and even giving the model a bounding box or red box doesn't push GPT-4o much past that on the natural split. That is a worthwhile data point for anyone building or relying on LVLMs for perceptual grounding.\n\nWhat's genuinely new: the three task formulations (detection, referring, visual referring) are a natural way to separate visual from language grounding, and the few-shot results are interesting because they show that in-context examples don't help and sometimes hurt. The training experiment in Section 5 is also a legitimately useful negative result: synthetic saliency data doesn't transfer to the natural split, even when the model is fine-tuned on generated images that look like the test set.\n\nThe soft spots are real but mostly fixable. The biggest is the missing human baseline. The abstract and intro lean hard on 'obvious' and 'trivial for humans,' but no human subjects were run on SalBench. The supplementary actually concedes that the focus attribute is hard for humans too, which undercuts the headline. If human-human agreement on multi-label attribute sets is well below 100%—and for focus, it probably is—the human-model gap shrinks. That doesn't erase the result; it changes the claim from 'can't see the obvious' to 'struggle on odd-one-out under exact-match multi-label scoring.'\n\nThere are also several smaller issues: the abstract calls the F1 score 'accuracy' (matching accuracy in the supplementary is 45.2%, not 47.6%); the synthetic task description lists 'shape' where P3 actually uses size; and the 'open-source' claim is premature since no code or data is on the website yet. The multi-label exact-match evaluation is strict, so per-category F1 scores matter for interpretation; the paper does report those, which helps.\n\nThe core observation survives my skepticism: performance is low across models, and the size/focus categories are genuinely hard for current LVLMs. What doesn't survive is the unmeasured human 'obviousness' framing. I'd send this to a serious referee, with the expectation that revisions should add a human baseline on a subset, correct the metric labeling, and release the benchmark artifacts. The paper is useful for the LVLM evaluation and agent/robotics communities.\n\nMy bottom line: worth engaging, worth citing as a benchmark, but the authors need to either test humans or soften the language.","headline":"SalBench is a useful odd-one-out benchmark with a solid negative result, but the 'obvious to humans' claim is undersupported without a human baseline.","tokens_in":34362,"tokens_out":3872,"would_cite":true,"duration_ms":45179,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that state-of-the-art vision-language models largely fail to detect visually obvious odd-one-out features, with the strongest model scoring only 47.6% F1 on natural images.","keywords":["saliency benchmark","large vision-language models","odd-one-out detection","visual search","low-level visual features","feature integration theory","perceptual alignment","LVLM evaluation"],"falsifier":"Run the same three SalBench tasks with human participants on a sample of the O3 natural images and compare their feature-naming accuracy to the 47.6% F1 reported for the strongest model; if human accuracy on hard focus and size categories is far below near-perfect, the obviousness premise fails. Conversely, a model that reaches human-level accuracy on the natural split would refute the claim that current LVLMs cannot see the obvious.","tokens_in":33447,"feed_emoji":"👀","tokens_out":7266,"duration_ms":74043,"temperature":0.7,"pith_summary":"The paper sets out to test whether large vision-language models can detect the low-level visual features that make an object conspicuously different from its surroundings, such as a large circle among small ones or a red object among green ones. It introduces SalBench, a benchmark built from synthetic 7x7 search arrays and natural odd-one-out photos, with three tasks: naming the odd feature, naming it given a text bounding box, and naming it given a red-box highlight. Across fourteen models, the paper finds that even the strongest evaluated model reaches only 47.6% F1 on natural-image detection, and no model substantially exceeds half on natural images. The authors argue this reveals a structural gap between LVLMs and human visual attention, and that the bottleneck lies more in the vision encoder than in the language model's knowledge of saliency.","feed_headline":"Even GPT-4o misses the visually obvious at 47.6%","feed_subtitle":"On natural odd-one-out images, leading vision-language models score below half on simple color, size, and orientation detection.","key_machinery":"The central object is SalBench, a benchmark that repurposes the P3 synthetic visual-search arrays (2,514 seven-by-seven grids varying in color, orientation, or size) and the O3 natural odd-one-out photos (2,001 images with distractors and a singleton differing in color, shape, size, focus, orientation, location, or pattern). The three tasks — Odd-One-Out Detection, Referring Odd-One-Out, and Visual Referring Odd-One-Out — are designed to separate pure perception from language-guided attention: detection requires spotting the odd feature without local cues, referring gives a text bounding box, and visual referring highlights the target with a red box. Multi-label classification with exact-match accuracy and per-category F1 is the scoring mechanism, and a feature-integration-theory prompt supplies the closed set of feature categories. The key diagnostic machinery is a two-part ablation: an LLM knowledge test shows language models know feature integration theory well, while a zero-shot retrieval test on vision embeddings shows the visual representations are not discriminative enough, locating the failure in perception rather than reasoning.","core_discovery":"The central discovery is that current large vision-language models largely fail at odd-one-out perception when the target differs from distractors in low-level attributes such as size, orientation, focus, or pattern, and that the difficulty is not a matter of language understanding. On natural images, the strongest model scores 47.6% F1 on the detection task and most models score between 20% and 45%; on synthetic grids the same models reach 70–90%, showing that simplified arrays are far easier. Models are strongly biased toward color, which is near-ceiling on easy synthetic items, while size and focus degrade sharply at hard difficulty; adding more distractors monotonically lowers accuracy. A training experiment with one million synthetic saliency image-caption pairs does not fix the problem, and a vision-backbone retrieval probe shows the evaluated vision embeddings carry little discriminative information for the natural-image categories. The paper concludes that the perceptual alignment of LVLMs with human attention is still missing.","pith_inferences":["Beyond the paper, a human-baseline study on the O3 natural images would test the central premise of obviousness; if humans are far from perfect on the hard focus and size categories, the headline gap shrinks.","The same benchmark could be used with a different training objective, such as contrastive learning on odd-one-out pairs, to test whether the vision-encoder bottleneck can be fixed without new architectures.","The distractor-count and difficulty-level trends suggest a scaling law for perceptual search that might predict when an LVLM will fail in the wild, which would be useful for safety cases in agents.","The low retrieval accuracy of the evaluated vision encoders on natural images implies that current vision-language pretraining objectives under-weight precisely the low-level features human attention uses, a hypothesis worth testing on newer backbones."],"forward_implications":["SalBench can serve as a perceptual alignment probe for any LVLM, complementing benchmarks that measure high-level reasoning.","Current LVLMs deployed as robotic or GUI agents should not be assumed to notice salient scene changes, such as a person being swapped in a conversation.","Providing bounding boxes or red-box highlights only marginally helps on natural images, so the limitation is not primarily in following referring instructions.","Few-shot examples are not a reliable remedy and sometimes degrade performance, meaning simple in-context prompting will not close the gap.","The near-ceiling color performance alongside poor size and focus performance indicates that models rely on low-level pixel color rather than on relational geometry, pointing to a specific architectural weakness."],"supporting_citations":[{"why":"Supplies the P3 synthetic grids and O3 natural odd-one-out images that SalBench augments with language instructions.","marker":"[24]"},{"why":"Defines feature integration theory, from which the benchmark draws the low-level feature categories and the assumption of pop-out saliency.","marker":"[49]"},{"why":"The model that anchors the headline 47.6% F1 result on natural-image detection.","marker":"[42]"},{"why":"Vision encoder used in the zero-shot retrieval diagnosis and in the saliency-trained variants.","marker":"[43]"},{"why":"Second vision encoder used in the retrieval diagnosis and in the trained variants.","marker":"[58]"},{"why":"Provides instruction-tuning data in the training experiment and is used as a vision-centric benchmark comparison.","marker":"[47]"},{"why":"The training recipe reused for the saliency-trained model variants.","marker":"[31]"},{"why":"Motivates the paper's premise that low-level perception can be harder than high-level reasoning.","marker":"[40]"}],"fun_headline_variants":["Even GPT-4o fails at spotting the obvious odd-one-out","Vision-language models miss the obvious: GPT-4o at 47.6%","Why can't GPT-4o see the big circle? SalBench reveals weakness","Synthetic easy, natural hard: LVLMs fail real-world saliency","Training on 1M images doesn't fix LVLMs' saliency blindness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's central premise is that the odd-one-out targets are visually obvious to humans; no human observers were tested on the natural images, so if humans also find some of these images ambiguous, the claim that models cannot see the obvious would be overstated.","fun_headline_variants_meta":{"raw":{"variants":["Even GPT-4o fails at spotting the obvious odd-one-out","Vision-language models miss the obvious: GPT-4o at 47.6%","Why can't GPT-4o see the big circle? SalBench reveals weakness","Synthetic easy, natural hard: LVLMs fail real-world saliency","Training on 1M images doesn't fix LVLMs' saliency blindness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000567,"raw_usage":{"total_tokens":2674,"prompt_tokens":921,"completion_tokens":1753,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":1664}},"tokens_in":537,"tokens_out":1753,"duration_ms":14301,"temperature":1.0,"reasoning_tokens":1664,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:40:45.201677+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three SalBench tasks with human participants on a sample of the O3 natural images and compare their feature-naming accuracy to the 47.6% F1 reported for the strongest model; if human accuracy on hard focus and size categories is far below near-perfect, the obviousness premise fails. Conversely, a model that reaches human-level accuracy on the natural split would refute the claim that current LVLMs cannot see the obvious.","supporting_citations":[{"cited_title":"A feature- integration theory of attention","cited_arxiv_id":null,"evidence_quote":"Defines feature integration theory, from which the benchmark draws the low-level feature categories and the assumption of pop-out saliency."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The model that anchors the headline 47.6% F1 result on natural-image detection."},{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Vision encoder used in the zero-shot retrieval diagnosis and in the saliency-trained variants."},{"cited_title":"Sigmoid loss for language im- age pre-training","cited_arxiv_id":null,"evidence_quote":"Second vision encoder used in the retrieval diagnosis and in the trained variants."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"The training recipe reused for the saliency-trained model variants."},{"cited_title":"Mind children: The future of robot and human intelligence","cited_arxiv_id":null,"evidence_quote":"Motivates the paper's premise that low-level perception can be harder than high-level reasoning."}],"review_version":1}