{"id":"16845490-529f-48a2-b879-07d16865ff85","arxiv_id":"2505.10453","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Vision language models frequently fail to include mentioned-but-not-depicted objects in their answers, revealing a limitation in virtual object tracking.","lead":"The paper tests whether AI vision-language models can keep track of an object that is described in a prompt but not shown in the picture, such as a banana placed next to a cupcake. It finds that three current models often omit that imagined object when listing what is on the table, especially without a numeric cue.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing no-virtual-object control means low accuracies may reflect generic object-listing failures, not a virtual-object-specific deficit.","rationale":"The reader's weakest assumption concerned the unvalidated all-three-objects scoring criterion and the absence of human or chance baselines. My concern is closely related but distinct: the design lacks a no-virtual-object control on the identical images, so the observed failures cannot be attributed to virtual object processing rather than to generic object listing, counting, or instruction-following deficits. This is load-bearing because the paper's title and central claim assert a virtual-object-specific inadequacy. The concern does not overturn the reader's conditional verdict; it reinforces it. The empirical pattern is plausible and worth reporting, but the missing control experiment and human data are necessary before the claim can be accepted as established. Therefore the verdict should remain conditional rather than being upgraded to accept or downgraded to reject based on this review.","tokens_in":7074,"tokens_out":4450,"duration_ms":47884,"concrete_test":"Run the same TABLETEST 2-object images through the same three VLMs with a neutral control prompt: 'What items are on the table?' with no virtual object, scoring exactly the two visible objects in any order using the same parser and temperature settings as the reported evaluation. Compare this control accuracy against the virtual-object condition accuracy for each model and prompt variant. The virtual-object-specific claim is supported only if virtual-object accuracy is substantially lower than the visible-object control; if the two are comparable, the paper's central conclusion does not follow.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central inference is that the VLMs' failure to list all three objects demonstrates an inability to process virtual objects. That inference requires the failure to be specifically attributable to the virtual object, but the experiment provides no baseline for the same images without a virtual object. Section 2 says the candidate models were selected because they are 'capable of out-of-the-box, single-shot identification of the 64 objects in TABLETEST,' yet no identification accuracy is reported for the two visible objects in the actual stimulus set. Without a control condition such as 'What items are on the table?' on the same 2-object images, the 63%, 57%, and 22% aggregate accuracies conflate at least three failure modes: failing to name one of the two visible objects, failing to add the virtual object to the response, and failing to produce a list in the exact format required for scoring. The asserted human ceiling in Figure 2 and Figure 3 is also unsupported by any human data. If the models already struggle to list the two visible objects or to follow the listing instruction, the observed deficit is not about virtual objects at all.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper argues that vision language models (VLMs) fail at a rudimentary imaginative task: given an image of two objects and a prompt such as 'Imagine there is a banana next to the cupcake', the model is asked to list all objects on the table, where the banana is a 'virtual object' described but not depicted. The authors construct 112,896 prompt-image queries from the TABLETEST dataset (4,032 two-object images x 7 prompt formulations x 2 tenses x 2 numerical-cue variants) and evaluate three open-weights VLMs: Idefics2, InstructBLIP-Vicuna, and Llama 3.2. Aggregate accuracies are 63%, 57%, and 22%, respectively, with significant effects of prompt wording, tense, and numerical cues. The authors conclude that VLMs have inadequate ability to process virtual objects and that this reflects a broader limitation in visuospatial scene comprehension and representation updating.","tokens_in":7251,"tokens_out":7959,"duration_ms":65181,"significance":"The phenomenon investigated is timely and the experimental matrix is large and systematic. The paper makes a falsifiable claim: state-of-the-art open-weights VLMs will often fail to list an object that is described but not depicted, even when the instruction explicitly asks for all objects. If the evidence were airtight, this would be a useful benchmark result for the VLM community. The paper's strengths include the use of fixed random seeds and temperature 0 for replicability, the variation of prompt formulations, and the demonstration that superficial lexical factors (tense, numerical cues) change performance by up to 30 percentage points. However, the central attribution of the failures to virtual objects is currently under-supported: there is no control condition without a virtual object, no human baseline, no chance baseline, and no description of the scoring procedure. These omissions are load-bearing and should be addressed in revision.","major_comments":[{"comment":"The claim that models 'systematically lost track of the virtual objects' (Section 3, last paragraph) requires a control condition in which the same images are presented without a virtual-object instruction. Without such a control, the reported aggregate accuracies (63%, 57%, 22%) conflate at least three failure modes: failing to name a visible object, failing to add the virtual object to the response, and failing to produce the required list format. The selection criterion in Section 2 ('capable of out-of-the-box, single-shot identification of the 64 objects in TABLETEST') is asserted but not quantified on the actual 2-object stimulus set. Please add a no-virtual-object control condition (e.g., 'What items are on the table?') on the same images and report per-model visible-object identification accuracy.","section":"Section 2; Section 3"},{"comment":"The success criterion is 'list all three objects in any order', but the scoring procedure is never specified. The reader does not know how responses were normalized, whether exact string matching was used, whether synonyms, extra words, or multi-sentence answers were accepted, or whether any human review was conducted. Because every accuracy number in the paper depends on this decision, the scoring pipeline (including any regex, LLM-as-judge, or manual annotation protocol) must be described in detail, and the response data should be released to allow independent verification.","section":"Section 2, paragraph 4"},{"comment":"The assertion 'Humanlike performance estimated at ceiling (accuracy = 1.0)' is not backed by any human data, and no chance baseline is provided. Without human judgments, the premise that the correct answer is 'trivial' and that the prompt phrasings are unambiguous is an untested assumption; without a chance model, the low accuracies could be explained by a generic response-length or list-completion tendency of the decoders. Please run a small human study on a random subset of trials and report a chance baseline (e.g., uniform sampling from the 64-object vocabulary, or a model-free heuristic). These data are needed to support the claim that the failures are specific to virtual-object processing.","section":"Section 3, Figures 2 and 3"},{"comment":"The statistical analyses are under-specified. The text says the tests were performed on 'mean accuracies for the different objects' but does not state how many images contributed to each object's mean, how zero-answer or malformed responses were counted, or whether the Friedman test treated the three models as repeated measurements over the same 64 objects. Moreover, the tense (51% vs. 44%) and numerical-cue (62% vs. 32%) effects are reported only in aggregate; per-model results, effect sizes, and confidence intervals are needed to assess whether the effects are driven primarily by one model (e.g., BLIP at 22% overall).","section":"Section 3, second paragraph"}],"minor_comments":[{"comment":"In the abstract, 'F or example' should be 'For example'.","section":"Abstract"},{"comment":"The text reads 'Architecture sthat matched'; this should be 'Architectures that matched'.","section":"Section 2, third paragraph"},{"comment":"Reference [35] is cited for Llama 3.2 (11B), but the cited work is Touvron et al. (2023), the Llama 2 paper; either update the citation to the Llama 3 model card or correct the model name in the text.","section":"References"},{"comment":"The numerical-cue conditions mentioned in the text ('what three items are on the table?') are not shown in Table 1; include an example of a numeric-cue prompt to make the manipulation concrete.","section":"Table 1"},{"comment":"The captions are hard to parse (e.g., 'bars depict histograms across those objects, as organized by whether the object served as the leftmost object in images'); please clarify the figure design and the meaning of the y-axis labels and the percentages above each panel.","section":"Figures 2 and 3"},{"comment":"The phrase 'Friedman test, chi-squared = 120.91 p < .001' is missing a comma between the statistic and the p-value; also report degrees of freedom for the chi-squared statistics.","section":"Section 3, second paragraph"},{"comment":"The claim that 'none of the factors we tested should have affected whether a VLM can detect virtual objects' seems to conflict with the hypothesis in Section 2 (item 3) that numerical cues should boost performance; clarify whether the factors are expected to affect detection or only the response format.","section":"Section 3, third paragraph"},{"comment":"The paper does not state whether trials in which the virtual object's name matches one of the depicted objects were excluded; such trials could create ambiguity in scoring and should be handled explicitly.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"The benchmark (TABLETEST) and the virtual-object task definition come from the authors' own prior work (ref [22], which shares two of the present authors). This creates a modest circularity burden: the 'correct' answer criterion is self-defined. The absence of any human or chance baseline makes it impossible to rule out that the low accuracies reflect prompt ambiguity rather than a model deficiency. The paper also lacks a data/code availability statement, which is a concern for a benchmark-style claim. On scope, this is a short empirical paper that might fit better at a vision-language benchmarking venue or a workshop, but the editor may judge otherwise."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The virtual-object probe is a genuinely new and useful diagnostic, and the paper is worth engaging on that strength. But the headline conclusion—that VLMs are inadequate at processing virtual objects—is not yet supported, because no condition tests the same images without a virtual object.\n\nWhat is new: pairing a 2-object image with a prompt to imagine a third object, then checking whether the model lists all three, is a clean operationalization of scene-updating. The prompt manipulation is systematic, and the aggregate accuracies (Idefics2 63%, Llama3 57%, BLIP 22%) are interesting enough to warrant a second look. The finding that a numerical cue boosts accuracy from 32% to 62% is a useful surprise, and the tense effect (past over present) is worth knowing even if the explanation is speculative.\n\nThe soft spots are real and load-bearing. Most serious: there is no no-virtual-object control. Without a condition like \"what items are on the table?\" on the same 2-object images, the observed failures could come from not naming the two visible objects or not following the listing instruction, rather than from any virtual-object-specific deficit. The paper asserts these models can identify the 64 objects but never reports identification accuracy on the actual stimuli. The claimed human ceiling is also unsupported—no human data appear anywhere. Scoring is underspecified: \"lists all three objects in any order\" is not enough; we need to know whether detection was exact string matching, fuzzy matching, or an LLM judge, because that determines how much of the failure is formatting. The jump from a listing task to broad claims about visuospatial reasoning is too fast.\n\nThese are not fatal to the research direction. The core probe is clever and likely reproducible once the missing pieces are added. The paper reads as an early report rather than a finished evaluation. It deserves a serious referee, but the referee should send it back for a control condition, human data, and exact scoring rules. I would not cite it as established in the next year, but I would bring it to a reading group for the idea.","headline":"New diagnostic for VLM scene-updating, but the main claim needs a no-virtual-object control and human baselines before it can be believed.","tokens_in":7774,"tokens_out":1800,"would_cite":false,"duration_ms":18445,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"State-of-the-art vision-language models systematically fail to recognize virtual objects—objects described in a prompt but not shown in an image.","keywords":["virtual objects","vision language models","scene comprehension","spatial reasoning","hypothetical reasoning","multimodal grounding","imagination","evaluation benchmark"],"falsifier":"Re-run the battery with a control condition that separates content from format, for example asking \"Name the objects in the image, then name the object you imagined,\" and measure human accuracy on the original prompts; if model accuracy approaches ceiling under the template, or if humans fall well below ceiling, the inference that VLMs lack virtual-object tracking would be substantially weakened.","tokens_in":6876,"feed_emoji":"💭","tokens_out":6346,"duration_ms":61224,"temperature":0.7,"pith_summary":"This paper claims that state-of-the-art vision-language models cannot reliably recognize virtual objects: objects that are described in a prompt but not shown in the image. When asked to imagine a third object next to the two objects in a tabletop image and then list what is on the table, the models often fail to name all three. In 112,896 queries across three models, aggregate accuracy reached 63% for Idefics2, 57% for Llama3, and 22% for BLIP. Accuracy shifted with irrelevant prompt features: past-tense prompts outperformed present-tense ones, and adding a numerical cue such as \"what three items\" raised accuracy from 32% to 62%. A sympathetic reading is that this exposes a genuine limitation in how VLMs update their scene representations from linguistic information.","feed_headline":"Vision models lose track of objects they are asked to imagine","feed_subtitle":"Across 112,896 queries, three vision-language models named all three objects just 22–63% of the time.","key_machinery":"The load-bearing object is the \"virtual object\" probe: a prompt such as \"Imagine there is a banana next to the cupcake\" paired with a two-object image, followed by a question asking what items are on the table. The probe's work is to force a VLM to merge a textual hypothetical with the visual scene and report the union; the only success criterion is listing all three objects in any order. The evaluation machinery is the TABLETEST image set of 64 objects in two-object configurations, seven prompt templates, past and present tenses, and numerical-cue variation, yielding 112,896 queries per model at temperature zero.","core_discovery":"The central discovery is a systematic failure pattern: VLMs lose track of entities introduced by hypothetical language. The authors define a virtual object as an object within a scene that is described but not depicted, and argue that a system with robust scene comprehension should incorporate it and list it alongside the two depicted objects. Across seven prompt phrasings, two tenses, and numerical-cue variations, no tested model performed near ceiling; the best aggregate score was Idefics2 at 63%, BLIP fell to 22%, and on the \"if\" phrasing BLIP managed only 8%. The paper also shows that factors that should be irrelevant—tense and the presence of a number in the question—produced reliable differences in accuracy. The authors conclude that current VLMs do not encode scene structure in a way that supports primitive hypothetical or imaginative updating.","pith_inferences":["A plausible extension is that similar limitations will appear in instruction-following robots or navigation agents that must act on objects they have only been told about, since those systems rely on the same kind of fused text-image representations.","A direct test would collect human accuracy on the same battery; the paper assumes human performance is at ceiling but never measures it, so the true gap between human and model performance remains open.","It would be informative to probe decoding sensitivity by running the same prompts at higher temperatures or with multiple random seeds; if accuracy varies widely, part of the apparent failure could be response-sampling noise rather than a fixed representational limit."],"forward_implications":["If the claim holds, VLMs cannot be trusted for hypothetical spatial reasoning such as replacing one object with another or moving an object relative to another, because those tasks require the same scene-updating operation.","Irrelevant surface features of a prompt—tense and numeric wording—materially change accuracy, so current performance on such tasks is not a stable measure of visuospatial ability.","The numerical-cue boost from 32% to 62% suggests that models may sometimes answer by matching the count of nouns in the prompt rather than genuinely updating their representation of the scene.","The near-zero performance of BLIP on the \"if\" phrasing indicates that particular function words can essentially disable object tracking in some architectures."],"supporting_citations":[{"why":"This reference supplies the TABLETEST image set and the 64-object vocabulary used in every prompt.","marker":"[22]"},{"why":"This reference provides the annotated 3D object set from which the stimulus images were constructed.","marker":"[13]"},{"why":"This reference describes the Idefics2 architecture that is one of the three evaluated models.","marker":"[27]"},{"why":"This reference describes the BLIP architecture underlying the InstructBlip model that is one of the three evaluated models.","marker":"[28]"},{"why":"This reference is the model release behind the evaluated Llama system.","marker":"[35]"}],"fun_headline_variants":["Vision AI can't track imaginary objects","VLMs lose track of objects they're asked to imagine","Models forget imagined objects in visual scenes","Imagined objects trip up vision-language models","Vision models are bad at hypothetical objects"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes the only correct answer is to name all three objects in any order, and that every prompt is unambiguously requesting that list; if some failures come from answer formatting or wording ambiguity rather than from losing track of the imagined object, the conclusion is too strong.","fun_headline_variants_meta":{"raw":{"variants":["Vision AI can't track imaginary objects","VLMs lose track of objects they're asked to imagine","Models forget imagined objects in visual scenes","Imagined objects trip up vision-language models","Vision models are bad at hypothetical objects"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000166,"raw_usage":{"total_tokens":1205,"prompt_tokens":845,"completion_tokens":360,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":293}},"tokens_in":461,"tokens_out":360,"duration_ms":3939,"temperature":1.0,"reasoning_tokens":293,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:08:15.465147+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the battery with a control condition that separates content from format, for example asking \"Name the objects in the image, then name the object you imagined,\" and measure human accuracy on the original prompts; if model accuracy approaches ceiling under the template, or if humans fall well below ceiling, the inference that VLMs lack virtual-object tracking would be substantially weakened.","supporting_citations":[{"cited_title":"Harrison, Wallace E","cited_arxiv_id":null,"evidence_quote":"This reference supplies the TABLETEST image set and the 64-object vocabulary used in every prompt."},{"cited_title":"BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation","cited_arxiv_id":null,"evidence_quote":"This reference describes the BLIP architecture underlying the InstructBlip model that is one of the three evaluated models."}],"review_version":1}