{"id":"291711d8-ac9f-4baf-96c6-2428f3d7d1bd","arxiv_id":"2501.03957","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Vision language models align poorly with human judgments of image relevance, but occasionally produce value-aware responses that human annotators missed.","lead":"This paper tests whether five vision language models identify the same 'element of relevance' in twelve home images that human annotators choose. Alignment was low, with the best model scoring 0.42, and the authors argue the models show untapped potential for values-aware AI.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Annotator agreement is unreported and the majority-vote reference is unstable, so the low alignment scores may not measure model-human misalignment.","rationale":"The paper's aim is to measure alignment between VLM outputs and human judgments of relevance. For that measurement to be meaningful, the human reference must be reliable. The reference is a majority vote over 14 annotators, but the paper gives no agreement statistic and explicitly resorts to random tie-breaking. The dog-image example shows a plurality of only 5/14, making the reference weakly supported. Because each model response is scored as binary against this single reference, low scores could simply reflect annotator ambiguity rather than model deficiency. The paper's own observation that models sometimes outperformed the majority of annotators further shows that the reference is not treated as a gold standard, which is internally inconsistent. The reader's weakest assumption identifies exactly this issue, so I agree. The proposed re-scoring test would directly show whether the conclusion changes when multiple reasonable answers are accepted. Since the reader already rendered a conditional verdict based on this concern, no verdict change is needed.","tokens_in":6633,"tokens_out":3037,"duration_ms":29113,"concrete_test":"Re-score all 180 model responses (5 models x 12 images x 3 completions) against the full distribution of normalized annotator responses rather than the single majority element: count a response as correct if its element matches any annotator's element that was named by at least 2 of the 14 annotators. If the recalculated alignment scores rise substantially above 0.42 or reorder the models, the low-alignment conclusion is an artifact of the unstable reference; if scores and ranking are essentially unchanged, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim that VLMs are poorly aligned with humans depends on the correctness of the single 'element of relevance' reference per image derived in Section 2 from 14 annotators. The paper never reports inter-annotator agreement, and the example it gives (dog image) has only 5/14 annotators agreeing on 'dog is sick'; other images required random tie-breaking. Section 3.2 then scores every model response as binary correct/incorrect against this one arbitrarily selected reference. If multiple elements are plausibly relevant, or if annotator disagreement is high, the Table 2 scores (0.14-0.42) reflect reference noise rather than model misalignment. The paper's own Section 4.2 reports cases where models gave 'a more satisfying result than the annotators' (e.g., concern for the woman's health after slipping), which directly undermines treating the majority response as the ground truth. Without a reliability check on the annotation procedure, the headline conclusion is not robust.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a small empirical study of whether current vision-language models (VLMs) align with human annotators in identifying the 'element of relevance' in twelve AI-generated home-scene images. Fourteen annotators each named the relevant element for every image, and the most common response was taken as the reference. Five models (GPT-4o and four LLaVA variants) were then prompted with the same task, each producing three answers per image, and each answer was scored as correct/incorrect against the single majority reference. The paper reports alignment scores of 0.14 to 0.42, with LLaVA-34B highest, and a Cochran's Q test giving p = 0.077. The discussion analyzes the response types and argues that the models show potential for detecting value-laden elements despite the low alignment.","tokens_in":6752,"tokens_out":6241,"duration_ms":61493,"significance":"The question asked is timely: VLMs are increasingly deployed in social robotics and assistive technology, where knowing what to attend to matters as much as generating fluent text. The paper contributes a transparently described small benchmark, explicit model versions, and a qualitative error analysis showing that models sometimes produce 'more aware' answers than the majority of human annotators. These qualitative observations are genuinely interesting and could motivate follow-up work. The main quantitative claim, however, is not yet supported: the reference labels are unstable, the binary scoring is too coarse, and the model ranking is statistically indistinguishable. If the measurement issues are fixed, the paper would be a useful preliminary study; in its current form, the headline conclusion outruns the evidence.","major_comments":[{"comment":"The single-reference evaluation does not measure model-human alignment reliably. The 'perceived element of relevance' in Table 1 is a majority label from 14 annotators, but no inter-annotator agreement is reported, and Section 2 states that ties are broken randomly. For the dog image, only 5/14 annotators chose the dog's health; the other nine answers were split among four other categories, so the reference is a plurality, not a consensus. Section 3.2 then awards a binary score to every model answer against this one label, which conflates model-human disagreement with label noise. The problem is visible in the controls: a model saying 'none' for the woman-eating dinner image is scored wrong even though the image was designed to contain no element of relevance, and Section 4.2 itself reports that 16% of misaligned answers were 'more aware' than the majority annotation. Please report annotator agreement (e.g., Fleiss' kappa) and evaluate with multiple accepted labels or a soft scoring rule.","section":"Section 2; Section 3.2"},{"comment":"The model ranking in Table 2 and the abstract's phrase 'LLaVA 34B showing the highest performance' are not statistically supported. The paper's own Cochran's Q test returns p = 0.077, so the observed differences among the five models are within sampling noise. The caveat in Section 3.3 is appropriate, but the abstract and conclusion do not carry it. The paper should either remove the best-model claim or support it with pairwise comparisons, effect sizes, and confidence intervals. In addition, the Cochran's Q implementation is underspecified: with three completions per image per model, the independence assumptions of the test are not obvious and should be described.","section":"Section 3.3; Abstract"},{"comment":"The title and the Section 4.3 discussion frame the contribution as 'VLMs as values detectors,' but the experiment does not operationalize 'value' or 'value-laden.' The prompt asks only for the element that needs attention, the annotation procedure does not ask about values, and the response-type taxonomy in Section 4.1 was constructed after seeing the annotator data. The value-laden interpretation rests on selected examples (smoking near a child, medicines, slipping) and on a general citation about values and perception [8], rather than on a measured value-detection construct. This section should be reframed as an exploratory hypothesis, or a separate validation step for value detection should be added.","section":"Section 4.3; Title"}],"minor_comments":[{"comment":"The model is called 'LLaVA 34B' in the abstract and Table 2 but 'LLaVA 36B' in Section 4.2 and the Conclusion; please standardize the name.","section":"Abstract; Section 4.2; Conclusion"},{"comment":"The prompt was chosen after initial testing on one image, which risks prompt overfitting; this should be listed as a limitation, especially because the paper later suggests that better prompts would improve alignment.","section":"Section 3.1"},{"comment":"The paper keeps only the first sentence of each model completion; if any answer was a list, a conditional statement, or a qualification, this truncation could remove the element actually identified, and the paper does not report how often truncation occurred.","section":"Section 3.2"},{"comment":"The caption does not state how the 95% confidence intervals were computed (e.g., binomial intervals over the 36 binary responses per model), and the fact that the intervals overlap is itself further evidence that the model ordering is not reliable.","section":"Figure 2"},{"comment":"Reference [3] is attributed to 'Daniel, K.'; the correct author is Kahneman, D., and the title is Thinking, Fast and Slow.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a short exploratory study with 12 images and 14 annotators. The main quantitative conclusion is not yet robust, but the problems are addressable within the scope of the paper: report inter-annotator agreement, evaluate with multiple accepted labels or soft scoring, describe the statistical test more carefully, and soften the 'values detector' framing. With those changes, the paper could make a modest but useful empirical contribution to the VALE workshop proceedings."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Carl,\n\nYou asked about 2501.03957. It's a small exploratory study that applies five VLMs to a 12-image dataset of home scenes, with the task of identifying the 'element that needs attention.' Fourteen annotators provided reference answers, and the models get scored on whether they match the majority vote. The top model, LLaVA 34B, scores 0.42, and a Cochran's Q gives p=0.077, so no significant ranking.\n\nThe new thing here is the dataset and the specific comparison. Twelve generated images is small, but the scenarios are chosen to be value-laden (sick dog, crying child, smoking near a child), and the authors carefully sort annotator responses into four types. The observation that models sometimes gave more caring answers than the human majority—expressing concern for the woman's health after a fall, or noticing the child near the smoker—is the most interesting result. The authors are also honest about the non-significant test and about their own expectations being violated.\n\nThe main weakness is the ground truth. The reference is the most common answer among 14 annotators, with no inter-annotator agreement reported and ties broken randomly. A dog image has only 5/14 agreeing on 'the dog is sick.' Scoring every model answer as binary correct/incorrect against that single reference makes the alignment scores partly a measure of reference noise. The paper itself admits that in 16% of misaligned answers, the model gave a more aware response than the majority of annotators. That directly undercuts the headline claim that the models are misaligned. Also, the prompt and temperature were not systematically varied, there are no baselines beyond the models themselves, and no code/data are provided.\n\nStill, the central observation—that raw VLMs don't consistently latch onto the same salient element as humans do—probably holds up, even if the exact scores are soft. The qualitative taxonomy of response types is a useful contribution.\n\nWho should read this: people working on VLM evaluation for social robotics or assistive systems. It's a workshop-level contribution that deserves a serious referee, but with major revision. I'd ask for a reliability analysis of the annotations, a softer scoring scheme that allows multiple relevant elements, and the data/code. If those are addressed, the paper would be a solid empirical data point.\n\nI'd bring it to a reading group as an example of how fragile VLM-human alignment evaluation can be, and I'd probably cite it if I were writing about VLM evaluation pitfalls.\n\nMy recommendation: accept for review, not desk reject, but condition acceptance on substantial revisions.","headline":"A small, honest exploratory study whose quantitative claims are fragile due to noisy ground truth, but whose qualitative observations about VLM sensitivity to values are worth attention.","tokens_in":7294,"tokens_out":2849,"would_cite":true,"duration_ms":27417,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that current vision-language models are not aligned with human annotators in detecting the element of relevance in home images, with the best model, LLaVA 34B, reaching only 0.42 alignment, and argues that value-aware…","keywords":["vision-language models","value detection","human-AI alignment","image relevance","LLaVA","GPT-4o","home environment scenarios","annotator agreement"],"falsifier":"Compute inter-annotator agreement on the 168 responses, for example with Fleiss' kappa. If agreement is low across the twelve images, the single \"element of relevance\" per image is not a stable ground truth, and the observed scores from 0.14 to 0.42 would not establish human-model misalignment; conversely, high agreement would support the paper's interpretation of the scores.","tokens_in":6392,"feed_emoji":"🖼️","tokens_out":4431,"duration_ms":41336,"temperature":0.7,"pith_summary":"This paper asks whether vision-language models and human viewers agree on what deserves attention in an everyday scene, and it answers with a measured no. Using twelve generated home images and fourteen annotators, the authors define for each image the element of relevance that most annotators named, then score five models (GPT-4o and four LLaVA variants) against that same reference. The best model, LLaVA 34B, matches the human majority only 42% of the time, and the spread across models is not statistically significant. The authors argue that the failure is partly a descriptive bias in the models and that value-aware fine-tuning and more explicit prompts could turn VLMs into useful detectors of value-laden situations for social robotics and assistive technology.","feed_headline":"Best VLM still scores only 0.42 on spotting what matters","feed_subtitle":"Home-scene tests show models often describe the wrong element, and the authors say value-aware prompts and fine-tuning could close the gap.","key_machinery":"The evaluation rests on a small bespoke dataset: twelve diffusion-generated images of domestic situations, ten designed to carry value-relevant content and two controls. Fourteen annotators each named the element needing attention, and the most frequent answer (with random tie-breaking) became the reference element of relevance. The models were given the same request in the same prompt, their first sentence was kept, and each answer received a binary score against the reference. A qualitative response-type taxonomy — descriptive, value-aware, \"none\", or comment — is what carries the interpretation of the scores, distinguishing mere misalignment from promising value detection.","core_discovery":"The central claim is that current vision-language models are not aligned with human annotators in identifying the element of relevance in images: the highest alignment score is 0.42, below the midpoint of the scale. The same data show a potential the paper treats as real: in 16% of disagreements the models gave a more value-aware answer than the human majority, noticing health risks or distress that annotators did not state, and in 29% of disagreements they said \"none\" where humans found something relevant. The paper's conclusion is that the bottleneck is training and prompting rather than the architecture: with value-laden fine-tuning data and prompts that ask the model to consider values, vision LLMs could become detectors of value-laden scenarios.","pith_inferences":["Because the reference element is a single majority answer with random tie-breaking, the alignment scores should be read relative to annotator agreement; a model that agreed with a randomly chosen annotator would also score well below 1.","A natural next experiment is to score models against the full distribution of annotator answers rather than the majority, treating any annotator-named element as acceptable; this would separate genuine value detection from disagreement about which object is central.","The value-detection hypothesis is testable directly: ask models to label the value at stake (health, safety, family bonds) in each image and compare those labels to annotators' stated concerns, bypassing the ambiguity of a single \"element\".","The same protocol could be run with real photographs and a larger, culturally balanced annotator pool to see whether the 0.42 ceiling is an artifact of generated images or of the prompt."],"forward_implications":["At current capability levels, off-the-shelf VLMs should not be trusted to focus attention on the element a human would consider relevant in a home environment.","LLaVA 34B's 0.42 score is the ceiling in this test, so any deployment relying on VLM attention in assistive or robotic systems needs a human-in-the-loop or a more targeted model.","Models answer descriptively when a value-aware answer is expected in about a third of disagreements, pointing to training-data bias toward description rather than interpretation.","Value-aware prompts and fine-tuning on value-laden images are concrete, untested-in-this-paper paths the authors predict would raise alignment.","The two control images produced a desirable outcome in 14% of misaligned answers, where models answered \"none\" while annotators described irrelevant objects."],"supporting_citations":[{"why":"Supplies the LLaVA architecture and the visual-instruction paradigm that the evaluation targets.","marker":"[7]"},{"why":"Source of the home scenarios and the social value alignment framing the study builds on.","marker":"[2]"},{"why":"Theoretical premise that personal values act as selective factors in perception, motivating the idea of VLMs as value detectors.","marker":"[8]"},{"why":"Supplies the CLIP visual encoder used by LLaVA, i.e., the visual backbone under test.","marker":"[9]"},{"why":"Supplies the LLaMA language backbone used by the LLaVA variants.","marker":"[11]"},{"why":"Prior application of VLMs as success detectors, used as motivation that VLMs can act as detectors in practice.","marker":"[4]"}],"fun_headline_variants":["Best VLM matches humans on image relevance only 42%","Vision LLMs see values humans miss, but miss key elements","VLM-human alignment at 0.42, prompting could boost it","Home-scene test: VLMs need value-aware training to align","Models spot value-laden details, but not the main relevance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reference answer for each image is whatever the largest group of fourteen annotators happened to say, with ties broken randomly and no check on whether the annotators themselves agreed; if that reference is arbitrary, the alignment scores and model ranking inherit that arbitrariness.","fun_headline_variants_meta":{"raw":{"variants":["Best VLM matches humans on image relevance only 42%","Vision LLMs see values humans miss, but miss key elements","VLM-human alignment at 0.42, prompting could boost it","Home-scene test: VLMs need value-aware training to align","Models spot value-laden details, but not the main relevance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000174,"raw_usage":{"total_tokens":1245,"prompt_tokens":874,"completion_tokens":371,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":282}},"tokens_in":490,"tokens_out":371,"duration_ms":3923,"temperature":1.0,"reasoning_tokens":282,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:42:55.398300+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute inter-annotator agreement on the 168 responses, for example with Fleiss' kappa. If agreement is low across the twelve images, the single \"element of relevance\" per image is not a stable ground truth, and the observed scores from 0.14 to 0.42 would not establish human-model misalignment; conversely, high agreement would support the paper's interpretation of the scores.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LLaVA architecture and the visual-instruction paradigm that the evaluation targets."},{"cited_title":"A., Marchesi, S., Wykowska, A., and Belpaeme, T","cited_arxiv_id":null,"evidence_quote":"Source of the home scenarios and the social value alignment framing the study builds on."},{"cited_title":"S., and McGinnies, E","cited_arxiv_id":null,"evidence_quote":"Theoretical premise that personal values act as selective factors in perception, motivating the idea of VLMs as value detectors."},{"cited_title":"W., Hallacy, C., Ramesh, A., Goh, G., Agar w al, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP visual encoder used by LLaVA, i.e., the visual backbone under test."},{"cited_title":"Llama: Open and efficient foundation language models, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the LLaMA language backbone used by the LLaVA variants."},{"cited_title":"Vision-language models as success detectors, 2023","cited_arxiv_id":null,"evidence_quote":"Prior application of VLMs as success detectors, used as motivation that VLMs can act as detectors in practice."}],"review_version":1}