{"id":"594950b4-484a-4f06-9ad9-f36e59edcb8f","arxiv_id":"2607.20092","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new taxonomy and 1,500-item dataset, ENTRAP-VL, lets researchers measure whether vision-language models are entrained by textual and visual context separately.","lead":"This paper presents ENTRAP-VL, a manually curated 1,500-item dataset and taxonomy for measuring how vision-language models get pulled by irrelevant textual or visual context. It argues that studying this 'contextual entrainment' in multimodal models needs a purpose-built, two-stream instrument, and provides one.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ENTRAP-VL's central contradictory/counterfactual distinction is misapplied in the paper's own examples, and the manual curation is unvalidated; the taxonomy's instantiation is unsupported.","rationale":"The reader identified the weakest assumption as the lack of inter-annotator agreement and independent validation of the manual curation. My stress-test agrees and provides concrete evidence that this is not a hypothetical worry: the paper's own flagship figure mislabels items on the very axis the taxonomy introduces. The contradictory/counterfactual distinction is the paper's chief novel contribution and the reason a purpose-built VLM instrument is said to be needed; if the authors cannot apply this distinction consistently in their illustrative examples, the ENTRAP-VL dataset's 1,500 items are unlikely to be reliably labeled, and the evaluation protocols built on those labels lose their validity. This directly threatens the position that ENTRAP-VL is a valid instrument. However, the flaw is addressable through an independent annotation study and possible correction of the dataset, so conditional acceptance rather than rejection is the appropriate outcome. The reader's verdict already reflects this, so no change is recommended.","tokens_in":13775,"tokens_out":9877,"duration_ms":92358,"concrete_test":"Recruit two or three annotators who were not involved in curation. Give them the taxonomy documentation (Sec. 4) and ask them to label a stratified random sample of at least 200 items from each stream into the eight/three conditions, without seeing the authors' labels. Compute chance-corrected agreement on the full label and separately on the veracity axis (true/contradictory/counterfactual). The concern lands if agreement on the veracity axis is below κ = 0.6, or if a majority of annotators judge Fig. 1's 'There are dragons' as counterfactual and 'the grass is blue' as contradictory, contradicting the paper. If κ is high and the figure examples are judged as the authors label them, the concern fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that ENTRAP-VL is a purpose-built instrument for dual contextual entrainment depends on the dataset actually instantiating the taxonomy, especially the novel veracity axis. The only implementation evidence is the authors' manual curation, which Sec. 7 states is done 'by co-authors and reviewed by the leads' with no inter-annotator agreement or independent check. This is not merely a missing statistic: the paper's own illustrative data violate the taxonomy's core distinction. In Fig. 1, the Taj Mahal item lists 'There are dragons' under relatable_contradictory (false of this scene but possible in the world), yet dragons are not possible in the world; under the paper's own definitions this is counterfactual (false in the world). Similarly, 'the grass is blue' for the dog item is listed under relatable_counterfactual, but blue grass exists, so if the depicted grass is green the statement is contradictory, not counterfactual. If the authors cannot reliably apply the contradictory/counterfactual boundary in the figure designed to exhibit the taxonomy, the 1,500-item dataset very likely contains systematic label noise in exactly the dimension the paper introduces. Because the protocols and comparisons in Sec. 6 presuppose that conditions are correctly labeled — e.g., the within-item variance 'internal reliability estimate' assumes the three statements in a condition are homogeneous — the measurement logic is undermined. This is a validity concern, not a matter of current consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that studying contextual entrainment in vision-language models requires a purpose-built, dual-modality instrument, and it presents ENTRAP-VL as that instrument: a manually curated 1,500-item dataset organized along two axes (association and veracity), with eight textual-stream conditions and three visual-stream conditions. The paper explicitly declines to measure entrainment in any model and instead contributes the taxonomy, the dataset, and the evaluation protocols that the dataset is designed to support. The central claim is that ENTRAP-VL's construction—manual, item-aware, taxonomically structured, and split into separate textual and visual streams—is necessary to investigate VLM entrainment rigorously, and that the dataset realizes this requirement.","tokens_in":14083,"tokens_out":6458,"duration_ms":63205,"significance":"If the dataset and taxonomy are validated, this would be a genuinely useful resource: the textual/visual stream separation is a real conceptual advance over unimodal distraction benchmarks, and the scene-relative contradictory/counterfactual distinction is a novel and interesting axis that has no direct counterpart in text-only world-knowledge formulations. The design invariants—short-form triggers always being wrong answers, lexical disjointness in the visual stream, and explicit attention to confounds such as watermarking and tonal register—show careful thought, and the paper is appropriately modest in not claiming to have measured entrainment in any model. The significance, however, is conditional: the paper's central artifact is an unvalidated dataset, and the paper's own illustrative examples suggest that the very veracity distinction it introduces is being applied inconsistently. The contribution will become credible only with direct evidence that the annotated conditions reliably instantiate the taxonomy.","major_comments":[{"comment":"The dataset is the instrument, but no evidence is provided that the eight textual and three visual conditions reliably instantiate the taxonomy. Section 7 states that judgments of relatability, local-vs-global falsehood, and clean distractors are made by co-authors and reviewed by the leads, and the only listed checks are structural (schema, resolution, watermarking, identifier uniqueness, disjointness). There is no pilot, no inter-annotator agreement, and no independent audit of label correctness. This matters for the measurement logic in Sec. 6: the proposed 'internal reliability estimate' assumes that the three statements within a condition are homogeneous and correctly labeled. The revision should include at least a sampled inter-annotator agreement study and a full label audit, especially on the veracity axis.","section":"Sec. 7; Sec. 6"},{"comment":"The boundary that makes the taxonomy novel is contradicted by the paper's own examples. In Sec. 4.2, 'the grass is blue' is listed under relatable_counterfactual as world-level-false, but blue grass exists (e.g., blue fescue); if the depicted grass is green, this is relatable_contradictory by the paper's own definition (false of the scene, possible in the world). Conversely, Fig. 1 places 'There are dragons' under relatable_contradictory, yet dragons are not possible in the natural world and should be relatable_counterfactual. If the illustrative examples cannot be reliably classified, the 1,500-item dataset likely contains systematic label noise in exactly the dimension the paper introduces. The authors need an explicit disambiguation protocol and a re-annotation pass, with post-hoc examples consistent with that protocol.","section":"Sec. 4.2; Fig. 1"},{"comment":"The visual stream's reduction from eight to three conditions depends on the claim that 'A photograph, by construction, is world-knowledge-consistent.' This is not tenable as stated: photographs can be staged, edited, or depict unusual or rare states of affairs. The subsequent inference that the veracity axis 'collapses in imagery' is load-bearing for the dual-modality asymmetry the paper emphasizes. Unless the curation protocol explicitly excludes manipulated or atypical images and verifies this on the released set, the visual stream is not actually protected from the veracity distinctions the taxonomy assigns to the textual stream. Please specify a concrete exclusion rule and report compliance on the released images.","section":"Sec. 4.3"},{"comment":"The paper calls the per-item variance across the three statements an 'internal reliability estimate.' Variance across three non-parallel statements is at best a dispersion measure, not reliability in a psychometric sense; reliability requires parallel forms, repeated measures, or an appropriate internal-consistency statistic. If the protocol is to support item-level measurement, the design needs either more controlled repetition or a clearly defined estimator. As written, this claim overstates what the dataset and protocol can support.","section":"Sec. 6 (Measurements per item and per condition)"}],"minor_comments":[{"comment":"Typo: 'doesn't satisfies' should be 'doesn't satisfy'.","section":"Fig. 2 caption"},{"comment":"The figure is very dense and the alignment between condition names and the two sub-rows is hard to follow. Consider a clearer layout that visually groups the full-sentence, short-form, and counterfactual conditions.","section":"Fig. 1"},{"comment":"The phrase 'is not straightforwardly comparable to it in the same veracity sense' is awkward and could be rewritten for clarity.","section":"Sec. 4.3"},{"comment":"Reference [6] is a 2026 preprint by the first author. If it is not load-bearing, consider removing it or explicitly explaining its connection; as written it appears to be a self-citation to unpublished work.","section":"References"},{"comment":"The two streams are called 'mirror streams,' but the visual stream has three conditions while the textual stream has eight. The text already explains this asymmetry, but the word 'mirror' may mislead readers; consider 'complementary streams.'","section":"Sec. 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a position/resource paper rather than an empirical study, so the absence of model results is not a defect by itself. However, the central artifact is the dataset, and its validity is currently unsupported. The paper's own examples misclassify the core contradictory/counterfactual distinction, which makes the need for validation concrete rather than procedural. I am recommending major revision rather than rejection because the taxonomy is coherent and the problems are fixable with additional annotation work, an IAA study, and a label audit. If the authors cannot supply such validation, the paper should not be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this is a position-plus-resource paper, not an empirical study. The core contribution is a taxonomy: eight textual conditions and three visual conditions for contextual entrainment, built on two axes (association and veracity). The genuinely new piece is the 'contradictory' level — false of the depicted scene but possible in the world — which has no real unimodal counterpart. The dual-stream design is also well motivated, and the authors are honest that they are not claiming to measure entrainment in any model.\n\nWhat the paper does well: the writing is clear, the scope is carefully delimited, and the design invariants are thoughtful — short-form triggers are always wrong answers, distractor names are lexically disjoint from the answer, and images are tonally matched. The relation to Niu et al. is precisely drawn, and the taxonomy is internally coherent on the page.\n\nThe soft spots are real, and the stress-test is right about Fig. 1. 'There are dragons' above the Taj Mahal is listed as contradictory, but dragons are not possible in the world, so under the paper's own definition it is counterfactual. 'The grass is blue' for the running dog is listed as counterfactual, but blue grass exists; if the depicted grass is green, that statement is contradictory. If the authors cannot reliably apply their central new boundary in the figure meant to exhibit it, the 1,500-item dataset likely contains systematic label noise in exactly that dimension. The curation section admits judgments were made by co-authors with no inter-annotator agreement, and no pilot is reported. The visual stream also rests on an unexamined claim that photographs are world-knowledge-consistent by construction. These are validity concerns, not minor quibbles, though they are fixable.\n\nThe no-model-results choice is fine for a resource paper, but it means the impact depends entirely on the resource being trustworthy. At 1,500 English-only items, it is a focused diagnostic, not a large benchmark.\n\nWho this is for: researchers who want a controlled probe for measuring entrainment in VLMs. If the dataset is corrected and released with reliability data, this could be genuinely useful. Right now, I would want to see a revision that fixes the examples and reports at least a small inter-annotator agreement study before building on it.\n\nRecommendation: send it to peer review — the taxonomy deserves refereeing — but the referee should require the example corrections and some validation evidence before acceptance.","headline":"A thoughtful taxonomy for dual-modality entrainment, but the paper's own figure misapplies its central new distinction, and the dataset has no reliability evidence yet.","tokens_in":14579,"tokens_out":2365,"would_cite":false,"duration_ms":23645,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Studying contextual entrainment in vision-language models requires a purpose-built dual-modality probe, and ENTRAP-VL supplies one.","keywords":["contextual entrainment","vision-language models","taxonomy","multimodal evaluation","dataset","distraction","veracity","behavioral probing"],"falsifier":"Have a second, independent curation team re-label a random sample of the 1,500 items using only the published taxonomy and schema, without seeing the intended labels; substantial disagreement with the released labels (for example, Cohen's kappa well below 0.8) would show that the manual judgments do not consistently instantiate the taxonomy and the paper's central resource claim would fail.","tokens_in":13636,"feed_emoji":"🖼️","tokens_out":5994,"duration_ms":56992,"temperature":0.7,"pith_summary":"Contextual entrainment is the tendency of a model to let auxiliary context in its input pull its output, even when the context is irrelevant, false, or meaningless. Prior work identified and mechanistically explained this in text-only language models; this paper argues that in vision-language models the phenomenon is genuinely different: it can be driven independently by textual and by visual context, and a depicted scene creates a distinction between context that is false of the scene but possible in the world and context that is impossible outright. To make that claim actionable, the paper builds ENTRAP-VL, a manually curated 1,500-item dataset with a taxonomy of eight textual context conditions and three visual image conditions, and publishes evaluation protocols. The paper deliberately reports no model results; its contribution is the instrument and the demonstration that a purpose-built probe, rather than a ported text benchmark, is needed.","feed_headline":"1500-item probe separates textual from visual entrainment in VLMs","feed_subtitle":"A dual-stream probe that isolates each pull channel — and tests a scene-relative falsehood no text benchmark can.","key_machinery":"The load-bearing component is the taxonomy itself, specified on two axes: association with the item at hand (relatable vs. random) and relationship to truth (true, contradictory, counterfactual). The taxonomy yields eight textual context conditions and three visual image conditions, and it is realized through the dual-stream construction: in the textual stream the image and query are fixed while injected text is varied, and in the visual stream the query is fixed while the candidate image is varied. A key design invariant is that every short-form distractor is constructed to be a wrong answer, so any pull toward it is cleanly attributable to entrainment rather than coincidental correctness.","core_discovery":"The paper argues that contextual entrainment becomes a dual phenomenon in vision-language models: it can be driven independently by textual context and by visual context, which is structurally different from the single-channel phenomenon in language models. It also argues that a perceivable scene introduces a veracity category — contradictory, false of the depicted scene yet possible in the world — that has no counterpart in text-only, world-knowledge-only settings. To support rigorous study of this, the paper introduces ENTRAP-VL, a manually curated 1,500-item dataset organized by a two-axis taxonomy (association: relatable vs. random; truth: true, contradictory, counterfactual), split into","pith_inferences":["If the instrument works as intended, a natural next question the paper leaves open is whether a given model's pull strengths differ between the textual and visual channels on matched items; the dataset is built to detect exactly that, but the paper does not predict the direction.","An entrainment effect isolated in the contradictory condition would be a signature of vision-specific entrainment, not an artifact of world-knowledge conflict; this could connect the probe to hallucination and sycophancy studies, which the paper lists as adjacent but does not integrate.","The weakest link is curation consistency; a natural first use of the released documentation is a replication study in which independent curators re-label a random sample, giving the dataset a reliability score it currently lacks.","The two-axis taxonomy could be exported beyond vision-language pairs, for example to audio-language question answering, wherever a perceptual referent gives rise to a local-versus-global falsehood distinction."],"forward_implications":["Researchers can measure textual entrainment and visual entrainment separately in the same model, and compare their strengths, because the dataset separates the two channels by construction.","The contradictory condition allows a test of scene-relative falsehood distinct from counterfactual impossibility, a distinction that text-only benchmarks cannot express.","Three statements per condition enable item-level estimates of entrainment through within-item variance, rather than only corpus-level aggregates.","The short-form distractor contrast isolates the effect of scene association while keeping the trigger content as a bare entity, and the form contrast (full sentence vs. bare word) isolates sentential scaffolding.","The dataset is intended strictly for behavioral evaluation, not training or fine-tuning; using it for training would contaminate it as an evaluation instrument."],"fun_headline_variants":["New probe exposes how VLMs get dragged by text and image context","Dual-context trap: 1500-item probe tests VLM entrainment","ENTRAP-VL: 1500 items to catch VLM context biases","Probe isolates text vs image pull in vision-language models","Trap for VLMs: 1500 items test visual and textual context"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The dataset is the instrument, and its validity rests on the co-authors' manual judgments of relatability and of local versus global falsehood, made without an independent agreement check, being consistently correct across all 1,500 items.","fun_headline_variants_meta":{"raw":{"variants":["New probe exposes how VLMs get dragged by text and image context","Dual-context trap: 1500-item probe tests VLM entrainment","ENTRAP-VL: 1500 items to catch VLM context biases","Probe isolates text vs image pull in vision-language models","Trap for VLMs: 1500 items test visual and textual context"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000165,"raw_usage":{"total_tokens":1151,"prompt_tokens":876,"completion_tokens":275,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":177}},"tokens_in":620,"tokens_out":275,"duration_ms":3387,"temperature":1.0,"reasoning_tokens":177,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T10:46:54.587799+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a second, independent curation team re-label a random sample of the 1,500 items using only the published taxonomy and schema, without seeing the intended labels; substantial disagreement with the released labels (for example, Cohen's kappa well below 0.8) would show that the manual judgments do not consistently instantiate the taxonomy and the paper's central resource claim would fail.","supporting_citations":[],"review_version":1}