{"id":"55645c84-8c62-4e7f-9057-3dfd37618ffd","arxiv_id":"2608.13167","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"TRAPSBench shows that across 16 vision-language models, answerability is decodable from hidden states while spontaneous abstention remains poor, pointing to an output-stage bottleneck in epistemic restraint.","lead":"A new benchmark of 1,404 paired physics videos shows that vision-language models can internally detect when an answer cannot be known, yet still answer anyway. The paper demonstrates that the failure is in expressing restraint rather than in perceiving uncertainty, using hidden-state probes and activation steering.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Steering lacks null-direction controls: the void direction may be a perceptual or formatting artifact, so the causal specificity of the epistemic-signal claim is untested.","rationale":"The paper is a well-executed benchmark with three converging lines of evidence, and I agree with the reader's conditional verdict. My stress-test targets the same weakest assumption the reader identified: the void direction may be a low-level perceptual or question-text artifact rather than an epistemic signal. This is the most load-bearing concern because the causal steering experiment is the only evidence that the encoded distinction is causally usable; if the steering effect is driven by a generic off-manifold perturbation or by a 'video looks different' vector, the headline claim that the bottleneck is expressive rather than perceptual is not established. The missing control is concrete and easy to add: null-direction steering with random vectors, control-control directions, and question-text-only directions. The paper's cross-modality transfer and Cf→Cf restriction are genuinely helpful and should be credited, but they do not substitute for null controls. The reader also noted the per-pair best-layer probe selection as an inflation risk; I regard that as a secondary issue because the Cf→Cf restriction and cross-dataset transfer would likely survive a fixed-layer re-analysis, whereas the steering result has no comparable safeguard. My recommended verdict remains CONDITIONAL, matching the reader's, pending the null-direction controls and, secondarily, a fixed-layer probe re-analysis.","tokens_in":39917,"tokens_out":5833,"duration_ms":61370,"concrete_test":"Run the identical steering protocol of Section 5.5/Appendix I (same 50-pair cells, same layer, same judge panel, guided inference, α = 0, 2, 5, 10) with three null directions: (a) a random unit vector of the same dimensionality; (b) v computed from two control sets with different outcomes but identical answerability (e.g., control-control pairs from different scenarios); and (c) v computed from two well-posed questions on identical control videos, so the contrast is question-text-only without an answerability difference. If any null direction drives control abstention to levels comparable to the void direction (e.g., >30% at α=10) or suppresses void abstention similarly, the steering result does not isolate epistemic content.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the bottleneck is expression, not perception, rests on the causal steering result in Section 5.5 and Appendix I: adding the void direction v_l = (h_void - h_control)/||h_void - h_control|| at layer 20 induces abstention on control videos and suppresses it on void videos. The paper's weakest point is that this experiment only compares the void direction against the α=0 baseline; it never includes null-direction controls. Because h_void and h_control differ systematically in low-level ways (occlusion: an occluder is present; chaotic: the video is truncated and therefore shorter), v_l could encode 'visual difference,' 'sequence length,' or 'presence of an occluder' rather than answerability. Adding such a vector at every token position with α up to 10 could push activations off-manifold and trigger generic refusal or hedging, which a text judge would label as abstention. The cross-modality oc ip→ch result (Table 22) is the strongest safeguard, since oc ip uses videos identical to control; but that direction is computed from a question-text difference and could be a generic 'unanswerable or nonsensical question' axis that causes broad hedging on any input. The paper's own construct caveat in Appendix I.5 acknowledges this, but the cited converging evidence (cross-dataset transfer, Cf→Cf restriction, ill-posed conditions) does not include the one experiment that would rule out a non-epistemic direction: steering with null directions matched in norm and in low-level visual/linguistic properties. Without such controls, the causal claim 'steering a single-layer void direction causally induces or suppresses abstention' does not establish that the manipulated quantity is an epistemic signal rather than a generic perturbation or perceptual artifact.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TRAPSBench, a procedurally generated video benchmark of matched physics pairs in which a single modification (occlusion, chaotic truncation, or an ill-posed question) makes the outcome undeterminable, together with a new metric, Penalized Epistemic Calibration Score (PECS), that rewards correct answering on answerable controls and selective abstention on unanswerable voids. Across 16 VLMs, spontaneous abstention is poor (best PECS 0.292). The central claim is that the bottleneck is expressive rather than perceptual: linear probes decode the void/control distinction from hidden states with cross-dataset AUROC up to 0.91, and activation steering with a single-layer void direction causally induces or suppresses abstention. These mechanistic results are replicated on three open-weight families (Qwen3-VL-8B, Gemma 4 E4B, LLaVA-NeXT-Video-7B). The paper also reports a visual-vs-textual asymmetry and reasoning-dependent effects on calibration.","tokens_in":40182,"tokens_out":5011,"duration_ms":46051,"significance":"If the central claim holds, the paper makes a valuable contribution: TRAPSBench is a well-motivated, procedurally generated benchmark; PECS is a sensible conjunction metric whose degenerate-strategy behavior is explicitly verified; the Cf-to-Cf restriction is a strong control for behavioral confounds in probing; and the cross-family replication on three open-weight families is commendable. The transparent evaluation-provenance appendix is also a positive feature. However, the headline mechanistic claims rest on two load-bearing methodological choices—target-label-based best-layer selection in probing, and a steering experiment without null-direction controls—that currently leave the strength of the 'encode but fail to express' conclusion uncertain. The benchmark and metric contributions are likely to be useful to the community regardless, but the central mechanistic claim needs additional controls before it can be accepted as stated.","major_comments":[{"comment":"The reported probe AUROC values are per-pair maxima over layers, with the best layer selected using target-dataset labels. As the authors acknowledge in the text, 'best-layer selection does' see target labels. This is a form of test-set selection that can inflate AUROC, especially across 37 layers, and it directly affects the abstract's 'up to 0.91 AUROC' claim. Please report transfer AUROC with a layer chosen without target labels (e.g., a fixed layer, a layer chosen by source-side validation, or the average over layers), and apply the same protocol to the Cf-to-Cf restriction. The current numbers are not unbiased estimates of transferable decodability.","section":"Section 5.4, Table 3"},{"comment":"The steering experiment compares +alpha and -alpha only against the alpha=0 baseline; it does not include any null-direction control. Because h_void and h_control differ in low-level properties (presence of an occluder, truncated video length, question text), the void direction v_l could encode a generic 'different input' or 'hedging' axis rather than an epistemic signal. Adding such a vector at every token position with alpha up to 10 could push activations off-manifold and produce generic refusal that a text judge labels as abstention. The paper's construct caveat in Appendix I.5 is appropriate but the cited converging evidence does not include the one experiment that would rule out a non-epistemic direction. Please add norm-matched random directions, control-vs-control difference directions, and ideally an unrelated task direction, and show that abstention induction is specific to the void direction and is not simply a degradation artifact at high alpha.","section":"Section 5.5, Appendix I.5"},{"comment":"The paper mixes two evaluation epochs in its headline results: the PECS leaderboard (Table 1) and per-dataset tables use the corrected occlusion/chaotic splits but reuse original-evaluation chaotic ill-posed measurements, while the visual-vs-textual asymmetry (Table 2, Figure 3b) and the activation-steering analyses are reported entirely on the original evaluation. The appendix discloses this clearly, but the main text presents the abstract's 4x asymmetry claim and the PECS numbers without this provenance distinction, and the steering results were not recomputed on the corrected benchmark. Because the benchmark correction changed questions and ground truths, results from the two epochs are not directly comparable. Please either recompute all headline numbers on a single benchmark version or clearly separate, in the main text, which claims rest on which evaluation.","section":"Appendix P"}],"minor_comments":[{"comment":"The abstract states '1,404 matched physics pairs', but Appendix P indicates that the released and evaluated set comprises 1,354 pairs per model per regime (202 occlusion, 500 chaotic, 202 occlusion ill-posed, 450 chaotic ill-posed). Please reconcile the pair count.","section":"Abstract"},{"comment":"Table 6 lists Chaotic Ill-Posed as N=500, while Appendix F reports analyses on N=450 and the text says 50 restored Plinko items have no ill-posed measurements. Please clarify the released versus evaluated number.","section":"Table 6 and Appendix F"},{"comment":"Several table entries omit the leading zero (e.g., '.292', '.568'), which is inconsistent with the rest of the tables that use '0.292'. Please standardize numeric formatting.","section":"Table 1 and throughout"},{"comment":"The statement that the steering recompute 'was blocked by a build-environment incompatibility' should be moved into the main-text limitations section, since activation steering is a central causal claim and the paper should not rely on an appendix-only disclosure for this point.","section":"Appendix P, steering"},{"comment":"Please define Acc explicitly as control-accuracy in the equation caption or surrounding text, since the metric is applied only to answerable control videos.","section":"Equation (1)"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for COLM and the benchmark/metric contributions are solid, but the central mechanistic claim needs the two controls described in the major comments: unbiased layer selection for the probes and null-direction controls for the steering. The multi-epoch evaluation provenance is also a robustness concern that should be unified in revision. I would not recommend acceptance without these fixes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a serious referee. The paper does something genuinely new: a procedurally generated video benchmark of 1,404 matched physics pairs where a single targeted change makes the answer undeterminable, plus a metric (PECS) that is well designed and tested against degenerate strategies. The central claim—that VLMs encode answerability internally but fail to express it—is supported by probing, steering, and replication across three open-weight families. The Cf→Cf restriction is a good control, and the cross-modality steering transfer (oc ip→ch) is a clever piece of evidence. I came away believing that VLMs contain a decodable answerability signal that standard outputs suppress.\n\nSoft spots, in proportion. The steering experiment lacks null-direction controls. The void direction v_l = h_void − h_control can bake in low-level visual differences: chaotic voids are truncated and therefore shorter, occlusion voids contain an occluder. Adding such a vector at α up to 10 could push activations off-manifold and trigger generic hedging that the judge labels as abstention. The cross-modality direction (identical videos, different question text) is the strongest safeguard, but it is one path and could reflect a generic unanswerable-question axis. The authors acknowledge the construct caveat in Appendix I.5, but they never run the decisive control: steering with norm-matched random or perceptual directions. That is the main technical gap.\n\nThe probe AUROC uses per-pair best-layer selection with target labels, so the 0.91 headline is optimistic; fixed-layer or cross-validated selection would be more credible. The headline aggregates mix corrected and original evaluation runs—Appendix P is transparent about this, but ratios like the 4× asymmetry in the abstract come from the original run, so the numbers do not all come from the same benchmark state. Probe and steering results also lack error bars. None of these are fatal, but they should be fixed.\n\nWho this is for: VLM reliability, calibration, and interpretability researchers. It deserves peer review; I would send it out with revision requests focused on steering controls, probe layer selection, and consistent evaluation provenance.","headline":"Genuinely new video unanswerability benchmark with a plausible representation–output gap story; the steering claim needs null-direction controls before the 'expression, not perception' framing fully lands.","tokens_in":40757,"tokens_out":2469,"would_cite":true,"duration_ms":23941,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VLMs internally know when video evidence is insufficient but generate confident guesses anyway.","keywords":["vision-language models","epistemic restraint","selective abstention","physical reasoning","video understanding","activation steering","linear probing","uncertainty calibration"],"falsifier":"Create control videos that match the void videos’ low-level appearance (same motion energy, same occlusion footprint, same rendered object sizes) but remain deterministically answerable; retrain the probe and the steering direction on this matched set. If cross-dataset AUROC drops to chance and steering no longer changes abstention, the decoded direction was a visual artifact rather than an epistemic signal.","tokens_in":1855,"feed_emoji":"🧠","tokens_out":8481,"duration_ms":139699,"temperature":0.7,"pith_summary":"Vision-language models that watch short physics videos would rather guess than admit they cannot know: across sixteen models and five families, no model exceeds a PECS of 0.292 under ordinary prompting, even though PECS is defined to reward exactly the combination of correct answering when evidence suffices and abstention when it does not. The paper's central claim is that this failure is expressive, not perceptual. Linear probes decode the answerable/unanswerable distinction from hidden states and transfer it across physics domains at up to 0.91 AUROC, and adding a single-layer “void direction” to hidden states causally induces abstention on answerable videos and suppresses it on unanswerable ones. The same signature appears in three open-weight families with no shared training pipeline, so the bottleneck sits between an internal epistemic signal and the autoregressive output that fails to express it.","feed_headline":"VLMs know when to abstain, then guess anyway","feed_subtitle":"Probes read the unanswerable/answerable split at up to 0.91 AUROC; one steering direction turns abstention on and off.","key_machinery":"The load-bearing object is the void direction: for a given layer $\\ell$, $v_\\ell = (\\bar{h}^{\\mathrm{void}}_\\ell - \\bar{h}^{\\mathrm{control}}_\\ell)/\\|\\bar{h}^{\\mathrm{void}}_\\ell - \\bar{h}^{\\mathrm{control}}_\\ell\\|$, the normalized difference between mean hidden states on unanswerable and answerable videos. The paper uses this vector in two ways: a linear probe tests whether the direction separates void from control across datasets, and activation steering adds $\\pm \\alpha v_\\ell$ during generation to show causation. Around it sit TRAPSBench, a MuJoCo-generated set of 1,404 matched answerable/unanswerable video pairs across occlusion, chaotic sensitivity, and ill-posed questions, and PECS = $\\mathrm{Acc} \\times \\max(0, \\mathrm{AbsRec} - \\mathrm{FalseAbs})$, a conjunction metric that zeros both always-answer and always-abstain policies.","core_discovery":"On TRAPSBench—1,404 matched MuJoCo video pairs in which a control clip has a deterministic outcome and a near-identical void clip is unanswerable via occlusion, chaotic sensitivity, or an ill-posed question—the paper establishes that models' spontaneous restraint is poor across all sixteen VLMs tested. The new conjunction metric PECS = Acc × max(0, AbsRec − FalseAbs) forces selectivity: six degenerate strategies including never-abstain and always-abstain score zero, and the best unguided score is 0.292. Yet the void/control distinction is linearly decodable from frozen hidden states and transfers across domains, across visual and textual void types, and even to the specific void samples the model confabulates on (AUROC far above chance). Steering one layer's hidden states along the normalized void-minus-control direction turns abstention on for control videos and off for void videos, replicating across three open-weight families. The authors conclude that VLMs encode answerability but fail to express it, that visual evidence gaps are about four times harder to detect than textual impossibility, and that closing this gap will require output-stage interventions.","pith_inferences":["If the paper’s claim holds, an untested consequence is that a decoding-side probe could serve as an abstention head for models whose weights are not open, reading hidden states to veto confident answers—the paper only demonstrates direct steering in open-weight models.","The geometric result—occlusion directions transfer across domains while chaotic directions are near-orthogonal—predicts that benchmarks built on visible occluders will overstate model restraint relative to benchmarks with implicit evidence gaps.","The visual/textual asymmetry is consistent with training-data statistics: text corpora include many explicit “cannot determine” statements, whereas video corpora rarely label missing evidence. If so, finetuning on void-labeled video pairs should narrow the gap—a test the paper does not run.","The success of single-layer steering in three unrelated open-weight families suggests the mechanism is architectural rather than pipeline-specific, but it also means the claim cannot be checked in closed models with current API access."],"forward_implications":["Reliability work should shift toward output-stage interventions—steering, decoding constraints, or explicitly permitted abstention—because the epistemic representation already exists internally.","Simply adding an “I don’t know” option to the system prompt is a cheap, reproducible fix that raises abstention recall a median 1.9× across main-family video-native models without hurting control accuracy.","Chain-of-thought reasoning is not a reliable calibration aid: in at least one family, thinking overrides the model’s own expressed doubt and increases confabulation, so training rewards must penalize speculation.","Evaluations of VLM reliability need visual-evidence voids, not just textual tricks: unanswerability from missing visual evidence is detected roughly four times less readily than question-side impossibility.","Internal monitoring is a viable safety signal: because the void/control distinction is decodable even on confabulated outputs, a probe reading hidden states could flag should-abstain cases that the text gets wrong."],"supporting_citations":[{"why":"MuJoCo’s deterministic rigid-body engine generates the control/void pairs with simulator ground truth.","marker":"Todorov et al. (2012)"},{"why":"Supplies the J statistic (AbsRec − FalseAbs) that measures selective abstention in PECS.","marker":"Youden (1950)"},{"why":"Provides the unanswerable-question abstention protocol that the judge panel adapts.","marker":"Kirichenko et al. (2025)"},{"why":"Shows latent knowledge is linearly decodable, the probing premise this paper extends to visual uncertainty.","marker":"Burns et al. (2023)"},{"why":"Establishes linear truth geometry in hidden states, supporting the interpretation of the void direction as semantic.","marker":"Marks & Tegmark (2024)"},{"why":"Demonstrates a single direction mediates refusal, the template for single-layer abstention steering.","marker":"Arditi et al. (2024)"},{"why":"Supplies the activation-steering technique used to add or subtract the void direction.","marker":"Turner et al. (2023)"},{"why":"Contrastive activation addition, the steering baseline the protocol builds on.","marker":"Rimsky et al. (2024)"}],"fun_headline_variants":["VLMs sense unanswerable videos, then guess anyway","Probes decode VLM uncertainty, yet models still bluff","Steering one hidden direction makes VLMs abstain on cue","VLMs encode doubt but never express it: TRAPSBench","Models detect missing visuals, then guess anyway"],"cache_read_input_tokens":42752,"weakest_assumption_plain":"The story stands on the assumption that the computed void direction—the normalized difference between hidden states on void and control videos—is an epistemic signal about answerability rather than a low-level perceptual or question-text artifact; if that premise fails, the conclusion that models internally encode answerability independently of output behavior is not established.","fun_headline_variants_meta":{"raw":{"variants":["VLMs sense unanswerable videos, then guess anyway","Probes decode VLM uncertainty, yet models still bluff","Steering one hidden direction makes VLMs abstain on cue","VLMs encode doubt but never express it: TRAPSBench","Models detect missing visuals, then guess anyway"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001266,"raw_usage":{"total_tokens":5216,"prompt_tokens":1012,"completion_tokens":4204,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":4124}},"tokens_in":628,"tokens_out":4204,"duration_ms":28975,"temperature":1.0,"reasoning_tokens":4124,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:16:13.023254+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Create control videos that match the void videos’ low-level appearance (same motion energy, same occlusion footprint, same rendered object sizes) but remain deterministically answerable; retrain the probe and the steering direction on this matched set. If cross-dataset AUROC drops to chance and steering no longer changes abstention, the decoded direction was a visual artifact rather than an epistemic signal.","supporting_citations":[{"cited_title":"2012 , organization=","cited_arxiv_id":null,"evidence_quote":"MuJoCo’s deterministic rigid-body engine generates the control/void pairs with simulator ground truth."}],"review_version":1}