{"id":"c6a0e3d2-c08e-489e-869d-374185a679c9","arxiv_id":"2607.09544","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"VLMs encode correct counts in misaligned internal subspaces; probe-based error detectors enable selective self-correction that improves accuracy up to 15.6 absolute points without parameter updates.","lead":"Vision-language models often encode the correct object count in their hidden activations even when they output the wrong number. A lightweight internal error detector can selectively re-prompt them, raising counting accuracy by up to 15.6 points with no training.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Best-layer selection plus limited causal evidence leave open whether probes recover genuine count directions or dataset-specific correlations.","rationale":"The reader's weakest_assumption correctly isolates the softest joint of the argument: best-layer selection + narrow causal evidence + the authors' own confound note in Appendix C. The multi-probe + SVCCA + selective re-prompt package is otherwise clean and useful, and CountBench results help, but the strongest mechanistic claim (\"the count is there but misaligned\") is only weakly causally supported outside one small model on synthetic data. A fixed-layer, multi-model re-test on real data would settle whether the concern lands; until then the paper remains a strong empirical contribution whose causal interpretation should be scoped more carefully, moving the verdict from unconditional ACCEPT to CONDITIONAL.","tokens_in":27859,"tokens_out":492,"duration_ms":6213,"concrete_test":"Fix the intervention layer a priori (e.g., middle third of the decoder, or the layer of peak gt_probe F1 on a held-out synthetic split never used for selection) and re-run the full steering sweep of Section 4 plus detector-guided correction of Table 1 on both InternVL2-4B and Qwen3-VL-8B for CountBench; if probe-derived steering no longer beats random directions by >5 pp or selective re-prompt gains fall below Always/Random-K, the causal-count-subspace claim weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that VLMs encode the correct count internally (and that misalignment, not absence, explains failures) rests on probes and detectors trained/selected at the single best layer by held-out F1 (Section 3.3), plus causal steering reported only for InternVL2-1B on the four synthetic sets (Section 4, Figure 4). Appendix C already shows vision-encoder error detectors can exploit scene-complexity confounds rather than true error signals. Because layer choice is post-hoc and the only causal test is narrow, high probe F1 and the 15.6 pp re-prompt gains (Table 1) could partly reflect selection of layers/datasets where count co-varies with other visual statistics, rather than a general, causally relevant count subspace that is merely misaligned with generation.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper argues that VLM counting failures often reflect a misalignment between internal count representations and verbalized outputs rather than absent knowledge. Across four VLMs and five datasets (four synthetic regimes plus CountBench), nonlinear probes on intermediate activations recover ground-truth counts and detect errors even when the model is wrong. SVCCA on probe weight matrices shows that ground-truth and output readout subspaces are only partially shared and often misaligned, especially for nonlinear probes. Causal activation steering along the ground-truth probe direction improves accuracy on synthetic data for InternVL2-1B (with random-direction controls), and a detector-guided selective re-prompting method yields up to +15.6 absolute percentage points without parameter updates, outperforming budget-matched random intervention and correlating with detector F1.","tokens_in":28069,"tokens_out":869,"duration_ms":9436,"significance":"If the result holds, the work supplies both a practical inference-time correction tool for a known VLM weakness and a mechanistic account of the internal-knowledge vs. output gap. Strengths include multi-model and multi-dataset evaluation, three random seeds, random-steering and Random-K/Always-Reprompt baselines, explicit correlation between detector quality and correction gain, and extension of the multi-probe arithmetic framework of Sun et al. (2025) to VLMs. The combination of probing, subspace comparison, causal intervention, and selective correction is a coherent contribution for the field.","major_comments":[{"comment":"Section 4 and Figure 4 report the only causal steering experiment, and only for InternVL2-1B on the four synthetic datasets. The central claim that recovered directions are causally relevant count subspaces (not merely predictive or dataset-specific) therefore rests on a narrow intervention. The manuscript should either extend steering to at least one larger model and/or CountBench, or substantially qualify the causal claim and treat steering as supporting evidence for InternVL2-1B only.","section":null},{"comment":"Section 3.3 selects the best layer l* by held-out F1 for each probe/detector, and all subsequent SVCCA, steering, and correction results use that post-hoc choice. Combined with Appendix C’s own admission that vision-encoder error detectors can exploit scene-complexity confounds, this leaves open whether high probe F1 and the Table 1 gains partly reflect selection of layers where count co-varies with other visual statistics. A fixed-layer or early-stopped protocol (or an ablation showing gains persist under non-oracle layer choice) is needed to make the “count is there but misaligned” claim robust.","section":null}],"minor_comments":[{"comment":"Abstract and §1 state “up to 15.6% absolute percentage points”; Table 1 correctly uses percentage points. Standardize the wording.","section":null},{"comment":"Figure 1 caption and pipeline text are dense; a short legend distinguishing the three probe targets (gt / output / error) would help.","section":null},{"comment":"τ is fixed at 0.5 (§5) with no sensitivity analysis; a brief threshold sweep or ROC-style note would strengthen the method section.","section":null},{"comment":"Limitations (§7) correctly flag the 1–9 range, model size, and single-model steering; these should be echoed more clearly when interpreting Table 1 gains for Qwen models, which are already strong.","section":null},{"comment":"Typographical inconsistencies (e.g., “V o et al.”, mixed “15.6%” vs “15.6 absolute percentage points”) should be cleaned in a revision.","section":null}],"recommendation":"major_revision","confidential_remarks":"The reader’s accept recommendation is understandable given the multi-model empirical package, but the load-bearing causal and layer-selection points are real and fixable within revision. I would not reject; major revision is the proportionate bar. Scope fits a solid CV/ML venue interested in VLM reliability and interpretability."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The punchline is simple: VLMs often have the right count in intermediate activations even when they say the wrong number; the gt and output readouts sit in partially shared but misaligned subspaces, and you can exploit that with either steering or a cheap error-detector gate for re-prompting.\n\nWhat is new is the clean transfer of the three-probe (gt / output / error) setup from LLM arithmetic to VLMs, the weight-space SVCCA comparison of readout directions, the causal steering check, and the practical detector-guided self-correction that beats budget-matched Random-K and often recovers most of Always-Reprompt without the cost or the flip risk. They run four models, four synthetic regimes plus CountBench, three seeds, and show a clear ρ≈0.80 link between detector F1 and correction gain. The layer-wise plots make the temporal asymmetry obvious: gt information plateaus early, output probes peak late. That is useful for anyone working on multimodal reliability or interpretability.\n\nSoft spots exist but are scoped. Best-layer selection by held-out F1 is post-hoc, and the main causal steering figure is only InternVL2-1B on the synthetic sets; Appendix C already flags that vision-encoder error detectors can latch onto scene complexity. Models stop at 8B, counts at 1–9, and no code is released. Those are real limits, not fatal ones. The CountBench numbers and the random-direction controls keep the central claim from collapsing into pure dataset correlation. The free parameters (layer, α, τ) are ordinary for this style of work and are reported transparently.\n\nMath and citation pattern look fine: SVCCA is applied correctly to probe weights rather than activations, baselines are sensible, and the literature (Sun, Paiss, Alghisi, etc.) is engaged honestly. This is for people who care about VLM grounding failures and inference-time fixes. I would bring it to reading group, cite the probe-and-correct result, and send it to referees.","headline":"Solid multi-probe extension of Sun et al. to VLM counting: count is often present but misaligned, with causal steering and selective re-prompt gains up to 15.6 pp that hold on CountBench.","tokens_in":28694,"tokens_out":513,"would_cite":true,"duration_ms":7268,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"VLMs often already know the correct object count inside their activations; the failure is a misaligned readout that simple probes can detect and selectively correct.","keywords":["vision-language models","object counting","activation probing","SVCCA","activation steering","self-correction","error detection","representational alignment"],"falsifier":"If error detectors trained on the same activations fail to predict counting mistakes better than chance on held-out real images, or if steering along the recovered count direction never improves accuracy above random-direction controls across models and datasets, the claim that the count is present but misaligned would be falsified.","tokens_in":28758,"feed_emoji":"🔢","tokens_out":630,"duration_ms":7118,"temperature":0.7,"pith_summary":"Vision-language models still fail at basic object counting even when they handle harder multimodal tasks. This paper asks whether the correct count is simply missing from the model’s internal representations or whether the count is present but fails to reach the final verbal answer. Across four models and five datasets, nonlinear probes trained on intermediate activations recover the ground-truth count and flag counting errors with high reliability, even on trials where the model’s own output is wrong. Weight-space SVCCA shows that the subspace aligned with the true count and the subspace aligned with the model’s verbalized count are only partially shared and often nearly orthogonal for nonlinear probes. Strengthening the count-aligned direction by activation steering improves accuracy, and a practical detector-guided self-correction loop that re-prompts only when an internal error detector fires raises counting accuracy by as much as 15.6 absolute points with no parameter updates. The work therefore treats counting failures as a readout-alignment problem rather than a pure knowledge gap, and turns that diagnosis into an inference-time fix.","feed_headline":"VLMs already know the count; the readout is just misaligned","feed_subtitle":"Probes catch the error and selective re-prompting lifts accuracy up to 15.6 points with no training","key_machinery":"Multi-probe activation analysis (ground-truth count probe, output-count probe, and binary error detector) together with weight-space SVCCA comparison of their readout subspaces and detector-gated re-prompting at the best detector layer.","core_discovery":"Nonlinear probes on intermediate VLM activations can recover the true object count and detect counting errors even when the model utters the wrong number; the ground-truth and output readout subspaces are partially shared but misaligned, and strengthening the count direction (by steering or selective re-prompting) improves counting performance without any weight updates.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Probes reveal VLMs encode correct counts despite wrong outputs","Count knowledge is present but readout directions are misaligned","Error probes plus selective re-prompting gain 15.6 points","Internal activations hold the true count; steering fixes readout","Nonlinear probes detect counting failures VLMs already know"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"High probe accuracy and successful steering at the single best layer chosen by held-out F1 truly reflect causally relevant count representations rather than dataset correlations or post-hoc layer selection.","fun_headline_variants_meta":{"raw":{"variants":["Probes reveal VLMs encode correct counts despite wrong outputs","Count knowledge is present but readout directions are misaligned","Error probes plus selective re-prompting gain 15.6 points","Internal activations hold the true count; steering fixes readout","Nonlinear probes detect counting failures VLMs already know"]},"model":"grok-4.5","effort":"low","cost_usd":0.003582,"raw_usage":{"total_tokens":1163,"prompt_tokens":753,"num_sources_used":0,"completion_tokens":63,"cost_in_usd_ticks":35820000,"prompt_tokens_details":{"text_tokens":753,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":347,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":753,"tokens_out":63,"duration_ms":3434,"temperature":1.0,"reasoning_tokens":347,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T02:14:49.753587+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"If error detectors trained on the same activations fail to predict counting mistakes better than chance on held-out real images, or if steering along the recovered count direction never improves accuracy above random-direction controls across models and datasets, the claim that the count is present but misaligned would be falsified.","supporting_citations":[],"review_version":1}