Failing to See or Failing to Know? Attributing Errors in Vision-Language Models

Artem Shelmanov; Artem Vazhentsev; Khang Nhat Hoang Vo; Timothy Baldwin; Yova Kementchedjhieva

A model's wrong answer in a vision-language Q&A can be traced to one of four failure points before the model generates a single word.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 04:27 UTC pith:MHAHMI7K

load-bearing objection Solid descriptive core on pre-generation failure attribution, but the intervention gains are uncontrolled and can't yet be credited to routing. the 4 major comments →

arxiv 2607.04683 v4 pith:MHAHMI7K submitted 2026-07-06 cs.CV cs.CL

Failing to See or Failing to Know? Attributing Errors in Vision-Language Models

Khang Nhat Hoang Vo , Artem Vazhentsev , Artem Shelmanov , Timothy Baldwin , Yova Kementchedjhieva This is my paper

classification cs.CV cs.CL

keywords vision-language modelsknowledge-intensive VQAfailure attributionpre-generation representationsentity recognitionfactual accesstargeted interventionsoperational diagnostics

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that when a vision-language model gives a wrong answer to a knowledge-based question about an image, the error is not one monolithic failure. Sometimes the model never recognizes the thing in the picture; sometimes it recognizes the thing but cannot pull up the fact; sometimes it would succeed if the entity were named; sometimes the image itself is too degraded. The paper builds a decision tree that sorts each wrong answer into one of these categories using simple behavioral tests, then shows that signals hidden inside the model — before it generates any text — can predict which category an example will fall into. It then demonstrates that routing each predicted failure to a targeted fix (repairing the image, naming the entity, rewriting the question, adding a fact) lifts accuracy by 30–39 percentage points.

Core claim

The central claim is that VLM errors in knowledge-intensive visual question answering are not monolithic but decompose along a small set of operational decisions: whether the entity is recognized, whether controlled visual degradation breaks recognition, whether the original question is answered correctly after recognition, and whether the answer becomes correct when the entity is named explicitly. Each decision leaves a distinct trace in the model's pre-generation hidden states, so a separate lightweight probe per decision can predict the failure before decoding begins. The paper shows that visual-token hidden states are the strongest signal for image-side decisions, while prompt-boundary h

What carries the argument

The attribution tree is the central object: four local binary decisions arranged hierarchically, mapping each image-question pair to one of five operational outcomes (visual-evidence failure, unknown entity, success, unknown fact, unrecallable fact). Its work is to turn an unstructured wrong answer into a structured prediction target that can use different internal signals at different nodes, and to enable targeted interventions rather than a single abstention decision. The probes that do the predicting read pre-generation features — the final visual-token hidden state (VIS), the final prompt-token hidden state (EOP), the concatenated hidden states of the last eight prompt tokens (LAST8), an

Load-bearing premise

The labels produced by the controlled behavioral tests faithfully represent the true failure causes — that the yes/no recognition probe with distractors really measures entity recognition, that the entity-explicit rewrite really separates missing knowledge from failed recall, and that synthetic degradation stands in for natural visual failure; if any proxy is unfaithful, every downstream attribution and intervention result is conditional on the labeling protocol.

What would settle it

Take a sample of wrong answers where humans independently label the underlying cause (entity not seen, fact absent from the model, image degraded in a natural way such as occlusion or glare) and compare against the tree's operational labels on the same model outputs; if the tree's predicted failure type disagrees with the human diagnosis on a substantial fraction, the framework is measuring its own probes rather than the failure process. Concretely, test the visual-evidence probes on naturally degraded images — if the 97.0 PR-AUC achieved on synthetic corruption drops to chance under occlusion

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

Wrong answers can be sorted into pre-recognition and post-recognition branches, and the two branches require different remedies.
Pre-decoding diagnostics can replace blanket abstention: instead of refusing to answer, the model can be routed to a targeted fix.
Visual-token features carry a strong signal for recognition failures, while prompt-boundary hidden states carry a useful but weaker signal for whether an answer will be factually correct.
Factual-access attribution (unknown fact vs. unrecallable fact) remains hard; only a weak pre-generation signal exists, close to what answer-level uncertainty provides.
The labels are model-specific: the same image-question pair can receive different failure labels for different target models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

This suggests a deployment pattern where every VLM answer carries a predicted 'failure type', letting an interactive system decide whether to ask a clarifying question, retrieve a fact, or show a repaired image — rather than silently accepting a wrong answer.
The visual-evidence branch is built from synthetic corruption (Gaussian blur, noise, JPEG compression, downscaling); natural failures such as occlusion, glare, and unusual viewpoint may not be detectable by the same feature, so the 97.0 PR-AUC on controlled degradation may not transfer to the wild.
The entity-explicit rewrite test may not cleanly separate 'fact absent from memory' from 'fact present but not recalled'; the weak factual-access signal could reflect this labeling ambiguity rather than an intrinsic limit of the model's internals.
A natural extension is to couple this attribution tree with retrieval-augmented generation: use the predicted failure type to decide when to retrieve external evidence, which is currently applied indiscriminately in many systems.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Solid descriptive core on pre-generation failure attribution, but the intervention gains are uncontrolled and can't yet be credited to routing.

read the letter

Colleague,

This is a careful empirical paper on attributing VLM failures in knowledge-intensive VQA. The new thing is a tree-structured operational taxonomy built from behavioral tests that mostly exist in prior work, with per-node pre-generation probes. The descriptive results are the strong part: visual-token features dominate recognition-side decisions (86.7 PR-AUC recognition, 97.0 visual evidence), prompt-boundary hidden states are best for post-recognition answer success (62.7), and factual access is genuinely hard (53.5 versus 49.8 random). The pattern holds across four open-weight VLMs and two datasets, and the appendices are thorough. The paper is also honest about the weak factual-access signal and clear that its labels are operational diagnostics, not causal ground truth.

The main soft spot is the intervention experiment in Table 4. The 30.8–39.3 point accuracy gains come from applying a branch-specific intervention after the tree's prediction, but there is no control: no random routing, no best-single intervention, no apply-all. Every non-visual branch augments the prompt with GPT-5 output, so a large share of the gain could come from the intervention itself rather than from choosing the right branch. That is load-bearing for the applied claim, and the stress-test note lands.

Other concerns are secondary. No error bars or seed averaging anywhere, which matters given how thin the factual-access margin is. Labels are derived from the target model's own behavior, so probes partly measure self-consistency; the paper acknowledges this, but it limits the attribution claims. And the 97.0 PR-AUC on visual evidence comes from a synthetic corruption schedule with post-hoc balancing; the paper itself notes this is not about natural visual failures.

None of this breaks the paper. The descriptive core is solid and useful, but the routing value proposition is not yet established. Anyone building VLM failure diagnostics or selective answering gets value here.

This deserves a serious referee. A good reviewer would ask for the missing controls and seed variance, and the authors could plausibly supply both. I would not cite the routing gains; I might cite the diagnostic framework as a pre-generation probing baseline.

Send to review with expectations of moderate revision.

Referee Report

4 major / 4 minor

Summary. The paper proposes an operational attribution tree for failures in knowledge-intensive VQA, assigning each incorrect answer to one of four failure classes (visual-evidence failure, unknown entity, unknown fact, unrecallable fact) or success, using model-specific behavioral tests. The authors extract pre-generation features (visual-token states, prompt-boundary hidden states, attention maps) and train local binary probes at each tree node. They report that visual-token features best predict recognition-side decisions, prompt-boundary hidden states best predict answer success, factual-access prediction is weak, tree-composed prediction beats a flat 5-label classifier, and attribution-guided routing to targeted interventions yields 30.8–39.3 point accuracy gains. The paper is unusually honest about the operational, model-relative nature of its labels and about the weak factual-access signal.

Significance. If the central empirical pattern is robust, the paper makes a useful contribution: it moves beyond monolithic hallucination/abstention to a fine-grained, model-specific failure taxonomy, and it provides concrete evidence that different pre-generation representation families carry decision-dependent information. The multi-model, multi-dataset design and the explicit acknowledgment that factual-access attribution is weak are strengths. The intervention claim, however, is the part that would most distinguish this framework, and the current experiment is not controlled enough to support the routing conclusion. The taxonomy itself is also only as informative as its operational tests, which are acknowledged but not validated. As such, the paper is a promising diagnostic study whose main applied claim needs substantially stronger evidence.

major comments (4)

[§5, Table 4] The attribution-guided intervention experiment lacks the controls needed to attribute the 30.8–39.3 point gains to the routing mechanism. Table 4 only compares original accuracy to accuracy after GPT-5 executes the branch selected by the predicted label. Because every non-visual branch augments the prompt with GPT-5 output (Appendix G) and each intervention is intended to be helpful, the gains could largely occur even under random or uniform routing. Please add: (i) random routing among the four interventions for non-SUCCESS examples; (ii) best-single intervention applied to all non-SUCCESS examples; (iii) apply-all (providing all supports at once); and (iv) a generic prompt-augmentation control with no branch-specific support. Without these, the paper's strongest applied claim — that fine-grained pre-generation attribution enables the improvement — is unsupported.
[§3.2–§3.3, Limitations] The operational labels are defined by the target VLM's own behavior: UNKNOWN ENTITY is determined by yes/no probe answers to one ground-truth and three distractor prompts; VISUAL-EVIDENCE FAILURE is defined as a flip under a synthetic corruption schedule; UNKNOWN FACT vs. UNRECALLABLE FACT is determined by whether an entity-explicit rewrite succeeds. The paper states these are diagnostic indicators, not validated causal ground truth, but all downstream conclusions (probe rankings, tree-vs-flat comparison, intervention gains) are conditional on this protocol. The manuscript should either empirically validate the protocol (e.g., human sanity-checking of predicted labels, robustness of the probe rankings to alternative distractor counts/thresholds or to natural corruptions like occlusion/viewpoint) or explicitly restrict the claims to the operational definitions. Currently, the framework's
[Tables 1, 2, and 4] No error bars, confidence intervals, or seed variance are reported anywhere. Several key comparisons are close: in Table 2, LAST8 Transformer for factual access is 53.5 versus ORACLE-UQ 52.4 and random 49.8; in Table 1, recognition differences between VIS and other features are smaller for some models. The qualitative claim that visual-token features are best for image-side decisions and prompt-boundary states are best for post-recognition decisions would be more convincing with multiple seeds or bootstrap intervals, and the weak factual-access claim should be framed with explicit uncertainty. Without variance estimates, a reader cannot tell whether the decision-dependent pattern is reliable across feature families, especially for the 1–2 point gaps.
[Table 3] The tree-vs-flat comparison is not fully matched. Tree composition selects the best local probe per node using validation PR-AUC, while the flat classifier is a single model with its own hyperparameter grid. The flat classifier may be disadvantaged by the representational bottleneck, but it may also be disadvantaged by hyperparameter selection for a different objective (macro PR-AUC). Please report flat-classifier results using the same feature families and heads, or at least a per-feature comparison (e.g., LAST8 linear flat vs. tree-composed with LAST8 linear at all nodes), to show that the improvement comes from tree structure rather than from model-selection freedom or the ability to use different features at different nodes.

minor comments (4)

[§1, Fig. 1] The example in the bottom panel says 'No, it’s the Forbidden City' for a Mukden Palace recognition check, which is a plausible distractor, but the figure caption could clarify that this is illustrative rather than an actual model output. Minor but helpful for reproducibility.
[§3.2, Visual-evidence failure] The corruption schedule in Table 12 applies a fixed bundle of blur, noise, JPEG, and downscale at each level. It would be useful to report how many examples flip at each level, and whether the flip is dominated by one operation, since the operational definition of 'visual-evidence failure' depends on this schedule.
[Appendix I.5] The balancing of visual-evidence failures to match unknown-entity counts is mentioned, but the exact post-balancing class sizes per model are not given. Since class balance affects PR-AUC and the random baseline in Tables 1–2, reporting these numbers would increase transparency.
[Appendix G] For the UNRECALLABLE FACT branch, the target VLM is first asked to identify the entity, then GPT-5 rewrites the question. If the entity-identification step is wrong, the rewritten question may be misleading. Please report the entity-identification accuracy of the target VLMs, or at least discuss this potential error source in the limitations.

Circularity Check

1 steps flagged

Probe analysis is self-contained; one intervention branch is the same operation that defines its label, making part of the routing gain definitional.

specific steps

self definitional [§3.2 (Unrecallable fact vs. Unknown fact) and §5 (Attribution-Guided Intervention, UNRECALLABLE FACT bullet)]
"If the model fails on the original question but succeeds on the entity-explicit version, we assign UNRECALLABLE FACT... Under UNRECALLABLE FACT, no external factual evidence is added. We first ask the target VLM to identify the entity, GPT-5 then rewrites the question to state the predicted entity explicitly, without adding factual information or answering the question."

The label UNRECALLABLE FACT is defined by success on an entity-explicit rewrite. The intervention for that branch is the same entity-explicit rewrite (using the target VLM's predicted entity). For examples where the predicted entity matches the ground-truth entity used in labeling, the intervention is exactly the operation that produced the label, so the observed gain on this branch is inherited from the labeling protocol rather than providing independent evidence for attribution-guided routing. The paper acknowledges labels are operational, but Table 4/Figure 4 still count this branch's gain as support for the routing claim.

full rationale

The core probe experiments are not circular: features (VIS/EOP/LAST8/ATTN) are defined independently of the labels and are evaluated on held-out examples, so high PR-AUC is an empirical result rather than an identity. The labels are admittedly operational diagnostics, and predicting a model's own behavior from its hidden states is a legitimate supervised-probing setup. Self-citations (Shelmanov et al. 2025, Vazhentsev et al. 2025, Vashurin et al. 2025) appear only as baselines/feature-motivation and are not load-bearing. The intervention experiment, however, has one definitional component: the UNRECALLABLE FACT intervention is the same entity-explicit rewrite used to define that label. The absence of random-routing/single-intervention controls is a separate experimental-design concern, not a circularity. Overall, the central attribution-signal claim retains independent content, so the circularity is partial rather than total.

Axiom & Free-Parameter Ledger

5 free parameters · 7 axioms · 1 invented entities

The framework rests on behavioral-test definitions (Section 3.2), a two-hop task decomposition (Sections 2/3.1), a synthetic degradation schedule (Table 12), an 8B LLM judge (Appendix B), and the assumption that late-layer pre-generation representations encode the operational decisions (Section 3.3). None of these are externally validated against human causal annotation; the paper repeatedly disclaims causal status. Free parameters are modest and mostly protocol choices, but they directly shape the label distribution and therefore the headline probe scores.

free parameters (5)

Corruption schedule (blur sigma, noise std, JPEG quality, downscale per level) = Levels 1-4: sigma=2.5-5.5, noise=25-60, JPEG=60-25, downscale=0.75-0.40 (Table 12)
Hand-chosen severity levels define the visual-evidence failure boundary; different schedules would change the label distribution and all probe scores on that branch.
Recognition protocol thresholds = yes to 1 ground-truth prompt, no to all 3 distractors
Choice of distractor count and the strict conjunction determines the UNKNOWN ENTITY label; a looser or tighter rule would shift the recognition split.
Visual-evidence balancing target = Downsample so visual-evidence count matches unknown-entity count per model
Class balance in the recognition-failure split is forced to match the other class, affecting reported PR-AUC for recognition-side decisions.
Feature geometry choices = LAST8 window=8 tokens, attention lookback k=4, layer offsets {-1,-4,-8,-12}, top-m probabilities
Ad hoc geometry inherited from prior probing work; results may shift with other window sizes or layer selections.
Probe hyperparameters = Grid in Appendix E.1 (lr 1e-4..1e-3, batch 64-256, epochs 5-30)
Selected on validation PR-AUC; no seed averaging is reported, so validation-selection noise is unquantified.

axioms (7)

domain assumption Knowledge-intensive VQA is a two-hop process: recognition then factual access.
The whole tree structure depends on this decomposition (attributed to Venhoff et al. 2025). If failures intertwine, the tree misattributes them.
domain assumption Forced-choice yes/no recognition probes with distractors measure recognition reliability.
Section 3.2: the authors abandoned open-question probes because they were hard to evaluate; labels are therefore protocol-dependent.
domain assumption Entity-explicit question rewrites separate missing knowledge from failed recall.
Section 3.2/I.7, following Cohen et al. 2025. The paper admits rewrites may not fully separate the two causes.
domain assumption Synthetic degradation (blur/noise/JPEG/downscale) represents visual-evidence failures.
Section 3.2/I.5; explicitly acknowledged as not covering occlusion, viewpoint, cropping, or domain artifacts.
domain assumption A Meta-Llama-3-8B text-only judge gives valid correctness labels.
Appendix B: answer-success and factual-access labels all filter through this judge; judge errors propagate into every label.
domain assumption White-canvas ablation identifies insufficiently image-grounded questions.
Section 3.2/I.3: items any model answers without the image are removed, shaping the curated set and the reported low success rates.
domain assumption Late-layer hidden states of the target VLM are probeable proxies for the operational decisions.
Section 3.3/Appendix D: the entire method presumes the signal is accessible at pre-generation prompt-boundary representations.

invented entities (1)

Five-label operational attribution taxonomy (VISUAL-EVIDENCE FAILURE, UNKNOWN ENTITY, SUCCESS, UNKNOWN FACT, UNRECALLABLE FACT) no independent evidence
purpose: Organize VLM errors into routable decision leaves for targeted interventions
Explicitly operational diagnostics, not independently validated causal ground truth. The labels are defined by the target model's own behavioral tests, so no falsifiable handle exists outside the paper's protocol.

pith-pipeline@v1.3.0-alltime-deepseek · 26320 in / 16403 out tokens · 145494 ms · 2026-08-04T04:27:20.325947+00:00 · methodology

0 comments

Cite this review

Pith. "Pith review of Failing to See or Failing to Know? Attributing Errors in Vision-Language Models." pith.science (2026). https://pith.science/paper/MHAHMI7K

@misc{pith2026260704683,
  author       = {Pith},
  title        = {Pith review of: Failing to See or Failing to Know? Attributing Errors in Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MHAHMI7K}},
  note         = {Machine review of arXiv:2607.04683}
}

Share X Bluesky LinkedIn Reddit HN

read the original abstract

Vision-language models (VLMs) can recognize entities in clear images yet still fail when answering questions that require factual knowledge beyond what is directly observable. Prior work has either examined individual failure modes in isolation or treated incorrect answers as monolithic, binary failures. We propose a tree-structured framework that organizes failures in knowledge-intensive visual question answering into model-specific operational outcomes. Across two datasets and four VLMs, we observe consistent distributions of operational outcomes: some failures occur before entity recognition, while others persist after the relevant entity is recognized. Visual token representations are most informative for recognition-related decisions. Prompt hidden states predict answer success more effectively, although factual-access attribution remains difficult and exhibits only a weak signal. These pre-generation signals support attribution-guided routing to targeted interventions, including image repair, entity support, question rewriting, and factual evidence.