{"id":"b7a35fd3-cd13-45c4-9a7b-7be2ce0db0c5","arxiv_id":"1908.06306","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"U-CAM uses gradients of aleatoric and predictive uncertainty losses to sharpen visual attention maps and improve VQA accuracy over standard baselines.","lead":"The paper proposes U-CAM, a training and explanation recipe that combines uncertainty estimates with gradient-based attention maps for visual question answering. It reports modest accuracy gains and attention maps that correlate better with human attention, but the results lack code, error bars, and precise hyperparameter details.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The decisive missing experiment is a non-uncertainty control for the gradient surgery in Eqs. 6-9; without it, the claimed certainty mechanism is indistinguishable from generic gradient noise or regularization.","rationale":"The paper does something genuinely useful: it evaluates attention quality against human attention maps, provides a systematic ablation, and shows consistent gains on a standard VQA benchmark. Those are real strengths, and they justify taking the empirical claim seriously. However, the central claim is causal: the uncertainty content of the gradients must be what improves attention and accuracy. The design as reported does not isolate that variable. Every ablation replaces one uncertainty-based loss with another uncertainty-based loss; none removes the uncertainty semantics while keeping the gradient surgery intact. Without that control, the most plausible alternative explanation, that the elementwise product and ReLU/normalization surgery simply acts as a strong regularizer or sharpening operator, remains live. The internal inconsistencies in the method definition, Eq. 8 versus Algorithm 1 and the absence of η∂Lu/∂fi in the stated update, reinforce that the modified gradient is an ad hoc construction rather than a derived optimization signal, exactly the reader's weakest assumption. I therefore agree with the reader that the paper should be conditionally accepted: the missing control experiment, together with code and exact hyperparameters, is needed to establish that the certainty mechanism, rather than generic gradient surgery, is responsible for the reported gains. These missing pieces do not by themselves make the results implausible, and the human-attention benchmark is appropriate, so rejection would be too strong at this stage.","tokens_in":14826,"tokens_out":6290,"duration_ms":69801,"concrete_test":"Run four VQA-v1 training runs on the same MCB backbone with identical hyperparameters and seeds: (1) P-GCA as specified; (2) Sham-GCA, replacing Lu in Eq. 6 with a scalar noise loss whose gradient wrt fi is i.i.d. Gaussian, renormalized per sample to match ||∂Lu/∂fi||; (3) Self-GCA, replacing ∂Lu/∂fi by ∂Ly/∂fi so only the product/sign-flip structure remains; and (4) baseline MCB. Evaluate RC and EMD on VQA-HAT and accuracy on test-dev over three random seeds. If (2) or (3) matches (1) within one standard error, the certainty mechanism is not load-bearing; if (1) is significantly better, the concern is answered. In the same runs, test both Eq. 8 normalization and softmax normalization to confirm the method is uniquely specified.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that the uncertainty-specific content of the modified gradient in Eq. 9, not the surgery itself, drives both the VQA-HAT rank-correlation gains (Table 3) and the accuracy gains (Tables 1 and 4). The paper's ablations compare VE, UDL, AUL, PUL and their combinations, but every auxiliary gradient is derived from an uncertainty loss. There is no control in which ∂Lu/∂fi in Eq. 6 is replaced by a gradient with the same shape, scale, and noise level but no uncertainty semantics, such as i.i.d. Gaussian noise or the gradient of an unrelated auxiliary task. If such a control reproduces P-GCA's RC of 0.334 and accuracy of 66.5, then the improvement is a property of the elementwise product, sign-flip, ReLU, and normalization surgery rather than of the uncertainty estimates, and the paper's recipe claim is unsupported. The method is also underspecified exactly at this point: Eq. 8 normalizes by the spatial sum, while Algorithm 1 line 25 applies softmax; and the stated cost C = Ly + ηLu is not reflected in Algorithm 1 line 27, which updates θf with ∂Ly/∂fi + ∇'''u and never adds η∂Lu/∂fi. Both ambiguities should be resolved before the conditional claim can be evaluated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes U-CAM, a method for visual question answering (VQA) that combines gradients of the standard classification loss with gradients of aleatoric and predictive uncertainty losses with respect to the attended feature representation. The resulting 'certainty gradients' (Eqs. 6-9) are used both as visual explanation masks and as a modified training signal. Experiments on VQA-v1 and VQA-v2, evaluated against human attention in VQA-HAT and VQA-X, show a monotone improvement in rank correlation from the baseline (0.2790) to P-GCA (0.3341) in Table 3, and accuracy gains from 63.8 to 66.5 for P-GCA in Table 1. The paper claims this is a general recipe for obtaining improved certainty estimates and explanations for deep learning models.","tokens_in":15138,"tokens_out":5752,"duration_ms":52590,"significance":"If the results hold, the contribution is practically valuable: a simple, model-agnostic addition that improves both VQA accuracy and agreement with human attention, without training on human attention data. The use of an external human-attention benchmark that is not part of the training objective rules out direct circularity for the correlation claims, and the ablation series from VE through P-GCA is a useful empirical mapping. The method is applicable to at least two existing attention architectures (MCB and SAN). However, the significance is currently limited by unresolved specification issues, missing control experiments, and the absence of error bars, which prevent a reliable assessment of the claimed uncertainty-based mechanism.","major_comments":[{"comment":"The central claim that the uncertainty-specific content of the modified gradient drives the gains is not tested. Every proposed variant (VE, UDL, AUL, PUL, A-GCA, P-GCA) uses the same elementwise product, sign flip, ReLU, and normalization surgery, so the monotone improvements in Tables 1 and 3 could in principle come from the surgery itself rather than from the uncertainty estimates. A control experiment that replaces ∂Lu/∂fi in Eq. (6) with a gradient of unrelated semantics but matched shape and scale (e.g., gradient of a random auxiliary loss, or structured noise) is missing. If such a control reproduces the RC of 0.3341 and accuracy of 66.5, the paper's 'recipe' claim is unsupported.","section":"Section 4.1.3, Eq. (6); Tables 1 and 3"},{"comment":"There are two load-bearing specification gaps in the training procedure. First, Eq. (8) normalizes ∇''y by its spatial sum, whereas Algorithm 1 line 25 applies a softmax; these are different operations and the paper does not say which one was used in the reported experiments. Second, the cost function in Section 4.2 is C = Ly + ηLu, but Algorithm 1 line 27 updates θf with ∇y = ∂Ly/∂fi + ∇'''u and never adds η∂Lu/∂fi. As written, the update is not a gradient step on C, so the objective actually being optimized is unclear; this directly affects the claim that uncertainty is minimized and that the heatmaps reflect certainty.","section":"Section 4.2, Algorithm 1, Eqs. (8)-(9)"},{"comment":"The abstract and Section 5.2 claim that the uncertainty estimates show 'strong correlation with misclassification' and that the method improves certainty estimates, but no quantitative correlation coefficient or statistical test is reported; Figure 5(a,b) are qualitative plots. Since this is one of the two main claimed benefits, a number (such as Spearman or Pearson correlation between uncertainty and error, with confidence intervals) should be provided.","section":"Section 5.2, Figure 5"},{"comment":"No error bars, standard deviations, or multiple-seed results are reported for any table, and the values of the free hyperparameters λ (Eq. 6), γ (Eq. 7), η (Section 4.2), α (Eq. 11), σ0 (Eq. 10), and T (Eqs. 2 and 5) are never given, despite being 'fine-tuned using the validation set.' The reported differences are small (e.g., Table 3: PUL 0.3012 vs PUL+UDL 0.3243; Table 1: AUL 64.7 vs PUL 64.9), so the monotone ablation pattern could be within run-to-run variation. Means over at least three seeds and a listing of all hyperparameter values are necessary for the claims to be reproducible.","section":"Tables 1-5 and Section 5.5"}],"minor_comments":[{"comment":"Equation (13) writes the solution as arg max of the cost function, but the text describes training by minimizing a loss; this should be arg min.","section":"Eq. (13)"},{"comment":"The sentence 'Data uncertainty in a multi-modal setting, Uncer- tainty in VQA task is two-fold.' is a broken fragment and should be rewritten.","section":"Section 1, paragraph after contributions"},{"comment":"The caption of Table 2 says 'VQA-X' and the text says 'This human explanation mask only available for VQA-v2 dataset.' Please clarify the relationship between VQA-X and VQA-v2, and state which dataset each row of Table 2 is evaluated on.","section":"Table 2 and surrounding text"},{"comment":"For each column in Figure 6, state explicitly whether the displayed map is the raw attention weight, the gradient mask ∂Ly/∂fi, or the modified certainty gradient, so the qualitative comparison is interpretable.","section":"Figure 6"},{"comment":"Algorithm 1 line 14 references 'expI' and Eq. (10), but Eq. (10) defines exp(σ0^2); align the notation between the algorithm and the equation.","section":"Algorithm 1, line 14 and Eq. (10)"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful empirical contribution, but the missing control for the gradient surgery and the training-objective ambiguity are the key technical hurdles. The authors should also moderate the claim of a general 'recipe' given that only two base architectures (MCB and SAN) were tested and that the P-GCA additions to SAN underperform the stronger baselines in Table 4 (though they improve over SAN itself). The manuscript would benefit from a clearer separation of claims that are quantified (attention correlation, accuracy) from those that are only qualitative (uncertainty-misclassification correlation)."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is worth a look: combine aleatoric and predictive uncertainty losses with a Grad-CAM-style gradient mask, and use the result to steer VQA attention. That specific recipe is new as far as I can tell from the cited work, and the paper demonstrates it on two baselines (SAN and MCB), not just one. The monotone improvement in rank correlation with human attention across the ablation series in Table 3 is genuinely encouraging, and using VQA-HAT as an external benchmark rather than a training signal lowers the circularity concern. Accuracy gains of 2.7% on VQA-v1 are modest but consistent with the story.\n\nThe soft spots are concentrated exactly where the stress-test note points. Equations 6–9 define the gradient surgery, but there is no control where the uncertainty gradient is replaced by something with the same shape and scale but no uncertainty semantics. Without that control, the gains could just as well come from extra regularization or from the elementwise product and sign-flip operation itself. The ablations among VE, UDL, AUL, and PUL suggest that different auxiliary losses behave differently, which is evidence but not proof that uncertainty content matters. I would not call this fatal, but it is the load-bearing missing experiment, and the paper should not claim the certainty mechanism while it is unperformed.\n\nThere are also reproducibility gaps that are easy to name: no code, no error bars or multiple seeds, and the MCB baseline is described only as a \"variant.\" The discrepancy between the L2-style normalization in Eq. 8 and the softmax in Algorithm 1 is uncomfortable but likely minor; more serious is that the cost function C = Ly + ηLu does not show up in Algorithm 1, where the update uses ∇Lu without η. That underspecification needs to be resolved before anyone can reproduce the numbers. The SOTA claim is also a bit strong given that the eye-tracking-trained saliency method mentioned in the paper achieves 0.49 RC, though the authors do qualify it as task-independent.\n\nOn balance, I think this paper deserves a serious referee. The recipe is clearly stated, the ablation logic is sound, and the human-attention benchmark is a good choice. But referees should treat the mechanism claim as unproven and require the noise control, full hyperparameters, and code before accepting. It is a solid conditional accept, not a reject.\n\nFor my own use, I would probably cite this as an example of uncertainty-aware attention modification, but I would not rely on the certainty mechanism claim until the control is done. I would bring it to a reading group as a case study in how a plausible-looking ablation series can still miss the decisive control.","headline":"The paper has a plausible new recipe and a clean external benchmark, but the missing noise control leaves the central mechanism claim unproven.","tokens_in":15657,"tokens_out":1737,"would_cite":true,"duration_ms":19846,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding uncertainty-loss gradients to the classifier gradient produces visual attention maps that align with human gaze and improves visual question answering accuracy.","keywords":["visual question answering","uncertainty estimation","aleatoric uncertainty","predictive uncertainty","attention maps","class activation maps","explainability","Grad-CAM"],"falsifier":"Train the same VQA model with the same uncertainty-loss hyperparameters but replace the elementwise product and sign flip in Equation 6 with a randomly signed copy of the uncertainty gradient. If rank correlation with human attention on VQA-HAT does not drop materially, the product and sign asymmetry are not the operative cause of the reported gains.","tokens_in":14618,"feed_emoji":"🎯","tokens_out":4320,"duration_ms":42891,"temperature":0.7,"pith_summary":"The paper proposes U-CAM, a recipe for visual question answering that modifies the classification gradient during training using gradients of aleatoric and predictive uncertainty losses. The modified gradient steers the network toward attention regions the model is certain about, producing explanation masks that correlate better with human attention and boosting answer accuracy. Sympathetically read, the claim is that uncertainty-aware gradient surgery is a general way to obtain both improved certainty estimates and visual explanations for deep learning models.","feed_headline":"Uncertainty gradients yield human-like attention in VQA","feed_subtitle":"Adding aleatoric and predictive uncertainty loss gradients lifts VQA accuracy up to 2.7% and better matches human gaze.","key_machinery":"The mechanism is the Gradient-based Certainty Attention (GCA) mask. The uncertainty gradient $\\partial L_u/\\partial f_i$ is multiplied pointwise with the classification gradient $\\partial L_y/\\partial f_i$, negated with hyperparameter $\\lambda$, then passed through an asymmetric ReLU that keeps positive activations and scales negative ones by a large negative $\\gamma$. After normalizing to a probability distribution, this certainty mask is added back to the original gradient $\\partial L_y/\\partial f_i$ to form the effective gradient used to update the attention network. The uncertainty losses themselves come from a Gaussian-perturbed logit reparameterization: aleatoric loss uses a per-input variance, and predictive loss adds the entropy of Monte Carlo sampled probabilities.","core_discovery":"The central claim is that if you compute the gradient of an aleatoric or predictive uncertainty loss with respect to the attended image feature and combine it with the standard cross-entropy gradient through an elementwise product, a sign flip, an asymmetric ReLU, and a residual connection, the resulting 'certainty map' improves both the attention masks and the predictive accuracy of a VQA model. On VQA-v1 test-dev, the Predictive-GCA variant raises the MCB baseline accuracy from 63.8% to 66.5%, and on the VQA-HAT benchmark it raises rank correlation with human attention from 0.279 to 0.334.","pith_inferences":["The asymmetric ReLU with a large negative multiplier for negative activations is an ad hoc detail; if it is the actual engine of the accuracy gain, a simpler sign-aware regularizer might reproduce the effect without the full gradient surgery.","A natural stress test is to perturb images or add distractors and see whether the P-GCA attention maps remain grounded on the correct object; success would separate true grounding gains from generic gaze mimicking.","The uncertainty-correlation claim suggests a testable extension: P-GCA attention maps might be more robust to domain shift than standard Grad-CAM maps, because the certainty signal is trained rather than inferred post hoc.","The comparison to human attention is correlational; an evaluation on a downstream task requiring precise localization, such as providing the bounding box of the answered object, would test whether the improved rank correlation translates to better spatial grounding."],"forward_implications":["On VQA-v1 test-dev, P-GCA improves the MCB baseline by 2.7 points overall, with the largest gain on yes/no questions (82.2% to 84.7%).","On VQA-HAT, rank correlation of the produced attention maps with human attention rises from 0.279 (baseline) to 0.334 (P-GCA), while the Earth Mover's Distance decreases from 0.3931 to 0.3721.","The recipe transfers to other VQA models: P-GCA grafted on to the Counter model reaches 69.2% on VQA-v2 test-dev, approaching the then state of the art BAN model.","Uncertainty estimates correlate with misclassification: correctly predicted samples are typically more certain, and the margin between top-2 softmax scores grows for similar answer classes.","The method can be viewed as a plug-in that improves any VQA architecture outfitted with the uncertainty-estimation branches."],"supporting_citations":[{"why":"Supplies the gradient-based localization approach (Grad-CAM) that is extended to use uncertainty gradients.","marker":"[9]"},{"why":"Provides the aleatoric and epistemic uncertainty decomposition and the reparameterization trick used to define the uncertainty losses.","marker":"[24]"},{"why":"Contributes the Gaussian logit perturbation and the multi-task uncertainty weighting used in Equations 2 and 3.","marker":"[25]"},{"why":"Establishes the Monte Carlo dropout framework used to approximate the predictive uncertainty in Equation 5.","marker":"[16]"},{"why":"Releases the VQA-HAT human attention dataset used to measure rank correlation and Earth Mover Distance.","marker":"[8]"},{"why":"Defines the MCB baseline model that the proposed gradient certainty method is grafted onto.","marker":"[14]"},{"why":"Introduces the VQA-v1/v2 datasets and the answer-accuracy evaluation metric used throughout the experiments.","marker":"[2]"},{"why":"Provides the DVQA baseline that the method compares against on attention correlation and accuracy.","marker":"[37]"}],"fun_headline_variants":["Uncertainty gradients sharpen VQA attention","VQA attention improved via uncertainty gradients","Aleatoric loss gradients align VQA gaze","Gradient certainty maps boost VQA accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The modified gradient in Equation 9, which uses the elementwise product and sign flip of uncertainty gradients with hyperparameters lambda and gamma, is taken to be a valid optimization signal, even though it is not derived from any single loss function.","fun_headline_variants_meta":{"raw":{"variants":["Uncertainty gradients sharpen VQA attention","VQA attention improved via uncertainty gradients","Aleatoric loss gradients align VQA gaze","Gradient certainty maps boost VQA accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000137,"raw_usage":{"total_tokens":1084,"prompt_tokens":815,"completion_tokens":269,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":431,"completion_tokens_details":{"reasoning_tokens":214}},"tokens_in":431,"tokens_out":269,"duration_ms":3430,"temperature":1.0,"reasoning_tokens":214,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:49:23.575922+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same VQA model with the same uncertainty-loss hyperparameters but replace the elementwise product and sign flip in Equation 6 with a randomly signed copy of the uncertainty gradient. If rank correlation with human attention on VQA-HAT does not drop materially, the product and sign asymmetry are not the operative cause of the reported gains.","supporting_citations":[{"cited_title":"Moura, Stefan Lee, and Dhruv Batra","cited_arxiv_id":null,"evidence_quote":"Supplies the gradient-based localization approach (Grad-CAM) that is extended to use uncertainty gradients."},{"cited_title":"What uncertainties do we need in bayesian deep learning for computer vision? In Advances in neural information processing systems, pages 5574–5584, 2017","cited_arxiv_id":null,"evidence_quote":"Provides the aleatoric and epistemic uncertainty decomposition and the reparameterization trick used to define the uncertainty losses."},{"cited_title":"Multi-task learning using uncertainty to weigh losses for scene geome- try and semantics","cited_arxiv_id":null,"evidence_quote":"Contributes the Gaussian logit perturbation and the multi-task uncertainty weighting used in Equations 2 and 3."},{"cited_title":"Lawrence Zitnick, Devi Parikh, and Dhruv Batra","cited_arxiv_id":null,"evidence_quote":"Releases the VQA-HAT human attention dataset used to measure rank correlation and Earth Mover Distance."},{"cited_title":"Namboodiri","cited_arxiv_id":null,"evidence_quote":"Provides the DVQA baseline that the method compares against on attention correlation and accuracy."}],"review_version":1}