{"id":"a464d568-347a-4e7c-b8a4-6a87e3718bfe","arxiv_id":"2505.07496","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DocVXQA learns sparsity-regularized masks guided by ColPali so a Pix2Struct-based DocVQA model answers from masked document images and thereby produces visual explanations.","lead":"DocVXQA is a new framework that makes document question-answering models show their work by highlighting the relevant parts of the document image. It trains the model to answer from masked images, using sparsity pressure and signals from a separate retrieval model to shape the highlights, and human raters preferred these highlights over baseline explanations.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'context-aware' claim is inherited from ColPali: Eq. (3)'s L_MSE term is the sole source of context, and the paper admits ColPali heatmaps often misalign with human intuition; without independent evidence that ColPali relevance equals answer-justification context, the central claim is not…","rationale":"The reader identified the ColPali-prior dependency as the weakest assumption, and my analysis converges on the same point: the 'context-aware' property central to the paper's novelty is introduced exclusively through the L_MSE term in Eq. (3). The ablation in Table 2 demonstrates that this term is necessary for the reported accuracy/minimality trade-off; without it, the mask is answer-only and utility degrades. Therefore, the scientific content of the context-awareness claim reduces to the validity of ColPali's heatmaps as proxies for answer-justifying context. I considered alternative concerns: the circular sufficiency metric (the model is trained to answer from masks) genuinely weakens the quantitative evidence but does not directly attack the method's existence; the unmasked accuracy drop on PFL-DocVQA (0.57 vs 0.80) is understated but the paper does not claim competitive unmasked accuracy. The human preference study is a positive, but it compares DocVXQA against ColPali+Pix2Struct, not against an independent notion of context, and the confidence-rating protocol in the first human study is confounded by displaying the ground-truth answer. Because the reader's verdict is already CONDITIONAL—and this concern is precisely the kind of condition that must be met—no verdict adjustment is needed. The proposed concrete test would settle whether the context-awareness is genuinely learned or merely copied from ColPali.","tokens_in":16828,"tokens_out":5878,"duration_ms":58515,"concrete_test":"Train two variants of DocVXQA with Eq. (3) modified: (a) replace the ColPali prior Mp with a matched random smooth prior, and (b) replace it with a Grad-CAM saliency map computed from the frozen Pix2Struct itself, keeping all hyperparameters and postprocessing identical. Then re-run the human context/clarity evaluation on the same 10 images and the preference study on the same 21 question-answer pairs. Additionally, compute the IoU between DocVXQA masks and ColPali heatmaps at matched pixel ratios on a held-out set. If the human context scores drop significantly without the ColPali prior, or if the IoU with ColPali is near 1, this confirms that the 'context-aware' property is inherited from ColPali rather than learned by DocVXQA. If both are false, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that DocVXQA learns 'contextually sufficient' explanations rests on the L_MSE(Mp; M) term in Eq. (3), which pulls the learned mask toward ColPali's retrieval heatmaps. The paper's own Sec. 3.4 states that these visualizations 'often fail to align with human intuition or provide comprehensive explanations,' and the ablation in Table 2 shows that without the token-interactions term (S+M only) the mask collapses to a sparse answer-only region (pixel ratio 0.02) with degraded accuracy. Thus the context-awareness that distinguishes DocVXQA from answer-localization baselines is imported entirely from ColPali; the model has no independent mechanism to learn what context is needed to justify an answer, it can only refine ColPali's notion of relevance. If ColPali's retrieval relevance differs from human-judged context for answer justification—a possibility the paper itself acknowledges—then the 'contextually sufficient' property is not learned by DocVXQA but inherited, and is bounded by whatever ColPali highlights. The only direct evidence for context quality is a small human study on 10 images where participants were shown the ground-truth answer alongside the mask and asked for self-reported confidence, which can reward masks that merely contain the answer string rather than demonstrate reasoning context. Without an evaluation against human-annotated context regions or a demonstration that the method does not depend on the specific choice of ColPali as prior, the central claim is not supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DocVXQA, a self-explainable framework for document visual question answering. A mask head on a frozen Pix2Struct backbone produces a relevance map; the masked image is fed to the same model to answer the question. The training objective (Eq. 3) combines a cross-entropy sufficiency term, an L1 plus total-variation minimality term, and an MSE term that aligns the learned mask with a ColPali-derived prior. The authors claim that this is the first self-explainable DocVQA model producing contextually sufficient and representation-efficient explanations, and they support this with quantitative comparisons against attention, gradient, and retrieval baselines, ablations, a human rating study, and a human preference study. Code is released.","tokens_in":17129,"tokens_out":4080,"duration_ms":38370,"significance":"If the central claims are sustained, the paper would contribute a practical way to attach learned visual explanations to an existing DocVQA model with modest architectural changes, and the release of code and use of an external pretrained prior are commendable. The human preference result (64.7% preference over ColPali+Pix2Struct) is a useful signal. However, the significance is currently bounded by three load-bearing issues: the actual objective (Eq. 3) is not the information-bottleneck objective announced in Eq. (1); the sufficiency evaluation partly measures the model's own training objective; and the 'context-aware' property is imported from ColPali rather than learned independently. These issues affect the core claims, not just the presentation.","major_comments":[{"comment":"The paper announces an information-bottleneck objective in Eq. (1) and lists 'formalize explainability as an explicit learning objective with the information bottleneck principle' as a contribution, but the implemented objective in Eq. (3) replaces I(X;X⊙M) with L1 sparsity plus total-variation smoothness and adds an MSE term to a ColPali prior. Section 3.3 explicitly states that the kernel-density-estimation approach to mutual information 'did not yield the expected results.' No derivation is given to show that L1 plus TV approximates the mutual information term I(X;X⊙M), and the objective in Eq. (3) is not an information bottleneck in any formal sense. Please either provide a formal or empirical justification that the surrogate losses implement the IB trade-off, or revise the contribution and abstract to describe the actual objective (sparsity, smoothness, prior alignment) without claiming the IB principle.","section":"Sec. 3.1 and Sec. 3.3, Eqs. (1) and (3)"},{"comment":"The sufficiency evaluation is partly circular. In the protocol, all explanation masks are applied to the input and the resulting image is passed to 'our fine-tuned Pix2Struct'—the model that was trained with L_CE on images masked by our own learned mask. Thus our mask is co-optimized with the evaluator model, while baseline masks (raw attention, Grad-CAM, ColPali) are not. The high accuracy of Ours at a given pixel ratio therefore reflects not only mask quality but also that the model has specialized to its own mask distribution. Please evaluate all methods with a fixed evaluator that has not been trained on masked inputs (e.g., the original pretrained Pix2Struct), or at least report the accuracy of random masks and of S-only masks under the same evaluator to bound the effect.","section":"Sec. 4.1 and Table 1"},{"comment":"The central 'context-aware' claim is inherited from ColPali rather than learned. The ablation in Table 2 shows that without the token-interactions term (S+M only) the mask collapses to a tiny answer-only region (pixel ratio 0.02) with degraded accuracy, and the only source of context in Eq. (3) is L_MSE pulling the mask toward ColPali's retrieval heatmaps. Section 3.4 itself states that ColPali visualizations 'often fail to align with human intuition or provide comprehensive explanations.' The paper therefore needs evidence that ColPali's retrieval relevance coincides with the context needed to justify a DocVQA answer. Please provide (i) an evaluation against human-annotated context regions, or (ii) a comparison replacing the ColPali prior with a different prior (or a random fixed prior) to show the method's context-awareness is not just a copy of ColPali's biases.","section":"Sec. 3.4, Eq. (3), and Table 2"},{"comment":"The human evaluation of context-awareness is confounded by showing participants the ground-truth answer before they rate the masked image. The question 'How confident are you (1-5) to answer the question using only this masked image?' becomes, in effect, 'How easy is it to locate the answer string in this mask?' when the GT answer is displayed, so a mask that highlights any occurrence of the answer token can score highly even if it omits reasoning context. Please re-run the study with the GT answer hidden, or ask participants to produce the answer from the mask and then assess correctness separately.","section":"Sec. 4.3, Table 3"}],"minor_comments":[{"comment":"The threshold values for each baseline are chosen per method (e.g., Ours at 0.7, ColPali+Pix2Struct at 0.5), but no criterion for selecting these thresholds is stated. Please report the full threshold sweeps for all methods so the reader can see whether the reported operating points are comparably chosen.","section":"Sec. 4.1 and Table 1"},{"comment":"The appendix says the background-removal and connected-region postprocessing steps are applied to all methods in Table 1, but the main text (Sec. 3.6) says postprocessing is applied only at inference. Please state clearly which steps are applied to which methods.","section":"Appendix A.1"},{"comment":"There is a typo: 'text is extrtacted with OCR systems' should be 'extracted'.","section":"Sec. 2.1"},{"comment":"Typos: 'envolve' and 'throught' should be 'involves' and 'through'.","section":"Appendix A.1"},{"comment":"Strictly, cross-entropy CE(Y;Y_hat) equals H(Y|X⊙M) only in the ideal case; in general it is an upper bound. This is a standard surrogate, but the equality in Eq. (2) is written as if exact.","section":"Sec. 3.2, Eq. (2)"},{"comment":"The paper does not describe how the ColPali heatmap is extracted, resized, and aligned to the mask resolution for the L_MSE term. Please add these implementation details to the appendix.","section":"Sec. 3.4 and Fig. 2"},{"comment":"The human preference result is reported as 'p<< 0.001'; please report the actual p-value or exact binomial test statistic.","section":"Sec. 4.3"},{"comment":"The model-agnostic claim is supported only by qualitative examples with Donut. Please include quantitative results or state explicitly that the model-agnostic demonstration is qualitative.","section":"Sec. 4.3 and Appendix B.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is readable and the code release is a plus, but the gap between the announced information-bottleneck formulation and the actual objective, the circularity in the sufficiency evaluation, and the reliance on ColPali for the context-aware property are all central to the claimed contributions. The human-evaluation confound (GT answer shown before rating) further weakens the key evidence. These are fixable in a major revision, but the current version overstates the results. I would also encourage the authors to check the novelty claim 'first self-explainable DocVQA' against DLAVA and similar recent works more carefully."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a real systems paper, not a theory paper. The authors build a learnable mask head on Pix2Struct, train it with a three-term loss, and show that masked inputs still answer DocVQA questions while producing heatmaps people prefer over attention maps and ColPali heatmaps. That is a genuinely useful contribution. Code is available, ablations are informative, and the human preference result (64.7%, p<0.001, 12/12 participants) is a meaningful signal.\n\nWhat's actually new: to my knowledge this is the first self-explainable DocVQA model where the explanation mask is learned end-to-end with the answer. The combination of a sufficiency loss, an L1 sparsity/smoothness term, and a ColPali prior loss is new. The model-agnostic demo with Donut is a nice extra.\n\nNow the soft spots, in order.\n\nFirst, the information-bottleneck framing is decorative. Eq. (1) is the IB objective, but Eq. (3) replaces the MI terms with L_CE and L1. The paper admits the MI approach didn't work. That's fine — but the \"IB principle\" language overstates what is actually L1 + cross-entropy + MSE.\n\nSecond, the sufficiency metric is partly circular. You measure sufficiency by feeding the masked image to the model and checking accuracy, but the model is trained with L_CE to answer from masked images. So the metric rewards exactly what the objective optimizes. That doesn't make the masks useless, but it means the quantitative sufficiency numbers overstate generalization to unseen explanation quality.\n\nThird, the context-awareness is imported. The L_MSE term is the only thing pulling the mask beyond the answer region; the ablation (S+M row) shows the mask collapses to a tiny answer-only region without it. The paper even admits ColPali heatmaps often misalign with human intuition. So the \"contextually sufficient\" claim is really \"ColPali's notion of relevance, refined by L_CE and L1.\" The stress-test is right about this. It's not fatal — using an external prior is a legitimate design — but the framing should be toned down, and the authors should show sensitivity to the choice of prior.\n\nFourth, evaluation details. Table 1 compares methods at different thresholds (Ours at 0.7/0.8 vs ColPali+Pix2Struct at 0.5). The differences are modest on DocVQA, and on PFL-DocVQA our model's unmasked accuracy drops from 0.80 to 0.57 — a big degradation that is only mentioned in passing. The human study shows the GT answer alongside the mask, which can reward masks that simply contain the answer string.\n\nOverall: this is a credible, useful paper with an overclaimed narrative. It deserves a serious referee, but the authors need to tighten the evaluation and soften the IB/context claims. It's a good baseline for anyone working on document AI interpretability.\n\nRecommendation: send to peer review. Ask for matched thresholds, an independent sufficiency metric (e.g., human-annotated context regions), and an analysis of how mask quality changes with different priors.","headline":"A credible mask-based self-explainable DocVQA system with a real human-preference signal, but the 'context-aware' and IB claims outrun the evidence.","tokens_in":17723,"tokens_out":4015,"would_cite":false,"duration_ms":33006,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DocVXQA is the first self-explainable DocVQA framework: it learns answer-justifying masks during training, and people rate those masks clearer and more context-aware than attention heatmaps.","keywords":["document visual question answering","self-explainable model","visual explanations","relevance mask","information bottleneck","explainability","vision-language models","context-aware explanations"],"falsifier":"Build a set of questions where the answer string appears in several semantically different places in one document (for example, the same number under 'Total' and 'Subtotal', or the same name in two contract clauses) and ask which one the question means. A mask that merely encircles any occurrence of the answer will still let the model answer correctly, as the paper itself notes for its 'total amount' example, so the discriminating test is whether the learned mask consistently includes the disambiguating label or layout context. Running this probe with the released code would settle whether the masks carry reasoning context or track the retrieval prior regardless of where the true justification lies.","tokens_in":16592,"feed_emoji":"🔍","tokens_out":11578,"duration_ms":97626,"temperature":0.7,"pith_summary":"DocVXQA claims that a document question answering (DocVQA) system can be made self-explainable: together with each answer, the model learns a mask over the document that highlights the regions justifying that answer, and the masked document is literally what the model reads when producing the answer. The paper's central thesis is that this explainability can be trained directly — sufficiency by requiring the answer to stay correct on the masked image, minimality by penalizing how much of the image is highlighted — and that a retrieval model's heatmaps can be used as a prior to make the masks context-aware rather than mere reproductions of the answer text. This matters because current DocVQA explanations are post-hoc attention maps that often highlight every occurrence of the answer without showing why it was selected, which is an obstacle to trusting automated document reading in finance, healthcare, and administration. The reported experiments, including human ratings, support the claim that the learned masks carry enough context for people to answer from the masked image alone and that people prefer them to attention-based explanations.","feed_headline":"Self-explaining DocVQA masks beat attention heatmaps in human tests","feed_subtitle":"People can still answer from the masked evidence alone, and prefer these masks over attention maps.","key_machinery":"The load-bearing machinery is a learnable relevance mask M treated as an information-bottleneck representation: the masked image T = X⊙M is the only input the answering network — Pix2Struct, a pretrained OCR-free visual-language model — reads, so explanation and prediction are inseparable. Three losses carry the argument: cross-entropy with the ground-truth answer enforces sufficiency (the mask must retain what is needed to answer), an L1 term with an anisotropic total-variation continuity loss enforces minimality (sparse, coherent highlighted regions), and an MSE term pulls the mask toward a retrieval model's heatmaps to add context. The mask head is a lightweight MLP over encoder tokens, decoder attentions, and positional embeddings, so the whole apparatus attaches to an already-fine-tuned DocVQA model with only a new head and three loss terms, and the framework ports to other backbones such as Donut.","core_discovery":"The paper's claim is that a self-explainable DocVQA model can be built with small changes to a pretrained OCR-free vision-language model: a mask head turns encoded image tokens, decoder attention, and positional embeddings into a relevance mask M, and the masked image X⊙M is the bottleneck representation that the network reads to answer. The training objective unites three terms: a cross-entropy loss that keeps answers correct on masked inputs (sufficiency), an L1 plus anisotropic total-variation loss that keeps highlighted regions sparse and coherent (minimality), and an MSE loss that pulls the mask toward the question-document heatmaps of ColPali, a pretrained retrieval model whose late-interaction matching localizes question-relevant patches (context-awareness). The paper reports that without the third term the mask degenerates into a copy of the answer text, and that with it the method beats raw attention, attention rollout, Grad-CAM, and the retrieval-heatmap baseline in masked-input accuracy at comparable pixel ratio, while scoring highest in human context and clarity ratings; in a forced-choice study 163 of 252 trials preferred DocVXQA's masks.","pith_inferences":["A sharper test than Table 1's aggregates: build questions where the answer string appears in several places with different meanings (for instance, the same number under 'Total' and 'Subtotal') and check whether the mask includes the disambiguating header or layout; the paper's own 'total amount' example shows that highlighting any number can pass a sufficiency check without being a real justificat","Because the context prior is a single retrieval model, the explanations' ceiling is set by what retrieval relevance considers important; swapping in priors with different grounding, or an ensemble of priors, is an untested lever that could make masks better aligned with human reasoning.","The mask-then-answer loop suggests an iterative training direction the paper does not pursue: feeding the learned mask back as a hard attention constraint during further fine-tuning could push the underlying DocVQA model to rely on context it otherwise ignores."],"forward_implications":["If the central claim holds, DocVQA deployments can provide built-in justifications without post-hoc saliency tools or datasets annotated with answer locations.","The masked-image protocol turns 'does the explanation explain?' into a testable property: a mask is sufficient if the model — or a human — can still answer correctly from only the masked document.","The same mask head and losses attach to different DocVQA backbones, so the recipe is not bound to one architecture.","Threshold and postprocessing choices trade utility against minimality, giving practitioners a dial between revealing more of the document (higher accuracy) and revealing only the key evidence (higher interpretability)."],"supporting_citations":[{"why":"Supplies the pretrained Pix2Struct DocVQA backbone that the mask head and losses attach to.","marker":"Lee et al., 2023"},{"why":"Provides the ColPali retrieval heatmaps used as the context prior in the MSE term of the objective.","marker":"Faysse et al., 2024"},{"why":"Gives the information-bottleneck formulation that frames the minimality-sufficiency tradeoff.","marker":"Tishby & Zaslavsky, 2015"},{"why":"Supplies the self-explainable training paradigm and the minimality/sufficiency principle formalization reused here.","marker":"Choi et al., 2024"},{"why":"Provides the DocVQA dataset used for fine-tuning and evaluation.","marker":"Mathew et al., 2021"},{"why":"Provides the PFL-DocVQA dataset used for the second evaluation.","marker":"Tito et al., 2024"},{"why":"Donut, the alternative backbone used to demonstrate the framework's model-agnostic design.","marker":"Kim et al., 2021"}],"fun_headline_variants":["Context-aware masks beat attention maps in DocVQA","DocVXQA's self-explaining masks win human tests","Masks from DocVXQA preferred over attention heatmaps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method takes a pretrained retrieval model's heatmaps as the ground truth for 'context,' so the whole context-aware property rests on the assumption that what retrieval deems relevant for finding a document is also what a human needs to see in order to trust an answer, and that a pixel-level pull toward those heatmaps imports context rather than the retrieval model's own biases.","fun_headline_variants_meta":{"raw":{"variants":["Context-aware masks beat attention maps in DocVQA","DocVXQA's self-explaining masks win human tests","Masks from DocVXQA preferred over attention heatmaps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000473,"raw_usage":{"total_tokens":2337,"prompt_tokens":916,"completion_tokens":1421,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":1367}},"tokens_in":532,"tokens_out":1421,"duration_ms":10129,"temperature":1.0,"reasoning_tokens":1367,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:15:16.142606+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a set of questions where the answer string appears in several semantically different places in one document (for example, the same number under 'Total' and 'Subtotal', or the same name in two contract clauses) and ask which one the question means. A mask that merely encircles any occurrence of the answer will still let the model answer correctly, as the paper itself notes for its 'total amount' example, so the discriminating test is whether the learned mask consistently includes the disambiguating label or layout context. Running this probe with the released code would settle whether the masks carry reasoning context or track the retrieval prior regardless of where the true justification lies.","supporting_citations":[],"review_version":1}