Look Again Before You Abstain:Budgeted Conformal Evidence Acquisition for Reliable Vision-Language Model

Delu Zeng; Jian Xu; John Paisley; Qibin Zhao; Yanning Wu

REVIEW 2 major objections 5 minor 20 references

BCEA shows that a vision-language model can be made to 'look again'—acquiring zoomed, cropped, or flipped evidence—without losing the finite-sample guarantee that the hallucination rate among asserted claims stays below a target level.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Folding visual evidence acquisition into the conformal score and re-calibrating on post-acquisition scores preserves the hallucination-rate guarantee while recovering coverage in LVLM selective prediction.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection A genuinely useful take on conformal evidence acquisition, but the algorithm as written doesn't match the theorem that's supposed to guarantee it. the 2 major comments →

arxiv 2606.16667 v4 pith:OZWFGNG7 submitted 2026-06-15 cs.CV

Look Again Before You Abstain:Budgeted Conformal Evidence Acquisition for Reliable Vision-Language Model

Jian Xu , Yanning Wu , Delu Zeng , John Paisley , Qibin Zhao This is my paper

classification cs.CV

keywords vision-language modelshallucinationconformal predictionselective predictionevidence acquisitionrecalibrationrisk controlclaim-type interventions

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper shows that guaranteed selective prediction for vision-language models is unnecessarily conservative: to certify a low hallucination rate, conformal filters abstain on most claims, sometimes over 80%. It introduces Budgeted Conformal Evidence Acquisition (BCEA), which replaces the binary answer/abstain decision with a three-way choice—answer, abstain, or acquire additional visual evidence by re-examining the image—under a bounded compute budget. The central fix is to fold the entire acquisition policy into the score function and recalibrate the conformal threshold on post-acquisition scores, which provably restores the distribution-free hallucination guarantee while improving coverage. The paper also shows that claim-type-specific interventions, such as a horizontal flip for left/right relations, recover signal that global ungrounding scores miss, and gives a precise condition: acquisition helps exactly when it improves the score's ROC curve.

Core claim

The central discovery is that naively plugging evidence acquisition into a calibrated conformal filter breaks the statistical guarantee—realized hallucination rate can overshoot the target by up to 17 points—because acquisition changes the score distribution and destroys the exchangeability between calibration and test that conformal validity relies on. The fix is to treat the acquisition policy as part of the score function and calibrate the threshold on the post-acquisition scores of the calibration claims (Theorem 1), so that calibration and test pass through an identical pipeline. This restores the finite-sample guarantee while still recovering coverage. A second discovery is that the gl

What carries the argument

The evidence-sufficiency score s(x,c) = ℓ(x) − ℓ(T(x)) measures how much a claim's likelihood drops when the image is removed (T = blank) or a claim-specific intervention is applied; acquisition forms a set of zoomed crops and takes the max likelihood, s_acq(x,c) = max(ℓ(x), max_b ℓ(x^(b))) − ℓ(∅). The load-bearing identity is Theorem 1: applying a fixed, label-independent map A to every claim preserves exchangeability, so the Clopper-Pearson fixed-sequence procedure on post-acquisition scores controls selective risk at level α with probability 1−δ. Theorem 2 converts 'does acquisition help' into a falsifiable ROC-dominance criterion: coverage at fixed risk is a point on the ROC curve, so ac

Load-bearing premise

The guarantee rests on exchangeability of test claims with the calibration set after the acquisition policy is applied; if the deployment distribution drifts or the policy is tuned using calibration labels, the finite-sample risk bound no longer holds. Equally load-bearing is the existence of a claim-type-specific, in-distribution, truth-flipping intervention—without one (counting, color binding, above/below), the method's coverage collapses to zero.

What would settle it

Tune the acquisition budget B (or a stopping threshold) on the same calibration labels used to select the conformal threshold, without a held-out split, and measure the 90th-percentile realized selective risk on exchangeable test claims. If it exceeds α by more than Monte Carlo error, the fixed-sequence bound no longer controls risk. Alternatively, run BCEA on a test set drawn from a different distribution than the calibration set and check whether the realized risk exceeds α, which would show the exchangeability assumption is load-bearing.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Guaranteed selective prediction for vision-language models can become substantially less conservative: a model may re-examine the image instead of abstaining, while the hallucination rate among asserted claims remains provably bounded.
The invariance principle—fold any fixed, label-independent acquisition policy into the score and recalibrate—extends beyond vision: any evidence-gathering step (querying a tool, running a verifier, asking a follow-up) can be added to a conformal predictor without losing validity, as long as calibration and test traverse the identical pipeline.
Claim-type-specific interventions become a design knob: for any claim family with a truth-flipping, in-distribution intervention, coverage can be recovered without sacrificing the guarantee; for families where no such intervention exists (counting, color binding in current backbones), the honest behavior is to abstain.
Theorem 2 gives a measurable acceptance test for acquisition policies: compute whether the post-acquisition ROC dominates the pre-acquisition ROC; if not, acquisition may not improve coverage at any fixed risk level.
Anytime-valid calibration (Proposition 4) means a deployed reliability monitor can keep collecting labeled claims and stop based on observed coverage without invalidating the guarantee.
The paper's explicit negative result—per-claim budget allocation is coverage-neutral without predictable heterogeneous gains—delimits where adaptive compute spending can help.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The paper evaluates probed atomic claims; composing BCEA with a claim parser for free-form generation is an untested extension. If the parser introduces noise or correlated claims, the exchangeability assumption in Theorem 1 may be violated, and the guarantee would need re-examination—the paper flags this gap explicitly.
The failure of vertical flip and the collapse on counting/color suggest a general principle for intervention design: an intervention is useful only if it is a symmetry of the image distribution that flips the truth value of the claim. This suggests intervention libraries could be built automatically by searching for transformations that are plausible and truth-altering on a small labeled probe set
Theorem 2's ROC criterion suggests a practical model-selection rule: when choosing between acquisition policies (e.g., uniform grid vs. model-guided crops), compare AUROC or the full ROC on a held-out split, not just coverage at one α, because the equivalence is pointwise in α.
The anytime-valid extension implies that one can monitor risk continuously on live traffic and keep calibrating until the desired coverage is reached, without a pre-fixed sample size—useful for deployment where data arrive sequentially.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A genuinely useful take on conformal evidence acquisition, but the algorithm as written doesn't match the theorem that's supposed to guarantee it.

read the letter

The core idea is right: when you acquire extra visual evidence, fold the acquisition policy into the score and recalibrate the conformal threshold on the post-acquisition scores. That principle is standard, but the paper earns its keep by showing how badly naive acquisition breaks the guarantee (coverage jumps, risk overshoots by 20+ points), and by demonstrating that a cheap, claim-type-specific intervention—horizontal flip for left/right relations—recovers signal where the global ungrounding score is at chance. The negative result on per-claim budget allocation is also clarifying: it doesn't help because the aggregate score doesn't become more informative. The limitations section is admirably honest: several claim types (counting, binding, above/below) simply have no usable intervention, so coverage collapses there. That is the faithful behavior, not a cover-up.

The main problem is a mismatch between Algorithm 1 and Theorem 1. Theorem 1 and its proof describe a fixed-sequence scan from the most to least conservative threshold, stopping at the first threshold whose Clopper–Pearson bound is ≤ α. That procedure controls FWER and gives a finite-sample guarantee. Algorithm 1, however, selects the *smallest* threshold (most permissive) where the bound ≤ α. With non-monotone selective risk, which the paper itself notes in Proposition 2, the smallest-safe-threshold rule is not covered by the fixed-sequence guarantee. Either the algorithm is wrong and the guarantee as stated does not apply to the deployed method, or the implementation actually uses the fixed-sequence scan and the reported coverage numbers are optimistic (the first accepted threshold would be the most conservative one, not the most permissive). This needs to be resolved before the central claim—"BCEA preserves the finite-sample guarantee"—can be taken at face value.

Minor issues: the abstract's "up to 17 points" overshoot doesn't match Table 1's largest overshoot of 21 points; the POPE table reports coverage but not the claimed risk control; and the paper lacks error bars (though it does report 90th-percentile risk over 300 splits, which partly addresses this).

Overall, the paper deserves a serious referee. The mechanism is sound, the interventions are clever, and the empirical work is honest about its boundaries. But the theory-algorithm gap is load-bearing for the guarantee claim, and the authors need to either patch Algorithm 1 to match the fixed-sequence proof or justify the smallest-t rule with a proper multiple-testing correction. I'd want to see that fixed before recommending acceptance.

Referee Report

2 major / 5 minor

Summary. The paper proposes BCEA, a method for adding budgeted visual evidence acquisition to conformal selective prediction for large vision-language models (LVLMs). The central idea is to turn the binary answer/abstain decision into a three-way decision in which borderline claims trigger acquisition of additional visual evidence (CLIP-guided crops, or claim-type-specific interventions such as horizontal flip), and then to fold the acquisition policy into the score function and recalibrate the conformal threshold on post-acquisition scores. The paper proves (Theorem 1) that this restores a finite-sample selective-risk guarantee in the exchangeable setting, characterizes when acquisition improves coverage via an ROC-dominance criterion (Theorem 2), and reports experiments on object-existence and spatial-relation claims across four open VLMs showing that BCEA controls risk near target while improving coverage over guaranteed abstention. The paper explicitly states its scope limits: it evaluates probed atomic claims rather than free-form generation, and it documents claim families for which no truth-flipping intervention recovers signal.

Significance. The problem is timely and important: conformal filters for LVLM hallucination provide distribution-free guarantees but are extremely conservative, and merely adding evidence acquisition destroys the guarantee. The paper makes a clear conceptual contribution by identifying the failure mode of naive acquisition and fixing it through post-acquisition recalibration. The theoretical machinery is standard but cleanly applied: Lemma 1 is the correct exchangeability-transport argument, Theorem 1 is a proper instantiation of fixed-sequence Clopper-Pearson risk control, and Theorem 2's ROC characterization is an elegant, falsifiable criterion for when acquisition helps. The empirical study is broad: four VLMs, the POPE benchmark, multiple claim families, 300 calibration/test splits, and honest reporting of 90th-percentile risk as the matched quantity. The paper also includes a valuable negative result on per-claim budget allocation and an explicit limitations section. If the theorem-algorithm gap described below is resolved, this is a solid contribution to reliable VLM inference.

major comments (2)

[§4.3 / Algorithm 1 / Appendix A] There is a direct mismatch between the validity theorem and the implemented algorithm. Theorem 1 (and its proof in Appendix A) establishes validity for a fixed-sequence procedure: scan candidate thresholds from largest to smallest and stop at the first threshold whose Clopper-Pearson upper bound satisfies Rδ ≤ α; this controls the family-wise error rate. Algorithm 1 (line 7), however, selects the 'smallest t' with Rδ ≤ α, i.e., the most permissive threshold satisfying the bound. Because Proposition 2 explicitly states that the selective risk R(τ) is not monotone in τ, the set of safe thresholds is not necessarily a suffix, so the fixed-sequence argument does not cover global-minimum selection. Appendix B repeats the 'smallest value' wording while calling the procedure a 'monotone fixed-sequence test,' which is contradictory. The authors must either (i) implement the fixed-sequence scan a
[§4.4 / §5.4] The paper asserts that 'each extra view weakly improves the ROC' and presents this as an empirical counterpart of Proposition 2 and Theorem 2. This assertion is not proven and does not follow from Proposition 2. Proposition 2 shows that at a fixed threshold τ, both TPR and FPR are non-decreasing with budget and the acceptance sets are nested. Nested acceptance does not imply pointwise ROC dominance: for a fixed FPR of the base score, the augmented score may require a higher threshold, and its TPR at that threshold can be lower. Theorem 2 makes ROC dominance the exact criterion for coverage improvement, so the budget-sweep discussion in §5.4 relies on an unproven theoretical premise. The empirical monotonicity in Figure 6 can stand on its own as an experimental finding, but the paper should either prove ROC dominance under the specific max-of-likelihoods structure or explicitly label the

minor comments (5)

[Abstract / §1 / Table 1] The quantitative claim about naive acquisition's risk overshoot is inconsistent. The abstract says 'up to 17 points,' §1 says 'up to 13 points,' but Table 1 shows overshoots of 14, 20, and 21 points at α = 0.05, 0.10, and 0.20, respectively. Please harmonize these numbers.
[§4.3, Proposition 4] The statement that 'selecting the smallest τ whose confidence-sequence upper bound is ≤α controls the selective risk at level α uniformly over all t' needs a simultaneous guarantee over the threshold grid. The confidence sequence C_t as defined is for a fixed τ; selecting τ after looking at the data requires a union bound or an e-process indexed by τ. This proposition is not used in the experiments, but the claim should be made precise or flagged as a conjecture.
[§6] The sentence 'coverage collapses to zero on those families' for counting, color, and binding is not quantified. A table or a specific number would make the limitation concrete.
[Notation / Figure 9] Figure 9's y-axis label 'acquisition gain sacq sglob' appears to be missing a minus sign; presumably it should be 'sacq − sglob'.
[General] No code availability statement is included. If the source code is released, adding a link would aid reproducibility of the CLIP-guided crop selection and the conformal calibration procedure.

Circularity Check

0 steps flagged

No significant circularity: the derivation chain is self-contained standard conformal risk control; no fitted input is renamed a prediction.

full rationale

The paper's central validity claim (Theorem 1) is a direct application of split-conformal selective risk control to the post-acquisition score s_A. The proof is included in Appendix A and rests only on Lemma 1, which states that a label-independent measurable map preserves i.i.d./exchangeability. This is a standard mathematical fact, not an assumption of the conclusion. The conformal threshold is the intended calibration output, not a fitted parameter later presented as a prediction; coverage and risk are evaluated on held-out test claims across random calibration/test splits (Section 5.3, Table 1). The 'naive acquisition is invalid' result (Proposition 1) follows from the pointwise inequality s_acq >= s_glob, which is explicit in Eq. (3), and is not circular. Theorem 2 ('acquisition helps iff it improves the ROC') is a mathematical identity derived from the definition of selective risk and ROC geometry, not an empirical fit. The cited prior work on folding label-independent transformations into conformal pipelines (Shanmugam et al. 2025; Wang et al. 2026) is supporting context, but Theorem 1 is proved directly, so the citation is not load-bearing. No self-citation chain forces the result. The paper also reports honest limitations (e.g., vertical flip giving AUROC 0.50, coverage collapse for counting/color/binding), which further indicates the empirical claims are not manufactured by construction. The identified mismatch between Algorithm 1's 'smallest t' selection and Theorem 1's fixed-sequence scan is a correctness/implementation concern, not a circularity of the derivation; it does not make the theorem equivalent to its inputs.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The central claim rests on the standard conformal exchangeability assumption plus the empirical assumption that a truth-flipping, in-distribution intervention exists for the claim type. No new physical entities are postulated. The free parameters are experimental settings and the intended calibration threshold, not hidden fits.

free parameters (4)

conformal threshold τ = e.g., 0.49 at α=0.10 (Table 2)
Chosen on calibration data to control selective risk; this is the method's calibration statistic, not an ad hoc fit, but it is data-determined.
acquisition budget B = 5 extra views per claim
Hand-chosen number of crops; coverage improves with B but no principled selection is given.
crop window size = 62%
Hand-chosen size for overlapping crops (four corners + center).
confidence level δ = 0.1
Hand-chosen; sets the calibration guarantee to 1−δ = 0.9.

axioms (6)

domain assumption Calibration and test claims are exchangeable draws from a fixed distribution over (image, claim, label).
Section 3 and Lemma 1 assume i.i.d. or exchangeable claims; POPE adversarial split may strain this, though the paper reports risk still controlled.
domain assumption The LVLM log-likelihood ℓ(v) is a label-independent measurable score computable per claim.
Section 4.1; the score s = ℓ(x) − ℓ(∅) and the post-acquisition max over crops rely on this.
domain assumption A blank/all-black image ∅ removes image grounding without changing the linguistic prior.
Section 4.1, Eq. 2; the global ungrounding score's discrimination (AUROC 0.838) is partial empirical support, but the assumption is built into the score.
domain assumption For spatial relations, a horizontal flip is an in-distribution intervention that flips the truth value of left/right claims.
Section 5.5; empirically shown to lift AUROC to 0.765, but the authors note vertical flip fails because it is out-of-distribution, so this property is empirical, not guaranteed.
standard math Clopper-Pearson bound and fixed-sequence testing control family-wise error.
Theorem 1 proof invokes Angelopoulos et al. 2025; valid for the fixed monotone scan over calibration scores.
standard math Betting confidence sequences are anytime-valid for optional stopping of the calibration stream.
Proposition 4 uses martingale Ville's inequality (Ramdas et al. 2023; Jiang and Yu 2024).

reviewed 2026-08-02 · how reviews work

0 comments

Cite this review

Pith. "Pith review of Look Again Before You Abstain:Budgeted Conformal Evidence Acquisition for Reliable Vision-Language Model." pith.science (2026). https://pith.science/paper/OZWFGNG7

@misc{pith2026260616667,
  author       = {Pith},
  title        = {Pith review of: Look Again Before You Abstain:Budgeted Conformal Evidence Acquisition for Reliable Vision-Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OZWFGNG7}},
  note         = {Machine review of arXiv:2606.16667}
}

Share X Bluesky LinkedIn Reddit HN

abstract

Large vision-language models (LVLMs) hallucinate: they assert visual details that the image does not support. A principled remedy is selective prediction with a distribution-free guarantee-verify each claim and abstain when the claim is not grounded, so that the hallucination rate among asserted claims is provably bounded. We show, however, that this guarantee is bought at a brutal price: to keep the hallucination rate below $5\%$ on a balanced object-existence benchmark, a state-of-the-art conformal filter must abstain on more than $80\%$ of claims. We argue that abstention is wasteful when more visual evidence is cheaply available, and introduce Budgeted Conformal Evidence Acquisition (BCEA), which replaces the binary answer/abstain decision with a three-way choice: answer, abstain, or acquire additional visual evidence by re-examining the image (zooming, cropping, or applying a claim-specific intervention) under a bounded

Figures

Figures reproduced from arXiv: 2606.16667 by Delu Zeng, Jian Xu, John Paisley, Qibin Zhao, Yanning Wu.

**Figure 2.** Figure 2: BCEA replaces answer/abstain with a three-way decision. A claim is scored by evidence sufficiency; borderline claims [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Figure 3: Evidence sufficiency on real images (LLaVA-1.5, mean claim log-likelihood [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

**Figure 4.** Figure 4: Occlusion sensitivity for existence claims (LLaVA-1.5). [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

**Figure 5.** Figure 5: Rescued claims (LLaVA-1.5, α=0.10). Three truly-supported claims (GT box in green) whose full-image evidence score falls below the conformal threshold τ , so guaranteed abstention must decline them (red, “No-Acq”). A single CLIP-guided crop zooms onto the object, raising the post-acquisition score above the recalibrated threshold, so BCEA correctly asserts (green). These are exactly the questions a guarant… view at source ↗

**Figure 6.** Figure 6: Guaranteed coverage grows with acquisition budget [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗

**Figure 8.** Figure 8: Uniform grid crops (orange) spread blindly; CLIP [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗

**Figure 9.** Figure 9: Acquisition gain sacq−sglob by object size. Uniform grid crops give size-independent gains; CLIP-guided crops nearly double the gain and, crucially, rescue small objects the grid crops away. Together with structured, claim-type-specific interventions that see what global ungrounding cannot, this turns a binary answer/abstain filter into a budgeted, guaranteed, evidenceseeking one. References Angelopoulos,… view at source ↗

discussion (0)

Reference graph

Works this paper leans on

20 extracted references · 7 linked inside Pith

[1]

N.; Bates, S.; Cand \`e s, E

Angelopoulos, A. N.; Bates, S.; Cand \`e s, E. J.; Jordan, M. I.; and Lei, L. 2025. Learn then test: Calibrating predictive algorithms to achieve risk control. The Annals of Applied Statistics, 19(2): 1641--1662

2025
[2]

Bai, S.; Cai, Y.; Chen, R.; Chen, K.; Chen, X.; Cheng, Z.; Deng, L.; Ding, W.; Gao, C.; Ge, C.; et al. 2025. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631

Pith/arXiv arXiv 2025
[3]

Guan, T.; Liu, F.; Wu, X.; Xian, R.; Li, Z.; Liu, X.; Wang, X.; Chen, L.; Huang, F.; Yacoob, Y.; et al. 2024. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 14375--14385

2024
[4]

He, X.; Zhang, Q.; Jin, A.; Yuan, Y.; Yiu, S.-M.; et al. 2024. TUBench: Benchmarking large vision-language models on trustworthiness with unanswerable questions. arXiv preprint arXiv:2410.04107

Pith/arXiv arXiv 2024
[5]

Huang, Q.; Dong, X.; Zhang, P.; Wang, B.; He, C.; Wang, J.; Lin, D.; Zhang, W.; and Yu, N. 2024. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13418--13427

2024
[6]

Jiang, R.; and Yu, K. 2024. Estimating means of bounded random variables by betting. JOURNAL OF THE ROYAL STATISTICAL SOCIETY SERIES B-STATISTICAL METHODOLOGY, 86(1)

2024
[7]

Kaul, P.; Li, Z.; Yang, H.; Dukler, Y.; Swaminathan, A.; Taylor, C.; and Soatto, S. 2024. Throne: An object-based hallucination benchmark for the free-form generations of large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 27228--27238

2024
[8]

Leng, S.; Zhang, H.; Chen, G.; Li, X.; Lu, S.; Miao, C.; and Bing, L. 2024. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13872--13882

2024
[9]

Li, Y.; Du, Y.; Zhou, K.; Wang, J.; Zhao, X.; and Wen, J.-R. 2023. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, 292--305

2023
[10]

J.; Cui, W.; Li, B.; Zhang, J.; and Malin, B

Li, Z.; Yan, C.; Jackson, N. J.; Cui, W.; Li, B.; Zhang, J.; and Malin, B. A. 2025. Towards statistical factuality guarantee for large vision-language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 11446--11467

2025
[11]

Liu, H.; Li, C.; Li, Y.; and Lee, Y. J. 2024. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 26296--26306

2024
[12]

Mohri, C.; and Hashimoto, T. 2024. Language models with conformal factuality guarantees. In Proceedings of the 41st International Conference on Machine Learning, 36029--36047

2024
[13]

Ramdas, A.; Gr \"u nwald, P.; Vovk, V.; and Shafer, G. 2023. Game-theoretic statistics and safe anytime-valid inference. Statistical Science, 38(4): 576--601

2023
[14]

Shanmugam, D.; Lu, H.; Sankaranarayanan, S.; and Guttag, J. 2025. Test-time augmentation improves efficiency in conformal prediction. In Proceedings of the Computer Vision and Pattern Recognition Conference, 20622--20631

2025
[15]

selective prediction

Srinivasan, T.; Hessel, J.; Gupta, T.; Lin, B. Y.; Choi, Y.; Thomason, J.; and Chandu, K. 2024. Selective “selective prediction”: Reducing unnecessary abstention in vision-language reasoning. In Findings of the Association for Computational Linguistics: ACL 2024, 12935--12948

2024
[16]

Tayebati, S.; Kumar, D.; Darabi, N.; Jayasuriya, D.; Krishnan, R.; and Trivedi, A. R. 2025. Learning conformal abstention policies for adaptive risk management in large language and vision-language models. arXiv preprint arXiv:2502.06884

Pith/arXiv arXiv 2025
[17]

Wang, X.; Suresh, A.; Zhang, A.; More, R.; Jurayj, W.; Van Durme, B.; Farajtabar, M.; Khashabi, D.; and Nalisnick, E. 2026. Conformal Thinking: Risk Control for Reasoning on a Compute Budget. arXiv preprint arXiv:2602.03814

Pith/arXiv arXiv 2026
[18]

Xu, Y.; Guo, W.; and Wei, Z. 2025. Selective Conformal Risk Control. arXiv preprint arXiv:2512.12844

Pith/arXiv arXiv 2025
[19]

A.; Kuzborskij, I.; Stutz, D.; Gy \"o rgy, A.; Fisch, A.; Doucet, A.; Beloshapka, I.; Weng, W.-H.; Yang, Y.-Y.; Szepesv \'a ri, C.; et al

Yadkori, Y. A.; Kuzborskij, I.; Stutz, D.; Gy \"o rgy, A.; Fisch, A.; Doucet, A.; Beloshapka, I.; Weng, W.-H.; Yang, Y.-Y.; Szepesv \'a ri, C.; et al. 2024. Mitigating llm hallucinations via conformal abstention. arXiv preprint arXiv:2405.01563

Pith/arXiv arXiv 2024
[20]

Zhang, J.; Khayatkhoei, M.; Chhikara, P.; and Ilievski, F. 2023. Towards perceiving small visual details in zero-shot visual question answering with multimodal llms. arXiv preprint arXiv:2310.16033

Pith/arXiv arXiv 2023

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.

[1] [1]

N.; Bates, S.; Cand \`e s, E

Angelopoulos, A. N.; Bates, S.; Cand \`e s, E. J.; Jordan, M. I.; and Lei, L. 2025. Learn then test: Calibrating predictive algorithms to achieve risk control. The Annals of Applied Statistics, 19(2): 1641--1662

2025

[2] [2]

Bai, S.; Cai, Y.; Chen, R.; Chen, K.; Chen, X.; Cheng, Z.; Deng, L.; Ding, W.; Gao, C.; Ge, C.; et al. 2025. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631

Pith/arXiv arXiv 2025

[3] [3]

Guan, T.; Liu, F.; Wu, X.; Xian, R.; Li, Z.; Liu, X.; Wang, X.; Chen, L.; Huang, F.; Yacoob, Y.; et al. 2024. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 14375--14385

2024

[4] [4]

He, X.; Zhang, Q.; Jin, A.; Yuan, Y.; Yiu, S.-M.; et al. 2024. TUBench: Benchmarking large vision-language models on trustworthiness with unanswerable questions. arXiv preprint arXiv:2410.04107

Pith/arXiv arXiv 2024

[5] [5]

Huang, Q.; Dong, X.; Zhang, P.; Wang, B.; He, C.; Wang, J.; Lin, D.; Zhang, W.; and Yu, N. 2024. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13418--13427

2024

[6] [6]

Jiang, R.; and Yu, K. 2024. Estimating means of bounded random variables by betting. JOURNAL OF THE ROYAL STATISTICAL SOCIETY SERIES B-STATISTICAL METHODOLOGY, 86(1)

2024

[7] [7]

Kaul, P.; Li, Z.; Yang, H.; Dukler, Y.; Swaminathan, A.; Taylor, C.; and Soatto, S. 2024. Throne: An object-based hallucination benchmark for the free-form generations of large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 27228--27238

2024

[8] [8]

Leng, S.; Zhang, H.; Chen, G.; Li, X.; Lu, S.; Miao, C.; and Bing, L. 2024. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13872--13882

2024

[9] [9]

Li, Y.; Du, Y.; Zhou, K.; Wang, J.; Zhao, X.; and Wen, J.-R. 2023. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, 292--305

2023

[10] [10]

J.; Cui, W.; Li, B.; Zhang, J.; and Malin, B

Li, Z.; Yan, C.; Jackson, N. J.; Cui, W.; Li, B.; Zhang, J.; and Malin, B. A. 2025. Towards statistical factuality guarantee for large vision-language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 11446--11467

2025

[11] [11]

Liu, H.; Li, C.; Li, Y.; and Lee, Y. J. 2024. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 26296--26306

2024

[12] [12]

Mohri, C.; and Hashimoto, T. 2024. Language models with conformal factuality guarantees. In Proceedings of the 41st International Conference on Machine Learning, 36029--36047

2024

[13] [13]

Ramdas, A.; Gr \"u nwald, P.; Vovk, V.; and Shafer, G. 2023. Game-theoretic statistics and safe anytime-valid inference. Statistical Science, 38(4): 576--601

2023

[14] [14]

Shanmugam, D.; Lu, H.; Sankaranarayanan, S.; and Guttag, J. 2025. Test-time augmentation improves efficiency in conformal prediction. In Proceedings of the Computer Vision and Pattern Recognition Conference, 20622--20631

2025

[15] [15]

selective prediction

Srinivasan, T.; Hessel, J.; Gupta, T.; Lin, B. Y.; Choi, Y.; Thomason, J.; and Chandu, K. 2024. Selective “selective prediction”: Reducing unnecessary abstention in vision-language reasoning. In Findings of the Association for Computational Linguistics: ACL 2024, 12935--12948

2024

[16] [16]

Tayebati, S.; Kumar, D.; Darabi, N.; Jayasuriya, D.; Krishnan, R.; and Trivedi, A. R. 2025. Learning conformal abstention policies for adaptive risk management in large language and vision-language models. arXiv preprint arXiv:2502.06884

Pith/arXiv arXiv 2025

[17] [17]

Wang, X.; Suresh, A.; Zhang, A.; More, R.; Jurayj, W.; Van Durme, B.; Farajtabar, M.; Khashabi, D.; and Nalisnick, E. 2026. Conformal Thinking: Risk Control for Reasoning on a Compute Budget. arXiv preprint arXiv:2602.03814

Pith/arXiv arXiv 2026

[18] [18]

Xu, Y.; Guo, W.; and Wei, Z. 2025. Selective Conformal Risk Control. arXiv preprint arXiv:2512.12844

Pith/arXiv arXiv 2025

[19] [19]

A.; Kuzborskij, I.; Stutz, D.; Gy \"o rgy, A.; Fisch, A.; Doucet, A.; Beloshapka, I.; Weng, W.-H.; Yang, Y.-Y.; Szepesv \'a ri, C.; et al

Yadkori, Y. A.; Kuzborskij, I.; Stutz, D.; Gy \"o rgy, A.; Fisch, A.; Doucet, A.; Beloshapka, I.; Weng, W.-H.; Yang, Y.-Y.; Szepesv \'a ri, C.; et al. 2024. Mitigating llm hallucinations via conformal abstention. arXiv preprint arXiv:2405.01563

Pith/arXiv arXiv 2024

[20] [20]

Zhang, J.; Khayatkhoei, M.; Chhikara, P.; and Ilievski, F. 2023. Towards perceiving small visual details in zero-shot visual question answering with multimodal llms. arXiv preprint arXiv:2310.16033

Pith/arXiv arXiv 2023

Look Again Before You Abstain:Budgeted Conformal Evidence Acquisition for Reliable Vision-Language Model

The reading

Core claim §

What carries the argument

Load-bearing premise §

What would settle it §

If this is right

Where Pith is reading between the lines

discussion (0)

Core claim

Load-bearing premise

What would settle it