Pith. sign in

REVIEW 3 major objections 6 minor 43 references

Asymptotic Risk Calibration for Selective Question Answering

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A post-hoc calibration rule for selective question answering asymptotically caps the error rate among accepted LLM answers at a user-specified tolerance, without retraining the model.

desk verdict A useful and honest synthesis of LEC and CRC for selective QA that deserves review once the asymptotic proof is tightened and artifacts are released. read the letter →

arxiv 2608.12008 v1 pith:IWMQZVRN submitted 2026-08-12 cs.CL

Shufan Lin , Sijin Dong This is my paper · ORCID
classification cs.CL
keywords selectivequestionansweringuncertaintyquantificationriskcalibrationabstentionconformalcontrolasymptoticthresholdlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models can produce fluent but wrong answers, and a hand-picked uncertainty threshold gives no statistical control over how often the answers actually returned are wrong. This paper proposes A-CRC-QA, a post-hoc calibration rule that turns a held-out calibration set and a user-specified error tolerance $\alpha$ into one acceptance threshold; under exchangeable data, the expected error rate among accepted answers is asymptotically at most $\alpha$. The key move is to rewrite selection-conditioned error control as a linear expectation constraint and then calibrate that constraint through a monotonized empirical risk, even though the per-example loss moves in opposite directions for correct and incorrect answers. The authors present this as an asymptotic, marginal guarantee rather than a finite-sample one, and experiments on CoQA and MedMCQA show it retains more answers than conservative confidence-bound baselines at comparable average risk.

What carries the argument

The load-bearing object is the monotonized empirical risk $\hat{g}^{\uparrow}_n(\lambda)=\sup_{t\ge\lambda}\hat{g}_n(t)$, an upper envelope of the average linear loss over all more conservative thresholds. The per-example loss $L(\lambda)=S(\lambda)(E-\alpha)$ is not monotone in $\lambda$, because rejecting a correct answer raises the loss while rejecting an incorrect answer lowers it; monotonizing the empirical curve, not each instance loss, restores a monotone population-level target. The calibrated threshold is $\hat{\lambda}_n=\inf\{\lambda:\hat{g}^{\uparrow}_n(\lambda)+\gamma_n\le 0\}$ with $\gamma_n=(1-\alpha)/(n+1)$, and the proof uses uniform Glivenko-Cantelli convergence of bounded one-dimensional threshold functions to pass from the finite-sample constraint to a limiting expectation constraint. The equivalence $g(\lambda)=\mathbb{E}[S(\lambda)(E-\alpha)]\le 0 \iff \mathrm{SCER}(\lambda)\le\alpha$ (when selection probability is positive) connects the threshold choice to the user's error target.

What would settle it

Run the calibration on a stream where the test distribution is deliberately shifted from the calibration distribution (calibrate on one medical topic and test on another) and check whether the long-run average SCER stays below $\alpha$ as the calibration size grows; a systematic exceedance falsifies the claimed asymptotic control. Within the theorem's own assumptions, simulate bounded right-continuous losses with fixed tie-breaking and verify directly whether $\limsup_{n\to\infty}\mathbb{E}[L_{n+1}(\hat{\lambda}_n)]$ remains non-positive.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that the calibrated decision rule provides asymptotic control of the selection-conditioned error rate: if calibration examples and a future test example are independent and identically distributed and the per-example losses are bounded and right-continuous under a fixed tie convention, then the threshold $\hat{\lambda}_n$ chosen from Eq. (11) satisfies $\limsup_{n\to\infty}\mathbb{E}[L_{n+1}(\hat{\lambda}_n)]\le 0$, so accepted answers error no more than $\alpha$ in expectation as the calibration set grows. The loss being calibrated is $L(\lambda)=S(\lambda)(E-\alpha)$, where $S$ is the accept indicator and $E$ is the error indicator; controlling its expectation is exactly the same as controlling $\mathrm{SCER}(\lambda)\le\alpha$ whenever selection has positive probability. Because this loss is not monotone in the threshold, the paper targets the limit rather than a finite-sample guarantee and explicitly warns that the result should not be presented as conditional or high-probability finite-sample validity.

Load-bearing premise

The guarantee rests entirely on calibration and test examples being exchangeable draws from the same distribution, with bounded right-continuous losses; if deployment data comes from a shifted distribution, the calibrated threshold has no stated control over the accepted-answer error rate.

Editorial extensions

If this is right

  • If Theorem 1 holds, a deployment can specify a target accepted-answer error rate $\alpha$ and, with enough calibration data from the same distribution, the long-run average error among returned answers will not exceed $\alpha$; the residual finite-sample risk is visible in the empirical violation rate across calibration-test splits.
  • Because the calibration is post-hoc and needs only a scalar score, the same rule attaches a statistically meaningful operating point to any existing uncertainty estimator, so better uncertainty ranking and better calibration compound rather than compete.
  • Monotonizing the empirical risk cuts the violation rate roughly in half relative to direct linear-expectation calibration on the two tested benchmarks, at a cost of about 3 percentage points of acceptance rate; compared with confidence-bound calibration, it accepts roughly 7 percentage points more answers.
  • At small target error rates the feasible set may be empty, meaning the underlying model and uncertainty signal are not reliable enough to support that operating point; the method then abstains on everything rather than pretending the risk is achievable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because the theorem needs only i.i.d. calibration data and bounded right-continuous losses on a one-dimensional threshold class, the same monotonized-envelope calibration should transfer to any scalar-gated decision rule (retrieval acceptance, model routing) whose per-example loss is non-monotone; this is an extension, not a claim the paper makes.
  • Beyond the paper: the violation rates at small calibration sizes (for instance 26% at $n=100$ on CoQA) imply that the asymptotic guarantee does not say when a finite dataset is large enough; a practical extension would be a calibration-size diagnostic that reports the largest $\alpha$ supportable at a given $n$.
  • Beyond the paper: distribution shift breaks the guarantee, so a natural testable extension is sliding-window recalibration or reweighted exchangeability schemes; the paper lists distribution shift only as future work.
  • Beyond the paper: the ablation that monotonizes each instance loss accepts fewer than 19% of answers, which suggests a finite-sample guarantee for this problem, if one exists, will require a different loss design than per-instance monotone envelopes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes A-CRC-QA, a post-hoc calibration procedure for selective question answering. The method converts the selection-conditioned error rate SCER(λ)=Pr(E=1|S(λ)=1) into a linear expectation constraint via the instance-wise loss L(λ)=S(λ)(E−α), monotonizes the empirical risk by taking a supremum over more conservative thresholds, adds a vanishing correction γ_n, and chooses the smallest threshold satisfying the monotone envelope constraint. The central theoretical claim is Theorem 1 in Section 3.7: under i.i.d. calibration and test examples and bounded right-continuous losses, the expectation of the test loss at the calibrated threshold satisfies limsup E[L_{n+1}(lambda_hat_n)]≤0, implying asymptotic SCER control when the expected selection probability is bounded away from zero. The experiments on CoQA and MedMCQA with two LLMs and several uncertainty estimators compare the method with Fixed-50, Empirical, Hoeffding and Clopper-Pearson UCB baselines, and LEC-Direct. A-CRC-QA achieves average SCER below the 0.15 target with higher acceptance rates than the UCB baselines and lower violation rates than LEC-Direct, and the calibration-size sweep shows convergence toward the target as n grows.

Significance. The paper addresses a real problem: heuristic uncertainty thresholds do not control the error rate among accepted LLM answers. If Theorem 1 can be made rigorous, the contribution is valuable because the selective loss is non-monotone in the threshold, which blocks the standard finite-sample CRC theorem; an asymptotic calibration rule with a simple closed-form correction is a useful addition to COIN and LEC. The empirical protocol is a strength: 100 calibration–test splits, explicit violation-rate and infeasibility-rate metrics, a calibration-size sweep, and an honest Remark 1 that the guarantee is asymptotic and marginal rather than finite-sample. The ablation separating monotonization from the finite-sample correction is informative. The main weakness is that the proof of the central theorem is only a sketch and contains a genuine attainment gap, so the paper's theoretical contribution is not yet established. The experimental results are consistent with the intended behavior and support the practical recommendation, but they do not by themselves establish Eq. (15).

major comments (3)
  1. [Section 3.7, Eq. (11) and proof sketch] The proof's key step that the monotonized empirical risk at lambda_hat_n is at most -gamma_n is not justified. Since lambda_hat_n is defined as an infimum in Eq. (11), the feasible set can be open at its left endpoint. In the notation of Section 3.6, this happens when the monotonized empirical risk jumps downward at an observed threshold: all thresholds strictly above a score satisfy the constraint, but the score itself does not, so the infimum falls at the score and the constraint fails at the selected threshold. The algorithm in Eq. (14) avoids this by searching only over observed scores, so the theorem's object is not the object whose behavior is demonstrated. Please supply an attainment lemma for Eq. (11), for example by restricting Lambda to observed thresholds, by choosing any feasible point instead of the infimum, or by proving the feasible set is closed at its left endpoint under the tie convention actually used. Alternatively, the proof can be repaired by bounding the jump at the infimum by O(1/n) and showing the resulting violation vanishes; but as written this step is missing and Eq. (15) is not established.
  2. [Section 3.7, Eq. (4) and Eq. (1) (continuity assumption)] Theorem 1 assumes the losses are right-continuous under a fixed deterministic tie convention, but the loss defined in Eq. (4) with S(λ)=1{r≥λ} is left-continuous in λ: at λ=r, S jumps from 1 to 0 as λ increases past r, so the right limit at the jump is 0 while the value is 1. Section 3.6 processes ties as blocks and accepts scores equal to the threshold, which is the left-continuous convention. The manuscript does not define the tie convention that would make the loss right-continuous. This matters because the infimum-attainment problem in the previous comment is controlled by the continuity convention: with a right-continuous loss the feasible set is closed at its left endpoint, while with the left-continuous loss used in the paper it need not be. The theorem's assumptions should be matched to the actual selection rule, or the theorem and Section 3.6 should be changed together.
  3. [Sections 3.5 and 3.7 (empty feasible set)] Theorem 1 does not cover the case in which the feasible set in Eq. (11) is empty, although the procedure explicitly handles that case by the all-abstention rule lambda_bottom. This is not a vacuous corner case: Table 2 reports infeasibility rates of 16% and 22% at alpha=0.05. The theorem should state a convention for lambda_hat_n when Eq. (11) is empty (e.g., lambda_hat_n=lambda_bottom, for which L=0 and the bound holds trivially), and the proof should mention this case. Without such a convention, the statement 'let lambda_hat_n be obtained from Eq. (11)' has no object in precisely the settings where the method is exercised.
minor comments (6)
  1. [Abstract and Section 3.7] The abstract says the guarantee holds 'under exchangeability', but Theorem 1 assumes i.i.d. calibration and test examples; please harmonize the wording.
  2. [Section 3.6, Eq. (13)] Eq. (13) would be clearer if it defined C_0=0 and M_0=0 to represent the all-abstain prefix, since thresholds above the largest observed reliability score are otherwise not represented in the prefix-maximum notation.
  3. [Section 4.3, Table 2] At alpha=0.05, the paper reports SCER and AR alongside IF=16–22%; please clarify whether SCER and AR are averaged over all splits or only over feasible splits, and how undefined SCER values (no accepted answers) are handled.
  4. [Section 4.1] Please provide the exact split construction or a code artifact; the protocol is described in sufficient detail that reproduction is plausible, but the random-split seeds and any deduplication or filtering steps are not specified.
  5. [Author affiliation] The author affiliation line lists 'Zhangjiang University' in 'Hangzhou, Zhangjiang, China'; this geographic identification should be verified and corrected.
  6. [Title page] The ACM template metadata in the header says 2018 while the arXiv submission is dated 2026; please update the conference and copyright information.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the calibrated threshold is fit only to held-out calibration data, and the asymptotic guarantee is a standard uniform-convergence argument rather than a reduction of the target risk to the calibration constraint.

full rationale

The derivation chain is self-contained. The calibrated threshold λhat_n is selected on a held-out calibration set through Eq. (11), while Theorem 1 evaluates the expected loss on an independent future test example; test labels never enter the calibration step. The equivalence g(λ)≤0 ⇔ SCER(λ)≤α is an algebraic restatement of the selection-conditioned error rate, not a fitted relation. The asymptotic guarantee relies on uniform Glivenko–Cantelli convergence of the empirical risk to the population risk, an external mathematical fact, and no load-bearing result is imported from the authors' own prior work; the references to CRC and LEC are external. The proof sketch's infimum-attainment step ('By construction, ĝ↑_n(λhat_n)≤−γ_n') is not fully justified and is a correctness risk, but it is not circular because the theorem would still be an asymptotic consistency statement rather than an identity. Remark 1 explicitly limits the claim to asymptotic marginal control and disclaims finite-sample guarantees, which supports a non-circular reading.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the exchangeability assumption for uniform convergence, the boundedness and right-continuity of the loss, and the standard Glivenko-Cantelli property of threshold functions. The correction gamma_n and target risk alpha are hand-chosen rather than fitted. No invented entities are introduced.

free parameters (2)
  • gamma_n correction = (1-alpha)/(n+1)
    Hand-chosen finite-sample correction from CRC; not fitted to data; vanishes as n grows.
  • target risk alpha = 0.15 default; 0.05-0.25 in sweep
    User-specified risk level; not fitted.
assumptions (3)
  • domain assumption Calibration and test examples are independent and identically distributed (exchangeable).
    Invoked in Theorem 1; needed for Glivenko-Cantelli uniform convergence of the empirical risk.
  • standard math The instance-wise loss L_i(lambda)=S_i(lambda)(E_i-alpha) is bounded and right-continuous in lambda under a fixed deterministic tie convention.
    Boundedness (Eq. 7) is used for the expectation argument; right-continuity is a technical condition for the threshold selection.
  • standard math The class of one-dimensional threshold functions is a Glivenko-Cantelli class, so sup_lambda |g-hat_n(lambda)-g(lambda)| -> 0 almost surely.
    Used in the proof sketch of Theorem 1 to pass from empirical to population risk.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Asymptotic Risk Calibration for Selective Question Answering." pith.science (2026). https://pith.science/paper/IWMQZVRN

@misc{pith2026260812008,
  author       = {Pith},
  title        = {Pith review of: Asymptotic Risk Calibration for Selective Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IWMQZVRN}},
  note         = {Machine review of arXiv:2608.12008}
}
read the original abstract

Large language models (LLMs) may generate fluent but incorrect answers, making uncertainty quantification important for reliable question answering. However, heuristic uncertainty scores cannot perfectly distinguish correct predictions from incorrect ones, and directly applying a fixed uncertainty threshold provides no statistical control over the error rate among accepted answers. To address this limitation, we propose A-CRC-QA, a post-hoc calibration framework for uncertainty-aware selective question answering. The proposed method reformulates selection-conditioned error control as a linear expectation constraint and applies a monotonized empirical-risk calibration procedure inspired by conformal risk control. Since the resulting instance-wise loss is generally non-monotone with respect to the acceptance threshold, our framework targets asymptotic rather than finite-sample risk control. A-CRC-QA is model-agnostic, requires no additional training, and can be combined with different uncertainty estimators. Experiments on CoQA and MedMCQA demonstrate its applicability to both open-ended and closed-ended question answering, achieving a favorable trade-off between accepted-answer reliability and answer retention compared with uncalibrated and confidence-bound-based baselines.

Figures

Figures reproduced from arXiv: 2608.12008 by the authors.

Figure 1
Figure 1. Risk control and acceptance-rate trade-offs across target risk levels. Curves show the mean over repeated calibration– [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 20 canonical work pages

  1. [1]

    Anastasios Nikolas Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. 2024. Conformal Risk Control. InThe Twelfth International Conference on Learning Representations

    work page 2024

  2. [2]

    Aniri, Jinhe Bi, Peng Liao, Zengjie Jin, Volker Tresp, Fei Shen, Yunpu Ma, and Tat-Seng Chua. 2026. OPD-V: Visual On-Policy Self-Distillation with Modality Balance. arXiv:2608.05131 [cs.CV] https://arxiv.org/abs/2608.05131

    work page Pith review arXiv 2026

  3. [3]

    Jinhe Bi, Aniri, Minglai Yang, Xingcheng Zhou, Wenke Huang, Sikuan Yan, Yujun Wang, Zixuan Cao, Michael Färber, Xun Xiao, Volker Tresp, and Yunpu Ma. 2026. EchoRL: Reinforcement Learning via Rollout Echoing. InForty-third International Conference on Machine Learning. https://openreview.net/forum?id=A6az59SGtF

    work page 2026

  4. [4]

    Jinhe Bi, Yujun Wang, Haokun Chen, Xun Xiao, Artur Hecker, Volker Tresp, and Yunpu Ma. 2025. LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ek...

    work page 2025

  5. [5]

    Jinhe Bi, Yifan Wang, Danqi Yan, Xun Xiao, Artur Hecker, Volker Tresp, and Yunpu Ma. 2025. PRISM: Self-Pruning Intrinsic Selection Method for Training- Free Multimodal Data Selection.ArXivabs/2502.12119 (2025). https://api. semanticscholar.org/CorpusID:276421326

    arXiv 2025

  6. [6]

    Jinhe Bi, Danqi Yan, Yifan Wang, Wenke Huang, Haokun Chen, Guancheng Wan, Mang Ye, Xun Xiao, Hin rich Schuetze, Volker Tresp, and Yunpu Ma. 2025. CoT-Kinetics: A Theoretical Modeling Assessing LRM Reasoning Process.ArXiv abs/2505.13408 (2025). https://api.semanticscholar.org/CorpusID:278769227

    arXiv 2025

  7. [7]

    Jinhe Bi, Chennan Zhou, Zengjie Jin, Aniri, Shuo Lu, Wenke Huang, Hu Cao, Xun Xiao, Zhihong Zhu, Volker Tresp, Fei Shen, Yunpu Ma, and Tat-Seng Chua. 2026. ReflectRL: Learning from Golden Negative Trajectories via Reflective-to-Direct Reasoning. arXiv:2608.03972 [cs.AI] https://arxiv.org/abs/2608.03972

    arXiv 2026

  8. [8]

    Charles J Clopper and Egon S Pearson. 1934. The use of confidence or fiducial limits illustrated in the case of the binomial.Biometrika26, 4 (1934), 404–413

    1934

Show all 43 references
  1. [9]

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy.Nature630, 8017 (2024), 625–630

    2024

  2. [10]

    Yonatan Geifman and Ran El-Yaniv. 2019. Selectivenet: A deep neural network with an integrated reject option. InInternational conference on machine learning. PMLR, 2151–2159

    2019

  3. [11]

    Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024. A survey of confidence estimation and calibration in large lan- guage models. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lin...

    2024

  4. [12]

    Yu Gui, Ying Jin, and Zhimei Ren. 2024. Conformal alignment: Knowing when to trust foundation models with guarantees.Advances in Neural Information Processing Systems37 (2024), 73884–73919

    2024

  5. [13]

    Wassily Hoeffding. 1963. Probability inequalities for sums of bounded random variables.Journal of the American statistical association58, 301 (1963), 13–30

    1963

  6. [14]

    Anqi Hu, Zhiyuan Wang, Zijun Jia, and Bo Fu. 2026. MiRD: Reliable Set-Valued Prediction for Open-Ended Question Answering via Miscoverage Risk Decompo- sition.arXiv preprint arXiv:2605.27091(2026)

    2026 arXiv

  7. [15]

    Xingyue Huang, Rishabh, Gregor Franke, Ziyi Yang, Jiamu Bai, Weijie Bai, Jinhe Bi, Zifeng Ding, Yiqun Duan, Chengyu Fan, Wendong Fan, Xin Gao, Ruohao Guo, Yuan He, Zhuangzhuang He, Xianglong Hu, Neil Johnson, Bowen Li, Fangru Lin, Siyu Lin, Tong Liu, Yunpu Ma, Hao Shen, Hao Su...

    2025 arXiv

  8. [16]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation.ACM computing surveys55, 12 (2023), 1–38

    2023

  9. [17]

    Zijun Jia, Yuanchang Ye, Sen Jia, Yiyao Qian, Haoning Wang, Baojie Chen, Diyin Tang, Jinsong Yu, and Zhiyuan Wang. 2026. Balancerag: Joint risk calibration for cascaded retrieval-augmented generation.arXiv preprint arXiv:2605.20084 (2026)

    2026 arXiv

  10. [18]

    Kailin Jiang, Hongbo Jiang, Ning Jiang, Zhi Gao, Jinhe Bi, Yuchen Ren, Bin Li, Yuntao Du, Lei Liu, and Qing Li. 2025. KORE: Enhancing Knowledge Injec- tion for Large Multimodal Models via Knowledge-Oriented Augmentations and Constraints. arXiv:2510.19316 [cs.CL] https://arxiv....

    2025 arXiv

  11. [19]

    Kailin Jiang, Ning Jiang, Yuntao Du, Yuchen Ren, Yuchen Li, Yifan Gao, Jinhe Bi, Yunpu Ma, Qingqing Liu, Xianhao Wang, Yifan Jia, Hongbo Jiang, Yaocong Hu, Bin Li, and Lei Liu. 2025. MINED: Probing and Updating with Multimodal Time- Sensitive Knowledge for Large Multimodal Mod...

    2025 arXiv

  12. [20]

    Jaehun Jung, Faeze Brahman, and Yejin Choi. 2025. Trust or escalate: Llm judges with provable guarantees for human agreement. InInternational Conference on Learning Representations, Vol. 2025. 3101–3125

    2025

  13. [21]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran- Johnson, et al. 2022. Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221(2022)

    2022 arXiv

  14. [22]

    Ye Li, Anqi Hu, Yuanchang Ye, Shiyan Tong, Zhiyuan Wang, and Bo Fu. 2026. Set-valued prediction for large language models with feasibility-aware coverage guarantees.arXiv preprint arXiv:2603.22966(2026)

    2026

  15. [23]

    Xiaowen Ma, Yunpu Ma, Chenyang Lin, Sikuan Yan, Jinhe Bi, Zixuan Cao, Yi- jun Tian, Volker Tresp, and Hinrich Schuetze. 2026. Self-Evolving Multi-Agent Systems via Textual Backpropagation. InFindings of the Association for Compu- tational Linguistics: ACL 2026, Maria Liakata, ...

    2026 doi

  16. [24]

    Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. Selfcheckgpt: Zero- resource black-box hallucination detection for generative large language models. InProceedings of the 2023 conference on empirical methods in natural language processing. 9004–9017

    2023

  17. [25]

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. InConference on health, inference, and learning. PMLR, 248– 260

    2022

  18. [26]

    Tianfan Peng, Yuntao Du, Pengzhou Ji, Shijie Dong, Kailin Jiang, Mingchuan Ma, Yijun Tian, Jinhe Bi, Qian Li, Wei Du, Feng Xiao, and Lizhen Cui. 2025. Can Visual Input Be Compressed? A Visual Token Compression Benchmark for Large Multimodal Models. arXiv:2511.02650 [cs.CV] htt...

    2025

  19. [27]

    Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S Jaakkola, and Regina Barzilay. 2024. Conformal Language Modeling. InThe Twelfth International Conference on Learning Representations

    2024

  20. [28]

    Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversa- tional question answering challenge.Transactions of the Association for Compu- tational Linguistics7 (2019), 249–266

    2019

  21. [29]

    Xuankun Rong, Wenke Huang, Jian Liang, Jinhe Bi, Xun Xiao, Yiming Li, Bo Du, and Mang Ye. 2026. Backdoor Cleaning without External Guidance in MLLM Fine- tuning. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=os4QYDf3Ms

    2026

  22. [30]

    Binyu Tan, Zhiyuan Wang, Jinhao Duan, Kaidi Xu, Heng Tao Shen, Xiaoshuang Shi, and Fumin Shen. 2025. Conformal Lesion Segmentation for 3D Medical Images.arXiv preprint arXiv:2510.17897(2025)

    2025

  23. [31]

    Yijun Tian, Shaoyu Chen, Zhichao Xu, Yawei Wang, Jinhe Bi, Peng Han, and Wei Wang. 2025. Reinforcement Mid-Training. arXiv:2509.24375 [cs.CL] https: //arxiv.org/abs/2509.24375

    2025

  24. [32]

    Aditya Prakash, Yizhou Sun, and Wei Wang

    Guancheng Wan, Lucheng Fu, Haoxin Liu, Yiqiao Jin, Hui Yi Leong, Eric Hanchen Jiang, Hejia Geng, Jinhe Bi, Yunpu Ma, Xiangru Tang, B. Aditya Prakash, Yizhou Sun, and Wei Wang. 2025. Beyond Magic Words: Sharpness-Aware Prompt Evolving for Robust Large Language Models with TARE....

    2025

  25. [33]

    Qingni Wang, Tiantian Geng, Zhiyuan Wang, Teng Wang, Bo Fu, and Feng Zheng

  26. [34]

    Yujun Wang, Aniri , Jinhe Bi, Soren Pirk, and Yunpu Ma. 2026. ASCD: Attention- Steerable Contrastive Decoding for Reducing Hallucination in MLLM.Proceedings of the AAAI Conference on Artificial Intelligence40, 12 (Mar. 2026), 10306–10314. doi:10.1609/aaai.v40i12.38000

    2026 doi

  27. [35]

    Zhiyuan Wang, Aniri, Tianlong Chen, Yue Zhang, Heng Tao Shen, Xiaoshuang Shi, and Kaidi Xu. 2026. LEC: Linear Expectation Constraints for Selection- Conditioned Risk Control in Selective Prediction and Routing Systems. InForty- third International Conference on Machine Learning

    2026

  28. [36]

    Zhiyuan Wang, Jinhao Duan, Lu Cheng, Yue Zhang, Qingni Wang, Xiaoshuang Shi, Kaidi Xu, Heng Tao Shen, and Xiaofeng Zhu. 2024. ConU: Conformal Uncertainty in Large Language Models with Correctness Coverage Guarantees. Asymptotic Risk Calibration for Selective Question Answering...

    2024

  29. [37]

    Zhiyuan Wang, Jinhao Duan, Qingni Wang, Xiaofeng Zhu, Tianlong Chen, Xi- aoshuang Shi, and Kaidi Xu. 2026. Coin: Uncertainty-guarding selective question answering for foundation models with provable risk guarantees. InProceedings of the AAAI Conference on Artificial Intelligen...

    2026

  30. [38]

    Zhiyuan Wang, Jinhao Duan, Chenxi Yuan, Qingyu Chen, Tianlong Chen, Yue Zhang, Ren Wang, Xiaoshuang Shi, and Kaidi Xu. 2025. Word-sequence entropy: Towards uncertainty estimation in free-form medical question answering appli- cations and beyond.Engineering Applications of Arti...

    2025

  31. [39]

    Zhiyuan Wang, Qingni Wang, Yue Zhang, Tianlong Chen, Xiaofeng Zhu, Xi- aoshuang Shi, and Kaidi Xu. 2025. SConU: Selective Conformal Uncertainty in Large Language Models. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P...

    2025

  32. [40]

    Minglai Yang, Xinyan Velocity Yu, Pengyuan Li, Xinyu Guo, Zhenting Qi, Konwoo Kim, Longtian Ye, Xiaolong Luo, Jinhe Bi, Henry Zhang, Haris Riaz, Xuan Zhang, Yunze Xiao, Bangya Liu, Tom Tang, Yunfei Zhao, Qunshu Lin, Zihan Wang, Minghao Liu, Michael Lingzhi Li, Yilun Du, Jesse ...

    2026 arXiv

  33. [41]

    Xuanle Zhao, Qiushi Sun, Jingyu Xiao, Xuexin Liu, Haoyue Yang, Qiaosheng Chen, Xianzhen Luo, Jing Huang, Yufeng Zhong, Lei Chen, Shuai Fu, Zhenlin Wei, Jinhe Bi, Lei Jiang, Haibo Qiu, Siqi Yang, Peng Shi, Jian Hu, and Zhixiong Zeng

  34. [2025]

    InInternational Conference on Learning Representations, Vol

    Sample then identify: A general framework for risk control and assessment in multimodal large language models. InInternational Conference on Learning Representations, Vol. 2025. 64280–64297

    2025

  35. [2026]

    arXiv:2606.15932 [cs.CL] https://arxiv.org/abs/2606.15932

    Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence. arXiv:2606.15932 [cs.CL] https://arxiv.org/abs/2606.15932

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.