Pith. sign in

REVIEW 2 major objections 4 minor 35 references

Consistency Has a Computable Blind Spot: A Commutation Theory of Label-Free Reliability for Vision-Language Figure Reading

T0 review · 2 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proves that the blind spot of label-free consistency tests is a computable algebraic object: an error is invisible to an edit exactly when it commutes with the edit's answer-transform, and the errors a suite misses form its…

desk verdict Core centralizer theory is sound and useful; the abstract's 'no suite of swaps' claim is false, but the paper's specific fixed-pair result and the rest of the framework survive. read the letter →

arxiv 2608.05675 v1 pith:44K2UFXG submitted 2026-08-06 cs.LG

Rasul Khanbayov , Hasan Kurban This is my paper · ORCID
classification cs.LG
keywords vision-languagemodelshallucinationdetectionlabel-freeevaluationequivariancechartquestionansweringconsistencycentralizermetamorphictesting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to prove that the blind spot of label-free reliability tests for vision-language models is not a mystery but a computable algebraic object. It claims that a systematic misreading escapes an input edit exactly when the misreading commutes with the edit's known answer-transform, so the set of errors any suite of edits can miss is the joint centralizer of those transforms. Because invariance edits have the identity transform, which everything commutes with, every invariance-based method carries the same blind spot, and adding invariance edits cannot repair it. On the constructive side, the paper shows that two matched equivariance edits, scaling by c and adding δ, catch every affine reading error, while for label errors no swap suite is complete and cyclic relabeling provably shrinks the gap. The payoff is a design rule: which edit suite to use can be decided in advance rather than guessed.

What carries the argument

The central object is the commutation relation between a systematic error map e and an edit's answer-transform φ_g. The paper defines the residual ρ_g = d(ŷ_g, φ_g(ŷ)) between the model's answer on the edited figure and the transform of its own base answer; Lemma 1 shows ρ_g vanishes exactly when e and φ_g commute at the true answer. This turns the undetectable set of a suite into the joint centralizer of its transforms, a set that shrinks monotonically as edits are added and can be computed without knowing the model's internal state.

What would settle it

On a matched m=6 sample, if cyclic relabeling does not catch roughly 48/6 = 8 times as many label errors as a swap edit, Corollary 6's design rule fails; alternatively, a model whose scale and offset residuals both vanish while its misreadings are affine and non-identity would refute Lemma 1 and Theorem 4.

Watch

Extended reading notes

Core claim

The central claim is that detectability of a systematic reading error by an equivariance edit is decided by a commutation condition. Writing φ_g for the transform an edit g applies to the correct answer, a style-invariant error map e is invisible to g exactly when e∘φ_g = φ_g∘e (up to the edit's tolerance), so the residuals the suite cannot see form the joint centralizer ∩_g C_τ(φ_g) (Theorem 2). Consequently invariance, whose transform is the identity, has the full error class as its centralizer and can never see the systematic errors it certifies; combining invariance with an equivariance suite leaves the undetectable set unchanged (Corollary 3). Within affine errors, the pair {×c, +δ} with c≠1, δ≠0 is complete (Theorem 4); within label permutations, a swap leaves 2(m−2)! errors undetectable while a cyclic relabeling of order m leaves only m, giving a design rule for m≥5 (Proposition 5, Corollary 6).

Load-bearing premise

The entire characterization assumes the model applies the same deterministic, style-invariant error map to the base rendering and to every edited rendering; if mistakes depend on the input or the style, the commutation residual no longer equals the distance between e(φ_g(a*)) and φ_g(e(a*)), and the completeness results do not transfer.

Editorial extensions

If this is right

  • Invariance-family reliability signals (re-rendering, prompt perturbation, metamorphic robustness) all share the same blind spot, and combining them with equivariance edits cannot remove it.
  • A two-edit suite of scale and offset detects every affine reading error; detecting a pure offset bias needs the scale edit and a pure scale bias needs the offset edit.
  • For label questions, swap edits cannot be complete; at m≥5 categories, replacing swaps with cyclic relabeling shrinks the undetectable set from 2(m−2)! to m, predicting an 8× gain at m=6.
  • ECS flags a substantial part of the invariance-blind error mass at a matched false-positive rate, improving combined AUROC and calibration in the high-confidence region.
  • The same characterization explains why a classifier metamorphic-testing study reports the reverse ordering over relation families: detectability is a joint property of the relation and the fault class.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the commutation characterization is right, suite design becomes a group-theory search: the ideal suite for a known error class is a set of transforms whose joint centralizer is the identity, and the paper's methods give a way to search for such sets beyond affine and permutation errors.
  • A natural extension is to stochastic models: replace the deterministic residual with an expectation over repeated draws, and ask whether the ordering in Theorem 7 survives; the paper states this extension as a conjecture rather than a result.
  • Appendix A states that the proofs were drafted with AI assistance and that the human authors must re-derive them independently before submission, so the completeness theorems are best treated as machine-checked results pending that re-derivation.
  • The paper itself reports that delete-max and offset edits are measurably easier than base on all three models, so absolute detection rates for those edits carry a difficulty confound; the theory-driven ordering over edit types is the more robust deliverable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. This paper develops a commutation-theoretic account of label-free reliability for chart-reading vision-language models. An edit of the underlying chart data is paired with a known answer-transform phi_g; a systematic reading error e is invisible to the edit exactly when e commutes with phi_g (Lemma 1), and the errors invisible to a suite G form the joint centralizer of its transforms (Theorem 2). The authors prove that scale and offset edits are jointly complete for affine errors (Theorem 4), compute centralizers for label permutations (Proposition 5), propose cyclic relabeling as a better design choice (Corollary 6), and instantiate the theory in the ECS score and the REND-EQUIV benchmark. Experiments across three models, a hand-labeled control population, and a real-figure pilot are presented as confirmations of the predicted ordering.

Significance. If the theorem-level claims are taken in their correct scope, the paper provides a genuinely useful closed-form answer to a question that the metamorphic-testing literature treats only combinatorially: which systematic errors can a given edit suite detect? The central algebraic steps are clean, the proofs are machine-checked with exhaustive enumeration for the permutation case, the predictions were fixed before measurement and include predicted failures, and the released artifact pairs matched invariance and equivariance sets. These are real strengths. However, the paper's headline incompleteness claim for swap suites is overbroad as written, and one empirical 'closed-form prediction' match is not actually derived from the theory. With those two points corrected, the central characterization remains valuable and the empirical design-rule comparison is still meaningful, albeit more qualitatively than the current text claims.

major comments (2)
  1. [Abstract; §1 Contribution 2; §4.3 Proposition 5; Figure 2] The claim that 'no suite of swap edits is complete for label permutations' is false for m >= 3. Any generating set of transpositions in S_m has trivial joint centralizer, so a suite of appropriately chosen distinct swaps detects every non-identity label permutation. For example, in S_6 the adjacent transpositions (12), (23), (34), (45), (56) jointly commute only with the identity, so that six-edit swap suite has Upsilon_0 = {id}. Proposition 5 proves only the weaker statement that no suite of swaps on a fixed pair is complete; its proof (the transposition lies in its own centralizer) does not extend to suites of different swaps. Because the stronger statement appears in the abstract, Contribution 2, and Figure 2, and because Appendix A reports exhaustive enumeration of S_m for m <= 6 (which would exhibit the counterexample), this is an internal inconsistency. The corrected statement still leaves the empirical swap-versus-cycle comparison meaningful as a comparison of single edits, but the advertised 'no suite' conclusion must be removed or explicitly restricted.
  2. [§4.3 Corollary 6; §6] The claimed quantitative match between the 8x detection improvement and the closed-form ratio 2(m-2)!/m = 48/6 is not a consequence of Proposition 5. Centralizer sizes are counts of invisible errors, not detection rates; converting them into detection probabilities requires an error distribution, and under the natural uniform distribution over S_m the predicted detection rates would be 1 - 48/720 = 93.3% for a swap and 1 - 6/720 = 99.2% for a cycle, a ratio of about 1.06, not 8. The observed rates (5.3% and 44.0%) are far lower, so the empirical error distribution is strongly non-uniform, and nothing in the theory as stated predicts that the ratio of detected counts equals the ratio of centralizer sizes. The qualitative prediction (cyclic relabeling detects more label errors than a single swap) is confirmed, but the quantitative 'almost exactly' claim should be removed or derived from an explicit distributional model.
minor comments (4)
  1. [§6, Table 2] The perturbation-uncertainty confirmation of Corollary 3 rests on 0 detections in a 10-instance IBEM sample; the exact binomial 95% confidence interval reaches about 26%, so the statement that the method 'matches REA exactly' is much weaker than the surrounding text suggests. Please report the interval or obtain a larger sample.
  2. [§6, Table 1] The claim that the predicted per-edit-type ordering holds across three models is not documented in the main text, which shows only Qwen2.5-VL-7B; the other two models' per-edit-type rates are referenced but not displayed. Please add the per-model tables to an appendix or point explicitly to where they appear.
  3. [§3 and §4] Because the deployed ECS uses a 5% relative tolerance for continuous answers while Theorem 4 is stated at tau = 0, the theorem guarantees a nonzero residual, not detection at the implemented threshold. A sentence noting that tau > 0 enlarges the undetectable set beyond {id} would prevent readers from over-reading the completeness claim.
  4. [Appendix A; §10] The appendix sentence stating that the human authors 'must re-derive them independently before submission' is an artifact of the drafting process and conflicts with the claim that the proofs are machine-checked. Please clarify the verification status or remove the sentence.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity: the one acknowledged identity-edit selection circularity is local and removed by a REA-independent hand-labeled check.

  1. other [Section 6, Table 1 footnote a and IBEM definition]
    "IBEM is defined offline as instances with REA≥0.75 whose base answer is wrong. ... Partly forced: IBEM is selected at REA≥0.75, itself re-rendering agreement, so this row corroborates Theorem 2 without testing it on its own; a hand-labeled, REA-independent check below removes this circularity."

    The identity/zoom edit is an invariance-family perturbation, and the IBEM population is defined by REA≥0.75, i.e., stability under cosmetic re-renderings. Therefore the identity row's near-zero detection is partly built into the selection rule rather than independently testing Theorem 2. The paper concedes this ('partly circular') and supplies a hand-labeled REA-independent population that reproduces the ordering, so the circularity is acknowledged and locally mitigated, not load-bearing for the main claim.

full rationale

The derivation chain is largely self-contained algebra: Lemma 1 and Theorem 2 restate the residual definition in commutation language, while the completeness and incompleteness counts (Theorem 4, Proposition 5, Corollary 6) are direct computations with no fitted constants. The empirical ordering was fixed before measurement and includes predicted failures, so it is not reverse-engineered from the data. The only genuine circularity the paper concedes is the identity-edit row of Table 1: IBEM is selected by REA≥0.75, which is itself re-rendering agreement, so testing the zoom/identity edit on that population is partly forced. The paper explicitly labels this and removes it with a REA-independent hand-labeled 200-instance sample that reproduces the ordering, so the central claim does not rest on the circular row. REA [4] is a self-citation, but it is used as a baseline and independently corroborated by perturbation uncertainty and the hand-labeled population, so it is not load-bearing. The overbroad 'no suite of swaps is complete' wording is a mathematical correctness risk (a generating set of transpositions has trivial joint centralizer), not a circularity, and is outside this score.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central theory assumes known answer-transforms (Eq. 1), deterministic style-invariant error maps, and roughly matched task difficulty. The free parameters are the 5% tolerance, the ECS threshold, and the chosen edit magnitudes; no constants are fitted to make the theorems work. The permutation-class results use standard group-theoretic facts about centralizers. No new physical or latent entities are introduced.

free parameters (3)
  • ECS continuous tolerance tau_g = 0.05 relative error
    Hand-chosen 5% relative tolerance held constant across operators and chart families (Section 3, Eq. 2). It is not fitted but affects ECS pass/fail and is a free design constant.
  • ECS decision threshold t = 0.5, with cross-validated alternative 0.70
    ECS flags answers when ECS<t. The paper uses 0.5 as theory-grounded and reports that cross-validation on seed 0 selects 0.70 (Appendix C).
  • Edit magnitudes (scale factor c, offset delta) = e.g., c=2 in ChartQA pilot; not specified uniformly
    Specific magnitudes are design choices; Theorem 4 needs only c != 1 and delta != 0, but empirical residuals and tolerances depend on the chosen magnitudes.
assumptions (5)
  • domain assumption A deterministic program computes the exact answer a*(D,q), and each edit g induces a known answer-transform phi_g satisfying Eq. 1.
    Section 3 defines the setting; the entire commutation theory operates on phi_g. If an edit's effect on the correct answer is not known or computable, the equivariance residual cannot be formed.
  • domain assumption A systematic error e is deterministic and style-invariant: the model applies the same error map to base and edited renderings.
    Assumed throughout Section 4 and in Theorem 7's idealizations; Lemma 1's residual identity depends on it. The paper validates it only indirectly via greedy inference and difficulty checks.
  • domain assumption Edits leave task difficulty unchanged.
    Section 5 validates this for scale, swap-labels, and zoom, but delete-max and offset are measurably easier than base (6.9-12.5 and 3.8-8.4 points), so the assumption fails for two of five operators.
  • standard math For label-permutation results, the error class E = S_m is a group under composition.
    Used for the subgroup clause of Theorem 2 and Proposition 5's centralizer counts.
  • domain assumption Real chart-reading errors fall into affine or permutation classes.
    The empirical hand-label of 200 instances classifies 75.5% affine and 24.5% permutation with zero third category, but this is a finite sample labeled by the authors, not a theorem.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Consistency Has a Computable Blind Spot: A Commutation Theory of Label-Free Reliability for Vision-Language Figure Reading." pith.science (2026). https://pith.science/paper/44K2UFXG

@misc{pith2026260805675,
  author       = {Pith},
  title        = {Pith review of: Consistency Has a Computable Blind Spot: A Commutation Theory of Label-Free Reliability for Vision-Language Figure Reading},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/44K2UFXG}},
  note         = {Machine review of arXiv:2608.05675}
}
read the original abstract

Label-free reliability for vision-language models rests on invariance: perturb the input and a faithful reader's answer should not change. This has a known blind spot, a systematic misreading survives the perturbation and gets certified wrong, which we show is computable, not just real: an error is invisible to an edit exactly when the two commute, so the errors a suite cannot reach form its joint centralizer, a set that shrinks as edits are added and can be written down rather than guessed at. We act on the complementary relation, equivariance: edit a figure's data and the correct answer must change by a computable amount. Two matched edits are provably complete for affine reading errors; no suite of swap edits is complete for label permutations, and cyclic relabeling closes most of that gap. We instantiate the theory as the Equivariance-Consistency Score, a label-free, training-free detector, and release REND-EQUIV, pairing matched invariance and equivariance sets over identical data. The predicted ordering holds across three models and a hand-labeled population immune to the one circularity in how it is selected; a second invariance-family method confirms the blind spot belongs to the relation, not to any implementation; and cyclic relabeling delivers its predicted gain on a matched real sample. The same characterization explains a reported inversion of this ordering in the classifier metamorphic-testing literature: detectability is a joint property of the relation and the fault class, never of the relation alone.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 19 canonical work pages

  1. [1]

    Consistency and uncertainty: Identifying unreliable responses from black- box vision-language models for selective visual question answering

    Zaid Khan and Yun Fu. Consistency and uncertainty: Identifying unreliable responses from black- box vision-language models for selective visual question answering. In2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10854–10863. IEEE, 2024

    work page 2024

  2. [2]

    Decompose and Compare Consistency: Measuring VLMs' Answer Reliability via Task-Decomposition Consistency Comparison

    Qian Yang, Weixiang Yan, and Aishwarya Agrawal. Decompose and compare consistency: Measuring vlms’ answer reliability via task-decomposition consistency comparison, 2024. URL https://arxiv. org/abs/2407.07840

    work page Pith review arXiv 2024

  3. [3]

    Zoom consistency: A free confidence signal in multi-step visual grounding pipelines.arXiv preprint arXiv:2604.15376, 2026

    Keon Kim and Krish Chelikavada. Zoom consistency: A free confidence signal in multi-step visual grounding pipelines.arXiv preprint arXiv:2604.15376, 2026

    arXiv 2026

  4. [4]

    When does consensus mean correctness? measuring the agreement–accuracy coupling with semantics-preserving re-rendering, 2026

    Rasul Khanbayov and Hasan Kurban. When does consensus mean correctness? measuring the agreement–accuracy coupling with semantics-preserving re-rendering, 2026

    work page 2026

  5. [5]

    Vl-uncertainty: Detecting hallucination in large vision-language model via uncertainty estimation.arXiv preprint arXiv:2411.11919, 2024

    Ruiyang Zhang, Hu Zhang, and Zhedong Zheng. Vl-uncertainty: Detecting hallucination in large vision-language model via uncertainty estimation.arXiv preprint arXiv:2411.11919, 2024

    arXiv 2024

  6. [6]

    Metara: Metamorphic robustness assessment for multimodal large language model-based visual question answering systems.arXiv preprint arXiv:2605.19307, 2026

    Quanxing Xu, Yuhao Tian, Ling Zhou, Xian Zhong, Xiaohua Huang, Rubing Huang, and Chia-Wen Lin. Metara: Metamorphic robustness assessment for multimodal large language model-based visual question answering systems.arXiv preprint arXiv:2605.19307, 2026

    arXiv 2026

  7. [7]

    Bidirectional Empowerment of Metamorphic Testing and Large Language Models: A Systematic Survey

    Zheng Zheng, Zenghui Zhou, Yinwang Xu, Daixu Ren, and Tsong Yueh Chen. Bidirectional em- powerment of metamorphic testing and large language models: A systematic survey.arXiv preprint arXiv:2605.13898, 2026

    work page Pith review arXiv 2026

  8. [8]

    Minimum Complete MR Subsets under Semantic-Mutation Fault Models: A Support-Set Domination Boundary

    Meng Li, Xiaohua Yang, Jie Liu, and Shiyu Yan. Minimum complete mr subsets under semantic-mutation fault models: A support-set domination boundary, 2026. URLhttps://arxiv.org/abs/2606.08269

    work page Pith review arXiv 2026

Show all 35 references
  1. [9]

    Learning more from less: Exploiting counterfactuals for data-efficient chart understanding

    Jianzhu Bao, Haozhen Zhang, Kuicai Dong, Bozhi Wu, Sarthak Ketanbhai Modi, Zi Pong Lim, Yon Shin Teo, and Wenya Wang. Learning more from less: Exploiting counterfactuals for data-efficient chart understanding. InProceedings of the 64th Annual Meeting of the Association for Com...

    2026

  2. [10]

    What if the tv was off? examining counterfactual reasoning abilities of multi-modal language models

    Letian Zhang, Xiaotong Zhai, Zhongkai Zhao, Yongshuo Zong, Xin Wen, and Bingchen Zhao. What if the tv was off? examining counterfactual reasoning abilities of multi-modal language models. In2024 13 CONSISTENCY HAS A COMPUTABLE BLIND SPOTKurban Intelligence Lab IEEE/CVF Confere...

    2024

  3. [11]

    Pixels versus priors: Controlling knowledge priors in vision-language models through visual counterfacts

    Michal Golovanevsky, William Rudman, Michael A Lepori, Amir Bar, Ritambhara Singh, and Carsten Eickhoff. Pixels versus priors: Controlling knowledge priors in vision-language models through visual counterfacts. InProceedings of the 2025 Conference on Empirical Methods in Natur...

    2025

  4. [12]

    Unraveling the truth: Do vlms really understand charts? a deep dive into consistency and robustness,

    Srija Mukhopadhyay, Adnan Qidwai, Aparna Garimella, Pritika Ramu, Vivek Gupta, and Dan Roth. Unraveling the truth: Do vlms really understand charts? a deep dive into consistency and robustness,

  5. [13]

    Losing the plot: How vlm responses degrade on imperfect charts.arXiv preprint arXiv:2509.18425, 2025

    Philip Wootaek Shin, Jack Sampson, Vijaykrishnan Narayanan, Andres Marquez, and Mahantesh Halappanavar. Losing the plot: How vlm responses degrade on imperfect charts.arXiv preprint arXiv:2509.18425, 2025

    2025

  6. [14]

    Chart-rvr: Re- inforcement learning with verifiable rewards for explainable chart reasoning, 2025

    Sanchit Sinha, Oana Frunza, Kashif Rasul, Yuriy Nevmyvaka, and Aidong Zhang. Chart-rvr: Re- inforcement learning with verifiable rewards for explainable chart reasoning, 2025. URL https: //arxiv.org/abs/2510.10973

    2025

  7. [15]

    On robustness and chain-of-thought consistency of rl-finetuned vlms, 2026

    Rosie Zhao, Anshul Shah, Xiaoyu Zhu, Xinke Deng, Zhongyu Jiang, Yang Yang, Joerg Liebelt, and Arnab Mondal. On robustness and chain-of-thought consistency of rl-finetuned vlms, 2026. URL https://arxiv.org/abs/2602.12506

    2026 arXiv

  8. [16]

    Metamorphic testing of large language models for natural language processing

    Steven Cho, Stefano Ruberto, and Valerio Terragni. Metamorphic testing of large language models for natural language processing. In2025 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 174–186. IEEE, 2025

    2025

  9. [17]

    Test adequacy for metamorphic testing: Criteria, measurement, and implication, 2024

    An Fu, Chang ai Sun, Jiaming Zhang, and Huai Liu. Test adequacy for metamorphic testing: Criteria, measurement, and implication, 2024. URLhttps://arxiv.org/abs/2412.20692

    2024 arXiv

  10. [18]

    Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In2024 I...

    2024

  11. [19]

    Nguyen, Yifan Shen, Tianjiao Yu, and Ismini Lourentzou

    Xinzhuo Li, Adheesh Juvekar, Jiaxun Zhang, Xingyou Liu, Muntasir Wahed, Kiet A. Nguyen, Yifan Shen, Tianjiao Yu, and Ismini Lourentzou. Counterfactual segmentation reasoning: Diagnosing and mitigating pixel-grounding hallucination, 2026. URLhttps://arxiv.org/abs/2506.21546

    2026 arXiv

  12. [20]

    Charthal: A fine- grained framework evaluating hallucination of large vision language models in chart understanding,

    Xingqi Wang, Yiming Cui, Xin Yao, Shijin Wang, Guoping Hu, and Xiaoyu Qin. Charthal: A fine- grained framework evaluating hallucination of large vision language models in chart understanding,

  13. [21]

    Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

    2022 arXiv

  14. [22]

    Vl-calibration: Decoupled confidence calibration for large vision-language models reasoning, 2026

    Wenyi Xiao, Xinchi Xu, and Leilei Gan. Vl-calibration: Decoupled confidence calibration for large vision-language models reasoning, 2026. URLhttps://arxiv.org/abs/2604.09529

    2026 arXiv

  15. [23]

    On the foundations of noise-free selective classification.Journal of Machine Learning Research, 11(5), 2010

    Ran El-Yaniv et al. On the foundations of noise-free selective classification.Journal of Machine Learning Research, 11(5), 2010

    2010

  16. [24]

    On calibration of modern neural networks

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. InInternational conference on machine learning, pages 1321–1330. PMLR, 2017

    2017

  17. [25]

    selective prediction

    Tejas Srinivasan, Jack Hessel, Tanmay Gupta, Bill Yuchen Lin, Yejin Choi, Jesse Thomason, and Khy- athi Chandu. Selective “selective prediction”: Reducing unnecessary abstention in vision-language reasoning. InFindings of the Association for Computational Linguistics: ACL 2024...

    2024

  18. [26]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024. URLhttps://arxiv.org/abs/2408.03314

    2024 arXiv

  19. [27]

    Chart-coca: Self-improving chart understanding of vision lms via code-driven synthesis and candidate-conditioned answering, 2025

    Gongyao Jiang and Qiong Luo. Chart-coca: Self-improving chart understanding of vision lms via code-driven synthesis and candidate-conditioned answering, 2025. URL https://arxiv.org/abs/25 08.11975

    2025

  20. [28]

    Qwen2.5-vl technical report, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Han...

    2025 arXiv

  21. [29]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.Science China Information Sciences, 67(12):220101, 2024

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.Science China Information Sciences, 67(12):220101, 2024

    2024

  22. [30]

    Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation.arXiv preprint arXiv:2302.09664, 2023

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation.arXiv preprint arXiv:2302.09664, 2023

    2023 arXiv

  23. [31]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Jia Qing Tan, Shafiq Joty, Enamul Hoque, et al. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. InFindings of the association for computational linguistics: ACL 2022, pages 2263–2279, 2022

    2022

  24. [32]

    Testing and validating machine learning classifiers by metamorphic testing.Journal of Systems and Software, 84(4):544–558, 2011

    Xiaoyuan Xie, Joshua WK Ho, Christian Murphy, Gail Kaiser, Baowen Xu, and Tsong Yueh Chen. Testing and validating machine learning classifiers by metamorphic testing.Journal of Systems and Software, 84(4):544–558, 2011

    2011

  25. [33]

    how many countries have the same value

    Potsawee Manakul, Adian Liusie, and Mark Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. InProceedings of the 2023 conference on empirical methods in natural language processing, pages 9004–9017, 2023. APPENDIX A Proof...

    2023

  26. [2024]

    URLhttps://arxiv.org/abs/2407.11229

    arXiv

  27. [2025]

    URLhttps://arxiv.org/abs/2509.17481

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.