Pith. sign in

REVIEW 3 major objections 4 minor 30 references

ConfBench measures whether a vision-language model's confidence scores are trustworthy enough to route document extractions to automation or to a human reviewer.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 20:38 UTC pith:BELLFSLY

load-bearing objection ConfBench is a genuinely useful benchmark and ECARB is a sensible metric, but the undisclosed binary-correctness threshold means the headline numbers need rechecking before you trust the rankings. the 3 major comments →

arxiv 2608.01792 v1 pith:BELLFSLY submitted 2026-08-03 cs.AI cs.CL

Can You Trust the Confidence? ConfBench for Vision-Language Models on Document Extraction

Priyashree Roy , Sujitha Martin , Mohammad Rostami , Spencer Romo , Renhao Xue , Bob Strahan , Diego A. Socolinsky , Boyi Xie
show 1 more author
Md Mofijul Islam
This is my paper
classification cs.AI cs.CL
keywords confidence calibrationvision-language modelskey information extractiondocument degradationcalibration benchmarkECARBhuman-in-the-loop reviewintelligent document processing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Systems that process invoices and contracts automatically rely on the model's confidence score to decide which extractions are safe to automate and which need a human reviewer, so a confidence score that cannot be trusted silently turns into errors. The paper argues that no existing document benchmark could test that trust, because benchmarks built on clean scans produce almost all-correct predictions, leaving the low-accuracy region where confidence actually matters too sparse to measure. It introduces ConfBench, built by running 20 controlled degradation pipelines (blur, skew, stains, bleed-through, JPEG noise, and more) over 75 manually verified FCC invoices to create 1,346 degraded variants whose extraction accuracy spans the full spectrum, with over 70,000 entity-level evaluations. On this substrate the paper reports the first systematic comparison of seven vision-language models, two confidence-estimation families (verbalized self-report and token log-probability), and three input modalities: confidence quality varies from near-perfect (AUROC 0.84, ECE 0.05) to severely overconfident (AUROC 0.58, ECE 0.31), is driven mainly by model capability rather than parameter count, and is uniformly best when the model receives both OCR text and the image. The paper's practical claim is that a deployment team can now measure, before rollout, whether a model's confidence is trustworthy enough to route work to automation, and can quantify the human-review savings with the new ECARB metric.

Core claim

The central claim is that confidence calibration for document extraction is a measurable, deployment-relevant property that varies widely and predictably. Feeding a model both the OCR text and the document image is uniformly the strongest input configuration for confidence quality, with the largest gap over image-only appearing in smaller models. Within the Claude family, confidence quality scales monotonically with capability, with Opus 4.6 attaining AUROC 0.84 and ECE 0.05, close to ideal calibration without any post-hoc correction, while Haiku 4.5 falls to 0.74 and 0.17. Across families parameter count is a poor predictor, since a 27-billion-parameter model outranks a 235-billion-paramete

What carries the argument

The argument is carried by three objects. ConfBench: 20 controlled degradation pipelines built with an open-source document-augmentation library—ink-level (bleed-through, dithering, low ink), paper-level (stains, moiré patterns, watermarks), and capture-level (JPEG compression, skew, shadows) artifacts—applied to 75 manually verified FCC invoices to yield 1,346 variants whose entity-level extraction accuracy spans the full spectrum, the regime where calibration can actually be distinguished from luck. The confidence estimators: 'verbalized' 1S-TopK, where the model emits its top four candidate values with probabilities in a single call, and log-probability, where confidence is read off the t

Load-bearing premise

Every calibration number in the paper depends on turning each entity's continuous similarity score into a binary right/wrong label with a per-field acceptance threshold that is never stated; different thresholds would change the AUROC, ECE, Brier, and ECARB values and could reorder the models.

What would settle it

Recompute the paper's metrics on the released ConfBench images and ground truth using several explicit acceptance thresholds (for example, exact match, 90% similarity, and 70% similarity) to create the correct/incorrect labels, and check whether the reported orderings survive—Opus above Sonnet above Haiku, verbalized above log-probability for Qwen but below it for Gemma. If the rankings change with the threshold, the headline confidence-quality results are artifacts of an unpublished choice; if they hold, the claims are robust to it.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Before rollout, a deployment team can run ConfBench on its candidate model and, from AUROC plus ECARB at its own review budget, decide whether confidence-gated automation is safe or whether humans must review.
  • OCR+Image should be the default input for confidence-sensitive extraction: it beats OCR-only and image-only on both accuracy and confidence quality, most clearly for smaller models.
  • Calibration is situational, not a blanket requirement: per-model post-hoc rescaling fixes absolute confidence values for threshold-based routing but leaves ranking-based metrics (AUROC, ECARB) unchanged, so teams should rescale only where an absolute threshold gates automation.
  • For models that expose token probabilities, first-token log-probability is the preferred confidence aggregation over mean-token and margin; for closed models, verbalized confidence is usable and, for the strongest Claude model, nearly well-calibrated.
  • Model capability, not parameter count, predicts confidence quality: within the Claude family confidence scales with tier, while across families a much smaller model can outperform a much larger one.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The published rankings should be treated as provisional until the per-field acceptance thresholds that convert similarity scores into correct/incorrect labels are released; because the benchmark's images and ground truth are public, any reader can re-derive the metrics once those thresholds are known.
  • The OCR+Image result suggests a cheap production rule worth testing: add per-token OCR confidence to the prompt rather than relying on the image alone, since the textual signal appears to carry most of the discriminative information, especially for weaker models.
  • The paper's two-point evidence on which confidence family wins implies the honest deployment practice is per-model evaluation, and also points to a natural next experiment the paper names: ensembling verbalized and log-probability scores, which draw on different information.
  • The tiered degradation structure makes ConfBench a ready-made testbed for calibration transfer—whether a calibrator fitted on mild degradation holds on severe degradation—a question the paper lists as future work but that the released data can answer immediately.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces ConfBench, a benchmark for evaluating confidence calibration of vision-language models (VLMs) on key information extraction (KIE) from documents. ConfBench is built by applying 20 controlled degradation pipelines to 75 verified FCC invoices, producing 1,346 document variants and 70K+ entity-level evaluations across the accuracy spectrum. The authors evaluate seven VLMs with verbalized confidence (1S-TopK) and two open-weight models with three log-probability aggregation methods, across OCR-only, image-only, and OCR+image input modalities. They also propose ECARB, a review-budget metric that measures how much confidence-guided review improves over random sampling. The main findings are that OCR+image is the strongest modality for both accuracy and confidence quality; model capability, rather than parameter count, dominates confidence quality; calibration quality varies widely across models; and first-token log-probability aggregation outperforms mean-token and margin. The benchmark and code are publicly released.

Significance. If the empirical findings hold, ConfBench addresses a genuine gap: existing document benchmarks concentrate on clean, high-accuracy samples and are not suitable for calibration assessment. The controlled degradation strategy, the explicit scoping of claims, and the public release of the benchmark, augmentation configurations, and code are notable strengths. ECARB is a practical contribution that connects discriminative confidence quality to human-in-the-loop review economics. The paper is transparent about many limitations, including the small source corpus, the two-model basis for the verbalized-vs-logprob comparison, and the lack of post-hoc calibration benchmarks. However, the central quantitative claims currently rest on an undisclosed binary-correctness threshold and on single-point metric estimates without uncertainty quantification, which limits the reproducibility and evidentiary strength of the headline rankings.

major comments (3)
  1. [Appendix A.5, §4.2] The conversion of continuous per-field similarity scores into binary correctness labels uses a 'per-field acceptance threshold' (Appendix A.5), but the threshold values are never stated. AUROC, ECE, Brier, and ECARB are all computed from these binary labels: E in Eq. (1) counts entities with acc_i=0, ECE bins use binary accuracy, and AUROC partitions correct from incorrect. The reported AUROC range 0.58–0.84, ECE 0.05–0.36, and all model/modality orderings in Table 3 can shift if the string-similarity cutoff, numeric tolerance, or date equivalence rule is changed. Please report the exact per-field thresholds used by STICKLER, state whether the same threshold is applied across fields and runs, and provide a sensitivity analysis (e.g., thresholds at several similarity levels) showing that the model rankings and ECARB conclusions are stable. Without this, the central empirical findings are
  2. [Table 3, §5.1–5.3] The main results table reports a single value per metric per configuration, with no error bars, confidence intervals, or significance tests. The paper elsewhere uses nine runs to average accuracy drop (Table 1, Figure 7), yet the confidence metrics in Table 3 appear to come from a single evaluation pass. Claims such as 'OCR+Image is uniformly the strongest modality' and 'within the Claude family confidence quality scales monotonically' are ordinal claims that may be sensitive to run-to-run variability, especially for small gaps (e.g., Haiku 0.74 vs Qwen 3.6-27B 0.72 AUROC; Sonnet 0.77 vs Opus 0.84 AUROC). The authors appropriately caution against reading small gaps as strict orderings (Appendix A.6), but the headline findings need a quantitative measure of uncertainty. Please report per-run variability or bootstrap confidence intervals for the key metric values and rankings.
  3. [§4.2, Eq. (1)] The claim that 'because ECARB depends on the ranking confidence induces rather than on absolute confidence values, model rankings are largely stable across budgets' is not supported by the reported evidence. Only b=30% is reported; no ECARB curves or values at other budgets are shown. Since the ECARB ceiling min(1/b, n/E) depends on b and the composition of the low-confidence slice changes with b, ranking stability is an empirical assertion, not a consequence of the definition. Please report ECARB at multiple review budgets (e.g., 10%, 20%, 50%) or provide a formal argument for budget invariance.
minor comments (4)
  1. [Appendix A.5] Minor typographical and consistency issues: 'CONFBENCH' appears in uppercase in Appendix A.5; Table 2's '#p' column is ambiguous for ConfBench (1,346 documents vs. 6.7K pages). Please clarify the unit.
  2. [Eq. (1)] ECARB is undefined when E=0 (no errors) or when k=0. State the convention for these degenerate cases, since near-perfect extraction is a plausible regime.
  3. [§5.2 / Appendix A.6] The discussion of coupling between extraction and confidence estimation would be clearer if it explicitly stated that the proposed decoupled two-stage verbalized variant is not evaluated in this paper; as written, a reader may mistakenly infer it was part of the experimental design.
  4. [Reference list] Several references are incomplete or inconsistent in formatting (e.g., Shrivastava et al. 2024 has no venue; multiple arXiv entries lack arXiv IDs). Please standardize.

Circularity Check

0 steps flagged

No circularity: ConfBench is an empirical benchmark study; the headline results are measurements computed from held-out model outputs against a released ground-truth dataset, not a derivation that reduces to its own inputs.

full rationale

ConfBench is an empirical measurement study rather than a derivation chain. Its headline results—AUROC/ECE/Brier/ECARB ranges, modality ordering, and model-family rankings—are computed by applying standard metrics to model outputs scored against ground-truth annotations whose provenance is a released dataset. No quantity the paper calls a prediction is obtained by fitting a parameter and then re-measuring that parameter on the same data. The closest candidates do not rise to circularity. (1) The ground truth is inherited from RealKIE-FCC-Verified (Amazon AGI Team, 2024), a self-citation; but this is dataset provenance, not a load-bearing argument. The benchmark's validity depends on the annotations being accurate, which is an externally checkable property of a released dataset, and the paper's confidence-quality findings are not defined in terms of that dataset's construction. (2) The binary correctness labels underlying all calibration metrics depend on an undisclosed per-field acceptance threshold (Appendix A.5). This is a serious specification and reproducibility gap that can shift AUROC/ECE/ECARB results and rankings, but it is not circular: the threshold is a measurement convention, not a fitted input that the paper then presents as a prediction. (3) ECARB is defined to reward ranking quality, and the paper explicitly states that it 'is a ranking-driven quantity and therefore tracks AUROC' (Appendix A.7). That is a transparent mathematical relationship between two metrics computed from the same labels, not a hidden reduction of a derived result to its own input. The paper also openly scopes its limitations—small source corpus, two-model strategy comparisons, partial cross-model controls—and none of these admissions reveals a step that is equivalent to its assumptions by construction. Overall, the central claims are self-contained empirical findings with no significant circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The empirical evaluation rests on a small set of design choices, chiefly the undisclosed correctness threshold, on ground truth authored by the same group, and on a synthetic degradation distribution created by the authors. The ECARB metric is a new construct with no external validation.

free parameters (3)
  • Per-field acceptance threshold for binary correctness labels
    STICKLER computes a continuous similarity score per entity, then 'each score is converted to a binary correctness label using a per-field acceptance threshold' (Appendix A.5). The threshold value is never disclosed. All calibration metrics (AUROC, ECE, Brier, ECARB) and the error count E in ECARB depend on this threshold; changing it can reorder models.
  • ECE bin count M = 5
    Adaptive quantile binning at M=5 (Section 4.2) is a design choice; ECE values are bin-count-sensitive.
  • ECARB review budget b = 0.30
    ECARB@0.30 is reported as a representative operating point (Section 4.2); absolute values change with b, though rankings are claimed largely stable.
axioms (4)
  • domain assumption Ground-truth annotations in RealKIE-FCC-Verified are correct and remain valid after augmentation.
    The benchmark preserves ground-truth labels across all augmented variants and relies on them to label every entity correct/incorrect (Section 3.1). Annotation errors would corrupt every reliability bin, as the Limitations section itself notes.
  • domain assumption Augraphy degradation pipelines with the chosen parameters produce document variants whose accuracy distribution spans the full spectrum without altering ground truth.
    Section 3.1 describes OCR-safe parameter constraints and quality filtration; this is assumed, not validated against real-world degradation.
  • domain assumption Per-entity pooling of calibration metrics across varying entity counts yields unbiased cross-model comparisons.
    Appendix A.6 argues differing entity counts do not bias per-entity metrics, but it assumes the entity distributions (line-item lengths, difficulty) are comparable across models.
  • standard math Standard definitions of AUROC, ECE (quantile binning), Brier score, and Levenshtein-based similarity are accepted as measures of calibration and extraction quality.
    Used in Section 4.2 without derivation.
invented entities (1)
  • ECARB (Error Capture at Review Budget) no independent evidence
    purpose: Operational metric that quantifies how many more errors confidence-ordered review catches versus random sampling at a given review budget.
    Introduced in Section 4.2 and Appendix A.7. It is a simple lift ratio and is not externally benchmarked or validated against other HITL metrics; its behavior is shown only on this study's data.

pith-pipeline@v1.3.0-daily-deepseek · 22070 in / 15305 out tokens · 149567 ms · 2026-08-04T20:38:26.198693+00:00 · methodology

0 comments
read the original abstract

Intelligent document processing (IDP) with vision-language models (VLMs) hinges on confidence scores trustworthy enough to route extractions between automation and human review. Existing document benchmarks are dominated by clean, high-quality samples, leaving low accuracy regions too sparse for calibration assessment. We introduce ConfBench, the first calibration-specific benchmark for key information extraction (KIE), built by applying 20 controlled degradation pipelines to a diverse document set, yielding 1,346 variants and 70K+ entity-level evaluations spanning the full accuracy spectrum. We evaluate four proprietary and three open-weight VLMs under verbalized and log-probability confidence estimation methods across three input modalities, and find: (i) OCR+Image modality results in more accurate confidence estimates; (ii) model capability is the dominant factor: within the Claude family confidence quality scales monotonically with capability, while across families parameter count is a poor predictor; (iii) calibration quality varies widely across models, from near-perfect to severely overconfident, and per-model post-hoc correction rescales these absolute confidence values for threshold-based routing without altering ranking-based operational metrics; and (iv) log-probability with first-token aggregation consistently outperforms mean-token and margin aggregations. We also introduce ECARB, a review-budget metric translating discriminative gains into operational savings. We release ConfBench publicly to enable systematic study of confidence estimators and calibration methods for trustworthy IDP application deployment.

Figures

Figures reproduced from arXiv: 2608.01792 by Bob Strahan, Boyi Xie, Diego A. Socolinsky, Md Mofijul Islam, Mohammad Rostami, Priyashree Roy, Renhao Xue, Spencer Romo, Sujitha Martin.

Figure 1
Figure 1. Figure 1: Distribution of per-document mean entity [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Experimental design overview. Verbalized: A single model jointly extracts entities and estimates confidence in one call, producing top candidate guesses with continuous probabilities (0 to 1) per entity; the highest-probability candidate is selected. Logprob: A single-stage extraction pass produces structured output via greedy decoding, and confidence is derived from the token-level log-probabilities of ea… view at source ↗
Figure 3
Figure 3. Figure 3: Confidence quality (AUROC) vs. extrac￾tion accuracy (WOA) across all three input modalities for both verbalized and logprob first-token strategies. OCR+Image consistently occupies the upper-right re￾gion, with OCR-only ranking second and image-only third across all models. 5 Results and Discussion The following analysis isolates three factors that influence extraction accuracy and confidence mea￾sure: inpu… view at source ↗
Figure 5
Figure 5. Figure 5: Effect of model size and family on confidence [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Document 1220d048d2a75d910c52e7f290e32ca0 from our ConfBench dataset under three degradation levels. (a) Original clean scan. (b) Custom15: mild degradation (<5% accuracy drop) simulates Moiré pattern on colored paper. (c) Custom12: moderate degradation (6–10% accuracy drop) simulates dirty printer drum and rollers. (d) Custom22: high degradation (21% accuracy drop) represents dithered print with dot-matri… view at source ↗
Figure 7
Figure 7. Figure 7: Average accuracy drop in information ex [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Reliability diagrams for all verbalized (1S-TopK) configurations. Rows: models ordered by AUROC (top [PITH_FULL_IMAGE:figures/full_fig_p024_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Reliability diagrams for Gemma 3-12B: verbalized vs logprob strategies. Rows (top to bottom): verbalized, [PITH_FULL_IMAGE:figures/full_fig_p025_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Reliability diagrams for Qwen 3.6-27B: verbalized vs logprob strategies. Rows (top to bottom): [PITH_FULL_IMAGE:figures/full_fig_p026_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 16 canonical work pages · 1 internal anchor

  1. [1]

    Amazon AGI Team . 2024. Realkie-fcc-verified: A verified test set for enterprise key information extraction. https://huggingface.co/datasets/amazon-agi/RealKIE-FCC-Verified. Manually annotated and corrected subset of the RealKIE benchmark

    work page 2024

  2. [2]

    Sergio A Balanya, Juan Maronas, and Daniel Ramos. 2024. https://doi.org/10.1007/s00521-024-09505-4 Adaptive temperature scaling for robust calibration of deep neural networks . Neural Computing and Applications, 36(14):8073--8095

    doi:10.1007/s00521-024-09505-4 2024

  3. [3]

    Glenn W. Brier. 1950. https://doi.org/10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2 Verification of forecasts expressed in terms of probability . Monthly Weather Review, 78(1):1--3

    doi:10.1175/1520-0493(1950)078 1950

  4. [4]

    DL Dimmick, MD Garris, and CL Wilson. 1992. https://www.nist.gov/publications/nist-special-database-6-structured-forms-database-ii-users-guide Nist special database 6: Structured forms database 2 . Technical report, National Institute of Standards and Technology

    work page 1992

  5. [5]

    Alexander Groleau, Kok Wei Chee, Stefan Larson, Samay Maini, and Jonathan Boarman. 2023. https://doi.org/10.1007/978-3-031-41682-8_24 Augraphy: A data augmentation library for document images . In Document Analysis and Recognition -- ICDAR 2023 , Lecture Notes in Computer Science, pages 384--401. Springer

    work page doi:10.1007/978-3-031-41682-8_24 2023

  6. [6]

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. https://proceedings.mlr.press/v70/guo17a.html On calibration of modern neural networks . In Proceedings of the 34th International Conference on Machine Learning ( ICML ) , volume 70 of Proceedings of Machine Learning Research, pages 1321--1330. PMLR

    work page 2017

  7. [7]

    Adam W Harley, Alex Ufkes, and Konstantinos G Derpanis. 2015. https://doi.org/10.1109/ICDAR.2015.7333910 Evaluation of deep convolutional nets for document image classification and retrieval . In 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pages 991--995. IEEE

    arXiv 2015

  8. [8]

    Md Mofijul Islam, Md Sirajus Salekin, Nivedha Balakrishnan, Vincil C Bishop III, Niharika Jain, Spencer Romo, Bob Strahan, Boyi Xie, and Diego A Socolinsky. 2026. https://arxiv.org/abs/2602.15958 Docsplit: A comprehensive benchmark dataset and evaluation approach for document packet recognition and splitting . Computing Research Repository, arXiv:2602.15958

    work page arXiv 2026

  9. [9]

    Yifan Ji, Zhipeng Xu, Zhenghao Liu, Zulong Chen, Qian Zhang, Zhibo Yang, Junyang Lin, Yu Gu, Ge Yu, and Maosong Sun. 2026. https://arxiv.org/abs/2602.07038 Unikie-bench: Benchmarking large multimodal models for key information extraction in visual documents . Computing Research Repository, arXiv:2602.07038

    Pith/arXiv arXiv 2026

  10. [10]

    Tom Joy, Francesco Pinto, Ser-Nam Lim, Philip HS Torr, and Puneet K Dokania. 2023. https://doi.org/10.1609/aaai.v37i12.26742 Sample-dependent adaptive temperature scaling for improved calibration . In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 14919--14926

    work page doi:10.1609/aaai.v37i12.26742 2023

  11. [11]

    Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. 2022. https://doi.org/10.1007/978-3-031-19815-1_29 Ocr-free document understanding transformer . In Computer Vision -- ECCV 2022 , Lecture Notes in Computer Science, pages 498--517. Springer

    doi:10.1007/978-3-031-19815-1_29 2022

  12. [12]

    Jayant Kumar and David Doermann. 2013. https://doi.org/10.1109/ICDAR.2013.248 Unsupervised classification of structurally similar document images . In 2013 12th International Conference on Document Analysis and Recognition, pages 1225--1229. IEEE

    work page doi:10.1109/icdar.2013.248 2013

  13. [13]

    Moxin Li, Wenjie Wang, Fuli Feng, Fengbin Zhu, Qifan Wang, and Tat-Seng Chua. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.693 Think twice before trusting: Self-detection for large language models through comprehensive answer reflection . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 11858--11875. Association for...

    doi:10.18653/v1/2024.findings-emnlp.693 2024

  14. [14]

    L Rodney Long, Sameer K Antani, and George R Thoma. 2005. https://doi.org/10.1016/j.compmedimag.2004.09.015 Image informatics at a national research center . Computerized Medical Imaging and Graphics, 29(2-3):171--193

    work page doi:10.1016/j.compmedimag.2004.09.015 2005

  15. [15]

    Huan Ma, Jingdong Chen, Joey Tianyi Zhou, Guangyu Wang, and Changqing Zhang. 2025. https://arxiv.org/abs/2502.00290 Estimating LLM uncertainty with evidence . Computing Research Repository, arXiv:2502.00290

    Pith/arXiv arXiv 2025

  16. [16]

    Shreekant Mandvikar. 2023. https://doi.org/10.14445/22312803/IJCTT-V71I10P110 Augmenting intelligent document processing (idp) workflows with contemporary large language models (llms) . International Journal of Computer Trends and Technology, 71(10):80--91

    work page doi:10.14445/22312803/ijctt-v71i10p110 2023

  17. [17]

    Thisanaporn Mungmeeprued, Yuxin Ma, Nisarg Mehta, and Aldo Lipani. 2022. https://doi.org/10.1145/3558100.3563852 Tab this folder of documents: page stream segmentation of business documents . In Proceedings of the 22nd ACM Symposium on Document Engineering, pages 1--10

    work page arXiv 2022

  18. [18]

    Sriraam Natarajan, Saurabh Mathur, Sahil Sidheekh, Wolfgang Stammer, and Kristian Kersting. 2025. https://doi.org/10.1609/aaai.v39i27.35083 Human-in-the-loop or ai-in-the-loop? automate or collaborate? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 28594--28600

    doi:10.1609/aaai.v39i27.35083 2025

  19. [19]

    Shiyu Ni, Keping Bi, Lulu Yu, and Jiafeng Guo. 2025. https://doi.org/10.1007/978-981-96-1710-4_10 Are large language models more honest in their probabilistic or verbalized confidence? In Information Retrieval: 30th China Conference (CCIR 2024), Lecture Notes in Computer Science. Springer

    work page doi:10.1007/978-981-96-1710-4_10 2025

  20. [20]

    Linke Ouyang, Yuan Qu, Hongbin Zhou, Jiawei Zhu, Rui Zhang, Qunshu Lin, Bin Wang, Zhiyuan Zhao, Man Jiang, Xiaomeng Zhao, and 1 others. 2025. https://doi.org/10.1109/CVPR52734.2025.02313 Omnidocbench: Benchmarking diverse pdf document parsing with comprehensive annotations . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

    arXiv 2025

  21. [21]

    Vaishnavi Shrivastava, Percy Liang, and Ananya Kumar. 2024. https://openreview.net/forum?id=D8DAQhpznu Llamas know what GPT s don't show: Surrogate models for selective classification

    work page 2024

  22. [22]

    S t e p \'a n S imsa, Milan S ulc, Michal U r i c \'a r , Yash Patel, Ahmed Hamdi, Mat e j Koci \'a n, Maty \'a s Skalick \`y , Ji r \' Matas, Antoine Doucet, Micka \"e l Coustaty, and 1 others. 2023. Docile benchmark for document information localization and extraction. arXiv preprint arXiv:2302.05658

    Pith/arXiv arXiv 2023

  23. [23]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.330 Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback . In Proceedings of the 2023 Conference on Em...

    doi:10.18653/v1/2023.emnlp-main.330 2023

  24. [24]

    Benjamin Townsend, Madison May, Katherine Mackowiak, and Christopher Wells. 2024. https://arxiv.org/abs/2403.20101 Realkie: Five novel datasets for enterprise key information extraction . Computing Research Repository, arXiv:2403.20101

    work page arXiv 2024

  25. [25]

    Weijie Tu, Weijian Deng, Dylan Campbell, Stephen Gould, and Tom Gedeon. 2024. https://arxiv.org/abs/2402.07417 An empirical study into what matters for calibrating vision-language models . Computing Research Repository, arXiv:2402.07417

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  26. [26]

    Chen, Yoonho Lee, Eric Mitchell, and Chelsea Finn

    Johnathan Xie, Annie S. Chen, Yoonho Lee, Eric Mitchell, and Chelsea Finn. 2024 a . https://doi.org/10.18653/v1/2024.emnlp-main.1007 Calibrating language models with adaptive temperature scaling . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics

    doi:10.18653/v1/2024.emnlp-main.1007 2024

  27. [27]

    Liangru Xie, Hui Liu, Jingying Zeng, Xianfeng Tang, Yan Han, Chen Luo, Jing Huang, Zhen Li, Suhang Wang, and Qi He. 2024 b . https://arxiv.org/abs/2412.12767 A survey of calibration process for black-box llms . Computing Research Repository, arXiv:2412.12767

    Pith/arXiv arXiv 2024

  28. [28]

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2024. https://openreview.net/forum?id=gjeQKFxFpZ Can LLM s express their uncertainty? A n empirical evaluation of confidence elicitation in LLM s . In The Twelfth International Conference on Learning Representations ( ICLR )

    work page 2024

  29. [29]

    Yunpu Zhao, Rui Zhang, Junbin Xiao, Ruibo Hou, Jiaming Guo, Zihao Zhang, Yifan Hao, and Yunji Chen. 2025. https://arxiv.org/abs/2504.14848 Object-level verbalized confidence calibration in vision-language models via semantic perturbation . Computing Research Repository, arXiv:2504.14848

    Pith/arXiv arXiv 2025

  30. [30]

    Guangyu Zhu and David Doermann. 2007. https://doi.org/10.1109/ICDAR.2007.4377038 Automatic document logo detection . In Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), volume 2, pages 864--868. IEEE

    work page arXiv 2007