Beyond Logprobs: A Multi-Signal Confidence Engine for LLM-Based Document Field Extraction

Nitesh Kumar

arxiv: 2606.24420 · v1 · pith:3443LEDA · submitted 2026-06-23 · cs.CL

Beyond Logprobs: A Multi-Signal Confidence Engine for LLM-Based Document Field Extraction

Nitesh Kumar This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-25 23:50 UTCgrok-4.3pith:3443LEDA record.json open to challenge →

classification cs.CL

keywords LLM confidence estimationdocument field extractionselective predictionmulti-signal fusionHunter Mapper callsOCR layout signalszero-shot transfercalibration

0 comments

The pith

ExtractConf uses disagreement between two structurally different LLM readings of a document to estimate per-field extraction trustworthiness.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to show that token log-probabilities alone fail to separate trustworthy from untrustworthy LLM field extractions in document pipelines, collapsing toward over-confident outputs. It introduces ExtractConf, which runs a field-guided Hunter call and a document-guided Mapper call on the same input; their disagreement supplies an independent signal of error that is fused with LLM uncertainty, OCR quality, and layout features inside a simple classifier. On the DocILE invoice benchmark with a 26 percent baseline failure rate, the fused model reaches 0.928 ROC AUC and lowers selective-prediction risk by 70 percent relative to a log-probability mean. At 80 percent coverage the method yields 99.1 percent accuracy, and the same signals transfer zero-shot to the CORD receipt set at 0.858 AUC. Lightweight recalibration on the new domain further reduces expected calibration error by 89 percent.

Core claim

ExtractConf is a field-agnostic classifier that grounds confidence in the structural asymmetry between a schema-slot Hunter call and a holistic Mapper call on the identical document. Because the two calls induce distinct failure modes, their disagreement is treated as an additional informative feature alongside internal LLM uncertainty, OCR output, image quality, and spatial layout; the resulting model requires no domain rules or retraining and directly supports selective automation versus human review.

What carries the argument

The ExtractConf classifier that fuses Hunter-Mapper disagreement with LLM uncertainty, OCR, and layout signals.

If this is right

On DocILE invoices the fused model reaches 0.928 ROC AUC and cuts selective-prediction risk 70 percent versus log-probability mean.
At 80 percent coverage the method attains 99.1 percent accuracy, supporting a practical human-in-the-loop workflow.
Zero-shot application to CORD receipts yields 0.858 AUC without retraining.
Lightweight Lasso recalibration on the target domain reduces ECE by 89 percent and Brier score by 43 percent.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same dual-call structure could be applied to other structured LLM outputs where multiple prompt framings are feasible.
The approach indicates that deliberate prompt asymmetry can serve as a general, training-free source of uncertainty information.
Further testing on tables, forms, or multi-page contracts would clarify how far the disagreement signal generalizes beyond invoices and receipts.

Load-bearing premise

The disagreement produced by the two different LLM calls on the same document supplies information about extraction errors that is not already captured by the other fused signals.

What would settle it

If a new document corpus with comparable field-extraction failure rates shows that ExtractConf ROC AUC is no higher than the log-probability-mean baseline, the value of the cross-call disagreement signal would be refuted.

Figures

Figures reproduced from arXiv: 2606.24420 by Nitesh Kumar.

**Figure 1.** Figure 1: EXTRACTCONF architecture. A document feeds two asymmetric LLM calls in parallel: a field-guided Hunter (blue) and a document-guided Mapper (orange). Both report extracted values, token logprobs, and the neighbouring text they attended to. Value agreement, neighbourhood text overlap, and spatial centroid divergence (from OCR bounding boxes) form the cross-call reliability signals. OCR confidence and image q… view at source ↗

**Figure 2.** Figure 2: Group 4 spatial feature extraction pipeline. Nearby tokens from each call are matched against OCR tokens to construct per-call bounding boxes, from which label neighbourhood match scores, bounding box OCR confidence, and Laplacian sharpness are derived. Centroid distance between the two boxes measures spatial divergence between Hunter and Mapper groundings. Group 3. Cross-Call Value Agreement. Exact match:… view at source ↗

**Figure 3.** Figure 3: Calibration vs. selective prediction quality across all methods [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

**Figure 4.** Figure 4: (a) Feature importances confirm that document quality signals (HV>90, HV>75, H_OCR_LBL) dominate, followed by the Mapper cross-call signal (M_NLS), spatial centroid divergence (CENT_DIST), and logprob features. No single group is sufficient; the full fusion is required. (b) The accuracy–coverage curve operationalises the AURC metric: deferring even a modest fraction of low-confidence predictions yields nea… view at source ↗

read the original abstract

In high-stakes document processing pipelines, including financial reconciliation, compliance verification, and procurement automation, an LLM extraction that is silently wrong is more dangerous than one that is visibly absent. The central challenge is not extraction accuracy alone but reliable confidence estimation: knowing, field by field, whether an extraction can be trusted for automation or deferred to human review. Token-level log-probabilities, verbalized confidence, and multi-sample self-consistency all collapse toward all-positive behaviour at practical thresholds, offering no reliable separation between trustworthy and untrustworthy extractions. We present ExtractConf, a cross-domain, field-agnostic confidence engine that grounds confidence estimation in two structurally different readings of the same document. A field-guided Hunter call extracts each field under schema-slot completion pressure; a document-guided Mapper call scans holistically and surfaces values grounded in document content. This asymmetry yields different failure modes: Hunter hallucinates values for absent fields, while Mapper misses visually non-salient ones. Their disagreement is independently informative. ExtractConf fuses cross-call disagreement, LLM-internal uncertainty, OCR, image quality, and spatial layout into a classifier requiring no domain-specific rules or retraining. On DocILE (55-field invoices, 26% failure rate), it achieves 0.928 ROC AUC and reduces selective prediction risk by 70% over logprob-mean. At 80% coverage, accuracy reaches 99.1%, enabling a practical human-in-the-loop workflow. Zero-shot transfer to CORD receipts achieves 0.858 AUC; lightweight Lasso recalibration reduces ECE by 89% and Brier by 43%, confirming the signals generalise across document domains.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

ExtractConf's Hunter-Mapper disagreement signal looks like a useful addition for confidence scoring, but the abstract gives no evidence it contributes independently of the other fused features.

read the letter

The paper's main move is to run two differently structured LLM calls on the same document—one field-guided Hunter that fills schema slots and one document-guided Mapper that scans holistically—then treat their disagreement as a signal for whether an extraction is trustworthy. It folds that signal together with logprobs, OCR quality, image features, and layout into a single classifier that needs no domain rules.

This is a direct response to the practical failure of logprobs and self-consistency, which the abstract notes tend to stay overconfident at usable thresholds. Reporting on DocILE invoices and zero-shot transfer to CORD receipts, with numbers like 0.928 AUC and 70% risk reduction at 80% coverage, gives the work a concrete empirical anchor on public data.

The soft spot is exactly the one the stress-test flags: nothing in the abstract shows an ablation, correlation check, or feature-importance result that isolates the disagreement signal from the rest. If the gains come mostly from OCR and layout, the asymmetry idea does not carry the load the central claim assigns it. Classifier training details, feature definitions, and data splits are also absent, so the reported ECE and Brier improvements after recalibration cannot be checked for leakage or post-hoc tuning.

The work is aimed at teams running LLM extraction in financial or compliance pipelines who already have access to layout and OCR pipelines and want a better rejection threshold. A reader who needs a field-agnostic method that generalizes across document types will find the setup worth examining.

It should go to peer review. The benchmarks are public, the problem statement is sharp, and the method is specific enough that referees can test whether the new signal actually adds information once the full experiments are available.

Referee Report

2 major / 2 minor

Summary. The paper introduces ExtractConf, a field-agnostic confidence engine for LLM-based document field extraction. It grounds confidence in disagreement between a field-guided Hunter LLM call (under schema-slot pressure) and a document-guided Mapper call (holistic scan), fused with LLM-internal uncertainty, OCR quality, image quality, and spatial layout signals into a classifier. On the DocILE dataset (55-field invoices, 26% failure rate), it reports 0.928 ROC AUC, 70% selective prediction risk reduction versus logprob-mean baseline, and 99.1% accuracy at 80% coverage. Zero-shot transfer to CORD yields 0.858 AUC, with Lasso recalibration reducing ECE by 89% and Brier score by 43%.

Significance. If the empirical results and independence of the Hunter-Mapper disagreement signal hold after verification, the work offers a practical, cross-domain method for reliable per-field confidence in high-stakes LLM document extraction pipelines. This could meaningfully improve human-in-the-loop workflows by reducing silent errors in financial and compliance applications. The reported gains over logprob baselines and the zero-shot transfer with recalibration are notable strengths if reproducible; the multi-signal fusion approach, if shown to leverage the claimed structural asymmetry, would advance confidence estimation beyond single-signal methods.

major comments (2)

[Abstract and experimental results] Abstract and experimental results: the claim that Hunter-Mapper disagreement is 'independently informative' and drives the reported 0.928 AUC and 70% risk reduction lacks supporting evidence such as ablation studies (removing the disagreement feature), feature-importance scores, or correlation analysis with the other fused signals (LLM uncertainty, OCR, layout). Without this, the performance could be explained by the non-asymmetry signals alone, leaving the core methodological contribution unverified.
[Methods and results sections] Methods and results sections: no details are provided on data splits, classifier training procedure (e.g., which model, hyperparameters, cross-validation), exact feature definitions for the disagreement metric, or post-hoc selection of thresholds. These omissions make the AUC, risk-reduction, and coverage numbers impossible to reproduce or assess for overfitting, directly undermining the central empirical claims.

minor comments (2)

[Abstract] The abstract mentions 'lightweight recalibration' on CORD but does not specify the exact procedure or whether it was applied only to the test set; clarify to avoid appearance of data leakage.
[Methods] Notation for the fused classifier (e.g., how disagreement is quantified as a numeric feature) is not introduced in the provided abstract; ensure consistent definitions appear early in the methods.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback highlighting the need for stronger empirical support of the core contribution and improved methodological transparency. We will revise the manuscript to incorporate the requested evidence and details.

read point-by-point responses

Referee: [Abstract and experimental results] Abstract and experimental results: the claim that Hunter-Mapper disagreement is 'independently informative' and drives the reported 0.928 AUC and 70% risk reduction lacks supporting evidence such as ablation studies (removing the disagreement feature), feature-importance scores, or correlation analysis with the other fused signals (LLM uncertainty, OCR, layout). Without this, the performance could be explained by the non-asymmetry signals alone, leaving the core methodological contribution unverified.

Authors: We agree that the manuscript currently lacks explicit ablation studies, feature-importance analysis, or correlation metrics isolating the Hunter-Mapper disagreement signal. While the paper provides theoretical motivation based on the structural asymmetry in failure modes (Hunter under schema pressure vs. Mapper holistic scan), this does not substitute for empirical verification. In revision we will add: (1) an ablation removing the disagreement feature and reporting the AUC and risk-reduction delta; (2) feature-importance scores (e.g., from the trained classifier); and (3) pairwise correlation analysis among all signals. These additions will directly test whether the disagreement signal contributes independently. revision: yes
Referee: [Methods and results sections] Methods and results sections: no details are provided on data splits, classifier training procedure (e.g., which model, hyperparameters, cross-validation), exact feature definitions for the disagreement metric, or post-hoc selection of thresholds. These omissions make the AUC, risk-reduction, and coverage numbers impossible to reproduce or assess for overfitting, directly undermining the central empirical claims.

Authors: We acknowledge the absence of these implementation details in the current version, which prevents full reproducibility. In the revised manuscript we will expand the Methods section to specify: the exact train/validation/test splits on DocILE (and how CORD was used for zero-shot evaluation), the classifier architecture and hyperparameters together with cross-validation procedure, precise mathematical definitions of every feature (including the disagreement metric), and the criterion used for threshold selection. We will also release the feature-extraction code and trained model weights to support verification. revision: yes

Circularity Check

0 steps flagged

No circularity; empirical results on external benchmarks

full rationale

The paper presents ExtractConf as a fused classifier using disagreement between Hunter and Mapper calls plus other signals, then reports ROC AUC, risk reduction, and calibration metrics on the public DocILE and CORD datasets. No equations, fitted parameters, or self-citations are shown to define the target quantities by construction; the central claim rests on external evaluation rather than internal redefinition or tautological prediction. The absence of ablations on signal independence is an evidence gap, not a circularity reduction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Abstract-only view yields a minimal ledger; the central domain assumption is that dual asymmetric LLM readings supply non-redundant information about trustworthiness. No explicit free parameters or invented entities are described.

axioms (1)

domain assumption Disagreement between field-guided Hunter and document-guided Mapper calls is independently informative of extraction quality
This asymmetry is presented as the source of the new signal that enables the classifier.

pith-pipeline@v0.9.1-grok · 5827 in / 1313 out tokens · 23912 ms · 2026-06-25T23:50:21.601972+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

29 extracted references · 11 canonical work pages · 2 internal anchors

[1]

Perot et al.LMDX: Language Model-based Document Information Extraction and Localization

V . Perot et al.LMDX: Language Model-based Document Information Extraction and Localization. ACL Findings, 2024; arXiv:2309.10952

work page arXiv 2024
[2]

Šimsa et al.DocILE Benchmark for Document Information Localization and Extraction.ICDAR, 2023; arXiv:2302.05658

S. Šimsa et al.DocILE Benchmark for Document Information Localization and Extraction.ICDAR, 2023; arXiv:2302.05658

work page arXiv 2023
[3]

Reflect then Learn: Active Prompting for Information Extraction Guided by Introspective Confusion

(Authors).Reflect then Learn: Active Prompting for IE Guided by Introspective Confusion. arXiv:2508.10036, 2025

work page internal anchor Pith review Pith/arXiv arXiv 2025
[4]

Geifman and R

Y . Geifman and R. El-Yaniv.Selective Prediction in Deep Neural Networks.NeurIPS, 2017

2017
[5]

Corbière et al.Addressing Failure Prediction by Learning Model Confidence.NeurIPS, 2019

C. Corbière et al.Addressing Failure Prediction by Learning Model Confidence.NeurIPS, 2019

2019
[6]

Geifman and R

Y . Geifman and R. El-Yaniv.SelectiveNet: A Deep Neural Network with an Integrated Reject Option. ICML, 2019

2019
[7]

H. S. Seung, M. Opper, and H. Sompolinsky.Query by Committee.COLT, 1992

1992
[8]

Blum and T

A. Blum and T. Mitchell.Combining Labeled and Unlabeled Data with Co-training.COLT, 1998

1998
[9]

Lakshminarayanan, A

B. Lakshminarayanan, A. Pritzel, and C. Blundell.Simple and Scalable Predictive Uncertainty Estima- tion using Deep Ensembles.NeurIPS, 2017

2017
[10]

Li et al.Evaluating Object Hallucination in Large Vision-Language Models.EMNLP, 2023

Y . Li et al.Evaluating Object Hallucination in Large Vision-Language Models.EMNLP, 2023

2023
[11]

Min et al.FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation.EMNLP, 2023

S. Min et al.FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation.EMNLP, 2023

2023
[12]

Gero et al.Self-Verification Improves Few-Shot Clinical Information Extraction.ACL Workshop on Clinical NLP, 2023

K. Gero et al.Self-Verification Improves Few-Shot Clinical Information Extraction.ACL Workshop on Clinical NLP, 2023

2023
[13]

H. T. Nguyen et al.HYCEDIS: HYbrid Confidence Engine for Deep Document Intelligence System. KDD Workshop on Document Intelligence, 2021; arXiv:2206.02628

work page arXiv 2021
[14]

Rombach and M

M. Rombach and M. Mehdiyev.Beyond Accuracy: Understanding Model Confidence in Key Informa- tion Extraction with Conformal Prediction.IJDAR, 2026; doi:10.1007/s10032-026-00572-y

work page doi:10.1007/s10032-026-00572-y 2026
[15]

Hemmer et al.Confidence-Aware Document OCR Error Detection.DAS/ICDAR, 2024; arXiv:2409.04117

A. Hemmer et al.Confidence-Aware Document OCR Error Detection.DAS/ICDAR, 2024; arXiv:2409.04117

work page arXiv 2024
[16]

C. L. Willis et al.Automated Information Extraction from Plant Specimen Labels using OCR and Large Language Models.PMC12835874, 2025

2025
[17]

J. H. Chen and J. Mueller.Quantifying Uncertainty in Answers from any Language Model and Enhanc- ing their Trustworthiness.ACL, 2024; arXiv:2308.16175

work page arXiv 2024
[18]

Goh and J

J. Goh and J. Mueller.Real-Time Trustworthiness Scoring for LLM Structured Outputs and Data Extraction.Cleanlab, 2026; arXiv:2603.18014

work page arXiv 2026
[19]

Farquhar et al.Detecting Hallucinations in Large Language Models using Semantic Entropy

S. Farquhar et al.Detecting Hallucinations in Large Language Models using Semantic Entropy. Nature, 630, 2024

2024
[20]

Language Models (Mostly) Know What They Know

S. Kadavath et al.Language Models (Mostly) Know What They Know.arXiv:2207.05221, 2022

work page internal anchor Pith review Pith/arXiv arXiv 2022
[21]

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback

K. Tian et al.Just Ask for Calibration.EMNLP, 2023; arXiv:2305.14975

work page Pith review arXiv 2023
[22]

Jagannatha and H

A. Jagannatha and H. Yu.Calibrating Structured Output Predictors for NLP .ACL, 2020; arXiv:2004.04361

work page arXiv 2020
[23]

Nixon et al.Measuring Calibration in Deep Learning.CVPR Workshops, 2019

J. Nixon et al.Measuring Calibration in Deep Learning.CVPR Workshops, 2019

2019
[24]

Kull et al.Beyond Temperature Scaling.NeurIPS, 2019

M. Kull et al.Beyond Temperature Scaling.NeurIPS, 2019

2019
[25]

Park et al.CORD: A Consolidated Receipt Dataset for Post-OCR Parsing.NeurIPS Workshop, 2019

S. Park et al.CORD: A Consolidated Receipt Dataset for Post-OCR Parsing.NeurIPS Workshop, 2019. 10

2019
[26]

Huang et al.ICDAR 2019 Competition on Scanned Receipt OCR and Information Extraction.ICDAR, 2019

Z. Huang et al.ICDAR 2019 Competition on Scanned Receipt OCR and Information Extraction.ICDAR, 2019

2019
[27]

Prokhorenkova et al.CatBoost: Unbiased Boosting with Categorical Features.NeurIPS, 2018

L. Prokhorenkova et al.CatBoost: Unbiased Boosting with Categorical Features.NeurIPS, 2018

2018
[28]

Wang et al.Self-Consistency Improves Chain of Thought Reasoning.ICLR, 2023

X. Wang et al.Self-Consistency Improves Chain of Thought Reasoning.ICLR, 2023

2023
[29]

Feature names correspond to column headers in the pipeline output and to the feature importance plot in Figure 4a

Feature Reference Table Table 3 provides a complete reference for all features used in EXTRACTCONF, grouped by signal category. Feature names correspond to column headers in the pipeline output and to the feature importance plot in Figure 4a. Table 3:Complete feature reference for EXTRACTCONF. H =Huntercall, M =Mappercall. Feature Name Group Description G...

[1] [1]

Perot et al.LMDX: Language Model-based Document Information Extraction and Localization

V . Perot et al.LMDX: Language Model-based Document Information Extraction and Localization. ACL Findings, 2024; arXiv:2309.10952

work page arXiv 2024

[2] [2]

Šimsa et al.DocILE Benchmark for Document Information Localization and Extraction.ICDAR, 2023; arXiv:2302.05658

S. Šimsa et al.DocILE Benchmark for Document Information Localization and Extraction.ICDAR, 2023; arXiv:2302.05658

work page arXiv 2023

[3] [3]

Reflect then Learn: Active Prompting for Information Extraction Guided by Introspective Confusion

(Authors).Reflect then Learn: Active Prompting for IE Guided by Introspective Confusion. arXiv:2508.10036, 2025

work page internal anchor Pith review Pith/arXiv arXiv 2025

[4] [4]

Geifman and R

Y . Geifman and R. El-Yaniv.Selective Prediction in Deep Neural Networks.NeurIPS, 2017

2017

[5] [5]

Corbière et al.Addressing Failure Prediction by Learning Model Confidence.NeurIPS, 2019

C. Corbière et al.Addressing Failure Prediction by Learning Model Confidence.NeurIPS, 2019

2019

[6] [6]

Geifman and R

Y . Geifman and R. El-Yaniv.SelectiveNet: A Deep Neural Network with an Integrated Reject Option. ICML, 2019

2019

[7] [7]

H. S. Seung, M. Opper, and H. Sompolinsky.Query by Committee.COLT, 1992

1992

[8] [8]

Blum and T

A. Blum and T. Mitchell.Combining Labeled and Unlabeled Data with Co-training.COLT, 1998

1998

[9] [9]

Lakshminarayanan, A

B. Lakshminarayanan, A. Pritzel, and C. Blundell.Simple and Scalable Predictive Uncertainty Estima- tion using Deep Ensembles.NeurIPS, 2017

2017

[10] [10]

Li et al.Evaluating Object Hallucination in Large Vision-Language Models.EMNLP, 2023

Y . Li et al.Evaluating Object Hallucination in Large Vision-Language Models.EMNLP, 2023

2023

[11] [11]

Min et al.FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation.EMNLP, 2023

S. Min et al.FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation.EMNLP, 2023

2023

[12] [12]

Gero et al.Self-Verification Improves Few-Shot Clinical Information Extraction.ACL Workshop on Clinical NLP, 2023

K. Gero et al.Self-Verification Improves Few-Shot Clinical Information Extraction.ACL Workshop on Clinical NLP, 2023

2023

[13] [13]

H. T. Nguyen et al.HYCEDIS: HYbrid Confidence Engine for Deep Document Intelligence System. KDD Workshop on Document Intelligence, 2021; arXiv:2206.02628

work page arXiv 2021

[14] [14]

Rombach and M

M. Rombach and M. Mehdiyev.Beyond Accuracy: Understanding Model Confidence in Key Informa- tion Extraction with Conformal Prediction.IJDAR, 2026; doi:10.1007/s10032-026-00572-y

work page doi:10.1007/s10032-026-00572-y 2026

[15] [15]

Hemmer et al.Confidence-Aware Document OCR Error Detection.DAS/ICDAR, 2024; arXiv:2409.04117

A. Hemmer et al.Confidence-Aware Document OCR Error Detection.DAS/ICDAR, 2024; arXiv:2409.04117

work page arXiv 2024

[16] [16]

C. L. Willis et al.Automated Information Extraction from Plant Specimen Labels using OCR and Large Language Models.PMC12835874, 2025

2025

[17] [17]

J. H. Chen and J. Mueller.Quantifying Uncertainty in Answers from any Language Model and Enhanc- ing their Trustworthiness.ACL, 2024; arXiv:2308.16175

work page arXiv 2024

[18] [18]

Goh and J

J. Goh and J. Mueller.Real-Time Trustworthiness Scoring for LLM Structured Outputs and Data Extraction.Cleanlab, 2026; arXiv:2603.18014

work page arXiv 2026

[19] [19]

Farquhar et al.Detecting Hallucinations in Large Language Models using Semantic Entropy

S. Farquhar et al.Detecting Hallucinations in Large Language Models using Semantic Entropy. Nature, 630, 2024

2024

[20] [20]

Language Models (Mostly) Know What They Know

S. Kadavath et al.Language Models (Mostly) Know What They Know.arXiv:2207.05221, 2022

work page internal anchor Pith review Pith/arXiv arXiv 2022

[21] [21]

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback

K. Tian et al.Just Ask for Calibration.EMNLP, 2023; arXiv:2305.14975

work page Pith review arXiv 2023

[22] [22]

Jagannatha and H

A. Jagannatha and H. Yu.Calibrating Structured Output Predictors for NLP .ACL, 2020; arXiv:2004.04361

work page arXiv 2020

[23] [23]

Nixon et al.Measuring Calibration in Deep Learning.CVPR Workshops, 2019

J. Nixon et al.Measuring Calibration in Deep Learning.CVPR Workshops, 2019

2019

[24] [24]

Kull et al.Beyond Temperature Scaling.NeurIPS, 2019

M. Kull et al.Beyond Temperature Scaling.NeurIPS, 2019

2019

[25] [25]

Park et al.CORD: A Consolidated Receipt Dataset for Post-OCR Parsing.NeurIPS Workshop, 2019

S. Park et al.CORD: A Consolidated Receipt Dataset for Post-OCR Parsing.NeurIPS Workshop, 2019. 10

2019

[26] [26]

Huang et al.ICDAR 2019 Competition on Scanned Receipt OCR and Information Extraction.ICDAR, 2019

Z. Huang et al.ICDAR 2019 Competition on Scanned Receipt OCR and Information Extraction.ICDAR, 2019

2019

[27] [27]

Prokhorenkova et al.CatBoost: Unbiased Boosting with Categorical Features.NeurIPS, 2018

L. Prokhorenkova et al.CatBoost: Unbiased Boosting with Categorical Features.NeurIPS, 2018

2018

[28] [28]

Wang et al.Self-Consistency Improves Chain of Thought Reasoning.ICLR, 2023

X. Wang et al.Self-Consistency Improves Chain of Thought Reasoning.ICLR, 2023

2023

[29] [29]

Feature names correspond to column headers in the pipeline output and to the feature importance plot in Figure 4a

Feature Reference Table Table 3 provides a complete reference for all features used in EXTRACTCONF, grouped by signal category. Feature names correspond to column headers in the pipeline output and to the feature importance plot in Figure 4a. Table 3:Complete feature reference for EXTRACTCONF. H =Huntercall, M =Mappercall. Feature Name Group Description G...