{"id":"794b6e98-e9ba-4e4a-a3e9-0ea89d4794cc","arxiv_id":"2608.01792","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Vision-language models vary widely in how trustworthy their confidence scores are on document extraction, with stronger models and OCR-plus-image input helping most, as measured on the new ConfBench benchmark.","lead":"The paper introduces ConfBench, a benchmark that applies 20 controlled degradations to real scanned invoices to test how well AI document readers know when they are wrong. It evaluates seven AI models and introduces a review-budget metric, ECARB, that translates confidence quality into savings in human review effort.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Binary-correctness acceptance threshold is undisclosed; every calibration metric and ranking in Table 3 depends on it.","rationale":"The paper's central claim is an empirical one about relative confidence quality across models, modalities, and strategies, and that claim is operationalized entirely through binary correctness labels derived from an undisclosed per-field acceptance threshold. This is exactly the reader's weakest assumption, and it is indeed load-bearing: all four calibration metrics depend on the binary labels, and the paper gives no threshold values, no per-field breakdown, and no sensitivity analysis. The concern is concrete and addressable, not a rejection of the benchmark's value. The authors are otherwise careful: they release the data, use an open-source evaluation library, state limitations explicitly, and avoid over-generalizing from two logprob-capable models. Those strengths support a conditional acceptance, not a rejection. Since the reader already assigned CONDITIONAL for essentially this reason, my read does not move the verdict.","tokens_in":22319,"tokens_out":2368,"duration_ms":28316,"concrete_test":"Using the released ConfBench images and the authors' evaluation code, recompute Table 3 under a sweep of per-field acceptance thresholds (e.g., similarity >= 0.5, >= 0.8, >= 1.0, and the STICKLER default) for the OCR+Image verbalized condition, and report AUROC, ECE, and ECARB@30% for every model. Then compare model rankings and the OCR+Image vs OCR-only ordering across the sweep. If the rankings and the claimed modality/capability orderings remain identical, the concern is resolved; if any rank change occurs, the central findings are threshold-dependent and must be re-reported with explicit threshold values and a robustness analysis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix A.5 states that for calibration, each continuous similarity score is converted to a binary correctness label using a per-field acceptance threshold, but the threshold values are never reported. AUROC, ECE, Brier, and ECARB are all computed from these binary labels, so the threshold choice propagates into every headline number: it changes which entities count as errors, the error count E in ECARB, per-bin accuracies in ECE, and the correct/incorrect partition underlying AUROC. Since the paper's central claims are that confidence quality varies widely across models (AUROC 0.58–0.84), is driven by capability and modality, and that OCR+Image is strongest, a threshold shift that reclassifies even a few percent of entities could alter model rankings, the modality ordering, or the ECARB gains. The paper does not state whether the threshold is the same across fields, what values were used for string-similarity, numeric-tolerance, and date comparators, or whether results were checked for sensitivity to this choice. This is a concrete specification gap, not a philosophical objection: the benchmark and code are released, so the missing numbers are recoverable, but the empirical findings as printed are not fully pinned down.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ConfBench, a benchmark for evaluating confidence calibration of vision-language models (VLMs) on key information extraction (KIE) from documents. ConfBench is built by applying 20 controlled degradation pipelines to 75 verified FCC invoices, producing 1,346 document variants and 70K+ entity-level evaluations across the accuracy spectrum. The authors evaluate seven VLMs with verbalized confidence (1S-TopK) and two open-weight models with three log-probability aggregation methods, across OCR-only, image-only, and OCR+image input modalities. They also propose ECARB, a review-budget metric that measures how much confidence-guided review improves over random sampling. The main findings are that OCR+image is the strongest modality for both accuracy and confidence quality; model capability, rather than parameter count, dominates confidence quality; calibration quality varies widely across models; and first-token log-probability aggregation outperforms mean-token and margin. The benchmark and code are publicly released.","tokens_in":22592,"tokens_out":4006,"duration_ms":44855,"significance":"If the empirical findings hold, ConfBench addresses a genuine gap: existing document benchmarks concentrate on clean, high-accuracy samples and are not suitable for calibration assessment. The controlled degradation strategy, the explicit scoping of claims, and the public release of the benchmark, augmentation configurations, and code are notable strengths. ECARB is a practical contribution that connects discriminative confidence quality to human-in-the-loop review economics. The paper is transparent about many limitations, including the small source corpus, the two-model basis for the verbalized-vs-logprob comparison, and the lack of post-hoc calibration benchmarks. However, the central quantitative claims currently rest on an undisclosed binary-correctness threshold and on single-point metric estimates without uncertainty quantification, which limits the reproducibility and evidentiary strength of the headline rankings.","major_comments":[{"comment":"The conversion of continuous per-field similarity scores into binary correctness labels uses a 'per-field acceptance threshold' (Appendix A.5), but the threshold values are never stated. AUROC, ECE, Brier, and ECARB are all computed from these binary labels: E in Eq. (1) counts entities with acc_i=0, ECE bins use binary accuracy, and AUROC partitions correct from incorrect. The reported AUROC range 0.58–0.84, ECE 0.05–0.36, and all model/modality orderings in Table 3 can shift if the string-similarity cutoff, numeric tolerance, or date equivalence rule is changed. Please report the exact per-field thresholds used by STICKLER, state whether the same threshold is applied across fields and runs, and provide a sensitivity analysis (e.g., thresholds at several similarity levels) showing that the model rankings and ECARB conclusions are stable. Without this, the central empirical findings are","section":"Appendix A.5, §4.2"},{"comment":"The main results table reports a single value per metric per configuration, with no error bars, confidence intervals, or significance tests. The paper elsewhere uses nine runs to average accuracy drop (Table 1, Figure 7), yet the confidence metrics in Table 3 appear to come from a single evaluation pass. Claims such as 'OCR+Image is uniformly the strongest modality' and 'within the Claude family confidence quality scales monotonically' are ordinal claims that may be sensitive to run-to-run variability, especially for small gaps (e.g., Haiku 0.74 vs Qwen 3.6-27B 0.72 AUROC; Sonnet 0.77 vs Opus 0.84 AUROC). The authors appropriately caution against reading small gaps as strict orderings (Appendix A.6), but the headline findings need a quantitative measure of uncertainty. Please report per-run variability or bootstrap confidence intervals for the key metric values and rankings.","section":"Table 3, §5.1–5.3"},{"comment":"The claim that 'because ECARB depends on the ranking confidence induces rather than on absolute confidence values, model rankings are largely stable across budgets' is not supported by the reported evidence. Only b=30% is reported; no ECARB curves or values at other budgets are shown. Since the ECARB ceiling min(1/b, n/E) depends on b and the composition of the low-confidence slice changes with b, ranking stability is an empirical assertion, not a consequence of the definition. Please report ECARB at multiple review budgets (e.g., 10%, 20%, 50%) or provide a formal argument for budget invariance.","section":"§4.2, Eq. (1)"}],"minor_comments":[{"comment":"Minor typographical and consistency issues: 'CONFBENCH' appears in uppercase in Appendix A.5; Table 2's '#p' column is ambiguous for ConfBench (1,346 documents vs. 6.7K pages). Please clarify the unit.","section":"Appendix A.5"},{"comment":"ECARB is undefined when E=0 (no errors) or when k=0. State the convention for these degenerate cases, since near-perfect extraction is a plausible regime.","section":"Eq. (1)"},{"comment":"The discussion of coupling between extraction and confidence estimation would be clearer if it explicitly stated that the proposed decoupled two-stage verbalized variant is not evaluated in this paper; as written, a reader may mistakenly infer it was part of the experimental design.","section":"§5.2 / Appendix A.6"},{"comment":"Several references are incomplete or inconsistent in formatting (e.g., Shrivastava et al. 2024 has no venue; multiple arXiv entries lack arXiv IDs). Please standardize.","section":"Reference list"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ConfBench is a real contribution: the first benchmark built specifically to test calibration for document KIE, using 20 controlled degradation pipelines on a verified invoice corpus so that the low-accuracy region is populated. The design logic is clear, and the release includes the data, ground truth, pipeline configs, and metadata, which makes it reproducible and extensible. ECARB is a sensible operational metric — it translates rank-based discrimination into review-budget terms, and the worked example in the appendix does what a good metric explanation should do.\n\nThe stress-test note is right: the binary-correctness threshold is the load-bearing weak point. Appendix A.5 says each continuous similarity score is converted to a binary label using a per-field acceptance threshold, but the actual threshold values never appear. AUROC, ECE, Brier, and ECARB all depend on which entities count as correct, so a reclassification of even a few percent of entities could shift model rankings or modality ordering. This is a specification gap, not a philosophical objection — the code is released, so the numbers are recoverable — but as printed, the central empirical findings are not fully pinned down.\n\nThe paper also has the usual small-sample issues: Table 3 has no error bars, the verbalized-vs-logprob comparison rests on two open-weight models, and the abstract's 'consistently outperforms' about first-token aggregation is stronger than the data — two models is a pattern, not a law. The authors scrupulously acknowledge most of this in the limitations, which makes the threshold omission stand out more.\n\nCredit where due: the paper is unusually honest about scope — it explicitly says the MoE observation is not an architecture claim, that cross-model token budgets differ, and that the corpus is deliberately small in exchange for verified annotations. That candor is rare.\n\nWho should read this: anyone building or evaluating VLM extraction pipelines with confidence-based routing. The benchmark and ECARB are worth adopting now; the comparative findings need re-verification with disclosed thresholds, error bars, and the exact prompts. This deserves a serious referee — send it out, with a request for the threshold values and sensitivity analysis.","headline":"ConfBench is a genuinely useful benchmark and ECARB is a sensible metric, but the undisclosed binary-correctness threshold means the headline numbers need rechecking before you trust the rankings.","tokens_in":23106,"tokens_out":2962,"would_cite":true,"duration_ms":31868,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ConfBench measures whether a vision-language model's confidence scores are trustworthy enough to route document extractions to automation or to a human reviewer.","keywords":["confidence calibration","vision-language models","key information extraction","document degradation","calibration benchmark","ECARB","human-in-the-loop review","intelligent document processing"],"falsifier":"Recompute the paper's metrics on the released ConfBench images and ground truth using several explicit acceptance thresholds (for example, exact match, 90% similarity, and 70% similarity) to create the correct/incorrect labels, and check whether the reported orderings survive—Opus above Sonnet above Haiku, verbalized above log-probability for Qwen but below it for Gemma. If the rankings change with the threshold, the headline confidence-quality results are artifacts of an unpublished choice; if they hold, the claims are robust to it.","tokens_in":22237,"feed_emoji":"📊","tokens_out":10236,"duration_ms":90606,"temperature":0.7,"pith_summary":"Systems that process invoices and contracts automatically rely on the model's confidence score to decide which extractions are safe to automate and which need a human reviewer, so a confidence score that cannot be trusted silently turns into errors. The paper argues that no existing document benchmark could test that trust, because benchmarks built on clean scans produce almost all-correct predictions, leaving the low-accuracy region where confidence actually matters too sparse to measure. It introduces ConfBench, built by running 20 controlled degradation pipelines (blur, skew, stains, bleed-through, JPEG noise, and more) over 75 manually verified FCC invoices to create 1,346 degraded variants whose extraction accuracy spans the full spectrum, with over 70,000 entity-level evaluations. On this substrate the paper reports the first systematic comparison of seven vision-language models, two confidence-estimation families (verbalized self-report and token log-probability), and three input modalities: confidence quality varies from near-perfect (AUROC 0.84, ECE 0.05) to severely overconfident (AUROC 0.58, ECE 0.31), is driven mainly by model capability rather than parameter count, and is uniformly best when the model receives both OCR text and the image. The paper's practical claim is that a deployment team can now measure, before rollout, whether a model's confidence is trustworthy enough to route work to automation, and can quantify the human-review savings with the new ECARB metric.","feed_headline":"Benchmark reveals which VLMs overstate extraction confidence","feed_subtitle":"ConfBench grades seven models on 1,346 degraded invoices, from near-perfect calibration to severe overconfidence.","key_machinery":"The argument is carried by three objects. ConfBench: 20 controlled degradation pipelines built with an open-source document-augmentation library—ink-level (bleed-through, dithering, low ink), paper-level (stains, moiré patterns, watermarks), and capture-level (JPEG compression, skew, shadows) artifacts—applied to 75 manually verified FCC invoices to yield 1,346 variants whose entity-level extraction accuracy spans the full spectrum, the regime where calibration can actually be distinguished from luck. The confidence estimators: 'verbalized' 1S-TopK, where the model emits its top four candidate values with probabilities in a single call, and log-probability, where confidence is read off the t","core_discovery":"The central claim is that confidence calibration for document extraction is a measurable, deployment-relevant property that varies widely and predictably. Feeding a model both the OCR text and the document image is uniformly the strongest input configuration for confidence quality, with the largest gap over image-only appearing in smaller models. Within the Claude family, confidence quality scales monotonically with capability, with Opus 4.6 attaining AUROC 0.84 and ECE 0.05, close to ideal calibration without any post-hoc correction, while Haiku 4.5 falls to 0.74 and 0.17. Across families parameter count is a poor predictor, since a 27-billion-parameter model outranks a 235-billion-paramete","pith_inferences":["The published rankings should be treated as provisional until the per-field acceptance thresholds that convert similarity scores into correct/incorrect labels are released; because the benchmark's images and ground truth are public, any reader can re-derive the metrics once those thresholds are known.","The OCR+Image result suggests a cheap production rule worth testing: add per-token OCR confidence to the prompt rather than relying on the image alone, since the textual signal appears to carry most of the discriminative information, especially for weaker models.","The paper's two-point evidence on which confidence family wins implies the honest deployment practice is per-model evaluation, and also points to a natural next experiment the paper names: ensembling verbalized and log-probability scores, which draw on different information.","The tiered degradation structure makes ConfBench a ready-made testbed for calibration transfer—whether a calibrator fitted on mild degradation holds on severe degradation—a question the paper lists as future work but that the released data can answer immediately."],"forward_implications":["Before rollout, a deployment team can run ConfBench on its candidate model and, from AUROC plus ECARB at its own review budget, decide whether confidence-gated automation is safe or whether humans must review.","OCR+Image should be the default input for confidence-sensitive extraction: it beats OCR-only and image-only on both accuracy and confidence quality, most clearly for smaller models.","Calibration is situational, not a blanket requirement: per-model post-hoc rescaling fixes absolute confidence values for threshold-based routing but leaves ranking-based metrics (AUROC, ECARB) unchanged, so teams should rescale only where an absolute threshold gates automation.","For models that expose token probabilities, first-token log-probability is the preferred confidence aggregation over mean-token and margin; for closed models, verbalized confidence is usable and, for the strongest Claude model, nearly well-calibrated.","Model capability, not parameter count, predicts confidence quality: within the Claude family confidence scales with tier, while across families a much smaller model can outperform a much larger one."],"supporting_citations":[{"why":"Supplies the 75 manually corrected, re-annotated FCC invoice documents whose ground-truth annotations are preserved across all ConfBench variants; the benchmark's accuracy and calibration labels all rest on this verified corpus.","marker":"Amazon AGI Team, 2024"},{"why":"Source RealKIE benchmark whose FCC subset ConfBench builds on, and the contrast case: its annotation quality varies, which motivates the verified subset as calibration ground truth.","marker":"Townsend et al., 2024"},{"why":"Provides the augmentation framework and eight pre-configured archetype pipelines that, with twelve custom pipelines, generate the 1,346 degraded variants spanning the accuracy spectrum.","marker":"Groleau et al., 2023"},{"why":"The TopK verbalized protocol (top-four candidates with probabilities, no chain-of-thought) that the paper's 1S-TopK confidence method is built on.","marker":"Tian et al., 2023"},{"why":"Defines the calibration problem and the overconfidence failure mode that motivates auditing confidence scores before trusting them for routing.","marker":"Guo et al., 2017"},{"why":"Defines the Brier score, one of the four calibration metrics used to evaluate every configuration.","marker":"Brier, 1950"},{"why":"Supplies the AUROC convention and adaptive quantile binning (M=5) used for the ECE computation.","marker":"Shrivastava et al., 2024"}],"fun_headline_variants":["Confidence in VLMs: bigger models aren't better calibrated","OCR+image input sharpens VLM confidence for document extraction","ConfBench: calibration varies wildly across 7 VLMs","Which VLM overstates extraction confidence? New benchmark says","For trustworthy document AI, OCR+image beats image alone"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"Every calibration number in the paper depends on turning each entity's continuous similarity score into a binary right/wrong label with a per-field acceptance threshold that is never stated; different thresholds would change the AUROC, ECE, Brier, and ECARB values and could reorder the models.","fun_headline_variants_meta":{"raw":{"variants":["Confidence in VLMs: bigger models aren't better calibrated","OCR+image input sharpens VLM confidence for document extraction","ConfBench: calibration varies wildly across 7 VLMs","Which VLM overstates extraction confidence? New benchmark says","For trustworthy document AI, OCR+image beats image alone"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000678,"raw_usage":{"total_tokens":2948,"prompt_tokens":801,"completion_tokens":2147,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":2074}},"tokens_in":545,"tokens_out":2147,"duration_ms":19364,"temperature":1.0,"reasoning_tokens":2074,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T20:38:26.198693+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the paper's metrics on the released ConfBench images and ground truth using several explicit acceptance thresholds (for example, exact match, 90% similarity, and 70% similarity) to create the correct/incorrect labels, and check whether the reported orderings survive—Opus above Sonnet above Haiku, verbalized above log-probability for Qwen but below it for Gemma. If the rankings change with the threshold, the headline confidence-quality results are artifacts of an unpublished choice; if they hold, the claims are robust to it.","supporting_citations":[],"review_version":1}