{"id":"ba9085e0-2e4b-4215-b5c7-e20de2b62a84","arxiv_id":"2505.23854","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Across 80 LLMs on MMLU-Pro, linguistic verbal uncertainty judged by another LLM gives better calibration and error ranking on average than token-probability or numeric self-reported uncertainty, with exceptions.","lead":"The paper tests three ways to estimate how uncertain an LLM is, on 80 different models and the challenging MMLU-Pro benchmark. It finds that asking a second model to judge the hedging words in an answer, called linguistic verbal uncertainty, gives the most reliable uncertainty scores, which matters for deciding when to trust or abstain.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LVU's headline advantage may be an artifact of the judge scoring instruction violations as uncertainty; Section 3.1 admits this inflates small-model AUROC, so 'consistently outperforms' is unsupported.","rationale":"The reader's weakest assumption was the general reliability of the LVU judge, validated only on 20 non-verifiable sentences. I agree that judge validity is load-bearing, but I identify a more specific and more immediately testable problem: the paper's own Section 3.1 admits that the judge assigns high LVU uncertainty to instruction-violating outputs, and that this produces strong discriminative performance for small models for reasons unrelated to genuine uncertainty. Since the 80-model set includes many small and weak models, this admitted artifact can mechanically inflate the aggregate LVU averages that support the headline claim. The per-model results in Tables 3 and 4 reinforce the concern: numerous models, including several strong ones, have substantially lower ECE under TPU than under LVU, so 'consistently' is not supported by the paper's own data. This does not require rejecting the paper outright; a weaker 'on average' claim, with per-model win rates, exclusions reported per method, and the judge-compliance confound analyzed, could still be defensible. That is why I keep the reader's CONDITIONAL verdict rather than moving to REJECT: the empirical corpus is valuable and the central mechanism may survive the proposed test, but the current headline overstates what the evidence shows. The proposed concrete test would settle whether the LVU advantage is genuine or an artifact of the judge conflating uncertainty with format non-compliance.","tokens_in":21300,"tokens_out":8166,"duration_ms":73452,"concrete_test":"Recompute ECE and AUROC for LVU, NVU, and TPU on the subset of responses that are fully format-compliant for all three methods (valid answer letter, valid numeric confidence, and non-empty judge output), stratified by model size. Then compute pairwise per-model win rates and a paired bootstrap confidence interval for mean differences. If the LVU advantage over TPU/NVU drops below significance, or if the win rate among models at or above 30B parameters is below 0.5, the headline should be replaced with a scale-dependent finding and the Section 3.1 artifact is confirmed as a driver. Additionally, run the judge on 500 compliant versus 500 non-compliant MMLU-Pro responses matched by true correctness to quantify how much of LVU AUROC comes from format failures rather than linguistic hedging.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LVU 'consistently outperforms' TPU and NVU rests on the LLaMA-4-Maverick judge converting hedged text into a reliable uncertainty score. Appendix H validates only that 13 hedge words are ordinally ordered on 20 synthetic daily-life sentences; it never tests whether judge scores track correctness on MMLU-Pro responses. More importantly, Section 3.1 contains an admission that undercuts the headline: for smaller models, the judge 'tends to assign high LVU uncertainty to such non-compliant outputs, many of which are indeed incorrect. This results in strong discriminative uncertainty for smaller models—not because their uncertainty estimates are intrinsically better, but because instruction violations correlate strongly with failure.' Because the 80-model roster is dominated by small/weak models, this artifact can drive the average AUROC/ECE advantage claimed in Section 3.6. Published tables confirm the aggregate hides many per-model defeats: e.g., GPT-4.1 TPU ECE 0.074 vs LVU 0.166; Gemini-2.0-flash TPU 0.091 vs LVU 0.162; Qwen3-32B-think TPU 0.042 vs LVU 0.092. Thus the word 'consistently' is contradicted by the paper's own results even before questioning judge reliability.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a large empirical benchmark of uncertainty estimation for 80 LLMs on MMLU-Pro, comparing token probability-based uncertainty (TPU), numerical verbal uncertainty (NVU), and linguistic verbal uncertainty (LVU). The authors report that LVU consistently outperforms TPU and NVU in calibration (ECE) and selective classification (AUROC), and they examine the effects of model scale, post-training, reasoning mode, MoE architecture, quantization, and task type. The evaluation uses a shared prompt with chain-of-thought and confidence elicitation, LLaMA-4-Maverick as the LVU judge, and includes code and full per-model results.","tokens_in":21605,"tokens_out":9058,"duration_ms":80954,"significance":"If the central result were established, the paper would make a valuable practical contribution: LVU is a black-box, interpretable, single-pass uncertainty signal that does not require token-level access. The benchmark's breadth (80 models, multiple families and scales, per-subject analysis, code release) is a genuine resource for the field. However, the headline conclusion is not currently supported by the evidence: the paper's own tables contain many per-model cases where LVU loses to TPU, and the authors explicitly identify an instruction-violation artifact that inflates LVU for small models. The judge validation is too thin to carry the load. The data and framing are salvageable, but the central claims need re-analysis and careful qualification.","major_comments":[{"comment":"The abstract and Section 3.6 state that LVU 'consistently outperforms' TPU and NVU, but Tables 3 and 4 contain numerous per-model counterexamples. For example, GPT-4.1 has TPU ECE 0.074 versus LVU ECE 0.166; Gemini-2.0-flash has TPU ECE 0.091 versus LVU ECE 0.162; Qwen3-32b-think has TPU AUROC 0.791 versus LVU AUROC 0.726 and TPU ECE 0.042 versus LVU ECE 0.092; Qwen3-14b-think has TPU AUROC 0.809 versus LVU AUROC 0.752; and Phi-4-reasoning has TPU AUROC 0.640 versus LVU AUROC 0.601. The word 'consistently' is therefore not a faithful summary of the data. I recommend reporting the full distribution of per-model differences (e.g., win rates, paired bootstrap or signed-rank tests) and either removing the word 'consistently' or explicitly discussing which model classes drive the aggregate advantage.","section":"§3.6, Tables 3–4"},{"comment":"Section 3.1 acknowledges a serious confound: for smaller models, the judge 'tends to assign high LVU uncertainty to such non-compliant outputs, many of which are indeed incorrect,' producing strong discriminative uncertainty 'not because their uncertainty estimates are intrinsically better, but because instruction violations correlate strongly with failure.' Since the 80-model roster includes many small or weak models, this artifact can dominate the average AUROC/ECE advantage claimed in Section 3.6. The concern is compounded by the extraction protocol in Appendices F.3 and G: outputs with failed extractions are dropped, and the paper reports that on average 17% of responses are dropped; because NVU requires a parsable numeric confidence value while the LVU judge can score free text, the LVU evaluation set may contain non-compliant responses that NVU/TPU exclude. I ask the authors to recompute all three methods on a common subset of responses with successful extractions for every method, and to report LVU results stratified by instruction-compliance (e.g., compliant versus non-compliant outputs) so that the artifact is separated from genuine uncertainty signal.","section":"§3.1, §F.3, §G"},{"comment":"The validity of LVU as a general uncertainty measure rests on the LLaMA-4-Maverick judge, but the only validation (Appendix H) shows that the judge ordinally orders individual hedge words on 20 non-verifiable daily-life sentences. It does not show that judge scores track actual correctness of model responses on MMLU-Pro, nor that the mapping is stable across domains, model families, or output styles. The Conclusion explicitly says 'its general reliability across tasks, domains, and model families remains unclear,' which is in tension with the abstract's 'consistently outperforms' claim. The authors should validate the judge against correctness on the benchmark (e.g., calibration of judge scores with true accuracy per model), or provide a sensitivity analysis using an alternative LVU judge, before claiming LVU is a practical general-purpose uncertainty signal.","section":"§2.1, Appendix H, §4"}],"minor_comments":[{"comment":"The caption contains a typo: 'T oken Probability-Based Uncertainty' should be 'Token Probability-Based Uncertainty.'","section":"Figure 1 caption"},{"comment":"Reference [8] appears to be a BBC article on the US debt ceiling, but the text cites it for the story of a lawyer relying on ChatGPT for legal research; please replace it with the correct source.","section":"Reference [8]"},{"comment":"The statement that Qwen3-30B-A3B 'consistently outperforms' Qwen3-32B in both reasoning and non-reasoning tasks under LVU is contradicted by Table 4 for the non-reasoning pair, where LVU ECE is 0.243 for Qwen3-30b-a3b versus 0.218 for Qwen3-32b; please adjust the claim.","section":"§3.4, Table 4"},{"comment":"Section 3.5 describes Qwen3-235B-A22B (Reasoning) as having 'moderate accuracy (67%),' but Table 4 lists its accuracy as 0.880 for the think variant and 0.728 for the non-think variant; please reconcile the numbers.","section":"§3.5, Table 4"},{"comment":"The '/' entries for TPU values are not explained in the table captions; Appendix E notes that DeepSeek does not provide valid token probabilities, but a general legend for missing values should be added.","section":"Tables 3–4"},{"comment":"The stratified-sampling validation is performed with GPT-4.1-Mini only; the claim that 50% subsets are sufficient for all models should be presented as a single-model analysis rather than a general guarantee.","section":"Appendix G"},{"comment":"The claim that reasoning mode reduces the proportion of highly overconfident predictions by 'more than 20%' is not supported by a numerical table or statistical test; please provide the underlying numbers.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"This is a benchmark paper with useful data, but the central claim is currently overstated. A major revision that reanalyzes the aggregate comparisons, controls for instruction compliance, and strengthens judge validation could make it suitable. I would not recommend acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a genuinely broad benchmark—80 models, MMLU-Pro, three black-box single-pass uncertainty methods—and that breadth is the contribution. The headline that LVU \"consistently\" beats TPU and NVU is not supported by their own numbers; on per-model tables, TPU wins AUROC on several Qwen3-think and Phi-4-reasoning rows, and TPU ECE beats LVU on GPT-4.1, Gemini-2.0-flash, and Qwen3-32B-think. What survives is an on-average advantage, which is still a useful result for a first large-scale map.\n\nWhat the paper does well: the model coverage is real, the prompt design is shared and reproducible, they report extraction failures (17% dropped) instead of hiding them, and the auxiliary findings—post-training helps, reasoning mode reduces overconfidence, calibration and AUROC are weakly correlated—are clearly stated and backed by reliability diagrams and stratified-sampling controls. The distinction between calibration and ranking is worth taking seriously.\n\nSoft spots in proportion: (1) The \"consistently\" wording is refuted by their own Tables 3 and 4. That is a fixable overstatement, but it is the abstract's central claim, so it matters. (2) The LVU judge is validated on 20 synthetic sentences with 13 hedge words; that validates ordinal sensitivity to hedges, not whether judge scores track correctness on MMLU-Pro responses. The paper itself admits in the conclusion that judge reliability across tasks and domains is unclear. That does not kill the comparison, but it makes \"LVU is a practical tool\" conditional. (3) Section 3.1 contains the important admission: for small models, high LVU AUROC partly reflects instruction violations correlating with failure, not better uncertainty. Since the 80-model set is heavy on small weak models, the aggregate LVU advantage is partly artifact. The authors should re-analyze excluding non-compliant outputs and report results by model capability strata. (4) No confidence intervals around the averaged AUROC/ECE, so \"roughly 10% improvement\" is asserted without variance.\n\nWho this is for: people building selective answering and abstention systems who need a map of which uncertainty signal works where, and benchmark builders who want a baseline suite. It deserves a serious referee; the data collection is expensive and worth preserving. My recommendation: send to peer review, but require the authors to soften the headline, add per-model error bars or significance tests, validate the judge on the actual task, and separate the instruction-following artifact from genuine LVU signal.","headline":"A genuinely broad and useful benchmark whose 'LVU consistently wins' headline is overstated; the honest version is that LVU wins on average, with judge-validity and instruction-following artifacts still to be resolved.","tokens_in":22169,"tokens_out":1489,"would_cite":true,"duration_ms":15323,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that across 80 large language models, the most reliable uncertainty signal is the model's own hedging language—words like \"probably\" and \"might\"—outperforming token probabilities and numerical self-scores on calibration…","keywords":["uncertainty estimation","large language models","calibration","selective classification","linguistic verbal uncertainty","MMLU-Pro","expected calibration error","AUROC"],"falsifier":"Re-evaluate LVU on a sample of MMLU-Pro responses using a different judge, or human-annotated hedging scores, and check whether LVU still beats NVU and TPU by the reported margin; if the ranking flips or the advantage shrinks to noise, the central claim fails.","tokens_in":21104,"feed_emoji":"🤖","tokens_out":5893,"duration_ms":56544,"temperature":0.7,"pith_summary":"This paper tries to establish that the most reliable black-box way to know when a large language model is uncertain is to listen to the words it uses, not its token probabilities or its self-reported numbers. The authors evaluate 80 models from 0.6B to 671B parameters on the MMLU-Pro benchmark, comparing three single-pass methods: token probability-based uncertainty (TPU), numerical verbal uncertainty (NVU), and linguistic verbal uncertainty (LVU). Their central result is that LVU consistently beats the other two methods on both calibration (lower expected calibration error) and selective classification (higher AUROC), by roughly 10% over the second-best method, while also being more interpretable. A sympathetic reader would care because LVU works with any API-only model and produces a signal humans can understand, making it a plausible practical tool for deciding when to abstain, flag, or defer.","feed_headline":"Hedging words beat token odds for LLM uncertainty","feed_subtitle":"Across 80 models on MMLU-Pro, linguistic hedges give the best calibration and error ranking.","key_machinery":"The load-bearing mechanism is the LVU judge: a separate LLM, LLaMA-4-Maverick-17B-128E-Instruct, that reads a model's response and converts hedging language into a confidence score in [0,1], after the numeric confidence line has been stripped from the response. This is compared against TPU, defined as one minus the exponentiated average log-likelihood of the generated response, and NVU, the model's self-reported numeric score extracted by regular expression from a prompt that asks for confidence from 0 to 100. All three methods share the same single sampled response, prompted with a concise chain-of-thought template, and are scored by expected calibration error (ECE) with 10 bins and by AUROC for selective classification.","core_discovery":"On the paper's own terms, the discovery is that natural-language hedging is a stronger and more general uncertainty signal than either token-level likelihood or explicit numerical self-assessment. Across all 80 models and all subjects in MMLU-Pro, responses containing hedging expressions such as \"probably\", \"might\", and \"possibly\" are scored by a separate judge model and turn out to be better calibrated and better ranked than the two competing baselines. The paper also reports that model size, post-training, reasoning mode, and quantization each shift uncertainty quality, that reasoning tasks yield more reliable uncertainty estimates than knowledge-heavy tasks by over 10% AUROC, and that good calibration does not imply good error ranking. It takes these findings as evidence that accuracy is not a proxy for uncertainty reliability and that LVU deserves a central place in future uncertainty-aware LLM design.","pith_inferences":["If the LVU advantage comes partly from a judge that flags format violations as uncertainty, then models that follow instructions poorly may look uncertainty-aware for the wrong reasons; this is a testable failure mode the paper notes only for small models under LVU AUROC.","If hedging language genuinely tracks uncertainty, a natural next step the paper leaves implicit is to train models to hedge appropriately during post-training, rather than only detecting hedges after generation.","A different judge model—or human-annotated hedging scores—could reverse the reported LVU advantage, so the headline result should be read as contingent on the specific judge used.","Because the judge was validated on only 20 non-verifiable sentences, extending LVU to open-ended generation or new domains would require a standardized judge-evaluation protocol before deployment."],"forward_implications":["LVU can be used out-of-the-box with any API-only model, since it needs no logits, hidden states, or multiple generations.","Reasoning-enhanced models are better calibrated without necessarily being more accurate, so reasoning mode is a practical lever for improving uncertainty.","Accuracy should not be used as a proxy for reliability: high-accuracy models can be poorly calibrated, while moderate-accuracy models can have top-tier uncertainty estimates.","Calibration and selective classification are distinct capabilities, so deployment evaluation needs both metrics rather than either one alone.","Uncertainty signals are more trustworthy on reasoning tasks than on knowledge-heavy tasks, implying that abstention policies should be task-aware."],"supporting_citations":[{"why":"Defines token probability-based uncertainty as the complement of the exponentiated average log-likelihood, which is the TPU baseline.","marker":"[10]"},{"why":"Introduces numerical verbal uncertainty elicitation and the prompting approach the paper adapts for NVU.","marker":"[17]"},{"why":"Introduces linguistic verbal uncertainty, the central method the paper argues outperforms the baselines.","marker":"[18]"},{"why":"Supplies the hedging-word perception experiment replicated in Appendix H to validate the LVU judge.","marker":"[19]"},{"why":"Is the LLaMA-4 model family source for LLaMA-4-Maverick-17B-128E-Instruct, the judge used for all LVU scoring.","marker":"[23]"},{"why":"Is MMLU-Pro, the benchmark dataset on which all 80 models are evaluated for calibration and selective classification.","marker":"[28]"},{"why":"Defines expected calibration error, the calibration metric used throughout the study.","marker":"[32]"},{"why":"Defines selective classification and AUROC, the ranking metric used throughout the study.","marker":"[33]"}],"fun_headline_variants":["Linguistic hedging beats token odds for LLM uncertainty","Verbal uncertainty outperforms token probability in LLMs","Natural-language hedges give best calibration in LLMs","LVU wins over TPU and NVU for LLM uncertainty","Hedging words beat token probabilities in LLM study"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire LVU result rests on the judge model reading hedging words as a faithful sign of uncertainty; the paper validates that judge on only 20 crafted sentences and concedes its general reliability is unclear.","fun_headline_variants_meta":{"raw":{"variants":["Linguistic hedging beats token odds for LLM uncertainty","Verbal uncertainty outperforms token probability in LLMs","Natural-language hedges give best calibration in LLMs","LVU wins over TPU and NVU for LLM uncertainty","Hedging words beat token probabilities in LLM study"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000324,"raw_usage":{"total_tokens":1822,"prompt_tokens":951,"completion_tokens":871,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":790}},"tokens_in":567,"tokens_out":871,"duration_ms":7874,"temperature":1.0,"reasoning_tokens":790,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:55:17.791786+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-evaluate LVU on a sample of MMLU-Pro responses using a different judge, or human-annotated hedging scores, and check whether LVU still beats NVU and TPU by the reported margin; if the ranking flips or the advantage shrinks to noise, the central claim fails.","supporting_citations":[{"cited_title":"Semantic calibration of llms through the lens of temperature scaling","cited_arxiv_id":null,"evidence_quote":"Defines token probability-based uncertainty as the complement of the exponentiated average log-likelihood, which is the TPU baseline."},{"cited_title":"Llama 4: Multimodal intelligence","cited_arxiv_id":null,"evidence_quote":"Is the LLaMA-4 model family source for LLaMA-4-Maverick-17B-128E-Instruct, the judge used for all LVU scoring."},{"cited_title":"Benchmarking uncertainty disen- tanglement: Specialized uncertainties for specialized tasks.Advances in neural information processing systems, 37:50972–51038, 2024","cited_arxiv_id":null,"evidence_quote":"Defines selective classification and AUROC, the ranking metric used throughout the study."}],"review_version":1}