Pith. sign in

REVIEW 9 cited by

Benchmarking LLMs via Uncertainty Quantification

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.12794 v3 pith:KVEJ77XW submitted 2024-01-23 cs.CL

Fanghua Ye , Mingming Yang , Jianhui Pang , Longyue Wang , Derek F. Wong , Emine Yilmaz , Shuming Shi , Zhaopeng Tu This is my paper · ORCID
classification cs.CL
keywords llmsuncertaintyevaluationbenchmarkinglanguagequantificationaccuracyapproach
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The proliferation of open-source Large Language Models (LLMs) from various institutions has highlighted the urgent need for comprehensive evaluation methods. However, current evaluation platforms, such as the widely recognized HuggingFace open LLM leaderboard, neglect a crucial aspect -- uncertainty, which is vital for thoroughly assessing LLMs. To bridge this gap, we introduce a new benchmarking approach for LLMs that integrates uncertainty quantification. Our examination involves nine LLMs (LLM series) spanning five representative natural language processing tasks. Our findings reveal that: I) LLMs with higher accuracy may exhibit lower certainty; II) Larger-scale LLMs may display greater uncertainty compared to their smaller counterparts; and III) Instruction-finetuning tends to increase the uncertainty of LLMs. These results underscore the significance of incorporating uncertainty in the evaluation of LLMs.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Uncertainty-Aware Complex Scientific Table Data Extraction

    cs.IR 2025-07 conditional novelty 6.0 of 10

    Applying conformal prediction to TSR+OCR outputs flags incorrect table cells; the paper reports 53% labor savings and a 30-point accuracy gain, but the flagging threshold is fitted to the test data.

  2. PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Models

    cs.LG 2025-06 reject novelty 6.0 of 10

    PARC measures prompt sensitivity in VLMs, showing semantic changes hurt most and InternVL2 models are most robust.

  3. Is Your Model Fairly Certain? Uncertainty-Aware Fairness Evaluation for LLMs

    cs.CL 2025-05 conditional novelty 6.0 of 10

    UCerF scores LLM fairness by both correctness and confidence, and SynthBias provides 31,756 gender-occupation coreference samples for benchmark testing.

  4. Prune 'n Predict: Optimizing LLM Decision-making with Conformal Prediction

    cs.LG 2024-12 conditional novelty 6.0 of 10

    Conformal pruning of answer choices plus a second LLM pass improves MCQ accuracy in most tested settings, and learned scores yield smaller prediction sets.

  5. Learning Conformal Abstention Policies for Adaptive Risk Management in Large Language and Vision-Language Models

    cs.LG 2025-02 reject novelty 5.0 of 10

    CAP tunes conformal thresholds with RL to switch between single answers, sets, and abstention, but its test-set-fitting undermines the claimed statistical guarantees.

  6. Predictive Inference With Fast Feature Conformal Prediction

    cs.LG 2024-12 conditional novelty 5.0 of 10

    FFCP approximates feature conformal prediction with a gradient-normalized score, cutting runtime about 50x while maintaining coverage guarantees.

  7. Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Survey

    cs.CL 2025-02 conditional novelty 4.0 of 10

    A survey organizes current research on trustworthy RAG into six pillars, reliability, privacy, safety, fairness, explainability, and accountability, and maps methods, metrics, and open problems for each.

  8. A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directions

    cs.CL 2024-12 conditional novelty 4.0 of 10

    A review that organizes LLM uncertainty quantification into token-level, self-verbalized, semantic-similarity, and mechanistic interpretability categories.

  9. Evolution of Thought: Diverse and High-Quality Reasoning via Multi-Objective Optimization

    cs.NE 2024-11 reject novelty 4.0 of 10

    EoT applies multi-objective evolutionary search with crossover, mutation, and clustering to MLLM reasoning and reports improved Pass@K accuracy on MathVista, Math-Vision, and GSM8K.

Pith tools