REVIEW 5 cited by
Semantic Density: Uncertainty Quantification for Large Language Models through Confidence Measurement in Semantic Space
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
With the widespread application of Large Language Models (LLMs) to various domains, concerns regarding the trustworthiness of LLMs in safety-critical scenarios have been raised, due to their unpredictable tendency to hallucinate and generate misinformation. Existing LLMs do not have an inherent functionality to provide the users with an uncertainty/confidence metric for each response it generates, making it difficult to evaluate trustworthiness. Although several studies aim to develop uncertainty quantification methods for LLMs, they have fundamental limitations, such as being restricted to classification tasks, requiring additional training and data, considering only lexical instead of semantic information, and being prompt-wise but not response-wise. A new framework is proposed in this paper to address these issues. Semantic density extracts uncertainty/confidence information for each response from a probability distribution perspective in semantic space. It has no restriction on task types and is "off-the-shelf" for new models and tasks. Experiments on seven state-of-the-art LLMs, including the latest Llama 3 and Mixtral-8x22B models, on four free-form question-answering benchmarks demonstrate the superior performance and robustness of semantic density compared to prior approaches.
Forward citations
Cited by 5 Pith papers
-
UProp: Investigating the Uncertainty Propagation of LLMs in Multi-Step Agentic Decision-Making
UProp estimates extrinsic uncertainty in multi-step LLM decision-making by approximating pointwise mutual information over trajectory samples, improving AUROC for identifying incorrect agent outputs by 2.3 to 11% over...
-
A Survey on Semantic Communication for Vision: Categories, Frameworks, Enabling Techniques, and Applications
A survey that classifies visual semantic communication into preservation, expansion, and refinement categories and reviews their machine-learning components and applications.
-
Eliminating Hallucination-Induced Errors in LLM Code Generation with Functional Clustering
Functional clustering uses self-generated tests to group LLM code by exact output behavior, and abstains when the largest group is too small, trading answer coverage for lower error rates.
-
A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directions
A review that organizes LLM uncertainty quantification into token-level, self-verbalized, semantic-similarity, and mechanistic interpretability categories.
-
Assessing GPT Model Uncertainty in Mathematical OCR Tasks via Entropy Analysis
The paper reports that GPT-4o's token-level uncertainty, computed as the negative log-likelihood of its output, rises monotonically as image resolution falls from 300 to 72 dpi on a single test page.
Discussion (0). Continue with ORCID to comment.