Pith. sign in

REVIEW 1 cited by

Not yet reviewed by Pith; the record is open.

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.15814 v2 pith:HV2QZYPE submitted 2024-07-22 cs.CL cs.AIcs.LG

Perceptions of Linguistic Uncertainty by Language Models and Humans

classification cs.CL cs.AIcs.LG
keywords languagemodelsuncertaintyexpressionshumansstatementlinguisticprior
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

_Uncertainty expressions_ such as "probably" or "highly unlikely" are pervasive in human language. While prior work has established that there is population-level agreement in terms of how humans quantitatively interpret these expressions, there has been little inquiry into the abilities of language models in the same context. In this paper, we investigate how language models map linguistic expressions of uncertainty to numerical responses. Our approach assesses whether language models can employ theory of mind in this setting: understanding the uncertainty of another agent about a particular statement, independently of the model's own certainty about that statement. We find that 7 out of 10 models are able to map uncertainty expressions to probabilistic responses in a human-like manner. However, we observe systematically different behavior depending on whether a statement is actually true or false. This sensitivity indicates that language models are substantially more susceptible to bias based on their prior knowledge (as compared to humans). These findings raise important questions and have broad implications for human-AI and AI-AI communication.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs

    cs.CL 2026-07 conditional novelty 6.0

    A large-scale multilingual evaluation of LLM uncertainty estimation methods across 22 languages and 9 models finds that English reasoning closes the UE gap for low-resource languages and that optimal UE method choice ...