Pith. sign in

REVIEW 5 cited by

I Think, Therefore I am: Benchmarking Awareness of Large Language Models Using AwareBench

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.17882 v2 pith:TIGPFGYR submitted 2024-01-31 cs.CL

Yuan Li , Yue Huang , Yuli Lin , Siyuan Wu , Yao Wan , Lichao Sun This is my paper · ORCID
classification cs.CL
keywords llmsawarenessmodelsawarebenchdatasetdimensionsexhibitintelligence
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Do large language models (LLMs) exhibit any forms of awareness similar to humans? In this paper, we introduce AwareBench, a benchmark designed to evaluate awareness in LLMs. Drawing from theories in psychology and philosophy, we define awareness in LLMs as the ability to understand themselves as AI models and to exhibit social intelligence. Subsequently, we categorize awareness in LLMs into five dimensions, including capability, mission, emotion, culture, and perspective. Based on this taxonomy, we create a dataset called AwareEval, which contains binary, multiple-choice, and open-ended questions to assess LLMs' understandings of specific awareness dimensions. Our experiments, conducted on 13 LLMs, reveal that the majority of them struggle to fully recognize their capabilities and missions while demonstrating decent social intelligence. We conclude by connecting awareness of LLMs with AI alignment and safety, emphasizing its significance to the trustworthy and ethical development of LLMs. Our dataset and code are available at https://github.com/HowieHwong/Awareness-in-LLM.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges

    cs.CL 2026-05 conditional novelty 6.0 of 10

    Per-bias selection of a cross-family LLM auditor lifts biased-judgment accuracy from 0.805/0.824 baselines to 0.884.

  2. Truly Self-Improving Agents Require Intrinsic Metacognitive Learning

    cs.AI 2025-06 conditional novelty 5.0 of 10

    The paper proposes that self-improving agents must learn to manage their own learning processes, framing this as intrinsic metacognitive learning, and argues it is necessary for sustained and generalized improvement.

  3. Agents Require Metacognitive and Strategic Reasoning to Succeed in the Coming Labor Markets

    cs.AI 2025-05 conditional novelty 5.0 of 10

    AI agents in future labor markets will need metacognitive and strategic reasoning because incomplete information creates adverse selection, moral hazard, and reputation effects.

  4. Self-Critique-Guided Curiosity Refinement: Enhancing Honesty and Helpfulness in Large Language Models via In-Context Learning

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Adding a self-critique and refinement step to curiosity-driven prompting improves GPT-4o-judged honesty and helpfulness scores on HONESET by 1.4% to 4.3% across ten LLMs.

  5. Domain Specific Benchmarks for Evaluating Multimodal Large Language Models

    cs.LG 2025-06 conditional novelty 3.0 of 10

    A review paper that organizes domain-specific MLLM benchmarks into an eight-discipline taxonomy, with summary tables and performance highlights.

Pith tools