REVIEW 7 cited by
On Benchmarking Human-Like Intelligence in Machines
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Recent advances in Artificial Intelligence (AI) have yielded powerful computational models that, by learning from vast amounts of human-generated data, are increasingly posited as approximate models of human cognition. However, we argue that many current evaluation paradigms for AI are insufficient for assessing human-like cognitive capabilities in these models. We identify a set of key shortcomings: a lack of human-validated labels, inadequate representation of human response variability and uncertainty, and reliance on simplified and ecologically invalid tasks. We support our claims by conducting a human evaluation study on nine existing AI benchmarks, suggesting major limitations in task and label designs. To address these limitations, we propose five concrete recommendations for future AI evaluation efforts that will enable more rigorous and meaningful understanding of human-like cognitive capacities in AI models.
Forward citations
Cited by 7 Pith papers
-
Language-Informed Synthesis of Rational Agent Models for Grounded Theory-of-Mind Reasoning On-The-Fly
LIRAS synthesizes PDDL world models and agent configurations from language, parses video frames into symbolic states, and runs Bayesian inverse planning (SIAM) to produce human-like graded social inferences across fiv...
-
Predicting Biased Human Decision-Making with Large Language Models in Conversational Settings
LLMs can partially predict individual and sample-level biased decisions in conversational settings, with GPT-4 aligning best, but reproduction of cognitive-load interactions is inconsistent.
-
Modeling Open-World Cognition as On-Demand Synthesis of Probabilistic Models
A hybrid language-model and probabilistic-program architecture predicts human judgments on novel open-world reasoning vignettes better than language-model-only baselines.
-
Are Large Language Models Reliable AI Scientists? Assessing Reverse-Engineering of Black-Box Systems
LLMs struggle to use passive observations for reverse engineering, but active intervention improves performance, largely through the process of generating queries rather than the data obtained.
-
What's in the Box? Reasoning about Unseen Objects from Multimodal Cues
A neurosymbolic pipeline combining LLM parsing, audio classification, and Bayesian reasoning achieves r=0.78 correlation with human judgments on a new hidden-object guessing task.
-
Using LLMs to Advance the Cognitive Science of Collectives
A position paper arguing that LLMs can help cognitive scientists study collective behavior along structural, interactional, and individual complexity axes, with cautions about bias and reproducibility.
-
Belief Attribution as Mental Explanation: The Role of Accuracy, Informativity, and Causality
When people say what an agent believes, they prefer beliefs that are causally relevant to the agent's actions, more than beliefs that are merely accurate or informative.
Discussion (0). Continue with ORCID to comment.