Pith. sign in

REVIEW 2 major objections 2 minor 13 references

Human Learning about AI

T0 review · 2 major / 2 minor · reviewed 2026-05-24 · grok-4.3

Challenge this review Download PDF Read on arXiv ↗

Pith's one-line read People project human task difficulty onto AI, overestimating performance on easy tasks and underestimating it on hard ones.

desk verdict The paper formalizes human projection of difficulty onto AI, with lab and field tests showing distorted beliefs and all-or-nothing adoption, but the isolation from priors and exposure is the main unverified spot. read the letter →

arxiv 2406.05408 v3 submitted 2024-06-08 econ.GN q-fin.EC

Bnaya Dreyfuss , Raphaël Raux This is my paper
classification econ.GNq-fin.EC
keywords humanprojectionAIadoptionbeliefformationtaskdifficultyanthropomorphismchatbotengagementperformanceevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper establishes that humans apply their own standards of task difficulty and mistake reasonableness when assessing AI capabilities. This projection produces systematic errors in beliefs, such as over-updating after easy failures or hard successes, and treats performance as evidence of a single underlying ability. As a result, adoption decisions tend toward all-or-nothing patterns even when AI outperforms humans only selectively. A field experiment with a parenting chatbot shows that mistakes perceived as less reasonable trigger sharper drops in trust and usage. These patterns arise because people use human evaluation frameworks for AI, which misaligns expectations when AI capabilities are jagged rather than ordered like human ones.

What carries the argument

Human Projection (HP), the mechanism of applying human difficulty and reasonableness standards to AI, which drives cross-task generalization and binary adoption.

What would settle it

An experiment presenting AI performance data without human-comparable task labels or cues and checking whether belief updates and adoption shift from all-or-nothing to selective patterns.

Watch

Extended reading notes

Core claim

Human Projection (HP) is the tendency to evaluate AI using the same frameworks applied to humans, where task difficulty and the reasonableness of mistakes serve as diagnostics of overall ability. This produces overestimation on human-easy tasks, underestimation on human-hard tasks, and over-updating after easy failures and hard successes. It also induces single-index interpretations of performance, leading to all-or-nothing adoption even when superiority is task-specific, with anthropomorphic cues strengthening the effect and a field setting confirming larger trust losses from unreasonable errors.

Load-bearing premise

The lab and field tasks isolate projection of human difficulty from other factors such as prior AI exposure or task-specific knowledge.

Editorial extensions

If this is right

  • When AI performance order differs from human difficulty order, beliefs become systematically misspecified.
  • Removing human-like cues from AI reduces cross-task generalization and over-adoption.
  • Mistakes that appear unreasonable by human standards produce larger drops in trust and engagement.
  • Anthropomorphic design choices amplify the projection effect on adoption.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Interfaces that present AI results without human difficulty framing could improve calibration of user expectations.
  • The same projection may apply to other non-human systems whose error patterns diverge from human norms.
  • Selective adoption could increase if users receive explicit task-by-task performance breakdowns rather than overall ability signals.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. This paper studies Human Projection (HP): the tendency to evaluate AI using human frameworks for task difficulty and mistake reasonableness. It formalizes HP and its consequences for equilibrium adoption, then tests predictions experimentally. Key findings include projection of human difficulty onto AI (overestimating performance on easy tasks, underestimating on hard ones, and over-updating after easy failures/hard successes), all-or-nothing adoption even when AI outperforms humans on only some tasks, and a field experiment with a parenting-advice chatbot showing that less reasonable mistakes reduce trust and engagement more. Anthropomorphic design is argued to amplify these effects.

Significance. If the results hold after addressing design details, the work offers a useful framework for understanding systematic biases in beliefs about AI capabilities and their impact on adoption. Strengths include the formalization of HP, the combination of lab experiments with a field test in a realistic setting, and the focus on design implications. These elements provide a foundation for further research on human-AI interaction.

major comments (2)
  1. [Abstract] Abstract: The description of multiple experiments and the parenting-chatbot field test provides no information on sample sizes, statistical power, pre-registration, or controls for alternative explanations such as prior AI exposure or task-specific priors. This is load-bearing for the central claim that belief distortions (overestimation on easy tasks, underestimation on hard ones, all-or-nothing adoption) are attributable to the HP mechanism rather than confounders.
  2. [Experimental design sections] Experimental design sections: The assumption that the tasks and field setting isolate projection from other factors (e.g., heterogeneous beliefs about AI jaggedness or differential exposure) is unverified as load-bearing. Without pre-treatment measurement of priors or regression controls, observed patterns could arise independently of HP, weakening attribution to the proposed mechanism.
minor comments (2)
  1. [Abstract] Abstract: The summary of results is information-dense; consider separating the theoretical predictions from the empirical findings for improved readability.
  2. Notation and terminology: Ensure consistent use of 'HP' and related terms across the formalization and empirical sections to avoid ambiguity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their detailed and constructive report. We address each major comment below. We agree that greater transparency on experimental details and robustness to alternative mechanisms will strengthen the manuscript and plan revisions accordingly.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The description of multiple experiments and the parenting-chatbot field test provides no information on sample sizes, statistical power, pre-registration, or controls for alternative explanations such as prior AI exposure or task-specific priors. This is load-bearing for the central claim that belief distortions (overestimation on easy tasks, underestimation on hard ones, all-or-nothing adoption) are attributable to the HP mechanism rather than confounders.

    Authors: We agree that the abstract should convey more information on these elements to support attribution to the HP mechanism. The main text and appendix report sample sizes, power analyses, and pre-registration status for each study, along with discussion of design features intended to limit confounding. In revision we will expand the abstract to include summary sample sizes, note pre-registration, and reference the controls for prior exposure and task priors that are detailed in the experimental sections. revision: yes

  2. Referee: [Experimental design sections] Experimental design sections: The assumption that the tasks and field setting isolate projection from other factors (e.g., heterogeneous beliefs about AI jaggedness or differential exposure) is unverified as load-bearing. Without pre-treatment measurement of priors or regression controls, observed patterns could arise independently of HP, weakening attribution to the proposed mechanism.

    Authors: We acknowledge that explicit pre-treatment measurement of priors and additional regression controls would provide stronger evidence that the observed patterns are driven by HP rather than heterogeneous beliefs about AI capabilities or differential exposure. The current designs rely on randomization across conditions and careful task selection to isolate the mechanism, but we will revise the experimental sections to include pre-treatment elicitation of relevant priors and report regressions that control for prior AI exposure and task-specific beliefs. These additions will directly test robustness to the alternative explanations raised. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: experimental paper with no derivation chain reducing to inputs

full rationale

The paper is an experimental study that formalizes Human Projection (HP) conceptually and tests its predictions via lab and field experiments. No closed-form derivations, equations, or fitted parameters are presented whose outputs reduce by construction to the inputs. The central claims rest on empirical isolation of projection effects rather than any self-referential modeling step. Self-citations, if present, are not load-bearing for any uniqueness theorem or ansatz. This is the expected outcome for a primarily empirical contribution.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Abstract-only review yields no explicit free parameters, axioms, or invented entities; the central construct (Human Projection) is presented as a behavioral tendency rather than a new postulated entity with independent evidence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Human Learning about AI." pith.science (2026). https://pith.science/paper/2406.05408

@misc{pith2026240605408,
  author       = {Pith},
  title        = {Pith review of: Human Learning about AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2406.05408}},
  note         = {Machine review of arXiv:2406.05408}
}
read the original abstract

We study \emph{Human Projection} (HP): people's tendency to evaluate AI using the same frameworks they use for humans -- treating features such as task difficulty and the reasonableness of mistakes as diagnostic of overall ability. We formalize HP and its consequences for equilibrium adoption, testing its predictions experimentally. First, people project human difficulty onto AI, overestimating performance on human-easy tasks, underestimating it on human-hard ones, and over-updating after easy failures and hard successes -- leading to systematic misspecification when AI performance is jagged rather than human-ordered. Second, HP interprets observed performance through a single ability index, inducing all-or-nothing adoption even when AI outperforms humans on only some tasks; experimentally stripping AI of human-like cues weakens cross-task generalization and reduces over-adoption. Finally, a field experiment with a parenting-advice chatbot shows that less humanly reasonable mistakes cause larger drops in trust and future engagement. Anthropomorphic AI design can amplify HP, misaligning beliefs and distorting adoption.

Figures

Figures reproduced from arXiv: 2406.05408 by the authors.

Figure 12
Figure 12. Adoption Path under HP vs. Optimal Adoption Path [PITH_FULL_IMAGE:figures/full_fig_p048_12.png] view at source ↗
Figure 17
Figure 17. CDF of Prior Beliefs in Performance 0 .2 .4 .6 .8 1 Cumulative probability 0 20 40 60 80 100 Beliefs in Agent Performance Humans AI Notes: The figure plots cumulative distribution functions of prior beliefs elicited in the second part of the experiment. On x-axis is the belief in %. Controlling for ability priors. In [PITH_FULL_IMAGE:figures/full_fig_p062_17.png] view at source ↗
Figure 20
Figure 20. Instructions Screen - Initial Test 65 [PITH_FULL_IMAGE:figures/full_fig_p066_20.png] view at source ↗
Figures from the paper (14 more)
Figure 27
Figure 27. Figure 27: Adoption Experiment Design Flowchart Endline Survey Adoption Decision Black box framing Anthrop. framing Instructions Delegate Problems Prior beliefs x60 Human AI Success OR Failure Training Phase Posterior beliefs Notes: The treatment variation is contained within in…
Figure 29
Figure 29. Figure 29: Example Prompt Screen - Anthropomorphic [PITH_FULL_IMAGE:figures/full_fig_p076_29.png]
Figure 30
Figure 30. Figure 30: Delegation Screen (Blue Task) - Anthropomorphic [PITH_FULL_IMAGE:figures/full_fig_p076_30.png]
Figure 31
Figure 31. Figure 31: Delegation Screen (Green Task) - Anthropomorphic [PITH_FULL_IMAGE:figures/full_fig_p077_31.png]
Figure 32
Figure 32. Figure 32: Performance Reveal Screen 76 [PITH_FULL_IMAGE:figures/full_fig_p077_32.png]
Figure 33
Figure 33. Figure 33: Prior Beliefs Screen - Anthropomorphic 77 [PITH_FULL_IMAGE:figures/full_fig_p078_33.png]
Figure 34
Figure 34. Figure 34: Final Adoption Screen - Anthropomorphic 78 [PITH_FULL_IMAGE:figures/full_fig_p079_34.png]
Figure 35
Figure 35. Figure 35: Histograms of queries and conversations per user [PITH_FULL_IMAGE:figures/full_fig_p080_35.png]
Figure 36
Figure 36. Figure 36: Histogram of match score 0 1 2 3 4 Share (%) .5 .6 .7 .8 .9 1 Dewey Score Among Exact Matches Notes: Upon receiving a user query, Dewey matches it to all questions in its database based on this score, and displays the top-ranked match if above a confidence threshold (…
Figure 39
Figure 39. Figure 39: Treatment Effect on Engagement Measures - Conditional Engagement [PITH_FULL_IMAGE:figures/full_fig_p085_39.png]
Figure 40
Figure 40. Figure 40: Screenshot of Reasonableness Elicitation [PITH_FULL_IMAGE:figures/full_fig_p091_40.png]
Figure 42
Figure 42. Figure 42: Flowchart of Engagement Experiment Instructions 3 useful conversations Endline Survey Engagement Link 2 unreasonable mistakes 2 reasonable mistakes Engagement Ecological Chatbot Webpage Notes: Useful conversations are held fixed across treatments, and are presented in…
Figure 44
Figure 44. Figure 44: Screenshot of Engagement Decision 96 [PITH_FULL_IMAGE:figures/full_fig_p097_44.png]
Figure 45
Figure 45. Figure 45: Screenshot of Dewey’s Webpage 97 [PITH_FULL_IMAGE:figures/full_fig_p098_45.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 13 canonical work pages

  1. [1]

    Pr(t = 1 | tδ− = 1) < Pr(t = 1 | tδ+ = 1)

    work page

  2. [2]

    Pr(t = 1 | tδ− = 0) < Pr(t = 1 | tδ+ = 0) Proposition 2 holds for posterior rates on any task, and for any difference in human difficulty on the observed task. Through the MLRP , the posterior distribution on ability following an observed success on the harder task first-order stochastically dominates the one following a success on the easier task (and co...

    work page

  3. [3]

    X is the power set of {e, d}

    Actions (X): Principal chooses AI or human for each task. X is the power set of {e, d}

    work page

  4. [4]

    Consequences (Y ∈ {0, 1}2): Observed binary performance (success/failure) tasks

    work page

  5. [5]

    Payoff (R : Y → R): Increases with Y elements

    work page

  6. [6]

    Bivariate uncorrelated Bernoulli with success rates based on the chosen agent for each task according to x

    Objective distribution (Q(·|x)): True distribution given an actionx. Bivariate uncorrelated Bernoulli with success rates based on the chosen agent for each task according to x

    work page

  7. [7]

    Bivariate independent Bernoulli with success rates ( ˆpD(θ; x), ˆpE(θ; x))

    Subjective distribution ( Qθ(·|x)): Distribution given an action x and a belief about AI ability θ. Bivariate independent Bernoulli with success rates ( ˆpD(θ; x), ˆpE(θ; x))

    work page

  8. [8]

    best explain

    Subjective expected payoff (Eθ(R|x)): Expected payoff given action x and belief θ. Each adoption action x ∈ X induces an objective distribution over consequences Q(· | x) ∈ ∆(Y), which is bi-variate uncorrelated Bernoulli with success rates ˜pE(x), and ˜pD(x) where ˜pk(x) is the success rate of the agent chosen to perform task k according to action x.36 A...

    work page 2016

Show all 13 references
  1. [9]

    For τ ∈ [0, τ1], no adoption is optimal

  2. [10]

    τ ∈ (τ1, τ2] partial adoption dominates no adoption, and is optimal for a subset of [τ1, τ2]

  3. [11]

    For τ ∈ [τ2, τ], full adoption is optimal. 39Notice that after the first period where partial adoption is optimal, the economy could, in principle, alternate between different types of partial adoption and full adoption, but it will eventually converge to full adoption. 46 We ...

  4. [12]

    If τ < ˜τ, then for any pA ∈ P ∗(τ), no adoption is the unique Berk-Nash equilibrium

  5. [13]

    Full Adoption*

    If τ ≥ ˜τ then for any pA ∈ P ∗(τ), full adoption is a Berk-Nash equilibrium. This theorem shows that (i) compared to the optimal path, adoption is delayed, but that (ii) at the early stages of adoption, over-adoption arises. In particular, for τ ∈ [τ1, ˜τ], some adoption is o...

    2015

Pith tools

Reviewed May 24, 2026 · model on record in the stance chip above.