Pith. sign in

REVIEW 4 major objections 5 minor 34 references

Scaling up the think-aloud method

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that an automated pipeline can transcribe and code think-aloud speech into search graphs at thousands-of-participants scale, making process-level analysis of human reasoning practical again.

desk verdict Solid proof-of-concept for automated think-aloud coding, with real scale, but the load-bearing downstream findings rest on differential-error assumptions the paper never tests. read the letter →

arxiv 2505.23931 v1 pith:GL22QTFI submitted 2025-05-29 cs.CL cs.AI

Daniel Wurgaft , Ben Prystawski , Kanishk Gandhi , Cedegao E. Zhang , Joshua B. Tenenbaum , Noah D. Goodman This is my paper
classification cs.CLcs.AI
keywords think-aloudverbalprotocolslanguagemodelssearchgraphsGameof24reasoningprocessesautomatedtranscriptioninter-raterreliability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that the think-aloud method, long abandoned by cognitive science because human transcription and annotation are too slow, can be run at scale with automated tools. The authors had 640 participants think aloud while playing the Game of 24, transcribed the recordings with Whisper, and used a large language model to convert each transcript into a search graph of the arithmetic states and operations the participant explored. They argue that LLM-coded graphs, despite lower agreement with human coders than humans have with each other, are close enough to support large-scale process-level analyses of reasoning. If right, think-aloud studies can return to cognitive science as a practical source of data about the time course of reasoning, not just its outcomes. To show the payoff, the paper uses the graphs to measure operation frequencies, subgoal use, and predictors of success such as whether participants considered division.

What carries the argument

The load-bearing mechanism is a search-graph representation of a verbal protocol, built by prompting a large language model to write Python code against a custom GraphBuilder class. Nodes in the graph are game states, the sets of numbers still available; edges are arithmetic operations that combine two numbers; and subgoals are represented as backward edges from the goal state. An automatic checker detects common coding errors, such as computing a wrong resulting state or using numbers that do not exist in the current state, and an iterative repair loop asks the model to fix its code up to five times, keeping the graph with the fewest errors. This object turns free-form speech into a standardized structure that can be compared across participants and models via normalized graph edit distance.

What would settle it

Hand-code a random sample of the 4,947 trials and check whether the reported effects, including division-only problems solved 20% of the time versus 59% for non-division problems, 65% versus 53% success with one subgoal, and the Gini clustering for operation sequences, appear in the human-coded graphs; if they disappear, the pipeline's systematic coding errors, not human reasoning, produced them.

Watch

Extended reading notes

Core claim

The paper's central claim is that an automated pipeline can turn think-aloud speech into structured search graphs at a scale that manual coding cannot reach, with enough fidelity to support psychological analysis. On nearly 5,000 trials of the Game of 24, a speech-to-text model transcribed participants' spoken arithmetic, and a large language model converted each transcript into a graph whose nodes are game states and whose edges are operations or subgoals. The paper reports that the best model, Claude 3.5 Sonnet, produces graphs that are qualitatively similar to human-coded graphs, preserve the root-to-goal path, and agree with human coders well enough that the paper uses its annotations for all downstream analyses. This allows the paper to describe human search behavior, such as which operations people try, how often they set subgoals, and how search patterns differ between success and failure, from thousands of traces rather than dozens.

Load-bearing premise

The psychological conclusions assume that Claude 3.5 Sonnet's graph annotations are accurate enough that aggregate patterns, such as subgoal use predicting success, reflect the participants' actual reasoning; model-human agreement is significantly lower than human-human agreement, and the paper notes that the key patterns were weaker or non-significant when other models coded the graphs.

Editorial extensions

If this is right

  • Think-aloud studies can be run at thousands of participants, with coding time and cost reduced from roughly 816 person-hours to about two hours and $74.
  • Prompting people to think aloud does not appear to change which problems are hard, though it slows responses, supporting the non-reactivity of verbal protocols.
  • Successful and unsuccessful search differ in breadth: incorrect trials contain more explored operations, and people who fail often never try division.
  • Subgoal use is rare overall, but setting one subgoal is associated with higher success while setting multiple subgoals is not, suggesting backward search helps only when it works.
  • Human search is much more clustered than random search on operation sequences of length two and three, indicating consistent multi-step reasoning strategies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper leaves implicit: the same transcription-plus-graph-coding pipeline could be applied to archived think-aloud corpora from chess or physics, retrofitting process-level scale to classic small-n studies.
  • The dependence on a closed-weights model for the reported patterns suggests a concrete robustness check: re-run the downstream analyses with an open-weights model once its coding agreement matches Claude, and require the subgoal and division effects to survive.
  • A natural follow-up would compare the button-press action logs from the silent control condition with the search graphs from the think-aloud condition, testing whether verbalization changes the structure of search rather than only its speed.
  • The paper's aggregate conclusions would be placed on firmer ground if the reported effects were shown to hold on graphs coded by several independent strong models, since the authors note the key patterns were weaker or non-significant with other models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper develops an automated pipeline for coding think-aloud verbal protocols into search graphs, using Whisper for speech transcription and LLMs to generate Python code that constructs a graph representation of a participant's search process. The authors validate the pipeline on 50 trials manually coded by two human raters from a larger experiment (640 think-aloud participants, ~5,000 trials) and report that model-human graph edit distances are higher than human-human distances but with substantial overlap. They then use graphs produced by Claude 3.5 Sonnet for all downstream analyses, which characterize operation frequencies, sequential consistency (Gini indices), difficulty of division problems, and the association between subgoal setting and success, accompanied by a comparison of think-aloud versus control conditions. The paper emphasizes the time and cost savings over manual coding and frames the work as a proof of concept for scaling think-aloud analysis.

Significance. If the reliability concerns are addressed, this is a valuable methodological contribution. The paper provides the largest think-aloud dataset that I am aware of, with open code and data and a preregistered design. The pipeline is concrete and reproducible in structure, and the inclusion of multiple LLMs, an automatic graph checker, and a random-agent baseline are strengths. The honest reporting of limitations (e.g., incompleteness of verbal reports and graph representation) is commendable. However, the central validation is underreported, and the downstream psychological conclusions rest on a single LLM coder without a test for whether its coding errors are systematically related to the outcome variables of interest. These issues are load-bearing for the paper's claims and should be resolved before the findings can be fully accepted.

major comments (4)
  1. [Reliability and validity (Figure 1c)] The validation section reports only p-values and a figure, with no numeric edit distances or confidence intervals. This makes it impossible to assess the magnitude of the gap between model-human and human-human agreement, or to compare models quantitatively. Please report the mean normalized graph edit distances and confidence intervals for the human-human comparison and for each model-human comparison, along with effect sizes and, if possible, the full distributional summaries.
  2. [Footnote 3 and 'Predictors of success and failure in reasoning'] The downstream findings, including the subgoal-success association (65% vs. 53%, p<0.0001), are computed exclusively from Claude 3.5 Sonnet's graphs. The paper does not test whether the model's coding errors are independent of trial outcomes. If, for example, Claude detects or hallucinates subgoals more often in successful trials, the reported association could be inflated. The footnote's admission that these patterns are weaker or non-significant with DeepSeek V3 and Llama 4 Maverick makes this concern concrete. Please add a differential-error analysis: on the 50 human-coded trials, compare the key aggregate findings using human graphs versus Claude graphs, and/or stratify model-human agreement by trial success and by the presence of subgoals.
  3. [Inter-rater reliability] Claude 3.5 Sonnet is selected as the production coder based on its agreement with human coders on the same 50 trials used for the validation, yet the reported permutation p-value for Claude's agreement does not account for this model-selection step. This is a multiple-comparisons issue. Either correct the p-value for the number of models evaluated or validate the selected model on a held-out subset of the 50 human-coded trials.
  4. [Reliability and validity] The manuscript states that model-human agreement is 'close enough' to be informative but does not provide a pre-specified or justified criterion for what level of agreement is sufficient for the intended downstream aggregate analyses. Given that model-human agreement is significantly lower than human-human agreement, please specify a benchmark (e.g., a threshold tied to effect sizes in the downstream analyses) or otherwise justify the sufficiency, for example by showing that the main conclusions are robust when using graphs from the other top-performing models.
minor comments (5)
  1. [Figure 2b] The two example graphs (low vs. high edit distance) are not self-explanatory; please add a legend or a more detailed caption that indicates what features distinguish the low-disagreement case from the high-disagreement case.
  2. [Predictors of success and failure in reasoning] The interpretation that setting multiple subgoals 'might indicate that a participant struggled' is speculation; please label it explicitly as a hypothesis rather than a conclusion.
  3. [Filtering irrelevant transcripts] The relevance classification performed by Llama 3.3 70B is not validated; please report its agreement with human judgments on a sample, or at least provide a description of how the prompt was designed and tested.
  4. [Consistency and variation in search patterns] The Gini indices are computed only over subsequences that occur at least once in the data; this can inflate Gini values, particularly for short subsequences. Please discuss the direction and potential magnitude of this bias.
  5. [Keywords and model names] There is a missing space in 'Keywords:think-aloud' in the abstract, and the model name 'Llama 3.3 70b' uses an inconsistent lowercase 'b' compared with other model names.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the pipeline is validated against external human annotations, and the lone self-citation is context-setting rather than load-bearing.

full rationale

This paper's derivation chain is self-contained and validated against external human annotations, so no circular step rises to the level defined by the review criteria. The central reliability claim is established by comparing LLM-produced graphs with manually annotated graphs from 50 trials, an external ground truth produced by human coders, and the model chosen (Claude 3.5 Sonnet) is selected by that external agreement rather than by any quantity derived from the model itself. The downstream analyses of operation frequencies, subgoal rates, and success predictors are aggregate descriptions of the Claude-coded graphs; although the paper candidly notes in footnote 3 that these patterns were weaker or non-significant with other models, this is a robustness limitation, not circularity. The single self-citation (Gandhi et al., 2024, which shares authors) is used only to motivate Game of 24 as a reasoning benchmark and is not load-bearing for the method or conclusions. Human coders used the authors' prompt and graph checker, which anchors the coding scheme, but this is standard practice for establishing inter-rater reliability of a coding scheme, not a reduction of the prediction to its inputs. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no known result is merely relabeled. Consequently, the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the modeling assumption that reasoning in this task is a search process representable by arithmetic operations on states, and on the accuracy of the chosen LLM's coding. The free parameters are hand-chosen thresholds in the data processing pipeline that shape the dataset. No new entities are postulated.

free parameters (4)
  • Trial exclusion threshold = at least half of a participant's trials excluded
    Participants with at least half of trials lacking relevant information were fully excluded; this post hoc threshold shapes the final dataset of 4,947 trials.
  • Audio truncation limit = 181 seconds
    Trials above 181 seconds were truncated and marked incorrect; chosen by hand to allow one second of lag beyond the 3-minute limit.
  • Silence threshold for transcription = 20 seconds
    Used in Whisper transcription to skip long silences; chosen by the authors.
  • Temperature increase on retry = +0.1 per failed iteration
    When the LLM failed to improve, temperature was increased by 0.1; this is a procedural choice that affects graph corrections.
assumptions (4)
  • domain assumption Human reasoning in Game of 24 can be represented as a search graph where each utterance maps to an arithmetic operation or subgoal on the current state.
    Section 'Coding transcripts as graphs'; the authors acknowledge this is a theoretical commitment but build only general operations to avoid specific commitments.
  • domain assumption The LLM-generated graphs, after automatic error checking, are accurate enough for aggregate analysis.
    The validation shows model-human agreement is lower than human-human agreement; downstream analyses assume the errors are not systematically biasing the aggregate patterns.
  • domain assumption The random policy agent is an appropriate null model for comparing search clustering.
    In 'Consistency and variation', Gini indices are compared to an agent that explores random legal actions matched in number of operations; this assumes random search is the relevant baseline.
  • domain assumption The preregistered exclusion rules and LLM-based relevance filtering do not introduce systematic bias.
    1,240 trials were excluded based on an LLM's relevance judgment; this assumes the filtering is unbiased.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling up the think-aloud method." pith.science (2026). https://pith.science/paper/GL22QTFI

@misc{pith2026250523931,
  author       = {Pith},
  title        = {Pith review of: Scaling up the think-aloud method},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GL22QTFI}},
  note         = {Machine review of arXiv:2505.23931}
}
read the original abstract

The think-aloud method, where participants voice their thoughts as they solve a task, is a valuable source of rich data about human reasoning processes. Yet, it has declined in popularity in contemporary cognitive science, largely because labor-intensive transcription and annotation preclude large sample sizes. Here, we develop methods to automate the transcription and annotation of verbal reports of reasoning using natural language processing tools, allowing for large-scale analysis of think-aloud data. In our study, 640 participants thought aloud while playing the Game of 24, a mathematical reasoning task. We automatically transcribed the recordings and coded the transcripts as search graphs, finding moderate inter-rater reliability with humans. We analyze these graphs and characterize consistency and variation in human reasoning traces. Our work demonstrates the value of think-aloud data at scale and serves as a proof of concept for the automated analysis of verbal reports.

Figures

Figures reproduced from arXiv: 2505.23931 by the authors.

Figure 1
Figure 1. Overview of the data processing pipeline. (a) Participants solve the task while speaking aloud. We transcribe their data using a speech-to-text model and use an LLM to translate the transcripts into Python code that produces a graph representation of their search. (b) Example graphs for one problem in the dataset. (c) Normalized graph edit distances between the graphs produced by human and language model coders. Low… view at source ↗
Figure 2
Figure 2. Analysis of model-generated graphs. (a) Counts of each type of automatically detected error for the three models that agree most with human coders. (b) Example graphs for trials with low and high mean human-model edit distance using Claude 3.5 Sonnet. Results Our analyses are organized around two goals: assessing the reliability and validity of the think-aloud method in our set￾ting and examining characteristics of … view at source ↗
Figure 3
Figure 3. Summary of participant data. (a) Proportion of correct answers for every problem in the dataset. (b) Mean number of uses of each operation in trials where the participant was correct and incorrect. (c) Proportion of subgoals that are of each type, defined by how one can reach the goal from them. suggest a change to the underlying reasoning processes. We computed split-half correlations for mean accuracy by prob￾lem … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: (a) Gini indices for sequences of operations of lengths 1 through 5 in humans (blue) and random search agents (gray). [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 31 canonical work pages

  1. [1]

    , Raveaux, R

    abu2015exact APACrefauthors Abu-Aisheh, Z. , Raveaux, R. , Ramel, J Y. \ Martineau, P. APACrefauthors \ 2015 . An exact graph edit distance algorithm for solving pattern recognition problems An exact graph edit distance algorithm for solving pattern recognition problems . 4th International Conference on Pattern Recognition Applications and Methods 2015. 4...

    work page 2015

  2. [2]

    APACrefauthors \ 2024

    anthropic2024claude APACrefauthors Anthropic . APACrefauthors \ 2024 . Claude 3.5 Sonnet Model Card Addendum. Claude 3.5 sonnet model card addendum. APACrefURL https://www-cdn.anthropic.com/fed9cc193a14b84131812372d8d5857f8f304c52/Model_Card_Claude_3_Addendum.pdf APACrefURL

    work page 2024

  3. [3]

    , Blach, N

    besta2024graph APACrefauthors Besta, M. , Blach, N. , Kubicek, A. , Gerstenberger, R. , Podstawski, M. , Gianinazzi, L. others APACrefauthors \ 2024 . Graph of thoughts: Solving elaborate problems with large language models Graph of thoughts: Solving elaborate problems with large language models . Proceedings of the AAAI Conference on Artificial Intellige...

    work page 2024

  4. [4]

    , Mann, B

    brown2020language APACrefauthors Brown, T. , Mann, B. , Ryder, N. , Subbiah, M. , Kaplan, J D. , Dhariwal, P. others APACrefauthors \ 2020 . Language models are few-shot learners Language models are few-shot learners . Advances in neural information processing systems 33 1877--1901

    work page 2020

  5. [5]

    \ Simon, H A

    ericsson1993protocol APACrefauthors Ericsson, K A. \ Simon, H A. APACrefauthors \ 1993 . Protocol Analysis: Verbal Reports as Data Protocol analysis: Verbal reports as data . The MIT Press . APACrefDOI doi:10.7551/mitpress/5657.001.0001 APACrefDOI

    work page doi:10.7551/mitpress/5657.001.0001 1993

  6. [6]

    , Ericsson, K A

    fox2011procedures APACrefauthors Fox, M C. , Ericsson, K A. \ Best, R. APACrefauthors \ 2011 . Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods. Do procedures for verbal reporting of thinking have to be reactive? a meta-analysis and recommendations for best reporting methods...

    work page 2011

  7. [7]

    , Lee, D H J

    gandhi2024stream APACrefauthors Gandhi, K. , Lee, D H J. , Grand, G. , Liu, M. , Cheng, W. , Sharma, A. \ Goodman, N. APACrefauthors \ 2024 . Stream of Search (SoS): Learning to Search in Language Stream of search (sos): Learning to search in language . First Conference on Language Modeling. First conference on language modeling. APACrefURL https://openre...

    work page 2024

  8. [8]

    , Dubey, A

    grattafiori2024llama APACrefauthors Grattafiori, A. , Dubey, A. , Jauhri, A. , Pandey, A. , Kadian, A. , Al-Dahle, A. others APACrefauthors \ 2024 . The llama 3 herd of models The llama 3 herd of models . arXiv preprint arXiv:2407.21783

    arXiv 2024

Show all 34 references
  1. [9]

    , Swart, P J

    hagberg2008exploring APACrefauthors Hagberg, A. , Swart, P J. \ Schult, D A. APACrefauthors \ 2008 . Exploring network structure, dynamics, and function using NetworkX Exploring network structure, dynamics, and function using networkx \ . Los Alamos National Laboratory (LANL),...

    2008

  2. [10]

    , Tenenbaum, J B

    hartshorne2018critical APACrefauthors Hartshorne, J K. , Tenenbaum, J B. \ Pinker, S. APACrefauthors \ 2018 . A critical period for second language acquisition: Evidence from 2/3 million English speakers A critical period for second language acquisition: Evidence from 2/3 mill...

    2018

  3. [11]

    APACrefauthors \ 2024

    hertzum2024concurrent APACrefauthors Hertzum, M. APACrefauthors \ 2024 . Concurrent or Retrospective Thinking Aloud in Usability Tests? A Meta-Analytic Review Concurrent or retrospective thinking aloud in usability tests? A meta-analytic review . ACM Transactions on Computer-H...

    2024

  4. [12]

    \ Simon, H A

    kaplan1990search APACrefauthors Kaplan, C A. \ Simon, H A. APACrefauthors \ 1990 . In search of insight In search of insight . Cognitive psychology 22 3 374--419

    1990

  5. [13]

    , McDermott, J

    larkin1980expert APACrefauthors Larkin, J. , McDermott, J. , Simon, D P. \ Simon, H A. APACrefauthors \ 1980 . Expert and novice performance in solving physics problems Expert and novice performance in solving physics problems . Science 208 4450 1335--1342

    1980

  6. [14]

    , Feng, B

    liu2024deepseek APACrefauthors Liu, A. , Feng, B. , Xue, B. , Wang, B. , Wu, B. , Lu, C. others APACrefauthors \ 2024 . Deepseek-v3 technical report Deepseek-v3 technical report . arXiv preprint arXiv:2412.19437

    2024 arXiv

  7. [15]

    APACrefauthors \ 1982

    marr1982vision APACrefauthors Marr, D. APACrefauthors \ 1982 . Vision: A computational investigation into the human representation and processing of visual information Vision: A computational investigation into the human representation and processing of visual information . MIT press

    1982

  8. [16]

    , Kirsh, D

    mccarthy2023consistency APACrefauthors McCarthy, W P. , Kirsh, D. \ Fan, J E. APACrefauthors \ 2023 . Consistency and Variation in Reasoning About Physical Assembly Consistency and variation in reasoning about physical assembly . Cognitive Science 47 12 e13397

    2023

  9. [17]

    , Ungermann, P

    mechera2024using APACrefauthors Mechera-Ostrovsky, T. , Ungermann, P. \ Donkin, C. APACrefauthors \ 2024 . Using machine learning to automate the collection, transcription, and analysis of verbal-report data Using machine learning to automate the collection, transcription, and...

    2024

  10. [18]

    \ Simon, H A

    newell1961gps APACrefauthors Newell, A. \ Simon, H A. APACrefauthors \ 1961 . GPS , a program that simulates human thought GPS , a program that simulates human thought . Rand Corporation Santa Monica, CA

    1961

  11. [19]

    \ Simon, H A

    newell_simon1972human APACrefauthors Newell, A. \ Simon, H A. APACrefauthors \ 1972 . Human problem solving Human problem solving . Englewood Cliffs, NJ Prentice-Hall

    1972

  12. [20]

    \ Newell, B R

    ostrovsky2024verbal APACrefauthors Ostrovsky, T. \ Newell, B R. APACrefauthors \ 2024 . Verbal reports as data revisited: Using natural language models to validate cognitive models. Verbal reports as data revisited: Using natural language models to validate cognitive models. Decision

    2024

  13. [21]

    , Bourgin, D D

    peterson2021using APACrefauthors Peterson, J C. , Bourgin, D D. , Agrawal, M. , Reichman, D. \ Griffiths, T L. APACrefauthors \ 2021 . Using large-scale experiments and machine learning to discover theories of human decision-making Using large-scale experiments and machine lea...

    2021

  14. [22]

    , Kim, J W

    radford2023robust APACrefauthors Radford, A. , Kim, J W. , Xu, T. , Brockman, G. , McLeavey, C. \ Sutskever, I. APACrefauthors \ 2023 . Robust speech recognition via large-scale weak supervision Robust speech recognition via large-scale weak supervision . International confere...

    2023

  15. [23]

    , Johnson, E J

    russo1989validity APACrefauthors Russo, J E. , Johnson, E J. \ Stephens, D L. APACrefauthors \ 1989 . The validity of verbal protocols The validity of verbal protocols . Memory & cognition 17 759--769

    1989

  16. [24]

    APACrefauthors \ 2025 04

    qwen2025qwen APACrefauthors Team, Q. APACrefauthors \ 2025 04 . Qwen3: Think Deeper , Act Faster . Qwen3: Think Deeper , Act Faster . https://qwenlm.github.io/blog/qwen3/

    2025

  17. [25]

    , Yang, J

    tong2014card APACrefauthors Tong, L. , Yang, J. , Han, X. \ Velasquez, L. APACrefauthors \ 2014 . The card game 24 and its application to math education The card game 24 and its application to math education . International Journal of Mathematical Education in Science and Tech...

    2014

  18. [26]

    \ Nyamsuren, E

    van2017cognitive APACrefauthors van der Maas, H L. \ Nyamsuren, E. APACrefauthors \ 2017 . Cognitive analysis of educational games: The number game Cognitive analysis of educational games: The number game . Topics in cognitive science 9 2 395--412

    2017

  19. [27]

    APACrefauthors \ 2017

    van2017examining APACrefauthors Van de Wiel, M W. APACrefauthors \ 2017 . Examining Expertise Using Interviews and Verbal Protocols. Examining expertise using interviews and verbal protocols. Frontline Learning Research 5 3 112--140

    2017

  20. [28]

    , Barnard, Y F

    van1994think APACrefauthors Van Someren, M. , Barnard, Y F. \ Sandberg, J. APACrefauthors \ 1994 . The think aloud method: a practical approach to modelling cognitive processes The think aloud method: a practical approach to modelling cognitive processes . Academic Press

    1994

  21. [29]

    \ Newell, A

    waterman1971protocol APACrefauthors Waterman, D A. \ Newell, A. APACrefauthors \ 1971 . Protocol analysis as a task for artificial intelligence Protocol analysis as a task for artificial intelligence . Artificial intelligence 2 3-4 285--318

    1971

  22. [30]

    APACrefauthors \ 1994

    wilson1994proper APACrefauthors Wilson, T D. APACrefauthors \ 1994 . The proper protocol: Validity and completeness of verbal reports The proper protocol: Validity and completeness of verbal reports . Psychological Science 5 5 249--252

    1994

  23. [31]

    , Xiong, H

    xie2024evaluating APACrefauthors Xie, H. , Xiong, H. \ Wilson, R. APACrefauthors \ 2024 1 . Evaluating Predictive Performance and Learning Efficiency of Large Language Models with Think Aloud in Risky Decision Making Evaluating predictive performance and learning efficiency of...

    2024

  24. [32]

    , Xiong, H

    xie2023text2decision APACrefauthors Xie, H. , Xiong, H. \ Wilson, R C. APACrefauthors \ 2023 . Text2Decision: Decoding latent variables in risky decision making from think aloud text Text2decision: Decoding latent variables in risky decision making from think aloud text . Neur...

    2023

  25. [33]

    , Xiong, H D

    xie2024strategic APACrefauthors Xie, H. , Xiong, H D. \ Wilson, R. APACrefauthors \ 2024 2 . From Strategic Narratives to Code-Like Cognitive Models: An LLM-Based Approach in A Sorting Task From strategic narratives to code-like cognitive models: An llm-based approach in a sor...

    2024

  26. [34]

    yao2023tree APACrefauthors Yao, S. , Yu, D. , Zhao, J. , Shafran, I. , Griffiths, T. , Cao, Y. \ Narasimhan, K. APACrefauthors \ 2023 . Tree of Thoughts : Deliberate Problem Solving with Large Language Models Tree of Thoughts : Deliberate problem solving with large language mo...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.