Can LLMs Help Improve Analogical Reasoning For Strategic Decisions? Experimental Evidence from Humans and GPT-4

Maciej Workiewicz; Phanish Puranam; Prothit Sen

REVIEW 4 major objections 6 minor 6 references

Can LLMs Help Improve Analogical Reasoning For Strategic Decisions? Experimental Evidence from Humans and GPT-4

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Humans remain better than GPT-4 at the matching step of analogical reasoning, where choosing among plausible analogies requires causal structure, not just retrieval.

desk verdict The two-source/two-target design is a welcome extension, but the headline human-matching advantage is almost certainly a response-format artifact: GPT-4 lists both stories, so recall is mechanically 1.0 and precision sits near 0.5. read the letter →

arxiv 2505.00603 v1 pith:O7AREJ4Z submitted 2025-05-01 cs.AI cs.HC

Phanish Puranam , Prothit Sen , Maciej Workiewicz This is my paper · ORCID

classification cs.AIcs.HC

keywords analogicalreasoningstrategicdecisionmakinglargelanguagemodelsGPT-4matchingproblemprecisionandrecallcausalmappinghuman-AIcomplementarity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether GPT-4 can do the full job of analogical reasoning in strategic decisions—not just retrieve candidate analogies but pick the one whose causal structure fits the problem. In an experiment where two source stories (a converging-radiation technique and a survivorship-bias tale about dolphins) are matched against two business problems, GPT-4 retrieves nearly every applicable analogy but also applies many wrong ones, while human participants select fewer analogies and almost always select the right one. The paper concludes that the matching stage—adjudicating among superficially plausible analogues—is where human strategic reasoners keep a decisive advantage over current large language models. If correct, the practical consequence is a division of labor: language models as broad analogy generators, humans as evaluative filters.

What carries the argument

The load-bearing instrument is the experimental design itself. It turns the classic one-source-to-one-target analogy experiment into a matching problem: two source stories (the split-and-converge radiation technique and the dolphins-as-survivorship-bias story) pair with two target problems (city factory logistics and an HR training pilot), with known correct mappings from story to problem. Because the correct mappings are known in advance, every response can be placed in a confusion matrix and scored for precision, recall, F1, and accuracy. Directed acyclic graphs of each story's causal schema serve as the coding standard for whether an analogy was applied correctly, and the resulting errors are classified as surface-level or structural. This setup separates retrieval (did the agent use any analogy?) from matching (was it the structurally right one?), which is what allows the paper to locate the human advantage in the evaluative phase.

What would settle it

Re-run GPT-4 on the same two-story protocol for many more trials across varied sampling settings and random seeds, then inspect the distribution of claimed analogies in each condition. If the high-recall/low-precision pattern is stable across seeds, the paper's account is supported; if the outputs collapse to a few repeated texts or flip sharply with settings, the 15-run cells were not independent observations and the reported metrics overstate confidence in the model's behavior.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that human and machine analogical reasoning fail in mirror-image ways, and that the decisive bottleneck is the evaluative matching step. With an explicit hint that one of the two stories is relevant, GPT-4 achieved perfect recall (1.00) for both stories but precision of only 0.52 and 0.50, whereas humans achieved precision of 0.67 and 0.75 with recall of only 0.26 and 0.39. Without a hint, GPT-4 never spontaneously used the correct analogy, while humans did, with perfect precision but low recall. Coding of the wrong analogies shows that GPT-4's errors are driven mainly by surface similarity, such as linking the dolphin story to a coastal factory through the sea, whereas human errors come from applying an incorrect causal schema. The paper argues that this makes matching a first-order component of strategic analogical reasoning, and grounds a human-in-the-loop division of labor.

Load-bearing premise

The central claim rests on treating each of the 60 GPT-4 runs as an independent observation, but the paper reports little variation in the model's output and deems 15 runs per condition sufficient; if the sampling is near-deterministic, the reported precision and recall could describe one response pattern rather than stable properties of the model.

Editorial extensions

If this is right

In time-sensitive settings where missing a valid analogy is costlier than chasing false leads, GPT-4's perfect recall makes it a useful first-pass generator of candidate analogies.
In high-stakes, hard-to-reverse strategic choices, the human role as evaluative filter should be preserved, because human precision is markedly higher.
Because adding a choice among sources lowers solvability relative to one-to-one analogy tasks, performance claims from single-source analogy studies overstate what either humans or LLMs can do in realistic multi-source settings.
As LLMs expand the candidate set, training managers in structural comparison and causal mapping becomes more valuable rather than less.
The complementary error profiles (surface-driven for LLMs, causal-schema for humans) suggest ensembles that combine both agents could reduce overall error, though the paper does not identify the optimal ensemble design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If GPT-4's output is as near-deterministic as the paper suggests, the perfect recall under the hint may partly reflect the model complying with the cue rather than genuine retrieval breadth; testing with withheld hints and more distractors would separate these.
Scaling the design from two sources to dozens could erode the human precision advantage, since evaluative capacity is itself limited; the proposed division of labor may then need a second AI filter before human review.
The paper's error classification relies on written justifications; asking both agents to justify why they rejected the non-chosen source would test whether the surface-versus-structural asymmetry is in reasoning or only in post-hoc explanation.
A direct implication for future model design is that the most promising lever is causal-schema discrimination, not wider retrieval.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The setup is genuinely new: extending Gick and Holyoak to a matching task with two sources and two targets, and running the same vignettes with humans and GPT-4. The error decomposition into surface versus structural misapplication is a useful lens, and the proposed division of labor (AI generates analogies, humans filter) is a natural implication if the asymmetry were real.

But the core empirical claim does not hold up under scrutiny. The stress-test note is right. In the hint condition, GPT-4 almost always mentions both source stories. Table 5 shows TP=15 and FP=14 or 15 for both stories. That means the model lists both candidates regardless of which is correct. When a system always lists both, recall is mechanically 1.0 and precision sits near 0.5. Human participants, by contrast, typically report one story or none. So the precision/recall gap is telling us about response format and the model's over-inclusive style, not about its capacity to adjudicate structural match. The paper itself invokes a "demand effect," but that is an explanation, not a test.

Other soft spots: no significance tests on the precision or recall differences; the AI sample is effectively very small because the 15 runs per cell show "little variation in output" and may be near-identical responses; and the abstract's high-recall claim applies only to the hint condition—without a hint, GPT-4 used no analogies at all. That no-hint result is interesting (LLMs apparently do not spontaneously retrieve analogies), but it is not about matching.

What the paper does well: the careful construction of two structurally distinct source–target pairs, the DAG-based coding scheme, and the honest presentation of the confusion matrices. The limitations section is transparent about sample and task simplicity, though it misses the response-format confound.

Is this worth a serious referee? Yes. The research question is important and the design has potential. But the current manuscript's headline conclusion is not supported. The fix is feasible: a forced-choice follow-up where the model must pick the single best analogy, or a scoring method that does not reward listing both. With that re-test, the asymmetry might still show up; right now, it is an artifact.

My recommendation: invite revision conditional on a forced-choice experiment, not accept the current evidence.

Referee Report

4 major / 6 minor

Summary. This paper compares analogical reasoning in business contexts between human participants (n = 199) and GPT-4 runs (n = 60) using an experimental design with two source stories (radiation problem; dolphins/survivorship bias) and two target problems (factory logistics; HR pilot study). It measures correct analogy use via DAG-based coding and computes precision, recall, F1, and accuracy from confusion matrices in no-hint and hint conditions. The paper reports that, in the hint condition, GPT-4 shows high recall but low precision, while humans show high precision but low recall, and concludes that the matching stage is where humans retain a decisive advantage over LLMs. The paper also reports an error-type analysis distinguishing surface- from structure-based misapplications and discusses implications for human–AI complementarity in strategic decision making.

Significance. If the reported results were robust, the paper would make a useful contribution by extending the classical analogical-transfer paradigm to a two-source, two-target matching setting in naturalistic business vignettes, and by providing one of the first direct human–LLM comparisons on the full retrieval–mapping–evaluation pipeline. The pre-specification of correct analogies via DAGs, the inclusion of error-type coding, and the transparent discussion of limitations and the pre-registered attention check are strengths. However, the measurement and inference issues described below currently prevent the headline claim from being supported, so the paper's significance depends on whether those issues can be resolved in revision.

major comments (4)

[§4.3, Tables 5 and 6] The central claim of a human matching advantage in the hint condition is not supported by the confusion-matrix counts because of a response-format confound. The AI counts (TP = 15, FP = 14 for Radiation; TP = 15, FP = 15 for Dolphin) are exactly what one would observe if every GPT-4 run listed both candidate source stories in its response; the per-claim scoring scheme used in the paper would then mechanically assign recall = 1.00 and precision ≈ 0.50 regardless of any real ability to distinguish structural from surface matches. Human participants, who typically report one story or none, automatically exhibit higher precision and lower recall under the same scoring. The paper acknowledges a 'demand effect' in §4.3 but does not control for it. A forced-choice protocol, or a scoring rule that considers only the participant's single best-supported match (or a top-1 precision metric), is needed before the data can be said to show that humans are better at matching.
[§3.4, Tables 5 and 6] The AI sample cannot be treated as 15 independent observations per condition. The paper states in §3.4 that there was 'little variation in the AI output to the prompt' and that 15 observations per condition were therefore deemed sufficient. If the model's sampling is near-deterministic at the chosen settings, the 15 runs are not independent draws, and the reported precision/recall values (e.g., 0.52/1.00 for Radiation with hint) may reflect a single response pattern rather than a stable property of the model. The manuscript does not report the temperature or sampling parameters, nor the number of distinct response patterns per cell. The authors should report the effective number of unique responses and, if necessary, run additional trials with higher stochasticity or treat the model as a fixed effect in the analysis.
[§4.3, Tables 4 and 6] The precision/recall comparisons are presented without any statistical tests or confidence intervals. Claims such as a 'decisive advantage' in the Discussion (§5) are not supported by formal inference. The paper should add tests appropriate to the data structure (each participant responds to one target problem under one hint condition, and AI runs are non-independent as noted above), or at minimum report exact binomial confidence intervals for the proportions underlying precision and recall. Without this, it is unclear whether the observed differences are within sampling variation.
[Abstract and §4.3, Table 7] The abstract and Discussion characterize GPT-4 as having 'high recall' as a general property, but the perfect recall (1.00) is observed only in the hint condition; in the no-hint condition, GPT-4's recall is 0.00 for both stories (Table 4). Table 7 makes this condition-dependence explicit. The paper should qualify the high-recall claim to the hint condition, or explain why the no-hint behavior should be disregarded for the theoretical argument about matching. Otherwise the claim as stated is misleading.

minor comments (6)

[§3.4] There is a typo: 'we man that' should read 'we mean that'.
[Table 4 note] Reporting 0/0 as zero is statistically questionable; it would be more appropriate to report the metric as undefined or to exclude that cell from the summary.
[§5, Limitations] The paper claims that intercoder reliability checks were used, but no reliability statistic (e.g., Cohen's kappa) is reported for the coding of correct analogical transfer or for the surface/structural error classification; please provide those values.
[References] The Miller and Lin (2015) reference appears twice with identical details; the duplicate should be removed.
[Figure 3] The y-axis label contains a typo: 'Distriubution' should be 'Distribution'.
[§4.2] The comparison of solvability with Gick and Holyoak (1983) is purely descriptive; consider adding a formal test or at least a caveat that the materials differ substantially between the studies.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's precision/recall results are empirical measurements, not derivations that reduce to their own inputs.

full rationale

The paper does not fit parameters and then rename them as predictions, nor does it invoke a load-bearing self-citation or import a uniqueness theorem from the authors' prior work. Its central empirical claim—that GPT-4 shows high recall and low precision while humans show high precision and low recall on an analogical matching task—is supported by coded response data and confusion matrices, not by definitional identity. The closest candidate for circularity is the concern that GPT-4's hint-condition precision is an artifact of the model tending to mention both candidate stories, which would mechanically produce recall near 1 and precision near 0.5. That is a measurement-validity and response-format critique, not circularity: the confusion-matrix counts are observations, and the conclusion is not entailed by the way the variables are defined. Likewise, the design's inclusion of a two-source/two-target matching step is an experimental manipulation, and observing that matching adds difficulty is an empirical result rather than a tautology. Self-citations in the paper (e.g., a footnote to Puranam & Sen 2019 and a mention of Choudhary et al. 2025) are not load-bearing for the main claim. The analysis is therefore self-contained against external benchmarks, and any weaknesses belong to construct validity or statistical independence of AI runs, not circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the validity of the author-constructed DAG coding scheme and the representativeness of GPT-4 runs; no free parameters or invented entities are involved.

assumptions (4)

domain assumption Analogical reasoning decomposes into retrieval, mapping, and matching/evaluation.
The paper's triadic model extends the canonical retrieval/mapping dyad; it is a theoretical framing assumed from Section 2.1 and used to interpret results.
ad hoc to paper The DAGs constructed by the authors capture the correct causal schemas for the source stories and target problems.
Section 3.5 states correct analogical transfer was pre-defined using author-built DAGs; if these DAGs are not the intended or only valid schemas, the correctness coding is invalid.
domain assumption Textual responses can be reliably coded for analogy use, correctness, and surface/structural error type.
Section 4.4 relies on qualitative coding per Holyoak and Koh definitions; limitations admit this involves qualitative judgment.
ad hoc to paper GPT-4 runs with 'little variation' can be treated as independent observations.
Section 3.4 deems 15 observations per condition sufficient despite little variation, but near-deterministic outputs mean effective sample size may be far below 60.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Can LLMs Help Improve Analogical Reasoning For Strategic Decisions? Experimental Evidence from Humans and GPT-4." pith.science (2026). https://pith.science/paper/O7AREJ4Z

@misc{pith2026250500603,
  author       = {Pith},
  title        = {Pith review of: Can LLMs Help Improve Analogical Reasoning For Strategic Decisions? Experimental Evidence from Humans and GPT-4},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O7AREJ4Z}},
  note         = {Machine review of arXiv:2505.00603}
}

read the original abstract

This study investigates whether large language models, specifically GPT4, can match human capabilities in analogical reasoning within strategic decision making contexts. Using a novel experimental design involving source to target matching, we find that GPT4 achieves high recall by retrieving all plausible analogies but suffers from low precision, frequently applying incorrect analogies based on superficial similarities. In contrast, human participants exhibit high precision but low recall, selecting fewer analogies yet with stronger causal alignment. These findings advance theory by identifying matching, the evaluative phase of analogical reasoning, as a distinct step that requires accurate causal mapping beyond simple retrieval. While current LLMs are proficient in generating candidate analogies, humans maintain a comparative advantage in recognizing deep structural similarities across domains. Error analysis reveals that AI errors arise from surface level matching, whereas human errors stem from misinterpretations of causal structure. Taken together, the results suggest a productive division of labor in AI assisted organizational decision making where LLMs may serve as broad analogy generators, while humans act as critical evaluators, applying the most contextually appropriate analogies to strategic problems.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

6 extracted references · 3 canonical work pages

[1]

Blanchette, I., & Dunbar, K. (2001). Analogy use in naturalistic settings: The influence of audience, emotion, and goals. Memory & Cognition, 29(5), 730-735. Camposampiero, G., Hersche, M., Wattenhofer, R., Sebastian, A., & Rahimi, A. (2025). Can Large Reasoning Models do Analogical Reasoning under Perceptual Uncertainty?. arXiv preprint arXiv:2503.11207....

arXiv 2001
[30]

S., Matlen, B

Vendetti, M. S., Matlen, B. J., Richland, L. E., & Bunge, S. A. (2015). Analogical reasoning in the classroom: Insights from cognitive science. Mind, Brain, and Education, 9(2), 100-106. Webb, T., Fu, S., Bihl, T., Holyoak, K. J., & Lu, H. (2023). Zero -shot visual reasoning through probabilistic analogical mapping. Nature Communications, 14(1),

work page 2015
[60]

B., & Gentner, D

Markman, A. B., & Gentner, D. (1993). Structural alignment during similarity comparisons. Cognitive psychology, 25(4), 431-467. Miller, K. D., & Lin, S. J. (2015). Analogical reasoning for diagnosing strategic issues in dynamic and complex environments. Strategic Management Journal, 36(13), 2000-2020. Miller, K. D., & Lin, S. J. (2015). Analogical reasoni...

work page 1993
[168]

M., & Eisenhardt, K

Santos, F. M., & Eisenhardt, K. M. (2009). Constructing markets and shaping boundaries: Entrepreneurial power in nascent fields. Academy of Management Journal, 52(4), 643-671. Sun, B., & Saenko, K. (2016). Deep coral: Correlation alignment for deep domain adaptation. In Computer vision–ECCV 2016 workshops: Amsterdam, the Netherlands, October 8 -10 and 15-...

work page 2009
[1103]

D., Gentner, D., & Law, K

Forbus, K. D., Gentner, D., & Law, K. (1995). MAC/FAC: A model of similarity‐based retrieval. Cognitive science, 19(2), 141-205. Gary, M. S., Wood, R. E., & Pillinger, T. (2012). Enhancing mental models, analogical transfer, and performance in strategic decision making. Strategic Management Journal, 33(11), 1229-1246. Gavetti, G., Levinthal, D. A., & Rivk...

arXiv 1995
[5144]

Yuan, S., Chen, J., Ge, X., Xiao, Y ., & Yang, D. (2023). Beneath surface similarity: Large language models make reasonable scientific analogies after structure abduction. arXiv preprint arXiv:2305.12660. Zollo, M., & Winter, S. G. (2002). Deliberate learning and the evolution of dynamic capabilities. Organization science, 13(3), 339-351

arXiv 2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Blanchette, I., & Dunbar, K. (2001). Analogy use in naturalistic settings: The influence of audience, emotion, and goals. Memory & Cognition, 29(5), 730-735. Camposampiero, G., Hersche, M., Wattenhofer, R., Sebastian, A., & Rahimi, A. (2025). Can Large Reasoning Models do Analogical Reasoning under Perceptual Uncertainty?. arXiv preprint arXiv:2503.11207....

arXiv 2001

[2] [30]

S., Matlen, B

Vendetti, M. S., Matlen, B. J., Richland, L. E., & Bunge, S. A. (2015). Analogical reasoning in the classroom: Insights from cognitive science. Mind, Brain, and Education, 9(2), 100-106. Webb, T., Fu, S., Bihl, T., Holyoak, K. J., & Lu, H. (2023). Zero -shot visual reasoning through probabilistic analogical mapping. Nature Communications, 14(1),

work page 2015

[3] [60]

B., & Gentner, D

Markman, A. B., & Gentner, D. (1993). Structural alignment during similarity comparisons. Cognitive psychology, 25(4), 431-467. Miller, K. D., & Lin, S. J. (2015). Analogical reasoning for diagnosing strategic issues in dynamic and complex environments. Strategic Management Journal, 36(13), 2000-2020. Miller, K. D., & Lin, S. J. (2015). Analogical reasoni...

work page 1993

[4] [168]

M., & Eisenhardt, K

Santos, F. M., & Eisenhardt, K. M. (2009). Constructing markets and shaping boundaries: Entrepreneurial power in nascent fields. Academy of Management Journal, 52(4), 643-671. Sun, B., & Saenko, K. (2016). Deep coral: Correlation alignment for deep domain adaptation. In Computer vision–ECCV 2016 workshops: Amsterdam, the Netherlands, October 8 -10 and 15-...

work page 2009

[5] [1103]

D., Gentner, D., & Law, K

Forbus, K. D., Gentner, D., & Law, K. (1995). MAC/FAC: A model of similarity‐based retrieval. Cognitive science, 19(2), 141-205. Gary, M. S., Wood, R. E., & Pillinger, T. (2012). Enhancing mental models, analogical transfer, and performance in strategic decision making. Strategic Management Journal, 33(11), 1229-1246. Gavetti, G., Levinthal, D. A., & Rivk...

arXiv 1995

[6] [5144]

Yuan, S., Chen, J., Ge, X., Xiao, Y ., & Yang, D. (2023). Beneath surface similarity: Large language models make reasonable scientific analogies after structure abduction. arXiv preprint arXiv:2305.12660. Zollo, M., & Winter, S. G. (2002). Deliberate learning and the evolution of dynamic capabilities. Organization science, 13(3), 339-351

arXiv 2023