Pith. sign in

REVIEW 4 major objections 6 minor 18 references

WHODUNIT: Evaluation benchmark for culprit detection in mystery stories

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read WhoDunIt, a new benchmark built from public-domain mystery stories, shows that GPT-4-family models solve unmodified texts well but lose accuracy when all character names are swapped and recover when names are replaced with famous figures.

desk verdict A small but real benchmark for narrative deduction, undermined by an abstract/body mismatch and an unvalidated name-swap procedure that the main comparisons depend on. read the letter →

arxiv 2502.07747 v1 pith:NKPF3R7B submitted 2025-02-11 cs.CL cs.AI

Kshitij Gupta This is my paper · ORCID
classification cs.CLcs.AI
keywords deductivereasoningmysterystoriesLLMevaluationbenchmarkculpritidentificationnamesubstitutionchain-of-thoughtpromptingself-reflectionGPT-4
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces WhoDunIt, a benchmark that asks large language models to read public-domain mystery stories and name the culprit. Its central goal is to see whether models truly reason through narrative clues or lean on memorized associations with famous story titles and character names. To test this, the authors alter character names in four ways—swapping all names, and replacing them with well-known fictional or celebrity names—and measure accuracy under four prompting styles. The reported results show strong baseline performance, a drop when every name is swapped, and recovery when names are replaced with familiar figures. If the benchmark works as intended, it gives the field a reusable way to separate memorization from deduction in narrative understanding.

What carries the argument

The benchmark's central object is a set of story-level name augmentation transforms applied to public-domain mystery texts: original names, a full permutation swap of all character names, replacement with well-known fictional characters, replacement with Hollywood celebrities, and replacement with Bollywood celebrities. These transforms are designed to break memorized name-to-story links while keeping plot and clue structure intact. The evaluation machinery is ten repeated model calls per configuration with majority-vote answer selection, across four prompting styles; culprit labels come from story-study notes or manual reading.

What would settle it

Run the same mystery set through two matched name-substitution conditions—one using widely known fictional or celebrity names and one using equally short, equally frequent but unfamiliar names—keeping story text otherwise identical. If accuracy is equal in both conditions, the paper's familiarity explanation fails; also inspect all augmented stories for leftover old-name references, initials, or title mismatches, since finding any such inconsistency would explain accuracy loss without invoking memory of famous names.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that GPT-4-family models can identify culprits in unaltered public-domain mystery stories with high accuracy, but that performance depends on how familiar character names are. The authors report that swapping all names caused a noticeable accuracy drop, while replacing names with well-known fictional or celebrity names restored accuracy to levels similar to, or above, the swapped condition, which they read as evidence that pre-training associations help the model. They also report that Chain-of-Thought and Self-Reflection prompting, alone or combined, improve accuracy over basic prompting. In the paper's results, the original-text condition produced the highest accuracy, and the body text describes the familiar-name conditions more favorably than the abstract's summary of diminished accuracy under wide-recognition substitutions.

Load-bearing premise

The load-bearing assumption is that name substitutions leave all clues, relationships, and character references intact, so any accuracy change comes from the names themselves rather than from contradictions introduced by the swap.

Editorial extensions

If this is right

  • Accuracy on unaltered mystery stories cannot by itself certify deductive ability; future benchmarks should include name-augmented control conditions to separate memorization from reasoning.
  • Chain-of-Thought and Self-Reflection prompting, alone or in combination, should be part of any standard evaluation protocol for narrative deduction tasks.
  • Model comparisons in this setting should control for document length, since the smallest tested model shows a steeper accuracy drop on longer stories.
  • A complete character-name swap is a simple, reusable robustness probe: a model that solves the unaltered story but fails the swap has not fully tracked the causal roles behind names.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If familiar replacement names genuinely help accuracy, then a sharper test would compare them against equally simple but unfamiliar made-up names matched for length and frequency; the familiarity effect should survive that control.
  • The paper's abstract and its Section 5.3 point in different directions for wide-recognition substitutions; if the body text is accurate, the defensible result is a swap penalty plus a familiarity benefit, not a general decline under familiar names.
  • Before relying on the augmentation results, a checker should verify that swapped or replaced names never conflict with initials, titles, aliases, or pronoun agreement in the story text; any such conflict would be a confound for the name-familiarity interpretation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces WHODUNIT, a dataset of public-domain mystery stories with identified culprits, and evaluates GPT-4o, GPT-4-turbo, and GPT-4o-mini on the task of naming the culprit. Four prompting styles and five text conditions (original, name-swap, Harry Potter, Hollywood, Bollywood) are tested with majority voting over ten trials. The authors report overall accuracies of 83.5%, 82.7%, and 74.1% for the three models, find that name-swapping hurts accuracy while familiar-name substitutions are less harmful, and conclude that prompting, especially Chain-of-Thought combined with reflection, helps. The dataset is released on HuggingFace.

Significance. If properly validated, the benchmark would be a useful resource for studying the interplay between memorization and contextual reasoning in long-narrative tasks, and the augmentation scheme is a plausible probe of reliance on name recognition. The paper's public-domain sources, labeled culprits from CliffNotes/manual reading, and use of majority voting are admirable. However, the current manuscript does not yet provide the evidence needed to support the headline claim: the augmentation procedure is not audited, experimental details are missing, and the abstract and body disagree on the main result. The under-specification and internal inconsistency mean the benchmark's validity and the paper's conclusions cannot yet be assessed.

major comments (4)
  1. [Abstract; §5.3] The abstract's claim that accuracy 'diminishes with certain name substitutions, particularly those with wide recognition' is in direct conflict with §5.3, which states that the Harry Potter, Hollywood, and Bollywood versions produced increased accuracy and that the model benefits from contexts associated with well-known entities. Please reconcile the two, and report the actual accuracy for each augmentation condition (with numbers) so the direction and size of the effect can be assessed.
  2. [§3 (augmentation procedure); Appendix A.1, A.2] The augmentation method is not specified beyond 'replace all character names', and no check is provided that the replaced texts preserve all clues. For 'A Case of Identity', the culprit James Windibank uses the alias Hosmer Angel; for 'Silver Blaze', John Straker is also called 'Straker'. If replacements do not apply to aliases and partial names, the augmented text will contain contradictory identity information, and any accuracy drop could be due to corrupted clues rather than the intended manipulation. Please provide the replacement code/rule, a manual or automatic audit of all augmented stories for alias/coreference/pronoun consistency, and report how many stories had to be corrected.
  3. [§4 (Experimental Setup), §5 (Results)] Critical experimental details are missing: the exact number of stories in the dataset, the prompt templates, the answer-parsing/scoring procedure, and how the '10-shot' majority vote is defined (10 examples vs. 10 trials). Without these, the reported accuracies of 83.5%, 82.7%, and 74.1% cannot be reproduced, and the figures (2–5) do not give per-condition error bars. Please include a full results table with per-model, per-augmentation, per-prompt accuracy and standard errors, and describe the evaluation protocol in enough detail for replication.
  4. [§3, §5.3 (memorization confound)] The authors acknowledge that models have likely memorized these public-domain stories and blog posts about them, but their name substitutions do not conclusively remove the memorization channel: a full plot, even with swapped names, can uniquely identify a famous story and let the model retrieve the culprit from a memorized summary. Please add a control experiment, e.g., asking the model to identify the original title/author from an augmented text or to answer 'who is the culprit?' on a redacted plot summary that omits names, to measure the residual memorization effect.
minor comments (6)
  1. [§3] The paper says it obtained a list of 500+ titles but never states how many stories are actually in WHODUNIT; please give the final count and a breakdown by author/length.
  2. [§4.1] The phrase '10-shot prompting' is ambiguous; clarify whether it means 10 in-context examples or 10 sampled model calls (the latter is what '10 independent calls' in Section 5 suggests).
  3. [Figures 2–5] Figures 2–5 lack axis labels and units; Figure 4 should be accompanied by a table of numerical accuracies since the text references values not visible in the plot.
  4. [Appendix A.3] Appendix A.3 lists 'Silver Blaze' both as a Novel and as a Short Story; this inconsistency, and the general story-type (novel vs. short story) labels, should be corrected.
  5. [Footnote 7] Footnote 7 states that code and results are available on GitHub but no URL is provided; please include the link or state the repository identifier.
  6. [References] The Vaswani (2017) entry lacks the full proceedings title/volume, and the Huang and Sutskever (2023) interview entry omits the full title; please complete these.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the benchmark compares external labels against model outputs and no claim reduces to its inputs.

full rationale

WHODUNIT is an empirical benchmark paper rather than a derivation, so the standard circularity patterns do not apply. The dataset labels are obtained from external sources (Project Gutenberg texts and Cliffnotes, supplemented by manual reading), and model accuracy is measured against those labels; there is no fitted parameter, self-citation chain, or uniqueness theorem that forces the reported accuracies. The paper explicitly acknowledges the main validity threat, that public-domain stories are likely in the training data, and attempts to address it through name augmentations; this is a memorization confound, not a circular reduction, and is therefore a correctness risk rather than a circularity finding. The skeptical concern that find-and-replace name swaps may break coreference or alias consistency (e.g., 'A Case of Identity' and 'Silver Blaze') is also an empirical validity threat to the augmentation comparison in Section 5.3, not evidence that any result is equivalent to its inputs by construction. The inconsistency between the abstract's claim that wide-recognition substitutions diminish accuracy and Section 5.3's statement that HP, Hollywood, and Bollywood versions increased accuracy is an internal consistency problem, not circularity. No circular step can be quoted because none of the paper's claimed findings are defined in terms of the outputs they predict.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the accuracy of the ground-truth labels, the preservation of story logic under name substitution, and the reliability of majority voting. These are domain assumptions rather than derived results; no new theoretical entities are introduced.

free parameters (1)
  • trial_count = 10
    Number of independent model calls per condition used for majority voting; chosen without an ablation study, and no variance across trials is reported.
assumptions (5)
  • domain assumption Culprit labels from Cliffnotes and self-reading are correct for all stories.
    Section 3 states labels come from Cliffnotes or the author's own reading, with no independent verification or inter-annotator agreement.
  • domain assumption Public-domain mystery stories are present in LLM pretraining corpora, making memorization a plausible confound.
    Section 3 acknowledges this and motivates the augmentations, but does not measure the degree of memorization.
  • domain assumption Majority voting over 10 trials gives a stable estimate of model accuracy.
    Section 4.1 uses 10-shot prompting with majority selection, but no analysis of inter-trial variance or confidence intervals is provided.
  • domain assumption Character-name substitution preserves the narrative logic and the identifiability of the culprit.
    Section 3 describes the augmentations but does not verify that clues like initials, titles, or aliases are consistently updated in the substituted text.
  • domain assumption Model outputs are parsed correctly to determine whether the predicted culprit matches the label.
    The paper does not describe the answer-extraction or scoring procedure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of WHODUNIT: Evaluation benchmark for culprit detection in mystery stories." pith.science (2026). https://pith.science/paper/NKPF3R7B

@misc{pith2026250207747,
  author       = {Pith},
  title        = {Pith review of: WHODUNIT: Evaluation benchmark for culprit detection in mystery stories},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NKPF3R7B}},
  note         = {Machine review of arXiv:2502.07747}
}
read the original abstract

We present a novel data set, WhoDunIt, to assess the deductive reasoning capabilities of large language models (LLM) within narrative contexts. Constructed from open domain mystery novels and short stories, the dataset challenges LLMs to identify the perpetrator after reading and comprehending the story. To evaluate model robustness, we apply a range of character-level name augmentations, including original names, name swaps, and substitutions with well-known real and/or fictional entities from popular discourse. We further use various prompting styles to investigate the influence of prompting on deductive reasoning accuracy. We conduct evaluation study with state-of-the-art models, specifically GPT-4o, GPT-4-turbo, and GPT-4o-mini, evaluated through multiple trials with majority response selection to ensure reliability. The results demonstrate that while LLMs perform reliably on unaltered texts, accuracy diminishes with certain name substitutions, particularly those with wide recognition. This dataset is publicly available here.

Figures

Figures reproduced from arXiv: 2502.07747 by the authors.

Figure 1
Figure 1. , the stories vary in length, covering short, medium and full narratives, providing a broad spec￾trum of text. By including works from different writers and narrative traditions, we ensure that the models encounter a variety of narrative structures, reasoning styles, and linguistic expressions used to describe mystery and crime [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Accuracy distribution across the number of [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 2
Figure 2. Accuracy comparison across models 5.2 Impact of Document Length on Model Accuracy [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Accuracy across different prompting tech [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 3 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Prithviraj Ammanabrolu, Wesley Cheung, William Broniec, and Mark O Riedl. 2021. Automated storytelling via causal, commonsense plot ordering. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 5859--5867

    work page 2021

  3. [3]

    Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165

    arXiv 2020

  4. [4]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    arXiv 2020

  5. [5]

    Jensen Huang and Ilya Sutskever. 2023. Interview with jensen huang and ilya sutskever. https://www.youtube.com/watch?v=GI4Tpi48DlA

    work page 2023

  6. [6]

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, page 2. Minneapolis, Minnesota

    2019

  7. [7]

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110

    arXiv 2022

  8. [8]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36

    2024

Show all 18 references
  1. [9]

    OpenAI. 2022. Introducing chatgpt. https://openai.com/index/chatgpt/

    2022

  2. [10]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog

    2019

  3. [11]

    Hannah Rashkin, Asli Celikyilmaz, Yejin Choi, and Jianfeng Gao. 2020. Plotmachines: Outline-conditioned generation with dynamic plot state tracking. arXiv preprint arXiv:2004.14967

    2020 arXiv

  4. [12]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36

    2024

  5. [13]

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv p...

    2022 arXiv

  6. [14]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems

    2017

  7. [15]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

    2022 arXiv

  8. [16]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  9. [17]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  10. [18]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.