{"id":"7173c506-02fc-4c28-aa16-953bfc7f6d8e","arxiv_id":"2505.07601","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A multi-LLM workflow extracted and synthesized distinctive investigative trait profiles for seven fictional detectives, achieving 91.43% reverse-identification accuracy across the same models.","lead":"This paper uses fifteen AI language models to list the investigative habits of seven famous fictional detectives, then tests whether the models can name each detective from those traits alone. It reports 91.43% accuracy and proposes the approach as a scalable way to profile characters for AI-driven storytelling.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 91.43% reverse-identification accuracy is a self-consistency result: the same 15 LLMs that generated the trait descriptions are the ones asked to recognize them, so the central effectiveness claim is not externally validated.","rationale":"I agree with the reader's weakest assumption: the reverse-identification validation is circular because the generating and evaluating models share the same training knowledge of these canonical detectives. This is the load-bearing point because the paper's abstract and Section 4.2 explicitly use the 91.43% accuracy as evidence of 'effectiveness.' The issue is not that the profiles are worthless; the pipeline is transparent, all prompts are provided, and the trait lists largely match familiar critical descriptions. But the evaluation design cannot distinguish a faithful extraction of distinctive methods from a self-consistent paraphrase of common knowledge. The proposed leave-one-out test would settle this by checking whether profiles built without a given model can still be recognized by that model. If they can, the original result is more credible; if not, the claim should be downgraded to 'self-consistent trait extraction.' Therefore I keep the reader's conditional verdict unchanged.","tokens_in":15862,"tokens_out":5497,"duration_ms":57522,"concrete_test":"Leave-one-model-out cross-validation: for each of the 15 LLMs, regenerate all detective descriptions using only the other 14 models, redo trait extraction and semantic grouping with a model other than GPT-4o, and then have the held-out model perform reverse identification alone. Compare the held-out model's accuracy with the reported 91.43% (Table 2). If the held-out accuracy is substantially lower (e.g., below 75%) or close to the prior-based accuracy for identifying seven famous detectives, the headline result reflects self-consistency rather than transferable distinctiveness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the workflow 'demonstrat[es] the method's effectiveness in capturing the distinctive investigative approaches of each detective' (Abstract; Section 4.2). That claim rests on the reverse-identification accuracy of 91.43% (Section 3.6, Table 2). This result is not an independent validation. The 15 LLMs that perform the identification are the same model set that generated the source descriptions in Section 3.2, and GPT-4o alone performed trait extraction (Section 3.3) and semantic grouping (Section 3.4). The test is a 7-way forced choice among extremely famous detectives whose methods are heavily represented in these models' training data, with the candidate names supplied in the prompt (Prompt 4). High agreement therefore shows that the pipeline preserves the models' prior consensus, not that the extracted traits are externally accurate or distinctive. The consistency threshold of 20% (Section 3.5) is arbitrary and low (3 of 15 models), and the comparison to literary analyses in Section 4.1 is a qualitative, post-hoc alignment with selected references rather than a systematic evaluation. Without a held-out evaluator, a non-informative control, or human/external ground truth, the 91.43% figure does not support the strong claim that the profiles reliably capture each detective's investigative method.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-phase LLM workflow to characterize the investigative methods of seven fictional detectives: 15 LLMs independently generate short descriptions; GPT-4o extracts candidate traits and semantically groups them; traits are filtered by cross-model consensus at a 20% threshold; and the resulting profiles are evaluated by asking the same 15 LLMs to identify the detective from the trait list. The authors report 91.43% overall reverse-identification accuracy and a qualitative alignment of the trait profiles with literary criticism, interpreting these as evidence that the method reliably captures distinctive investigative approaches for reuse in narrative generation.","tokens_in":16117,"tokens_out":2617,"duration_ms":27088,"significance":"If the central claim held, the paper would supply a scalable, reusable protocol for character profiling in computational narratology, with a clearly documented prompt set, a concrete pipeline, and full trait lists in the appendix. The authors are transparent about their prompts, model versions, temperature settings, and the confusion matrix, which supports reproducibility. The main weakness is that the headline validation is circular: the same models that generate the descriptions are the ones asked to recognize them, and the literary comparison is post-hoc and qualitative. These issues are load-bearing for the claim that the profiles capture externally valid, distinctive investigative methods, so the manuscript needs additional validation before the effectiveness claim can be accepted.","major_comments":[{"comment":"The reverse-identification test uses the same 15 LLMs that generated the descriptions in Section 3.2, with GPT-4o also performing trait extraction (Section 3.3) and semantic grouping (Section 3.4). High accuracy therefore largely measures self-consistency within the ensemble: the models are recognizing trait lists derived from their own outputs. The phrase in Section 4.2 that this 'demonstrat[es] the method's effectiveness' overstates what the test can establish. This is not an independent validation of distinctiveness or external accuracy. I recommend adding a held-out evaluator (e.g., human raters or a different set of models not involved in generation), a non-informative control condition, or an external ground-truth comparison to support the claim.","section":"Section 3.6 and Section 4.2, Prompt 4 and Table 2"},{"comment":"The 20% consistency threshold is introduced without justification and is quite permissive: a trait is retained if only 3 of the 15 models independently contributed it to the group. Because this threshold directly determines which traits enter the final profiles and thus affects the downstream reverse-identification results, the paper should provide a sensitivity analysis showing how the threshold affects both the synthesized profiles and the reported accuracy. Without such an analysis, the central quantitative result depends on an arbitrary free parameter.","section":"Section 3.5, consistency threshold"},{"comment":"The claimed 'validation against existing literary analyses' is a qualitative, post-hoc alignment with selected sources rather than a systematic evaluation. For each detective, the paper finds references that match the extracted traits, including sources likely present in the LLMs' training data, so agreement with these references does not distinguish a genuine extraction from memorized common critical commentary. A pre-registered or independent-annotation protocol (e.g., human raters judging whether each trait is distinctive and accurate for the detective) would be needed to support the validation claim.","section":"Section 4.1, literary comparison"}],"minor_comments":[{"comment":"The abstract and Section 4.1 state that traits were 'validated against existing literary analyses,' but the paper presents no formal validation metric or protocol for that comparison; consider softening the wording to 'compared with' or 'consistent with selected literary analyses.'","section":"Abstract and Section 4.1"},{"comment":"The phrase 'indirect evaluation of the clarity, uniqueness, and informativeness' attributes three properties to the profiles, but the reverse-identification task only directly tests whether the models can map the trait list to the correct name; uniqueness and informativeness are not separately measured.","section":"Section 3.6"},{"comment":"There is a typo in 'the 15 LLMs presented in Section3.6' — a missing space before the section number.","section":"Section 4.2"},{"comment":"Reference [2] is incomplete: the author field appears as 'T. B. and' with no full name or title for the first author; this should be corrected.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The circularity concern is the main obstacle: the paper's headline result is a self-consistency score. However, the pipeline is clearly described and the underlying idea has value as a proof of concept. A major revision that adds a non-circular validation component, a sensitivity analysis for the threshold, and a more careful framing of the claims would bring it to an acceptable level. The literary-comparison section is currently too selective to function as validation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper describes a five-phase LLM pipeline: 15 models write short descriptions of seven fictional detectives, GPT-4o extracts and semantically groups traits, a 20% consensus threshold filters them, and then the same 15 models are asked to name each detective from the trait list. The headline is 91.43% reverse-identification accuracy. The work is clearly written, the prompts are included, and the appendix gives the full trait lists. Credit where it's due: the methodology is transparent, the detective sample is sensible, and the misclassification analysis, including the DeepSeek R1 reasoning trace, is honest and informative. The trait profiles themselves align well with established literary scholarship, which is reassuring but not novel.\n\nThe soft spot is exactly where the stress test lands. The reverse identification test is circular in a real sense: the same 15 LLMs that generated the source descriptions are the ones doing the recognition, with the candidate names supplied in the prompt. These are seven iconic detectives heavily represented in the models' training data, and the task is a 7-way forced choice. So 96/105 correct mostly shows that the pipeline preserves the models' prior consensus about these characters. That is not nothing—for an LLM-based narrative generation system, a profile that the ensemble can recognize is arguably usable—but it does not support the abstract's claim that the method's effectiveness in capturing distinctive investigative approaches has been demonstrated. There is no baseline (random traits, a held-out evaluator, or human judgment), no sensitivity analysis for the 20% threshold, and the literary comparison is qualitative post-hoc alignment. The threshold is also low: three of fifteen models are enough to keep a trait. These issues are not fatal to the workflow, but they are load-bearing for the validation claim.\n\nBottom line: this is a reasonable, incremental contribution to computational narratology. A serious referee should engage it because the pipeline is well specified and the main flaw is fixable. I would ask for an external human or held-out evaluation, a control condition, threshold sensitivity analysis, and release of intermediate data. As it stands, treat the 91.43% as a self-consistency metric, not proof of external validity.","headline":"A transparent multi-LLM workflow for detective trait profiles, but the 91.43% accuracy is ensemble self-consistency rather than external validation; worth a serious referee, not a desk reject.","tokens_in":16621,"tokens_out":2618,"would_cite":false,"duration_ms":27114,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A five-phase LLM workflow derives consensus trait profiles for seven fictional detectives, and reverse identification succeeds in 91.43% of trials.","keywords":["detective fiction","computational narratology","large language models","investigative traits","reverse identification","character analysis","narrative generation","LLM ensemble"],"falsifier":"A concrete check is to run the reverse-identification test on a fresh set of detectives that the fifteen models did not help profile — for example, two detectives created from the workflow's own trait lists with labels withheld — and see whether accuracy stays near 91 percent; if it collapses, the original score was measuring memorized canon rather than trait distinctiveness. A sharper version would present the same trait lists with one trait removed at a time and measure how much each trait contributes to correct identification.","tokens_in":15679,"feed_emoji":"🕵️","tokens_out":7660,"duration_ms":67884,"temperature":0.7,"pith_summary":"This paper claims that a multi-phase workflow built from fifteen large language models can systematically capture the distinctive investigative methods of fictional detectives and compress them into reusable trait profiles. The workflow was run on seven iconic detectives — Hercule Poirot, Sherlock Holmes, William Murdoch, Columbo, Father Brown, Miss Marple, and Auguste Dupin — and the resulting profiles were checked against existing literary analyses. The authors then reversed the process: each profile was shown to all fifteen models, which had to name the detective from the trait list alone, and 91.43% of those identifications were correct. If the claim holds, computational narratology gains a scalable way to turn character interpretation into structured data that story-generation systems can reuse.","feed_headline":"Trait profiles let LLMs name 7 detectives 91% of the time","feed_subtitle":"A five-phase workflow turns 15 models' descriptions into consensus trait lists that pass a reverse identification test.","key_machinery":"The load-bearing mechanism is a five-phase LLM ensemble workflow. Each of fifteen models independently writes a constrained five-sentence description of a detective's method; a single extraction model turns each description into a bullet list of traits; a grouping model merges semantically equivalent traits across models; a consensus threshold keeps only trait groups supported by at least 20 percent of the models (three of fifteen); and the surviving trait list is fed back to all fifteen models, which must identify the detective from the list alone. The reverse-identification phase is the step that makes distinctiveness measurable rather than merely asserted.","core_discovery":"On the paper's own terms, the central discovery is that the multi-model consensus process yields trait lists specific enough for models to recover the detective's identity from the traits alone. Across 105 reverse-identification trials, 96 were correct. Four detectives — Poirot, Columbo, Father Brown, and Miss Marple — were identified perfectly by all fifteen models; Sherlock Holmes and William Murdoch each had one misclassification; Auguste Dupin was the weak point, identified correctly only 8 times out of 15, with most errors placing him as Sherlock Holmes, a confusion the paper attributes to Dupin's documented role as the prototype for Holmes. The authors read this pattern as evidence that the profiles capture real investigative styles, including their historical overlaps.","pith_inferences":["The reverse-identification test would be stronger if run against a control set of non-canonical or lesser-known detectives, because the seven chosen characters are heavily represented in LLM training data; a high score could then be partly recognition rather than distinctiveness.","Stability of the profiles across different LLM panels is testable: repeating the workflow with a different set of models would show whether the consensus traits drift, and drift would indicate which traits are model-dependent rather than character-dependent.","The semantic grouping step deliberately generalizes specific reasoning terms, so the final profiles can lose the vocabulary of reasoning type (for example, abduction disappears under 'logical reasoning'); a profile intended for narrative generation might want to retain such distinctions.","A direct extension would use these profiles as conditioning input in a story generator and measure whether readers can identify which detective is acting, turning the reverse-ID test from a model self-check into a reader-facing evaluation."],"forward_implications":["The synthesized trait profiles are reusable character models: a narrative generation system could use them to keep a detective acting in character across a generated story.","The workflow is prompt-generic, so the same five phases can be pointed at other detectives, at non-detective roles such as criminals or victims, or at characters in other genres.","Because the consensus filter drops traits supported by only one model, the final profiles represent cross-model agreement rather than any single model's idiosyncratic reading.","The Dupin/Holmes confusion suggests that near-neighbour detective pairs will be the limiting case of the method, and that is where profile refinement would matter most.","The reported 91.43% accuracy is a property of this seven-detective set; adding more detectives would make the identification task harder and test the profiles' discriminative power more severely."],"supporting_citations":[{"why":"Supplies the literary analysis of Poirot and Marple used to confirm that the extracted traits match scholarly characterization.","marker":"[37]"},{"why":"Supplies detective archetypes and the account of Father Brown's moral-intuitive method used as validation ground truth.","marker":"[32]"},{"why":"Supplies the analysis of Columbo's rhetorical questioning technique that the extracted Columbo traits are checked against.","marker":"[7]"},{"why":"Supplies Bargainnier's description of Miss Marple's method of analogy and observation used in validating her profile.","marker":"[6]"},{"why":"Supplies Grella's characterization of Dupin's dual poetic-mathematical temperament used to validate the Dupin traits.","marker":"[29]"},{"why":"Provides the abduction and ratiocination framework connecting Dupin and Holmes, used to explain their shared reasoning traits and the main misclassification.","marker":"[24]"},{"why":"Documents the historical influence of Dupin on the creation of Holmes, used to explain why Dupin is most often misidentified as Holmes.","marker":"[45]"},{"why":"Establishes computational narratology as the motivating framework for turning literary character analysis into narrative generation input.","marker":"[11]"}],"fun_headline_variants":["LLM trait profiles identify 7 detectives at 91% accuracy","Reverse ID test: LLMs name detectives from traits 91% correctly","15 LLMs reach 91% accuracy on detective identification","Detective traits by LLMs pass reverse ID test at 91%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an LLM correctly naming a detective from a trait list is evidence that the traits are distinctive and accurate, even though the same models generated those traits from their own training knowledge of the same seven detectives.","fun_headline_variants_meta":{"raw":{"variants":["LLM trait profiles identify 7 detectives at 91% accuracy","Reverse ID test: LLMs name detectives from traits 91% correctly","15 LLMs reach 91% accuracy on detective identification","Detective traits by LLMs pass reverse ID test at 91%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1368,"prompt_tokens":937,"completion_tokens":431,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":356}},"tokens_in":553,"tokens_out":431,"duration_ms":4659,"temperature":1.0,"reasoning_tokens":356,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:11:51.245293+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check is to run the reverse-identification test on a fresh set of detectives that the fifteen models did not help profile — for example, two detectives created from the workflow's own trait lists with labels withheld — and see whether accuracy stays near 91 percent; if it collapses, the original score was measuring memorized canon rather than trait distinctiveness. A sharper version would present the same trait lists with one trait removed at a time and measure how much each trait contributes to correct identification.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the literary analysis of Poirot and Marple used to confirm that the extracted traits match scholarly characterization."},{"cited_title":"Ketovi´c","cited_arxiv_id":null,"evidence_quote":"Supplies detective archetypes and the account of Father Brown's moral-intuitive method used as validation ground truth."},{"cited_title":"Berzsenyi","cited_arxiv_id":null,"evidence_quote":"Supplies the analysis of Columbo's rhetorical questioning technique that the extracted Columbo traits are checked against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Bargainnier's description of Miss Marple's method of analogy and observation used in validating her profile."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Grella's characterization of Dupin's dual poetic-mathematical temperament used to validate the Dupin traits."},{"cited_title":"Eco and T","cited_arxiv_id":null,"evidence_quote":"Provides the abduction and ratiocination framework connecting Dupin and Holmes, used to explain their shared reasoning traits and the main misclassification."},{"cited_title":"van Laethem","cited_arxiv_id":null,"evidence_quote":"Documents the historical influence of Dupin on the creation of Holmes, used to explain why Dupin is most often misidentified as Holmes."},{"cited_title":"Cavazza and D","cited_arxiv_id":null,"evidence_quote":"Establishes computational narratology as the motivating framework for turning literary character analysis into narrative generation input."}],"review_version":1}