{"id":"14fd3f0b-8578-45a7-8bac-9794b7f6c1d8","arxiv_id":"2506.10231","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Experts annotated 817 short first-person narratives for narrator unreliability, and LLMs classified them with modest macro-F1 scores: best on verbal-tic detection, worst on abstract character-trope matching.","lead":"This paper introduces a new task, dataset, and set of LLM benchmarks for identifying unreliable narrators in first-person text, whether the unreliability appears as verbal tics, contradictions with other characters, or matching literary character tropes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim assumes expert annotators' notion of unreliability transfers to ordinary readers of real-world texts; the paper reports inter-expert kappa but no evidence of construct validity, so reported F1 scores may measure agreement with an annotation artifact.","rationale":"The reader's weakest_assumption is construct validity, and that is the concern I find most load-bearing. I considered the missing majority baseline and the inter-narrational prompt inconsistency; both are real and worth fixing, but they affect the magnitude and interpretation of specific numbers rather than the identity of what is being measured. The construct-validity question sits upstream: if the labels do not correspond to a property that readers of real-world text perceive, then even a perfect F1 against those labels would not support the paper's central application claim. The paper's own Section 3 assumption makes the reader the arbiter, but the annotators are experts; no evidence is provided that the two coincide. This is a testable empirical claim, not a philosophical objection. My recommendation is unchanged CONDITIONAL: the resource is valuable and honestly documented, but it should be accepted with the condition that the construct-validity check be run or the claims re-scoped to expert-defined unreliability rather than reader-perceived unreliability in everyday text.","tokens_in":29006,"tokens_out":10504,"duration_ms":133248,"concrete_test":"Recruit a stratified sample of 30 texts from each of Blog, Subreddit, and Review, plus 30 from the Fiction test set. Have three non-expert annotators per text (not English-literature majors) apply the exact annotation instructions from Appendix B.1, and also rate trust in the narrator on a 5-point scale. Compute (i) pairwise Cohen's kappa among non-experts per task, and (ii) agreement (macro-F1 or kappa) between non-expert labels and the resolved expert gold labels. If non-expert kappa is below about 0.4 or non-expert/expert agreement is at chance for the inter-narrational and inter-textual tasks, the gold labels are an expert artifact and the real-world claims need re-scoping; if non-experts reproduce the expert labels substantially, the construct-validity concern is retired.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's operating definition (Section 3) is that a narrator is unreliable when 'the reader notices explicit or implicit information indicating unreliability.' The dataset is annotated only by English-literature experts using a taxonomy built from literary fiction (Hansen 2007; Riggan 1978). The reported kappas (0.71-0.75) establish that experts can apply the annotation manual consistently, but not that the labels track what ordinary readers of hotel reviews, Reddit posts, or blogs perceive as unreliable narration. This matters because the paper's motivation and abstract are explicitly about real-world texts and the claim of 'potential for using LLMs to identify unreliable narrators.' If non-expert readers do not spontaneously recognize the expert labels---especially the inter-narrational and inter-textual categories, which require abstract inferences about tropes---then every macro-F1 in Tables 3-5 measures agreement with an expert-convention artifact, and the transfer-to-real-world conclusion is unsupported. The paper itself flags no validation of this kind; Section 8 lists only length, language, gender, and dataset-size limitations. This is not to say the dataset is useless: an expert-defined task can still be coherent. But the central claim is about potential for identifying unreliable narrators in ordinary text, so construct validity is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TUNA, a new expert-annotated dataset of first-person narratives drawn from fiction, blog posts, subreddit posts, and hotel reviews, labeled for three types of narrator unreliability (intra-narrational, inter-narrational, inter-textual) borrowed from the narratology of Hansen (2007) and Riggan (1978). It defines binary and multi-class classification tasks and evaluates several open-weight and proprietary LLMs under zero-shot, few-shot, LoRA fine-tuning, and curriculum learning. The core claims are that the dataset provides reliable expert labels (Cohen's kappa 0.71-0.75), that the tasks form a difficulty gradient from verbal tics to abstract trope matching, that training on fiction can transfer to real-world domains with macro-F1 up to about 57, and that fine-tuned models generally outperform zero-shot prompting, with curriculum learning often beating plain fine-tuning.","tokens_in":29186,"tokens_out":4553,"duration_ms":57776,"significance":"If the dataset and task definition are valid, this is a novel and potentially useful resource: it is, to the authors' knowledge, the first dataset and experimental benchmark for automated unreliable-narrator detection, and the annotation protocol is careful, with expert annotators, reported agreement, disagreement resolution, and fully disclosed prompts and code. The main weakness is that the paper's central motivation is about real-world readers and real-world texts, but the labels are produced by English-literature experts applying a literary-fiction taxonomy, with no validation that ordinary readers perceive the annotated phenomena. The reported F1 scores may therefore measure agreement with an expert-convention artifact rather than a property that transfers to naturalistic use. The paper also overstates several experimental conclusions that its own tables do not support.","major_comments":[{"comment":"Construct validity is load-bearing for the abstract and introduction's real-world framing. Section 3 assumes that 'given a text, it contains explicit or implicit information that can be leveraged to ascertain the narrator's unreliability' and that a narrator is reliable 'until the reader notices' unreliability. Yet the dataset is annotated solely by English-literature experts applying Hansen's and Riggan's literary taxonomy, and Section 8 lists only length, language, gender, and dataset-size limitations, with no check that non-expert or ordinary readers perceive the same unreliability. If expert labels do not generalize to the readers of blogs, reviews, and Reddit posts who motivate the task, every macro-F1 in Tables 3-5 measures agreement with an annotation artifact. Please add a validation study with non-expert readers, or at minimum report agreement between expert labels and an independent set of lay annotations and temper the real-world transfer claims accordingly.","section":"Section 3, Section 3.4, Section 8"},{"comment":"The blanket claim that 'methods using training samples (i.e., CL, fine-tuning, few-shot) outperform the zero-shot method' is contradicted by the paper's own tables. In Table 3, for Llama3.1-8B the zero-shot inter-narrational Subreddit score is 30.97, higher than CL's 30.91 and fine-tuning's 25.59; the inter-textual Blog score is 24.23 for zero-shot versus 19.58 for CL and 18.99 for fine-tuning. In Table 4, Llama3.3-70B zero-shot intra-narrational (54.20) and inter-textual (28.52) exceed both CL and fine-tuned scores. The claim should be restricted to specific models, tasks, and domains, or replaced by a systematic significance comparison across all conditions.","section":"Section 5.1, Tables 3-4"},{"comment":"The claim that curriculum learning outperforms fine-tuning 'for most cases' is not supported by Table 3. For example, inter-narrational Fiction (34.59 CL vs. 34.63 FT), inter-narrational Review (35.29 vs. 36.59), inter-textual Fiction (27.42 vs. 28.59), and inter-textual Review (16.72 vs. 17.54) all show fine-tuning at least as good as CL. Because the table reports row-wise significance comparisons only against columns to the right, it is unclear whether the CL-vs-FT differences are statistically reliable; please report the relevant paired comparisons and adjust the conclusion accordingly.","section":"Section 5.1, Table 3"},{"comment":"The evaluation prompt for inter-narrational unreliability adds a condition not present in the task definition: '<B>: Other character contradiction: Another character contradicts narrator who has demonstrated at least one form of intra-narrational unreliability.' Section 3.2 defines other-character contradiction without requiring prior intra-narrational unreliability, and the annotator instructions in Appendix B.1 do not include this requirement either. Since the annotation labels and the LLM prompts may therefore operationalize different definitions, the reported F1 scores could understate or overstate model performance in a way that is not attributable to the task. Please justify this extra condition or remove it, and verify that annotation and evaluation definitions coincide.","section":"Appendix D.3.3 vs. Section 3.2"}],"minor_comments":[{"comment":"In the task definition for inter-narrational unreliability, '<A>' is described as 'same reliable character over time', but Section 3.2 and Figure 2 call this 'same unreliable narrator over time'; the Section 4.1 wording appears to be a typo and should be corrected.","section":"Section 4.1"},{"comment":"The caption states 'For inter-tex unreliabilities: ... (C) → none', but the table's inter-textual columns are (A) through (E), and the 'none' class is (E). The caption label should be corrected to avoid ambiguity.","section":"Table 9 caption"},{"comment":"The phrasing 'out-of-domain performances ... are not better but good compared to in-domain performances' is unclear; the intended meaning appears to be 'not worse' or 'comparable', and the sentence should be rewritten to state the direction and magnitude of the comparison precisely.","section":"Section 5.1, takeaway 4"},{"comment":"The paper reports Cohen's kappa as a single value per task but does not explain whether this is the average over annotator pairs, a Fleiss-style multirater variant, or a value from a particular pair. Please specify the exact computation method.","section":"Section 3.4"}],"recommendation":"major_revision","confidential_remarks":"The dataset and task are novel and the annotation effort is real, but the manuscript's central real-world transfer claim rests on an unvalidated construct. The experimental overclaims in Section 5.1 are fixable by rewriting and adding targeted statistical comparisons, but the construct-validity issue needs a concrete validation experiment before the paper can support its stated motivation. I do not see this as a reject: the resource itself is useful, and the authors have already released code and detailed appendices, so the requested revision is within scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Jotting this down after reading arXiv:2506.10231. The short version: this is a genuinely new resource paper. TUNA is the first labeled dataset for unreliable narrator classification, built with care — ten English-literature experts, Cohen's kappa 0.71–0.75, disagreements resolved, full prompts and checkpoints in the appendix. The task decomposition into intra-, inter-narrational, and inter-textual unreliability comes straight from Hansen and Riggan and is a sensible operationalization. I believe the authors when they say no prior automated work exists; the related work on character understanding is adjacent but not on point. The evaluation is a clean held-out comparison against human labels. That is real and useful.\n\nThe soft spots are real but mostly mechanical. First, there is no majority-class baseline anywhere. The label distributions are heavily skewed (e.g., 3 vs 63 in some review classes), and macro-F1 without a majority baseline makes it hard to know whether a score like 19.30 on inter-textual is meaningful. A simple always-majority baseline would settle that. Second, the paper's second takeaway — 'methods using training samples outperform zero-shot' — is contradicted by its own Table 4 for Llama3.3-70B, Mistral-7B, and Phi3-medium, where zero-shot is competitive or better. The claim may hold for Llama3.1-8B, but as written it is overstated. Third, the inter-narrational definition in the prompt appendix adds a condition ('who has demonstrated at least one form of intra-narrational unreliability') that is not in the main-text definition; someone replicating from the appendix will run a slightly different task. Fourth, test classes with 2 to 7 positive examples (Reviews inter-narrational, Reviews inter-textual A and D) make those macro-F1 numbers noisy; the paper does not discuss this.\n\nOn the construct validity worry: the stress-test note worries that expert labels may not reflect ordinary readers' perceptions. That is a fair limitation, and the paper does not test it. But I do not think it is fatal. The dataset is explicitly an expert-annotated resource, and the paper's own claims are modest ('potential'). If someone wants to use this for real-world credibility assessment, they would need to validate against non-expert judgments. As a resource paper, it is fine.\n\nBottom line: the resource and task definition are solid, and the benchmark is useful despite the reporting flaws. I would send this to peer review. The fixes are straightforward: add majority baselines, qualify the training-vs-zero-shot claim, reconcile the inter-narrational definition, and flag the tiny classes.\n\nWho it is for: computational narrative understanding and anyone building datasets for subjective literary constructs. I would cite it if I worked on narrative NLP. Bring to reading group? Maybe.","headline":"First expert-labeled dataset for unreliable narrators, carefully built and honestly evaluated; the paper's own tables undercut one headline claim, but the resource and task definition survive.","tokens_in":29814,"tokens_out":3561,"would_cite":true,"duration_ms":40720,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that large language models can begin to classify unreliable narrators across fiction, blogs, Reddit, and reviews, and that the more abstract the cue, the worse they do.","keywords":["unreliable narrator","narratology","large language models","curriculum learning","text classification","first-person narrative","expert annotation","domain transfer"],"falsifier":"Re-annotate a random sample of TUNA texts with five ordinary readers who are not literature experts, giving them only the definitions from Section 3 and the same within-narrative instruction, and compare their resolved labels with the expert labels; if Cohen's kappa falls below about 0.4, the dataset captures an expert-specific convention rather than a general property of text. A second, complementary observation: a model that scores higher on inter-textual than on intra-narrational unreliability would contradict the paper's difficulty-gradient finding.","tokens_in":28706,"feed_emoji":"🎭","tokens_out":6900,"duration_ms":73621,"temperature":0.7,"pith_summary":"The paper introduces TUNA, an expert-annotated collection of first-person narratives from fiction, blogs, Reddit, and hotel reviews, and defines three increasingly abstract classification tasks taken from literary narratology: intra-narrational unreliability (surface verbal tics), inter-narrational unreliability (contradiction by another character or an unchanged unreliable past self), and inter-textual unreliability (matching the narrator to literary tropes such as the naïf, madman, pícaro, or clown). It then evaluates six large language models in zero-shot, few-shot, fine-tuned, and curriculum-learning settings. The central empirical claim is that the task is learnable but far from solved: expert agreement on the labels is substantial (Cohen's kappa 0.71–0.75), the best fine-tuned 8B model reaches macro-F1 around 57 on the surface-cue task across domains, and performance falls to about 19 on trope matching. The paper also argues that learning on fiction snippets transfers to real-world text domains, and that the difficulty of the three tasks forms a clear gradient tied to how abstract the required inference is.","feed_headline":"LLMs detect unreliable narrators, but abstract cues defeat them","feed_subtitle":"Surface verbal tics are learnable (macro-F1 ~57); literary trope matching stays near chance.","key_machinery":"The machinery is a trio: the narratological taxonomy of Hansen (2007) and Riggan (1978), which supplies the label sets and the definitions annotators apply; the TUNA dataset itself, with expert annotations, resolved disagreements, and written justifications for every label; and the curriculum-learning training setup, which orders fiction training samples by the number of plausible candidate labels an LLM detects, training first on unambiguous samples. The taxonomy defines the task, the dataset provides the ground truth the paper argues is reliable, and the curriculum ordering is the device that most consistently improves small-model performance, yielding the paper's best results on the surface-cue task.","core_discovery":"The paper's central claim is that unreliable narrator identification, borrowed from literary theory, can be operationalized as three separate classification tasks and that large language models can learn them, although none of the tasks is solved. The authors contend that expert annotators with degrees in English literature can reliably apply Hansen's taxonomy to short first-person texts, with Cohen's kappa between 0.71 and 0.75 for the three tasks. On the learning side, they find that fine-tuning and curriculum learning on fiction snippets consistently outperform zero-shot prompting, that an 8B model (Llama3.1-8B) with curriculum learning achieves macro-F1 of about 57 for intra-narrational unreliability averaged over domains, about 34 for inter-narrational, and about 19 for inter-textual, and that these out-of-domain results are broadly comparable to in-domain fiction results. The paper also reports that curriculum learning helps smaller models but not larger ones, that few-shot learning suffices for the 70B model, and that BERT-scale classifiers generalize poorly to out-of-domain texts.","pith_inferences":["One could test whether the difficulty gradient is driven by data sparsity rather than inference depth: the inter-textual classes are rare in some domains (for example, only 3 naïf labels in 100 reviews), so a class-balanced or synthetic augmentation experiment would separate data availability from task abstractness.","Because annotations were resolved through discussion among literature experts, the labels likely encode one interpretive community's norms; measuring label stability across lay readers, writers, and other languages would stress-test the construct validity of the dataset.","The paper's default that a narrator is reliable until cues appear suggests a complementary control task: prompting a model to justify why a narrator is reliable could expose systematic over-triggering on hedging language, which the paper's own error analysis already hints at.","The transfer design treats fiction as the source domain; inverting the direction and training on reviews or Reddit to detect fiction tropes would reveal whether the abstract cues are domain-general or literature-bound."],"forward_implications":["Expert annotators can apply the three-part taxonomy of unreliability to short real-world first-person texts with substantial agreement, making the task empirically tractable.","Training on fiction snippets with curriculum ordering lets an 8B model beat zero-shot prompting, reaching macro-F1 around 57 on verbal-tics detection over blogs, Reddit, reviews, and fiction.","For every tested model, the three tasks form a difficulty gradient from surface verbal tics to abstract trope matching, implying that model performance tracks the abstractness of the required inference.","Knowledge learned from fiction transfers to real-world domains, with out-of-domain scores comparable to in-domain ones for the same task.","At the largest model scale tested, few-shot learning matches fine-tuning, suggesting that fewer examples suffice as model size grows."],"supporting_citations":[{"why":"Supplies the three-part taxonomy of unreliability (intra-narrational, inter-narrational, inter-textual) that defines the labels and annotation instructions.","marker":"Hansen (2007)"},{"why":"Defines the four unreliable narrator tropes (naïf, madman, pícaro, clown) used as the inter-textual label set.","marker":"Riggan Jr (1978)"},{"why":"Provides the foundational definition of the unreliable narrator as unintentionally misleading, which the paper adopts and distinguishes from deliberate deception.","marker":"Booth (1961)"},{"why":"Supports the assumption that explicit or implicit information in the text is sufficient to ascertain narrator unreliability.","marker":"Chatman (1990)"},{"why":"Supports the default assumption that a narrator is reliable until explicit or implicit unreliability cues appear.","marker":"Wall (1994)"},{"why":"Introduces curriculum learning, the training-ordering method the paper uses to split easy and difficult samples.","marker":"Bengio et al. (2009)"},{"why":"Provides the LoRA parameter-efficient fine-tuning approach used in the supervised training experiments.","marker":"Hu et al. (2022)"},{"why":"Source of the hotel-review narratives in the Reviews domain of TUNA.","marker":"Ott et al. (2011, 2013)"},{"why":"Source of the blog post narratives in the Blog post domain of TUNA.","marker":"Lukin et al. (2016)"},{"why":"Source of the r/AITA subreddit narratives in the Subreddit domain of TUNA.","marker":"Vijjini et al. (2024)"}],"fun_headline_variants":["LLMs spot unreliable narrators, but only on the surface","Unreliable narrators: LLMs learn the tells, miss the lies","From fiction to Reddit: LLMs struggle with narrator trust","Narrator unreliability: LLMs solve surface, fail deep","LLMs fail at deep unreliable narrator cues"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole evaluation presupposes that a taxonomy developed for literary fiction transfers validly to short real-world narratives, and that expert readers applying only within-narrative evidence can reliably identify abstract phenomena such as trope matching; if the expert labels reflect shared literary conventions rather than properties ordinary readers perceive, every reported F1 score measures agreement with an artifact of the annotation procedure.","fun_headline_variants_meta":{"raw":{"variants":["LLMs spot unreliable narrators, but only on the surface","Unreliable narrators: LLMs learn the tells, miss the lies","From fiction to Reddit: LLMs struggle with narrator trust","Narrator unreliability: LLMs solve surface, fail deep","LLMs fail at deep unreliable narrator cues"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000364,"raw_usage":{"total_tokens":1968,"prompt_tokens":960,"completion_tokens":1008,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":921}},"tokens_in":576,"tokens_out":1008,"duration_ms":9423,"temperature":1.0,"reasoning_tokens":921,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:33:10.777441+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of TUNA texts with five ordinary readers who are not literature experts, giving them only the definitions from Section 3 and the same within-narrative instruction, and compare their resolved labels with the expert labels; if Cohen's kappa falls below about 0.4, the dataset captures an expert-specific convention rather than a general property of text. A second, complementary observation: a model that scores higher on inter-textual than on intra-narrational unreliability would contradict the paper's difficulty-gradient finding.","supporting_citations":[],"review_version":1}