StoryScope: Investigating idiosyncrasies in AI fiction

Chau Minh Pham; Jenna Russell; John Wieting; Mohit Iyyer; Rishanth Rajendhran

REVIEW 3 major objections 7 minor 4 cited by

StoryScope: Investigating idiosyncrasies in AI fiction

T0 review · 3 major / 7 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Narrative structure alone separates AI fiction from human fiction with 93.2% macro-F1, even after style is removed.

desk verdict A well-built empirical case for narrative-level AI detection, but the headline F1 is only as clean as the LLM annotator behind it. read the letter →

arxiv 2604.03136 v6 pith:OB35U2QS submitted 2026-04-03 cs.CL

Jenna Russell , Rishanth Rajendhran , Chau Minh Pham , Mohit Iyyer , John Wieting This is my paper · ORCID

classification cs.CL

keywords AI-generatedfictionnarrativefeaturesauthorshipdetectiondiscourseanalysisinterpretableclassificationLLMstorytellingoriginalitystorystructure

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that AI-generated fiction differs from human-written fiction not just in surface style but in deeper narrative decisions—how the plot is arranged, how themes are stated, how characters' choices are framed. It introduces StoryScope, a pipeline that turns more than sixty thousand stories into 304 interpretable narrative features, and finds that these features alone distinguish human from AI authorship at 93.2% macro-F1, retaining 97% of the performance of a model that also includes stylistic cues. The separation survives stylistic editing, and human stories occupy a rarer, more dispersed region of narrative space while the five AI models cluster together. If the claim holds, it gives readers and publishers a more durable, explainable basis for assessing whether a story was originally conceived by a human.

What carries the argument

The load-bearing mechanism is the StoryScope pipeline: stories are first converted into structured templates that abstract away surface wording along ten narrative dimensions; a comparative analysis stage identifies where sources diverge on the same prompt; a discovery stage turns those observations into 304 closed-form feature questions; and an LLM annotator scores every story on every feature. The resulting vectors feed gradient-boosted tree classifiers with per-feature attribution, which lets the authors isolate a small set of stable core features. What the machinery does is transfer the burden of detection from lexical surface to structural choices—causal continuity, chronological orderi

What would settle it

Take a held-out set of stories from the same six sources, have human narrative analysts score the 304 features (or use a non-LLM protocol to do so), retrain the classifier, and compare macro-F1 to the LLM-annotated result. If the separation collapses or drops far below 93.2%, the claim that narrative construction itself drives detection is falsified. A complementary check: classify stories from a new, unseen language model not in the training set; if narrative features fail to separate it from humans, the 'shared AI narrative space' is not general.

Watch

Extended reading notes

Core claim

The paper's central claim is that narrative construction carries a systematic, learnable signature of authorship. Using a parallel corpus in which each of 10,272 prompts was written by a human author and five language models, StoryScope induces 304 discourse-level features across ten narrative dimensions—character, plot, setting, time, revelation, perspective, and others—and classifies stories from these features alone. Narrative features reach 93.2% macro-F1 for human-versus-AI detection and 68.4% for six-way attribution; a compact set of 30 core features captures most of the binary signal. The paper also reports that AI stories over-explain themes, favor tidy single-track plots with protag

Load-bearing premise

Everything rests on the assumption that the 304 narrative features, scored entirely by an LLM, are valid measurements of real narrative structure; if those annotations encode annotator-specific biases, the observed human/AI separation may not generalize to features measured another way.

Editorial extensions

If this is right

A detector built only on narrative features keeps working after surface artifacts (clichés, purple prose) are edited out, suggesting it targets structure rather than style.
Because narrative choices are more expensive to alter than wording, such features may stay diagnostic longer as language models update.
The 30 core features give an interpretable checklist of how AI storytelling currently defaults: explicit themes, linear plots, protagonist-driven endings, embodied emotion.
Per-source fingerprints allow attributing a story to a specific model with 68.4% macro-F1, and the human class is the most separable.
Rarity in narrative feature space offers a quantitative proxy for originality, which could inform discussions of authorship and creative control.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: because the feature values come from LLM judges, part of the gap may reflect the judges' own stylistic or prompt-following biases rather than properties of the stories; a human-annotation or non-LLM replication would test this.
Editorial inference: if narrative features are learnable, adversarial authors could 'humanize' structure by adding subplots, time jumps, and ambiguous endings; the durability claim would then erode, and the method's value would be as a moving target rather than a fixed detector.
Editorial inference: the rarity measure could be used as a generation-time reward to push AI outputs toward less typical narrative combinations, effectively testing whether the observed cluster is an inevitable property of LLMs or just a current default.
Editorial inference: the same template-plus-features approach could be applied to other long-form creative domains such as screenplays or narrative nonfiction to see whether the human/AI structural divide generalizes beyond short fiction.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the short version: this is a well-built, unusually honest empirical paper that makes a real case for narrative-level AI detection, and the 93.2% F1 is likely a true effect. But the 'narrative' claim rests on an LLM annotation instrument, and the validation of that instrument is thin. If you read it as a study of what Gemini 3 Flash reliably reports about stories, it's solid; if you read it as proof about underlying narrative construction, it needs another round of evidence.

What's genuinely new: the pipeline that induces 304 interpretable narrative features via cross-source comparison, the parallel corpus of 10k prompts × six sources, and the demonstration that these features alone achieve 93.2% macro-F1. The paper does the right empirical work: prompt-level grouping, length-matched evaluation, topic sensitivity, memorization audit, and a careful style/non-style boundary. The core feature set is compact and plausible—AI over-explaining themes, humans using nonlinear time—and the rarity analysis is a nice operationalization of originality.

The soft spots are real but not fatal. The biggest one is the measurement loop: GPT-5.1 defines the features, Gemini 3 Flash assigns the values, and the same model family generated one of the five AI sources. The human validation is 12 stories and 240 items, with human–human kappa at 0.74. That's not enough to rule out the possibility that the annotator is picking up on stylistic regularities or its own generation priors. The LAMP robustness test edits surface prose but wouldn't remove that kind of latent sensitivity. Separately, the core-feature selection uses training labels, so the 84.8% core-only F1 is partly a fitted number, and the abstract's GPT-dream-sequence fingerprint doesn't show up in the reported fingerprint tables.

None of this kills the central result. The dataset is a contribution in itself, and the main finding—that discourse-level features carry signal independent of style—is worth taking seriously. But the paper should be pushed to validate the annotation instrument (e.g., a second annotator model or a larger human sample) and to soften the causal language. I'd send it to review; it's the kind of work that makes a subfield better.

Referee Report

3 major / 7 minor

Summary. StoryScope proposes an LLM-driven pipeline for extracting discourse-level narrative features from fiction. From 10,272 Books3 short stories, the authors generate mirror stories with five LLMs (61,608 total). A template representation is built with GPT-5.1, comparative analyses are distilled, and 304 features are discovered, then assigned to every story by Gemini 3 Flash. XGBoost on these features achieves macro-F1 93.2% for human/AI detection and 68.4% for 6-way authorship attribution. The paper identifies 30 core features and per-model fingerprints, and uses feature-space distance to argue that AI stories are more convergent and less rare than human stories. Code, prompts, and AI-generated narratives are released.

Significance. The contribution is substantial if the central interpretation holds. A narrative-level representation that is robust to surface style would be a valuable tool for AI authorship analysis and an interpretable complement to raw-text detectors. The evaluation is unusually careful in several respects: prompt-level train/test grouping, length-matched controls, topic-sensitivity tests, a memorization audit, repeatability measurements, and human validation of a subset of annotations. These design choices support the headline classification numbers as empirical facts about the constructed feature space. The main weakness is that the feature space is built and populated entirely by LLMs, which ties the interpretation to the annotator's objectivity. If the authors can address the annotation-validity concern, the paper would be a strong candidate for publication.

major comments (3)

[§2.1–2.2, Appendix C] The central claim that 93.2% macro-F1 reflects "underlying narrative construction" rather than surface style (Abstract, §4) depends on the validity of the feature assignments, but every feature value is generated by Gemini 3 Flash, the same model family as one of the five AI sources, and the feature definitions are induced by GPT-5.1 from LLM-written comparative analyses. Human validation covers only 12 stories and 240 feature-items (mean human–model κ=0.84; human–human κ=0.74, Table 7), which is too small to rule out that the annotator is answering from source-typical stylistic priors or its own generation preferences. The LAMP edit experiment (§4.2) targets surface artifacts and would not remove such latent sensitivity. Please add a cross-annotator experiment (e.g., annotate a held-out sample with a different LLM or with human raters) and report both annotation agreement and downstream
[§2.1, §B (style boundary)] The "Narrative" variant is defined by excluding 47 features rated as style-related by a GPT-5.4 audit, but this boundary is itself one of the load-bearing modeling choices. The paper does not report inter-auditor agreement for this rating, nor a sensitivity analysis around the boundary. If even a fraction of the excluded features are actually narrative (or if some retained features are stylistic in disguise), the 93.2% number changes. Please provide the audit prompt, run an independent second audit, and report results for a wider and more restrictive narrative set (e.g., ±10 features around the boundary).
[§2.2, §5, §G] The 30 core features and the rarity/divergence analyses are selected from the same data and feature space that was constructed by discriminative discovery. This makes the interpretive statements in §4.1 and §5 largely descriptive of the induced space, not independent evidence that human stories are inherently rarer or more complex. The feature-selection thresholds in §D are justified post hoc, and the Core-Only F1 may be optimistically biased by selection on the validation set. I do not dispute the held-out classification result for the full 257-feature model, but the paper should avoid essentialist language ("AI over-explains", "human authors subvert linearity") or support it with out-of-space validation—for example, human annotations of the core concepts on a larger sample.

minor comments (7)

[Table 2] For the ModernBERT baseline, specify how long stories are truncated or downsampled; with max length 512 tokens and stories averaging 4,753 words, the comparison may not be apples-to-apples.
[Figure 1] The figure says "~60k stories" but the exact count is 61,608; the "N" in the figure is also ambiguous. Please use precise numbers.
[Table 5] Gemini and DeepSeek fail the length instruction by about 3,000 words on average. The paper shows length-matching for the binary task, but it would be useful to discuss whether such systematic length differences could interact with narrative feature values in the attribution task.
[§4.2] The edit experiment uses only 278 Gemini-generated stories. Please clarify why only this subset was used and whether results generalize to the other four AI sources.
[§G] The term "rarity percentile" is used in Section 5 before it is formally defined in Section G. A forward reference would help the reader.
[§5] Centroid-distance statements such as "6.6 vs. 4.3" and "closest human-AI pair vs. farthest AI-AI pair" are presented without uncertainty intervals. Given the sensitivity of such claims, bootstrap confidence intervals should be reported.
[§2.1] The phrase "grounded in NarraBench" may overstate the grounding: the feature taxonomy is not a direct instantiation of NarraBench's twelve aspects but a new LLM-derived set. Please clarify the relationship and avoid implying that NarraBench provides a validated measurement instrument for these specific 304 features.

Circularity Check

0 steps flagged · score 0.0 of 10

No construction-level circularity: the 93.2% narrative-feature result is an empirical claim evaluated on a held-out test set, not an identity with the feature-discovery inputs.

full rationale

The central derivation chain is: (i) induce 304 narrative features from a held-out discovery pool of 600 stories using pairwise cross-source comparisons; (ii) assign feature values to all 61,608 stories with Gemini 3 Flash; (iii) train XGBoost on 8,788 prompts and evaluate on a disjoint held-out test set of 1,384 prompts. This is a supervised feature-learning pipeline, but the final macro-F1 is a genuine held-out prediction: the feature definitions do not encode the test labels, the discovery pool is excluded from the main corpus, and prompt-level grouping is used to prevent leakage. Core-feature selection uses bootstrap SHAP and thresholds on training data, but the reported test-set F1 (84.8% for core features) is not forced by those thresholds. The same holds for fingerprint features: they are selected by per-source SHAP concentration and then evaluated on unseen stories. The LLM-defined, LLM-assigned feature space is a serious validity caveat—human validation is small (12 stories, 240 items, mean human-model Cohen's kappa = 0.84, human-human kappa = 0.74) and the semantics of 'narrative' depend on GPT-5.1/Gemini 3 Flash judgment—but this is a measurement-validity threat, not circularity by construction. Self-citations (e.g., Pham et al. 2024; Russell et al. 2025; Thai et al. 2026) are contextual and not load-bearing for the main claim. External raw-text baselines and length-matched audits further anchor the result empirically. Accordingly, no specific circular step can be exhibited from the paper's own equations or definitions.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper's central claims rest on several domain assumptions about the representativeness of the corpus and the validity of LLM-based narrative annotation. The only truly ad hoc element is the purely statistical definition of 'rarity' as a proxy for originality. The free parameters are mostly hand-set thresholds and hyperparameters.

free parameters (5)

Discovery pool size = 100 prompts / 600 stories
Chosen to balance narrative diversity against GPT-5.1 cost; not derived from theory.
Deduplication cosine threshold = 0.85
Hand-set after observing 0.90 left duplicates and higher thresholds merged too aggressively (§B).
Core feature selection thresholds = stab>=0.55, top25>=0.60, gap>=0.20, spread<=0.35
Hand-set criteria in §D that determine which 30 features are called 'core'.
XGBoost hyperparameters = Binary: nest=420, depth=8, lambda=2.0, scale_pos_weight=5; 6-way: nest=500, depth=7, lambda=1.0
Grid-searched on validation split; standard practice but part of the modeling choices.
Rarity k = 25
Nearest-neighbor count for per-story rarity; chosen without sensitivity analysis (§G).

assumptions (5)

domain assumption NarraBench taxonomy is a valid and sufficiently comprehensive model of narrative structure for extracting discourse-level features.
The paper adopts 10 of NarraBench's 12 aspects as the grounding for all features (§2.1); if this taxonomy misses important narrative dimensions, the feature space is incomplete.
domain assumption Books3 short stories are representative of human-authored fiction, and reverse-engineered prompts preserve each story's essential narrative content.
The human corpus is the reference class for 'human writing'; prompt reconstruction via Gemini 2.5 Flash may introduce bias or lose narrative details (§2).
domain assumption LLM-generated stories under the same prompt are representative of 'AI fiction' and each model's output is typical of that model.
Single samples per model per prompt, with specific model versions; other models/temperatures might behave differently.
domain assumption Gemini 3 Flash's feature assignments are reliable measurements of the narrative properties defined by the features.
Validated on only 12 stories / 240 items with mean Cohen's κ=0.84 vs humans; the rest of the 61,608 stories are unvalidated (§C).
ad hoc to paper Mean Euclidean distance to 25 nearest neighbors in the feature space is a valid proxy for narrative originality.
This operationalization of 'originality' is introduced by the authors and tied to copyright law only loosely (§1, §G).

how reviews work

0 comments

Cite this review

Pith. "Pith review of StoryScope: Investigating idiosyncrasies in AI fiction." pith.science (2026). https://pith.science/paper/OB35U2QS

@misc{pith2026260403136,
  author       = {Pith},
  title        = {Pith review of: StoryScope: Investigating idiosyncrasies in AI fiction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OB35U2QS}},
  note         = {Machine review of arXiv:2604.03136}
}

read the original abstract

As AI-generated fiction becomes increasingly prevalent, questions of authorship and originality are becoming central to how written work is evaluated. While most existing work in this space focuses on identifying surface-level signatures of AI writing, we ask instead whether AI-generated stories can be distinguished from human ones without relying on stylistic signals, focusing on discourse-level narrative choices such as character agency and chronological discontinuity. We propose StoryScope, a pipeline that automatically induces a fine-grained, interpretable feature space of discourse-level narrative features across 10 dimensions. We apply StoryScope to a parallel corpus of 10,272 writing prompts, each written by a human author and five LLMs, yielding 61,608 stories, each ~5,000 words, and 304 extracted features per story. Narrative features alone achieve 93.2% macro-F1 for human vs. AI detection and 68.4% macro-F1 for six-way authorship attribution, retaining over 97% of the performance of models that include stylistic cues. A compact set of 30 core narrative features captures much of this signal: AI stories over-explain themes and favor tidy, single-track plots while human stories frame protagonist' choices as more morally ambiguous and have increased temporal complexity. Per-model fingerprint features enable six-way attribution: for example, Claude produces notably flat event escalation, GPT over-indexes on dream sequences, and Gemini defaults to external character description. We find that AI-generated stories cluster in a shared region of narrative space, while human-authored stories exhibit greater diversity. More broadly, these results suggest that differences in underlying narrative construction, not just writing style, can be used to separate human-written original works from AI-generated fiction.

Figures

Figures reproduced from arXiv: 2604.03136 by the authors.

**Figure 1.** Overview of the STORYSCOPE pipeline. Stories are converted into structured templates, then compared across sources writing to the same prompt to induce discriminative narrative features, and finally featurized across the full corpus for downstream detection and authorship experiments. Story inspired by "Tiny and the Monster" (Sturgeon, 1983). structure, and for good reason: these cues are highly discriminatory. AI-… view at source ↗

**Figure 2.** Projection of narrative feature vectors onto the first two linear discriminant [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Confusion matrix for authorship attribution (narrative model) as a percentage (%). Misclassifications concentrate among AI models, particularly DeepSeek–Gemini–Kimi. Gemini, DeepSeek, and Kimi are ‘triplets’. Despite forming a more confused cluster, each still has individual quirks. DeepSeek front-loads crucial context that other sources leave until later. Gemini produces the tidiest endings, extended denouements, … view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Boxplots of story lengths across all stories in the finalized six-sources dataset, [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]

**Figure 5.** Figure 5: Per-story narrative rarity percentiles by sources on the held-out test set. Solid lines [PITH_FULL_IMAGE:figures/full_fig_p023_5.png]

**Figure 6.** Figure 6: Prompt used to generate benchmark writing prompts from source stories [PITH_FULL_IMAGE:figures/full_fig_p028_6.png]

**Figure 7.** Figure 7: Representative benchmark story-generation prompt [PITH_FULL_IMAGE:figures/full_fig_p028_7.png]

**Figure 8.** Figure 8: Prompt for NarraBench template extraction [PITH_FULL_IMAGE:figures/full_fig_p030_8.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Generative AI floods and dilutes the market for books
cs.CL 2026-07 conditional novelty 7.0 of 10

AI-heavy self-published genre fiction reaches commercial scale (about 20% of sales by 2026 Q2) even though it sells less per title, and its rise tracks falling revenue per book for human-authored titles.
CraftAlign: Feature-Grounded Evaluation and Revision Guidance for AI Stories
cs.AI 2026-08 conditional novelty 6.0 of 10

CraftAlign uses 304 writing features and an energy model to choose which aspects of an AI story to revise, and its guidance improves perceived human-likeness in a small human study.
Pangram 4 Technical Report
cs.CL 2026-07 conditional novelty 5.0 of 10

Pangram 4 is a commercial MoE-based detector claiming 0.9916 AUROC, 0.0041% FPR, 0.3396% FNR, plus tokenwise human/AI-assisted/AI-generated labels and humanizer detection.
AI as a Tool for Simulation-Based Experiments in Literary Studies
cs.CL 2026-06 unverdicted novelty 4.0 of 10

Proposes AI-driven simulations for literary-historical experiments and reports preliminary text-generation results claiming the first limited in-distribution outputs matching human novels.

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.