Help Me Write a Story: Evaluating LLMs' Ability to Generate Writing Feedback

Elizabeth Clark; Fantine Huot; Hannah Rashkin; Mirella Lapata

REVIEW 3 major objections 5 minor 12 references

Help Me Write a Story: Evaluating LLMs' Ability to Generate Writing Feedback

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM story feedback is specific and mostly accurate, but it systematically misses the most salient writing problem and misjudges when praise is called for.

desk verdict Solid dataset and evaluation protocol for LLM story feedback, but the headline claim that models miss the biggest issue needs a human baseline on the same dimensions before it is fully supported. read the letter →

arxiv 2507.16007 v1 pith:WZ6424E3 submitted 2025-07-21 cs.CL

Hannah Rashkin , Elizabeth Clark , Fantine Huot , Mirella Lapata This is my paper · ORCID

classification cs.CL

keywords writingfeedbacklargelanguagemodelsstoryhumanevaluationerrordetectioncreativeassistancesynthetictextcorruptioncalibration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Can an off-the-shelf large language model act as a useful story-writing tutor? To answer this, the paper builds a controlled test bed: roughly 1,300 short stories, some original and some corrupted by repeated back-translation, sentence swapping, or sentence deletion, with feedback generated by eight LLMs under four prompting styles, producing 83,456 story-feedback pairs overall. Human annotators scored the feedback on whether it is well-formed, specific, correct, relevant to the induced corruption, and aimed at the biggest problem. The central finding is that current models produce concrete, plausible, mostly accurate feedback, but they often leave out the story's most salient problem and are not well calibrated about when to offer praise instead of criticism. If this is right, feedback quality for LLM writing assistants is limited less by fluency or specificity than by prioritization and calibration.

What carries the argument

The central object is the STORY FEEDBACK benchmark, built from 326 seed stories expanded to roughly 1,300 inputs by adding three synthetic corruption types (backtranslate, swap, delete) alongside untouched originals, and paired with feedback from eight LLMs under four prompts (bulleted list with and without a taxonomy, one sentence, spot the problem). The evaluation machinery that carries the argument is a seven-question human annotation protocol that separates sanity, feedback type, perfect-agree, correctness, error-detection, specificity, and relevance; the "perfect as-is" phrase serves as an automatic detector of the model's judgment, and the relevance question, asked with the uncorrupted original visible, ties feedback to the specific induced corruption. The conceptual distinction doing the work is correctness versus error-detection: a suggestion can be accurate without targeting the most salient problem, and the dataset is designed to expose exactly that gap.

What would settle it

Run the same human rating protocol on naturally occurring draft stories (for instance, first drafts by novice writers) with no injected corruptions and compare models' error-detection rates with their correctness rates; if the two rates converge, or if annotators cannot agree on a single biggest problem above chance, the claim that models systematically miss the most salient issue would fail to generalize.

Watch

Extended reading notes

Core claim

The paper establishes a measured separation between feedback-quality dimensions. In its controlled setup, human annotators scoring 1,920 feedback examples find that models are usually well-formed, specific to the story, and mostly correct, but their error-detection and relevance scores are markedly lower than their correctness scores: a suggestion can be accurate and story-specific yet miss what annotators view as the main problem, particularly when the corruption disrupts coherence through swapped or deleted sentences. The paper also establishes that "perfect as-is" judgments are unreliable, since annotators often still see fixable problems in stories the model praises. Its conclusion is that current LLMs are useful suppliers of plausible, concrete suggestions, but not yet reliable identifiers of the biggest writing issue or deciders of when praise is appropriate.

Load-bearing premise

The measurement assumes a corrupted story has a well-defined "biggest problem" and that the uncorrupted original is the right reference for judging relevance; if synthetic corruptions do not resemble naturally occurring writing issues, or if annotators cannot agree on the main issue, the measured gaps in error-detection and relevance may not transfer to real feedback settings.

Editorial extensions

If this is right

A writer using a current LLM as a story tutor should expect concrete, story-specific, and mostly accurate comments, but cannot assume the model will point at the draft's biggest flaw.
Even the strongest models leave a gap between correctness and error-detection, so improving model scale alone is unlikely to solve the prioritization problem.
Prompts that ask for a structured bulleted list help more than a direct "spot the problem" challenge, which produces the lowest correctness and error-detection among the tested prompts.
Two-shot prompting does not reliably help and can hurt specificity by making models imitate the examples.
Corruption type matters: models handle wording-level noise from back-translation better than coherence breaks from swapped or deleted sentences, so feedback research should treat narrative-level issues as a distinct challenge.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension is measuring top-1 hit rate of a model's suggestions against annotators' single biggest issue; the paper's correctness-versus-error-detection gap predicts this hit rate will be near chance for coherence-level corruptions.
The low inter-annotator agreement on error-detection (alpha 0.33) hints that "biggest problem" is partly subjective, so real-world evaluations may need rank-based or distributional metrics rather than one ground-truth issue.
An iterative revision stress test is a natural next step: after a writer applies the model's feedback, the model should shift its attention to the next most salient problem, and the single-turn design cannot reveal whether it does.
The different category distributions for model versus human feedback (models emphasize style and additions or removals, humans emphasize grammar and coherence) suggest an attention bias that targeted prompting or fine-tuning might correct.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a solid, useful evaluation paper whose headline conclusion runs ahead of its evidence. The dataset and annotation protocol are genuine contributions; the 'models miss the biggest issue' claim needs a human-feedback baseline on the same dimensions before it is the takeaway.

What's new: STORYFEEDBACK, roughly 84k model feedback pairs built from 1,300 controlled story versions (original, backtranslate, swap, delete), eight models, four prompt types, zero- and two-shot. The human evaluation covers seven dimensions with three annotators per item, and the authors validate their automatic metrics against the human judgments. The annotation pipeline is serious: tutorial, qualifying quiz, re-annotation of outlier workers. That is real, reproducible scaffolding, and the stated plan to release the data makes it a resource others can build on.

The soft spots are real but don't sink the paper. The central finding—specific and mostly correct, but missing the biggest issue—rests on absolute human scores for error-detection and relevance, with no human-written feedback scored under the same protocol. The 5,754 human entries collected in Section 6.4 are only categorized by Gemini, not rated for correctness, error-detection, specificity, or relevance. So 'models fail' is uncalibrated: the task may be hard for anyone, and the synthetic corruptions (especially swap and delete) are not the same as naturally occurring writing problems. The paper's own agreement numbers point the same way: Krippendorff's alpha is 0.33 for error-detection and 0.15 for perfect-agree, low enough that the 'biggest issue' target is not a stable construct. Averaging three raters helps but does not remove the concern. Also, mapping 'Both' to 0 for feedback-type and 'I don't see any issues' to 0 for error-detection pushes the reported rates down; the mappings are explained, but they shape the failure narrative.

I'd ask a reviewer for one addition: score a set of human-written feedback entries on the same dimensions, or at least re-run the main analysis with the agreement-weighted constructs. That would tell us whether the gap is an LLM deficiency or a task difficulty. Minor point: the perfect-agree numbers for Gemma/Gemini rest on tiny samples (n<20), so those rows should be interpreted cautiously.

Who this is for: people building or evaluating LLM writing tutors, and anyone designing feedback-quality evaluation protocols. It deserves a serious referee, and with the human baseline added it would be a much stronger benchmark. I'd send it out.

Bottom line: good paper, useful resource, headline slightly overclaimed.

Referee Report

3 major / 5 minor

Summary. The paper introduces STORY FEEDBACK, a new task and dataset for evaluating LLM-generated writing feedback on short stories. Starting from 326 seed stories, the authors create four versions per story (original plus backtranslate, swap, and delete corruptions), yielding ~1,300 stories, and collect 83,456 story-feedback pairs by prompting eight LLMs with four prompt types and zero- or two-shot settings. They evaluate the feedback with automatic metrics (perfect-as-is precision, trigram repetition) and a human annotation study of 1,920 outputs on seven dimensions: sanity, feedback-type, perfect-agree, correctness, error-detection, specificity, and relevance. The main findings are that models produce mostly well-formed, specific, and correct feedback, but that they often miss the most salient writing problems, score low on relevance to the induced corruptions, and are not discerning in agreeing that a story is perfect-as-is. The paper also collects 5,754 human-authored feedback entries but only analyzes their topical categories.

Significance. If its conclusions hold, the paper provides a useful benchmark resource and an evaluation framework for a relatively understudied task: open-ended story-writing feedback. The work has clear strengths: it is large-scale, uses controlled corruptions, covers a diverse set of models and prompts, and validates the automatic 'perfect-as-is' detection against human feedback-type labels (Pearson r ≈ 0.90). The dataset release is a concrete contribution that could support follow-up work. However, the headline deficiency claims—that models 'often leave out suggestions that would target the most salient writing problems' and are 'not very discerning with positive encouragement'—rest on absolute human scores without a human-written feedback baseline under the same protocol. Combined with the low inter-annotator agreement on the very dimensions that support those claims (error-detection α = 0.33; perfect-agree α = 0.15), the current evidence is not yet conclusive about LLM-specific failure. The resource is valuable, but the interpretive framing needs calibration before the paper can support its strongest conclusions.

major comments (3)

[§6.3, §6.4, App. D.1] The central claim that models 'often leave out suggestions that would target the most salient writing problems' is not calibrated against human-written feedback under the same evaluation protocol. The 5,754 human feedback entries described in §6.4 and App. D.1 are only categorized by an LLM (Figure 5, App. G); they are never scored on correctness, error-detection, specificity, or relevance. Since the error-detection scores in Table 3 range from 0.22 to 0.76, the observed gaps could reflect task difficulty rather than a deficiency specific to LLMs. The authors should either rate a sample of human feedback with the same annotation task and report a human baseline, or explicitly reframe the findings as descriptive of the task's difficulty rather than as evidence of LLM failure.
[Table 2] The inter-annotator agreement for the two constructs that support the headline negative findings is low: Krippendorff's alpha is 0.33 for error-detection and 0.15 for perfect-agree. The paper acknowledges subjectivity for these dimensions, but the headline conclusions in §6.3 and §7 rely on averages over only three raters. This is load-bearing because the claim that models 'miss the biggest issue' assumes that annotators themselves have a stable notion of the biggest issue. I recommend reporting results on a high-agreement subset (e.g., examples where all three raters agree on error-detection), and/or providing confidence intervals or variance components that reflect rater disagreement. The current presentation overstates the reliability of the target construct.
[App. D.4, Table 11] The error-detection question maps both 'No, following the feedback would not help fix the biggest problem' and 'I don't see any big problems in the story' to 0. This conflates the absence of a perceived problem with the model's failure to detect one, which can artificially lower error-detection scores. This is particularly relevant for original stories (Table 4 shows original error-detection of 0.485) and may also penalize valid feedback on corrupted stories when annotators do not perceive the induced corruption as a 'big problem.' The mapping should be separated or the analysis should condition on annotator agreement that at least one big problem exists, so that 'no issue perceived' does not count as an error-detection failure.

minor comments (5)

[§4, backtranslate description] The sentence 'The story is translated 3 to German and then back to English ten times' appears to contain a stray '3'; it should read 'translated to German' with the ten-round iteration described separately.
[§6.4] The phrase 'by prompting 8 Gemini 1.5 Flash' contains a stray '8' and should be 'by prompting Gemini 1.5 Flash.'
[Table 4] For the relevance row on original stories, 'nan±nan' should be replaced with a placeholder such as '—' since the relevance question is only administered for corrupted stories.
[Table 3] The note about perfect-agree scores for Gemma/Gemini being based on n<20 is helpful, but the paper would be easier to interpret if the per-cell sample sizes for all models and dimensions were reported, especially for perfect-agree and relevance.
[App. D.4, Table 11] The mapping of the feedback-type option 'Both' to 0 should be explicitly justified in the main text; if the variable is intended to measure 'feedback that is only positive,' that should be stated, because the same option is treated as 'positive' for the perfect-agree path in the annotation flow.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical evaluations against independent human ratings, with no fitted parameter renamed as a prediction and no load-bearing self-citation chain.

full rationale

This paper makes no derivation-style claim that reduces to its own inputs. The central findings, such as "models provide concrete and plausible suggestions but often leave out suggestions that would target the most salient writing problems" (Section 6.3) and the abstract's claim that models "often fail to identify the biggest writing issue in the story," are obtained by presenting stories and model-generated feedback to human annotators, who score correctness, error-detection, specificity, relevance, and perfect-agree on a protocol shown in Figure 4 and Appendix D.2. The error-detection label is defined by the annotator's own judgment of the biggest problem, not by the model output, and annotators wrote their own feedback before seeing the model feedback, so the target construct is not defined in terms of the model's predictions. The automatic metric "PAI prec" is also externally validated: Section E reports a Pearson correlation of ~0.90 between the presence of the phrase "perfect as-is" and human feedback-type labels. No model parameter is fitted to the evaluation data, and no prediction is statistically forced by a subset fit. The few self-citations (e.g., Huot et al. for trigram/sentence-splitting implementations, Du et al. for the feedback taxonomy) are methodological tools, not load-bearing evidence for the paper's conclusions. The skeptical concern that human feedback was never scored under the same protocol is a calibration gap: it means the paper cannot distinguish LLM-specific deficiency from general task difficulty, but it is not a circularity. Likewise, the low Krippendorff's alpha of 0.33 for error-detection is a measurement-reliability limitation, explicitly acknowledged by the authors in the Limitations section as "variations in judgment due to the natural ambiguity of the task," and it does not make the evaluation self-confirming. The paper's conclusions are therefore self-contained empirical findings, with weaknesses that belong under correctness risk or external validity rather than circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

Central claims rest on synthetic corruption validity, human judgment reliability, and seed story representativeness rather than on fitted parameters. No invented entities.

free parameters (1)

backtranslate iterations = 10
Number of round-trip translations used to introduce noise; chosen after observing noise on a small set (Section 4), not from a formal fit, and affects corruption severity.

assumptions (4)

domain assumption Synthetic corruptions (backtranslate, swap, delete) produce writing issues that are representative of real problems in short story drafts.
Used in Section 4 to create controlled data; the paper's Limitations notes this may not hold for naturally occurring errors.
domain assumption Human annotators' judgments of biggest issue, correctness, and relevance are valid ground truth for feedback quality despite subjectivity.
The evaluation in Section 6 relies on averaged ratings from 3 annotators; agreement is low on some dimensions (Table 2).
domain assumption The 326 seed stories from ROCStories and BIG-bench are representative of short story writing for this task.
Dataset construction in Section 4; acknowledged in Limitations as only English short stories.
domain assumption The revision taxonomy of Du et al. (2022b) transfers to story-writing feedback categories.
Used to define prompt categories in Section 4 and analysis categories in Section 6.4.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Help Me Write a Story: Evaluating LLMs' Ability to Generate Writing Feedback." pith.science (2026). https://pith.science/paper/WZ6424E3

@misc{pith2026250716007,
  author       = {Pith},
  title        = {Pith review of: Help Me Write a Story: Evaluating LLMs' Ability to Generate Writing Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WZ6424E3}},
  note         = {Machine review of arXiv:2507.16007}
}

read the original abstract

Can LLMs provide support to creative writers by giving meaningful writing feedback? In this paper, we explore the challenges and limitations of model-generated writing feedback by defining a new task, dataset, and evaluation frameworks. To study model performance in a controlled manner, we present a novel test set of 1,300 stories that we corrupted to intentionally introduce writing issues. We study the performance of commonly used LLMs in this task with both automatic and human evaluation metrics. Our analysis shows that current models have strong out-of-the-box behavior in many respects -- providing specific and mostly accurate writing feedback. However, models often fail to identify the biggest writing issue in the story and to correctly decide when to offer critical vs. positive feedback.

Figures

Figures reproduced from arXiv: 2507.16007 by the authors.

**Figure 1.** Examples of good and bad feedback for a short story self. Providing feedback is more open-ended and should help guide the human writer towards fixing the writing themselves. Therefore, this task requires a different skill-set from other creative tasks including being able to identify writing problems and articulate them in a clear and constructive way. However, there aren’t many existing resources for studying LLM … view at source ↗

**Figure 2.** An overview of the task. Short stories are [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Example with the corruption methods applied. Highlights indicate the portions of the story that are [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Outline of the human annotation task in which annotators can follow different question paths. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Distribution of categories of feedback written [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Types of feedback written by humans and generated by models. [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 9 canonical work pages

[1]

No, the text does not give any feedback

Well-formed feedback Task Instructions This question verifies that the feedback contains a critique of the story and stays on-topic. This isn’t asking about the quality of the feedback, just verifying that it is actually feedback for the given story, since we don’t want to evaluate it if it is not. Feedback that starts talking about another story or that ...

work page
[2]

perfect as-is

Feedback Type Task Instructions Does the feedback claim that the story is “perfect as-is” or that it doesn’t require any changes? If the feedback is simply, “The text is perfect as-is.”, please label it “Yes.” However, if the feedback says the text is perfect, but then also includes other suggestions, please respond “Both.” Q/A Format Is the feedback sayi...

work page
[3]

perfect as-is

Perfect-Agree Task Instructions If the feedback doesn’t suggest any changes or says the story is “perfect as-is”, do you agree? If you don’t see any problems or issues with the story, please mark “Yes.” If you think there are issues with the story that need to be improved, say “No.” Note that this may be subjective in some cases, so please use your best j...

work page
[4]

Every piece of feedback is correct

Correctness Task Instructions Does the feedback address weaknesses in the text? It’s ok if it doesn’t address all the problems in the text, as long as all the things it mentions would improve the story. Q/A Format Correctness: Does the feedback address a weakness in the story? ◦ “Every piece of feedback is correct.” ◦ “Some of the feedback is correct, but...

work page
[5]

perfect as-is

The GPT model output was generated using the OpenAI API11 in May 2024. The Gemma and Gemini models were generated from the publicly available checkpoints12 in December 2024. The feedback was generated using the API or model’s default settings to mimic scenarios where writers would use these systems out-of-the-box. All model outputs were sampled with tempe...

work page 2024
[6]

Yes, the feedback is about this story

Specificity Task Instructions Is it clear that the feedback is for this specific story? Or is it vague or general enough to apply to another story? Specific feedback might mention characters, events, or details from the story, or quote or rewrite pieces of the original story. For this question, it doesn’t matter whether the feedback is accurate or helpful...

work page
[7]

Somewhat

Relevance Task Instructions When you read the edited story draft, do the changes that were made from the original story reflect the suggestions from the feed- back? Or do the edits and the feedback seem unrelated? If it is a mix of both (i.e., some feedback is reflected in the edits and some is not OR some edits are related to the feedback and others aren...

work page
[10]

I don’t see any big problems in the story

Error-Detection Task Instructions If you feel there is a problem with the story, would following the feedback help fix that specific problem? Does any part of the feedback address what you consider to be the biggest issue in the text? As long as one part of the feedback addresses the problem, you can ignore the rest of the feedback. This may be somewhat s...

work page

Show all 12 references

[2020]

Transactions of the Associ- ation for Computational Linguistics, 8:264–280

Leveraging pre-trained checkpoints for se- quence generation tasks. Transactions of the Associ- ation for Computational Linguistics, 8:264–280. Lei Shu, Liangchen Luo, Jayakumar Hoskere, Yun Zhu, Simon Tong, JD Chen, and Lei Meng. 2024. Rewritelm: An instruction-tuned large la...

2024
[2022]

arXiv preprint arXiv:2210.13669

Help me write a poem: Instruction tuning as a vehicle for collaborative poetry writing. arXiv preprint arXiv:2210.13669. Eric Chamoun, Michael Schlichtkrull, and Andreas Vla- chos. 2024. Automated focused feedback generation for scientific writing assistance. In Findings of th...

2024 arXiv
[2023]

arXiv preprint arXiv:2309.14556

Art or artifice? large language models and the false promise of creativity. arXiv preprint arXiv:2309.14556. Tuhin Chakrabarty, Vishakh Padmakumar, Faeze Brah- man, and Smaranda Muresan. 2024. Creativity sup- port in the age of large language models: An empiri- cal study invol...

2024 arXiv
[2024]

LLM-as-a-tutor in EFL writing education: Fo- cusing on evaluation of student-LLM interaction. In Proceedings of the 1st Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U), pages 284–293, Miami,...

2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

No, the text does not give any feedback

Well-formed feedback Task Instructions This question verifies that the feedback contains a critique of the story and stays on-topic. This isn’t asking about the quality of the feedback, just verifying that it is actually feedback for the given story, since we don’t want to evaluate it if it is not. Feedback that starts talking about another story or that ...

work page

[2] [2]

perfect as-is

Feedback Type Task Instructions Does the feedback claim that the story is “perfect as-is” or that it doesn’t require any changes? If the feedback is simply, “The text is perfect as-is.”, please label it “Yes.” However, if the feedback says the text is perfect, but then also includes other suggestions, please respond “Both.” Q/A Format Is the feedback sayi...

work page

[3] [3]

perfect as-is

Perfect-Agree Task Instructions If the feedback doesn’t suggest any changes or says the story is “perfect as-is”, do you agree? If you don’t see any problems or issues with the story, please mark “Yes.” If you think there are issues with the story that need to be improved, say “No.” Note that this may be subjective in some cases, so please use your best j...

work page

[4] [4]

Every piece of feedback is correct

Correctness Task Instructions Does the feedback address weaknesses in the text? It’s ok if it doesn’t address all the problems in the text, as long as all the things it mentions would improve the story. Q/A Format Correctness: Does the feedback address a weakness in the story? ◦ “Every piece of feedback is correct.” ◦ “Some of the feedback is correct, but...

work page

[5] [5]

perfect as-is

The GPT model output was generated using the OpenAI API11 in May 2024. The Gemma and Gemini models were generated from the publicly available checkpoints12 in December 2024. The feedback was generated using the API or model’s default settings to mimic scenarios where writers would use these systems out-of-the-box. All model outputs were sampled with tempe...

work page 2024

[6] [6]

Yes, the feedback is about this story

Specificity Task Instructions Is it clear that the feedback is for this specific story? Or is it vague or general enough to apply to another story? Specific feedback might mention characters, events, or details from the story, or quote or rewrite pieces of the original story. For this question, it doesn’t matter whether the feedback is accurate or helpful...

work page

[7] [7]

Somewhat

Relevance Task Instructions When you read the edited story draft, do the changes that were made from the original story reflect the suggestions from the feed- back? Or do the edits and the feedback seem unrelated? If it is a mix of both (i.e., some feedback is reflected in the edits and some is not OR some edits are related to the feedback and others aren...

work page

[8] [10]

I don’t see any big problems in the story

Error-Detection Task Instructions If you feel there is a problem with the story, would following the feedback help fix that specific problem? Does any part of the feedback address what you consider to be the biggest issue in the text? As long as one part of the feedback addresses the problem, you can ignore the rest of the feedback. This may be somewhat s...

work page