Pith. sign in

REVIEW 3 major objections 3 minor 20 references

eRevise: Using Natural Language Processing to Provide Formative Feedback on Text Evidence Usage in Student Writing

T0 review · 3 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that NLP-selected formative feedback from rubric-based essay scoring helps fifth- and sixth-grade students improve their use of text evidence when revising drafts.

desk verdict A transparent pilot of a well-designed AWE system, but the abstract's causal claim outruns the no-control data; the planned control condition will settle it. read the letter →

arxiv 1908.01992 v1 pith:CN3QWL6N submitted 2019-08-06 cs.CL cs.AI

Haoran Zhang , Ahmed Magooda , Diane Litman , Richard Correnti , Elaine Wang , Lindsay Clare Matsumura , Emily Howe , Rafael Quintana This is my paper · ORCID
classification cs.CLcs.AI
keywords eReviseautomatedwritingevaluationformativefeedbacktextevidenceautomaticessayscoringresponsetoassessmentnaturallanguageprocessingrevision
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper describes eRevise, a web-based writing environment that automatically selects two of four formative feedback messages for upper elementary students based on natural-language-processing features of their first drafts. The central claim is that this adaptive feedback improves students' use of text evidence when they revise. In a pilot deployment across seven fifth- and sixth-grade classrooms, 143 students' revised essays received higher human rubric scores for evidence use than their first drafts, and the NLP measures of evidence quantity and specificity improved significantly. The paper argues that NLP-driven formative feedback can make a substantive writing dimension—using text evidence from a source—more learnable, without requiring extra teacher time.

What carries the argument

The load-bearing component is the feedback selection pipeline built on the SG automatic essay scoring model. The model represents each essay with two interpretable features: $\mathrm{NPE}$, the number of article topics a sliding window matches via word embeddings, and $\mathrm{SPC}$, a vector counting matched examples per category. eRevise pools $\mathrm{SPC}$ into $\mathrm{SPC_{total}}$, computes a duplication rate $\mathrm{DR}$ from merged unique examples, derives $\mathrm{SPC_{AWE}}$ for four primary topics, and buckets it into low, medium, or high. A lookup table built by content experts maps each pair of $\mathrm{NPE}$ and $\mathrm{SPC_{lmh}}$ to two of four predefined feedback messages addressing evidence quantity, specificity, explanation, and elaboration. These messages are what students see during revision.

What would settle it

A randomized trial comparing three revision conditions—eRevise adaptive feedback, generic feedback, and no feedback—would settle it. If students who revise with generic feedback or no feedback improve their evidence scores as much as students who receive eRevise's adaptive messages, the central claim fails.

Watch

Extended reading notes

Core claim

The paper reports that eRevise's rubric-aligned scoring features, specifically the number of evidence topics ($\mathrm{NPE}$) and a duplication-adjusted count of specific article examples ($\mathrm{SPC_{AWE}}$), can be converted into a small set of targeted feedback messages. In the pilot, human-scored Evidence quality improved from a mean of 2.62 to 2.72 ($p \le 0.08$), while the $\mathrm{NPE}$ feature rose significantly ($p \le 0.003$) and the merged specificity count rose significantly ($p \le 0.001$). Students whose drafts received the two least-sophisticated feedback messages showed the largest score gains ($p = 0.02$). The paper concludes that eRevise helped students improve text evidence usage after receiving formative feedback and engaging in revision.

Load-bearing premise

The improvements are attributed to the adaptive feedback itself, but the study has no control condition, so revising with any feedback—or simply revising—might have produced the same gains.

Editorial extensions

If this is right

  • If the result holds, a rubric-based automatic essay scoring system can double as an instructional tool, giving teachers a low-cost way to support evidence use in writing.
  • Students with the least sophisticated evidence use stand to benefit most, as the largest gains occurred in drafts receiving the first two feedback messages.
  • NLP feature counts such as $\mathrm{NPE}$ and $\mathrm{SPC_{Total\,Merged}}$ can detect improvement even when holistic rubric scores appear to hit a ceiling.
  • Because no other RTA dimensions improved, the feedback appears dimension-specific: targeting evidence use improves evidence use, not writing quality in general.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same selection pipeline could be applied to other response-to-text prompts if the topical word lists are generated automatically; the paper reports pilot data-driven extraction methods that still need refinement.
  • Hand-designed lookup tables could be learned from scored corpora, potentially replacing the content-expert mapping between feature values and feedback messages.
  • If students respond to feedback by copying more article text verbatim, evidence scores may rise mechanically; the paper acknowledges eRevise currently does not detect plagiarism, so future versions would need to address this.
  • The ceiling-effect analysis suggests fine-grained NLP features should complement rubric scores when evaluating writing interventions in elementary classrooms.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper presents eRevise, a web-based automated writing evaluation (AWE) environment for grades 5–6. eRevise uses an existing rubric-based automatic essay scoring (AES) system (the SG word-embedding model) to extract features — Number of Pieces of Evidence (NPE) and a merged specificity count (SPC Total Merged) — and then selects two of four predefined feedback messages via a hand-built lookup table. In a pilot deployment across seven classrooms (143 students), the authors test whether students improve from first to second drafts on human rubric Evidence scores (H1) and on the NLP feature counts (H2). They report a trending improvement on the human rubric (2.62 to 2.72, p ≤ 0.08), significant improvements on NPE (2.61 to 2.81, p ≤ 0.003) and SPC Total Merged (9.65 to 11.15, p ≤ 0.001), and no significant changes on the other four RTA dimensions. The authors acknowledge in the Current and Future Directions section that the next deployment will include a generic-feedback control condition. The central claim is that eRevise's adaptively selected feedback improves text evidence usage.

Significance. If the causal claim were established, eRevise would be a notable contribution to automated writing evaluation for upper elementary students, particularly for its focus on evidence use, its rubric-aligned interpretable features, and its use of NLP-generated feedback rather than scores alone. The paper has clear strengths: it validates the choice of the interpretable SG model over the better-performing CO-ATTN model, it provides a concrete feedback-selection algorithm with pre-specified thresholds, it reports classroom-deployment data with paired pre-post design and human rubric scoring, and it publicly acknowledges the lack of a control condition. The differential null results on non-targeted RTA dimensions provide some useful evidence that the observed gains are not merely a general writing-improvement effect. However, as presented, the evidence supports only a descriptive 'students improved after using eRevise' conclusion; the abstract's causal phrasing overstates what the data can establish.

major comments (3)
  1. [Experimental Deployment and Results, H2 analysis] The study lacks any control condition. The pre-post gains on the Evidence rubric (2.62 to 2.72, p ≤ 0.08), NPE (2.61 to 2.81, p ≤ 0.003), and SPC Total Merged (9.65 to 11.15, p ≤ 0.001) could be produced by the opportunity to revise, by teacher scaffolding in the second class period, or by task familiarity. The authors explicitly state in Current and Future Directions that a generic-feedback control condition has been added for the next deployment, which confirms that the current design cannot support the causal claim 'eRevise helped students improve their text evidence usage.' The abstract and conclusions should be reworded to describe a preliminary feasibility study with descriptive pre-post comparisons, or the authors should provide a compelling post-hoc argument ruling out these alternative explanations (e.g., using the non-targeted RTA dimensions as a control is only partially convincing because those dimensions did not receive feedback).
  2. [Experimental Deployment and Results] The H2 outcome measures (NPE and SPC Total Merged) are the very features used by the AWE feedback-selection algorithm. Students who receive 'Use more evidence' and 'Provide more details' can increase these counts by literally copying more phrases from the article, which the authors themselves note is a limitation (see the plagiarism discussion in Current and Future Directions). The significant NPE and SPC Total Merged gains are therefore partly mechanical compliance with the feedback, not evidence of improved evidentiary reasoning. The human-rubric result (H1) is not circular, but it is only trending (p ≤ 0.08). The paper should present H1 as the primary test of learning, treat H2 as a manipulation-check or compliance measure, and add an analysis that distinguishes, for example, copied phrases from paraphrased evidence, or that uses the human rubric's Relevance and Specificity subcomponents rather than the NLP counts.
  3. [Experimental Deployment and Results] The only statistically significant human-score improvement appears in the subgroup of 45 students receiving feedback messages 1 and 2 (2.33 to 2.64, p = 0.02), while the other two subgroups (messages 2 and 3; messages 3 and 4) show no improvement. Because this subgroup is selected by the feedback algorithm on the basis of the students' low initial evidence usage, the improvement may be inflated by regression to the mean. The paper should report effect sizes (e.g., Cohen's d) for the overall and subgroup comparisons, and should discuss regression-to-the-mean as a plausible alternative explanation for the subgroup result, especially given the overall p-value of 0.08.
minor comments (3)
  1. [AWE Feedback Selection] The column headers 'SCP5' through 'SCP8' appear to be typos for 'SPC5' through 'SPC8'; the text consistently uses 'SPC' elsewhere.
  2. [AWE Feedback Selection] The function 'RN D' is not defined; it should be stated as rounding to the nearest integer.
  3. [System Usage and Architecture] The text 'RT AM V P' has unusual spacing; it should read 'RTA_MVP' (and similarly for 'RTAspace' in Current and Future Directions).

Circularity Check

1 steps flagged · score 6.0 of 10

NLP-feature outcomes (H2) are the same variables used to select feedback, so the strongest reported gains are partly built into the intervention; the only independent outcome (H1 human rubric) is marginal, and the paper itself concedes that the pilot lacked an adaptive-vs-generic control.

  1. self definitional [AES Feature Extraction / AWE Feedback Selection (Eqs. 1-5, Table 5) and Experimental Deployment and Results (H2)]
    "In particular, two of the features used by SG for score prediction, namely Number of Pieces of Evidence (NPE) and Specificity (SPC), form the basis of eRevise's feedback selection algorithm. ... We thus explore the use of more fine-grained outcome measures that have a stronger relationship to the eRevise feedback that guided student revision. To test H2, we use the NPE and SPC Total Merged features as automatically computed by eRevise during its deployment to approximate evidence quantity and relevance/specificity, respectively."

    Feedback selection is driven by NPE and SPC Total Merged: NPE is the breadth-of-topics input to Table 5, and SPC_AWE (Eq. 4) is derived from SPC Total Merged via the duplication rate. H2 then evaluates the intervention using these same features as outcomes. The feedback itself instructs students to 'Choose at least three different pieces of evidence' (message 1) and 'Add more specific details about each piece of evidence' (message 2), so a compliant revision will by definition raise NPE and SPC Total Merged. The significant H2 gains are therefore partly mechanical confirmation of the feedback's literal instructions, not independent evidence that evidence-use quality improved.

full rationale

The paper is transparent about its design and reports an honest pilot. The NLP/AWE engineering is largely self-contained: the feedback-selection rules are explicit, and the AES model is prior work. However, the strongest statistical results (NPE p <= 0.003; SPC Total Merged p <= 0.001) use outcome measures that are the same constructs used to select the feedback, so those results are partly circular: the treatment tells students to do exactly what the measure counts. The human-rubric outcome is independent but only marginal, and the paper explicitly states in Current and Future Directions that the next deployment adds a generic-feedback control condition 'to better determine the benefit of using AES to adaptively guide revision', conceding that the present pilot cannot isolate the adaptive feedback's causal contribution. The null results on the other four RTA dimensions are consistent with evidence-specific improvement but do not replace a control arm. Overall, the central claim rests on one independent but weak result and one self-referential strong result, so partial circularity is present.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central evaluation depends on hand-crafted feature extraction resources (topic and example lists, thresholds, window size), the assumption that feedback messages map validly to feature values, and the absence of a control condition. The most consequential free parameters are the manually defined lookup table and the feature extraction lists, which are fitted to the training corpus and not published.

free parameters (6)
  • window_size = 6
    Sliding window size for NPE and SPC feature extraction, chosen to optimize AES performance on previously scored training essays.
  • word_similarity_threshold = not reported
    Two words are considered similar after thresholding word embedding similarity; the threshold value is not specified in the paper.
  • important_category_indices = not reported
    Start and end indices S and E for the four primary SPC categories (hospital, malaria, farming, school) in Equation 2; the exact indices are not given.
  • SPClmh_category_thresholds = 3 and 5
    Cutoffs in Equation 5 that map SPCAWE to low, medium, or high feedback categories.
  • feedback_lookup_table = Table 5 mapping
    Manually designed by content experts using a previously scored corpus as development data; not derived from theory.
  • manual_topic_and_example_lists = not reported
    Manually crafted lists of main topics and article examples used for NPE and SPC extraction; the lists are not included in the paper.
assumptions (6)
  • domain assumption The NPE and SPC Total Merged features accurately operationalize breadth and specificity of evidence use as defined by the RTA Evidence rubric.
    AWE feedback selection was guided by three assumptions, the first two being that NPE indicates breadth of unique topics and SPC Total Merged indicates the number of unique pieces of evidence.
  • domain assumption The four feedback messages and their grouping into pairs follow a valid developmental progression for evidence use.
    Messages were created by content experts based on research frameworks and analysis of previously scored essays; no empirical validation of the progression is provided.
  • domain assumption Students who receive feedback revise their essays in good faith and understand the feedback messages.
    The evaluation assumes students read and act on the displayed messages; no log data or compliance measure is reported.
  • domain assumption Improved scores on the RTA Evidence rubric reflect improved writing quality.
    The human rubric is treated as the ground truth for writing quality.
  • domain assumption Word embedding similarity thresholds and vector semantics transfer to elementary student writing.
    The SG model uses skip-gram embeddings to compute lexical and semantic similarity; behavior on noisy, misspelled student text is assumed adequate.
  • standard math Paired t-test assumptions are met for the pre-post comparisons.
    The paper uses paired t-tests on n=143 without reporting normality checks or effect sizes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of eRevise: Using Natural Language Processing to Provide Formative Feedback on Text Evidence Usage in Student Writing." pith.science (2026). https://pith.science/paper/CN3QWL6N

@misc{pith2026190801992,
  author       = {Pith},
  title        = {Pith review of: eRevise: Using Natural Language Processing to Provide Formative Feedback on Text Evidence Usage in Student Writing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CN3QWL6N}},
  note         = {Machine review of arXiv:1908.01992}
}
read the original abstract

Writing a good essay typically involves students revising an initial paper draft after receiving feedback. We present eRevise, a web-based writing and revising environment that uses natural language processing features generated for rubric-based essay scoring to trigger formative feedback messages regarding students' use of evidence in response-to-text writing. By helping students understand the criteria for using text evidence during writing, eRevise empowers students to better revise their paper drafts. In a pilot deployment of eRevise in 7 classrooms spanning grades 5 and 6, the quality of text evidence usage in writing improved after students received formative feedback then engaged in paper revision.

Figures

Figures reproduced from arXiv: 1908.01992 by the authors.

Figure 1
Figure 1. The architecture of the eRevise system. the Millenium Villages Project (MVP) to eradicate poverty in a Kenyan village.2 After the teacher discusses predefined vocabulary and asks standardized questions at designated points, there is a prompt at the end of the article which asks students: “Based on the article, did the author provide a con￾vincing argument that winning the fight against poverty is achievable in our l… view at source ↗
Figure 2
Figure 2. A formative feedback screenshot of the eRevise system. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. (a) RTA Evidence scores before and after revision. (b) Value changes for the NPE feature. (c) Value changes for the [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 19 canonical work pages

  1. [1]

    A.; Cotos, E.; and Lee, J

    Chapelle, C. A.; Cotos, E.; and Lee, J. 2015. Validity arguments for diagnostic assessment using automated writing evaluation. Language testing 32(3):385--405

    work page 2015

  2. [2]

    Condon, W. 2013. Large-scale assessment, locally-developed measures, and automated scoring of essays: Fishing for red herrings? Assessing Writing 18(1):100--108

    work page 2013

  3. [3]

    Correnti, R.; Matsumura, L.; Hamilton, L.; and Wang, E. 2013. Assessing students' skills at writing analytically in response to texts. Elementary School Journal 114:142--177

    work page 2013

  4. [4]

    Dikli, S. 2006. An overview of automated scoring of essays. The Journal of Technology, Learning and Assessment 5(1):4--35

    work page 2006

  5. [5]

    W., and Rosenstein, M

    Foltz, P. W., and Rosenstein, M. 2015. Analysis of a large-scale formative writing assessment system with automated feedback. In Proceedings of the Second (2015) ACM Conference on Learning@ Scale , 339--342. ACM

    work page 2015

  6. [6]

    R.; and Santangelo, T

    Graham, S.; Harris, K. R.; and Santangelo, T. 2015. based writing practices and the common core: Meta-analysis and meta-synthesis. The Elementary School Journal 115(4):498--522

    work page 2015

  7. [7]

    Liu, L.; Brew, C.; Blackmore, J.; Gerard, L.; Madhok, J.; and Linn, M. 2014. Automated scoring of constructed-response science items prospects and obstacles. Educational Measurement: Issues and Practice 33(2):19--28

    work page 2014

  8. [8]

    Loukina, A.; Zechner, K.; Chen, L.; and Heilman, M. 2015. Feature selection for automated speech scoring. In Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications , 12--19

    work page 2015

Show all 20 references
  1. [9]

    Perelman, L. 2012. Construct validity, length, score, and time in holistically graded writing assessments: The case against automated essay scoring. International advances in writing research: Cultures, places, measures 121--131

    2012

  2. [10]

    Rahimi, Z., and Litman, D. 2016. Automatically extracting topical components for a response-to-text writing assessment. In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications , 277--282

    2016

  3. [11]

    J.; Correnti, R.; Matsumura, L

    Rahimi, Z.; Litman, D. J.; Correnti, R.; Matsumura, L. C.; Wang, E.; and Kisa, Z. 2014. Automatic scoring of an analytical response-to-text assessment. In International Conference on Intelligent Tutoring Systems , 601--610. Springer

    2014

  4. [12]

    Rahimi, Z.; Litman, D.; Correnti, R.; Wang, E.; and Matsumura, L. C. 2017. Assessing students’ use of evidence and organization in response-to-text writing: Using natural language processing for rubric-based automated scoring. International Journal of Artificial Intelligence i...

    2017

  5. [13]

    D.; Allen, L

    Roscoe, R. D.; Allen, L. K.; Weston, J. L.; Crossley, S. A.; and McNamara, D. S. 2014. The writing pal intelligent tutoring system: Usability testing and development. Computers and Composition 34:39--59

    2014

  6. [14]

    D., and Burstein, J

    Shermis, M. D., and Burstein, J. C. 2003. Automated essay scoring: A cross-disciplinary perspective . Routledge

    2003

  7. [15]

    D., and Burstein, J

    Shermis, M. D., and Burstein, J. 2013. Handbook of automated essay evaluation: Current applications and new directions . Routledge

    2013

  8. [16]

    C.; and Correnti, R

    Wang, E.; Matsumura, L. C.; and Correnti, R. 2018. Student writing accepted as high-quality responses to analytic text-based writing tasks. The Elementary School Journal 118(3):357--383

    2018

  9. [17]

    Weigle, S. C. 2013. English language learners and automated scoring of essays: Critical considerations. Assessing Writing 18(1):85--99

    2013

  10. [18]

    Zhang, H., and Litman, D. 2017. Word embedding for response-to-text assessment of evidence. In Proceedings of ACL 2017, Student Research Workshop , 75--81

    2017

  11. [19]

    Zhang, H., and Litman, D. 2018. Co-attention based neural network for source-dependent essay scoring. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications , 399--409

    2018

  12. [20]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.