Word Embedding for Response-To-Text Assessment of Evidence

Diane Litman; Haoran Zhang

REVIEW 4 major objections 6 minor 19 references

Word Embedding for Response-To-Text Assessment of Evidence

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that adding a skip-gram word-embedding layer to the rubric feature extractor improves automated Evidence scoring of elementary students' essays on most corpora, with the largest gains on the noisiest responses.

desk verdict A modest, honest empirical extension of rubric-based RTA evidence scoring with word embeddings; the gains are plausible but rest on one fixed data split and small noisy corpora. read the letter →

arxiv 1908.01969 v1 pith:KJ64RSFL submitted 2019-08-06 cs.CL cs.AI

Haoran Zhang , Diane Litman This is my paper · ORCID

classification cs.CLcs.AI

keywords wordembeddingsskip-gramResponse-to-TextAssessmentevidencescoringrubric-basedlexicalmatchingquadraticweightedkappaelementarystudentessays

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a simple supplement to an interpretable evidence-scoring system—replacing exact word matches with word-embedding similarity—improves automated scoring of the Response-to-Text Assessment Evidence rubric. The motivation is practical: elementary students write noisy essays with misspellings and their own vocabulary, so exact lexical matching leaves valid evidence undetected. The claimed payoff is a modest but real increase in quadratic weighted kappa on most corpora, strongest for lower-grade responses, with no loss in interpretability. The paper also argues that embeddings trained on the target population generalize across age groups better than off-the-shelf embeddings.

What carries the argument

The central machinery is the skip-gram word embedding model, which maps each word to a vector so that words appearing in similar contexts sit close together. The feature extractor supplements the rubric's exact-match window search by computing cosine similarity between an essay word and each word in the hand-crafted topic and example lists, counting a match if similarity exceeds a threshold chosen on the development set. This is what lets "proverty" match "poverty", "went" match "go", and "mosquitoes" match "malaria", while still keeping the features themselves interpretable.

What would settle it

Take the subset of essays where the skip-gram model changed the score, remove every word matched only through embedding similarity, and have expert raters judge whether the removed evidence is rubric-relevant; if the matches are no more relevant than random pairs, the claimed improvement is an artifact.

Watch

Extended reading notes

Core claim

The paper's central claim is that a word-embedding layer can improve automated scoring of the Evidence dimension of the Response-to-Text Assessment without giving up the interpretability of the rubric features. Concretely, the skip-gram embedding trained on the students' own essays yields a higher or equal quadratic weighted kappa than the lexical-matching rubric baseline on most corpora, and the gain is significant on the lower-grade corpus and on the combined corpus; on the older-grade corpus the embedding model does not beat the baseline. A second claim is that this task-trained model generalizes across age groups better than either the rubric baseline or off-the-shelf embeddings. The paper attributes the gains to the model's ability to recognize misspellings, inflectional variants, and some paraphrases that exact string matching misses.

Load-bearing premise

The load-bearing premise is that nearness in the embedding space, above a threshold selected on a development set, tracks rubric-relevant evidence rather than mere topic relatedness; if that mapping fails, the method's score gains are not real evidence-detection.

Editorial extensions

If this is right

Evidence scoring can be improved by swapping the matching layer while leaving the rubric features unchanged, so the method can be dropped into existing RTA scoring pipelines.
On lower-grade corpora the improvement is significant, suggesting semantic matching matters most where writing is noisiest.
Task-trained skip-gram embeddings transfer across age groups better than both the rubric baseline and off-the-shelf embeddings, supporting the idea that population-matched embeddings are worth training even on small corpora.
On the older-student corpus the embedding model does not beat the rubric baseline, so for cleaner writing exact matching may already be sufficient.
Skip-gram consistently beats continuous bag-of-words, attributable to better handling of infrequent words that dominate student evidence vocabulary.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same thresholded-embedding layer could be applied to other RTA rubric dimensions that use lexical matching, such as Organization, but the paper does not test this.
A decisive test of the mechanism would compare embedding-matched evidence against a second human rater's judgments, since the paper evaluates only against first-rater scores.
The embedding space could double as a population-specific spell normalizer: because the model is trained on student typos, relations like "proverty" to "poverty" could be harvested to build a spelling-error dictionary.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is a legitimate incremental contribution: it takes Rahimi et al.'s rubric features for evidence scoring and relaxes exact lexical matching using word embeddings trained on the students' own essays. The new empirical comparison on three elementary RTA corpora is real, and the authors are honest about the mixed results. The skip-gram model improves QWKappa on the lower-grade corpus and matches or slightly beats the rubric baseline on most corpora, but not on M V PH. The cross-corpus generalization table is the most interesting part, and there the task-specific embedding does best. I think the central claim holds up as a promising direction, not as a definitive win.

What the paper does well: the motivation is concrete (misspellings, paraphrases, tense variants), the method is simple enough to reproduce from the description, and the authors discuss failure cases themselves, including the over-matching of "water" and "electricity". The comparison against two off-the-shelf embeddings is the right control, and the finding that a small in-domain corpus can beat a much larger generic one is worth reporting. The results are not oversold.

Soft spots: the stress-test note is on target. All reported results come from a single fixed 40/20/40 split. The 10 runs of 10-fold CV on the test partition capture classifier variance only; they do not capture variance in embedding training or threshold selection. On a corpus where first-rater kappa is 0.338 (Space), the measured QWKappa differences are small enough that a different split could easily change the ranking. The lack of code, data, and error bars makes this hard to check. I also find the significance claims hard to interpret: Table 3 marks significance over other models, but without variance across splits, these are conditional on one split. That said, the authors did use a development set for threshold selection, so the evaluation is not circular.

Who is this for: researchers working on rubric-based automated essay scoring, especially for noisy elementary writing. It is not a methods paper and does not claim to be.

Recommendation: it deserves a serious referee. The idea is sensible, the experiments are mostly transparent, and the published version of this line of work has been used by others. I would ask the authors for multiple splits, feature-level ablation, and ideally released code or at least the crafted word lists. A revision with those additions would be worth publishing.

Referee Report

4 major / 6 minor

Summary. The paper proposes augmenting the interpretable rubric-based features of Rahimi et al. (2014) for scoring the Evidence dimension of the Response-to-Text Assessment with word2vec embeddings (skip-gram and CBOW) trained on the task corpus. Cosine similarity above a development-tuned threshold is used to relax exact lexical matching between student essays and manually crafted topic/example lists. The authors compare against the original rubric features and two off-the-shelf embeddings (Mikolov, Baroni) on four RTA corpora (Space, MVP L, MVP H, MVP ALL) using quadratic weighted kappa, and they run cross-corpus transfer experiments. They report that the skip-gram embedding improves over the rubric baseline on Space, MVP L, and MVP ALL, but not MVP H, and that the task-trained embedding is competitive with Baroni's embedding.

Significance. If the claims hold, the paper offers a simple, interpretable, and low-cost way to improve evidence scoring by a small but consistent margin on lower-grade and lower-writing-proficiency corpora, while preserving the interpretability of the rubric features. The comparisons with off-the-shelf embeddings and the cross-corpus transfer experiments are informative for practitioners, and the honest reporting of the MVP H failure and the relatedness-versus-similarity limitation is a strength. The contribution is incremental over Rahimi et al. (2014), and the empirical basis is currently too fragile to constitute a strong demonstration: there is no code or data release, no multi-split variance analysis, no confidence intervals, and no analysis of the matching errors introduced by the embedding threshold. These gaps are fixable within the scope of the manuscript.

major comments (4)

[Section 6 (Experimental Setup)] The evaluation rests on a single 40/20/40 stratified split, with the 20% development set used to select both the embedding model and the cosine threshold; the final QWKappa numbers and the reported pairwise significance are computed within this one partition. The manuscript does not specify how the '10 runs of 10-fold cross validation in the final testing' relate to the held-out 40%: if the 40% partition is the test set, applying 10-fold CV to it is not a clean held-out evaluation, and if CV is run over the whole corpus, the classifiers see data used for embedding training and threshold selection. In either reading, the reported variability covers only classifier training, not embedding training, threshold selection, or partitioning. Given the small gains (e.g., Space 0.606 to 0.611 and MVP L 0.628 to 0.682) and the noisy labels (first-rater Kappa 0.338 on Space), the central claim is not yet robust. Please rerun the full pipeline over multiple independent splits or use bootstrap resampling, and report mean QWKappa with confidence intervals or error bars.
[Section 5 (Word Embedding Feature Extraction)] The only mechanism for relaxing exact matching is the cosine-similarity threshold, a single global number tuned on the development set; no threshold value, no distribution of cosine similarities, and no error analysis of the matches introduced by the threshold are reported. The authors themselves note in Section 8 that the embedding matches unwanted related pairs such as 'water' and 'electricity'. Because the rubric defines evidence relevance rather than distributional relatedness, the improvement could come from a fortuitous threshold level rather than from a meaningful semantic proxy. Please add an ablation or error analysis showing which newly matched pairs contribute to score changes, and report the threshold values selected on the development set.
[Section 7 (Results and Discussion)] The paper reports pairwise significance with superscripts but does not state which statistical test is used, whether it is corrected for multiple comparisons, or whether it accounts for the fact that all models are evaluated on the same folds. On MVP H, for example, skip-gram (0.583) is significantly worse than both the rubric baseline (0.599) and Baroni (0.593), which is consistent with the authors' hedge but makes the abstract-level phrase 'improving evidence scoring' too strong. More importantly, without standard deviations or confidence intervals, the reader cannot tell whether the differences in Table 3 are meaningful. Please report the exact test, effect sizes, and variability across runs.
[Section 3 (Data) and Table 1] The use of first-rater scores as the gold standard is understandable because the first rater graded more essays, but with Kappa as low as 0.338 on Space, the evaluation is against a noisy target. The paper should discuss the ceiling imposed by rater agreement and ideally report QWKappa against both raters on the double-rated portion, or at least discuss how rater noise affects the comparison between systems.

minor comments (6)

[Abstract] The phrase 'students writing quality' should be 'students' writing quality'.
[Throughout] The notation for corpus names is inconsistent: 'M V PH' appears with spaces in some places and without them in others; please use a single consistent notation, such as 'MVP-H'.
[Section 6 (Experimental Setup)] The sentence 'We use 10 runs of 10-fold cross validation in the final testing' is ambiguous and should be rewritten to state explicitly where the cross-validation is applied and how the 40/20/40 split relates to it.
[Section 5 (Word Embedding Feature Extraction)] The sentence 'students use their own vocabularies other than words in the crafted list' is grammatically awkward; consider 'students use their own vocabulary rather than the words in the crafted list'.
[References] Some references are formatted inconsistently, for example Rahimi et al. (2017) is cited in Table 2 as '(Rahimi et al., 2017)' but in the text as 'Rahimi et al. (2017)'; please standardize.
[Figure 1] The reproduced student essay contains intentional spelling errors, but a note in the caption clarifying that these are original student errors would help avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the embedding-augmented evidence scoring is evaluated on held-out test data with a development-set-selected threshold.

full rationale

The paper's central claim is that supplementing rubric-based feature extraction with word embeddings improves evidence scoring. The derivation chain is empirical rather than definitional. The word embedding model is trained on a 40% partition, the matching threshold is selected on a separate 20% development partition, and the reported QWKappa values are computed on a disjoint 40% test partition using 10 runs of 10-fold cross-validation. The threshold is not fitted to the test labels, and the final evaluation is not a restatement of any fitted quantity. The only substantial self-citations are to Rahimi et al. (2014), which supplies the rubric feature set and the baseline called 'Rubric'; this prior work is used as a comparison point and as the feature backbone, not as a justification that the new contribution succeeds. The paper also directly acknowledges the limitation that embeddings match unwanted related words (e.g., 'water' with 'electricity'), which is an empirical weakness rather than a circularity. No equation reduces a prediction to its input, and no parameter is renamed as a prediction. Thus no specific circular step can be quoted, and the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard NLP assumptions about embeddings, hand-crafted feature lists, and the validity of first-rater labels. The only fitted numeric parameters are the similarity threshold and embedding hyperparameters, both selected on the development set. No new entities are introduced.

free parameters (2)

cosine similarity threshold = not reported
Selected on the development set (Section 6). Controls which word pairs count as evidence matches; central to the method.
word embedding hyperparameters (dimension, window size, epochs, min count) = not reported
Multiple word embedding models with different parameters were trained and the best was selected on the development set (Section 6), so hyperparameters are effectively tuned to the data.

assumptions (4)

domain assumption Word2vec cosine similarity above a threshold is a valid proxy for evidence relevance in elementary essays.
Section 5 states words with high similarity are considered a match; this is the core modeling premise.
domain assumption The manually crafted topic and example lists are complete and correct.
Section 4 uses these lists for NPE, CON, and SPC features; they are not included in the paper.
domain assumption The first rater's score is an acceptable gold standard despite low inter-rater agreement.
Section 3 reports Kappa as low as 0.338 and uses only first rater scores as labels without addressing the implications.
domain assumption The development-set-selected threshold generalizes to test sets.
Section 6 selects the threshold on a 20% dev split and applies it to the 40% test split.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Word Embedding for Response-To-Text Assessment of Evidence." pith.science (2026). https://pith.science/paper/KJ64RSFL

@misc{pith2026190801969,
  author       = {Pith},
  title        = {Pith review of: Word Embedding for Response-To-Text Assessment of Evidence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KJ64RSFL}},
  note         = {Machine review of arXiv:1908.01969}
}

read the original abstract

Manually grading the Response to Text Assessment (RTA) is labor intensive. Therefore, an automatic method is being developed for scoring analytical writing when the RTA is administered in large numbers of classrooms. Our long-term goal is to also use this scoring method to provide formative feedback to students and teachers about students' writing quality. As a first step towards this goal, interpretable features for automatically scoring the evidence rubric of the RTA have been developed. In this paper, we present a simple but promising method for improving evidence scoring by employing the word embedding model. We evaluate our method on corpora of responses written by upper elementary students.

Figures

Figures reproduced from arXiv: 1908.01969 by the authors.

**Figure 1.** The prompt of RT AMV P and an example essay with score of 3. Finally, tenses used by students can sometimes be different from that of the article. Although a stemming algorithm can solve this problem, sometimes there are words that slip through the process. For example, “went” is the past tense of “go”, but stemming would miss this conjugation. Therefore, “go” and “went” would not be considered a match. To address… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 15 canonical work pages

[1]

Yigal Attali and Jill Burstein. 2006. Automated essay scoring with e-rater v. 2. The Journal of Technology, Learning and Assessment\/ 4(3)

work page 2006
[2]

Marco Baroni, Georgiana Dinu, and Germ \'a n Kruszewski. 2014. Don't count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In ACL (1)\/ . pages 238--247

work page 2014
[3]

Jill Burstein, Karen Kukich, Susanne Wolff, Chi Lu, and Martin Chodorow. 2001. Enriching automated essay scoring using discourse marking

work page 2001
[4]

Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 2002. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research\/ 16:321--357

work page 2002
[5]

Richard Correnti, Lindsay Clare Matsumura, Laura Hamilton, and Elaine Wang. 2013. Assessing students' skills at writing analytically in response to texts. The Elementary School Journal\/ 114(2):142--177

work page 2013
[6]

Tom Kenter and Maarten de Rijke. 2015. Short text similarity with word embeddings. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management\/ . ACM, pages 1411--1420

work page 2015
[7]

Douwe Kiela, Felix Hill, and Stephen Clark. 2015. Specializing word embeddings for similarity or relatedness. In EMNLP\/ . pages 2044--2048

work page 2015
[8]

Annie Louis and Derrick Higgins. 2010. Off-topic essay detection using short prompt texts. In Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications\/ . Association for Computational Linguistics, pages 92--95

work page 2010

Show all 19 references

[9]

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013 a . Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781\/

2013 arXiv
[10]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013 b . Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems\/ . pages 3111--3119

2013
[11]

Nathan Ong, Diane Litman, and Alexandra Brusilovsky. 2014. Ontology-based argument mining and automatic essay scoring. In Proceedings of the First Workshop on Argumentation Mining\/ . pages 24--28

2014
[12]

Zahra Rahimi, Diane Litman, Richard Correnti, Elaine Wang, and Lindsay Clare Matsumura. 2017. Assessing students’ use of evidence and organization in response-to-text writing: Using natural language processing for rubric-based automated scoring. International Journal of Artifi...

2017
[13]

Zahra Rahimi, Diane J Litman, Richard Correnti, Lindsay Clare Matsumura, Elaine Wang, and Zahid Kisa. 2014. Automatic scoring of an analytical response-to-text assessment. In International Conference on Intelligent Tutoring Systems\/ . Springer, pages 601--610

2014
[14]

Marek Rei and Ronan Cummins. 2016. Sentence similarity measures for fine-grained estimation of topical relevance in learner essays. arXiv preprint arXiv:1606.03144\/

2016 arXiv
[15]

Mark D Shermis and Jill C Burstein. 2003. Automated essay scoring: A cross-disciplinary perspective\/ . Routledge

2003
[16]

Ian H Witten, Eibe Frank, Mark A Hall, and Christopher J Pal. 2016. Data Mining: Practical machine learning tools and techniques\/ . Morgan Kaufmann

2016
[17]

Shasha Xie, Keelan Evanini, and Klaus Zechner. 2012. Exploring content features for automated speech scoring. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies\/ . Association for C...

2012
[18]

, " * write output.state after.block = add.period write newline

ENTRY address author booktitle chapter doi edition editor howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.se...
[19]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Yigal Attali and Jill Burstein. 2006. Automated essay scoring with e-rater v. 2. The Journal of Technology, Learning and Assessment\/ 4(3)

work page 2006

[2] [2]

Marco Baroni, Georgiana Dinu, and Germ \'a n Kruszewski. 2014. Don't count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In ACL (1)\/ . pages 238--247

work page 2014

[3] [3]

Jill Burstein, Karen Kukich, Susanne Wolff, Chi Lu, and Martin Chodorow. 2001. Enriching automated essay scoring using discourse marking

work page 2001

[4] [4]

Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 2002. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research\/ 16:321--357

work page 2002

[5] [5]

Richard Correnti, Lindsay Clare Matsumura, Laura Hamilton, and Elaine Wang. 2013. Assessing students' skills at writing analytically in response to texts. The Elementary School Journal\/ 114(2):142--177

work page 2013

[6] [6]

Tom Kenter and Maarten de Rijke. 2015. Short text similarity with word embeddings. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management\/ . ACM, pages 1411--1420

work page 2015

[7] [7]

Douwe Kiela, Felix Hill, and Stephen Clark. 2015. Specializing word embeddings for similarity or relatedness. In EMNLP\/ . pages 2044--2048

work page 2015

[8] [8]

Annie Louis and Derrick Higgins. 2010. Off-topic essay detection using short prompt texts. In Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications\/ . Association for Computational Linguistics, pages 92--95

work page 2010