{"id":"1da86b6a-491e-40bc-a7c4-8dc05e2ce2fa","arxiv_id":"1908.01969","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Using word embeddings to relax exact word matching in rubric features modestly improves automated evidence scoring for elementary response-to-text essays, with task-trained skip-gram best on lower-grade and cross-age tests.","lead":"This paper tests whether word embeddings, which represent word meanings as vectors, improve automatic grading of how well elementary students use evidence from a source text. The method helps on some student groups, but the gains are small and inconsistent across corpora.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single fixed 40/20/40 split underlies the reported QWKappa gains; rerunning the full pipeline on multiple splits is needed to confirm the claim.","rationale":"The paper's contribution is empirical: a task-dependent skip-gram embedding plus thresholded cosine matching improves rubric-based evidence scoring. The load-bearing condition is that the QWKappa advantage in Table 3 is real and reproducible. The current protocol does not demonstrate this because the word embeddings and threshold are fixed by one split; the repeated 10-fold CV on the same test slice only reduces classifier noise, not the variance from the embedding-training split or threshold selection. This concern is partially aligned with the reader's emphasis on the threshold as a fragile proxy, but it is more fundamental: even if thresholded cosine similarity were a perfect proxy for rubric relevance, a single favorable split could produce the reported significant differences. The paper's own conclusion concedes that the embedding matches are too broad (\"water\" vs. \"electricity\"), which makes split-sensitivity more likely. The proposed resampling test would settle whether the improvement generalizes. If it does, the paper is a modest but sound empirical contribution; if not, the central claim fails. The issue is the evaluation design, not the authors' intent.","tokens_in":7958,"tokens_out":8962,"duration_ms":93321,"concrete_test":"Repeat the entire procedure with 10 new random 40/20/40 splits stratified by score. For each split, retrain the word embeddings on the 40% slice, retune the threshold on the 20% dev slice, and evaluate the final classifier on the remaining 40% using the same 10x10-fold CV. Report the mean and 95% confidence interval of the QWKappa difference between skip-gram and Rubric baseline. If the CI for MVP L or MVP ALL includes zero or is negative, the Table 3 improvement is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 7 (Table 3) is that skip-gram embeddings improve or match the rubric baseline on most corpora. The experimental protocol in Section 6, however, uses one fixed stratified split: 40% for training word embeddings, 20% for selecting the model and threshold, and 40% for final testing. The reported significance comes from 10 runs of 10-fold CV on that single 40% test partition, so it only accounts for classifier variance, not for variance in embedding training or threshold selection. On small, noisy data (e.g., Space has first-rater Kappa=0.338), a different split could yield different embeddings and a different cosine threshold. Moreover, the threshold is a single global number tuned on 20% of the data, and the authors themselves note in the conclusion that the embedding matches unwanted related words such as \"water\" and \"electricity\". If the observed gains on MVP L and MVP ALL are the product of a favorable split or a lucky threshold, the central claim is not yet established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes augmenting the interpretable rubric-based features of Rahimi et al. (2014) for scoring the Evidence dimension of the Response-to-Text Assessment with word2vec embeddings (skip-gram and CBOW) trained on the task corpus. Cosine similarity above a development-tuned threshold is used to relax exact lexical matching between student essays and manually crafted topic/example lists. The authors compare against the original rubric features and two off-the-shelf embeddings (Mikolov, Baroni) on four RTA corpora (Space, MVP L, MVP H, MVP ALL) using quadratic weighted kappa, and they run cross-corpus transfer experiments. They report that the skip-gram embedding improves over the rubric baseline on Space, MVP L, and MVP ALL, but not MVP H, and that the task-trained embedding is competitive with Baroni's embedding.","tokens_in":8161,"tokens_out":6144,"duration_ms":63518,"significance":"If the claims hold, the paper offers a simple, interpretable, and low-cost way to improve evidence scoring by a small but consistent margin on lower-grade and lower-writing-proficiency corpora, while preserving the interpretability of the rubric features. The comparisons with off-the-shelf embeddings and the cross-corpus transfer experiments are informative for practitioners, and the honest reporting of the MVP H failure and the relatedness-versus-similarity limitation is a strength. The contribution is incremental over Rahimi et al. (2014), and the empirical basis is currently too fragile to constitute a strong demonstration: there is no code or data release, no multi-split variance analysis, no confidence intervals, and no analysis of the matching errors introduced by the embedding threshold. These gaps are fixable within the scope of the manuscript.","major_comments":[{"comment":"The evaluation rests on a single 40/20/40 stratified split, with the 20% development set used to select both the embedding model and the cosine threshold; the final QWKappa numbers and the reported pairwise significance are computed within this one partition. The manuscript does not specify how the '10 runs of 10-fold cross validation in the final testing' relate to the held-out 40%: if the 40% partition is the test set, applying 10-fold CV to it is not a clean held-out evaluation, and if CV is run over the whole corpus, the classifiers see data used for embedding training and threshold selection. In either reading, the reported variability covers only classifier training, not embedding training, threshold selection, or partitioning. Given the small gains (e.g., Space 0.606 to 0.611 and MVP L 0.628 to 0.682) and the noisy labels (first-rater Kappa 0.338 on Space), the central claim is not yet robust. Please rerun the full pipeline over multiple independent splits or use bootstrap resampling, and report mean QWKappa with confidence intervals or error bars.","section":"Section 6 (Experimental Setup)"},{"comment":"The only mechanism for relaxing exact matching is the cosine-similarity threshold, a single global number tuned on the development set; no threshold value, no distribution of cosine similarities, and no error analysis of the matches introduced by the threshold are reported. The authors themselves note in Section 8 that the embedding matches unwanted related pairs such as 'water' and 'electricity'. Because the rubric defines evidence relevance rather than distributional relatedness, the improvement could come from a fortuitous threshold level rather than from a meaningful semantic proxy. Please add an ablation or error analysis showing which newly matched pairs contribute to score changes, and report the threshold values selected on the development set.","section":"Section 5 (Word Embedding Feature Extraction)"},{"comment":"The paper reports pairwise significance with superscripts but does not state which statistical test is used, whether it is corrected for multiple comparisons, or whether it accounts for the fact that all models are evaluated on the same folds. On MVP H, for example, skip-gram (0.583) is significantly worse than both the rubric baseline (0.599) and Baroni (0.593), which is consistent with the authors' hedge but makes the abstract-level phrase 'improving evidence scoring' too strong. More importantly, without standard deviations or confidence intervals, the reader cannot tell whether the differences in Table 3 are meaningful. Please report the exact test, effect sizes, and variability across runs.","section":"Section 7 (Results and Discussion)"},{"comment":"The use of first-rater scores as the gold standard is understandable because the first rater graded more essays, but with Kappa as low as 0.338 on Space, the evaluation is against a noisy target. The paper should discuss the ceiling imposed by rater agreement and ideally report QWKappa against both raters on the double-rated portion, or at least discuss how rater noise affects the comparison between systems.","section":"Section 3 (Data) and Table 1"}],"minor_comments":[{"comment":"The phrase 'students writing quality' should be 'students' writing quality'.","section":"Abstract"},{"comment":"The notation for corpus names is inconsistent: 'M V PH' appears with spaces in some places and without them in others; please use a single consistent notation, such as 'MVP-H'.","section":"Throughout"},{"comment":"The sentence 'We use 10 runs of 10-fold cross validation in the final testing' is ambiguous and should be rewritten to state explicitly where the cross-validation is applied and how the 40/20/40 split relates to it.","section":"Section 6 (Experimental Setup)"},{"comment":"The sentence 'students use their own vocabularies other than words in the crafted list' is grammatically awkward; consider 'students use their own vocabulary rather than the words in the crafted list'.","section":"Section 5 (Word Embedding Feature Extraction)"},{"comment":"Some references are formatted inconsistently, for example Rahimi et al. (2017) is cited in Table 2 as '(Rahimi et al., 2017)' but in the text as 'Rahimi et al. (2017)'; please standardize.","section":"References"},{"comment":"The reproduced student essay contains intentional spelling errors, but a note in the caption clarifying that these are original student errors would help avoid confusion.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"To the editor: this is an incremental extension of the authors' earlier ITS-2014 system, and the empirical evidence is currently not strong enough for a journal-level claim. The central idea is plausible and the presentation is honest, but the single-split evaluation and the lack of reproducibility artifacts make the contribution difficult to verify. I would encourage a major revision requiring multi-split or bootstrap evaluation, an error analysis of embedding-based matches, and release of code and data if possible. I do not see grounds for rejection; nor is the manuscript ready for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a legitimate incremental contribution: it takes Rahimi et al.'s rubric features for evidence scoring and relaxes exact lexical matching using word embeddings trained on the students' own essays. The new empirical comparison on three elementary RTA corpora is real, and the authors are honest about the mixed results. The skip-gram model improves QWKappa on the lower-grade corpus and matches or slightly beats the rubric baseline on most corpora, but not on M V PH. The cross-corpus generalization table is the most interesting part, and there the task-specific embedding does best. I think the central claim holds up as a promising direction, not as a definitive win.\n\nWhat the paper does well: the motivation is concrete (misspellings, paraphrases, tense variants), the method is simple enough to reproduce from the description, and the authors discuss failure cases themselves, including the over-matching of \"water\" and \"electricity\". The comparison against two off-the-shelf embeddings is the right control, and the finding that a small in-domain corpus can beat a much larger generic one is worth reporting. The results are not oversold.\n\nSoft spots: the stress-test note is on target. All reported results come from a single fixed 40/20/40 split. The 10 runs of 10-fold CV on the test partition capture classifier variance only; they do not capture variance in embedding training or threshold selection. On a corpus where first-rater kappa is 0.338 (Space), the measured QWKappa differences are small enough that a different split could easily change the ranking. The lack of code, data, and error bars makes this hard to check. I also find the significance claims hard to interpret: Table 3 marks significance over other models, but without variance across splits, these are conditional on one split. That said, the authors did use a development set for threshold selection, so the evaluation is not circular.\n\nWho is this for: researchers working on rubric-based automated essay scoring, especially for noisy elementary writing. It is not a methods paper and does not claim to be.\n\nRecommendation: it deserves a serious referee. The idea is sensible, the experiments are mostly transparent, and the published version of this line of work has been used by others. I would ask the authors for multiple splits, feature-level ablation, and ideally released code or at least the crafted word lists. A revision with those additions would be worth publishing.","headline":"A modest, honest empirical extension of rubric-based RTA evidence scoring with word embeddings; the gains are plausible but rest on one fixed data split and small noisy corpora.","tokens_in":8607,"tokens_out":664,"would_cite":true,"duration_ms":9123,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adding a skip-gram word-embedding layer to the rubric feature extractor improves automated Evidence scoring of elementary students' essays on most corpora, with the largest gains on the noisiest responses.","keywords":["word embeddings","skip-gram","Response-to-Text Assessment","evidence scoring","rubric-based scoring","lexical matching","quadratic weighted kappa","elementary student essays"],"falsifier":"Take the subset of essays where the skip-gram model changed the score, remove every word matched only through embedding similarity, and have expert raters judge whether the removed evidence is rubric-relevant; if the matches are no more relevant than random pairs, the claimed improvement is an artifact.","tokens_in":7777,"feed_emoji":"📝","tokens_out":6948,"duration_ms":112190,"temperature":0.7,"pith_summary":"This paper tries to show that a simple supplement to an interpretable evidence-scoring system—replacing exact word matches with word-embedding similarity—improves automated scoring of the Response-to-Text Assessment Evidence rubric. The motivation is practical: elementary students write noisy essays with misspellings and their own vocabulary, so exact lexical matching leaves valid evidence undetected. The claimed payoff is a modest but real increase in quadratic weighted kappa on most corpora, strongest for lower-grade responses, with no loss in interpretability. The paper also argues that embeddings trained on the target population generalize across age groups better than off-the-shelf embeddings.","feed_headline":"Word embeddings improve evidence scoring in student essays","feed_subtitle":"Trained on students' own essays, skip-gram embeddings beat exact word matching on most RTA corpora.","key_machinery":"The central machinery is the skip-gram word embedding model, which maps each word to a vector so that words appearing in similar contexts sit close together. The feature extractor supplements the rubric's exact-match window search by computing cosine similarity between an essay word and each word in the hand-crafted topic and example lists, counting a match if similarity exceeds a threshold chosen on the development set. This is what lets \"proverty\" match \"poverty\", \"went\" match \"go\", and \"mosquitoes\" match \"malaria\", while still keeping the features themselves interpretable.","core_discovery":"The paper's central claim is that a word-embedding layer can improve automated scoring of the Evidence dimension of the Response-to-Text Assessment without giving up the interpretability of the rubric features. Concretely, the skip-gram embedding trained on the students' own essays yields a higher or equal quadratic weighted kappa than the lexical-matching rubric baseline on most corpora, and the gain is significant on the lower-grade corpus and on the combined corpus; on the older-grade corpus the embedding model does not beat the baseline. A second claim is that this task-trained model generalizes across age groups better than either the rubric baseline or off-the-shelf embeddings. The paper attributes the gains to the model's ability to recognize misspellings, inflectional variants, and some paraphrases that exact string matching misses.","pith_inferences":["The same thresholded-embedding layer could be applied to other RTA rubric dimensions that use lexical matching, such as Organization, but the paper does not test this.","A decisive test of the mechanism would compare embedding-matched evidence against a second human rater's judgments, since the paper evaluates only against first-rater scores.","The embedding space could double as a population-specific spell normalizer: because the model is trained on student typos, relations like \"proverty\" to \"poverty\" could be harvested to build a spelling-error dictionary."],"forward_implications":["Evidence scoring can be improved by swapping the matching layer while leaving the rubric features unchanged, so the method can be dropped into existing RTA scoring pipelines.","On lower-grade corpora the improvement is significant, suggesting semantic matching matters most where writing is noisiest.","Task-trained skip-gram embeddings transfer across age groups better than both the rubric baseline and off-the-shelf embeddings, supporting the idea that population-matched embeddings are worth training even on small corpora.","On the older-student corpus the embedding model does not beat the rubric baseline, so for cleaner writing exact matching may already be sufficient.","Skip-gram consistently beats continuous bag-of-words, attributable to better handling of infrequent words that dominate student evidence vocabulary."],"supporting_citations":[{"why":"It supplies the rubric-based feature extraction baseline and window-matching algorithm that the embedding layer modifies.","marker":"Rahimi et al., 2014"},{"why":"It introduces the Response-to-Text Assessment and the elementary-student corpora used for training and evaluation.","marker":"Correnti et al., 2013"},{"why":"It provides the word embedding model whose vectors drive the similarity matching.","marker":"Mikolov et al., 2013a"},{"why":"It supplies the 300-dimensional off-the-shelf vectors and the argument that skip-gram handles infrequent words better.","marker":"Mikolov et al., 2013b"},{"why":"It provides the 400-dimensional off-the-shelf embeddings compared against the task-trained models.","marker":"Baroni et al., 2014"},{"why":"It supplies the SMOTE oversampling method used to address imbalanced score classes.","marker":"Chawla et al., 2002"},{"why":"It supplies the Random Forest classifier and experimental setup used for the final evaluation.","marker":"Witten et al., 2016"}],"fun_headline_variants":["Word embeddings elevate essay evidence scoring","Student-trained embeddings outperform match scoring","Embedding model boosts RTA evidence scores","Skip-gram nets improve essay grading"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that nearness in the embedding space, above a threshold selected on a development set, tracks rubric-relevant evidence rather than mere topic relatedness; if that mapping fails, the method's score gains are not real evidence-detection.","fun_headline_variants_meta":{"raw":{"variants":["Word embeddings elevate essay evidence scoring","Student-trained embeddings outperform match scoring","Embedding model boosts RTA evidence scores","Skip-gram nets improve essay grading"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000214,"raw_usage":{"total_tokens":1350,"prompt_tokens":796,"completion_tokens":554,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":412,"completion_tokens_details":{"reasoning_tokens":504}},"tokens_in":412,"tokens_out":554,"duration_ms":6634,"temperature":1.0,"reasoning_tokens":504,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:57:17.583007+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the subset of essays where the skip-gram model changed the score, remove every word matched only through embedding similarity, and have expert raters judge whether the removed evidence is rubric-relevant; if the matches are no more relevant than random pairs, the claimed improvement is an artifact.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the rubric-based feature extraction baseline and window-matching algorithm that the embedding layer modifies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It introduces the Response-to-Text Assessment and the elementary-student corpora used for training and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the 400-dimensional off-the-shelf embeddings compared against the task-trained models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the SMOTE oversampling method used to address imbalanced score classes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the Random Forest classifier and experimental setup used for the final evaluation."}],"review_version":1}