{"id":"0cb73b50-1b42-4289-8785-50355d104f63","arxiv_id":"2506.02758","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs prompted with English Vocabulary Profile definitions outperform a part-of-speech baseline at assigning CEFR levels to words in learner sentences, though the advantage is concentrated on ambiguous words.","lead":"This paper tests whether large language models can assign English proficiency levels to individual words in second-language learner writing, using the English Vocabulary Profile as the reference. The best model beats a part-of-speech baseline on ambiguous words, but the comparison lacks error bars and the word-level annotations are not yet released.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's word-level accuracy may measure EVP sense-matching rather than L2 vocabulary assessment, because the gold labels and the LLM prompt options both come from the same EVP resource, applied to a readability corpus rather than learner writing.","rationale":"The reader's weakest assumption correctly identifies that the OneStopEnglish word-level labels are derived from the EVP and are not released, with no inter-annotator agreement. My stress-test sharpens this into a construct-validity concern: the LLM prompt contains the same EVP definitions used to create the gold labels, so high accuracy may reflect EVP sense retrieval rather than assessment of learner vocabulary. I add that OneStopEnglish is a readability corpus, not L2 learner writing, which weakens the abstract's claim even if the labels were clean. This is a partial agreement: the reader emphasizes label noise/circularity, while I emphasize that the evaluation task itself is not yet shown to be vocabulary assessment. A single independent human-annotated learner corpus test would settle whether the Table 2 gap reflects a real capability. The paper has strengths: the semantic-understanding experiment (Table 1) is a reasonable proof of concept, the method explicitly handles multi-word expressions, and the essay-level correlation results are suggestive. However, those strengths do not resolve the gold-standard problem for the headline word-level result. The appropriate verdict remains CONDITIONAL: the paper should release annotations, report agreement, and validate on learner-produced text with independent labels before the central claim is accepted.","tokens_in":14956,"tokens_out":2950,"duration_ms":37001,"concrete_test":"Construct an independent evaluation set by having two expert raters, blind to EVP options, assign word-level CEFR levels to individual words in 300 randomly sampled EFCAMDAT learner sentences. Report inter-annotator agreement (Cohen's kappa) and recompute Qwen 2.5 32B and PoS-baseline accuracy against these human labels. If the Qwen-over-PoS gap shrinks below roughly 3 points, or if rater agreement is low, the Table 2 advantage is an artifact of EVP-derived gold labels rather than a generalizable vocabulary-assessment capability.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central result is the 6.3-point accuracy gap between Qwen 2.5 32B (87.0%) and the PoS baseline (80.7%) on all OneStopEnglish words (Table 2, Section 4.5). The paper interprets this as evidence that LLMs are 'well-suited for the task of vocabulary assessment' (abstract). The load-bearing assumption is that the OneStopEnglish word-level CEFR labels are a valid gold standard for learner vocabulary skill. Section 3.2.1 states these labels were produced by the authors from the EVP, and Section 4.2 says the LLM prompt feeds the EVP definitions and PoS information as the candidate options. Thus the evaluation measures how well an LLM selects an EVP entry whose definition is already in the prompt, and the gold label is the CEFR level of that same EVP entry. This is close to reproducing the EVP mapping, not independently assessing whether a learner's use of a word is appropriate at their proficiency level. The risk is compounded by the corpus: OneStopEnglish is a readability/text-simplification corpus of parallel texts, not a corpus of L2 learner production, so word-level labels reflect text-difficulty design, not learner vocabulary use. The annotations are not released, no inter-annotator agreement is reported, and the comparison uses a single run without significance tests or variance estimates. If the gold labels are noisy or systematically aligned with the prompt content, the accuracy gap in Table 2 is not evidence for the paper's assessment claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"Bannò et al. propose combining large language models (LLMs) with the English Vocabulary Profile (EVP) for word-level, in-context CEFR proficiency prediction. The approach casts prediction as multiple-choice sense selection: for each word in a sentence, the EVP senses (with definitions and part-of-speech) are supplied as options, the LLM selects a sense from logits, and the selected sense's CEFR level is the prediction. Experiments cover (i) sense selection on EVP learner examples, (ii) word-level CEFR prediction on OneStopEnglish sentences annotated by the authors, (iii) essay-level proficiency prediction using the resulting word-level features on EFCAMDAT and ELLIPSE, and (iv) an EVP consistency check on two frequent words. The central result in Table 2 is that Qwen 2.5 32B reaches 87.0% accuracy on all words versus 80.7% for a part-of-speech baseline, with the largest advantage on ambiguous words (80.5% versus 66.7%). The paper concludes that LLMs are well-suited for vocabulary assessment.","tokens_in":15263,"tokens_out":9336,"duration_ms":91857,"significance":"The proposed method is a sensible and practical operationalization of in-context vocabulary assessment, and the use of logit-based multiple-choice selection with permutation averaging is a thoughtful design choice. Strengths include the use of a public standard resource (EVP), systematic comparison of five LLMs, and evaluation on two large learner corpora at the essay level. If the word-level evaluation were supported by an independent gold standard, the approach would be a useful building block for automated writing evaluation and for generating fine-grained vocabulary feedback. However, the current evidence is not yet convincing on its own: the word-level labels come from the same resource that provides the prompt options, the corpus is not learner production, and no uncertainty estimates or competitive baselines are supplied. These gaps are fixable within the scope of a revision.","major_comments":[{"comment":"The OneStopEnglish word-level labels are produced by the authors from the EVP with no inter-annotator agreement reported, and OneStopEnglish is a readability/text-simplification corpus rather than L2 learner production. Because the LLM prompt contains the EVP definitions and part-of-speech information as options, and the gold label is the CEFR level of the same EVP entry, Table 2 measures the model's ability to reproduce the EVP sense mapping. This may be a useful benchmark for sense selection, but it does not by itself establish that the model is 'well-suited for the task of vocabulary assessment' on learner writing (abstract). Please provide an independent validation set of learner essays annotated by human raters with agreement statistics, and report accuracy separately for senses with multiple CEFR levels versus those with a single level.","section":"Section 3.2.1 and Table 2"},{"comment":"No significance tests, confidence intervals, or variance estimates are reported. The central 6.3-point gap between Qwen 2.5 32B and the PoS baseline is based on a single run, and the sizes of the ambiguous and non-ambiguous subsets are not given. I recommend bootstrap resampling over words or sentences to obtain confidence intervals and per-category sample sizes; without this, the reported gap may be within noise.","section":"Section 4.5 and Table 2"},{"comment":"The PoS baseline is a deliberately weak heuristic: for ambiguous words it always assigns the lowest CEFR level, and for non-ambiguous words it uses only the matching PoS entry while the LLM receives all entries. The comparison therefore does not isolate the LLM's semantic reasoning from the extra information in the prompt (definitions and full option lists). The paper's footnote 10 says prompting without EVP information was 'significantly worse', but no numbers are given. A stronger baseline such as a lexical-overlap sense matcher, a BERT-based WSD classifier, or a sense-frequency majority baseline would make the claimed LLM advantage more convincing.","section":"Section 4.2 and Section 4.5"},{"comment":"The EVP consistency experiment is circular in a way that weakens its conclusion. The LLM is forced to choose among EVP entries, and the '≥ level' accuracy checks whether the chosen entry's level is at or below the essay's holistic level; since the candidate senses all come from the same EVP, high agreement is partly by construction. With only two words (work and like), no baseline, and no word-level ground truth, the claim of a 'strong degree of consistency' in the EVP is not supported. I suggest a larger sample of words, a comparison against chance (e.g., randomly choosing an EVP sense for the same word), and an evaluation on learner essays with independent word-level annotations.","section":"Section 4.7 and Table 5"},{"comment":"The abstract lists multi-word expressions as one of the key challenges addressed, but the Limitations section admits that no focused analysis of MWEs was conducted, and the only support is a single example in Appendix C. The paper should either provide quantitative results on an MWE subset or soften the claim. This matters because the treatment of MWEs is stated as a contribution and is currently unsubstantiated.","section":"Abstract and Limitations"}],"minor_comments":[{"comment":"The model name 'GPT-4omini' should be written as 'GPT-4o-mini' for consistency with Section 4.1.","section":"Section 4.4"},{"comment":"The annotation procedure is described in one sentence; please specify who annotated the OneStopEnglish sentences, what guidelines were used, how cases with multiple plausible EVP senses were resolved, and provide a release link for the annotations.","section":"Section 3.2.1"},{"comment":"The word-level prompt includes a 'None of the other options' choice, but the semantic-understanding prompt does not; please explain this asymmetry in the text.","section":"Appendix C"},{"comment":"The captions for Figures 5 and 6 should define 'normalised distribution', 'eCDF', and 'AUC' so that the figures are self-contained.","section":"Figures 5 and 6"},{"comment":"The N/A row in Table 9 should be defined in the text; the current wording in Section 3.2.1 ('excluding ... words not featured in the EVP') is ambiguous about whether such words are retained as N/A labels.","section":"Table 9"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the journal's scope and the proposed method is interesting, but the central word-level result currently rests on a self-produced gold standard with no independent validation and no uncertainty estimates. The authors should be encouraged to present the word-level task more modestly as EVP sense mapping with potential downstream use, and to add external validation and stronger baselines before the claim of suitability for vocabulary assessment can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a reasonable first-application study: it combines LLMs with the English Vocabulary Profile to predict CEFR levels of individual words in context. The main result, Qwen 2.5 32B at 87.0% vs. the PoS baseline at 80.7%, is plausible, and the idea of using sense-selection with EVP entries handles polysemy and multi-word expressions more cleanly than earlier rule-based tools. The side experiments on EFCAMDAT and ELLIPSE give some evidence that the extracted word-level features correlate with holistic proficiency scores, which is useful for automated feedback systems.\n\nWhat does not hold up as well is the evaluation. The word-level gold labels on OneStopEnglish were produced by the authors from the EVP, with no inter-annotator agreement, no release yet, and no significance tests or error bars anywhere in the paper. The corpus itself is a readability and text-simplification corpus, not a corpus of L2 learner writing, so the headline accuracy gap in Table 2 is at least partly measuring how well the LLM can reproduce the EVP mapping from the definitions already in the prompt. That is a legitimate task, but it is not the same as assessing whether a learner is using a word at the right level, and the abstract's phrasing overstates the evidence. The non-ambiguous-word comparison is also acknowledged to be unfair to the LLMs because they are given all PoS entries, so the strongest claim about LLM advantage rests on the ambiguous subset alone.\n\nThe EVP consistency experiment is thin: two words, essay-level holistic scores, and the same EVP source on both sides. It is presented as exploratory, but the title of the paper invites more serious validation. Also, the claim of 'fully replicable' is undercut by the annotations not yet being public.\n\nAll that said, the paper is honest about some of its own limitations, and the hybrid idea (LLM for ambiguous words, PoS for the rest) is sensible. For a reader in L2 assessment or intelligent tutoring, this is worth reading as a methodological starting point. It deserves a serious referee, but the review should require the data to be released, IAA for the annotations, significance tests, and ideally a word-level evaluation on genuine learner essays or at least a clear acknowledgment that OneStopEnglish is not learner writing.","headline":"Solid first-application paper: LLMs + EVP for word-level CEFR prediction, but the evaluation rests on self-annotated labels over a readability corpus and needs more rigorous validation before the claims hold.","tokens_in":15771,"tokens_out":2572,"would_cite":true,"duration_ms":30046,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs prompted with English Vocabulary Profile definitions assign CEFR levels to individual words in learner sentences, outperforming part-of-speech baselines.","keywords":["word-level vocabulary assessment","CEFR prediction","English Vocabulary Profile","large language models","polysemy","L2 learner writing","multi-word expressions","automated writing evaluation"],"falsifier":"Take the same OneStopEnglish sentences but have several independent human raters assign each word's CEFR level from the EVP with measured inter-annotator agreement, then score Qwen 2.5 32B against those raters rather than the authors' labels; if accuracy falls to within noise of the part-of-speech baseline, the claim that LLMs add semantic information fails.","tokens_in":14769,"feed_emoji":"📚","tokens_out":6287,"duration_ms":55399,"temperature":0.7,"pith_summary":"This paper tries to establish that word-level, in-context vocabulary assessment for second-language writing can be automated by combining large language models with the English Vocabulary Profile (EVP), a resource that links each sense of a word to a CEFR proficiency level. It matters because automated writing evaluation currently scores vocabulary mostly through word frequency and part of speech, ignoring polysemy and context. The paper's headline result is that multiple-choice sense selection with LLM prompts beats a part-of-speech baseline on word-level CEFR prediction, with Qwen 2.5 32B reaching 87.0 percent accuracy versus 80.7 percent for the baseline on the OneStopEnglish corpus. The authors also show that the predicted word-level levels correlate with essay-level proficiency on two large learner corpora, and that the approach can audit the consistency of the EVP's own level assignments.","feed_headline":"LLMs with dictionary definitions score 87% on word-level CEFR","feed_subtitle":"In-context sense selection beats part-of-speech baselines for assessing L2 vocabulary in learner essays.","key_machinery":"The central object is the EVP's per-lemma entry set, where each entry links a sense (base word plus guideword) to a CEFR level, a part of speech, and a definition; the machinery converts proficiency prediction into multiple-choice sense selection. A learner sentence is presented with the target word highlighted in square brackets, spaCy lemmatises it and matches the lemma to EVP entries, and those entries become numbered options alongside an added 'None of the other options' choice. The model's logits over the options are softmaxed and averaged over shuffled permutations of option order to remove positional bias, and the highest-probability option's CEFR level is the final prediction.","core_discovery":"The paper's central claim is that word-level CEFR prediction in L2 writing can be solved as a sense-selection problem: when a target word in a learner sentence is highlighted and the model is offered all matching English Vocabulary Profile entries as multiple-choice options, each with its guideword, definition, and part of speech, modern LLMs select the contextually correct sense and thereby the correct proficiency level. On OneStopEnglish, Qwen 2.5 32B reaches 87.0 percent accuracy over all words, against 80.7 percent for the part-of-speech baseline and 61.6 percent for random; on ambiguous words (same lemma and part of speech with several possible levels) it reaches 80.5 percent against 66.7 percent for the baseline. This finding is supported by a semantic-understanding experiment in which LLMs identify the intended EVP sense from learner example sentences, where GPT-4o leads at 84.4 percent average accuracy. The paper further claims that vocabulary features extracted this way transfer to essay-level proficiency prediction: on EFCAMDAT, Qwen-derived features raise SVR Pearson correlation from 0.734 (PoS features) to 0.771, and on ELLIPSE they correlate most strongly with the analytic Vocabulary score. Finally, the authors claim the same approach can probe the internal consistency of the EVP, reporting 88.6 percent and 89.6 percent accuracy for the words work and like when judged at their assigned level or above in EFCAMDAT essays.","pith_inferences":["The reported accuracy partly measures how well the LLM reproduces the EVP mapping, since the gold labels were produced from the same resource; a human-annotated held-out set with published inter-annotator agreement would separate sense-selection skill from measurement of learner ability.","The over-selection of the 'None of the other options' choice suggests a calibration issue; a small confidence threshold or option-order prior could improve accuracy without retraining.","A natural next test is whether the same prompting transfers to spoken learner output or to non-English L2s with their own graded profiles, which the authors list as future work but do not establish here.","If the accuracy holds against independent labels, the approach could power fine-grained learner feedback (for example, 'you used a C1 sense of run') and dynamic exercises that adapt to a learner's productive vocabulary."],"forward_implications":["Word-level, in-context vocabulary scoring can be automated at roughly 87 percent accuracy on learner writing, with the gains over part-of-speech baselines concentrated on polysemous words.","A hybrid system that lets an LLM handle ambiguous words and a part-of-speech lookup handle the rest would likely outperform either alone, since the simple baseline already beats the LLM on unambiguous words.","Predicted per-word CEFR distributions carry signal about holistic essay proficiency: Qwen-derived features improve naive and SVR correlations on EFCAMDAT, and on ELLIPSE they correlate most with the analytic Vocabulary score.","The same pipeline can audit the internal consistency of a proficiency resource, as demonstrated by the high 'level or above' accuracy for the words work and like in EFCAMDAT.","Because the method needs only the EVP entry set and a prompting loop, it transfers to any dataset with essay-level labels and to any language with a comparable sense-to-level resource."],"supporting_citations":[{"why":"Defines the English Vocabulary Profile, the sense-to-CEFR resource that supplies both the multiple-choice options and the gold-standard mapping.","marker":"Capel (2015)"},{"why":"Provides the OneStopEnglish parallel corpus from which the word-level evaluation sentences were annotated.","marker":"Vajjala and Lučić (2018)"},{"why":"Introduced EFCAMDAT, the large learner corpus used for essay-level transfer experiments.","marker":"Geertzen et al. (2013)"},{"why":"Documents the EFCAMDAT release and its proficiency levels used to define essay-level CEFR bands.","marker":"Huang et al. (2017)"},{"why":"Provides the ELLIPSE corpus with holistic and analytic scores used to test whether word-level features predict essay-level proficiency.","marker":"Crossley et al. (2023)"},{"why":"Supplies the permutation-averaging procedure the paper uses to remove positional bias from multiple-choice logits.","marker":"Liusie et al. (2024)"},{"why":"Describes the Duolingo CEFR checker, the prior automated vocabulary-scale model this work compares against in kind.","marker":"Settles et al. (2020)"},{"why":"Precedent for using the EVP to assign proficiency bands to learner essays, motivating the word-level extension.","marker":"Leńko-Szymańska (2015)"},{"why":"Documents the Cambridge Learner Corpus from which the EVP's learner examples and proficiency entries were built.","marker":"Nicholls (2003)"}],"fun_headline_variants":["LLMs pick correct word sense for 87% of L2 vocabulary","Sense selection lifts word-level CEFR prediction to 87%","LLMs leverage EVP senses for 87% accurate L2 word scoring","Word-level CEFR: LLMs choose sense, hit 87% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The OneStopEnglish word-level CEFR labels were derived by the authors from the English Vocabulary Profile itself, so the evaluation assumes these labels are a valid gold standard; if they are noisy or biased, the reported accuracy gap could shrink.","fun_headline_variants_meta":{"raw":{"variants":["LLMs pick correct word sense for 87% of L2 vocabulary","Sense selection lifts word-level CEFR prediction to 87%","LLMs leverage EVP senses for 87% accurate L2 word scoring","Word-level CEFR: LLMs choose sense, hit 87% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000823,"raw_usage":{"total_tokens":3650,"prompt_tokens":1048,"completion_tokens":2602,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":664,"completion_tokens_details":{"reasoning_tokens":2521}},"tokens_in":664,"tokens_out":2602,"duration_ms":17976,"temperature":1.0,"reasoning_tokens":2521,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:16:31.304162+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same OneStopEnglish sentences but have several independent human raters assign each word's CEFR level from the EVP with measured inter-annotator agreement, then score Qwen 2.5 32B against those raters rather than the authors' labels; if accuracy falls to within noise of the part-of-speech baseline, the claim that LLMs add semantic information fails.","supporting_citations":[],"review_version":1}