Pith. sign in

REVIEW 5 major objections 5 minor 25 references

Exploiting the English Vocabulary Profile for L2 word-level vocabulary assessment with LLMs

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMs prompted with English Vocabulary Profile definitions assign CEFR levels to individual words in learner sentences, outperforming part-of-speech baselines.

desk verdict Solid first-application paper: LLMs + EVP for word-level CEFR prediction, but the evaluation rests on self-annotated labels over a readability corpus and needs more rigorous validation before the claims hold. read the letter →

arxiv 2506.02758 v1 pith:7UPEZGMD submitted 2025-06-03 cs.CL cs.AI

Stefano Bannò , Kate Knill , Mark Gales This is my paper
classification cs.CLcs.AI
keywords word-levelvocabularyassessmentCEFRpredictionEnglishProfilelargelanguagemodelspolysemyL2learnerwritingmulti-wordexpressionsautomatedevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that word-level, in-context vocabulary assessment for second-language writing can be automated by combining large language models with the English Vocabulary Profile (EVP), a resource that links each sense of a word to a CEFR proficiency level. It matters because automated writing evaluation currently scores vocabulary mostly through word frequency and part of speech, ignoring polysemy and context. The paper's headline result is that multiple-choice sense selection with LLM prompts beats a part-of-speech baseline on word-level CEFR prediction, with Qwen 2.5 32B reaching 87.0 percent accuracy versus 80.7 percent for the baseline on the OneStopEnglish corpus. The authors also show that the predicted word-level levels correlate with essay-level proficiency on two large learner corpora, and that the approach can audit the consistency of the EVP's own level assignments.

What carries the argument

The central object is the EVP's per-lemma entry set, where each entry links a sense (base word plus guideword) to a CEFR level, a part of speech, and a definition; the machinery converts proficiency prediction into multiple-choice sense selection. A learner sentence is presented with the target word highlighted in square brackets, spaCy lemmatises it and matches the lemma to EVP entries, and those entries become numbered options alongside an added 'None of the other options' choice. The model's logits over the options are softmaxed and averaged over shuffled permutations of option order to remove positional bias, and the highest-probability option's CEFR level is the final prediction.

What would settle it

Take the same OneStopEnglish sentences but have several independent human raters assign each word's CEFR level from the EVP with measured inter-annotator agreement, then score Qwen 2.5 32B against those raters rather than the authors' labels; if accuracy falls to within noise of the part-of-speech baseline, the claim that LLMs add semantic information fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that word-level CEFR prediction in L2 writing can be solved as a sense-selection problem: when a target word in a learner sentence is highlighted and the model is offered all matching English Vocabulary Profile entries as multiple-choice options, each with its guideword, definition, and part of speech, modern LLMs select the contextually correct sense and thereby the correct proficiency level. On OneStopEnglish, Qwen 2.5 32B reaches 87.0 percent accuracy over all words, against 80.7 percent for the part-of-speech baseline and 61.6 percent for random; on ambiguous words (same lemma and part of speech with several possible levels) it reaches 80.5 percent against 66.7 percent for the baseline. This finding is supported by a semantic-understanding experiment in which LLMs identify the intended EVP sense from learner example sentences, where GPT-4o leads at 84.4 percent average accuracy. The paper further claims that vocabulary features extracted this way transfer to essay-level proficiency prediction: on EFCAMDAT, Qwen-derived features raise SVR Pearson correlation from 0.734 (PoS features) to 0.771, and on ELLIPSE they correlate most strongly with the analytic Vocabulary score. Finally, the authors claim the same approach can probe the internal consistency of the EVP, reporting 88.6 percent and 89.6 percent accuracy for the words work and like when judged at their assigned level or above in EFCAMDAT essays.

Load-bearing premise

The OneStopEnglish word-level CEFR labels were derived by the authors from the English Vocabulary Profile itself, so the evaluation assumes these labels are a valid gold standard; if they are noisy or biased, the reported accuracy gap could shrink.

Editorial extensions

If this is right

  • Word-level, in-context vocabulary scoring can be automated at roughly 87 percent accuracy on learner writing, with the gains over part-of-speech baselines concentrated on polysemous words.
  • A hybrid system that lets an LLM handle ambiguous words and a part-of-speech lookup handle the rest would likely outperform either alone, since the simple baseline already beats the LLM on unambiguous words.
  • Predicted per-word CEFR distributions carry signal about holistic essay proficiency: Qwen-derived features improve naive and SVR correlations on EFCAMDAT, and on ELLIPSE they correlate most with the analytic Vocabulary score.
  • The same pipeline can audit the internal consistency of a proficiency resource, as demonstrated by the high 'level or above' accuracy for the words work and like in EFCAMDAT.
  • Because the method needs only the EVP entry set and a prompting loop, it transfers to any dataset with essay-level labels and to any language with a comparable sense-to-level resource.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported accuracy partly measures how well the LLM reproduces the EVP mapping, since the gold labels were produced from the same resource; a human-annotated held-out set with published inter-annotator agreement would separate sense-selection skill from measurement of learner ability.
  • The over-selection of the 'None of the other options' choice suggests a calibration issue; a small confidence threshold or option-order prior could improve accuracy without retraining.
  • A natural next test is whether the same prompting transfers to spoken learner output or to non-English L2s with their own graded profiles, which the authors list as future work but do not establish here.
  • If the accuracy holds against independent labels, the approach could power fine-grained learner feedback (for example, 'you used a C1 sense of run') and dynamic exercises that adapt to a learner's productive vocabulary.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. Bannò et al. propose combining large language models (LLMs) with the English Vocabulary Profile (EVP) for word-level, in-context CEFR proficiency prediction. The approach casts prediction as multiple-choice sense selection: for each word in a sentence, the EVP senses (with definitions and part-of-speech) are supplied as options, the LLM selects a sense from logits, and the selected sense's CEFR level is the prediction. Experiments cover (i) sense selection on EVP learner examples, (ii) word-level CEFR prediction on OneStopEnglish sentences annotated by the authors, (iii) essay-level proficiency prediction using the resulting word-level features on EFCAMDAT and ELLIPSE, and (iv) an EVP consistency check on two frequent words. The central result in Table 2 is that Qwen 2.5 32B reaches 87.0% accuracy on all words versus 80.7% for a part-of-speech baseline, with the largest advantage on ambiguous words (80.5% versus 66.7%). The paper concludes that LLMs are well-suited for vocabulary assessment.

Significance. The proposed method is a sensible and practical operationalization of in-context vocabulary assessment, and the use of logit-based multiple-choice selection with permutation averaging is a thoughtful design choice. Strengths include the use of a public standard resource (EVP), systematic comparison of five LLMs, and evaluation on two large learner corpora at the essay level. If the word-level evaluation were supported by an independent gold standard, the approach would be a useful building block for automated writing evaluation and for generating fine-grained vocabulary feedback. However, the current evidence is not yet convincing on its own: the word-level labels come from the same resource that provides the prompt options, the corpus is not learner production, and no uncertainty estimates or competitive baselines are supplied. These gaps are fixable within the scope of a revision.

major comments (5)
  1. [Section 3.2.1 and Table 2] The OneStopEnglish word-level labels are produced by the authors from the EVP with no inter-annotator agreement reported, and OneStopEnglish is a readability/text-simplification corpus rather than L2 learner production. Because the LLM prompt contains the EVP definitions and part-of-speech information as options, and the gold label is the CEFR level of the same EVP entry, Table 2 measures the model's ability to reproduce the EVP sense mapping. This may be a useful benchmark for sense selection, but it does not by itself establish that the model is 'well-suited for the task of vocabulary assessment' on learner writing (abstract). Please provide an independent validation set of learner essays annotated by human raters with agreement statistics, and report accuracy separately for senses with multiple CEFR levels versus those with a single level.
  2. [Section 4.5 and Table 2] No significance tests, confidence intervals, or variance estimates are reported. The central 6.3-point gap between Qwen 2.5 32B and the PoS baseline is based on a single run, and the sizes of the ambiguous and non-ambiguous subsets are not given. I recommend bootstrap resampling over words or sentences to obtain confidence intervals and per-category sample sizes; without this, the reported gap may be within noise.
  3. [Section 4.2 and Section 4.5] The PoS baseline is a deliberately weak heuristic: for ambiguous words it always assigns the lowest CEFR level, and for non-ambiguous words it uses only the matching PoS entry while the LLM receives all entries. The comparison therefore does not isolate the LLM's semantic reasoning from the extra information in the prompt (definitions and full option lists). The paper's footnote 10 says prompting without EVP information was 'significantly worse', but no numbers are given. A stronger baseline such as a lexical-overlap sense matcher, a BERT-based WSD classifier, or a sense-frequency majority baseline would make the claimed LLM advantage more convincing.
  4. [Section 4.7 and Table 5] The EVP consistency experiment is circular in a way that weakens its conclusion. The LLM is forced to choose among EVP entries, and the '≥ level' accuracy checks whether the chosen entry's level is at or below the essay's holistic level; since the candidate senses all come from the same EVP, high agreement is partly by construction. With only two words (work and like), no baseline, and no word-level ground truth, the claim of a 'strong degree of consistency' in the EVP is not supported. I suggest a larger sample of words, a comparison against chance (e.g., randomly choosing an EVP sense for the same word), and an evaluation on learner essays with independent word-level annotations.
  5. [Abstract and Limitations] The abstract lists multi-word expressions as one of the key challenges addressed, but the Limitations section admits that no focused analysis of MWEs was conducted, and the only support is a single example in Appendix C. The paper should either provide quantitative results on an MWE subset or soften the claim. This matters because the treatment of MWEs is stated as a contribution and is currently unsubstantiated.
minor comments (5)
  1. [Section 4.4] The model name 'GPT-4omini' should be written as 'GPT-4o-mini' for consistency with Section 4.1.
  2. [Section 3.2.1] The annotation procedure is described in one sentence; please specify who annotated the OneStopEnglish sentences, what guidelines were used, how cases with multiple plausible EVP senses were resolved, and provide a release link for the annotations.
  3. [Appendix C] The word-level prompt includes a 'None of the other options' choice, but the semantic-understanding prompt does not; please explain this asymmetry in the text.
  4. [Figures 5 and 6] The captions for Figures 5 and 6 should define 'normalised distribution', 'eCDF', and 'AUC' so that the figures are self-contained.
  5. [Table 9] The N/A row in Table 9 should be defined in the text; the current wording in Section 3.2.1 ('excluding ... words not featured in the EVP') is ambiguous about whether such words are retained as N/A labels.

Circularity Check

1 steps flagged · score 4.0 of 10

Word-level prediction is independent, but the EVP-consistency check partially validates EVP using EVP-derived predictions.

  1. other [Section 4.7, Table 5 (EVP consistency experiment)]
    "Finally, to evaluate the consistency of the EVP, we reverse the approach used thus far. ... we compute Accuracy by checking whether a word appears in essays at its assigned level or above. ... The results show high accuracy and suggest a strong degree of consistency in the CEFR classification provided by the EVP."

    The 'assigned level' used to compute this accuracy is not an independent label; it is the CEFR level of the EVP entry selected by the LLM in Section 4.2 ('Finally, we select the CEFR level assigned to this option'). The experiment therefore claims to validate EVP's sense-level assignments using predictions that are themselves read off EVP. The only external input is the essay-level holistic score, and the 'at or above' criterion makes the test pass trivially whenever a low EVP level is selected for common words like work/like. The LLM's reliability was itself established against OneStopEnglish labels created from the same EVP (Section 3.2.1), so the chain reduces to using EVP-derived predictions to declare EVP consistent.

full rationale

The main word-level CEFR prediction task (Section 4.2, Table 2) is not circular: the LLM is given OneStopEnglish context plus EVP definitions and must select a sense, while the gold labels are human-applied EVP levels; no parameter is fitted to the test split, and the LLM's output is not constructed from the gold. The essay-level experiments use external holistic scores from EFCAMDAT and ELLIPSE, so those correlations are independent benchmarks. However, the EVP-consistency experiment in Section 4.7 contains a partially circular step: the 'assigned level' used to compute accuracy is the CEFR level of the EVP entry selected by the LLM, so the test evaluates EVP's internal consistency with predictions read from EVP; the only external signal is the essay-level score, and the lenient 'at or above' criterion makes low-level predictions almost always correct. This does not undermine the central accuracy comparison, but it weakens the secondary claim that EVP levels are strongly consistent. The self-citations in the paper (Bannò et al. 2024; Liusie et al. 2024) are contextual and not load-bearing; no fitted input is renamed as a prediction in the central word-level or essay-level experiments.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central evaluation leans on four domain assumptions: EVP levels as gold standard, correctness of the authors' OneStopEnglish annotations, spaCy's lemmatization and PoS quality, and validity of essay-level proficiency labels. No free parameters are fitted to the target data; the SVR uses fixed defaults. No new entities are postulated.

assumptions (5)
  • domain assumption EVP is a valid gold standard for word-level CEFR proficiency.
    All accuracy metrics compare LLM or PoS predictions to EVP-derived labels; the paper's own consistency check does not independently validate the EVP.
  • domain assumption The authors' OneStopEnglish word-level annotations are accurate.
    Section 3.2.1 says the authors annotated 293 triplets with EVP levels; no inter-annotator agreement is reported and the data are not yet released, so annotation errors directly affect Table 2.
  • domain assumption spaCy lemmatization and PoS tagging are accurate enough for EVP matching.
    Section 4.2 uses spaCy to map words to EVP entries; the limitations section acknowledges that learner errors can hinder lemmatization and matching.
  • domain assumption Softmax over LLM logits across option permutations estimates a stable choice probability.
    Section 4.1 averages probabilities over permutations to reduce positional bias, but no evidence of run-to-run stability across decoding temperatures is given.
  • domain assumption Essay-level proficiency labels in EFCAMDAT and ELLIPSE are valid proxies for learner proficiency.
    Sections 4.3 and 4.6 correlate word-level predicted CEFR distributions with holistic and analytic essay scores; if the essay labels are noisy, the reported correlations are lower bounds on the true relationships.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploiting the English Vocabulary Profile for L2 word-level vocabulary assessment with LLMs." pith.science (2026). https://pith.science/paper/7UPEZGMD

@misc{pith2026250602758,
  author       = {Pith},
  title        = {Pith review of: Exploiting the English Vocabulary Profile for L2 word-level vocabulary assessment with LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7UPEZGMD}},
  note         = {Machine review of arXiv:2506.02758}
}
read the original abstract

Vocabulary use is a fundamental aspect of second language (L2) proficiency. To date, its assessment by automated systems has typically examined the context-independent, or part-of-speech (PoS) related use of words. This paper introduces a novel approach to enable fine-grained vocabulary evaluation exploiting the precise use of words within a sentence. The scheme combines large language models (LLMs) with the English Vocabulary Profile (EVP). The EVP is a standard lexical resource that enables in-context vocabulary use to be linked with proficiency level. We evaluate the ability of LLMs to assign proficiency levels to individual words as they appear in L2 learner writing, addressing key challenges such as polysemy, contextual variation, and multi-word expressions. We compare LLMs to a PoS-based baseline. LLMs appear to exploit additional semantic information that yields improved performance. We also explore correlations between word-level proficiency and essay-level proficiency. Finally, the approach is applied to examine the consistency of the EVP proficiency levels. Results show that LLMs are well-suited for the task of vocabulary assessment.

Figures

Figures reproduced from arXiv: 2506.02758 by the authors.

Figure 1
Figure 1. Example for the word push from the EVP. Each EVP entry includes a base word, a guide￾word, its CEFR level(s), manually assigned PoS, topic, and additional details such as learner and dic￾tionary examples in context (see [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Annotation example on OneStopEnglish. S: stopword; P: punctuation; N/A: word not featured in the EVP. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. High-level diagram of the approach proposed [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Highlighting method. For each sentence we iterate through each word by highlighting it in square brackets, as shown in [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: eCDF of AUC values. Qwen 2.5 32B vs PoS [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 5
Figure 5. Figure 5: Predicted normalised distribution of words for [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: Distribution for word work (A1: 191; A2: 278; B1: 482; B2: 667; C1: 457; C2: 667) [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Distribution for word like (A1: 401 ; A2: 411; B1: 380; B2: 459; C1: 497; C2: 591). by checking whether a word appears in essays at its assigned level or above. It is also important to note that we are comparing word-level vocabulary levels with essay-level holistic sc…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 23 canonical work pages

  1. [1]

    [DEFINITION 1] - Additional infor- mation: [INFO] (PoS)

    work page

  2. [2]

    [DEFINITION 2] - Additional infor- mation: [INFO] (PoS)

    work page

  3. [3]

    sentence

    [DEFINITION 3] - Additional infor- mation: [INFO] (PoS) [...] n. None of the other options. where the additional information consists of a brief definition of the word. In round brackets, we also feed the information related to the manually assigned PoS as described in the EVP. See the ex- ample below for further information. The prompt for the experiment...

    work page

  4. [4]

    - Additional information: work (ACTIVITY) (Part of speech: noun)

    the activities that you have to do at school, for your job, etc. - Additional information: work (ACTIVITY) (Part of speech: noun)

    work page

  5. [5]

    - Additional in- formation: work (OPERATE) (Part of speech: verb)

    If a machine or piece of equipment works, it is not broken. - Additional in- formation: work (OPERATE) (Part of speech: verb)

    work page

  6. [6]

    when you use physical or mental effort to do something - Additional information: work (EFFORT) (Part of speech: noun)

    work page

  7. [7]

    - Additional information: work (SUCCEED) (Part of speech: verb)

    If something works, it is effective or successful. - Additional information: work (SUCCEED) (Part of speech: verb)

    work page

  8. [8]

    to exercise in order to improve the strength or appearance of your body - Additional information: work out (EX- ERCISE) (Part of speech: verb)

    work page

Show all 25 references
  1. [9]

    - Additional information: work (CRE- ATION) (Part of speech: noun)

    a painting, book, piece of music, etc. - Additional information: work (CRE- ATION) (Part of speech: noun)

  2. [10]

    the place where you go to do your job - Additional information: work (PLACE) (Part of speech: noun)

  3. [11]

    something you do as a job to earn money - Additional information: work (JOB) (Part of speech: noun)

  4. [12]

    to do a job, especially the job you do to earn money - Additional information: work (DO JOB) (Part of speech: verb)

  5. [13]

    - Additional information: work out (BECOME BETTER) (Part of speech: verb)

    If a problem or a complicated sit- uation works out, it ends in a success- ful way. - Additional information: work out (BECOME BETTER) (Part of speech: verb)

  6. [14]

    to know how to use a machine or piece of equipment - Additional informa- tion: can work sth; know how to work sth (Part of speech: verb)

  7. [15]

    to understand something or to find the answer to something by thinking about it - Additional information: work sth out or work out sth (UNDERSTAND) (Part of speech: verb)

  8. [16]

    For ambiguous cases, the PoS-based model performs reasonably well only at the A1 level

    None of the other options D Appendix D: Other results Table 9 reports the breakdown by word-level CEFR level in terms ofF1 score. For ambiguous cases, the PoS-based model performs reasonably well only at the A1 level. However, this result is partly influ- enced by the rule we ...

  9. [19]

    to try hard to achieve something - Additional information: work at sth (Part of speech: verb)

  10. [20]

    to spend time repairing or improv- ing something - Additional information: work on sth (Part of speech: verb)

  11. [21]

    to do a calculation to get an answer to a mathematical question - Additional information: work sth out or work out sth (Part of speech: verb)

  12. [2001]

    Burr Settles, Geoffrey T

    Developing and exploring the behaviour of two new versions of the V ocabulary Levels Test.Lan- guage Testing, 18(1):55–88. Burr Settles, Geoffrey T. LaFlair, and Masato Hagiwara

  13. [2018]

    Behavior research methods, 50(3):1030–1046

    The tool for the automatic analysis of lexi- cal sophistication (TAALES): version 2.0. Behavior research methods, 50(3):1030–1046. Kristopher Kyle and Scott A Crossley. 2015. Auto- matically assessing lexical sophistication: Indices, tools, findings, and application. TESOL Qua...

    2015 arXiv

  14. [2020]

    Transactions of the Association for computational Linguistics, 8:247–263

    Machine learning–driven language assessment. Transactions of the Association for computational Linguistics, 8:247–263. Yukio Tono. 2013. The CEFR-J handbook: A resource book for using CAN-DO descriptors for English lan- guage teaching. Tokyo: Taishukan. Jeanine Treffers-Daller...

    2013

  15. [2022]

    In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 6206–6219, Abu Dhabi, United Arab Emirates

    CEFR-based sentence difficulty annotation and assessment. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 6206–6219, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics. Melissa M. Baese-Berk, Shiloh D...

    2022

  16. [2023]

    Assessing Writing, 55:100688

    A multi-measure approach for lexical diversity in writing assessments: Considerations in measure- ment and timing. Assessing Writing, 55:100688. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, ...

  17. [2024]

    It was tough on the worn out employees

    Qwen2.5 Technical Report. arXiv preprint arXiv:2412.15115. Su-Youn Yoon, Suma Bhat, and Klaus Zechner. 2012. V ocabulary profile as a measure of vocabulary sophis- tication. In Proceedings of the seventh workshop on building educational applications using NLP, pages 180–189. G...

    2012 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.