Pith. sign in

REVIEW 4 major objections 5 minor 40 references

The Veln(ia)s is in the Details: Evaluating LLM Judgment on Latvian and Lithuanian Short Answer Matching

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Large open-weights LLMs, especially QWEN2.5 72B and LLaMa3.1 70B, distinguish matched from subtly altered short answers in Latvian and Lithuanian at near-perfect accuracy.

desk verdict A genuinely new low-resource dataset benchmark, but class-level label validation and sloppy dataset arithmetic mean the headline F1 claims need revision before they should be trusted. read the letter →

arxiv 2501.09164 v1 pith:X7ADFAMJ submitted 2025-01-15 cs.CL cs.AI

Yevhen Kostiuk , Oxana Vitman , {L}ukasz Gaga{l}a , Artur Kiulian This is my paper · ORCID
classification cs.CLcs.AI
keywords shortanswermatchingautomaticgradingLatvianLithuanianlow-resourcelanguageslargelanguagemodelsfew-shotpromptingalterationrules
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether open-weights language models can tell matched short answers apart from subtly altered non-matched ones in Latvian and Lithuanian, two low-resource Baltic languages where automated answer grading has little data. To test this, the authors built two Wikipedia-derived datasets—502 Latvian and 690 Lithuanian question-answer pairs—and generated matched and non-matched variants with six alteration rules, yielding 3,012 and 4,830 test triplets, with a small manually verified subset. They benchmarked seven open models in zero-shot and few-shot settings. Their central finding is that the largest models, QWEN2.5 72B and LLaMa3.1 70B, classify matched versus non-matched almost perfectly in both languages under both prompting settings, while smaller models vary: some improve with few-shot examples, and Mistral Nemo 12B is noticeably weaker on Lithuanian. If this holds, automatic short-answer grading for these languages is within reach, and the released datasets give the community a way to measure it.

What carries the argument

The mechanism that carries the argument is the alteration-rule taxonomy. Four matched rules—adding question-related entities (Ents), changing words to synonyms (Synonyms), adding background information (MoreInfo), and swapping style to exclamatory (Exclamatory)—generate answers that should be accepted; two non-matched rules—incorporating minor changes (IMC) and changing domain-related information (CDRI)—generate near-identical wrong answers by swapping a key word. Every test item is a (question, reference answer, generated answer) triple labeled as matched or non-matched, and each model must start its output with True or False. The per-rule breakdown turns the aggregate F1 into a diagnostic of which kinds of semantic alertness a model has, and the language-matched few-shot prompt is what separates the small models that recover with examples from those that do not.

What would settle it

Re-run the manual annotation on all 3,012 Latvian and 4,830 Lithuanian triplets, or on a much larger random sample stratified by alteration rule. If native-speaker rejection rates on the unverified items are substantially higher than the accept ratios reported in the manual evaluation, the near-perfect model F1 scores are partly measuring template artifacts rather than genuine semantic matching.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the relevant question is less whether LLMs can catch subtle answer mismatches in Latvian and Lithuanian than which model size and prompting recipe makes that reliable. In the released datasets, QWEN2.5 72B and LLaMa3.1 70B reach F1 around 0.99 on binary matched/non-matched classification in both zero-shot and few-shot conditions, with QWEN2.5 7B and Mistral 7B close behind despite their smaller size. Few-shot prompting is not uniformly helpful: it rescues EuroLLM 9B from zero-shot output that cannot even be parsed into True/False and lifts LLaMa3.1 8B, but it slightly lowers Mistral Nemo 12B and Mistral 7B. The rule-level analysis isolates the failure modes—Mistral Nemo misses synonym substitution and exclamatory-style matches in Lithuanian, while LLaMa3.1 8B and EuroLLM 9B struggle with added entities in Latvian—which is the paper's most useful output beyond aggregate accuracy.

Load-bearing premise

The load-bearing premise is that the automatically generated match/non-match labels are correct for the whole dataset, even though only 360 random samples per language were manually verified and the rest was kept based on aggregate accept ratios.

Editorial extensions

If this is right

  • Large open-weights models around 70B parameters can serve as short-answer matchers in Latvian and Lithuanian without task-specific training, at essentially ceiling-level accuracy on these datasets.
  • Few-shot prompting is a cheap remedy for some small models—EuroLLM 9B and LLaMa3.1 8B gain substantially—but it is not a general fix, since Mistral Nemo 12B and Mistral 7B score slightly worse with examples.
  • Per-rule failure patterns such as synonym detection and added-entity recognition give concrete targets for evaluation and fine-tuning rather than a single pass/fail score.
  • The released 3,012 Latvian and 4,830 Lithuanian triplets provide a reusable benchmark for future multilingual and Baltic-language models.
  • A model that cannot follow a simple True/False output instruction in zero-shot is not usable as an automatic grader regardless of its underlying language ability; few-shot examples can repair this for EuroLLM 9B.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the non-matched answers were generated from fixed templates, the near-perfect scores may partly reflect predictable surface patterns rather than robust semantic judgment; real student answers in Latvian and Lithuanian are likely messier and may lower accuracy.
  • The rule-level breakdown could be reused as a diagnostic suite: any model claiming support for these languages could be scored per rule, and synthetic data targeting the weakest rules could be used to improve small models.
  • The dataset design does not control for whether a model uses the question at all; swapping in an unrelated question while keeping the same reference and candidate answers would test whether the model is truly doing answer matching or just textual comparison.
  • Because only 360 samples per language were manually verified and the remaining labels were kept on aggregate accept ratios, the published accuracies are best read as upper bounds until the full labels are independently checked.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces new question-answer matching datasets for Latvian and Lithuanian, generated by applying six alteration rules to Wikipedia-derived reference answers with GPT-4o and LLaMa3, followed by a partial manual validation. It then evaluates seven open LLMs in zero-shot and few-shot settings to classify generated answers as matched or non-matched. The headline finding is that larger models (QWEN2.5 72b, LLaMa3.1 70b) achieve near-perfect F1 scores in both languages, while smaller models vary, with some benefiting from few-shot examples and others degrading. The authors release code and data on GitHub.

Significance. If the released datasets and labels are reliable, this is a useful resource for an under-resourced language pair: it provides multi-domain question-answer pairs with a transparent set of matched/non-matched alteration rules, and it gives a reproducible benchmark for open LLMs on a low-resource short-answer-matching task. The paper explicitly includes a manual validation component, which is a strength, and the evaluation protocol is simple enough to be replicated. The main value is empirical rather than methodological; the conclusions about model rankings and about which alteration rules are difficult are potentially useful for practitioners. However, the dataset-size arithmetic is inconsistent, and the label-validation procedure is too coarse to support the precision of the reported per-model differences. The paper needs a careful reconciliation of dataset figures and a clearer item-level audit before the accuracy claims can be accepted.

major comments (4)
  1. [Section 3.2] The dataset-size accounting is internally inconsistent. The text reports 3,012 Latvian triplets (1,506 matched / 1,506 non-matched) and 4,830 Lithuanian triplets (2,760 matched / 2,070 non-matched), then immediately states that the amount of matched answers is 3,697 and the amount of non-matched answers is 1,809. These two final numbers sum to 5,506, which matches neither the per-language total (7,842) nor the sum of the parenthetical matched and non-matched counts (4,266 matched / 3,576 non-matched). The paper must reconcile these figures and specify which numbers describe the final released dataset after duplicate removal and class filtering.
  2. [Section 3.3 / Appendix A] The ground-truth labels are validated only at class level, not at item level. Two annotators per language reviewed 360 random triplets, and Table 2 reports aggregate accept/reject counts per generation class; the final dataset is then produced by accepting or rejecting entire classes. Because the manual sample was drawn before duplicate removal and before the selective class filtering, the observed reject rates do not directly estimate the label-error rate on the final test items. A mislabeled cluster from one generation class would enter the benchmark unexamined, and models that happen to agree with the intended label would be scored correct even when the label is wrong. The reported differences between models are small enough (e.g., Mistral Nemo 12b at 0.94–0.96 versus QWEN2.5 72b at 0.99) that a few percent of systematic label noise could shift model rankings. The paper reports no inter-annotator agreement statistic, no per-rule item counts, and no confidence intervals. An item-level audit of the released labels, or at minimum per-class precision estimates with intervals, is needed before the headline accuracy claims can be accepted.
  3. [Section 5 / Table 1] The EuroLLM 9b zero-shot F1 scores conflate instruction-following with matching ability. The text states that in zero-shot settings EuroLLM 9b was not able to follow the output format at all and that its results were legible but impossible to parse, yet Table 1 reports F1 = 0.13 for Lithuanian ZS and F1 = 0.05 for Latvian ZS, and the discussion treats these numbers as a language-difference finding. A model whose outputs cannot be parsed should be excluded from the F1 computation or reported separately as a format-compliance failure. As written, the zero-shot comparison for EuroLLM is not interpretable as a matching-accuracy result.
  4. [Section 5 / Table 1 and Figures 3–4] The statistical support for the model comparisons is missing. Table 1 reports only one F1 number per model/setting/language, with no sample sizes, precision/recall, or confidence intervals, and Figures 3–4 show per-rule accuracy without numerical values or error bars. The conclusions that Mistral 7b performed weaker in few-shot experiments and that QWEN2.5 7b is comparable to the larger 70b models rest on differences as small as 0.01–0.02, which may be within sampling noise given the item counts. Please provide per-cell counts and interval estimates, or explicitly soften the comparative claims.
minor comments (5)
  1. [Section 3.2 / Appendix A] The generation model is referred to as LLaMa3 7b in Section 3.2 and as LLaMa3:7b in Table 2, while the evaluated models are LLaMa3.1 8b and 70b; the paper should clarify which model versions were used for generation and which for evaluation.
  2. [Section 5 / Table 1] The paper uses "accuracy", "F1 score", and "accuracy score" interchangeably; Table 1 is explicitly labeled F1, so the text should consistently report F1 or explicitly state that the reported values are F1.
  3. [Section 3.3] The statement that GPT-4o Match MoreInfo "was excluded by mistake" for Latvian is a data-construction error that should be acknowledged as such and analyzed for its effect on the per-rule comparisons, since readers cannot tell whether the omission is intentional.
  4. [Throughout] There are several typos and grammatical slips, including "perfromed", "MIstral", "similar the performance", and "flows" for "flaws"; a careful proofreading pass is needed.
  5. [Section 3.1 / Abstract] The abstract reports 502 Latvian and 690 Lithuanian question-answer pairs, and Section 3.1 reports 502 and 690 unique questions; these match, but it would be helpful to state explicitly that these counts refer to the base QA pairs before generated triplets are added.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports an external LLM benchmark on fixed labels, with no derivation step that reduces to its inputs.

full rationale

The paper's central claim is an empirical evaluation: several open-weight LLMs are prompted to classify matched versus non-matched short answers in two newly released datasets, and their F1 scores are reported. There is no fitted parameter that is later renamed a prediction, no uniqueness theorem imported from the authors' prior work, and no equation whose output equals its input by construction. The only self-citation is the use of the authors' earlier KID pipeline to generate Wikipedia-based question-answer pairs, which is a data-construction choice rather than a load-bearing theoretical premise. The fact that GPT-4o and LLaMa3 were used to generate candidate labels and that only a 360-sample subset per language was manually reviewed is a genuine data-quality limitation, but it is not circularity: the evaluated models are not the generators, the manual sample is an external check, and the F1 results are not derived from the annotation accept ratios. Any concern about aggregate-level filtering affecting label noise belongs to correctness risk, not to circularity, because the paper does not define its predictions in terms of those labels. The benchmark remains self-contained as an external evaluation artifact.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central benchmark rests on the reliability of automatically generated labels and on treating instruction-following failures as task failures. No numerical parameters are fitted to the evaluation data, so the ledger contains no free parameters, but the label-generation and filtering assumptions are load-bearing.

assumptions (4)
  • domain assumption The partially manual annotation (360 samples per language) is sufficient to certify the match/non-match labels of the full automatically generated dataset.
    Section 3.3 filters generation methods by accept ratio and then treats the remaining labels as ground truth; no full or second-pass verification is described.
  • domain assumption Reference answers extracted from Wikipedia and generated by Gemini 1.5 Pro are acceptable gold answers for the matching task.
    Section 3.1 relies on Gemini-generated questions and reference answers; factual correctness is explicitly declared irrelevant, but semantic acceptability is assumed.
  • domain assumption Counting an unparseable model output as an incorrect answer is a valid measurement of answer-matching ability.
    Section 4 says outputs not containing 'True' or 'False' are counted as incorrect; this assumption drives EuroLLM 9b's near-zero zero-shot scores.
  • domain assumption The defined alteration rules produce matched and non-matched answers that are genuinely subtle and meaningful for the benchmark.
    Section 3.2 defines the rules and the authors state they introduce 'small but meaningful changes,' but only a sample of the outputs was manually checked.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Veln(ia)s is in the Details: Evaluating LLM Judgment on Latvian and Lithuanian Short Answer Matching." pith.science (2026). https://pith.science/paper/X7ADFAMJ

@misc{pith2026250109164,
  author       = {Pith},
  title        = {Pith review of: The Veln(ia)s is in the Details: Evaluating LLM Judgment on Latvian and Lithuanian Short Answer Matching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X7ADFAMJ}},
  note         = {Machine review of arXiv:2501.09164}
}
read the original abstract

In this work, we address the challenge of evaluating large language models (LLMs) on the short answer matching task for Latvian and Lithuanian languages. We introduce novel datasets consisting of 502 Latvian and 690 Lithuanian question-answer pairs. For each question-answer pair, we generated matched and non-matched answers using a set of alteration rules specifically designed to introduce small but meaningful changes in the text. These generated answers serve as test cases to assess the ability of LLMs to detect subtle differences in matching of the original answers. A subset of the datasets was manually verified for quality and accuracy. Our results show that while larger LLMs, such as QWEN2.5 72b and LLaMa3.1 70b, demonstrate near-perfect performance in distinguishing matched and non-matched answers, smaller models show more variance. For instance, LLaMa3.1 8b and EuroLLM 9b benefited from few-shot examples, while Mistral Nemo 12b underperformed on detection of subtle text alteration, particularly in Lithuanian, even with additional examples. QWEN2.5 7b and Mistral 7b were able to obtain a strong and comparable performance to the larger 70b models in zero and few shot experiments. Moreover, the performance of Mistral 7b was weaker in few shot experiments.

Figures

Figures reproduced from arXiv: 2501.09164 by the authors.

Figure 1
Figure 1. Example of the element from Lithuanian generated dataset. text alteration rules (AR) that are minor when it comes to a text change, but semantically are major. For each rule, the different LLMs with a few shot generation process were used. Finally, to ensure the quality, we manually evaluated a sample of the data and filtered the final dataset based on it. We expect the models to obtain high, almost perfect, results… view at source ↗
Figure 2
Figure 2. Example of few-shot incorporating mi￾nor changes prompt for non-matched answers gen￾eration in Latvian. < . . . > indicate the sample that requires prediction. On the second stage, for each pair of question and answer, we defined a list of different alteration rules that rewrites reference answer to matched or non-matched (more details are provided in the Section 3.2). We used GPT-4o and LLaMa3 8b (see [PITH_FULL_I… view at source ↗
Figure 3
Figure 3. Accuracy scores per generated answer type for Latvian. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Accuracy scores per generated answer type for Lithuanian. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 23 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Sridevi Bonthu, S Rama Sree, and MHM Krishna Prasad. 2021. Automated short answer grading using deep learning: A survey. In Machine Learning and Knowledge Extraction: 5th IFIP TC 5, TC 12, WG 8.4, WG 8.9, WG 12.9 International Cross-Domain Conference, CD-MAKE 2021, Virtual Event, August 17--20, 2021, Proceedings 5, pages 61--78. Springer

    work page 2021

  4. [4]

    Steven Burrows, Iryna Gurevych, and Benno Stein. 2015. The eras and trends of automatic short answer grading. International journal of artificial intelligence in education, 25:60--117

    work page 2015

  5. [5]

    Changzhi Cai. 2019. Automatic essay scoring with recurrent neural network. In Proceedings of the 3rd International Conference on High Performance Compilation, Computing and Communications, pages 1--7

    work page 2019

  6. [6]

    Li-Hsin Chang and Filip Ginter. 2024. Automatic short answer grading for finnish with chatgpt. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 23173--23181

    work page 2024

  7. [7]

    Li-Hsin Chang, Jenna Kanerva, and Filip Ginter. 2022. https://doi.org/10.18653/v1/2022.bea-1.30 Towards automatic short answer assessment for F innish as a paraphrase retrieval task . In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), pages 262--271, Seattle, Washington. Association for Computati...

    doi:10.18653/v1/2022.bea-1.30 2022

  8. [8]

    Zhiyun Chen and Yuxin Zhou. 2019. Research on automatic essay scoring of composition based on cnn and or. In 2019 2nd International Conference on Artificial Intelligence and Big Data (ICAIBD), pages 13--18. IEEE

    work page 2019

Show all 40 references
  1. [9]

    Yucheng Chu, Hang Li, Kaiqi Yang, Harry Shomer, Hui Liu, Yasemin Copur-Gencturk, and Jiliang Tang. 2024. A llm-powered automatic grading framework with human-level guidelines optimization. arXiv preprint arXiv:2410.02165

    2024 arXiv

  2. [10]

    Roberts Dar \`g is, Ilze Auzi n a, Inga Kaija, Krist \= ne Lev \=a ne-Petrova, and Krist \= ne Pokratniece. 2022. Corpus based self-assessment platform for latvian language learners. Baltic Journal of Modern Computing

    2022

  3. [11]

    Roberts Dar g is, Ilze Auzi n a, Inga Kaija, Krist \= ne Lev \=a ne-Petrova, and Krist \= ne Pokratniece. 2022. https://aclanthology.org/2022.lrec-1.77 L a VA -- L atvian language learner corpus . In Proceedings of the Thirteenth Language Resources and Evaluation Conference, p...

    2022

  4. [12]

    Roberts Dar g is, Guntis B \=a rzdi n s , Inguna Skadi n a, and Baiba Saulite. 2024. https://doi.org/10.18653/v1/2024.nlp4dh-1.28 Evaluating open-source LLM s in low-resource languages: Insights from L atvian high school exams . In Proceedings of the 4th International Conferen...

    2024 doi

  5. [13]

    Arunima Divya, Vivek Haridas, and Jayasree Narayanan. 2023. Automation of short answer grading techniques: Comparative study using deep learning techniques. In 2023 Fifth International Conference on Electrical, Computer and Communication Technologies (ICECCT), pages 1--7. IEEE

    2023

  6. [14]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    2024 arXiv

  7. [15]

    Christian Gr \'e visse. 2024. Llm-based automatic short answer grading in undergraduate medical education. BMC Medical Education, 24(1):1060

    2024

  8. [16]

    Veronika Hackl, Alexandra Elena M \"u ller, Michael Granitzer, and Maximilian Sailer. 2023. Is gpt-4 a reliable rater? evaluating consistency in gpt-4's text ratings. In Frontiers in Education, volume 8, page 1272229. Frontiers Media SA

    2023

  9. [17]

    Rositsa V Ivanova and Siegfried Handschuh. 2024. Evaluating llms’ performance at automatic short-answer grading

    2024

  10. [18]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  11. [19]

    Artur Kiulian, Anton Polishko, Mykola Khandoga, Oryna Chubych, Jack Connor, Raghav Ravishankar, and Adarsh Shirawalmath. 2024. From bytes to borsch: Fine-tuning gemma and mistral for the ukrainian language representation. arXiv preprint arXiv:2404.09138

    2024 arXiv

  12. [20]

    Pernille Andresen Klevstuen. 2022. Assisting efficient and fair grading with information retrieval and text mining techniques. Master's thesis, NTNU

    2022

  13. [21]

    R Krithika and Jayasree Narayanan. 2015. Learning to grade short answers using machine learning techniques. In Proceedings of the Third International Symposium on Women in Computing and Informatics, pages 262--271

    2015

  14. [22]

    VD Lai, NT Ngo, APB Veyseh, H Man, F Dernoncourt, T Bui, and TH Nguyen. 2023. Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning. arxiv

    2023

  15. [23]

    Krist \= ne Lev \=a ne-Petrova, Roberts Darģis, Krist \= ne Pokratniece, and Viesturs J \=u lijs Lasmanis. 2023. http://hdl.handle.net/20.500.12574/84 Balanced corpus of modern latvian ( LVK2022 ) . CLARIN - LV digital library at IMCS , University of Latvia

    2023

  16. [24]

    Guerreiro, Ricardo Rei, Duarte M

    Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. 2024. http://ar...

    2024 arXiv

  17. [25]

    Plöger, and Jörn Hees

    Tim Metzler, Paul G. Plöger, and Jörn Hees. 2024. https://doi.org/10.18420/inf2024_121 Computer-assisted short answer grading using large language models and rubrics . In INFORMATIK 2024, pages 1383--1393. Gesellschaft für Informatik e.V., Bonn

    2024 doi

  18. [26]

    Atsushi Mizumoto and Masaki Eguchi. 2023. Exploring the potential of using an ai language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2):100050

    2023

  19. [27]

    Lekshmi R Pillai, G Veena, and Deepa Gupta. 2018. A combined approach using semantic role labelling and word sense disambiguation for question generation and answer extraction. In 2018 Second International Conference on Advances in Electronics, Computers and Communications (IC...

    2018

  20. [28]

    Johannes Schneider, Bernd Schenk, and Christina Niklaus. 2023. Towards llm-based autograding for short textual answers. arXiv preprint arXiv:2309.11508

    2023 arXiv

  21. [29]

    V Sreevidhya and Jayasree Narayanan. 2021. Short descriptive answer evaluation using word-embedding techniques. In 2021 12th international conference on computing communication and networking technologies (ICCCNT), pages 1--4. IEEE

    2021

  22. [30]

    Pavel Stefanovi c , Birut \.e Pliuskuvien \.e , Urt \.e Radvilait \.e , and Simona Ramanauskait \.e . 2024. Machine learning model for chatgpt usage detection in students’ answers to open-ended questions: Case of lithuanian language. Education and Information Technologies, pages 1--23

    2024

  23. [31]

    Chul Sung, Tejas Dhamecha, Swarnadeep Saha, Tengfei Ma, Vinay Reddy, and Rishi Arora. 2019. Pre-training bert on domain resources for short answer grading. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint...

    2019

  24. [32]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, A...

    2024 arXiv

  25. [33]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  26. [34]

    Maximilian Tornqvist, Mosleh Mahamud, Erick Mendez Guzman, and Alexandra Farazouli. 2023. Exasag: Explainable framework for automatic short answer grading. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 361--371

    2023

  27. [35]

    Rebecka Weegar and Peter Idestam-Almquist. 2024. Reducing workload in short answer grading using machine learning. International Journal of Artificial Intelligence in Education, 34(2):247--273

    2024

  28. [36]

    Nico Willms and Ulrike Pad \'o . 2022. A transformer for sag: What does it grade? In Swedish Language Technology Conference and NLP4CALL, pages 114--122

    2022

  29. [37]

    Kevin P Yancey, Geoffrey Laflair, Anthony Verardi, and Jill Burstein. 2023. Rating short l2 essays on the cefr scale with gpt-4. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 576--584

    2023

  30. [38]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

    2024 arXiv

  31. [39]

    Su-Youn Yoon. 2023. Short answer grading using one-shot prompting and text similarity scoring model. arXiv preprint arXiv:2305.18638

    2023 arXiv

  32. [40]

    Automatic short math answer grading via in-context meta-learning

    M Zhang, S Baral, N Heffernan, and A Lan. Automatic short math answer grading via in-context meta-learning. arxiv 2022. arXiv preprint arXiv:2205.15219

    2022 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.