Pith. sign in

REVIEW 3 major objections 6 minor 23 references

RETUYT-INCO at BEA 2025 Shared Task: How Far Can Lightweight Models Go in AI-powered Tutor Evaluation?

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that sub-billion-parameter models, fine-tuned on modest hardware, come within 6.46 to 13.13 exact-F1 points of the winning teams in the BEA 2025 AI tutor evaluation shared task, a gap the authors deem competitive.

desk verdict Honest, useful shared-task report showing sub-1B models land within 6-13 F1 points of winners; the 'competitive' label is generous but the data speaks for itself. read the letter →

arxiv 2506.11243 v1 pith:CUM5MUKP submitted 2025-06-12 cs.CL cs.AI

Santiago Góngora , Ignacio Sastre , Santiago Robaina , Ignacio Remersaro , Luis Chiruzzo , Aiala Rosá This is my paper
classification cs.CLcs.AI
keywords AI-poweredtutorevaluationpedagogicalabilityassessmentlightweightlanguagemodelssub-billionparametersharedtasklow-resourcecomputingDistilBERTQwen2.5
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reporting the team's entry in the BEA 2025 shared task on pedagogical ability assessment of AI-powered tutors, this paper tries to establish that a self-imposed restriction to models under 1 billion parameters costs little in performance. Across five tracks, the authors' best systems finished within 6.46 to 13.13 exact-F1 points of the winning teams, all while running on low-budget GPUs or no GPU. The claim matters because many research labs, particularly in the Global South, cannot access large-scale compute or paid APIs, and the authors argue the measured gap is an acceptable price for that accessibility. The paper presents the tradeoff as deliberate: sacrifice some performance to gain models that need far less training time, energy, and money.

What carries the argument

The central mechanism is a family of fine-tuning recipes for three-class judgment tasks that keep the active model tiny. DistilBERT is trained with the 'No' and 'To some extent' classes collapsed into one negative label, then the three-way prediction is recovered by thresholding the positive-class logit; Qwen2.5-0.5B-Instruct is fine-tuned with next-token prediction on prompt templates containing the conversation, rubric, and response, and the label is chosen by taking the highest logit among the three candidate first tokens, with per-dimension probability thresholds to correct under-prediction of 'To some extent'. Alongside these, sentence embeddings from multilingual-e5-large-instruct (561M parameters) feed a k-NN or a single-layer MLP, and classical TF-IDF classifiers (SVC, k-NN, Random Forest) serve as cheaper baselines. This menu lets the authors compare the 6-13 point gap against a preliminary LoRA fine-tuned Llama 3.1 8B, isolating model scale as the main variable under a fixed prompt and task.

What would settle it

Run the same task, data split, prompts, and tuning budget with the same architecture at several sizes (e.g., Qwen2.5 at 0.5B, 1.5B, and 7B) and compare exact F1 on the BEA 2025 test set; if the larger models show a gain much larger than the 6-13 point range, the paper's size explanation fails, whereas a small gain would corroborate it.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that lightweight, open models under 1 billion parameters are competitive with the state of the art in AI tutor evaluation. The best submission in each track—a balanced k-NN over e5 sentence embeddings for mistake identification, fine-tuned DistilBERT with logit thresholds for mistake location, fine-tuned Qwen2.5-0.5B-Instruct for providing guidance and actionability, and DistilBERT embeddings with XGBoost for tutor identification—finished 6.46, 10.24, 7.85, 9.56, and 13.13 exact-F1 points below the respective winners. The authors attribute this gap chiefly to the parameter-count restriction, pointing out that a preliminary LoRA fine-tune of Llama 3.1 8B was not significantly better and sometimes worse than their small models, and they present the tradeoff as deliberate: losing a few F1 points in exchange for models that run almost anywhere.

Load-bearing premise

Load-bearing premise: the 6.46-13.13 exact-F1 gap to the winners reflects mainly the sub-billion-parameter restriction, and not differences in ensembling, extra training data, or hyperparameter tuning, which the paper did not match or control for.

Editorial extensions

If this is right

  • If the gap is read as the price of the size constraint, then research groups without large GPU clusters or API budgets can deploy local AI tutor evaluation systems, keeping student data on-site.
  • The 6-13 point exact-F1 range provides a concrete published reference point for what sub-billion models can achieve on this task, which future low-resource systems can use as a baseline.
  • The finding that fine-tuned Qwen 0.5B roughly matches DistilBERT and the larger LoRA model suggests autoregressive models can serve as emergent classifiers at small scale, not just generators.
  • The observation that conversation history helps most for the more subtle dimensions (providing guidance, actionability) suggests lightweight systems should include context selectively rather than always or never.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the size explanation would be to run the same protocol with the same architecture at 0.5B, 1.5B, and 7B parameters, holding data, prompt, and tuning budget fixed; the paper's claim predicts only a modest gain from scale.
  • The threshold-based two-class-to-three-class trick could transfer to other coarse-grained categorical or ordinal NLP tasks with a rare middle class, provided logits are well-calibrated.
  • The paper's reported gaps are computed against the single winning team per track; computing the gap to the median or to the best lightweight team would give a fuller picture of where sub-billion models actually sit in the field.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper reports the RETUYT-INCO team's participation in the BEA 2025 shared task on pedagogical ability assessment of AI-powered tutors. The team imposed a self-reported restriction to models with fewer than 1B parameters (DistilBERT, Qwen2.5-0.5B, sentence embeddings, and classical ML), motivated by resource constraints common in the Global South. The authors describe their dev-set experiments, their five official test submissions, and the gaps between their best submissions and the winning team in each track according to exact F1. They report gaps ranging from 6.46 to 13.13 points and conclude that lightweight models are competitive with the state of the art. The paper also includes a dev-only comparison with a LoRA-tuned Llama 3.1 8B model, which did not clearly outperform the smaller models.

Significance. If the central claim is accepted, the paper provides a useful practical data point: sub-1B models and classical classifiers can achieve F1 scores within roughly 6-13 points of winning systems on this shared task, using no API access and modest computational resources. The paper's strengths are its reliance on official test scores, its transparency about quartile positions and limitations, and its use of simple, reproducible configurations. These strengths make it a valuable system description for low-resource NLP settings. However, the 'competitive' conclusion is currently supported mainly by gaps to a single winning team and needs additional distributional and attributional context before it can be considered established.

major comments (3)
  1. [Section 4.1, Table 3, Abstract] The central claim that models under 1B parameters are 'competitive' is based solely on exact-F1 gaps to the single winning team per track, but the paper's own team-level ranks in Table 3 are Q3 in every track (23/44, 21/32, 17/35, 17/29, 12/20). A gap to the leader is not by itself evidence of closeness to the state of the art: in a dense leaderboard a mid-pack system can be within a few F1 points of the winner while being beaten by many other teams, and in a sparse leaderboard the gap can be small even if the winner is an outlier. Since the paper does not report the distribution of team scores (median, interquartile range, second/third place scores, or the team's percentile), the abstract's 'competitive' conclusion is not directly supported. Please add this distributional context, or explicitly reframe the conclusion as 'moderate F1 gaps to the winner despite mid-pack rankings.'
  2. [Section 4, Tables 1-2, Limitations] The dev-to-test generalization is a load-bearing concern. Models and thresholds were selected on the authors' own 80/20 split (Section 2), and several best dev configurations drop substantially on the official test (e.g., Track 1 e5 k-NN balanced from 79.16 to 65.35 F1-macro; Track 2 DistilBERT thresholds from 56.30 to 49.58; Track 3 Qwen multi-dimension from 53.26 to 50.49). The authors acknowledge this in the Limitations but do not quantify the variability or verify the claimed cause ('differences in class distributions'). Because the reported test gaps in Table 3 are for systems chosen via dev-based selection, selection noise could affect the magnitude of those gaps. Please report repeated runs, confidence intervals, or an analysis of the dev/test distribution shift.
  3. [Section 3.5, Section 4.1] The paper implicitly attributes the modest gaps to model size, but winning teams very likely differed in ensembling, training data, and hyperparameter search. The only size comparison is the dev-only Llama 3.1 8B LoRA result in Table 1, which is not a controlled experiment: it uses a single LoRA configuration, no threshold tuning, and is not evaluated on the official test set. The conclusion that model size is the main driver of the performance difference is therefore not directly tested. Either provide a comparable setup for a larger model on the test set, or restrict the claim to 'our lightweight systems achieved these gaps' rather than 'lightweight models are competitive with state-of-the-art models.'
minor comments (6)
  1. [Abstract, Section 3.4.1] The abstract states that all of the approaches 'can be run on computers with a low-budget GPU or even without a GPU,' but the Qwen fine-tuning in Section 3.4.1 was trained on A100/A40 GPUs via ClusterUY. Please distinguish training from inference costs, since the sentence as written is misleading.
  2. [Section 3.5] There is a typo in the Track 4 bullet: 'apporach' should be 'approach.'
  3. [Section 4] Missing spaces in 'ourdev' and 'thetest' in the paragraph comparing dev and test sets.
  4. [Table 3] Minor formatting issue: '06.46' has a leading zero inconsistent with the other gaps; also the table caption says the table is built on team results while Table 2 counts submissions, which should be explained in the body text.
  5. [Tables 1-2] Metric names are inconsistent: Table 1 and Table 2 use 'F1-macro' while the abstract and Table 3 use 'exact F1'; please clarify which metric is the official one and label the tables accordingly.
  6. [Conclusions] In the Conclusions, 'QWEN 0.5B' should be spelled consistently as 'Qwen2.5-0.5B' or 'Qwen 0.5B.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the competitiveness claim is an interpretation of externally published shared-task scores, not a derived result.

full rationale

The paper's central empirical claim is that sub-1B models are competitive, supported by gaps to winning teams computed from the organizers' published exact F1 scores (Section 4.1, Table 3). These gaps are measurements against external shared-task results, not predictions derived from fitted parameters. The dev set is used for model selection and threshold tuning (Sections 3.3-3.5), which is standard practice and disclosed; the final test scores are held-out official evaluations. Self-citations to prior BEA participations are contextual and not load-bearing for the 2025 results. Thresholds are manually set based on validation statistics, and the paper acknowledges they may not generalize well; the target claim (test gap to winners) is not defined in terms of those thresholds. There is no equation in which the output reduces to the input, no fitted parameter renamed as a prediction, and no uniqueness theorem or ansatz smuggled in via citation. The skeptical concern that a gap to the winner is not enough to establish 'competitiveness' is an interpretive or correctness issue, not circularity, because the gap is an independently measured external quantity. This is a non-finding: the derivation chain is self-contained with respect to the shared-task evaluation.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central empirical claim is evaluated against external test scores, so it has no free parameters that directly define the outcome. The listed free parameters are standard hyperparameters and thresholds tuned on a validation split, which is normal practice. The key domain assumption is that the gap to the winners is a fair proxy for the gap due to model size.

free parameters (4)
  • Qwen threshold rules (Track 1-4) = Yes condition and No condition per dimension, e.g., Track 1: Yes>0.90 & No<0.05; Track 2: Yes>0.75 & No<0.15; etc.
    Manually defined thresholds tuned on a 20% validation split of the training data, used to separate Yes/No/To some extent classes in Qwen inference.
  • DistilBERT logit thresholds for Tracks 1-4 = Between -1 and +1, varied per track and target
    Two thresholds distinguish No, To some extent, and Yes from the logit of the positive class in the two-class DistilBERT model.
  • k values for k-NN classifiers = k=9 for e5 sentence embeddings; k=415, 540, 125, 96 for balanced k-NN per track
    Chosen based on dev set performance before the test submission.
  • TF-IDF n-gram ranges = Various, e.g., (1,2), (1,5), (1,7), (1,8) and (2,5)
    Selected after experimentation on the dev set; the best configurations are reported.
assumptions (3)
  • domain assumption The official shared task test set and its human-annotated gold labels are the ground truth.
    The paper uses the organizer-published scores and test labels as the basis for all conclusions about competitiveness.
  • domain assumption The performance difference between the winning teams and the RETUYT-INCO systems is primarily attributable to model size and resource constraints.
    This assumption underlies the interpretation of the F1 gaps as evidence that lightweight models are competitive, but confounding factors are not controlled.
  • domain assumption The human evaluation labels are consistent and unbiased across the dataset.
    The task defines the target labels, and the paper does not question the quality of the human annotations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RETUYT-INCO at BEA 2025 Shared Task: How Far Can Lightweight Models Go in AI-powered Tutor Evaluation?." pith.science (2026). https://pith.science/paper/CUM5MUKP

@misc{pith2026250611243,
  author       = {Pith},
  title        = {Pith review of: RETUYT-INCO at BEA 2025 Shared Task: How Far Can Lightweight Models Go in AI-powered Tutor Evaluation?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CUM5MUKP}},
  note         = {Machine review of arXiv:2506.11243}
}
abstract

In this paper, we present the RETUYT-INCO participation at the BEA 2025 shared task. Our participation was characterized by the decision of using relatively small models, with fewer than 1B parameters. This self-imposed restriction tries to represent the conditions in which many research labs or institutions are in the Global South, where computational power is not easily accessible due to its prohibitive cost. Even under this restrictive self-imposed setting, our models managed to stay competitive with the rest of teams that participated in the shared task. According to the $exact\ F_1$ scores published by the organizers, the performance gaps between our models and the winners were as follows: $6.46$ in Track 1; $10.24$ in Track 2; $7.85$ in Track 3; $9.56$ in Track 4; and $13.13$ in Track 5. Considering that the minimum difference with a winner team is $6.46$ points -- and the maximum difference is $13.13$ -- according to the $exact\ F_1$ score, we find that models with a size smaller than 1B parameters are competitive for these tasks, all of which can be run on computers with a low-budget GPU or even without a GPU.

Figures

Figures reproduced from arXiv: 2506.11243 by the authors.

Figure 1
Figure 1. Class balance in our train and dev sets. The columns are coupled according to classes. LLMs and use them in real applications. Moreover, this is not the only motivation for this self-imposed restriction, as one of the research lines of our lab is the application of NLP tools to aid teachers in rural areas (Chiruzzo et al., 2022; Rosá et al., 2025). In such contexts it is very unlikely to use state-of-the-art LLMs, d… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 12 canonical work pages

  1. [1]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  2. [2]

    Alexis Balad \'o n, Ignacio Sastre, Luis Chiruzzo, and Aiala Ros \'a . 2023. https://doi.org/10.18653/v1/2023.bea-1.61 RETUYT - I n C o at BEA 2023 shared task: Tuning open-source LLM s for generating teacher responses . In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 756--765, Toronto, ...

    work page doi:10.18653/v1/2023.bea-1.61 2023

  3. [3]

    Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785--794

    2016

  4. [4]

    Luis Chiruzzo, Laura Musto, Santiago Gongora, Brian Carpenter, Juan Filevich, and Aiala Rosa. 2022. https://doi.org/10.18653/v1/2022.nlp4pi-1.13 Using NLP to support E nglish teaching in rural schools . In Proceedings of the Second Workshop on NLP for Positive Impact (NLP4PI), pages 113--121, Abu Dhabi, United Arab Emirates (Hybrid). Association for Compu...

    work page doi:10.18653/v1/2022.nlp4pi-1.13 2022

  5. [5]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \' a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1911.02116 Unsupervised cross-lingual representation learning at scale . CoRR, abs/1911.02116

    arXiv 2019

  6. [6]

    Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Chukwunyere Osi, Prateek Sharma, Fan Chen, and Lei Jiang. 2024. Llmcarbon: Modeling the end-to-end carbon footprint of large language models. In The Twelfth International Conference on Learning Representations

    work page 2024

  7. [7]

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In International Conference on Learning Representations

    2022

  8. [8]

    Nwatu, Veronica Perez-Rosas, Siqi Shen, Zekun Wang, Winston Wu, and Rada Mihalcea

    Oana Ignat, Zhijing Jin, Artem Abzaliev, Laura Biester, Santiago Castro, Naihao Deng, Xinyi Gao, Aylin Ece Gunal, Jacky He, Ashkan Kazemi, Muhammad Khalifa, Namho Koh, Andrew Lee, Siyang Liu, Do June Min, Shinka Mori, Joan C. Nwatu, Veronica Perez-Rosas, Siqi Shen, Zekun Wang, Winston Wu, and Rada Mihalcea. 2024. https://aclanthology.org/2024.lrec-main.70...

    work page 2024

Show all 23 references
  1. [9]

    Ekaterina Kochmar, Kaushal Kumar Maurya, Kseniia Petukhova, K. V. Aditya Srivatsa, Anaïs Tack, and Justin Vasselli. 2025. Findings of the BEA 2025 Shared Task on Pedagogical Ability Assessment of AI-powered Tutors . In Proceedings of the 20th Workshop on Innovative Use of NLP ...

    2025

  2. [10]

    Vivian Liu and Yiqiao Yin. 2024. Green ai: exploring carbon footprints, mitigation strategies, and trade offs in large language model training. Discover Artificial Intelligence, 4(1):49

    2024

  3. [11]

    Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. 2023. Estimating the carbon footprint of bloom, a 176b parameter language model. Journal of Machine Learning Research, 24(253):1--15

    2023

  4. [12]

    Kaushal Kumar Maurya, Kv Aditya Srivatsa, Kseniia Petukhova, and Ekaterina Kochmar. 2025. https://aclanthology.org/2025.naacl-long.57/ Unifying AI tutor evaluation: An evaluation taxonomy for pedagogical ability assessment of LLM -powered AI tutors . In Proceedings of the 2025...

    2025

  5. [13]

    Sergio Nesmachnow and Santiago Iturriaga. 2019. Cluster-uy: Collaborative scientific high performance computing in uruguay. In Supercomputing, pages 188--202, Cham. Springer International Publishing

    2019

  6. [14]

    Nils Reimers and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/D19-1410 Sentence- BERT : Sentence embeddings using S iamese BERT -networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference...

    2019 doi

  7. [15]

    Aiala Rosá, Santiago Góngora, Juan Pablo Filevich, Ignacio Sastre, Laura Musto, Brian Carpenter, and Luis Chiruzzo. 2025. https://arxiv.org/abs/2504.20251 A platform for generating educational activities to teach english as a second language . Preprint, arXiv:2504.20251

    2025 arXiv

  8. [16]

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. https://arxiv.org/abs/1910.01108 Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter . Preprint, arXiv:1910.01108

    2020 arXiv

  9. [17]

    Ignacio Sastre, Leandro Alfonso, Facundo Fleitas, Federico Gil, Andr \'e s Lucas, Tom \'a s Spoturno, Santiago G \'o ngora, Aiala Ros \'a , and Luis Chiruzzo. 2024. https://aclanthology.org/2024.bea-1.56/ RETUYT - INCO at MLSP 2024: Experiments on language simplification using...

    2024

  10. [18]

    Matthew Shardlow, Fernando Alva-Manchego, Riza Batista-Navarro, Stefan Bott, Saul Calderon Ramirez, R \'e mi Cardon, Thomas Fran c ois, Akio Hayakawa, Andrea Horbach, Anna H \"u lsing, Yusuke Ide, Joseph Marvin Imperial, Adam Nohejl, Kai North, Laura Occhipinti, Nelson Per \'e...

    2024

  11. [19]

    Ana \"i s Tack, Ekaterina Kochmar, Zheng Yuan, Serge Bibauw, and Chris Piech. 2023. https://doi.org/10.18653/v1/2023.bea-1.64 The BEA 2023 shared task on generating AI teacher responses in educational dialogues . In Proceedings of the 18th Workshop on Innovative Use of NLP for...

    2023 doi

  12. [20]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  13. [21]

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672

    2024 arXiv

  14. [22]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

    2024 arXiv

  15. [23]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In International Conference on Learning Representations

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.