Pith. sign in

REVIEW 6 major objections 6 minor 21 references

BD at BEA 2025 Shared Task: MPNet Ensembles for Pedagogical Mistake Identification and Localization in AI Tutor Responses

T0 review · 6 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A hard-voting ensemble of ten fine-tuned MPNet classifiers identifies and localizes mistakes in AI tutor responses nearly as well as the top-ranked systems in the BEA 2025 shared task, reaching macro-F1 of 0.711 and 0.554.

desk verdict A credible shared-task system report with real organizer-scored results and a useful error taxonomy, but the method section has too many internal contradictions to be reproducible without cleanup. read the letter →

arxiv 2506.01817 v1 pith:JEWMFLRZ submitted 2025-06-02 cs.CL

Shadman Rohan , Ishita Sur Apan , Muhtasim Ibteda Shochcho , Md Fahim , Mohammad Ashfaq Ur Rahman , AKM Mahbubur Rahman , Amin Ahsan Ali This is my paper
classification cs.CL
keywords BEA2025sharedtaskmistakeidentificationlocationMPNetensemblelearningeducationaldialoguetutorresponseevaluationgroupedcross-validation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports a system for the BEA 2025 shared task on evaluating AI tutor responses. The central claim is that a hard-voting ensemble of ten fine-tuned MPNet classifiers, trained with class-weighted cross-entropy and dialogue-grouped cross-validation, classifies whether a tutor's response identifies a student's mistake (Track 1) and pins down its location (Track 2) almost as well as the best systems, with exact-match macro-F1 of about 0.7110 and 0.5543 on the official test set, ranking 5th of 44 and 7th of 31. The result matters because reliable automated evaluation of tutor feedback is a bottleneck for building AI tutors that can reflect on and improve their own responses. The paper also offers an error taxonomy showing that the hardest distinction is between full and partial mistake recognition.

What carries the argument

The load-bearing object is a 10-model hard-voting ensemble of MPNet classifiers. MPNet is a Transformer pretrained with a combination of masked and permuted language modeling; each model pools the [CLS] token's final-layer hidden state from the concatenated conversation history and cleaned tutor response, passes it through a dropout-plus-linear classification head, and is trained with a class-weighted cross-entropy loss that up-weights the rare 'To some extent' class. Grouped 10-fold cross-validation assigns entire dialogues to either training or validation in each fold, preventing context leakage, and the best checkpoint per fold is selected by validation macro-F1. Final predictions come from majority voting with softmax-confidence tie-breaking, which the paper reports improves macro-F1 by 2–3 points over individual models.

What would settle it

Retrain the same ensemble on the development set without the manual pruning step and with a shorter token limit, and compare Track 2 macro-F1 with the reported 0.5543; if performance does not drop meaningfully, the premise that pruning preserves the mistake's location is not load-bearing.

Watch

Extended reading notes

Core claim

On the shared task's official test set, the team's ensemble achieves exact-match macro-F1 of 0.7110 for Mistake Identification and 0.5543 for Mistake Location, placing it within 0.7 and 4.4 points of the top-ranked systems respectively, while exceeding the top Track-1 system in accuracy (0.877 versus 0.862). The paper argues that a moderate-size ensemble of MPNet classifiers, each fine-tuned on a different dialogue-grouped fold with a class-weighted loss, captures enough signal from about 2.5k labeled tutor responses to be competitive with much larger or more elaborate approaches. It further claims that the main systematic failure is confusion between the 'Yes' and 'To some extent' classes, driven by indirect or hedged tutor phrasing, and that the model's internal representations show these two classes are entangled while the 'No' class is compact.

Load-bearing premise

The most fragile load-bearing premise is that the preprocessed, truncated input still contains the specific step in the student's solution that the tutor's response refers to; if manual pruning of conversation history or the 300/512-token truncation cuts that referent, no classifier can locate the mistake.

Editorial extensions

If this is right

  • A 10-model MPNet ensemble is a viable, low-cost baseline for tutor-response evaluation: full training takes a few hours on one GPU and test-set inference under a minute.
  • Class-weighted loss and grouped cross-validation extract usable signal from a small (~2.5k) labeled corpus without allowing the same dialogue to appear in both training and validation.
  • The 'To some extent' category is the performance bottleneck, so any practical deployment should expect ambiguity between partial and full mistake recognition to be the dominant error source.
  • Because ensemble confidence is not well calibrated, the system's probability scores should not be used directly to trigger pedagogical interventions without post-hoc calibration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the confusion pattern generalizes, a two-stage cascade that first separates 'No' from the rest and then disambiguates 'Yes' from 'To some extent' could outperform the one-stage classifier, since the 'No' class appears compact in the embedding space.
  • The error taxonomy suggests cheap wins: surface features such as hedging phrases, question marks, and explicit references to a solution step could be added as features or as a rule-based pre-filter to lift the system without a larger model.
  • The finding that manual pruning was needed mainly for certain tutor sources hints that a learned or source-aware preprocessing filter could replace hand-written cleanup rules in future iterations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. This paper describes Team BD's submission to the BEA 2025 Shared Task on Pedagogical Ability Assessment of AI-powered Tutors, addressing Track 1 (Mistake Identification) and Track 2 (Mistake Location). The system fine-tunes MPNet with a class-weighted cross-entropy loss, uses grouped 10-fold cross-validation to avoid dialogue overlap, and combines the best model from each fold via hard voting with softmax-confidence tie-breaking. On the official test set, the authors report exact-match macro-F1 scores of approximately 0.7110 on Track 1 (ranked 5th of 44) and 0.5543 on Track 2 (ranked 7th of 31). The paper also presents preprocessing steps, confusion matrices, t-SNE visualizations, an error taxonomy, and a discussion of limitations.

Significance. If the reported test scores are correctly attributed to the described system, the paper provides a useful reference point: a moderate-size MPNet ensemble is competitive with the top systems on both tracks and exceeds the median leaderboard performance. The strengths of the paper include the use of externally produced official test scores, grouped cross-validation to prevent dialogue leakage, and a detailed error analysis that offers insight into the difficulty of distinguishing partial from full mistake recognition. However, the manuscript contains several internal contradictions in the method description that currently prevent verification of what was actually run; these need to be resolved before the central claim can be fully credited.

major comments (6)
  1. [Section 4.1 vs. Section 5.1] The maximum input length is stated inconsistently: Section 4.1 says inputs were constrained to a maximum of 512 MPNet tokens, while Section 5.1 says inputs were truncated to a maximum length of 300 tokens. Because truncation and manual pruning can remove the referent needed for Track 2, the paper must state which limit was used for the submitted runs and whether the same limit applied to both tracks.
  2. [Section 5.1 vs. Section 4.4 and Abstract] The ensemble size is described inconsistently: Section 4.4 defines N=10 models one from each fold for each track and the abstract says '10 MPNet classifiers', but Section 5.1 states 'Full ensemble training (10 models for Track 1 and 7 for Track 2)'. Please clarify the actual number of models per track and, if the Track 2 ensemble used only 7 models, explain how those 7 were selected from the 10 folds.
  3. [Section 5.1, class-weight formula] The formula w_c = N/(K·n_c) does not produce the reported weight vectors. Using the class counts given in the same section, the formula yields approximately [2.23, 4.74, 0.43] for Track 1 (No/Some/Yes) instead of [1.0, 3.0, 0.5], and similarly inconsistent values for Track 2. The Appendix gives a third formula, w_c = 1/log(f_c + epsilon) with epsilon = 1.05. Please provide the exact weight formula and values used in the submitted system, and clarify whether the reported weights were lightly tuned on the development set.
  4. [Table 1 vs. Table 5] The model selection narrative is contradicted by the reported results: Table 1 shows MPNet achieving the highest macro-F1 (0.6975) on Track 1 cross-validation and being selected, while Table 5 reports BERT outperforming MPNet on both tracks (0.8703 vs. 0.8639 for Mistake Identification and 0.7025 vs. 0.6203 for Mistake Location). Please reconcile these tables and specify the evaluation setup for each, since the claim that MPNet was the best-performing backbone is central to the method description.
  5. [Section 4.1, manual pruning and truncation] The preprocessing section states that inputs exceeding the token limit were manually pruned of 'low-value content' such as greetings. For Track 2, the model must connect the tutor's response to a specific step in the student's solution; pruning or truncation can remove exactly that referent. This is a load-bearing limitation for Track 2, and the paper should describe the pruning criteria in enough detail to be replicable and discuss the possible effect on Track 2 performance.
  6. [Appendix vs. Section 5.1] The hardware and hyperparameter details are contradictory: Section 5.1 reports a single NVIDIA Tesla V100 (16 GB), batch size 32, and 3–5 minutes per epoch, while the Appendix reports a single NVIDIA 3090 GPU, batch size 16, and approximately 25 minutes per fold. Please clarify which configuration produced the official test predictions and whether one of these descriptions refers to a different experiment.
minor comments (6)
  1. [Section 6.3] The citation for t-SNE appears as '(?)' and should be replaced with a proper reference.
  2. [Abstract and Section 3] The abstract says '2.5k examples' while Section 3 says 'over 2,480 tutor responses'; please use a single consistent number.
  3. [Section 2] Related Work mentions a 'Track 4' task, but the paper only addresses Tracks 1 and 2; please correct this reference or clarify the scope.
  4. [Section 5.2] The lenient evaluation that merges 'Yes' and 'To some extent' is mentioned as omitted; a one-sentence definition would help readers interpret the exact-match results.
  5. [Table 4] The caption says 'manual cleanup operations' but the row and column labels mix model names with operation categories; consider a clearer layout that separates models from operations.
  6. [Acknowledgments] The GitHub link is described as a future release ('will be released'); if the code is available at publication, please provide a persistent link or DOI.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's headline results are measured on an external, organizer-scored test set, and no load-bearing derivation reduces to its own inputs.

full rationale

The paper's central claim is a reporting of shared-task test performance (exact-match macro-F1 of approximately 0.7110 on Track 1 and 0.5543 on Track 2, Section 6.1), with test labels and scores provided by the BEA 2025 organizers. There is no derivation chain in which a fitted quantity is renamed a prediction: the class-weighted loss weights (Section 5.1) and learning-rate selection are tuned on the development set, and the final test metrics come from the official leaderboard, so the reported result is not constructed from the training inputs. The grouped cross-validation ensemble is a standard model-selection procedure, not a circular argument. No load-bearing self-citation appears: the cited evaluation taxonomy (Tack and Piech 2022b) and shared-task definition (SIGEDU 2025) are external prior work, and none of the authors' own prior results is invoked to force the method. The paper's internal inconsistencies (300 vs 512 token limit, 7 vs 10 ensemble members, V100 vs 3090, conflicting class-weight formulas, and Table 1 vs Table 5 model rankings) are reproducibility and verification concerns that make it hard to confirm exactly what was run, but they do not make any claim equivalent to its inputs by definition. The preprocessing concern that manual pruning might remove the mistake's referent would tend to depress Track 2 scores rather than manufacture them, so it is a potential limitation, not a circular step. Under the criteria of this review, no specific equation or cited result reduces to the target claim, and the appropriate finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central result is empirical, so the ledger is dominated by tuned hyperparameters rather than derived constants. Class weights, learning rate, token limit, and batch size are chosen by hand or grid search and directly affect the reported scores. The main domain assumptions are that the task labels are trustworthy, that the [CLS] vector captures the needed semantics, that input pruning preserves the mistake location, and that the development distribution matches the test distribution.

free parameters (5)
  • Track 1 class weights (w_No, w_Some, w_Yes) = [1.0, 3.0, 0.5]
    Used in the class-weighted cross-entropy loss in Section 5.1. The text gives formula N/(K*n_c), but that formula yields approximately [2.23, 4.74, 0.43] for the stated class counts, so the reported vector is a hand-tuned value, not the formula output.
  • Track 2 class weights (w_No, w_Some, w_Yes) = [0.8, 2.2, 0.9]
    Reported as 'derived from inverse class frequencies and lightly tuned' in Section 5.1. The appendix gives a different formula, so the exact values are chosen by hand.
  • Learning rate = 2e-5
    Selected by grid search over {1e-5, 2e-5, 3e-5} on the validation folds (Section 5.1 and Appendix). The final test result depends on this choice.
  • Maximum token length = 300 in Section 5.1, 512 in Section 4.1
    Inputs are truncated or manually pruned to this limit. The paper reports conflicting values, and the pruning policy can remove the referent needed for Track 2.
  • Batch size = 32 in Section 5.1, 16 in Appendix
    Training configuration with conflicting values; affects optimization and final model. Also early stopping patience 2 and dropout 0.1 are hand-set.
assumptions (4)
  • domain assumption The official BEA 2025 labels (Yes, To some extent, No) are reliable ground truth for the two tracks.
    Every metric and analysis in Sections 5 and 6 treats these labels as correct, while Section 7 acknowledges label ambiguity. This assumption is load-bearing and partially contested by the authors themselves.
  • domain assumption The final-layer [CLS] hidden state of MPNet is a sufficient representation of the combined conversation history and tutor response for classification.
    Section 4.2 defines h_T as the final-layer [CLS] vector and feeds only this vector to the classification head. No pooling or attention over tokens is used.
  • ad hoc to paper Truncating or manually pruning conversation history to the token limit preserves the information needed to identify and locate the student's mistake.
    Section 4.1 describes manual pruning of 'low-value' content such as greetings. Track 2 in particular requires the model to link the tutor's response to a specific step in the student's solution; if pruning removes that step, the task becomes unsolvable.
  • domain assumption The 10-fold grouped cross-validation models, when combined by majority vote, transfer to the held-out test dialogues.
    The ensemble's test performance in Section 6.1 assumes the test distribution is similar enough to the development dialogues that no domain shift invalidates the vote. This is a standard but unverified assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BD at BEA 2025 Shared Task: MPNet Ensembles for Pedagogical Mistake Identification and Localization in AI Tutor Responses." pith.science (2026). https://pith.science/paper/JEWMFLRZ

@misc{pith2026250601817,
  author       = {Pith},
  title        = {Pith review of: BD at BEA 2025 Shared Task: MPNet Ensembles for Pedagogical Mistake Identification and Localization in AI Tutor Responses},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JEWMFLRZ}},
  note         = {Machine review of arXiv:2506.01817}
}
read the original abstract

We present Team BD's submission to the BEA 2025 Shared Task on Pedagogical Ability Assessment of AI-powered Tutors, under Track 1 (Mistake Identification) and Track 2 (Mistake Location). Both tracks involve three-class classification of tutor responses in educational dialogues - determining if a tutor correctly recognizes a student's mistake (Track 1) and whether the tutor pinpoints the mistake's location (Track 2). Our system is built on MPNet, a Transformer-based language model that combines BERT and XLNet's pre-training advantages. We fine-tuned MPNet on the task data using a class-weighted cross-entropy loss to handle class imbalance, and leveraged grouped cross-validation (10 folds) to maximize the use of limited data while avoiding dialogue overlap between training and validation. We then performed a hard-voting ensemble of the best models from each fold, which improves robustness and generalization by combining multiple classifiers. Our approach achieved strong results on both tracks, with exact-match macro-F1 scores of approximately 0.7110 for Mistake Identification and 0.5543 for Mistake Location on the official test set. We include comprehensive analysis of our system's performance, including confusion matrices and t-SNE visualizations to interpret classifier behavior, as well as a taxonomy of common errors with examples. We hope our ensemble-based approach and findings provide useful insights for designing reliable tutor response evaluation systems in educational dialogue settings.

Figures

Figures reproduced from arXiv: 2506.01817 by the authors.

Figure 2
Figure 2. Confusion matrix for Track 2 (Mistake Loca [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 1
Figure 1. Confusion matrix for Track 1 (Mistake Iden [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 3
Figure 3. t-SNE projection of [CLS] embeddings from [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Histogram of prediction confidence values for [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Histogram of prediction confidence values [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 13 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Andersen, and Ted Briscoe

    Christopher Bryant, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. 2019. The bea-2019 shared task on grammatical error correction. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52--75. Association for Computational Linguistics

    work page 2019

  4. [4]

    Yangyi Chen, Lifan Yuan, Ganqu Cui, Zhiyuan Liu, and Heng Ji. 2023. https://aclanthology.org/2023.acl-long.75/ A close look into the calibration of pre-trained language models . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1343--1367, Toronto, Canada. Association for Computationa...

    work page 2023

  5. [5]

    Nico Daheim, Jakub Macina, Tanmay Sinha, and Mrinmaya Sachan. 2024. Stepwise verification and remediation of student reasoning errors with large language model tutors. arXiv preprint arXiv:2407.09136

    arXiv 2024

  6. [6]

    Dietterich

    Thomas G. Dietterich. 2000. Ensemble methods in machine learning. In International Workshop on Multiple Classifier Systems, pages 1--15. Springer

    work page 2000

  7. [7]

    Myroslava Dzikovska, Rodney Nielsen, Chris Brew, Claudia Leacock, Danilo Giampiccolo, Luisa Bentivogli, Peter Clark, Ido Dagan, and Hoa Trang Dang. 2013. Semeval-2013 task 7: The joint student response analysis and 8th recognizing textual entailment challenge. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings ...

    work page 2013

  8. [8]

    Fredrik Gustafsson, Martin Danelljan, and Thomas B. Schön. 2020. Evaluating scalable bayesian deep learning methods for robust computer vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 169--170

    work page 2020

Show all 21 references
  1. [10]

    Jakub Macina, Nico Daheim, Sankalan Pal Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. 2023. https://arxiv.org/abs/2305.14536 Mathdial: A dialogue tutoring dataset with rich pedagogical properties grounded in math reasoning problems . arXiv preprint ...

    2023 arXiv

  2. [11]

    Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The conll-2014 shared task on grammatical error correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pa...

    2014

  3. [12]

    Sculley, Sebastian Nowozin, Joshua Dillon, and Balaji Lakshminarayanan

    Yaniv Ovadia, Elad Fertig, Jae Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, and Balaji Lakshminarayanan. 2019. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. arXiv preprint arXiv:1906.02530

    2019 arXiv

  4. [13]

    Nils Reimers and Iryna Gurevych. 2020. sentence-transformers/all-mpnet-base-v2. https://huggingface.co/sentence-transformers/all-mpnet-base-v2

    2020

  5. [14]

    SIGEDU . 2025. Bea 2025 shared task: Pedagogical ability assessment of ai-powered tutors. https://sig-edu.org/sharedtask/2025

    2025

  6. [15]

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. https://proceedings.neurips.cc/paper/2020/hash/c3a690be93aa602ee2dc0ccab5b7b67e-Abstract.html Mpnet: Masked and permuted pre-training for language understanding . In Advances in Neural Information Processing Sys...

    2020

  7. [16]

    Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. https://aclanthology.org/J00-3003/ Dialogue act modeling for automatic tagging and recognition of conversati...

    2000

  8. [17]

    Anaïs Tack, Ekaterina Kochmar, Zheng Yuan, Serge Bibauw, and Chris Piech. 2023. https://aclanthology.org/2023.bea-1.64/ The bea 2023 shared task on generating ai teacher responses in educational dialogues . In Proceedings of the 18th Workshop on Innovative Use of NLP for Build...

    2023

  9. [18]

    Anaïs Tack and Chris Piech. 2022 a . https://arxiv.org/abs/2205.07540 The ai teacher test: Measuring the pedagogical ability of blender and gpt-3 in educational dialogues . In Proceedings of the 15th International Conference on Educational Data Mining (EDM)

    2022 arXiv

  10. [19]

    Anaïs Tack and Chris Piech. 2022 b . https://arxiv.org/abs/2412.09416 An evaluation taxonomy for pedagogical ability assessment of llm tutors . arXiv preprint arXiv:2412.09416

    2022 arXiv

  11. [20]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and ...

    2020 doi

  12. [21]

    Chen, Yoonho Lee, Eric Mitchell, and Chelsea Finn

    Johnathan Xie, Annie S. Chen, Yoonho Lee, Eric Mitchell, and Chelsea Finn. 2024. https://aclanthology.org/2024.emnlp-main.1007.pdf Calibrating language models with adaptive temperature scaling . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Pro...

    2024

  13. [22]

    Yue Zhang, Wei Wang, and Xiaojun Wan. 2023. https://papers.neurips.cc/paper_files/paper/2023/file/f2a11632520f4b7473d7838f074a7d25-Paper-Conference.pdf Boosting language-driven ordering alignment for ordinal classification . In Proceedings of the 37th Conference on Neural Info...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.