{"id":"b2583e73-7c09-4f65-92fa-2006bbd2ef67","arxiv_id":"2506.01817","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An ensemble of MPNet classifiers trained with grouped cross-validation and class-weighted loss reaches competitive macro-F1 on mistake identification and location in AI tutor responses.","lead":"Team BD fine-tuned ensembles of MPNet language models to judge whether AI tutors identify and locate student mistakes, scoring 0.711 and 0.554 macro-F1 on the BEA 2025 shared task test sets. The paper is useful as a competitive shared-task recipe and error analysis for automated evaluation of tutoring dialogue, though the method itself is a standard Transformer fine-tuning pipeline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Internal method inconsistencies (token limit 300 vs 512, ensemble size 10 vs 7, conflicting class-weight formulas and model rankings) prevent verification that the reported test scores were produced by the described MPNet pipeline.","rationale":"The reader's verdict of CONDITIONAL is appropriate. The reported test scores are externally produced by the shared task organizers, which is genuine evidence that a submission with these rankings existed. However, the paper's method description contains multiple mutually contradictory statements about core components: token truncation length, ensemble size per track, class-weight computation, GPU hardware, and even the relative performance of MPNet versus BERT. These contradictions do not prove the scores are wrong, but they do mean the central claim—that this particular described system achieved those scores—cannot be verified from the manuscript alone. The reader identified the preprocessing and truncation risk as the weakest load-bearing premise; I agree that this is a genuine risk, but I see it as secondary because it would only lower performance if it removed the referent. The more fundamental issue is the general unreliability of the method description, which makes the connection between the system and the reported leaderboard numbers unverifiable. A concrete check—obtaining the actual predictions and configuration, or running the released code—would settle whether the concern lands. I therefore keep the reader's CONDITIONAL verdict unchanged, with partial agreement on the specific weakest assumption.","tokens_in":12043,"tokens_out":5295,"duration_ms":55519,"concrete_test":"Request from the authors or the shared task organizers the exact submitted predictions for the official test set and the exact hyperparameter configuration (token limit, ensemble size, class weights, hardware). Independently recompute exact-match macro-F1 against the official labels. If the predictions or recomputed scores do not match 0.7110 / 0.5543, or if the claimed configuration does not match any consistent description in the paper, the central claim is not established. A lighter check: run the released repository (github.com/ShadmanRohan/team-bd-bea25) end-to-end with the paper's stated settings; if it cannot be run or yields different predictions, the claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a 10-model MPNet hard-voting ensemble with class-weighted loss and grouped cross-validation achieved exact-match macro-F1 of 0.7110 (Track 1) and 0.5543 (Track 2) on the official test set. Because these scores came from the organizers, the claim's validity hinges on the submitted system matching the paper's description. That link is weakened by multiple internal contradictions: Section 4.1 sets a 512-token maximum while Section 5.1 says 300 tokens; Section 5.1 says 10 models for Track 1 and 7 for Track 2, conflicting with Section 4.4's N=10 one-per-fold and the abstract's '10 MPNet classifiers'; Section 5.1's class-weight formula w_c = N/(K*n_c) does not produce the listed weights [1.0, 3.0, 0.5] or [0.8, 2.2, 0.9], and the Appendix gives a third formula w_c = 1/log(f_c + epsilon); hardware is V100 in Section 5.1 but 3090 in the Appendix; Table 1 selects MPNet over BERT-large, but Table 5 reports BERT > MPNet on both tracks. If the actual configuration (token truncation, ensemble size, weights, hardware) differs from what is claimed, the reported scores cannot be reliably attributed to the described method, and the paper's inference that a moderate MPNet ensemble is competitive becomes unverifiable. The reader's preprocessing concern is real but secondary: even if manual pruning removes the mistake's referent, that would depress Track 2 scores rather than inflate them. The load-bearing issue is that we cannot confirm what was actually run.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper describes Team BD's submission to the BEA 2025 Shared Task on Pedagogical Ability Assessment of AI-powered Tutors, addressing Track 1 (Mistake Identification) and Track 2 (Mistake Location). The system fine-tunes MPNet with a class-weighted cross-entropy loss, uses grouped 10-fold cross-validation to avoid dialogue overlap, and combines the best model from each fold via hard voting with softmax-confidence tie-breaking. On the official test set, the authors report exact-match macro-F1 scores of approximately 0.7110 on Track 1 (ranked 5th of 44) and 0.5543 on Track 2 (ranked 7th of 31). The paper also presents preprocessing steps, confusion matrices, t-SNE visualizations, an error taxonomy, and a discussion of limitations.","tokens_in":12429,"tokens_out":3757,"duration_ms":36189,"significance":"If the reported test scores are correctly attributed to the described system, the paper provides a useful reference point: a moderate-size MPNet ensemble is competitive with the top systems on both tracks and exceeds the median leaderboard performance. The strengths of the paper include the use of externally produced official test scores, grouped cross-validation to prevent dialogue leakage, and a detailed error analysis that offers insight into the difficulty of distinguishing partial from full mistake recognition. However, the manuscript contains several internal contradictions in the method description that currently prevent verification of what was actually run; these need to be resolved before the central claim can be fully credited.","major_comments":[{"comment":"The maximum input length is stated inconsistently: Section 4.1 says inputs were constrained to a maximum of 512 MPNet tokens, while Section 5.1 says inputs were truncated to a maximum length of 300 tokens. Because truncation and manual pruning can remove the referent needed for Track 2, the paper must state which limit was used for the submitted runs and whether the same limit applied to both tracks.","section":"Section 4.1 vs. Section 5.1"},{"comment":"The ensemble size is described inconsistently: Section 4.4 defines N=10 models one from each fold for each track and the abstract says '10 MPNet classifiers', but Section 5.1 states 'Full ensemble training (10 models for Track 1 and 7 for Track 2)'. Please clarify the actual number of models per track and, if the Track 2 ensemble used only 7 models, explain how those 7 were selected from the 10 folds.","section":"Section 5.1 vs. Section 4.4 and Abstract"},{"comment":"The formula w_c = N/(K·n_c) does not produce the reported weight vectors. Using the class counts given in the same section, the formula yields approximately [2.23, 4.74, 0.43] for Track 1 (No/Some/Yes) instead of [1.0, 3.0, 0.5], and similarly inconsistent values for Track 2. The Appendix gives a third formula, w_c = 1/log(f_c + epsilon) with epsilon = 1.05. Please provide the exact weight formula and values used in the submitted system, and clarify whether the reported weights were lightly tuned on the development set.","section":"Section 5.1, class-weight formula"},{"comment":"The model selection narrative is contradicted by the reported results: Table 1 shows MPNet achieving the highest macro-F1 (0.6975) on Track 1 cross-validation and being selected, while Table 5 reports BERT outperforming MPNet on both tracks (0.8703 vs. 0.8639 for Mistake Identification and 0.7025 vs. 0.6203 for Mistake Location). Please reconcile these tables and specify the evaluation setup for each, since the claim that MPNet was the best-performing backbone is central to the method description.","section":"Table 1 vs. Table 5"},{"comment":"The preprocessing section states that inputs exceeding the token limit were manually pruned of 'low-value content' such as greetings. For Track 2, the model must connect the tutor's response to a specific step in the student's solution; pruning or truncation can remove exactly that referent. This is a load-bearing limitation for Track 2, and the paper should describe the pruning criteria in enough detail to be replicable and discuss the possible effect on Track 2 performance.","section":"Section 4.1, manual pruning and truncation"},{"comment":"The hardware and hyperparameter details are contradictory: Section 5.1 reports a single NVIDIA Tesla V100 (16 GB), batch size 32, and 3–5 minutes per epoch, while the Appendix reports a single NVIDIA 3090 GPU, batch size 16, and approximately 25 minutes per fold. Please clarify which configuration produced the official test predictions and whether one of these descriptions refers to a different experiment.","section":"Appendix vs. Section 5.1"}],"minor_comments":[{"comment":"The citation for t-SNE appears as '(?)' and should be replaced with a proper reference.","section":"Section 6.3"},{"comment":"The abstract says '2.5k examples' while Section 3 says 'over 2,480 tutor responses'; please use a single consistent number.","section":"Abstract and Section 3"},{"comment":"Related Work mentions a 'Track 4' task, but the paper only addresses Tracks 1 and 2; please correct this reference or clarify the scope.","section":"Section 2"},{"comment":"The lenient evaluation that merges 'Yes' and 'To some extent' is mentioned as omitted; a one-sentence definition would help readers interpret the exact-match results.","section":"Section 5.2"},{"comment":"The caption says 'manual cleanup operations' but the row and column labels mix model names with operation categories; consider a clearer layout that separates models from operations.","section":"Table 4"},{"comment":"The GitHub link is described as a future release ('will be released'); if the code is available at publication, please provide a persistent link or DOI.","section":"Acknowledgments"}],"recommendation":"major_revision","confidential_remarks":"This is a standard shared-task system description, and the official test scores are externally produced, which is a strength. The main risk is that the internal contradictions in the method description make it impossible to verify which configuration actually produced the reported test scores. These issues are correctable with a careful revision and author responses, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a credible shared-task system report with real, organizer-scored leaderboard numbers, plus a genuinely useful error taxonomy. But the method section is self-contradictory in enough places (token limits, ensemble size, class-weight formulas, hardware, Table 5) that the paper cannot currently serve as a reproducible description of what was run.\n\nWhat's good: The main result — 0.711 macro-F1 on Track 1, 0.554 on Track 2, ranks 5/44 and 7/31 — comes from the official test set, so it is externally grounded and not a fitting exercise. The pipeline (MPNet fine-tuning, grouped CV, hard voting) is standard but sensible, and the authors are upfront about limitations: calibration, label ambiguity, and model scope all get an honest section. The error taxonomy in Table 3 is the most original part; it does real analytical work and would be useful to anyone building tutor-response classifiers. The t-SNE and confusion-matrix analysis are fine, if not deep.\n\nThe soft spots are mostly in the description, not the results. Section 4.1 says 512 tokens max; Section 5.1 says 300. Section 4.4 and the abstract say 10 models; Section 5.1 says 10 for Track 1 but 7 for Track 2. The class-weight formula w_c = N/(K*n_c) does not produce the listed vectors, and the appendix gives a different formula entirely. Hardware is V100 in one place, 3090 in another. Table 1 picks MPNet based on CV, but Table 5 reports BERT beating MPNet on both tracks. The code repo is not populated. These contradictions make it impossible for a reader to know what was actually submitted, and the stress-test note is right that this weakens the link between the described method and the scores. That matters for a system-description paper whose entire job is to say what was run.\n\nThe reader's concern about manual pruning cutting the referent for Track 2 is plausible but secondary. It would depress scores, not inflate them, and the scores are still respectable.\n\nBottom line: the core empirical claim is externally validated and the analysis is honest. The paper deserves peer review, not desk rejection, but it needs a serious cleanup pass: reconcile the method numbers, release code and predictions, and clarify which numbers are macro-F1. I'd send it to a workshop or shared-task track with the expectation of minor-to-major revision.","headline":"A credible shared-task system report with real organizer-scored results and a useful error taxonomy, but the method section has too many internal contradictions to be reproducible without cleanup.","tokens_in":12964,"tokens_out":1937,"would_cite":false,"duration_ms":19025,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A hard-voting ensemble of ten fine-tuned MPNet classifiers identifies and localizes mistakes in AI tutor responses nearly as well as the top-ranked systems in the BEA 2025 shared task, reaching macro-F1 of 0.711 and 0.554.","keywords":["BEA 2025 shared task","mistake identification","mistake location","MPNet","ensemble learning","educational dialogue","tutor response evaluation","grouped cross-validation"],"falsifier":"Retrain the same ensemble on the development set without the manual pruning step and with a shorter token limit, and compare Track 2 macro-F1 with the reported 0.5543; if performance does not drop meaningfully, the premise that pruning preserves the mistake's location is not load-bearing.","tokens_in":11856,"feed_emoji":"🎓","tokens_out":5931,"duration_ms":51513,"temperature":0.7,"pith_summary":"This paper reports a system for the BEA 2025 shared task on evaluating AI tutor responses. The central claim is that a hard-voting ensemble of ten fine-tuned MPNet classifiers, trained with class-weighted cross-entropy and dialogue-grouped cross-validation, classifies whether a tutor's response identifies a student's mistake (Track 1) and pins down its location (Track 2) almost as well as the best systems, with exact-match macro-F1 of about 0.7110 and 0.5543 on the official test set, ranking 5th of 44 and 7th of 31. The result matters because reliable automated evaluation of tutor feedback is a bottleneck for building AI tutors that can reflect on and improve their own responses. The paper also offers an error taxonomy showing that the hardest distinction is between full and partial mistake recognition.","feed_headline":"Ten-model MPNet ensemble ranks near top on AI tutor errors","feed_subtitle":"Macro-F1 of 0.711 and 0.554 lands 5th and 7th in the BEA 2025 shared task.","key_machinery":"The load-bearing object is a 10-model hard-voting ensemble of MPNet classifiers. MPNet is a Transformer pretrained with a combination of masked and permuted language modeling; each model pools the [CLS] token's final-layer hidden state from the concatenated conversation history and cleaned tutor response, passes it through a dropout-plus-linear classification head, and is trained with a class-weighted cross-entropy loss that up-weights the rare 'To some extent' class. Grouped 10-fold cross-validation assigns entire dialogues to either training or validation in each fold, preventing context leakage, and the best checkpoint per fold is selected by validation macro-F1. Final predictions come from majority voting with softmax-confidence tie-breaking, which the paper reports improves macro-F1 by 2–3 points over individual models.","core_discovery":"On the shared task's official test set, the team's ensemble achieves exact-match macro-F1 of 0.7110 for Mistake Identification and 0.5543 for Mistake Location, placing it within 0.7 and 4.4 points of the top-ranked systems respectively, while exceeding the top Track-1 system in accuracy (0.877 versus 0.862). The paper argues that a moderate-size ensemble of MPNet classifiers, each fine-tuned on a different dialogue-grouped fold with a class-weighted loss, captures enough signal from about 2.5k labeled tutor responses to be competitive with much larger or more elaborate approaches. It further claims that the main systematic failure is confusion between the 'Yes' and 'To some extent' classes, driven by indirect or hedged tutor phrasing, and that the model's internal representations show these two classes are entangled while the 'No' class is compact.","pith_inferences":["If the confusion pattern generalizes, a two-stage cascade that first separates 'No' from the rest and then disambiguates 'Yes' from 'To some extent' could outperform the one-stage classifier, since the 'No' class appears compact in the embedding space.","The error taxonomy suggests cheap wins: surface features such as hedging phrases, question marks, and explicit references to a solution step could be added as features or as a rule-based pre-filter to lift the system without a larger model.","The finding that manual pruning was needed mainly for certain tutor sources hints that a learned or source-aware preprocessing filter could replace hand-written cleanup rules in future iterations."],"forward_implications":["A 10-model MPNet ensemble is a viable, low-cost baseline for tutor-response evaluation: full training takes a few hours on one GPU and test-set inference under a minute.","Class-weighted loss and grouped cross-validation extract usable signal from a small (~2.5k) labeled corpus without allowing the same dialogue to appear in both training and validation.","The 'To some extent' category is the performance bottleneck, so any practical deployment should expect ambiguity between partial and full mistake recognition to be the dominant error source.","Because ensemble confidence is not well calibrated, the system's probability scores should not be used directly to trigger pedagogical interventions without post-hoc calibration."],"supporting_citations":[{"why":"Supplies MPNet, the pretrained Transformer backbone that the ensemble fine-tunes.","marker":"(Song et al., 2020)"},{"why":"Defines the shared task, the three-class taxonomy, the data, and the exact-match macro-F1 evaluation.","marker":"(SIGEDU, 2025)"},{"why":"Provides the specific pre-trained MPNet checkpoint the system fine-tunes from.","marker":"(Reimers and Gurevych, 2020)"},{"why":"Contributes the MathDial dialogues that form part of the training data.","marker":"(Macina et al., 2023)"},{"why":"Contributes the Bridge dialogues that form the other part of the training data.","marker":"(Kumar et al., 2024)"},{"why":"Motivates the pedagogical dimensions of the task.","marker":"(Tack and Piech, 2022a)"},{"why":"Justifies ensembling as a variance-reduction mechanism.","marker":"(Dietterich, 2000)"}],"fun_headline_variants":["MPNet ensemble hits top-5 in BEA tutor error challenge","10-model MPNet ensemble lands 5th and 7th in tutor tasks","Weighted-loss MPNet voting boosts tutor mistake ID to 0.71 F1","Ensemble of MPNets spots tutor errors, F1 0.711 and 0.554"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The most fragile load-bearing premise is that the preprocessed, truncated input still contains the specific step in the student's solution that the tutor's response refers to; if manual pruning of conversation history or the 300/512-token truncation cuts that referent, no classifier can locate the mistake.","fun_headline_variants_meta":{"raw":{"variants":["MPNet ensemble hits top-5 in BEA tutor error challenge","10-model MPNet ensemble lands 5th and 7th in tutor tasks","Weighted-loss MPNet voting boosts tutor mistake ID to 0.71 F1","Ensemble of MPNets spots tutor errors, F1 0.711 and 0.554"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000459,"raw_usage":{"total_tokens":2332,"prompt_tokens":1011,"completion_tokens":1321,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":1231}},"tokens_in":627,"tokens_out":1321,"duration_ms":11748,"temperature":1.0,"reasoning_tokens":1231,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:32:32.389280+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the same ensemble on the development set without the manual pruning step and with a shorter token limit, and compare Track 2 macro-F1 with the reported 0.5543; if performance does not drop meaningfully, the premise that pruning preserves the mistake's location is not load-bearing.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies MPNet, the pretrained Transformer backbone that the ensemble fine-tunes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the shared task, the three-class taxonomy, the data, and the exact-match macro-F1 evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the specific pre-trained MPNet checkpoint the system fine-tunes from."}],"review_version":1}