Pith. sign in

REVIEW 2 major objections 62 references

A teaching-feedback classification protocol stays durable across model generations and languages, so model choice is a deployment decision, not a property of the method.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 02:30 UTC pith:HKGLQ7OC

load-bearing objection Solid, carefully scoped durability benchmark: the protocol survives model generations and English transfer; model choice is framed as cost/auditability, not method property. the 2 major comments →

arxiv 2607.11873 v1 pith:HKGLQ7OC submitted 2026-07-13 cs.CL cs.LG

A Durability and Cross-Language Transfer Benchmark for a Validated Teaching-Feedback Classification Protocol

Esteban U. Vega Barajas This is my paper
classification cs.CL cs.LG
keywords teaching-evaluation feedbacktext classificationsentiment analysiscross-lingual transferlarge language modelseducational NLPreproducible benchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Institutions archive far more open-ended teaching-evaluation comments than they can read. A prior validated protocol classifies those comments by theme and sentiment using a written annotation guide, reliability checks, stratified validation, and a held-out test. This paper asks whether that protocol remains competitive when representations advance from sparse lexical features through frozen transformers to prompted large language models, and whether the sentiment task transfers to English. Holding the protocol fixed and varying only the model and language, the authors re-run the Spanish corpus across three representation generations and assemble a balanced 45,000-comment English sentiment sample. A 2026 frontier model leads on the hardest Spanish thematic task, yet shows no sentiment advantage over a cheaper model and no descriptive separation from it on English. The procedure itself carries; which model sits underneath it is a cost-and-auditability choice an institution can make.

Core claim

The validated teaching-feedback classification protocol is durable: when the identical annotation guide, splits, and held-out evaluation are held fixed, newer representations improve thematic accuracy on Spanish, but a 2026 frontier model buys no sentiment gain over a cheap model on either language, so model selection is a deployment decision rather than a property of the method.

What carries the argument

The carried-over protocol: a documented annotation guide (four thematic categories, three sentiment classes), test-retest intra-annotator reliability, stratified five-fold cross-validation for selection, and a reserved held-out set, applied unchanged to every arm and both languages.

Load-bearing premise

That single-annotator Spanish gold labels and noisy star-derived English labels are stable enough that F1 orderings under one fixed split can be read as properties of the protocol rather than of the labels themselves.

What would settle it

Re-annotate the Spanish held-out set with a second independent annotator and re-score every arm; if the cross-arm ordering or the frontier-versus-cheap sentiment parity collapses under inter-annotator gold, the durability claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The paper re-tests a previously validated teaching-feedback classification protocol (annotation guide, intra-annotator reliability, stratified CV, held-out evaluation) under two stresses: representation modernization and cross-language transfer. On the original Spanish institutional subset it compares sparse TF-IDF, frozen BETO-2019 and multilingual-e5 embeddings, and prompted LLMs (Haiku 4.5, Opus 4.8, Qwen2.5-1.5B) under a single fixed split; it then transfers the three-class sentiment task to a newly constructed, balanced 45k English RateMyProfessor sample whose star-derived labels are checked against EduRABSA human labels (κ=0.66). Treating all paired comparisons as descriptive, the authors conclude that the protocol is durable: LLMs raise thematic F1 on the hard Spanish task, yet a 2026 frontier model shows no sentiment advantage over a cheap model on either language, so model choice is a deployment (cost/auditability) decision rather than a property of the method. A reproducible English build recipe and cost/auditability co-reporting accompany the results.

Significance. If the descriptive durability claim holds, institutions that already invested in a documented annotation-and-validation pipeline can adopt newer encoders or LLMs without re-deriving the protocol, and can treat model selection as an explicit cost-auditability trade-off. The work is valuable as a carefully scoped educational-NLP benchmark rather than as a new SOTA model: it ships a fixed-seed English sample recipe, reports McNemar counts and bootstrap CIs without significance inflation, quantifies English label noise against an independent gold set, and co-reports inference cost and LIME auditability. These practices raise the bar for reusable SET-analysis methodology and make the contribution falsifiable and reusable by others.

major comments (2)
  1. Limitations and §3.1–3.2: all Spanish gold labels come from a single annotator (intra-annotator κ only, n=100 re-label sample). While the paper correctly scopes claims as descriptive and lists this as a limitation, the durability ordering in Table 1 and Figure 1 rests on that single-annotator gold under one 233-item split. A second annotator on at least the held-out set (or a reported inter-annotator κ) would materially strengthen the claim that the observed cross-arm orderings are properties of the protocol rather than of one annotator’s idiosyncrasies.
  2. §5 and Limitations: the Spanish durability ordering is read from a single stratified split (seed 20260616, n=233 held-out). Bootstrap CIs are reported, but robustness across resampled or repeated splits is left as future work. Because the central claim is that the protocol holds across representation generations, at least a leave-one-cycle or multi-seed sensitivity check on the Spanish subset would make the ordering less dependent on one draw.

Circularity Check

0 steps flagged

No significant circularity: durability claim is an empirical re-test of a fixed protocol under new models and language, not a result forced by definition or self-citation chain.

full rationale

The paper's central claim is that a previously documented protocol (annotation guide, intra-annotator reliability, stratified CV, held-out evaluation) remains usable when representation methods and language change. That claim is supported by new held-out F1 numbers, McNemar counts treated as descriptive only, cost/latency/auditability trade-offs, and an independent EduRABSA agreement check (kappa 0.66) on the star-to-sentiment rule. The prior thesis is cited only to identify the protocol being held fixed and to recover the original Spanish labeled subset; the durability ordering and English transfer results are produced by re-running that protocol on new arms and a newly constructed English sample, not by re-deriving or renaming the thesis numbers. There are no equations, fitted parameters re-labeled as predictions, uniqueness theorems, or ansatzes imported from the author's prior work that force the reported F1 orderings or the 'model choice is a deployment decision' conclusion. Self-citation of the protocol definition is ordinary and non-load-bearing for the new empirical content. The paper is therefore self-contained against its external benchmarks and exhibits no circular reduction.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

Empirical classification benchmark; almost no free parameters enter the central durability claim. The load-bearing premises are domain conventions (annotation guide, star-to-sentiment map) and experimental design choices (single seed/split, frozen vs. prompted arms) rather than fitted constants or invented physical entities.

free parameters (3)
  • stratified split seed = 20260616
    Fixed seed 20260616 used for the single Spanish held-out split and English sampling; all durability orderings rest on this one draw.
  • LinearSVC C and dual settings = C=0.1, dual=False
    Carried unchanged from prior study (C=0.1, dual=False for thematic); hyper-parameters are not re-tuned per arm, so they are free design choices that affect absolute F1.
  • star-to-sentiment thresholds = ≥4 / ≤2
    Positive if stars ≥4, negative if ≤2, neutral otherwise; chosen by hand to match the three-class Spanish scheme and validated post-hoc rather than optimized.
axioms (4)
  • domain assumption The written four-theme / three-sentiment annotation guide defines the only admissible labels and is held fixed across all arms and languages.
    §3.1; without a stable guide the protocol comparison is undefined.
  • domain assumption Intra-annotator test-retest κ (0.82 thematic, 0.88 sentiment) is an adequate reliability proxy for the Spanish gold set.
    §3.1 and Limitations; no inter-annotator measurement exists.
  • domain assumption Star-derived English labels are a usable (if noisy) proxy once agreement with EduRABSA human labels is reported.
    §3.4; κ=0.66 is accepted as 'reasonable but imperfect'.
  • standard math Paired McNemar counts and bootstrap F1 intervals may be reported purely descriptively without significance claims.
    §4; follows Dietterich / McNemar practice for classifier comparison under small n.

pith-pipeline@v1.1.0-grok45 · 19014 in / 2996 out tokens · 29946 ms · 2026-07-14T02:30:22.222715+00:00 · methodology

0 comments
read the original abstract

Institutions collect far more open-ended teaching-evaluation feedback than they read. A prior study introduced a validated protocol for classifying such comments by thematic category and sentiment, built from a documented annotation guide, an intra-annotator reliability measurement, stratified cross-validation, and a held-out evaluation on a Spanish institutional corpus with a frozen-encoder design. Two questions limit its reuse: whether a protocol fixed to 2019-era frozen embeddings stays competitive as representation methods advance, and whether it transfers to a second language. We re-run it on the original Spanish data across three representation generations, sparse lexical features, frozen transformer embeddings, and prompted large language models, and transfer its sentiment task to English with a balanced 45,000-comment corpus checked against an aspect-labeled education dataset. Treating paired comparisons as descriptive, we find the protocol durable: a 2026 frontier model posts the highest thematic F1 on the hardest Spanish task, yet shows no sentiment advantage over a cheap model and no descriptive separation from it on English, so model choice is a deployment decision, not a property of the method.

Figures

Figures reproduced from arXiv: 2607.11873 by Esteban U. Vega Barajas.

Figure 1
Figure 1. Figure 1: Spanish durability by representation gener￾ation. Weighted F1 on the 233-comment held-out set (single stratified split, seed 20260616), arms ordered by generation: sparse lexical, frozen embeddings, and large language models, separated by dashed rules. Thematic = navy solid circles, sentiment = red dashed squares; caps are descriptive 95% bootstrap confidence intervals (no significance is claimed). Themati… view at source ↗
Figure 2
Figure 2. Figure 2: Cost versus accuracy on Spanish sentiment. Each arm’s weighted F1 against API cost on a log scale, where the cost coordinate for each LLM marker is that model’s total API spend across the whole benchmark and local arms run at effectively zero API cost. The dashed line is the cost-accuracy frontier. Opus 4.8 sits below Haiku 4.5 despite about 7.7x the total API spend, while the frozen e5 encoder and the ope… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 6 linked inside Pith

  1. [1]

    , title =

    Vega Barajas, Esteban U. , title =

  2. [2]

    2025 , howpublished =

    2025

  3. [3]

    2026 , howpublished =

    2026

  4. [4]

    Transactions of the Association for Computational Linguistics , volume =

    Artetxe, Mikel and Schwenk, Holger , title =. Transactions of the Association for Computational Linguistics , volume =

  5. [5]

    Proceedings of LREC , pages =

    Barbieri, Francesco and Espinosa Anke, Luis and Camacho-Collados, Jose , title =. Proceedings of LREC , pages =

  6. [6]

    and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared and Dhariwal, Prafulla and others , title =

    Brown, Tom B. and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared and Dhariwal, Prafulla and others , title =. Advances in Neural Information Processing Systems 33 (NeurIPS) , pages =

  7. [7]

    PML4DC at ICLR , year =

    Ca\ nete, Jos\'e and Chaperon, Gabriel and Fuentes, Rodrigo and Ho, Jou-Hui and Kang, Hojin and P\'erez, Jorge , title =. PML4DC at ICLR , year =

  8. [8]

    and Schwenk, Holger and Stoyanov, Veselin , title =

    Conneau, Alexis and Rinott, Ruty and Lample, Guillaume and Williams, Adina and Bowman, Samuel R. and Schwenk, Holger and Stoyanov, Veselin , title =. Proceedings of EMNLP , pages =

  9. [9]

    Proceedings of ACL , pages =

    Conneau, Alexis and Khandelwal, Kartikay and Goyal, Naman and Chaudhary, Vishrav and Wenzek, Guillaume and Guzm\'an, Francisco and others , title =. Proceedings of ACL , pages =

  10. [10]

    Machine Learning , volume =

    Cortes, Corinna and Vapnik, Vladimir , title =. Machine Learning , volume =

  11. [11]

    IEEE Transactions on Education , volume =

    Cunningham-Nelson, Samuel and Baktashmotlagh, Mahsa and Boles, Wageeh , title =. IEEE Transactions on Education , volume =

  12. [12]

    Proceedings of NAACL-HLT , pages =

    Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina , title =. Proceedings of NAACL-HLT , pages =

  13. [13]

    , title =

    Dietterich, Thomas G. , title =. Neural Computation , volume =

  14. [14]

    Proceedings of the National Academy of Sciences , volume =

    Gilardi, Fabrizio and Alizadeh, Meysam and Kubli, Ma\"el , title =. Proceedings of the National Academy of Sciences , volume =

  15. [15]

    Journal of Computing in Higher Education , volume =

    Grimalt-\'Alvaro, Carme and Usart, Mireia , title =. Journal of Computing in Higher Education , volume =

  16. [16]

    Journal of Survey Statistics and Methodology , volume =

    Gweon, Hyukjun and Schonlau, Matthias , title =. Journal of Survey Statistics and Methodology , volume =

  17. [17]

    2020 , howpublished =

    He, Jibo , title =. 2020 , howpublished =

    2020

  18. [18]

    Assessment & Evaluation in Higher Education , volume =

    He, Jibo and Zheng, Xin and Liu, Min and Du, Yu and Liu, Gang and Cui, Jie and Su, Yang , title =. Assessment & Evaluation in Higher Education , volume =

  19. [19]

    Proceedings of LREC , pages =

    Herath, Missaka and Chamindu, Kushan and Maduwantha, Hashan and Ranathunga, Surangika , title =. Proceedings of LREC , pages =

  20. [20]

    and Bansal, Mohit , title =

    Hu, Yinuo and Zhang, Shiyue and Sathy, Viji and Panter, Abigail T. and Bansal, Mohit , title =. Proceedings of NAACL (System Demonstrations) , year =

  21. [22]

    Psychometrika , volume =

    McNemar, Quinn , title =. Psychometrika , volume =

  22. [27]

    Proceedings of EMNLP , year =

    Reimers, Nils and Gurevych, Iryna , title =. Proceedings of EMNLP , year =

  23. [28]

    Proceedings of KDD , year =

    Ribeiro, Marco Tulio and Singh, Sameer and Guestrin, Carlos , title =. Proceedings of KDD , year =

  24. [29]

    IEEE Access , volume =

    Shaik, Thanveer and Tao, Xiaohui and Li, Yan and Dann, Christopher and McDonald, Jacquie and Redmond, Petrea and Galligan, Linda , title =. IEEE Access , volume =

  25. [30]

    Journal of Documentation , volume =

    Sparck Jones, Karen , title =. Journal of Documentation , volume =

  26. [32]

    and Wirawarn, Pawan and Goodman, Noah and Demszky, Dorottya , title =

    Wang, Rose E. and Wirawarn, Pawan and Goodman, Noah and Demszky, Dorottya , title =. Proceedings of BEA at ACL , year =

  27. [33]

    Findings of NAACL , pages =

    Zhang, Wenxuan and Deng, Yue and Liu, Bing and Pan, Sinno Jialin and Bing, Lidong , title =. Findings of NAACL , pages =

  28. [34]

    Computational Linguistics , volume =

    Ziems, Caleb and Held, William and Shaikh, Omar and Chen, Jiaao and Zhang, Zhehao and Yang, Diyi , title =. Computational Linguistics , volume =

  29. [35]

    Anthropic . 2025. Claude Haiku 4.5 System Card . Anthropic. https://www.anthropic.com/claude-haiku-4-5-system-card

    2025

  30. [36]

    Anthropic . 2026. Claude Opus 4.8 System Card . Anthropic. https://www.anthropic.com/claude-opus-4-8-system-card

    2026

  31. [37]

    Mikel Artetxe and Holger Schwenk. 2019. Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond . Transactions of the Association for Computational Linguistics, 7:597--610

    2019

  32. [38]

    Francesco Barbieri, Luis Espinosa Anke, and Jose Camacho-Collados. 2022. XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond . In Proceedings of LREC, pages 258--266

    2022

  33. [39]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, and 1 others

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, and 1 others. 2020. Language Models are Few-Shot Learners . In Advances in Neural Information Processing Systems 33 (NeurIPS), pages 1877--1901

    2020

  34. [40]

    Jos\'e Ca\ nete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, and Jorge P\'erez. 2020. Spanish Pre-trained BERT Model and Evaluation Data . In PML4DC at ICLR

    2020

  35. [41]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm\'an, and 1 others. 2020. Unsupervised Cross-lingual Representation Learning at Scale . In Proceedings of ACL, pages 8440--8451

    2020

  36. [42]

    Bowman, Holger Schwenk, and Veselin Stoyanov

    Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating Cross-lingual Sentence Representations . In Proceedings of EMNLP, pages 2475--2485

    2018

  37. [43]

    Corinna Cortes and Vladimir Vapnik. 1995. Support-Vector Networks . Machine Learning, 20(3):273--297

    1995

  38. [44]

    Samuel Cunningham-Nelson, Mahsa Baktashmotlagh, and Wageeh Boles. 2019. Visualizing Student Opinion Through Text Analysis . IEEE Transactions on Education, 62(4):305--311

    2019

  39. [45]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding . In Proceedings of NAACL-HLT, pages 4171--4186

    2019

  40. [46]

    Dietterich

    Thomas G. Dietterich. 1998. Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms . Neural Computation, 10(7):1895--1924

    1998

  41. [47]

    Fabrizio Gilardi, Meysam Alizadeh, and Ma\"el Kubli. 2023. ChatGPT outperforms crowd workers for text-annotation tasks . Proceedings of the National Academy of Sciences, 120(30):e2305016120

    2023

  42. [48]

    Carme Grimalt-\'Alvaro and Mireia Usart. 2024. Sentiment analysis for formative assessment in higher education: a systematic literature review . Journal of Computing in Higher Education, 36:647--682

    2024

  43. [49]

    Hyukjun Gweon and Matthias Schonlau. 2024. Automated classification for open-ended questions with BERT . Journal of Survey Statistics and Methodology, 12(2):493--504

    2024

  44. [50]

    Jibo He. 2020. Big Data Set from RateMyProfessor.com for Professors' Teaching Evaluation . Mendeley Data, V2. CC BY 4.0 (sample); the full release is CC BY-NC-SA

    2020

  45. [51]

    Jibo He, Xin Zheng, Min Liu, Yu Du, Gang Liu, Jie Cui, and Yang Su. 2022. Reciprocity in College Teaching: A Big Data Study Based on Online Student Evaluation of 919,750 Professors . Assessment & Evaluation in Higher Education, 47(8):1401--1415

    2022

  46. [52]

    Missaka Herath, Kushan Chamindu, Hashan Maduwantha, and Surangika Ranathunga. 2022. Dataset and Baseline for Automatic Student Feedback Analysis . In Proceedings of LREC, pages 2042--2049

    2022

  47. [53]

    Panter, and Mohit Bansal

    Yinuo Hu, Shiyue Zhang, Viji Sathy, Abigail T. Panter, and Mohit Bansal. 2022. SETSum: Summarization and Visualization of Student Evaluations of Teaching . In Proceedings of NAACL (System Demonstrations)

    2022

  48. [54]

    Yan Cathy Hua, Paul Denny, J\"org Wicker, and Katerina Taskova. 2025. EduRABSA: An Education Review Dataset for Aspect-based Sentiment Analysis Tasks . arXiv preprint arXiv:2508.17008

    Pith/arXiv arXiv 2025

  49. [55]

    Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages . Psychometrika, 12(2):153--157

    1947

  50. [56]

    Nicholas Pangakis, Samuel Wolken, and Neil Fasching. 2023. Automated Annotation with Generative AI Requires Validation . arXiv preprint arXiv:2306.00176

    Pith/arXiv arXiv 2023

  51. [57]

    Furman, Franco Luque, Laura Alonso Alemany, and Mar\'ia Vanina Mart\'inez

    Juan Manuel P\'erez, Mariela Rajngewerc, Juan Carlos Giudici, Dami\'an A. Furman, Franco Luque, Laura Alonso Alemany, and Mar\'ia Vanina Mart\'inez. 2021. pysentimiento: A Python Toolkit for Opinion Mining and Social NLP tasks . arXiv preprint arXiv:2106.09462

    Pith/arXiv arXiv 2021

  52. [58]

    Qwen Team . 2024. Qwen2.5 Technical Report . arXiv preprint arXiv:2412.15115

    Pith/arXiv arXiv 2024

  53. [59]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks . In Proceedings of EMNLP

    2019

  54. [60]

    Michael V. Reiss. 2023. Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark . arXiv preprint arXiv:2304.11085

    Pith/arXiv arXiv 2023

  55. [61]

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. ``Why Should I Trust You?'': Explaining the Predictions of Any Classifier . In Proceedings of KDD

    2016

  56. [62]

    Thanveer Shaik, Xiaohui Tao, Yan Li, Christopher Dann, Jacquie McDonald, Petrea Redmond, and Linda Galligan. 2022. A Review of the Trends and Challenges in Adopting Natural Language Processing Methods for Education Feedback Analysis . IEEE Access, 10:56720--56739

    2022

  57. [63]

    Karen Sparck Jones. 1972. A Statistical Interpretation of Term Specificity and Its Application in Retrieval . Journal of Documentation, 28(1):11--21

    1972

  58. [64]

    Vega Barajas

    Esteban U. Vega Barajas. 2026. Evaluaci\'on automatizada de la pr\'actica docente mediante Procesamiento del Lenguaje Natural . Master's thesis, Universidad de Guadalajara

    2026

  59. [65]

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual E5 Text Embeddings: A Technical Report . arXiv preprint arXiv:2402.05672

    Pith/arXiv arXiv 2024

  60. [66]

    Wang, Pawan Wirawarn, Noah Goodman, and Dorottya Demszky

    Rose E. Wang, Pawan Wirawarn, Noah Goodman, and Dorottya Demszky. 2023. SIGHT: A Large Annotated Dataset on Student Insights Gathered from Higher Education Transcripts . In Proceedings of BEA at ACL

    2023

  61. [67]

    Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. 2024. Sentiment Analysis in the Era of Large Language Models: A Reality Check . In Findings of NAACL, pages 3881--3906

    2024

  62. [68]

    Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2024. Can Large Language Models Transform Computational Social Science? Computational Linguistics, 50(1):237--291

    2024