Pith. sign in

REVIEW 4 major objections 4 minor 50 references

Ratas framework: A comprehensive genai-based approach to rubric-based marking of real-world textual exams

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Splitting a rubric into a tree of simple rules lets a large language model grade long exams with near-human accuracy.

desk verdict The rubric knowledge tree and cascaded SR scoring are a real architectural contribution, but the headline accuracy rests on a rubric decomposition the paper never validates. read the letter →

arxiv 2505.23818 v1 pith:4XUZPL2W submitted 2025-05-27 cs.CL

Masoud Safilian , Amin Beheshti , Stephen Elbourn This is my paper
classification cs.CL
keywords automatedanswergradingrubric-basedscoringRubricKnowledgeTreelargelanguagemodelsGPT-4oexplainablefeedbacklonggenerativeAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that automated grading of long, open-ended exam answers becomes accurate and explainable when the rubric is first rebuilt as a tree of tiny, independently checkable rules. The proposed framework, RATAS, has a large language model split every rubric row into simplified leaf rules, judge each leaf as met or unmet, and propagate those judgments up the tree using weights inherited from the rubric. On a new dataset of 417 university project answers (average 366 words, up to 1,285 words) graded with a 34-criterion rubric, RATAS reports a mean absolute error of 0.0309, RMSE of 0.0443, and $R^2$ of 0.9627 against human scores, while direct GPT-4o scores the same answers with MAE 0.2355 and a negative $R^2$. If correct, this would mean rubric decomposition alone—without fine-tuning or domain-specific training—can bring LLM grading close to human levels and make every mark traceable to specific rubric criteria.

What carries the argument

The load-bearing object is the Rubric Knowledge Tree (RKT), built by an iterative rule-division process that turns each rubric row into Simplified Rules (SRs). The division must satisfy three criteria: the simpler rules collectively reconstruct the original rule, cover non-overlapping aspects, and have approximately equal influence on scoring. Scores are leaf-level $0$ or $1$ judgments made by a language model, then cascaded upward through the tree using each node's influence_on_scoring and score_source weights, which yields both the final numeric score and a structured reason for every internal node. This is what turns a holistic, judgment-heavy grading problem into many small yes/no classification tasks.

What would settle it

Grade a rubric that contains a conditional or capping rule (for example, 'if the formatting is wrong, the content score is capped at 50%') with RATAS; if the tree's additive weighted sum cannot reproduce the cap and the mean absolute error against human scores rises materially, the equal-weight, non-overlapping decomposition is where the method's validity stops.

Watch

Extended reading notes

Core claim

RATAS formalizes rubric-based grading as a scoring function over a Rubric Knowledge Tree (RKT), an object in which each row of an analytical rubric becomes a node and each node's rule is repeatedly decomposed until it reaches Simplified Rules (SRs) that cannot be simplified further. The decomposition is constrained to satisfy three criteria: the simpler rules must collectively reconstruct the original rule, cover distinct non-overlapping aspects, and carry approximately equal scoring importance. Scoring then reduces to asking a language model, for every SR, whether the answer fulfills it and which level of achievement is matched; the final score is obtained by cascading these $0$ or $1$ judgments up the tree, multiplying by each node's influence_on_scoring and score_source percentages. The paper reports that with GPT-4o handling all downstream NLP tasks through prompt engineering, RATAS achieves MAE 0.0309, RMSE 0.0443, $R^2$ 0.9627, and ICC 0.9662 on the 417-answer dataset, substantially outperforming direct GPT-4o while degrading less on answers longer than 600 words.

Load-bearing premise

The framework depends on the rule-division step preserving the original rubric's scoring semantics exactly: if a simplified rule misses part of the original requirement, overlaps with a sibling rule, or is given the wrong influence weight, the total score can be wrong even when every leaf judgment is individually correct.

Editorial extensions

If this is right

  • If the reported accuracy holds, rubric-based grading of long textual exams can be carried out by decomposing any rubric into a tree and letting a general language model judge each leaf, with no fine-tuning on domain-specific data.
  • The framework yields a structured, node-level reason for every score, so students can see which rubric criteria were met, partially met, or missed, and instructors can audit each mark.
  • The method handles answers far longer than existing automated short-answer and long-answer datasets (average 366 words versus the previous 120 words), with performance declining only modestly for answers over 600 words.
  • Because RATAS relies on prompt engineering rather than retraining, adapting it to a new course or rubric requires only a new rubric table and incurs per-leaf API costs.
  • The flexible scoring logic lets instructors write fewer, broader rubric rows while the tree expansion restores fine-grained scoring behavior, so rubric design is simplified without losing score resolution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The decisive untested step is whether the automatic rule division truly preserves scoring semantics; validating the three decomposition criteria with human annotation across multiple rubrics would turn a single-dataset demonstration into a general recipe.
  • Because every leaf requires a separate language-model call, token cost grows with rubric breadth; one could test whether sampling leaves or routing easy leaves to a smaller model preserves accuracy at lower cost.
  • The architecture separates rubric structure from scoring judgments, so the same Rubric Knowledge Tree could be reused across thousands of answers and across courses sharing a rubric, enabling longitudinal consistency checks that direct grading does not offer.
  • A stronger comparison than direct GPT-4o would be a fine-tuned LLM or an LLM prompted with the full rubric plus chain-of-thought; if RATAS still wins, the tree decomposition itself, rather than better prompting, is the active ingredient.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces RATAS (Rubric Automated Tree-based Answer Scoring), a framework that converts rubric rows into a tree of Simplified Rules (SRs), asks GPT-4o to judge each SR as met/unmet, and propagates binary leaf scores through the tree to produce final and partial scores with textual rationales. The authors formalize a rubric-based grading problem, describe the RKT construction and scoring pipeline, and evaluate on a self-constructed dataset of 417 selected responses from university project-based courses. They report strong accuracy for RATAS (MAE=0.0309, RMSE=0.0443, R²=0.9627) relative to direct GPT-4o (MAE=0.2355, RMSE=0.2923, R²=-0.6262), along with higher ICC. The central claim is that decomposing rubrics into simplified rules and scoring each rule with an LLM yields near-human, explainable grading on long real-world textual exams.

Significance. If the rubric-decomposition and the cascading scoring method are faithful to the original rubric semantics, RATAS would be a useful contribution to automated grading: it targets longer answers than most ASAG datasets, offers structured explainable feedback, and is subject-agnostic by construction. The authors also release code, which is a strength for reproducibility. However, the current evidence does not establish the central claim. The rule-division step is validated only by assertion, the formal scoring model is inconsistent with the implemented algorithm, and the evaluation is based on a self-selected dataset without confidence intervals or inter-rater reliability for the human ground truth. These gaps make the headline accuracy conditional on unverified design choices rather than a demonstrated property of the framework.

major comments (4)
  1. [§4.1] The load-bearing premise is that the LLMCTM rule-division process satisfies the three stated criteria (collective reconstruction, non-overlapping aspects, approximately equal scoring importance), but the paper provides no human annotation study, no reconstruction test, and no ablation in which the decomposition is replaced by a baseline. Because every leaf score is judged relative to the SRs generated by the same GPT-4o model, the reported MAE of 0.0309 in Table 3 is conditional on an unvalidated semantic-preservation assumption. Please report an independent verification where human raters confirm that the SRs reconstruct each Basic-Rule and that leaf influence weights reflect rubric intent, or at least an ablation with a different decomposition strategy.
  2. [§3 and Algorithm 1] The formal model defines SP and LQAP as continuous values via arg max over [0,1], but Algorithm 1 scores every leaf node as 0 or 1 and propagates via N.nodeScore = Σ C.infl × C.nodeScore with fixed equal influence. The paper never shows how intermediate Level-of-Achievement percentages (e.g., 50% in Table 1) are realized by binary leaves, nor how the continuous SP/LQAP equations relate to the implementation. This is not merely a presentational mismatch: it prevents the reader from verifying that the claimed floating-point scores follow from the stated rubric logic. The formalism and the pseudocode need to be reconciled, or the paper must explicitly state that the SP/LQAP equations are a conceptual idealization and Algorithm 1 is the operative method.
  3. [§6] The dataset consists of "417 selected responses from approximately 1,500 answers" with no description of the selection criteria. If the selection favors answers for which the rubric is unambiguous, the reported MAE underestimates deployment error. Moreover, no inter-rater reliability for the human ground-truth scores is reported; without it, the absolute scale of MAE=0.0309 cannot be interpreted, because human graders may themselves disagree by more than this amount. Please report the selection criteria, the number of raters, and the inter-rater agreement on the gold-standard scores.
  4. [§6 and Table 3] The headline metrics come from "three runs for each approach," but no variance, confidence intervals, or significance tests are reported, and the subset rows for >600 words are based on an unreported number of responses. It is also unclear what the reported ICC values (0.9662 vs 0.5984) are computed between: RATAS vs human, GPT-4o vs human, or run-to-run reliability? Without this information, the reliability claim is not verifiable. Please report the exact ICC definition, per-subset sample sizes, and per-run or bootstrap intervals for the main metrics.
minor comments (4)
  1. [Abstract, §2, §4] There are several typos and inconsistencies: "nstructors" in the abstract, "RiceCharm" for "RiceChem" in §6, "RATASframework" without a space in §4, and "Tabl 1" in §3. The acronym also alternates between RATG and RAGT; please choose one consistently.
  2. [§3] The displayed formula for AchievedScore lacks multiplication operators and is hard to parse; related_ls is also used before it is clearly defined. Please rewrite the formula with explicit operators and define all terms at first use.
  3. [§3] The definitions SP = arg max_{sp∈[0,1]} P(SP=sp|ri) and LQAP_{ij} = arg max over continuous values are formally ill-defined, since the probability of a specific continuous value is generally zero. If a modal or point estimate is intended, it should be stated explicitly.
  4. [§5.2 and Table 2] The text says "Table 1 compares key criteria" but the referenced table is Table 2; also, the prompts for LLMCTM, LLMCSC, and LLMSSR are not included in the paper, so the GitHub repository becomes essential for reproducibility. Please include the prompts or a detailed appendix, and clarify whether the evaluation dataset itself is released.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the scoring pipeline is not equivalent to its inputs and the evaluation is against external human grades, not a fitted parameter.

full rationale

The paper's derivation chain is not circular. Section 3 formalizes RATG by defining the total score as a sum over rubric rows, and Algorithm 1 computes that score by decomposing each rubric row into Simplified Rules, scoring each leaf binary, and propagating weighted sums up the RKT. No free parameter is fitted to the human scores, and no reported equation (MAE, RMSE, R²) is constructed from the framework's own outputs by definition. The evaluation in Section 6 compares RATAS's final score to manual human grading on a newly collected dataset, so the headline accuracy is an empirical result rather than a tautology. The rule-division criteria in Section 4.1 (reconstruction, non-overlap, approximate equal importance) are asserted rather than validated with human annotations or an ablation; this is a correctness and generalization risk, but it is not circularity, because an imperfect decomposition would produce a score that differs from the rubric's intended score rather than reproducing an input. The only self-citations ([12], [13]) appear as background examples of classical AI methods and are not load-bearing for the central claim. No equation reduces to a fitted value and no prediction is defined in terms of its own target, so the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on assumptions about rubric decomposition quality, LLM classification reliability, and the representativeness of the self-constructed dataset. None of these are independently validated in the paper, but none are hidden fitted parameters. The framework also introduces the Rubric Knowledge Tree as a computational structure, but it is a data structure rather than a new physical or ontological entity.

assumptions (3)
  • domain assumption LLM-generated simplified rules reconstruct original rubric rules exactly and with approximately equal scoring importance.
    Section 4.1 states the rule-division criteria but provides no validation that LLMCTM satisfies them for real rubrics.
  • domain assumption GPT-4o's binary classification of each Simplified Rule as met or unmet is accurate enough for grading.
    Section 4.2 and Algorithm 1 rely on LLMSSR to assign 0 or 1 scores to every leaf rule, but no human agreement study is reported.
  • domain assumption The 417 selected responses are a representative sample of real exams and the human scores used as ground truth are correct.
    Section 6 says 417 responses were selected from about 1,500 without describing the selection criteria or measuring inter-rater reliability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Ratas framework: A comprehensive genai-based approach to rubric-based marking of real-world textual exams." pith.science (2026). https://pith.science/paper/4XUZPL2W

@misc{pith2026250523818,
  author       = {Pith},
  title        = {Pith review of: Ratas framework: A comprehensive genai-based approach to rubric-based marking of real-world textual exams},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4XUZPL2W}},
  note         = {Machine review of arXiv:2505.23818}
}
read the original abstract

Automated answer grading is a critical challenge in educational technology, with the potential to streamline assessment processes, ensure grading consistency, and provide timely feedback to students. However, existing approaches are often constrained to specific exam formats, lack interpretability in score assignment, and struggle with real-world applicability across diverse subjects and assessment types. To address these limitations, we introduce RATAS (Rubric Automated Tree-based Answer Scoring), a novel framework that leverages state-of-the-art generative AI models for rubric-based grading of textual responses. RATAS is designed to support a wide range of grading rubrics, enable subject-agnostic evaluation, and generate structured, explainable rationales for assigned scores. We formalize the automatic grading task through a mathematical framework tailored to rubric-based assessment and present an architecture capable of handling complex, real-world exam structures. To rigorously evaluate our approach, we construct a unique, contextualized dataset derived from real-world project-based courses, encompassing diverse response formats and varying levels of complexity. Empirical results demonstrate that RATAS achieves high reliability and accuracy in automated grading while providing interpretable feedback that enhances transparency for both students and nstructors.

Figures

Figures reproduced from arXiv: 2505.23818 by the authors.

Figure 2
Figure 2. RATAS Conceptual Steps and Data Flow [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. RATAS High-Level Architecture. – id. Each node in the Rubric Knowledge Tree (RKT) is assigned a unique string￾based ID by the "RKT Constructing Core". The root ID is RID, while child IDs append “.i” (an integer) to the parent’s ID, ensuring a hierarchical structure. – leaf. This boolean attribute indicates whether a node is a leaf (1) or an internal node (0). It is set by “core of RKT Constructing” in each iteration… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 41 canonical work pages

  1. [1]

    The power of open-ended questions for kids,

    M. T . Cupboard, “The power of open-ended questions for kids,” Blog post, 2025, accessed: February 13, 2025. [Online]. Available: https://www. myteachingcupboard.com/blog/the-power-of-open-ended-questions-for-kids

    work page 2025

  2. [2]

    Eliciting self-explanations improves understanding,

    M. T . Chi, N. De Leeuw, M.-H. Chiu, and C. LaVancher, “Eliciting self-explanations improves understanding,” Cognitive science, vol. 18, no. 3, pp. 439–477, 1994

    work page 1994

  3. [3]

    Kolappan, Computer Assisted Short Answer Grading with Rubrics using Active Learning, 2023

    G. Kolappan, Computer Assisted Short Answer Grading with Rubrics using Active Learning, 2023

    work page 2023

  4. [4]

    Enhancing computer assisted assessment using rubrics in a qti editor,

    P . Santos, X. Colina, D. Hernández-Leo, J. Melero, and J. Blat, “Enhancing computer assisted assessment using rubrics in a qti editor,” in 2009 Ninth IEEE International Conference on Advanced Learning Technologies. IEEE, 2009, pp. 303–305

    work page 2009

  5. [5]

    Reducing workload in short answer grad- ing using machine learning,

    R. Weegar and P . Idestam-Almquist, “Reducing workload in short answer grad- ing using machine learning,” International Journal of Artificial Intelligence in Education, vol. 34, no. 2, pp. 247–273, 2024

    work page 2024

  6. [6]

    Automatic assessment of text-based responses in post-secondary education: A systematic review,

    R. Gao, H. E. Merzdorf, S. Anwar, M. C. Hipwell, and A. R. Srinivasa, “Automatic assessment of text-based responses in post-secondary education: A systematic review,”Computers and Education: Artificial Intelligence, vol. 6, p. 100206, 2024

    work page 2024

  7. [7]

    The eras and trends of automatic short answer grading,

    S. Burrows, I. Gurevych, and B. Stein, “The eras and trends of automatic short answer grading,” International journal of artificial intelligence in education , vol. 25, pp. 60–117, 2015

    work page 2015

  8. [8]

    Automated long answer grading with ricechem dataset,

    S. Sonkar, K. Ni, L. Tran Lu, K. Kincaid, J. S. Hutchinson, and R. G. Bara- niuk, “Automated long answer grading with ricechem dataset,” in International Conference on Artificial Intelligence in Education. Springer, 2024, pp. 163–176

    work page 2024

Show all 50 references
  1. [9]

    Analysis of algorithms in automated marking in education: a proposed hybrid algorithm,

    B. Prasain and S. K. Bajaj, “Analysis of algorithms in automated marking in education: a proposed hybrid algorithm,” in 2020 5th International Conference on Innovative Technologies in Intelligent Systems and Industrial Applications (CITISIA). IEEE, 2020, pp. 1–10

    2020

  2. [10]

    Inject rubrics into short answer grading system,

    T . Wang, N. Inoue, H. Ouchi, T . Mizumoto, and K. Inui, “Inject rubrics into short answer grading system,” in Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), 2019, pp. 175–182

    2019

  3. [11]

    Explainable automatic grading with neural addi- tive models,

    A. Condor and Z. Pardos, “Explainable automatic grading with neural addi- tive models,” in International Conference on Artificial Intelligence in Education. Springer, 2024, pp. 18–31

    2024

  4. [12]

    An approximation algorithm for the subpath planning problem

    M. Safilian, S. M. Hashemi, S. Eghbali, and A. Safilian, “An approximation algorithm for the subpath planning problem.” in IJCAI, 2016, pp. 669–675

    2016

  5. [13]

    Probabilistic roadmap planner with adaptive sampling based on clustering

    M. Safilian, S. Eghbali, A. Safilian, and A. Nouri, “Probabilistic roadmap planner with adaptive sampling based on clustering.”

  6. [14]

    Attention is all you need,

    A. Vaswani, “ Attention is all you need,”Advances in Neural Information Processing Systems, 2017

    2017

  7. [15]

    Alto,Modern Generative AI with ChatGPT and OpenAI Models: Leverage the capabilities of OpenAI’s LLM for productivity and innovation with GPT3 and GPT4

    V . Alto,Modern Generative AI with ChatGPT and OpenAI Models: Leverage the capabilities of OpenAI’s LLM for productivity and innovation with GPT3 and GPT4. Packt Publishing Ltd, 2023. 14 M. Safilian and A. Beheshti

    2023

  8. [16]

    (2024, Sep.) Global generative ai in educa- tion market

    MarketResearch.Biz. (2024, Sep.) Global generative ai in educa- tion market. [Online]. Available: https: //marketresearch.biz/report/ generative-ai-in-education-market /

    2024

  9. [17]

    uchemann, M. Bannert, D. Dementieva, F . Fischer, U. Gasser, G. Groh, S. G

    E. Kasneci, K. Sessler, S. K"uchemann, M. Bannert, D. Dementieva, F . Fischer, U. Gasser, G. Groh, S. G"unnemann, E. H"ullermeieret al., “Chatgpt for good? on opportunities and challenges of large language models for education,”Learning and individual differences, vol. 103, p....

    2023

  10. [18]

    Llms in short answer scoring: Limi- tations and promise of zero-shot and few-shot approaches,

    I. Chamieh, T . Zesch, and K. Giebermann, “Llms in short answer scoring: Limi- tations and promise of zero-shot and few-shot approaches,” in Proceedings of the 19th workshop on innovative use of nlp for building educational applications (bea 2024), 2024, pp. 309–315

    2024

  11. [19]

    Gpt-4o: The cutting-edge advancement in multi- modal llm,

    R. Islam and O. M. Moushi, “Gpt-4o: The cutting-edge advancement in multi- modal llm,” Authorea Preprints, 2024

    2024

  12. [20]

    The joint student response analysis and recognizing textual entailment challenge: making sense of student responses in educational applications,

    M. O. Dzikovska, R. D. Nielsen, and C. Leacock, “The joint student response analysis and recognizing textual entailment challenge: making sense of student responses in educational applications,” Language Resources and Evaluation , vol. 50, pp. 67–93, 2016

    2016

  13. [21]

    Learning to grade short answer ques- tions using semantic similarity measures and dependency graph alignments,

    M. Mohler, R. Bunescu, and R. Mihalcea, “Learning to grade short answer ques- tions using semantic similarity measures and dependency graph alignments,” in Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, 201...

    2011

  14. [22]

    Triplet loss based siamese networks for automatic short answer grading,

    N. Yeruva, S. Venna, H. Indukuri, and M. Marreddy , “Triplet loss based siamese networks for automatic short answer grading,” inProceedings of the 14th Annual Meeting of the Forum for Information Retrieval Evaluation, 2022, pp. 60–64

    2022

  15. [23]

    The automated grading of student open responses in mathematics,

    J. A. Erickson, A. F . Botelho, S. McAteer, A. Varatharaj, and N. T . Heffernan, “The automated grading of student open responses in mathematics,” in Proceedings of the tenth international conference on learning analytics & knowledge, 2020, pp. 615–624

    2020

  16. [24]

    Automatic short answer grading with semspace sense vectors and malstm,

    C. N. Tulu, O. Ozkaya, and U. Orhan, “Automatic short answer grading with semspace sense vectors and malstm,” IEEE Access, vol. 9, pp. 19 270–19 280, 2021

    2021

  17. [25]

    Fine-tuning chatgpt for automatic scoring,

    E. Latif and X. Zhai, “Fine-tuning chatgpt for automatic scoring,” Computers and Education: Artificial Intelligence, vol. 6, p. 100210, 2024

    2024

  18. [26]

    Automatic exam correction framework (aecf) for the mcqs, essays, and equations matching,

    H. M. Balaha and M. M. Saafan, “Automatic exam correction framework (aecf) for the mcqs, essays, and equations matching,” IEEE Access, vol. 9, pp. 32 368– 32 389, 2021

    2021

  19. [27]

    Automatic short-answer grading via bert-based deep neural networks,

    X. Zhu, H. Wu, and L. Zhang, “Automatic short-answer grading via bert-based deep neural networks,” IEEE Transactions on Learning Technologies, vol. 15, no. 3, pp. 364–375, 2022

    2022

  20. [28]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  21. [29]

    Kublik and S

    S. Kublik and S. Saboo, GPT-3: The Ultimate Guide to Building NLP Products with OpenAI API. Packt Publishing Ltd, 2023

    2023

  22. [30]

    Llms in short answer scoring: Limi- tations and promise of zero-shot and few-shot approaches,

    I. Chamieh, T . Zesch, and K. Giebermann, “Llms in short answer scoring: Limi- tations and promise of zero-shot and few-shot approaches,” in Proceedings of the 19th workshop on innovative use of nlp for building educational applications (bea 2024), 2024, pp. 309–315. Title Sup...

    2024

  23. [31]

    Short answer scoring with gpt-4,

    L. Jiang and N. Bosch, “Short answer scoring with gpt-4,” in Proceedings of the Eleventh ACM Conference on Learning@ Scale, 2024, pp. 438–442

    2024

  24. [32]

    Automated text scoring and real-time adjustable feedback: Supporting revision of scientific arguments involving uncertainty ,

    H.-S. Lee, A. Pallant, S. Pryputniewicz, T . Lord, M. Mulholland, and O. L. Liu, “ Automated text scoring and real-time adjustable feedback: Supporting revision of scientific arguments involving uncertainty ,”Science Education, vol. 103, no. 3, pp. 590–622, 2019

    2019

  25. [33]

    Machine learning-enabled automated feedback: Supporting students’ revision of scientific arguments based on data drawn from simulation,

    H.-S. Lee, G.-H. Gweon, T . Lord, N. Paessel, A. Pallant, and S. Pryputniewicz, “Machine learning-enabled automated feedback: Supporting students’ revision of scientific arguments based on data drawn from simulation,”Journal of Science Education and Technology, vol. 30, no. 2,...

    2021

  26. [34]

    Explaining black box text modules in natural language with language models,

    C. Singh, A. R. Hsu, R. Antonello, S. Jain, A. G. Huth, B. Yu, and J. Gao, “Explaining black box text modules in natural language with language models,” arXiv preprint arXiv:2305.09863, 2023

    2023 arXiv

  27. [35]

    Using criteria-based assessment rubrics for online marking: Technological and pedagogical challenges,

    S. El Boudamoussi, “Using criteria-based assessment rubrics for online marking: Technological and pedagogical challenges,” Journal of Higher Education Theory and Practice, vol. 22, no. 8, 2022

    2022

  28. [36]

    A comparison of the performance of analytic vs. holistic scoring rubrics to assess l2 writing,

    C. S. Wiseman, “A comparison of the performance of analytic vs. holistic scoring rubrics to assess l2 writing,” International Journal of Language Testing, vol. 2, no. 1, pp. 59–92, 2012

    2012

  29. [37]

    Assessment rubrics: towards clearer and more replicable design, research and practice,

    P . Dawson, “Assessment rubrics: towards clearer and more replicable design, research and practice,” Assessment & Evaluation in Higher Education, vol. 42, no. 3, pp. 347–360, 2017

    2017

  30. [38]

    Topic modeling algorithms and applications: A survey ,

    A. Abdelrazek, Y. Eid, E. Gawish, W . Medhat, and A. Hassan, “Topic modeling algorithms and applications: A survey ,”Information Systems, vol. 112, p. 102131, 2023

    2023

  31. [39]

    A survey of text classification algorithms,

    C. C. Aggarwal and C. Zhai, “A survey of text classification algorithms,” Mining text data, pp. 163–222, 2012

    2012

  32. [40]

    The falcon series of open language models,

    E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, É. Goffinet, D. Hesslow, J. Launay , Q. Malarticet al., “The falcon series of open language models,” arXiv preprint arXiv:2311.16867, 2023

    2023 arXiv

  33. [41]

    Pegasus: Pre-training with extracted gap-sentences for abstractive summarization,

    J. Zhang, Y. Zhao, M. Saleh, and P . Liu, “Pegasus: Pre-training with extracted gap-sentences for abstractive summarization,” in International conference on machine learning. PMLR, 2020, pp. 11 328–11 339

    2020

  34. [42]

    Llama: Open and efficient foundation language models,

    H. Touvron, T . Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T . Lacroix, B. Roz- ière, N. Goyal, E. Hambro, F . Azharet al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  35. [43]

    Gemma: Open models based on gemini research and technology ,

    G. Team, T . Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love et al., “Gemma: Open models based on gemini research and technology ,”arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  36. [44]

    Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,

    M. Lewis, Y . Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy , V . Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” arXiv preprint arXiv:1910.13461, 2019

    1910 arXiv

  37. [45]

    GPT-4o vs. GPT-4: How do they compare?

    TechTarget, “GPT-4o vs. GPT-4: How do they compare?” Jan 2025, accessed: 2024-07-24. [Online]. Available: https: //www.techtarget.com/ searchenterpriseai/feature/GPT-4o-vs-GPT-4-How-do-they-compare 16 M. Safilian and A. Beheshti

    2025

  38. [46]

    Putting gpt-4o to the sword: A comprehensive evaluation of language, vision, speech, and multimodal proficiency,

    S. Shahriar, B. D. Lund, N. R. Mannuru, M. A. Arshad, K. Hayawi, R. V . K. Bevara, A. Mannuru, and L. Batool, “Putting gpt-4o to the sword: A comprehensive evaluation of language, vision, speech, and multimodal proficiency,”Applied Sciences, vol. 14, no. 17, p. 7782, 2024

    2024

  39. [47]

    Hift: A hier- archical full parameter fine-tuning strategy ,

    Y . Liu, Y . Zhang, Q. Li, S. Feng, D. Wang, Y . Zhang, and H. Schütze, “Hift: A hier- archical full parameter fine-tuning strategy ,”arXiv preprint arXiv:2401.15207, 2024

    2024 arXiv

  40. [48]

    Prompt design and engineering: Introduction and advanced methods,

    X. Amatriain, “Prompt design and engineering: Introduction and advanced methods,” arXiv preprint arXiv:2401.14423, 2024

    2024 arXiv

  41. [49]

    Reframing instruc- tional prompts to gptk’s language,

    S. Mishra, D. Khashabi, C. Baral, Y . Choi, and H. Hajishirzi, “Reframing instruc- tional prompts to gptk’s language,” arXiv preprint arXiv:2109.07830, 2021

    2021 arXiv

  42. [50]

    Azure openai assistants - concepts,

    Microsoft, “Azure openai assistants - concepts,” Microsoft Learn, 2025, accessed: February 13, 2025. [Online]. Available: https://learn.microsoft.com/en-us/ azure/ai-services/openai/concepts/assistants

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.