Using Large Language Models to Assess Teachers' Pedagogical Content Knowledge

Shiyu Wang; Xiaoming Zhai; Yaxuan Yang

REVIEW 4 major objections 6 minor 21 references

Using Large Language Models to Assess Teachers' Pedagogical Content Knowledge

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Rater severity and scenario sensitivity, not scenario content, carry most of the construct-irrelevant variance when GPT-4, supervised ML, or humans score teachers' PCK, with GPT-4 the most lenient and ML the most severe.

desk verdict Useful, well-scoped study on LLM scoring bias, but the "LLM is most lenient" result is tied to a single prompt configuration. read the letter →

arxiv 2505.19266 v1 pith:DS5M7226 submitted 2025-05-25 cs.AI cs.CY

Yaxuan Yang , Shiyu Wang , Xiaoming Zhai This is my paper · ORCID

classification cs.AIcs.CY

keywords construct-irrelevantvarianceautomaticscoringpedagogicalcontentknowledgelargelanguagemodelsGPT-4many-facetRaschmodelgeneralizedlinearmixedraterseverity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Performance-based assessment of teachers' pedagogical content knowledge (PCK) is labor-intensive, and automated scoring is attractive only if it does not quietly add noise unrelated to what is being measured. This paper asks whether large language models introduce such construct-irrelevant variance (CIV) when they score teachers' video-based constructed responses, and how that compares with human raters and supervised machine learning. Using generalized linear mixed models—which split score variation among teachers, scenarios, raters, and rater-scenario combinations—on responses from 187 teachers to three classroom video scenarios, the paper finds that scenario variability is minimal in both analytic tasks, while rater severity and rater-by-scenario sensitivity contribute substantial CIV, especially in the more interpretive task of evaluating teacher responsiveness. The central result is a rater ordering: GPT-4 is the most lenient scorer, the supervised ML model the most severe and least scenario-sensitive, with human raters in between.

What carries the argument

The carrying mechanism is the many-facet Rasch model (MFRM) fitted as a generalized linear mixed model with a logit link: a measurement model that treats teachers, scenarios, raters, and rater-by-scenario interactions as separate random facets rather than assuming raters are interchangeable. Best linear unbiased predictions for each rater give a common logit-scale severity index, and the rater-by-scenario random effect gives each rater's context sensitivity. This machinery lets the paper separate how hard the scenario is, how strict the rater is, and how much the rater changes across scenarios; it also produces the singular-fit zero scenario variance in Task II.

What would settle it

A direct falsifier is to score the same 187-teacher responses with the same LLM under several prompts that differ only in the wording of the rubric restatement, the few-shot examples, or the temperature, then refit the generalized linear mixed model; if any prompt moves GPT-4's severity estimate close to or below the ML model's, the central ordering (LLM most lenient, ML most severe) is an artifact of the particular prompt.

Watch

Extended reading notes

Core claim

The paper's central discovery is that construct-irrelevant variance in these PCK scores is rater-dominated rather than scenario-dominated. In a many-facet Rasch model estimated as a generalized linear mixed model, scenario variance was tiny (0.021 on Task I; zero on Task II), while rater-severity variance grew from 0.249 to 6.343 and rater-by-scenario variance from 0.008 to 0.129 between the two tasks. On the logit severity scale, GPT-4 was the most lenient rater in both tasks (severity 0.93 and 5.19), the supervised ML model was the most severe (−0.37 and −1.60), and the three human raters sat in between; the gap widened sharply on Task II, the more interpretive construct of evaluating teacher responsiveness. The paper concludes that an LLM can score efficiently but imports its own systematic leniency and scenario sensitivity, so the choice of scoring source changes the meaning of the resulting scores.

Load-bearing premise

The paper treats a single GPT-4 run under one hand-designed prompt as a stable 'LLM rater,' and three pilot-chosen video clips as representative scenarios; if a different prompt or a different set of clips changed the estimated rater rankings or variance components, the headline conclusions would not hold.

Editorial extensions

If this is right

Score comparability between scoring sources holds only after adjusting for rater severity; without correction, the ML model would depress apparent teacher performance and the LLM would inflate it.
Rater-by-scenario sensitivity becomes the dominant CIV concern on interpretive tasks, so automatic scoring of such tasks needs context-level calibration, not just an overall leniency correction.
The near-zero scenario variance suggests the three classroom videos functioned as roughly equivalent elicitation contexts, shifting the practical validity burden from scenario selection to rater calibration.
Choosing an LLM as a scorer is itself a rater decision: the model carries a stable leniency profile within this prompt configuration, so prompt design is part of the measurement model, not a preprocessing step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves untested that LLM leniency is a property of the model rather than the prompt; a prompt-sweep experiment varying rubric wording, few-shot examples, and temperature would map how much the severity ordering moves.
With only three pilot-selected video clips, the near-zero scenario variance cannot rule out scenario-level CIV in a broader set of classroom contexts; a larger scenario sample could change the variance decomposition.
Because scores are binary, the analysis compresses the scoring scale; a multi-level rubric might show that LLM leniency concentrates at the boundary between score levels rather than uniformly.
A practical calibration strategy follows from the paper's result: estimate an LLM severity correction from a small human-scored subset and apply it to the LLM's scores, making automatic scoring both scalable and comparable to human judgments.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Here's my read on arXiv:2505.19266. The paper asks whether GPT-4 scoring of teachers' PCK responses introduces the same kinds of construct-irrelevant variance we already know about for human and ML raters. The new thing is the comparison: human vs. supervised ML vs. LLM on the same rater facets (severity, scenario sensitivity) within a multi-facet Rasch/GLMM framework. That specific comparison hasn't been done before, and it's a legitimate extension of the authors' earlier work on the CIV framework.

The paper does a few things well. The modeling approach is appropriate: binary scores, logit link, random effects for teacher, scenario, rater, and rater-by-scenario. The variance components and BLUPs are internally consistent with the qualitative story: scenario variance is tiny or zero, rater variance jumps in Task II (the more interpretive task), and the LLM is more lenient than humans, while the ML model is more severe. The likelihood ratio tests support the models. The paper is clearly written and appropriately cautious about inferring mechanisms. It also cites the relevant prior work, including the authors' own dataset and rubric, which is fine since those are published.

The soft spots are real but not fatal. The biggest one is the prompt. The LLM is represented by one GPT-4 run under one hand-designed prompt with three few-shot examples, no variation in wording, temperature, or sampling. The stress-test note is right: the 'LLM is most lenient' result is a comparative ranking, and a different prompt could shift the severity BLUP enough to reorder the raters. The paper itself cites work showing prompt design changes GPT scoring. So the headline result is conditional on that specific configuration. Also, only three human raters and three scenarios, all from one pilot selection, and no confidence intervals or data/code are provided. The Task II model had a singular fit with scenario variance estimated at zero, which makes those estimates somewhat unstable. These are standard limitations for a small study, not fatal flaws.

The central descriptive claim—that rater-related variance dominates scenario variance and that LLM scoring carries its own systematic leniency—holds up for the data as presented. I would send this to peer review, but the revision should include at least a prompt-sensitivity check (e.g., varying the prompt or running multiple samples) and ideally confidence intervals on the variance components. The paper is useful for assessment designers and validity researchers: it gives a concrete warning that LLM scoring is not rater-neutral, even if the magnitude and direction may be prompt-dependent.

Recommendation: engage with it; a serious referee can improve the robustness discussion. I'd read the revision.

Best.

Referee Report

4 major / 6 minor

Summary. This paper studies construct-irrelevant variance (CIV) in the automated and human scoring of teachers' pedagogical content knowledge (PCK) from video-based constructed-response tasks. Using a generalized linear mixed model / many-facet Rasch model applied to binary scores from three trained human raters, a supervised ML model, and a single run of GPT-4, the authors estimate variance components for scenario effects, rater severity, and rater-by-scenario sensitivity across two PCK tasks. They report that scenario variance is minimal, rater-related variance is large and especially pronounced in the more interpretive Task II, the ML model is the most severe and least scenario-sensitive rater, and the LLM is the most lenient rater. The paper discusses implications for rater training and automated scoring design.

Significance. If the central claims hold, the study makes a useful and non-obvious point: LLM-based scoring is not a neutral or noise-free alternative to human or ML scoring but introduces systematic rater tendencies of its own, whose direction and size differ across tasks. The choice of a GLMM/MFRM framework to decompose CIV into scenario, rater, and interaction facets is appropriate for binary scoring data, and the reporting of variance components and BLUPs is a strength. The replication of the previously reported pattern that the ML model is more severe but more stable than human raters is also valuable. However, the headline results rest on very limited empirical support: one GPT-4 run under one prompt, five raters in the rater facet, three scenarios, and no confidence intervals for any variance component or severity estimate. The manuscript therefore reports a plausible and interesting hypothesis rather than a fully established comparative claim.

major comments (4)

[Section 4.4, Table 3] The central claim that the LLM was the most lenient rater rests entirely on a single GPT-4 run with one hand-designed prompt, one set of three few-shot examples, and no reported decoding configuration. Because the comparison is a ranking of rater severity, a different prompt wording, different few-shot examples, or a different temperature could reorder the severity estimates relative to the human and ML raters. The authors themselves cite Lee et al. (2024) to argue that prompt configuration materially changes GPT-based scoring behavior, so a prompt-sensitivity analysis is not a nice extra but a necessary condition for the headline claim. Please vary the prompt wording, few-shot examples, and sampling temperature, report the distribution of severity BLUPs across runs, and state the decoding parameters used.
[Table 3 and Section 5, RQ2] No standard errors, confidence intervals, or other uncertainty measures are reported for the severity BLUPs or for the variance components in Table 2, even though the rater facet has only five levels and the scenario facet only three levels. Statements such as 'the LLM consistently had the highest severity estimates' and 'the ML model was the most severe' need uncertainty quantification: with this few clusters, the BLUPs can be unstable, and the apparent Task I/Task II differences in rater-by-scenario variance (0.008 vs. 0.129) may not be statistically distinguishable. Please report bootstrap or profile-likelihood intervals for the variance components and BLUPs.
[Section 4.1] The three video scenarios were selected from pilot data, but no selection criteria are stated. If the scenarios were chosen to be comparable in difficulty, then the finding of minimal scenario variance in Task I and zero scenario variance in Task II is partly a consequence of the selection procedure rather than an empirical property of the assessment context. Please report the selection criteria and, if possible, include a sensitivity analysis using the full pilot scenario set or discuss how selection could bias the scenario variance estimates.
[Section 5, Table 1] The Task II model is reported as a 'singular fit' with scenario variance estimated at zero, yet the likelihood-ratio test for this model is reported as significant. A variance component estimated at a boundary can make the standard LRT invalid and can also affect the estimates of the remaining variance components. Please discuss the boundary estimate explicitly, and consider using a non-boundary test (e.g., a simulation-based or score-based test) for the contribution of random effects in Task II.

minor comments (6)

[Section 4.4] The prompt text, the three few-shot examples, and the exact GPT-4 model version are not provided, which prevents replication. Please include the full prompt or an appendix with the prompt and examples.
[Section 4.2] It is not clear whether the three human raters' independent scores before consensus or the consensus scores were used in the GLMM. The phrase 'Three trained raters independently coded all responses' followed by 'A consensus-based approach was used to resolve discrepancies' leaves this ambiguous, and the choice affects whether the human rater variance is identifiable.
[Section 4.3] The section heading contains a spacing artifact in the manuscript ('V ariance'); please fix the typography.
[Table 3] The sign convention that larger severity estimates indicate greater leniency is counterintuitive for a Rasch model and should be stated directly in the table caption, not only in the text.
[Section 5] The text says 'in Task II, raters showed much more variability' and that human raters displayed more fluctuation, but Figure 2 is not quantitatively described in the text. Reporting the range and standard deviation of the rater-by-scenario BLUPs for each rater type would make the comparison more precise.
[Section 4.4] The paper does not report whether the LLM output was scored in a single batch or across multiple calls, nor whether any non-determinism in the API was observed. This is a minor reproducibility issue in addition to the prompt-sensitivity concern above.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the LLM severity comparison is an empirical GLMM fit to score data, not a construction that forces the conclusion.

full rationale

The paper's central claim—that the LLM was the most lenient scorer and the ML model the most severe—is an empirical output of a generalized linear mixed model fitted to binary scores. The GLMM/MFRM in Section 4.4 estimates variance components and rater-severity BLUPs from the observed scores; no equation defines the LLM's severity in terms of the research question, and no fitted parameter is renamed as a prediction. The scores themselves (human, ML, and LLM) are data entering the model, not derived quantities. The paper does rely on the authors' prior work for the dataset, rubrics, CIV framework, and the ML scores (e.g., refs. [19] and [20]), but this reuse is contextual and not load-bearing for the novel LLM comparison: the LLM scores are newly generated with GPT-4, and the GLMM is fit afresh. The framework does not constrain the ranking of raters, and no self-citation chain or uniqueness theorem is invoked to forbid alternatives. The prompt-dependence concern raised by the reader is a generalizability/validity limitation, not a circularity: the paper reports one prompt configuration, but the conclusion is not true by construction for that configuration. Overall, the derivation chain is self-contained for the claims made, and the severity ranking is an empirical finding rather than an artifact of definition.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The model fit itself is not a derivation; the reported variance components and severity BLUPs are estimates from data. The hand-selected prompt examples and unstated decoding settings are the main investigator-controlled choices that could change the LLM's apparent severity. The substantive assumptions are the GLMM distributional assumptions, the representativeness of the three video scenarios, and the inherited construct validity of the rubric.

free parameters (2)

Few-shot examples in the LLM prompt = 3 per task, content not fully reported
The examples were hand-picked by the authors to illustrate rubric interpretation; they can shift the model's severity, but no sensitivity analysis is reported (Section 4.4).
GPT-4 decoding configuration (temperature, top-p, sampling) = not reported
A single run is treated as a stable rater; stochastic decoding variability is not modeled or reported (Section 4.4).

assumptions (4)

domain assumption Random intercepts for teacher ability, rater severity, and rater-by-scenario sensitivity are normally distributed on the logit scale.
Assumed by the GLMM/MFRM fit in Section 4.4; severe violations would distort variance component estimates.
domain assumption The three video scenarios are a representative sample from a universe of classroom scenarios, permitting inference about scenario variability.
Only three clips are used (Section 4.1), and the Task II model is singular with scenario variance estimated as zero; variance estimates from three scenarios are fragile.
domain assumption The analytic rubrics and binary scores measure the intended PCK sub-constructs: analyzing student thinking and evaluating teacher responsiveness.
The construct-validity chain is inherited from prior work (Zhai et al., reference 19); if rubric scores carry construct-irrelevant content, the severity estimates are confounded.
domain assumption Human rater scores and consensus scores are suitable comparison baselines for machine scoring.
The paper uses human raters as the reference category for severity and sensitivity, with only three raters; their estimates are treated as stable enough for comparison.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Using Large Language Models to Assess Teachers' Pedagogical Content Knowledge." pith.science (2026). https://pith.science/paper/DS5M7226

@misc{pith2026250519266,
  author       = {Pith},
  title        = {Pith review of: Using Large Language Models to Assess Teachers' Pedagogical Content Knowledge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DS5M7226}},
  note         = {Machine review of arXiv:2505.19266}
}

read the original abstract

Assessing teachers' pedagogical content knowledge (PCK) through performance-based tasks is both time and effort-consuming. While large language models (LLMs) offer new opportunities for efficient automatic scoring, little is known about whether LLMs introduce construct-irrelevant variance (CIV) in ways similar to or different from traditional machine learning (ML) and human raters. This study examines three sources of CIV -- scenario variability, rater severity, and rater sensitivity to scenario -- in the context of video-based constructed-response tasks targeting two PCK sub-constructs: analyzing student thinking and evaluating teacher responsiveness. Using generalized linear mixed models (GLMMs), we compared variance components and rater-level scoring patterns across three scoring sources: human raters, supervised ML, and LLM. Results indicate that scenario-level variance was minimal across tasks, while rater-related factors contributed substantially to CIV, especially in the more interpretive Task II. The ML model was the most severe and least sensitive rater, whereas the LLM was the most lenient. These findings suggest that the LLM contributes to scoring efficiency while also introducing CIV as human raters do, yet with varying levels of contribution compared to supervised ML. Implications for rater training, automated scoring design, and future research on model interpretability are discussed.

Figures

Figures reproduced from arXiv: 2505.19266 by the authors.

**Figure 1.** Dot plot of rater severity across Task I and Task II. LLM severity increased markedly in Task II, indicating a shift in rating behavior Rater sensitivity across scenarios is shown in [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗

**Figure 2.** Rater-by-scenario sensitivity across tasks. Human raters showed greater variation across scenarios in Task II, while the LLM and ML scorers remained more stable, suggesting lower context-driven CIV in machine-based scoring These results suggest that while scenario features themselves contributed little to score variation, rater-related factors, especially severity and scenario sensitivity, introduced substantial CI… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 19 canonical work pages

[1]

arXiv preprint arXiv:2504.05736 (2025)

Cai, Y., Liang, K., Lee, S., Wang, Q., Wu, Y.: Rank-then-score: Enhancing large language models for automated essay scoring. arXiv preprint arXiv:2504.05736 (2025)

arXiv 2025
[2]

Research in Science & Technological Education34(2), 237–251 (2016)

Canbazoglu Bilici, S., Guzey, S.S., Yamak, H.: Assessing pre-service science teach- ers’ technological pedagogical content knowledge (tpack) through observations and lesson plans. Research in Science & Technological Education34(2), 237–251 (2016)

work page 2016
[3]

Teaching and teacher education34, 12–25 (2013)

Depaepe, F., Verschaffel, L., Kelchtermans, G.: Pedagogical content knowledge: A systematic review of the way in which the concept has pervaded mathematics educational research. Teaching and teacher education34, 12–25 (2013)

work page 2013
[4]

Using GPT-4 to Augment Unbalanced Data for Automatic Scoring

Fang, L., Lee, G.G., Zhai, X.: Using gpt-4 to augment unbalanced data for auto- matic scoring. arXiv preprint arXiv:2310.18365 (2023)

work page Pith review arXiv 2023
[5]

In: JSM Proceedings (2014)

Greenwood, M., Jesse, D.: Scoring and then analyzing or analyzing while scoring: An application of glmm to an education instrument development and analysis. In: JSM Proceedings (2014)

work page 2014
[6]

Educational Measurement: Issues and Practice23(1), 17–27 (2004)

Haladyna, T.M., Downing, S.M.: Construct-irrelevant variance in high-stakes test- ing. Educational Measurement: Issues and Practice23(1), 17–27 (2004)

work page 2004
[7]

Linguistic Research36 (2019)

Kim, H.J., Lee, J., You, H.J.: Analysis of rater effect in the evaluation of sec- ond language grammatical knowledge in the context of writing: Application of a generalized linear model. Linguistic Research36 (2019)

work page 2019
[8]

Studies in science education45(2), 169–204 (2009)

Kind, V.: Pedagogical content knowledge in science education: perspectives and potential for progress. Studies in science education45(2), 169–204 (2009)

work page 2009

Show all 21 references

[9]

Computers and Education: Artificial Intelligence6, 100210 (2024)

Latif, E., Zhai, X.: Fine-tuning chatgpt for automatic scoring. Computers and Education: Artificial Intelligence6, 100210 (2024)

2024
[10]

School Science and Mathematics107(2), 52–60 (2007)

Lee, E., Brown, M.N., Luft, J.A., Roehrig, G.H.: Assessing beginning secondary science teachers’ pck: Pilot year results. School Science and Mathematics107(2), 52–60 (2007)

2007
[11]

Computers and Education: Artificial Intelligence 6, 100213 (2024)

Lee, G.G., Latif, E., Wu, X., Liu, N., Zhai, X.: Applying large language models and chain-of-thought for automatic scoring. Computers and Education: Artificial Intelligence 6, 100213 (2024)

2024
[12]

ETS Research Report Series 1984(1), i–55 (1984)

Messick, S.: The psychology of educational measurement. ETS Research Report Series 1984(1), i–55 (1984)

1984
[13]

Language Testing41(3), 606–626 (2024)

Neittaanmäki, R., Lamprianou, I.: All types of experience are equal, but some are more equal: The effect of different types of experience on rater severity and rater consistency. Language Testing41(3), 606–626 (2024)

2024
[14]

Research in Science Education 48, 549–573 (2018)

Park, S., Suh, J., Seo, K.: Development and validation of measures of secondary science teachers’ pck for teaching photosynthesis. Research in Science Education 48, 549–573 (2018)

2018
[15]

European Journal of Science and Mathematics Education 1(2), 84–105 (2013)

Sothayapetch, P., Lavonen, J., Juuti, K.: Primary school teachers’ interviews re- garding pedagogical content knowledge (pck) and general pedagogical knowledge (gpk). European Journal of Science and Mathematics Education 1(2), 84–105 (2013)

2013
[16]

In: Frontiers in Education

Wahlen, A., Kuhn, C., Zlatkin-Troitschanskaia, O., Gold, C., Zesch, T., Horbach, A.: Automated scoring of teachers’ pedagogical content knowledge–a comparison between human and machine scoring. In: Frontiers in Education. vol. 5, p. 149. Frontiers Media SA (2020)

2020
[17]

Technology, Knowledge and Learning pp

Wu, X.,Saraf, P.P.,Lee, G., Latif,E., Liu, N., Zhai,X.: Unveiling scoringprocesses: Dissecting the differences between llms and human graders in automatic scoring. Technology, Knowledge and Learning pp. 1–16 (2025) 14 Yang et al

2025
[18]

arXiv preprint arXiv:2501.06704 (2025)

Yang, J., Latif, E., He, Y., Zhai, X.: Fine-tuning chatgpt for automatic scoring of written scientific explanations in chinese. arXiv preprint arXiv:2501.06704 (2025)

2025 arXiv
[19]

Studies in Educational Evaluation67, 100916 (2020)

Zhai, X., Haudek, K.C., Stuhlsatz, M.A., Wilson, C.: Evaluation of construct- irrelevant variance yielded by machine and human scoring of a science teacher pck constructed response assessment. Studies in Educational Evaluation67, 100916 (2020)

2020
[20]

In: Fron- tiers in Education

Zhai, X., Haudek, K.C., Wilson, C., Stuhlsatz, M.: A framework of construct- irrelevant variance for contextualized constructed response assessment. In: Fron- tiers in Education. vol. 6, p. 751283. Frontiers Media SA (2021)

2021
[21]

Journal of Sci- ence Education and Technology30, 361–379 (2021)

Zhai, X., Shi, L., Nehm, R.H.: A meta-analysis of machine learning-based science assessments: Factors impacting machine-human score agreements. Journal of Sci- ence Education and Technology30, 361–379 (2021)

2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

arXiv preprint arXiv:2504.05736 (2025)

Cai, Y., Liang, K., Lee, S., Wang, Q., Wu, Y.: Rank-then-score: Enhancing large language models for automated essay scoring. arXiv preprint arXiv:2504.05736 (2025)

arXiv 2025

[2] [2]

Research in Science & Technological Education34(2), 237–251 (2016)

Canbazoglu Bilici, S., Guzey, S.S., Yamak, H.: Assessing pre-service science teach- ers’ technological pedagogical content knowledge (tpack) through observations and lesson plans. Research in Science & Technological Education34(2), 237–251 (2016)

work page 2016

[3] [3]

Teaching and teacher education34, 12–25 (2013)

Depaepe, F., Verschaffel, L., Kelchtermans, G.: Pedagogical content knowledge: A systematic review of the way in which the concept has pervaded mathematics educational research. Teaching and teacher education34, 12–25 (2013)

work page 2013

[4] [4]

Using GPT-4 to Augment Unbalanced Data for Automatic Scoring

Fang, L., Lee, G.G., Zhai, X.: Using gpt-4 to augment unbalanced data for auto- matic scoring. arXiv preprint arXiv:2310.18365 (2023)

work page Pith review arXiv 2023

[5] [5]

In: JSM Proceedings (2014)

Greenwood, M., Jesse, D.: Scoring and then analyzing or analyzing while scoring: An application of glmm to an education instrument development and analysis. In: JSM Proceedings (2014)

work page 2014

[6] [6]

Educational Measurement: Issues and Practice23(1), 17–27 (2004)

Haladyna, T.M., Downing, S.M.: Construct-irrelevant variance in high-stakes test- ing. Educational Measurement: Issues and Practice23(1), 17–27 (2004)

work page 2004

[7] [7]

Linguistic Research36 (2019)

Kim, H.J., Lee, J., You, H.J.: Analysis of rater effect in the evaluation of sec- ond language grammatical knowledge in the context of writing: Application of a generalized linear model. Linguistic Research36 (2019)

work page 2019

[8] [8]

Studies in science education45(2), 169–204 (2009)

Kind, V.: Pedagogical content knowledge in science education: perspectives and potential for progress. Studies in science education45(2), 169–204 (2009)

work page 2009