Pith. sign in

REVIEW 3 major objections 4 minor 166 references

Automated item evaluation: Predicting item acceptance and rejection using LLM-generated critiques

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A text-only classifier trained on historical accept/reject decisions can predict which standardized-test items will be permanently rejected, and adding LLM-generated critiques improves that prediction.

desk verdict Real data and a useful screening result, but the critique-augmentation claim is confounded by metadata included only in the critique condition. read the letter →

arxiv 2608.06609 v1 pith:AKSOETQK submitted 2026-08-06 cs.AI

Hotaka Maeda , Yikai Lu This is my paper · ORCID
classification cs.AI
keywords automateditemevaluationrejectionpredictionLLM-generatedcritiquesDeBERTaV3Qwen3educationalassessmenttextclassificationdifficulty
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a single text-based classifier, trained on the historical fates of 52,759 standardized-test items, can act as a near-comprehensive screen for item quality before manual review and field testing. The authors report that a fusion model—one DeBERTaV3-large encoder reading the raw item text and a second reading a two-sentence critique generated by Qwen3—outperforms either encoder alone and reaches Accuracy .75, F1 .64, and AUC .80 on held-out items. The result matters because automated item generation is producing far more items than content experts can review, and a cheap text-only screen could prioritize which items deserve expert time. Prediction is much stronger for math (F1 .73, AUC .86) than for English language arts (F1 .51, AUC .72), and the model misses most bias, sensitivity, fairness, and accessibility rejections, so the authors conclude human review remains essential there.

What carries the argument

The carrying mechanism is a two-encoder fusion. Raw item text is tokenized and passed through a DeBERTaV3-large encoder, a 304-million-parameter Transformer, fine-tuned with LoRA as a binary classifier; independently, Qwen3-0.6B, a compact language model run in thinking mode, writes a two-sentence critique of the same item, and that critique text is passed through a second DeBERTaV3-large encoder. The 1024-dimensional final-layer representations of the two encoders are concatenated, fed through a 128-dimensional intermediate layer, and ended in a sigmoid head trained with class-weighted cross-entropy loss. The critique prompt supplies the item text together with subject, grade level, and aligned content standards, while the raw-text baseline receives only item text.

What would settle it

Run the same fusion training with a control condition in which the critique channel receives only the subject, grade, and standards strings, or a boilerplate sentence carrying them, with no meaningful evaluation of item quality; if its AUC matches the full-critique fusion AUC on the same held-out test split, the paper's attribution of the gain to critique content is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that the final operational decision about an item, accepted or permanently rejected, can be predicted from the item's text with practically useful accuracy, and that feeding a separate encoder an LLM-generated critique of the item improves that prediction. On the held-out test set the fusion model achieved Accuracy .75, F1 .64, AUC .80, sensitivity .64, and specificity .81, consistently beating both the raw text-only model (F1 .59) and the critique-only model (F1 .55). The gap between subjects is large: math items reached F1 .73 and AUC .86, while ELA items reached F1 .51 and AUC .72. Adding critiques improved detection across nearly all rejection reasons, and lowering the decision threshold to .25 raised sensitivity to .88 for ELA and .91 for math, at the cost of specificity (.31 and .56). The model also assigned higher rejection probabilities to more difficult items, with the strongest correlation against proportion correct ($r=-.30$).

Load-bearing premise

The load-bearing assumption is that the extra accuracy from critiques comes from what the critique says about quality; since the critique generator was also given the subject, grade, and standards while the text-only model was not, the gain could instead come from those extra details.

Editorial extensions

If this is right

  • In automated item generation, where producing an item is cheap but reviewing it is expensive, setting the decision threshold at .25 catches 88 to 91 percent of eventual rejections while accepting many false positives.
  • The same trained model can screen both human-written and LLM-generated items, because the label it learns is the operational accept/reject decision rather than any single property like difficulty.
  • Math items are predictable enough (AUC .86) to be useful operationally, while ELA items are not; the authors tie this to ELA rejections often stemming from passages, which were excluded from the model input.
  • The model's failure on bias, sensitivity, fairness, and accessibility items (sensitivity .27, mostly ELA) means those concerns still require expert human review.
  • The approach depends on a large item bank with recorded rejection decisions, so test programs without such history cannot train the model from scratch.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The decisive control the paper does not report is a metadata-only version of the critique channel: because subject, grade, and standards were fed to the critique model but not the raw-text baseline, part or all of the fusion gain could come from those extra details rather than from the critique's reasoning.
  • The model's learned rejection probability could be repurposed as a weak item-difficulty signal; the paper's $r=-.30$ correlation with proportion correct shows that signal is real but too weak to replace dedicated difficulty prediction.
  • A testable extension would be feeding passage text and scoring rubrics into the same fusion, which would directly test whether the ELA gap and the bias-detection failure shrink.
  • The threshold-shift result suggests a cheap triage workflow: run the fused classifier at a low threshold to send a small, high-yield subset of items to human experts, and use the default threshold for revision warnings in traditional item development.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes automated item evaluation (AIE) by fine-tuning DeBERTaV3-large encoders on raw item text and on Qwen3-generated critiques, plus a fusion of both, to predict permanent rejection of standardized test items using 52,759 ELA and mathematics items with a 34% rejection rate. The best fusion model is reported with Accuracy=.75, F1=.64, AUC=.80, with math predictions considerably stronger (F1=.73, AUC=.86) than ELA (F1=.51, AUC=.72). Additional analyses examine threshold tuning, sensitivity by rejection reason, correlations with item difficulty and discrimination, and critique sentiment. The authors conclude that text-based AIE is feasible in some areas, that LLM-generated critiques improve prediction across most rejection reasons, and that human review remains necessary for fairness-related flags.

Significance. If the claims hold, the paper provides a practical, text-based screening tool for large-scale assessment programs and for automated item generation pipelines, potentially reducing manual review and field-testing costs. The strength of the paper is its use of real operational acceptance/rejection labels in a large dataset, a held-out split with stimulus-group constraints, and transparent prompts in the appendices; the threshold analysis quantifies a practically useful sensitivity/specificity trade-off. The main empirical finding, however, currently rests on a confounded comparison and on point estimates without uncertainty quantification, so the significance of the contribution depends on a targeted control experiment and more careful statistical reporting.

major comments (3)
  1. [§3.2.2, §3.2.3, Appendix C] The claim that 'incorporating item critiques alongside raw item text improved performance' is confounded. The raw text-only model receives only item text (§3.2.2), whereas the critique prompt in Appendix C supplies the item's subject, grade, and aligned Common Core standards, and the example critiques in Appendix D restate this metadata. Since §3.1 excludes metadata from the item text used for modeling, the fusion model's gain over the raw-text baseline could be due entirely to the extra metadata rather than to the reasoning content of the critiques. A metadata-only control condition—for example, a fusion or text model whose input is raw item text plus subject/grade/standards as plain text, without any critique—is required before the abstract, Section 4, or Section 5 can attribute the improvement to LLM-generated critiques.
  2. [§4, Table 1] All model comparisons are point estimates from a single split, with no confidence intervals, bootstrap, or repeated-seed variation reported. The improvements over the raw text-only model are modest (overall F1 .59 vs .64, AUC .76 vs .80; math F1 .68 vs .73, AUC .80 vs .86), and without uncertainty measures the claim that the fusion model 'consistently outperformed' both alternatives is not established. Please report bootstrap confidence intervals for the metrics or variation across multiple fine-tuning seeds, at least for the key text-only/fusion comparison.
  3. [§4.2, Figure 3] The rejection-reason sensitivity analyses rest on only 19% of rejected test items having usable rejection reason data, and no sample sizes or confidence intervals are given for the per-category sensitivities. The statement that critiques 'improved detection across nearly all rejection categories' is therefore not quantitatively supported, especially for small categories such as bias (sensitivity .27). Add the number of items per rejection-reason category and uncertainty estimates, or at least report exact counts and avoid categorical improvement claims without them.
minor comments (4)
  1. [§3.1, Appendix A] There are small typos: 'exluded' should be 'excluded' in §3.1, and 'too difficulty' should be 'too difficult' in the Appendix A definition of psychometric rejection reasons.
  2. [§3.2.2, §3.2.3] The raw text-only model uses LoRA rank=32, while the critique-only model uses LoRA rank=50; since the two single-encoder models are compared directly, this hyperparameter difference is an additional uncontrolled variable. State whether the conclusions are robust to matching the rank across the two encoders.
  3. [§3.2.1] The zero-shot baseline prompt includes subject, grade, and standards information, whereas the raw text-only fine-tuned model does not; this makes the zero-shot comparison difficult to interpret, although it does not affect the main text-only versus fusion comparison.
  4. [§4.4] The sentiment correlations are very small in magnitude (r = −.05 to −.18) and some are significant only because of the large sample; the sentence in Section 4.4 that the results are 'consistent with the interpretation that the critiques contributed information beyond the item text itself' should be softened or supported with a direct ablation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central result is a held-out supervised-learning comparison, and the self-citations are not load-bearing.

full rationale

The paper's derivation chain is a standard supervised-learning pipeline: item text and Qwen3-generated critiques are featurized by DeBERTa encoders, fine-tuned on an 80/10/10 split, and evaluated on held-out test data. No fitted constant is recycled as a prediction, and the target label is not included in the critique prompt in Appendix C. The critique condition does include subject, grade, and aligned Common Core standards in the prompt, while the raw-text baseline excludes metadata as stated in Section 3.1, so part of the fusion model's gain could reflect metadata rather than critique reasoning. That is a potential confound requiring a metadata-only control condition, but it is not circularity: the held-out labels are independent of both feature sets, and the comparison is empirical rather than defined into existence. The self-citations (Maeda and Lu 2025, 2026; Maeda 2025) appear only in literature review and do not supply any premise on which the prediction claim depends. The paper also explicitly acknowledges limitations, including difficulty detecting bias and fairness concerns and rejection reasons that are inherently unpredictable from item text, which further confirms that the central result is not forced by the paper's own definitions or fitted inputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no fitted constants in a derivation sense; the listed axioms are the domain assumptions that hold together the empirical claim. The most fragile is the metadata confound, captured in red flags, where the critique condition receives information the raw-text baseline does not.

assumptions (4)
  • domain assumption Operational item status (accepted vs permanently rejected) is a reliable binary proxy for item quality.
    The entire approach trains on this label; if rejection reflects bank logistics, exposure, or passage-set decisions rather than text-detectable quality, the learned mapping is not a general item-quality evaluator. The authors acknowledge several non-content rejection reasons in Appendix A.
  • domain assumption Qwen3-generated critiques carry valid item-quality signal beyond the raw text.
    Central to the fusion model; critiques are generated by a 0.6B model from the same text plus metadata, and no independent validation of critique accuracy is provided.
  • domain assumption DeBERTaV3-large fine-tuned representations transfer from pretraining corpora to educational item text.
    Standard transfer-learning assumption; the paper provides no evidence on domain shift between pretraining data and assessment item text.
  • domain assumption Rejection reason labels produced by Claude Sonnet 4.6 plus manual fixes are accurate enough for sensitivity-by-reason analysis.
    No inter-rater reliability or validation against human expert labels is reported in Section 3.3.1, and only 19% of rejected items had usable comments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automated item evaluation: Predicting item acceptance and rejection using LLM-generated critiques." pith.science (2026). https://pith.science/paper/AKSOETQK

@misc{pith2026260806609,
  author       = {Pith},
  title        = {Pith review of: Automated item evaluation: Predicting item acceptance and rejection using LLM-generated critiques},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AKSOETQK}},
  note         = {Machine review of arXiv:2608.06609}
}
read the original abstract

Automated item evaluation (AIE) refers to the use of computational methods to assess item quality without requiring manual expert review or field testing of the items under evaluation. We aimed to build a near-comprehensive AIE model by predicting item acceptance and rejection from item text using historical rejection data from a large-scale standardized testing program. The dataset contained 52,759 English language arts (ELA) and mathematics items with 34% permanently rejected from future operational use. Rejection reasons included poor psychometric properties, content issues, bias and sensitivity concerns, and non-content issues. We fine-tuned a DeBERTaV3-large classifier on raw item text, a second DeBERTa classifier on Qwen3-generated item critiques, and a fusion model combining representations from both. The fusion model achieved the strongest overall performance (Accuracy = .75, F1 = .64, AUC = .80, Sensitivity = .64, Specificity = .81). Prediction for math (F1 = .73, AUC = .86) was considerably more accurate than ELA (F1 = .51, AUC = .72). Lowering the decision threshold from .5 to .25 raised average sensitivity for ELA and math to .88 and .91, while reducing specificity to .31 and .56, respectively, which may be preferable in automated item generation contexts where generating items is cheaper than evaluating them. Incorporating item critiques alongside raw item text improved performance across most rejection reasons. The model assigned higher rejection probabilities to more difficult items. However, the fusion model struggled to identify items flagged for bias, sensitivity, fairness, or accessibility, especially for ELA. These findings suggest that text-based AIE is feasible in some areas and may offer a practical tool for reducing the burden of manual review and field testing, while also underscoring the importance of human review for items with fairness concerns.

Figures

Figures reproduced from arXiv: 2608.06609 by the authors.

Figure 2
Figure 2. Cutoff threshold analysis for fusion model (raw text + critique) [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

166 extracted references · 54 canonical work pages

  1. [1]

    International Journal of Research in Marketing , year =

    Hartmann, Jochen and Heitmann, Mark and Siebert, Christian and Schamp, Christina , title =. International Journal of Research in Marketing , year =

  2. [2]

    Proceedings of the 2024 AAAI Conference on Artificial Intelligence , series =

    Current Evaluation Methods are a Bottleneck in Automatic Question Generation , author =. Proceedings of the 2024 AAAI Conference on Artificial Intelligence , series =. 2024 , publisher =

    2024

  3. [3]

    Educational Measurement: Issues and Practice , volume =

    Gorgun, Guher and Bulut, Okan , title =. Educational Measurement: Issues and Practice , volume =. doi:https://doi.org/10.1111/emip.12663 , url =. https://onlinelibrary.wiley.com/doi/pdf/10.1111/emip.12663 , abstract =

    doi:10.1111/emip.12663

  4. [4]

    and Pashley, Peter J

    van der Linden, Wim J. and Pashley, Peter J. Item Selection and Ability Estimation in Adaptive Testing. Computerized Adaptive Testing: Theory and Practice. 2000. doi:10.1007/0-306-47531-6_1

    work page doi:10.1007/0-306-47531-6_1

  5. [5]

    2020 , doi =

    Peng, Fang , title =. 2020 , doi =

    2020

  6. [6]

    Incompressible Knowledge Probes: Estimating Black-Box

    Bojie Li , year=. Incompressible Knowledge Probes: Estimating Black-Box. arXiv preprint arXiv:2604.24827 , archivePrefix=

    arXiv

  7. [7]

    2025 , eprint=

    Prompting Strategies for Language Model-Based Item Generation in K-12 Education: Bridging the Gap Between Small and Large Language Models , author=. 2025 , eprint=

    2025

  8. [8]

    Journal of Educational Data Mining , volume=

    Towards design-loop adaptivity: identifying items for revision , author=. Journal of Educational Data Mining , volume=

Show all 166 references
  1. [9]

    The Routledge International Handbook of Automated Essay Evaluation , pages=

    Automated short-response scoring for automated item generation in science assessments , author=. The Routledge International Handbook of Automated Essay Evaluation , pages=. 2024 , publisher=

    2024

  2. [10]

    Using generated rubrics to provide a window into item evaluation with multi-agent

    Wang, Yu and Gopalakrishnan, Madhumitha and Bergner, Yoav , booktitle=. Using generated rubrics to provide a window into item evaluation with multi-agent. 2025 , organization=

    2025

  3. [11]

    Automatic Item Generation, Evaluation, and Scale Construction of Non-Cognitive Measures with Generative Language Models , booktitle =

    Yu, Martin C and Burke, Maura I , editor =. Automatic Item Generation, Evaluation, and Scale Construction of Non-Cognitive Measures with Generative Language Models , booktitle =. 2026 , month =. doi:10.1093/9780197807309.003.0010 , url =

    2026 doi

  4. [12]

    Electronics , VOLUME =

    Prentzas, Jim and Binopoulou, Ariadni , TITLE =. Electronics , VOLUME =. 2025 , NUMBER =

    2025

  5. [13]

    and Monday, Onoja Abah , title =

    Oluoke, Omopekunola Moses and Gorbacheva, Anna M. and Monday, Onoja Abah , title =. International Journal of Evaluation and Research in Education , year =

  6. [14]

    International Journal of Assessment Tools in Education , volume=

    A review of automatic item generation techniques leveraging large language models , author=. International Journal of Assessment Tools in Education , volume=. 2025 , publisher=

    2025

  7. [15]

    Advances in Health Sciences Education , volume=

    Feasibility assurance: a review of automatic item generation in medical assessment , author=. Advances in Health Sciences Education , volume=. 2022 , publisher=

    2022

  8. [17]

    arXiv preprint arXiv:2410.21276 , archivePrefix=

    OpenAI , year=. arXiv preprint arXiv:2410.21276 , archivePrefix=

    arXiv

  9. [18]

    Youden, W. J. , title =. Cancer , year =. doi:10.1002/1097-0142(1950)3:1<32::aid-cncr2820030106>3.0.co;2-3 , pmid =

    1950 doi

  10. [19]

    2023 , journal=

    Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes , author=. 2023 , journal=

    2023

  11. [20]

    Two-Stage Reasoning-Infused Learning: Improving Classification with

    Mads Henrichsen and Rasmus Krebs , year=. Two-Stage Reasoning-Infused Learning: Improving Classification with. arXiv preprint arXiv:2507.00214 , archivePrefix=

    arXiv

  12. [21]

    Reasoning and Sampling-Augmented MCQ Difficulty Prediction via

    Wanyong Feng and Peter Tran and Stephen Sireci and Andrew Lan , year=. Reasoning and Sampling-Augmented MCQ Difficulty Prediction via. arXiv preprint arXiv:2503.08551 , archivePrefix=

    arXiv

  13. [22]

    2025 , howpublished =

    Anthropic , title =. 2025 , howpublished =

    2025

  14. [23]

    2026 , journal=

    Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models , author=. 2026 , journal=

    2026

  15. [24]

    2024 , journal=

    On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes , author=. 2024 , journal=

    2024

  16. [25]

    2025 , journal=

    Small Language Models: Survey, Measurements, and Insights , author=. 2025 , journal=

    2025

  17. [26]

    arXiv preprint arXiv:2605.18562 , year=

    Estimating Item Difficulty with Large Language Models as Experts , author=. arXiv preprint arXiv:2605.18562 , year=

  18. [27]

    Review of educational research , volume=

    Developing, analyzing, and using distractors for multiple-choice tests in education: A comprehensive review , author=. Review of educational research , volume=. 2017 , publisher=

    2017

  19. [28]

    educational measurement: issues and practice , volume=

    The longer-term impact of COVID-19 on K--12 student learning and assessment , author=. educational measurement: issues and practice , volume=. 2020 , publisher=

    2020

  20. [29]

    2025 , journal=

    Text-Based Approaches to Item Alignment to Content Standards in Large-Scale Reading & Writing Tests , author=. 2025 , journal=

    2025

  21. [30]

    Humanities and Social Sciences Communications , volume=

    Rater variability and reliability of constructed response questions in New York state high-stakes tests of English language arts and mathematics: implications for educational assessment policy , author=. Humanities and Social Sciences Communications , volume=. 2023 , publisher=

    2023

  22. [31]

    Journal of Development and Social Sciences , volume=

    Item Discrimination of Intelligence Test Items: A Quality Assurance Test for an Assessment Tool , author=. Journal of Development and Social Sciences , volume=

  23. [32]

    Haladyna and Steven M

    Thomas M. Haladyna and Steven M. Downing and Michael C. Rodriguez , title =. Applied Measurement in Education , volume =. 2002 , doi =

    2002

  24. [33]

    Haladyna and Steven M

    Thomas M. Haladyna and Steven M. Downing , title =. Applied Measurement in Education , volume =. 1989 , doi =

    1989

  25. [34]

    Educational and Psychological Measurement , year =

    Maeda, Hotaka and Lu, Yikai , title =. Educational and Psychological Measurement , year =. doi:10.1177/00131644261460779 , url =

    doi

  26. [35]

    2026 , journal=

    Efficient Detection of Bad Benchmark Items with Novel Scalability Coefficients , author=. 2026 , journal=

    2026

  27. [36]

    Predicting Item Survival for Multiple Choice Questions in a High-Stakes Medical Exam

    Yaneva, Victoria and Ha, Le An and Baldwin, Peter and Mee, Janet. Predicting Item Survival for Multiple Choice Questions in a High-Stakes Medical Exam. Proceedings of the Twelfth Language Resources and Evaluation Conference. 2020

    2020

  28. [37]

    2025 , journal=

    The Impact of Item-Writing Flaws on Difficulty and Discrimination in Item Response Theory , author=. 2025 , journal=

    2025

  29. [38]

    Advances in Health Sciences Education , volume=

    A suggestive approach for assessing item quality, usability and validity of Automatic Item Generation , author=. Advances in Health Sciences Education , volume=

  30. [39]

    arXiv preprint arXiv:2504.06465 , year =

    Ye Ma , title =. arXiv preprint arXiv:2504.06465 , year =

    arXiv

  31. [40]

    2025 , journal=

    Qwen3 Technical Report , author=. 2025 , journal=

    2025

  32. [41]

    Journal of Educational Measurement , year=

    Using item parameter predictions for reducing calibration sample requirements—A case study based on a high-stakes admission test , author=. Journal of Educational Measurement , year=

  33. [42]

    2023 , journal=

    Kosmos-2: Grounding Multimodal Large Language Models to the World , author=. 2023 , journal=

    2023

  34. [43]

    2025 , journal=

    OneLLM: One Framework to Align All Modalities with Language , author=. 2025 , journal=

    2025

  35. [44]

    Information Fusion , volume=

    A comprehensive survey on regularization strategies in machine learning , author=. Information Fusion , volume=. 2022 , publisher=

    2022

  36. [45]

    2025 , journal=

    Multimodal Alignment and Fusion: A Survey , author=. 2025 , journal=

    2025

  37. [46]

    Ward, Alex , title =

  38. [47]

    Research and practice in technology enhanced learning , volume=

    Controlling item difficulty for automatic vocabulary question generation , author=. Research and practice in technology enhanced learning , volume=. 2017 , publisher=

    2017

  39. [48]

    International conference on artificial intelligence in education , pages=

    A quantitative study of NLP approaches to question difficulty estimation , author=. International conference on artificial intelligence in education , pages=. 2023 , organization=

    2023

  40. [49]

    Proceedings of the 10th EAI International Conference on Mobile Multimedia Communications , pages=

    Masked loss residual convolutional neural network for facial keypoint detection , author=. Proceedings of the 10th EAI International Conference on Mobile Multimedia Communications , pages=

  41. [50]

    2026 , booktitle=

    Liver, vessel, and tumor segmentation from partially labeled CT and multi-label masked learning , author=. 2026 , booktitle=

    2026

  42. [51]

    Statistical theories of mental test scores , year=

    Some latent trait models and their use in inferring an examinee's ability , author=. Statistical theories of mental test scores , year=

  43. [52]

    ETS Research Report Series , volume=

    A generalized partial credit model: Application of an EM algorithm , author=. ETS Research Report Series , volume=. 1992 , publisher=

    1992

  44. [53]

    IEEE transactions on pattern analysis and machine intelligence , volume=

    Multimodal machine learning: A survey and taxonomy , author=. IEEE transactions on pattern analysis and machine intelligence , volume=. 2018 , publisher=

    2018

  45. [54]

    , author=

    Multimodal deep learning. , author=. ICML , volume=

  46. [55]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Smart: Simulated students aligned with item response theory for question difficulty prediction , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  47. [56]

    Ninth international conference on document analysis and recognition (ICDAR 2007) , volume=

    An overview of the Tesseract OCR engine , author=. Ninth international conference on document analysis and recognition (ICDAR 2007) , volume=. 2007 , organization=

    2007

  48. [57]

    The Cambridge handbook of multimedia learning , volume=

    An integrated model of text and picture comprehension , author=. The Cambridge handbook of multimedia learning , volume=. 2005 , publisher=

    2005

  49. [58]

    Psychology of learning and motivation , volume=

    Multimedia learning , author=. Psychology of learning and motivation , volume=. 2002 , publisher=

    2002

  50. [59]

    1990 , publisher=

    Mental representations: A dual coding approach , author=. 1990 , publisher=

    1990

  51. [60]

    Proceedings of the tenth international conference on learning analytics & knowledge , pages=

    R2DE: a NLP approach to estimating IRT parameters of newly generated questions , author=. Proceedings of the tenth international conference on learning analytics & knowledge , pages=

  52. [61]

    arXiv preprint arXiv:2304.07193 , year=

    Dinov2: Learning robust visual features without supervision , author=. arXiv preprint arXiv:2304.07193 , year=

    arXiv

  53. [62]

    Artificial Intelligence Review , volume=

    A survey of the vision transformers and their CNN-transformer based variants , author=. Artificial Intelligence Review , volume=. 2023 , publisher=

    2023

  54. [63]

    International conference on machine learning , pages=

    Learning transferable visual models from natural language supervision , author=. International conference on machine learning , pages=. 2021 , organization=

    2021

  55. [64]

    International conference on machine learning , pages=

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models , author=. International conference on machine learning , pages=. 2023 , organization=

    2023

  56. [65]

    International conference on machine learning , pages=

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation , author=. International conference on machine learning , pages=. 2022 , organization=

    2022

  57. [66]

    European Conference on Computer Vision , pages=

    Ocr-free document understanding transformer , author=. European Conference on Computer Vision , pages=. 2022 , organization=

    2022

  58. [67]

    arXiv preprint arXiv:1907.11692 , year=

    Roberta: A robustly optimized bert pretraining approach , author=. arXiv preprint arXiv:1907.11692 , year=

    1907 arXiv

  59. [68]

    Computers & Education , volume=

    Item difficulty estimation: An auspicious collaboration between data and judgment , author=. Computers & Education , volume=. 2012 , publisher=

    2012

  60. [69]

    Information Processing & Management , volume=

    Automated estimation of item difficulty for multiple-choice tests: An application of word embedding techniques , author=. Information Processing & Management , volume=. 2018 , publisher=

    2018

  61. [70]

    Educational and Psychological Measurement , volume=

    Item difficulty modeling using fine-tuned small and large language models , author=. Educational and Psychological Measurement , volume=. 2025 , publisher=

    2025

  62. [71]

    CBE—Life Sciences Education , volume=

    Multiple-choice exams: an obstacle for higher-level thinking in introductory science classes , author=. CBE—Life Sciences Education , volume=. 2012 , publisher=

    2012

  63. [72]

    Journal of Microbiology and Biology Education , volume=

    Question format is the best predictor of item discrimination: a multivariable analysis , author=. Journal of Microbiology and Biology Education , volume=. 2025 , publisher=

    2025

  64. [73]

    The Curriculum Journal , volume=

    Predicting item difficulty of science national curriculum tests: The case of key stage 2 assessments , author=. The Curriculum Journal , volume=. 2017 , publisher=

    2017

  65. [74]

    Mathematics , VOLUME =

    Yi, Xifan and Sun, Jianing and Wu, Xiaopeng , TITLE =. Mathematics , VOLUME =. 2024 , NUMBER =

    2024

  66. [75]

    arXiv preprint arXiv:2509.23486 , year=

    Text-Based Approaches to Item Difficulty Modeling in Large-Scale Assessments: A Systematic Review , author=. arXiv preprint arXiv:2509.23486 , year=

  67. [76]

    Information Sciences , volume=

    Attention pooling-based convolutional neural network for sentence modelling , author=. Information Sciences , volume=. 2016 , publisher=

    2016

  68. [77]

    arXiv preprint arXiv:2309.15698 , year=

    Deep model fusion: A survey , author=. arXiv preprint arXiv:2309.15698 , year=

    arXiv

  69. [78]

    Neural computation , volume=

    A survey on deep learning for multimodal data fusion , author=. Neural computation , volume=. 2020 , publisher=

    2020

  70. [79]

    arXiv preprint arXiv:2508.10104 , year=

    Dinov3 , author=. arXiv preprint arXiv:2508.10104 , year=

    arXiv

  71. [80]

    , author=

    Lora: Low-rank adaptation of large language models. , author=. ICLR , volume=

  72. [81]

    ACM Computing Surveys , volume=

    A survey on recent approaches to question difficulty estimation from text , author=. ACM Computing Surveys , volume=. 2023 , publisher=

    2023

  73. [82]

    International Journal of Artificial Intelligence in Education , volume=

    Text-based question difficulty prediction: A systematic review of automatic approaches , author=. International Journal of Artificial Intelligence in Education , volume=. 2024 , publisher=

    2024

  74. [83]

    Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Full Papers , pages=

    Leveraging Fine-tuned Large Language Models in Item Parameter Prediction , author=. Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Full Papers , pages=

  75. [84]

    Discover Computing , volume=

    Multimodal sentiment analysis using image and text fusion for emotion detection , author=. Discover Computing , volume=. 2025 , publisher=

    2025

  76. [85]

    Science Progress , volume =

    Szu-Yin Lin and Yen-Chiu Chen and Yu-Han Chang and Shih-Hsin Lo and Kuo-Ming Chao , title =. Science Progress , volume =. 2024 , doi =

    2024

  77. [86]

    2015 , issn =

    What makes an item more difficult? Effects of modality and type of visual information in a computer-based assessment of scientific inquiry abilities , journal =. 2015 , issn =. doi:10.1016/j.compedu.2015.01.007 , url =

    2015 doi

  78. [87]

    ETS Research Report Series , volume =

    Zwick, Rebecca , title =. ETS Research Report Series , volume =. doi:10.1002/j.2333-8504.2012.tb02290.x , url =. https://onlinelibrary.wiley.com/doi/pdf/10.1002/j.2333-8504.2012.tb02290.x , year =

    2012

  79. [88]

    , title =

    Birnbaum, A. , title =. Statistical theories of mental test scores , editor =

  80. [89]

    International Journal of Artificial Intelligence in Education , pages=

    Text-based question difficulty prediction: A systematic review of automatic approaches , author=. International Journal of Artificial Intelligence in Education , pages=. 2023 , publisher=

    2023

  81. [90]

    Educational and Psychological Measurement , year =

    Qizhou Duan and Ying Cheng , title =. Educational and Psychological Measurement , year =. doi:10.1177/00131644241280400 , URL =

    doi

  82. [91]

    2020 , journal=

    Language Models are Few-Shot Learners , author=. 2020 , journal=

    2020

  83. [92]

    Gierl and Hollis Lai and Simon R

    Mark J. Gierl and Hollis Lai and Simon R. Turner , doi =. Medical Education , title =

  84. [93]

    Psychological Test and Assessment Modeling , title =

    Jiao, Hong and He, Qiwei and Yao, Lihua , year =. Psychological Test and Assessment Modeling , title =

  85. [94]

    2024 , journal=

    The Rise of Artificial Intelligence in Educational Measurement: Opportunities and Ethical Challenges , author=. 2024 , journal=

    2024

  86. [95]

    Advancing natural language processing in educational assessment , pages=

    Psychometric considerations when using deep learning for automated scoring , author=. Advancing natural language processing in educational assessment , pages=. 2023 , publisher=

    2023

  87. [96]

    and Yaneva, Victoria and Lottridge, Susan and von Davier, Matthias and Harris, Deborah J

    Hao, Jiangang and von Davier, Alina A. and Yaneva, Victoria and Lottridge, Susan and von Davier, Matthias and Harris, Deborah J. , title =. Educational Measurement: Issues and Practice , volume =. doi:https://doi.org/10.1111/emip.12602 , url =. https://onlinelibrary.wiley.com/...

    doi

  88. [97]

    Applied Psychological Measurement , title =

    Gregory Camilli , doi =. Applied Psychological Measurement , title =

  89. [98]

    Frontiers in Education , title =

    Okan Bulut and Youngsuk Suh , doi =. Frontiers in Education , title =

  90. [99]

    British Journal of Psychology , title =

    William Brown , doi =. British Journal of Psychology , title =

  91. [100]

    Spearman , doi =

    C. Spearman , doi =. British Journal of Psychology, 1904‐1920 , title =

    1904

  92. [101]

    2021 , journal=

    DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing , author=. 2021 , journal=

    2021

  93. [102]

    Applied Psychological Measurement , title =

    Eiji Muraki , doi =. Applied Psychological Measurement , title =

  94. [103]

    Journal of the American Statistical Association , title =

    Nathan Mantel , doi =. Journal of the American Statistical Association , title =

  95. [104]

    2017 , journal=

    Attention Is All You Need , author=. 2017 , journal=

    2017

  96. [105]

    Holland and Dorothy T

    Paul W. Holland and Dorothy T. Thayer , doi =. Differential item performance and the mantel-haenszel procedure , year =. Test Validity , publisher =

  97. [106]

    and Wainer, Howard , booktitle =

    Zieky, Michael and Holland, Paul W. and Wainer, Howard , booktitle =. Practical Questions in the Use of DIF Statistics in Test Development , year =

  98. [107]

    Cognitive Computation , title =

    Vikas Hassija and Vinay Chamola and Atmesh Mahapatra and Abhinandan Singal and Divyansh Goel and Kaizhu Huang and Simone Scardapane and Indro Spinelli and Mufti Mahmud and Amir Hussain , doi =. Cognitive Computation , title =

  99. [108]

    arXiv preprint arXiv:2405.06064 , archivePrefix=

    Alexandra Zytek and Sara Pidò and Kalyan Veeramachaneni , year=. arXiv preprint arXiv:2405.06064 , archivePrefix=

    arXiv

  100. [109]

    2009 , doi=

    Differential item functioning , author=. 2009 , doi=

    2009

  101. [110]

    2012 , publisher=

    Differential item functioning , author=. 2012 , publisher=

    2012

  102. [111]

    Perspectives on differential item functioning methodology , volume =

    William H Angoff , journal =. Perspectives on differential item functioning methodology , volume =

  103. [112]

    Thayer and John Mazzeo , doi =

    Rebecca Zwick and Dorothy T. Thayer and John Mazzeo , doi =. Applied Measurement in Education , title =

  104. [113]

    Thayer , doi =

    Rebecca Zwick and Dorothy T. Thayer , doi =. Journal of Educational and Behavioral Statistics , title =

  105. [114]

    Gandomi and Fang Chen and Andreas Holzinger , doi =

    Jianlong Zhou and Amir H. Gandomi and Fang Chen and Andreas Holzinger , doi =. Electronics (Switzerland) , title =

  106. [115]

    ACM Transactions on Intelligent Systems and Technology , title =

    Haiyan Zhao and Hanjie Chen and Fan Yang and Ninghao Liu and Huiqi Deng and Hengyi Cai and Shuaiqiang Wang and Dawei Yin and Mengnan Du , doi =. ACM Transactions on Intelligent Systems and Technology , title =

  107. [116]

    Bennett and Paul Deane and Peter W

    Mo Zhang and Randy E. Bennett and Paul Deane and Peter W. van Rijn , doi =. Educational Measurement: Issues and Practice , title =

  108. [117]

    Axiomatic attribution for deep networks , volume =

    Mukund Sundararajan and Ankur Taly and Qiqi Yan , booktitle =. Axiomatic attribution for deep networks , volume =

  109. [118]

    Why Should I Trust You?

    "Why Should I Trust You?": Explaining the Predictions of Any Classifier , author=. 2016 , journal=

    2016

  110. [119]

    Axiomatic attribution for deep networks , urldate =

    Sundararajan, Mukund and Taly, Ankur and Yan, Qiqi , month = aug, year =. Axiomatic attribution for deep networks , urldate =. Proceedings of the 34th

  111. [120]

    CEUR Workshop Proceedings , title =

    Craig Pirie and Nirmalie Wiratunga and Anjana Wijekoon and Carlos Francisco Moreno-Garcia , issn =. CEUR Workshop Proceedings , title =

  112. [121]

    Advances in Neural Information Processing Systems , title =

    Adam Paszke and Sam Gross and Francisco Massa and Adam Lerer and James Bradbury and Gregory Chanan and Trevor Killeen and Zeming Lin and Natalia Gimelshein and Luca Antiga and Alban Desmaison and Andreas Köpf and Edward Yang and Zach DeVito and Martin Raison and Alykhan Tejani...

  113. [122]

    Michaelides , issn =

    Michalis P. Michaelides , issn =. Practical Assessment, Research and Evaluation , title =

  114. [123]

    Journal of the National Cancer Institute , title =

    Nathan Mantel and William Haenszel , doi =. Journal of the National Cancer Institute , title =

  115. [124]

    Lundberg and Su In Lee , issn =

    Scott M. Lundberg and Su In Lee , issn =. Advances in Neural Information Processing Systems , title =

  116. [125]

    2020 , journal=

    Captum: A unified and generic model interpretability library for PyTorch , author=. 2020 , journal=

    2020

  117. [126]

    Dorans and Paul W

    Neil J. Dorans and Paul W. Holland , doi =. ETS Research Report Series , title =

  118. [127]

    Jacob Devlin and Ming Wei Chang and Kenton Lee and Kristina Toutanova , booktitle =

  119. [128]

    LaFlair and Kevin Yancey and Alina A

    Jill Burstein and Geoffrey T. LaFlair and Kevin Yancey and Alina A. von Davier and Ravit Dotan , year=. Responsible. arXiv preprint arXiv:2409.07476 , archivePrefix=

    arXiv

  120. [129]

    2016 , journal=

    Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings , author=. 2016 , journal=

    2016

  121. [130]

    Educational and Psychological Measurement , volume =

    Hotaka Maeda , title =. Educational and Psychological Measurement , volume =. 2025 , doi =

    2025

  122. [131]

    ACM Computing Surveys , title =

    Ninareh Mehrabi and Fred Morstatter and Nripsuta Saxena and Kristina Lerman and Aram Galstyan , doi =. ACM Computing Surveys , title =

  123. [132]

    IEEE Transactions on Neural Networks and Learning Systems , author =

    A. IEEE Transactions on Neural Networks and Learning Systems , author =. 2021 , pages =. doi:10.1109/TNNLS.2020.3027314 , number =

    2021

  124. [133]

    FAIRFIL: CONTRASTIVE NEURAL DEBIASING METHOD FOR PRETRAINED TEXT ENCODERS , year =

    Pengyu Cheng and Weituo Hao and Siyang Yuan and Shijing Si and Lawrence Carin , booktitle =. FAIRFIL: CONTRASTIVE NEURAL DEBIASING METHOD FOR PRETRAINED TEXT ENCODERS , year =

  125. [134]

    Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021 , title =

    Anne Lauscher and Tobias Lüken and Goran Glavaš , doi =. Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021 , title =

    2021

  126. [135]

    2020 , journal=

    Towards Debiasing Sentence Representations , author=. 2020 , journal=

    2020

  127. [136]

    Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages=

    He is very intelligent, she is very beautiful? on mitigating social biases in language modelling and generation , author=. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages=. doi:10.18653/v1/2021.findings-acl.397 , year=

    2021 doi

  128. [137]

    arXiv preprint arXiv:2004.09456 , year=

    StereoSet: Measuring stereotypical bias in pretrained language models , author=. arXiv preprint arXiv:2004.09456 , year=

    2004 arXiv

  129. [138]

    Science , volume=

    Semantics derived automatically from language corpora contain human-like biases , author=. Science , volume=. 2017 , publisher=

    2017

  130. [139]

    2024 , journal=

    Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference , author=. 2024 , journal=

    2024

  131. [140]

    2024 , journal=

    Differential Transformer , author=. 2024 , journal=

    2024

  132. [141]

    arXiv preprint arXiv:2308.10149 , year=

    A survey on fairness in large language models , author=. arXiv preprint arXiv:2308.10149 , year=

    arXiv

  133. [142]

    Interpreting artificial intelligence models: A systematic review on the application of

    Vimbi, Vahe and Shaffi, Naseer and Mahmud, Mobashsher , journal=. Interpreting artificial intelligence models: A systematic review on the application of. 2024 , doi=

    2024

  134. [143]

    2021 , url=

    Transformers Interpret 0.5.2 , author=. 2021 , url=

    2021

  135. [144]

    arXiv preprint arXiv:2110.15733 , year=

    Detecting gender bias in transformer-based models: A case study on bert , author=. arXiv preprint arXiv:2110.15733 , year=

    arXiv

  136. [145]

    WIREs Data Mining and Knowledge Discovery , author =

    Explainable artificial intelligence: an analytical review , volume =. WIREs Data Mining and Knowledge Discovery , author =. 2021 , pages =. doi:10.1002/widm.1424 , language =

    2021 doi

  137. [146]

    and Bau, David and Yuan, Ben Z

    Gilpin, Leilani H. and Bau, David and Yuan, Ben Z. and Bajwa, Ayesha and Specter, Michael and Kagal, Lalana , month = oct, year =. Explaining. 2018. doi:10.1109/DSAA.2018.00018 , urldate =

    2018

  138. [147]

    Values of

    Owen, Guilliermo , editor =. Values of. Mathematical. 1977 , pages =. doi:10.1007/978-3-642-45494-3_7 , language =

    1977 doi

  139. [148]

    Nature Machine Intelligence , author =

    Improving performance of deep learning models with axiomatic attribution priors and expected gradients , volume =. Nature Machine Intelligence , author =. 2021 , note =. doi:10.1038/s42256-021-00343-w , language =

    2021 doi

  140. [149]

    Applied Stochastic Models in Business and Industry , author =

    Analysis of regression in game theory approach , volume =. Applied Stochastic Models in Business and Industry , author =. 2001 , keywords =. doi:10.1002/asmb.446 , language =

    2001 doi

  141. [150]

    International Journal of Educational Research , author =

    Item bias and item response theory , volume =. International Journal of Educational Research , author =. 1989 , pages =. doi:10.1016/0883-0355(89)90002-5 , number =

    1989 doi

  142. [151]

    Journal of Educational Measurement , author =

    An. Journal of Educational Measurement , author =. 1987 , pages =

    1987

  143. [152]

    ETS Research Report Series , author =

    Exploration of. ETS Research Report Series , author =. 1985 , pages =. doi:10.1002/j.2330-8516.1985.tb00127.x , language =

    1985

  144. [153]

    Item Bias and Test Multidimensionality

    Kok, Frank. Item Bias and Test Multidimensionality. Latent Trait and Latent Class Models. 1988. doi:10.1007/978-1-4757-5644-9_12

    1988 doi

  145. [154]

    Applied Measurement in Education , author =

    Assessing. Applied Measurement in Education , author =. 2001 , pages =. doi:10.1207/S15324818AME1403_3 , number =

    2001 doi

  146. [155]

    Traditional vs intersectional

    Albano, Tony and French, Brian F and Vo, Thao Thu , journal=. Traditional vs intersectional. 2024 , publisher=. doi:10.1080/08957347.2024.2311935 , pages=

    2024

  147. [156]

    An intersectional approach to

    Russell, Michael and Szendey, Olivia and Li, Zhushan , booktitle=. An intersectional approach to. 2023 , publisher=. doi:10.1080/10627197.2022.2094757 , pages=

    2023

  148. [157]

    Journal of Educational measurement , volume=

    Detecting differential item functioning using logistic regression procedures , author=. Journal of Educational measurement , volume=. 1990 , publisher=

    1990

  149. [158]

    Applied Psychological Measurement , volume=

    Empirical selection of anchors for tests of differential item functioning , author=. Applied Psychological Measurement , volume=. 2009 , publisher=

    2009

  150. [159]

    Use of Item Response Theory in the Study of Group Differences in Trace Lines

    Thissen, David and Steinberg, Lynne and Wainer, Howard , year =. Use of Item Response Theory in the Study of Group Differences in Trace Lines. , booktitle =. doi:10.1037/14047-004 , abstract =

    doi

  151. [160]

    and Donoghue, John R

    Allen, Nancy L. and Donoghue, John R. and Schoeps, Thomas L. , title =. 2001 , number =

    2001

  152. [161]

    de Ayala, R. J. , year =. The theory and practice of item response theory. , isbn =

  153. [162]

    and Algina, James , address =

    Crocker, Linda M. and Algina, James , address =. Introduction to classical and modern test theory , year =. Introduction to classical and modern test theory , isbn =

  154. [163]

    Predicting

    Marinho, Wemerson and Clua, Esteban Walter and Martí, Luis and Marinho, Karla , month = mar, year =. Predicting. doi:10.1145/3576050.3576139 , urldate =

  155. [164]

    Expert Systems with Applications , author =

    Design a personalized e-learning system based on item response theory and artificial neural network approach , volume =. Expert Systems with Applications , author =. 2009 , keywords =. doi:https://doi.org/10.1016/j.eswa.2008.10.080 , number =

    2009 doi

  156. [165]

    Automatic item generation: foundations and machine learning-based approaches for assessments , volume =

    Circi, Ruhan and Hicks, Juanita and Sikali, Emmanuel , month = may, year =. Automatic item generation: foundations and machine learning-based approaches for assessments , volume =. Frontiers in Education , publisher =. doi:10.3389/feduc.2023.858273 , language =

  157. [166]

    Srivastava, Nitish and Hinton, Geoffrey and Krizhevsky, Alex and Sutskever, Ilya and Salakhutdinov, Ruslan , title =. J. Mach. Learn. Res. , month = jan, pages =. 2014 , issue_date =

    2014

  158. [167]

    Journal of Educational Measurement , author =

    Finding words associated with. Journal of Educational Measurement , author =. 2025 , pages =. doi:10.1111/jedm.70017 , language =

    2025 doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.