Pith. sign in

REVIEW 4 major objections 6 minor 64 references

Text embeddings predict item difficulty — but only half the story

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 19:02 UTC pith:7M6Y3QDC

load-bearing objection Solid framework for text-based item parameter prediction; the two-ceiling idea is the real contribution the 4 major comments →

arxiv 2607.07141 v1 pith:7M6Y3QDC submitted 2026-07-08 cs.CL stat.ME

From Text to Parameters: Predicting Item Parameters from Embedding Regularization with Reliability and Design Ceilings

Shi-Ting Chen , Jinsong Chen This is my paper
classification cs.CL stat.ME
keywords item difficulty predictiontext embeddingsregularized regressionreliability ceilingitem response theorycross-validationpsychometric calibration
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that item difficulty in educational tests can be predicted from the text of the items themselves using modern language-model embeddings and regularized regression, achieving a cross-validated R² of about 0.53 on a mathematics item bank. The deeper claim, however, is methodological: the apparent hierarchy of predictability across psychometric parameters (difficulty is easy to predict, discrimination harder, pseudo-guessing nearly impossible) is largely an artifact of how noisy each parameter's calibration is, not of how much textual signal the embeddings capture. The authors introduce two upper bounds — a reliability ceiling (the maximum R² any predictor could achieve given the standard errors of the calibrated parameters) and a design ceiling (the maximum R² the finite sample and embedding matrix could yield even for a perfect signal) — and show that when observed performance is read against these ceilings, text recovers a strikingly uniform 57–63% of the reliable variance across all difficulty targets. The pseudo-guessing parameter, by contrast, has a reliability ceiling near zero: its between-item variance is smaller than its average calibration error, making it an unusable prediction target regardless of method. On a medical-licensure benchmark, the same pipeline matches leaderboard RMSE while explaining almost no variance, demonstrating that scale-dependent metrics like RMSE can mask the absence of genuine predictive signal.

Core claim

The central discovery is that the predictability hierarchy across IRT parameters — difficulty > discrimination > pseudo-guessing — dissolves when each parameter's reliability ceiling is accounted for. Text embeddings recover a nearly constant fraction (57–63%) of the reliable variance in every difficulty target, meaning the raw gap in R² across parameters reflects differential calibration noise rather than differential text signal. The 3PL pseudo-guessing parameter has an effective reliability ceiling of zero because its average sampling variance exceeds its between-item variance by a factor of six, certifying it as an unusable target at current calibration precision, not a failure of the文本-

What carries the argument

The framework has four components: (1) Elastic Net / Ridge / Lasso regression on item-text embeddings, treating the embedding as an automated design matrix in the spirit of the Linear Logistic Test Model; (2) repeated K-fold cross-validation (10 fold assignments) reporting mean and standard deviation of out-of-fold R², avoiding the 0.1–0.15 R² inflation a single split can produce; (3) a reliability ceiling R²_rel = Var(T) / (Var(T) + SE²), derived from the standard errors of the IRT calibration, representing the maximum population R² attainable against an estimated target; and (4) a design ceiling obtained by injecting synthetic signals of known R² into the principal components of the actual

Load-bearing premise

The reliability ceiling assumes that the standard errors from the IRT calibration accurately reflect the true sampling uncertainty of each item parameter. For the 3PL model specifically, the standard errors come from a refit that hit its EM iteration limit before full convergence, so the 3PL ceilings — including the claim that pseudo-guessing is an unusable target — rest on approximate standard errors from a non-converged estimation.

What would settle it

If the 3PL standard errors from the non-converged refit are systematically overestimated, the pseudo-guessing reliability ceiling could be artificially depressed toward zero, making the 'unusable target' claim an artifact of estimation failure rather than a genuine property of the parameter. A clean 3PL calibration with properly converged standard errors that yielded a substantially positive reliability ceiling for pseudo-guessing would undermine the central interpretive claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Item-difficulty benchmarks should report R² or RMSE/SD ratios alongside RMSE, and should include reliability ceilings whenever standard errors are available, so that near-zero R² on a noisily calibrated target is not mistaken for method failure.
  • Targets with reliability ceilings near zero (like 3PL pseudo-guessing at typical examinee sample sizes) should be excluded from prediction benchmarks altogether, or flagged as unusable until calibration precision improves.
  • Text-based item parameter predictions of the accuracy class reported here (R² ≈ 0.53 for difficulty) can serve as informative priors that materially reduce calibration sample requirements, rather than replacing empirical calibration.
  • Fixed benchmark train–test splits should be drawn by method-agnostic rules (random seeds or distribution matching), never selected to maximize a particular pipeline's holdout accuracy, and should be paired with repeated cross-validation on pooled items.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the reliability ceiling framework were applied retrospectively to published item-difficulty prediction studies, some reported 'failures' to predict discrimination or guessing parameters might be reinterpreted as targets being too noisy to predict, redirecting effort toward richer calibration data rather than richer models.
  • The uniform 57–63% recovery fraction across difficulty targets suggests a ceiling on what linear embeddings can extract, and the gap between this fraction and the reliability ceiling (roughly 37–43% of reliable variance unrecovered) quantifies the headroom for nonlinear prediction heads, fine-tuned representations, or multimodal encoders.
  • The design-ceiling simulation methodology could be adopted as a general pre-registration tool: before running a prediction study, researchers could simulate whether their sample size and embedding dimensionality are adequate to detect a signal of the strength they expect, preventing uninformative null results.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes an evaluation framework for predicting IRT item parameters from text embeddings. The framework combines regularized regression (Elastic Net) on item-text embeddings, repeated cross-validated R² with resampling standard deviations, and two upper bounds: a reliability ceiling derived from parameter standard errors, and a design ceiling derived from simulation-based power calibration. Applied to EEDI (mathematics) and BEA 2024 (medical licensure), the authors find that item difficulty is predictable from text (R² = 0.53 for 1PL difficulty), and that the apparent hierarchy of predictability (difficulty > discrimination > pseudo-guessing) is largely an artifact of target reliability rather than text signal strength. Text recovers a uniform 57–63% of reliable variance across difficulty targets, while the 3PL pseudo-guessing parameter has a reliability ceiling near zero. On BEA, the pipeline matches leaderboard RMSE while explaining almost no variance, illustrating the need for scale-free metrics. The paper also shows that single train-test splits can inflate R² by 0.1–0.15.

Significance. The paper makes a methodologically sound and practically valuable contribution to the psychometric and NLP communities working on item difficulty modeling. The core innovation is the dual-ceiling framework: the reliability ceiling (Equation 5) and the design ceiling (Section 3.4) together convert otherwise ambiguous R² values into interpretable fractions of attainable performance. The exact RMSE-SD-R² relationship (Equation 3) is a useful reminder for the field. The BEA 2024 analysis—showing that leaderboard-competitive RMSE corresponds to R² ≈ 0.07—is a compelling demonstration of why scale-free metrics matter. The reproducible code and fold-assignment seeds (OSF link) are a strength. The finding that text recovers a uniform 57–63% of reliable variance across difficulty targets, while the pseudo-guessing parameter is an unusable target due to near-zero reliability, is a substantive contribution to how item parameter prediction benchmarks should be constructed and interpreted.

major comments (4)
  1. Table 2, note c / Table 7: The 3PL reliability ceilings rest on a refit whose EM iteration limit was reached before full convergence. The authors note the refit reproduced original target estimates at correlations ≥ 0.99, but standard errors are more sensitive to convergence than point estimates. The most dramatic 3PL claim—pseudo-guessing has a ceiling near zero—depends on these approximate SEs. The paper explicitly flags this as approximate, and the claim has independent support from the low 2PL-3PL log-discrimination correlation (r = 0.16). However, the central interpretive contribution is the ceiling-adjusted recovery fraction, and for the 3PL targets these fractions (0.58 for difficulty, 0.14 for log-discrimination, undefined for logit pseudo-guessing) rest on potentially biased SEs. The authors should either (a) obtain converged 3PL standard errors (e.g., via MHRM or increasing the
  2. EM iteration limit substantially) or (b) provide a sensitivity analysis showing how the 3PL ceiling-adjusted fractions change under reasonable perturbations of the SEs. The core uniform-recovery claim for difficulty (57–63%) does not depend on the 3PL ceilings, so this is a localized rather than fatal concern, but it affects the 3PL rows of Table 7 which are presented as findings.
  3. Table 7 / Discussion, point 2: The characterization of 2PL log-discrimination recovery (51% of ceiling) as 'nearly as text-recoverable as difficulty' (57–63%) is a generous reading of a 6–12 percentage-point gap. This is an interpretive choice rather than a correctness issue, but the phrasing in the Discussion ('nearly as text-recoverable as difficulty') somewhat overstates the equivalence. The authors should consider softening this language or providing a more explicit justification for why a 51% vs. 57–63% gap is characterized as 'nearly' equivalent.
  4. Section 5.2 / Table 5: The BEA analysis is limited to CTT difficulty because examinee-level response data were unavailable, precluding IRT parameter estimation and reliability ceiling computation. This means the BEA results cannot be read against the reliability ceiling framework that is the paper's central methodological contribution. The BEA analysis serves as an external benchmark for metric comparison (RMSE vs. R²), which is valuable, but the authors should more clearly acknowledge that the ceiling framework cannot be applied to BEA, and that the conclusion about BEA ('difficulty labels carry little text-recoverable signal') is inferred from the high design ceiling and near-zero R² rather than from a reliability ceiling analysis.
minor comments (6)
  1. Table 1: The model name 'Qwen3-Embedding-8B' is cited to Zhang et al. (2025), but the reference list entry gives the title as 'Qwen3 embedding: Advancing text embedding and reranking through foundation models.' Minor inconsistency in naming.
  2. Section 3.4, step 2: The formula for σ uses R²_true but the subscript formatting is inconsistent with the rest of the paper (R²_true vs. R²_true). Ensure consistent LaTeX rendering.
  3. Appendix A: The rationale-generation prompt specifies 'two to four sentences' per explanation, but no validation is reported for whether the generated rationales actually met this constraint. A brief note on rationale quality control would strengthen reproducibility.
  4. Table 3: Monte Carlo SDs are stated to be 0.01–0.04 but are not reported in full in the table. The online supplement is referenced but not included in the manuscript. Consider including at least the range in a table note.
  5. Section 5.6: The weighted regression results (Table 8) show that the variance-stabilized weights w(3) are nearly uniform because τ² dominates SE². This is correctly explained, but the three weighting schemes could be more clearly motivated in the Methods section (Section 4.3) before the results are presented.
  6. Figure 1 and Figure 2: The figures are referenced but not visible in the text provided. Ensure they are clearly labeled and readable in the final version.

Circularity Check

0 steps flagged

No circularity: the framework's predictions, ceilings, and evaluations are derived from independent data sources and standard psychometric formulas.

full rationale

The paper's central claims are not circular. The reliability ceiling (Equation 5) is derived from the standard errors of the IRT calibration, which are computed from examinee response data—entirely independent of the text embeddings used as predictors. The design ceiling is derived from simulation on the embedding matrix without reference to the actual target values. The prediction R² is computed on held-out folds via repeated cross-validation, so the reported accuracy is not fitted to the evaluation data. The BEA leaderboard comparison uses external published results. The RMSE–SD–R² relationship (Equation 3) is a standard algebraic identity, not a fitted or self-referential result. The 3PL ceiling rests on approximate SEs from a non-converged EM refit, but the paper explicitly flags this as approximate (Table 2 note c), and the core uniform-recovery finding rests primarily on the well-converged 1PL and 2PL calibrations. No step in the derivation chain reduces to its own inputs by construction, and no self-citation is load-bearing for the central claims.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

No new entities are postulated. The free parameters are standard regularized-regression hyperparameters selected by cross-validation. The axioms are standard domain assumptions from psychometrics and statistics, not ad hoc constructions.

free parameters (3)
  • Regularization parameter λ = Selected by cross-validation within training data
    Standard CV-selected hyperparameter; not a free parameter in the circularity sense.
  • Elastic Net mixing parameter α = Selected by CV over {0, 0.1, ..., 1}
    Standard CV-selected hyperparameter.
  • Assumed signal dimensionality m for design ceiling = Grid: {1, 5, 20}
    Not fitted to data; reported as a grid to bracket uncertainty.
axioms (3)
  • domain assumption The IRT standard errors from mirt accurately reflect the sampling uncertainty of the item parameter estimates.
    Invoked in Section 3.3 (Equation 5) and Table 2. The reliability ceiling depends entirely on these SEs being correct. For 3PL, the refit did not fully converge.
  • domain assumption The linear regression model (Elastic Net on embeddings) is a sufficient prediction map for the signal in item text.
    Invoked in Section 3.1 (Equation 1). The design ceiling assumes the signal lies in the top principal components and is linear; nonlinear signals could exceed this bound.
  • domain assumption CTT difficulty and 1PL difficulty carry nearly identical ordering information.
    Stated in Section 4.2 based on r=0.990 in EEDI. Used to justify treating CTT and 1PL results as one finding.

pith-pipeline@v1.1.0-glm · 23705 in / 2289 out tokens · 270974 ms · 2026-07-09T19:02:04.999931+00:00 · methodology

0 comments
read the original abstract

Newly developed items must ordinarily be field tested before their psychometric properties are known, creating a cold start problem for item calibration. Predicting item parameters from features is a long standing measurement problem dating back to the Linear Logistic Test Model; modern text embeddings now automate the design matrices traditionally specified by hand. We propose an evaluation framework combining regularized regression on item text embeddings, repeated cross validated R squared reported with its resampling standard deviation, and two performance upper bounds: a reliability ceiling derived from parameter standard errors, and a design ceiling derived from simulation based power calibration. Applying this framework to a mathematics item bank (EEDI) and a medical licensure benchmark (BEA 2024), we find that item difficulty is highly predictable from text (repeated cross validated R squared = 0.53, or about 57% of its reliability ceiling), whereas discrimination and pseudo guessing appear less predictable. However, evaluating these results against our ceilings reveals that this apparent hierarchy stems from target reliability rather than text signal strength: text uniformly recovers 57 to 63% of the reliable variance across difficulty targets, whereas the 3PL pseudo guessing parameter has a reliability ceiling near zero, making it an unviable target at current precision. On BEA, embedding based regression matches leaderboard RMSE despite explaining almost no variance, highlighting the critical need for scale free metrics and explicit ceilings in benchmarking. Finally, we show that a single train and test split can inflate apparent accuracy by 0.1 to 0.15 in R squared, underscoring the necessity of repeated cross validation for calibration support applications and future benchmark construction.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

64 extracted references · 64 canonical work pages · 5 internal anchors

  1. [1]

    Medical care , volume=

    Classical test theory , author=. Medical care , volume=. 2006 , publisher=

    work page 2006

  2. [2]

    Applied Psychological Measurement , volume=

    Subject matter experts' assessment of item statistics , author=. Applied Psychological Measurement , volume=. 1983 , publisher=

    work page 1983

  3. [3]

    Journal of statistical Software , volume=

    mirt: A multidimensional item response theory package for the R environment , author=. Journal of statistical Software , volume=

    work page

  4. [4]

    Fifth international conference on information technology: New generations (itng 2008) , pages=

    Expert-driven knowledge discovery , author=. Fifth international conference on information technology: New generations (itng 2008) , pages=. 2008 , organization=

    work page 2008

  5. [5]

    International Conference on Learning Representations , volume=

    Nv-embed: Improved techniques for training llms as generalist embedding models , author=. International Conference on Learning Representations , volume=

    work page

  6. [6]

    Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models

    Qwen3 embedding: Advancing text embedding and reranking through foundation models , author=. arXiv preprint arXiv:2506.05176 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    URL https://huggingface.co/Salesforce/SFR-Embedding-2\_R , year=

    Sfr-embedding-2: Advanced text embedding with multi-stage training, 2024 , author=. URL https://huggingface.co/Salesforce/SFR-Embedding-2\_R , year=

    work page 2024

  8. [8]

    Longformer: The Long-Document Transformer

    Longformer: The long-document transformer , author=. arXiv preprint arXiv:2004.05150 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 2004

  9. [9]

    Educational measurement: issues and practice , volume=

    Comparison of classical test theory and item response theory and their applications to test development , author=. Educational measurement: issues and practice , volume=

    work page

  10. [10]

    Journal of the Royal Statistical Society Series B: Statistical Methodology , volume=

    Regression shrinkage and selection via the lasso , author=. Journal of the Royal Statistical Society Series B: Statistical Methodology , volume=. 1996 , publisher=

    work page 1996

  11. [11]

    2013 , publisher=

    An introduction to statistical learning: with applications in R , author=. 2013 , publisher=

    work page 2013

  12. [12]

    CRAN R Repositary , volume=

    Package ‘glmnet’ , author=. CRAN R Repositary , volume=

    work page

  13. [13]

    Behavior Research Methods , volume=

    Cross-validation and predictive metrics in psychological research: Do not leave out the leave-one-out , author=. Behavior Research Methods , volume=. 2025 , publisher=

    work page 2025

  14. [14]

    Advances in Methods and Practices in Psychological Science , volume=

    Making sense of model generalizability: A tutorial on cross-validation in R and Shiny , author=. Advances in Methods and Practices in Psychological Science , volume=. 2021 , publisher=

    work page 2021

  15. [15]

    Journal of computer-aided molecular design , volume=

    Estimation of influential points in any data set from coefficient of determination and its leave-one-out cross-validated counterpart , author=. Journal of computer-aided molecular design , volume=. 2013 , publisher=

    work page 2013

  16. [16]

    International Conference on Artificial Intelligence in Education , pages=

    Reasoning and sampling-augmented mcq difficulty prediction via llms , author=. International Conference on Artificial Intelligence in Education , pages=. 2025 , organization=

    work page 2025

  17. [17]

    Instructions and Guide for Diagnostic Questions: The NeurIPS 2020 Education Challenge

    Instructions and guide for diagnostic questions: The neurips 2020 education challenge , author=. arXiv preprint arXiv:2007.12061 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  18. [18]

    Large-scale Assessments in Education , volume=

    Linking errors introduced by rapid guessing responses when employing multigroup concurrent IRT scaling , author=. Large-scale Assessments in Education , volume=. 2025 , publisher=

    work page 2025

  19. [19]

    2024 , publisher=

    Exploring Psychometric Analysis of Textual Data with Large Language Models: Chances and Challenges , author=. 2024 , publisher=

    work page 2024

  20. [20]

    Amini, Aida and Gabriel, Saadia and Lin, Peter and Koncel-Kedziorski, Rik and Choi, Yejin and Hajishirzi, Hannaneh , journal =

    work page

  21. [21]

    arXiv preprint arXiv:2507.17442 , year=

    Confident RAG: Enhancing the Performance of LLMs for Mathematics Question Answering through Multi-Embedding and Confidence Scoring , author=. arXiv preprint arXiv:2507.17442 , year=

    work page arXiv

  22. [22]

    Technometrics , volume=

    Ridge regression: Biased estimation for nonorthogonal problems , author=. Technometrics , volume=. 1970 , publisher=

    work page 1970

  23. [23]

    Journal of the Royal Statistical Society Series B: Statistical Methodology , volume=

    Regularization and variable selection via the elastic net , author=. Journal of the Royal Statistical Society Series B: Statistical Methodology , volume=. 2005 , publisher=

    work page 2005

  24. [24]

    Journal of the Royal Statistical Society Series B: Statistical Methodology , volume=

    Model selection and estimation in regression with grouped variables , author=. Journal of the Royal Statistical Society Series B: Statistical Methodology , volume=. 2006 , publisher=

    work page 2006

  25. [25]

    Journal of statistical software , volume=

    Regularization paths for generalized linear models via coordinate descent , author=. Journal of statistical software , volume=

    work page

  26. [26]

    2005 , publisher=

    Linear models for optimal test design , author=. 2005 , publisher=

    work page 2005

  27. [27]

    2004 , publisher=

    Item response theory: Parameter estimation techniques , author=. 2004 , publisher=

    work page 2004

  28. [28]

    2025 , publisher=

    Item response theory: Foundations for psychologists and social scientists , author=. 2025 , publisher=

    work page 2025

  29. [29]

    Applied Psychological Measurement , volume=

    Cognitive assessment models with few assumptions, and connections with nonparametric item response theory , author=. Applied Psychological Measurement , volume=. 2001 , publisher=

    work page 2001

  30. [30]

    Journal of educational and behavioral statistics , volume=

    DINA model and parameter estimation: A didactic , author=. Journal of educational and behavioral statistics , volume=. 2009 , publisher=

    work page 2009

  31. [31]

    2007 , publisher=

    Cognitive diagnostic assessment for education: Theory and applications , author=. 2007 , publisher=

    work page 2007

  32. [32]

    Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024) , pages=

    Bert-irt: Accelerating item piloting with bert embeddings and explainable irt models , author=. Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024) , pages=

    work page 2024

  33. [33]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Smart: Simulated students aligned with item response theory for question difficulty prediction , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    work page 2025

  34. [34]

    Proceedings of the 2021 conference on empirical methods in natural language processing , pages=

    Jump-starting item parameters for adaptive language tests , author=. Proceedings of the 2021 conference on empirical methods in natural language processing , pages=

    work page 2021

  35. [35]

    International Journal of Artificial Intelligence in Education , volume=

    Text-based question difficulty prediction: A systematic review of automatic approaches , author=. International Journal of Artificial Intelligence in Education , volume=. 2024 , publisher=

    work page 2024

  36. [36]

    American Journal of Education and Learning , volume=

    Using item features to calibrate educational test items: Comparing artificial intelligence and classical approaches , author=. American Journal of Education and Learning , volume=. 2025 , publisher=

    work page 2025

  37. [37]

    Educational Measurement: Issues and Practice , volume=

    Rater cognition: Implications for validity , author=. Educational Measurement: Issues and Practice , volume=. 2012 , publisher=

    work page 2012

  38. [38]

    ETS Research Report Series , volume=

    Estimating item difficulty with comparative judgments , author=. ETS Research Report Series , volume=. 2014 , publisher=

    work page 2014

  39. [39]

    1960 , publisher =

    Rasch, Georg , title =. 1960 , publisher =

    work page 1960

  40. [40]

    Statistical Theories of Mental Test Scores , editor =

    Birnbaum, Allan , title =. Statistical Theories of Mental Test Scores , editor =. 1968 , publisher =

    work page 1968

  41. [41]

    2012 , publisher=

    Applications of item response theory to practical testing problems , author=. 2012 , publisher=

    work page 2012

  42. [42]

    Handbook of statistics , volume=

    18 multidimensional item response theory , author=. Handbook of statistics , volume=. 2006 , publisher=

    work page 2006

  43. [43]

    Predicting the Difficulty of Multiple Choice Questions in a High-stakes Medical Exam

    Ha, Le An and Yaneva, Victoria and Baldwin, Peter and Mee, Janet. Predicting the Difficulty of Multiple Choice Questions in a High-stakes Medical Exam. Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications. 2019. doi:10.18653/v1/W19-4402

    work page doi:10.18653/v1/w19-4402 2019

  44. [44]

    Proceedings of the tenth international conference on learning analytics & knowledge , pages=

    R2DE: a NLP approach to estimating IRT parameters of newly generated questions , author=. Proceedings of the tenth international conference on learning analytics & knowledge , pages=

    work page

  45. [45]

    Research and practice in technology enhanced learning , volume=

    Controlling item difficulty for automatic vocabulary question generation , author=. Research and practice in technology enhanced learning , volume=. 2017 , publisher=

    work page 2017

  46. [46]

    Proceedings of the 8th international conference on knowledge capture , pages=

    Automatically predicting quiz difficulty level using similarity measures , author=. Proceedings of the 8th international conference on knowledge capture , pages=

    work page

  47. [47]

    Transactions of the association for computational linguistics , volume=

    Predicting the difficulty of language proficiency tests , author=. Transactions of the association for computational linguistics , volume=. 2014 , publisher=

    work page 2014

  48. [48]

    Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications , pages=

    Candidate evaluation strategies for improved difficulty prediction of language tests , author=. Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications , pages=

    work page

  49. [49]

    Language testing , volume=

    Predicting item difficulty in a reading comprehension test with an artificial neural network , author=. Language testing , volume=. 1995 , publisher=

    work page 1995

  50. [50]

    2013 ieee workshop on hybrid intelligent models and applications (hima) , pages=

    Predicting item difficulty in a language test with an Adaptive Neuro Fuzzy Inference System , author=. 2013 ieee workshop on hybrid intelligent models and applications (hima) , pages=. 2013 , organization=

    work page 2013

  51. [51]

    Efficient Estimation of Word Representations in Vector Space

    Efficient estimation of word representations in vector space , author=. arXiv preprint arXiv:1301.3781 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  52. [52]

    Advances in neural information processing systems , volume=

    Distributed representations of words and phrases and their compositionality , author=. Advances in neural information processing systems , volume=

    work page

  53. [53]

    Information Processing & Management , volume=

    Automated estimation of item difficulty for multiple-choice tests: An application of word embedding techniques , author=. Information Processing & Management , volume=. 2018 , publisher=

    work page 2018

  54. [54]

    Bert: Pre-training of deep bidirectional transformers for language understanding , author=. Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) , pages=

    work page 2019

  55. [55]

    Educational and Psychological Measurement , volume=

    Field-testing multiple-choice questions with AI examinees: English grammar items , author=. Educational and Psychological Measurement , volume=. 2025 , publisher=

    work page 2025

  56. [56]

    Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024) , pages=

    Findings from the first shared task on automated prediction of difficulty and response time for multiple-choice questions , author=. Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024) , pages=

    work page 2024

  57. [57]

    Educational and Psychological Measurement , volume=

    Item difficulty modeling using fine-tuned small and large language models , author=. Educational and Psychological Measurement , volume=. 2025 , publisher=

    work page 2025

  58. [58]

    Acta Psychologica , volume=

    The linear logistic test model as an instrument in educational research , author=. Acta Psychologica , volume=

    work page

  59. [59]

    2004 , publisher=

    Explanatory item response models: A generalized linear and nonlinear approach , editor=. 2004 , publisher=

    work page 2004

  60. [60]

    Psychological Methods , volume=

    A cognitive design system approach to generating valid tests: Application to abstract reasoning , author=. Psychological Methods , volume=

    work page

  61. [61]

    Exploiting auxiliary information about items in the estimation of

    Mislevy, Robert J , journal=. Exploiting auxiliary information about items in the estimation of

    work page

  62. [62]

    Journal of Educational Measurement , volume=

    How to equate tests with little or no data , author=. Journal of Educational Measurement , volume=

    work page

  63. [63]

    Using item parameter predictions for reducing calibration sample requirements---

    Ulitzsch, Esther and Belov, Dmitry and L. Using item parameter predictions for reducing calibration sample requirements---. Journal of Educational Measurement , volume=

    work page

  64. [64]

    Documents Are People and Words Are Items: A Psychometric Approach to Textual Data with Contextual Embeddings

    Documents are people and words are items: A psychometric approach to textual data with contextual embeddings , author=. arXiv preprint arXiv:2509.08920 , year=

    work page internal anchor Pith review Pith/arXiv arXiv