LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment

Chenguang Wang; Dawei Zhou; Han Chen; Hong Jiao; Ming Li; Tianyi Zhou; Yijun Liang

REVIEW 2 major objections 1 minor 2 cited by

LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment

T0 review · 2 major / 1 minor · reviewed 2026-06-26 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Large language models show only weak alignment with human item discrimination scores in reading comprehension tests, reaching at most 0.24 Spearman correlation.

desk verdict LLMs show only weak alignment with human item discrimination (max 0.24 Spearman) and the synthetic-student method needs checking before the numbers can be read as evidence of model capability. read the letter →

arxiv 2606.18709 v2 pith:ALK7GNRD submitted 2026-06-17 cs.CL

Han Chen , Ming Li , Chenguang Wang , Yijun Liang , Dawei Zhou , Hong jiao , Tianyi Zhou This is my paper

classification cs.CL

keywords itemdiscriminationlargelanguagemodelsclassicaltesttheoryreadingcomprehensioneducationalassessmentsyntheticstudentresponsespsychometricevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests whether LLMs can estimate item discrimination, the property that determines whether a test question separates higher-proficiency students from lower-proficiency ones. It applies two zero-shot methods across 42 models: direct numerical prediction from item text and treating model answers as synthetic student data to compute discrimination via classical test theory. Both approaches produce only low correlations with human-calibrated values, with the stronger response-based method topping out at 0.241. This matters for assessment design because reliable synthetic discrimination would let models help create or validate tests without collecting large human response samples first.

What carries the argument

Item discrimination computed via Classical Test Theory from either direct LLM estimates or synthetic student response patterns.

What would settle it

An experiment that collects new human responses on the same reading items and checks whether the resulting human discrimination rankings match those derived from the best LLM synthetic pool at correlation above 0.5.

Watch

Extended reading notes

Core claim

Direct prediction of discrimination values from item content yields at most 0.152 Spearman correlation with human data, while response-based CTT calibration using an all-persona synthetic pool reaches 0.241; these results indicate that current LLMs hold non-random discrimination-relevant signal yet fall short of reliably reproducing how items distinguish human students of differing proficiency.

Load-bearing premise

LLM-generated answers can stand in for real human student responses when calculating item discrimination scores.

Editorial extensions

If this is right

LLMs contain measurable but limited discrimination-relevant information in zero-shot settings.
Response-based calibration using multiple synthetic personas outperforms direct numerical prediction.
Item discrimination remains harder for LLMs to capture than item difficulty has been in prior work.
Current models do not yet support fully synthetic psychometric calibration of reading comprehension items.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If synthetic discrimination scores improve, test developers could iterate item pools with far fewer live human pilots.
Persona diversity in prompting may be a practical lever for raising correlation without new model training.
The gap between 0.241 and usable levels suggests future work on fine-tuning or few-shot calibration rather than zero-shot alone.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

read the letter

The main point is that current LLMs contain some discrimination-relevant signal but fall well short of reliably capturing how items separate high- and low-proficiency students. Direct prediction tops out at 0.152 Spearman; the response-based CTT route reaches 0.241 with an all-persona pool.

What is new is the explicit separation of discrimination from difficulty and the first reported quantitative results on the former across 42 models. The two-method design is a clean way to test both explicit estimation and implicit simulation.

The paper does a service by putting concrete numbers on the table instead of leaving the question at the level of general claims about LLM psychometric ability.

The soft spot is the response-based approach. Treating LLM persona answers as synthetic student data only works if those answers produce accuracy patterns that vary across items in ways that parallel real human proficiency differences. If the variation instead comes from model artifacts or uniform capability ceilings, the resulting discrimination scores are not comparable to human CTT values, and the modest correlation does not demonstrate that LLMs capture the target property. The abstract gives no dataset size, item selection details, or checks on whether the personas actually induce differentiated error profiles, so this assumption stays untested in what we can see.

This is for researchers working on LLM use in educational measurement. A reader who needs to know the current ceiling on automated discrimination estimation will get value from the numbers.

It deserves peer review because the question is practical and the experiments are a direct first cut, even if the methods section will need close attention.

Referee Report

2 major / 1 minor

Summary. The paper evaluates whether 42 LLMs can capture item discrimination in reading comprehension assessments. It reports two zero-shot approaches: direct prediction of discrimination values (max Spearman correlation 0.152 with human benchmarks) and response-based Classical Test Theory calibration treating LLM persona answers as synthetic student responses (max Spearman 0.241 for an all-persona pool). The central claim is that LLMs contain non-random discrimination-relevant signal but do not yet reliably capture how items distinguish human students of different proficiency levels.

Significance. If the empirical results hold after addressing methodological details, the work provides a concrete benchmark across 42 models and two complementary methods, highlighting item discrimination as a distinct open challenge beyond item difficulty estimation. The use of external human benchmarks and standard correlation metrics is a strength that allows direct comparison to psychometric standards.

major comments (2)

[Methods (response-based CTT calibration)] The response-based CTT approach (abstract and corresponding methods section) interprets the Spearman correlation of 0.241 as evidence of non-random signal from synthetic responses. However, this requires that the persona-induced accuracy patterns vary across items in ways that parallel real human proficiency differences; no validation is reported showing that the synthetic pool produces differentiated error profiles rather than uniform capabilities or training artifacts, which directly affects whether the correlation supports the claim about capturing human discrimination.
[Abstract and Experiments section] No details are provided on dataset size (number of items or test-takers), item selection criteria, or statistical testing (e.g., p-values or confidence intervals) for the reported Spearman correlations of 0.152 and 0.241. These omissions make it difficult to assess the reliability and generalizability of the central empirical findings.

minor comments (1)

[Abstract] The abstract would benefit from briefly stating the number of items and the source of the human discrimination benchmarks to allow readers to immediately gauge the scale of the evaluation.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which help clarify key methodological aspects of our work. We address each major comment below and indicate where revisions will be made to strengthen the manuscript.

read point-by-point responses

Referee: [Methods (response-based CTT calibration)] The response-based CTT approach (abstract and corresponding methods section) interprets the Spearman correlation of 0.241 as evidence of non-random signal from synthetic responses. However, this requires that the persona-induced accuracy patterns vary across items in ways that parallel real human proficiency differences; no validation is reported showing that the synthetic pool produces differentiated error profiles rather than uniform capabilities or training artifacts, which directly affects whether the correlation supports the claim about capturing human discrimination.

Authors: We agree that explicit validation of differentiated error profiles in the synthetic responses would strengthen the interpretation. The positive correlation with human benchmarks provides indirect support for non-uniform patterns (uniform capabilities across items would be unlikely to produce a positive alignment with human discrimination values), but we acknowledge the absence of direct validation such as item-wise accuracy variance or profile comparisons. In the revised manuscript, we will add analyses of accuracy variance across items for the all-persona pool and, where feasible, compare synthetic response patterns to human data to address this concern. revision: partial
Referee: [Abstract and Experiments section] No details are provided on dataset size (number of items or test-takers), item selection criteria, or statistical testing (e.g., p-values or confidence intervals) for the reported Spearman correlations of 0.152 and 0.241. These omissions make it difficult to assess the reliability and generalizability of the central empirical findings.

Authors: We agree these details are necessary for assessing reliability. The full manuscript describes the dataset and experiments, but we will expand the abstract and experiments section in the revision to explicitly report the number of items and test-takers, item selection criteria, and statistical testing including p-values and confidence intervals for the reported Spearman correlations. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: direct empirical evaluation against external human benchmarks

full rationale

The paper performs straightforward empirical comparisons of LLM outputs (direct predictions and response-based CTT scores) against independently collected human item discrimination values, using standard Spearman correlations as the metric. No equations, fitted parameters, or results are defined in terms of themselves; the central claims rest on external human data as the ground truth benchmark rather than any internal derivation or self-citation chain. The evaluation is fully falsifiable outside the paper's own fitted values and contains no self-definitional, ansatz-smuggling, or renaming steps.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The study is an empirical evaluation relying on standard statistical tools and psychometric definitions without introducing new free parameters, axioms beyond basic correlation measures, or invented entities.

assumptions (1)

standard math Spearman rank correlation appropriately quantifies alignment between model outputs and human-calibrated discrimination values
The abstract uses this metric to report all performance results.

how reviews work

0 comments

Cite this review

Pith. "Pith review of LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment." pith.science (2026). https://pith.science/paper/ALK7GNRD

@misc{pith2026260618709,
  author       = {Pith},
  title        = {Pith review of: LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ALK7GNRD}},
  note         = {Machine review of arXiv:2606.18709}
}

read the original abstract

Existing work on LLM-based educational assessment has focused largely on item difficulty, but difficulty alone does not indicate whether an item meaningfully distinguishes higher- from lower-proficiency students. Item discrimination captures this complementary and fundamental psychometric property. We investigate whether LLMs can predict human item discrimination from assessment content. We evaluate 42 proprietary and open-weight LLMs using two complementary approaches. Direct discrimination prediction asks models to explicitly predict an item's discrimination value, while response-based proxy estimation treats LLM answers as synthetic responses and applies a Classical Test Theory (CTT)-inspired item-rest calculation. Direct predictions show weak alignment with human item discrimination. The response-based proxy provides a stronger but still limited ranking signal, reaching a CEFR-stratified rank correlation of 0.231. Further analysis shows that this correlation comes mainly from differences across models rather than proficiency prompts that reliably simulate students at different ability levels. Current LLMs therefore contain some discrimination-relevant information, but they do not yet reliably model the ability-conditioned human response behavior that gives item discrimination its psychometric meaning.

Figures

Figures reproduced from arXiv: 2606.18709 by the authors.

**Figure 2.** Distributions of human-calibrated and LLM-predicted item discrimination values for 20 representative [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Distributions of human-calibrated and LLM [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Prompt template for direct discrimination prediction. The gold answer was provided only in this setting. [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]

**Figure 5.** Figure 5: Prompt template for direct answer prediction used in response-based CTT calibration. The correct answer [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]

**Figure 6.** Figure 6: Complete distributions of human-calibrated and LLM-predicted item discrimination values for all 42 [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction
cs.CL 2026-06 unverdicted novelty 6.5 of 10

Structured cognitive-episode features from LRM reasoning traces, combined with item semantics, improve human item-difficulty prediction and show harder items drive more implementation-centered, iterative solving.
Representing Visual Evidence for Item Difficulty Prediction: Visual Textualization and Image-Native Modeling
cs.CL 2026-08 conditional novelty 6.0 of 10

Both visual interfaces, textualized diagrams and image-native VLM input, beat text-only difficulty prediction on point estimates, but their difference is not statistically reliable, and image-native gains depend on br...

Pith tools

Reviewed June 26, 2026 · model on record in the stance chip above.