{"id":"9a9d92f3-2c10-4ca0-b8e6-b8debffcb6bd","arxiv_id":"2507.08154","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Text-LENS, a text-aware version of the LENS variational autoencoder, improves predictions on unseen test items while matching LENS on items seen during training.","lead":"A new version of an assessment model reads the text of exam questions instead of just question numbers. Tested on real and simulated math data, it predicts how students do on brand-new questions better than the original model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The cleanest evidence for text-driven zero-shot prediction comes from LLM-Sim, where the difficulty-text link is injected by the generation prompt; no real-data check establishes that natural item text encodes difficulty strongly enough.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the synthetic LLM-Sim evaluation assumes that LLM-assigned difficulty labels are recoverable from generated text and that this recovery resembles real items. My independent reading confirms this is the point where the central claim is least secure. The model comparison alone cannot separate architecture quality from the artificially strong text-difficulty relationship manufactured by the generation prompt. The randomization control proves only that Text-LENS uses the injected relationship, not that the relationship exists in real assessment items. Section 7 explicitly acknowledges this limitation, which further supports treating the concern as real rather than manufactured. The Eedi results provide some real-world evidence, but the acknowledged OCR noise and loss of multimodal information make them weaker than the synthetic results, and no code or data link is provided for independent verification. Because the paper is honestly framed as a proof-of-concept and explicitly flags the need for higher-quality real datasets, the existing CONDITIONAL verdict remains appropriate. The proposed linear-probe test would isolate text-difficulty recoverability from the full VAE pipeline and directly settle whether the synthetic advantage transfers to real item text.","tokens_in":776,"tokens_out":769,"duration_ms":69450,"concrete_test":"Train a linear probe on mathBERT embeddings of LLM-Sim items to predict the assigned difficulty labels and measure held-out AUC. Train the same probe on clean human transcriptions of Eedi item text to predict IRT-estimated item difficulties (or binned difficulties estimated from training responses) and compare the two AUCs. If the Eedi probe AUC is substantially lower than the LLM-Sim probe AUC, the synthetic text-difficulty association is stronger than in real items and Text-LENS's LLM-Sim advantage cannot be taken as evidence for real-world zero-shot prediction; if the AUCs are comparable, this concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that text embeddings carry item difficulty and skill information sufficient for zero-shot prediction rests most heavily on the LLM-Sim experiment, but that experiment builds the text-difficulty link into the data. In Section 4, the LLM is prompted to generate ten easy, medium, and hard questions per subskill and to vary wording, and item difficulties are then assigned as -1.5, 0, or 1.5 from the LLM's self-reported difficulty level. The randomization control in Section 6.3 shows Text-LENS exploits this injected coupling, but it does not show that real math item text encodes difficulty with comparable strength. The paper's own Discussion (Section 7) concedes exactly this: 'it is unclear to what degree real-world questions encode difficulty in text and how that differs from an LLM specifically prompted to produce questions at different difficulties.' The real-data check on Eedi is weakened by acknowledged OCR noise and loss of spatial/graphical information, so the only clean demonstration of difficulty recovery may overstate real-world transfer. This is not an internal inconsistency; it is an external-validity risk in the strongest evidence for the paper's headline generalization claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Text-LENS, a modification of the LENS partial variational autoencoder in which item-ID embeddings are replaced by mathBERT text embeddings of item content. The goal is to enable predictions about student performance on previously unseen assessment items. The model is evaluated on two datasets: Eedi, a real-world math assessment dataset whose item images are converted to text via OCR, and LLM-Sim, a fully synthetic dataset in which GPT-4o generates math items at specified difficulty levels and student responses are simulated from a 3PL IRT model. The experimental design varies whether input items and query items were seen during training and whether the input items are on-target or off-target relative to the query skill, yielding eight conditions. Across these conditions, Text-LENS matches LENS on seen-query conditions and outperforms LENS on unseen-query conditions, with the largest advantage in off-target unseen-query conditions where LENS is at chance and Text-LENS reaches roughly 0.58 AUC. A randomization control on LLM-Sim, in which the text-difficulty relationship is broken, reduces Text-LENS to chance, suggesting the model exploits text-based difficulty cues.","tokens_in":7526,"tokens_out":6486,"duration_ms":70333,"significance":"If the result holds, Text-LENS addresses a real practical problem: the cold-start issue in ML-based educational assessment, where new items cannot be scored until student response data accumulate. The paper's strengths include a carefully structured eight-condition experimental matrix, 60 repetitions per condition, a novel synthetic dataset made publicly available, and a randomization control that directly tests the mechanism behind the unseen-query gains. These design choices give the internal comparison credibility. However, the headline generalization claim rests on an external-validity assumption that real math item text encodes difficulty as strongly as the LLM-generated prompts do; the paper's own discussion acknowledges this uncertainty. The real-world Eedi validation is weakened by acknowledged OCR noise and loss of spatial information. The lack of significance testing and the absence of any text-aware baseline beyond LENS also leave the magnitude and attribution of the improvements less certain than the text suggests.","major_comments":[{"comment":"The strongest evidence that text embeddings encode item difficulty is manufactured by construction. In §4, GPT-4o is explicitly prompted to generate easy, medium, and hard questions, and item difficulties are then set to -1.5, 0, or 1.5 on the basis of those self-reported levels before responses are simulated with a 3PL model. The §6.3 randomization control shows that Text-LENS exploits this injected text-difficulty coupling, but it does not establish that real mathematics item text encodes difficulty with comparable strength. The Discussion (§7) concedes exactly this point. Because the central generalization claim rests most heavily on LLM-Sim, the manuscript needs an additional real-data test in which item difficulties come from IRT calibration or professional annotation, and ideally where text is not degraded by OCR, to show that the injected-link result transfers to natural item text.","section":"§4 and §6.3"},{"comment":"The randomization control is described too briefly to be reproduced or fully interpreted. Please specify exactly how the text-difficulty relationship was broken (for example, permuting difficulty labels across items, shuffling text embeddings, or sampling from a null distribution), how many random seeds or repetitions were used, and which of the eight conditions were re-run. This control is load-bearing for the claim that text, rather than some other item property, drives the unseen-query gains, so the operational details should be in the methods rather than a single sentence in the results.","section":"§6.3"},{"comment":"No significance tests or confidence intervals are reported despite 60 repetitions per condition. Statements such as \"Text-LENS consistently matches or surpasses LENS\" and the condition 7/8 comparison (Text-LENS at about 0.58 AUC versus LENS at chance) need a paired statistical test, effect-size measure, or confidence interval across the 60 runs. This is particularly important because the parity claim on seen queries is the basis for arguing that text embeddings are not hurting performance; small differences in Figure 1 could easily be within sampling noise.","section":"§6, Figure 1"},{"comment":"The Eedi results are the only real-world validation, but the acknowledged OCR noise and the loss of graphical and spatial information mean this check may substantially understate what text-aware models can do with clean item text. To support the generalization claim, the paper should either report results on a subset of Eedi items whose OCR text was manually verified, use a multimodal LLM to summarize item images as the paper suggests in §7, or include a separate real-text dataset with IRT-calibrated difficulties. Without this, the only clean demonstration of difficulty recovery remains the synthetic LLM-Sim experiment.","section":"§5.1, §6, and §7"}],"minor_comments":[{"comment":"The condition numbering in the sentence \"LENS performs better when making inferences about unseen query items with unseen input items (condition 5) than with seen (condition 4)\" is inconsistent with the numbering implied by Figure 1 and the earlier discussion; under the figure's ordering, the comparison appears to be between condition 6 and condition 5. Please correct the numbering or clarify the table.","section":"§6.4"},{"comment":"Reference [4] is the original BERT paper, but the text says the embeddings were produced by mathBERT. Please cite the specific mathBERT model (for example, Peng et al., 2021) rather than the general BERT paper.","section":"§5.1"},{"comment":"Please clarify the relationship between the five Eedi skills and the five LLM-Sim skills used after filtering, and state whether the same skill taxonomy was used for both datasets. This would help readers assess how directly the synthetic results inform the real-data results.","section":"§4"},{"comment":"Because the headline comparisons are between models across conditions, consider plotting the full distribution of AUC values over the 60 repetitions (for example, with box plots or violin plots) rather than only the mean and standard error. This would make the variability and the overlap between LENS and Text-LENS more transparent.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a learning-analytics or educational-data-mining venue and the authors have made the synthetic dataset available, which is commendable. The main concern is external validity: the cleanest evidence for the central mechanism is generated by the authors' own LLM prompt, and the real-data evidence is weakened by OCR noise. I recommend asking for a real-text validation or a much more detailed characterization of the Eedi OCR quality before publication. The absence of significance testing should also be addressed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nText-LENS is a straightforward extension of LENS: swap item-ID embeddings for mathBERT text embeddings, then test on seen/unseen items in both directions. The paper does a few things well. The eight-condition protocol is clear, and 60 repetitions give stable estimates. The randomization control is the right check for the synthetic data: scramble the text-difficulty link and performance drops to chance, confirming the model is using that injected signal. They also contribute a new synthetic dataset, useful for controlled studies even if it is not real content.\n\nThe main result is credible: Text-LENS matches LENS on seen queries and beats it on unseen queries, with the largest gap in the off-target unseen condition. That is a real contribution to the cold-start problem in educational assessment. The paper is also candid about its limits—the Discussion explicitly says we do not know how much real items encode difficulty in text compared to an LLM prompted to do so.\n\nThe soft spots are real but not deal-breakers. First, the evidence for the headline generalization claim rests most heavily on LLM-Sim, where the difficulty-text link is built into the generation prompt and the IRT simulation. The randomization control proves the model exploits that link, not that natural math text carries the same information. Eedi is consistent, but the OCR is noisy and spatial/graphical content is lost, so it is a weaker check. Second, the only baseline is LENS itself. Prior work on text-based difficulty estimation and text-aware knowledge tracing is cited but not compared; you cannot tell whether the gain is specific to this architecture or generic to adding text. Third, there are no significance tests or confidence intervals beyond standard error bars—fine for a first look, thin for a generalization claim. Minor: no code or data link is actually given in the text despite the footnote saying the dataset is available.\n\nNone of this undermines the proof-of-concept. The paper knows what it can and cannot claim, and the limitation it flags is the load-bearing one. For a serious referee, I would send it out, but I would ask for at least one additional baseline (e.g., IRT with text-predicted difficulties, or a text-aware knowledge tracing model), significance tests or effect sizes, and a cleaner real-data artifact such as multimodal summaries. The LLM-Sim result should be framed as proof-of-concept, not as evidence about real items.\n\nWho is this for? People in educational data mining, psychometrics, and adaptive testing. I would not cite it in my own work this year, but I would bring it to a reading group as an example of a well-executed synthetic evaluation with an honest limitations section.\n\nRecommendation: engage with it; it deserves peer review with requested revisions.","headline":"Text-LENS is a clean, honest proof-of-concept that text embeddings help with unseen assessment items, but its strongest evidence is synthetic and the authors say so; worth reviewing, not ready as is.","tokens_in":8059,"tokens_out":2120,"would_cite":false,"duration_ms":22533,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that replacing item identifiers with question-text embeddings lets an assessment model predict student performance on test items it has never seen, matching or beating the identifier-based model on seen and unseen items.","keywords":["educational assessment","text embeddings","variational autoencoder","item cold start","item response theory","student modeling","unseen item generalization","LLM-simulated data"],"falsifier":"Train Text-LENS on a real assessment dataset with clean item text and item difficulties estimated from large response histories, then evaluate on unseen off-target query items; if the model's AUC is no better than chance while it remains above chance on the LLM-generated data, the claim that text carries enough difficulty information for zero-shot prediction is falsified.","tokens_in":7104,"feed_emoji":"📝","tokens_out":5458,"duration_ms":55480,"temperature":0.7,"pith_summary":"This paper tries to establish that an assessment model can make useful predictions about a student's performance on test items it has never seen, by reading the text of the questions instead of relying only on item identifiers. It replaces the item-ID embeddings in the LENS partial variational autoencoder with embeddings from a pretrained math-language transformer, producing Text-LENS. Across a synthetic dataset and a real public dataset, Text-LENS matches the identifier-based model on items seen in training and beats it on unseen items. The paper argues that text embeddings carry information about item difficulty and skill content, so a new question can be mapped into the same semantic space without retraining.","feed_headline":"Text-aware model predicts scores on questions it has never seen","feed_subtitle":"Replacing item IDs with text embeddings matches old models on known items and beats them on new ones.","key_machinery":"The load-bearing mechanism is the substitution of item-ID embeddings with fixed text embeddings from mathBERT, a pretrained transformer encoder for mathematical language, inside the LENS partial variational autoencoder. Instead of learning a discrete embedding per item from historical response data, Text-LENS uses the transformer to map question text into a continuous semantic space; new items enter that space without retraining, and the encoder and decoder can use text-derived difficulty and skill information to predict responses. The synthetic dataset is constructed so that difficulty labels assigned by a large language model are used to simulate student responses via a three-parameter logistic item response theory model, which lets the paper test exactly what information the model extracts from text.","core_discovery":"The central claim is that text embeddings are sufficient for zero-shot item inference in a partial-VAE assessment model: Text-LENS can take a new query question, embed its text, and predict whether a student will answer it correctly, even when neither the query nor the input items appeared in training. On the synthetic dataset, in the hardest condition where input items target a different skill and the query is unseen, Text-LENS reaches an AUC around 0.58 while the identifier-based baseline is at chance; when the relationship between text and difficulty is randomized, Text-LENS drops to chance, indicating that the improvement comes from reading difficulty out of the text. On the real public dataset, Text-LENS preserves the gain on unseen queries, with a smaller drop that likely reflects skill dependencies among items.","pith_inferences":["If text-difficulty recoverability is the binding constraint, then subjects with richer language than mathematics, such as reading comprehension or science, may show larger gains on unseen items; math text is often diagram-dependent and less linguistically rich.","On real items whose difficulty is estimated from response data rather than assigned by an LLM, the recoverability of difficulty from text is an empirical question, and a clean-text dataset with IRT-estimated difficulties would provide a direct test.","The synthetic-data recipe suggests a general strategy: use LLM-generated items with known difficulty to probe which textual features a model relies on before deploying it on real assessment items."],"forward_implications":["New assessment items can be scored immediately from their text, reducing or eliminating the field-testing period before an item becomes operational.","Instructors can modify or introduce questions during an assessment while maintaining predictive accuracy, enabling more dynamic and adaptive testing.","Models no longer need to be retrained from scratch when the item pool grows, because a new item's text embedding is available at inference time.","The same architecture works whether input items are seen or unseen, because text provides a common representation for old and new items.","The advantage on unseen off-target queries is bounded by how much difficulty information the text itself carries; when the text-difficulty link is removed, the advantage disappears."],"supporting_citations":[{"why":"Supplies the LENS partial variational autoencoder architecture that Text-LENS extends and the seen-item baseline it must match.","marker":"[3]"},{"why":"Supplies the pretrained transformer used to produce text embeddings that replace item-ID embeddings in Text-LENS.","marker":"[4]"},{"why":"Defines the three-parameter logistic IRT model used to simulate student responses in the synthetic dataset.","marker":"[10]"},{"why":"Supplies the real-world student response and item content data used for the non-synthetic evaluation.","marker":"[19]"},{"why":"Documents the field-testing and cold-start problem that motivates the need to predict from unseen items.","marker":"[6]"},{"why":"Surveys prior evidence that item difficulty can be estimated from text, which underpins the expectation that text embeddings carry difficulty information.","marker":"[1]"}],"fun_headline_variants":["Text embeddings let AI predict scores on unseen questions","AI reads item text to predict questions it never saw","No IDs needed: Text-only model predicts new items","Text-aware model beats ID baseline on new questions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the difficulty of a question is actually readable from its text: for the synthetic data this is true by construction, for the real data it is only available through noisy OCR, and if real item text does not encode difficulty strongly, the gains on unseen items will not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Text embeddings let AI predict scores on unseen questions","AI reads item text to predict questions it never saw","No IDs needed: Text-only model predicts new items","Text-aware model beats ID baseline on new questions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00028,"raw_usage":{"total_tokens":1607,"prompt_tokens":835,"completion_tokens":772,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":451,"completion_tokens_details":{"reasoning_tokens":723}},"tokens_in":451,"tokens_out":772,"duration_ms":8981,"temperature":1.0,"reasoning_tokens":723,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:25:22.482616+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train Text-LENS on a real assessment dataset with clean item text and item difficulties estimated from large response histories, then evaluate on unseen off-target query items; if the model's AUC is no better than chance while it remains above chance on the LLM-generated data, the claim that text carries enough difficulty information for zero-shot prediction is falsified.","supporting_citations":[{"cited_title":"The base LENS model uses a partial variational auto-encoder (V AE) architecture to produce probabilistic representations of students based on assessment performance [3]","cited_arxiv_id":null,"evidence_quote":"Supplies the LENS partial variational autoencoder architecture that Text-LENS extends and the seen-item baseline it must match."},{"cited_title":"significantly vary the wording of the questions, such as the type of information included in the question and the phrasing of the task itself","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained transformer used to produce text embeddings that replace item-ID embeddings in Text-LENS."},{"cited_title":"Educational Measurement: Issues and Practice 44(1), 73–83 (2025)","cited_arxiv_id":null,"evidence_quote":"Defines the three-parameter logistic IRT model used to simulate student responses in the synthetic dataset."},{"cited_title":"In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","cited_arxiv_id":null,"evidence_quote":"Supplies the real-world student response and item content data used for the non-synthetic evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the field-testing and cold-start problem that motivates the need to predict from unseen items."},{"cited_title":"cold start","cited_arxiv_id":null,"evidence_quote":"Surveys prior evidence that item difficulty can be estimated from text, which underpins the expectation that text embeddings carry difficulty information."}],"review_version":1}