{"id":"aaf945c7-75fd-4c0a-8588-71f4eb892ac7","arxiv_id":"2412.17427","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An LLM-based scorer predicts human-judged contextual informativeness in children's stories with a Spearman correlation of 0.4983, outperforming baselines and generalizing to adult text.","lead":"The authors introduce a dataset and task for measuring how well children's stories convey the meaning of vocabulary words, and they show that a large language model can estimate this contextual informativeness better than existing baselines. The work matters for automatically generating or selecting stories that help children learn new words.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The gold standard is an untested proxy: cloze-guess similarity to the target is never checked against independent informativeness ratings, even though the paper's own Kapelner re-annotation creates the data for that check.","rationale":"The reader's weakest assumption identifies the same construct-validity issue, and the Kapelner data provides a concrete, cheap test. I considered other concerns—non-independence of word instances, test-set-selected baselines, and no inter-annotator agreement—but they affect statistical confidence or reporting, not the identity of what is being measured. The proxy validity is the single point on which the entire evaluation rests. If the cloze-based gold is not a valid measure of contextual informativeness, then the headline numbers in Tables 1 and 2 do not support the paper's central claim, regardless of p-values. The concern is addressable; it does not require throwing out the dataset, so CONDITIONAL remains the right verdict, and my read does not change it.","tokens_in":16011,"tokens_out":4984,"duration_ms":45980,"concrete_test":"On the 200 Kapelner contexts annotated under both schemas, compute Spearman and Pearson correlations between (a) the original Likert-scale contextual informativeness scores and (b) the cloze-based gold scores from Section 5. Report the correlation, a scatterplot, and the top-20 disagreements. If the correlation is weak (say rho < 0.5), the gold standard's construct validity is unsupported and the central claim needs to be re-scoped. As a second, stronger check, collect Likert-style informativeness ratings on a random subset of 50 child-directed stories from the new dataset and compare to the cloze-based gold there as well.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 defines the gold score as the average ConceptNet Numberbatch cosine similarity between three annotators' guesses at masked target words and the true target. This operationalizes 'contextual informativeness' as guessability in embedding space. The paper never validates this proxy: no inter-annotator agreement is reported, and no correlation with an independent human rating of contextual informativeness is given. The absence is conspicuous because the paper already holds the needed data. In Section 5 and Appendix D, 200 contexts from Kapelner et al. (2018) are annotated under the cloze schema while original Likert-scale informativeness ratings exist for the same contexts. The authors report model performance under each schema separately (Tables 2 and 5) but never the direct correlation between the two gold standards. If the cloze proxy agrees with Likert ratings, the construct validity is supported; if not, the central result reduces to 'Gemini matches annotators' guesses in ConceptNet space' rather than measuring contextual informativeness. The shared embedding metric makes this especially consequential: both human gold and all model scores are computed as cosine similarity to the target in the same embedding space, so any systematic bias in that metric (e.g., rewarding broad hypernyms like 'animal' for 'dog') is shared by both sides and can inflate agreement without reflecting true semantic support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the task of measuring contextual informativeness in child-directed text: given a generated children's story and a target vocabulary word, predict how much semantic support the surrounding context provides. The authors construct a dataset by masking target words in 153 LLM-generated stories and scoring three annotators' guesses against the true target using ConceptNet Numberbatch cosine similarity. They propose two methods: a RoBERTa-based masked-language-model approach (RoBERTa-mult) and a Gemini-based prompting approach, and compare them with embedding-similarity baselines and the prior Nam et al. (2022) model. On the child-directed dataset, Gemini achieves Spearman's rho = 0.4983 versus 0.3534 for the strongest baseline; on a re-annotated subset of the adult-directed Kapelner dataset, Gemini again outperforms the baselines (rho = 0.3908). The authors conclude that LLM-based prediction of masked words is a promising automatic measure of contextual informativeness.","tokens_in":16262,"tokens_out":3146,"duration_ms":34770,"significance":"The paper addresses a real and understudied problem: automatically evaluating whether generated children's stories actually convey the meaning of target vocabulary items. The task definition is clean, and the dataset release is a useful resource. The main empirical claim, that an LLM can predict the proposed gold-standard scores better than simple embedding baselines, is plausible and supported by the reported correlations. The paper also demonstrates generalization to adult-directed text, which strengthens the claim that the method captures something beyond idiosyncrasies of the child-directed corpus. Strengths include a clear experimental setup, a publicly released dataset, and an explicit acknowledgement of the limitation that contextual informativeness is not the same as learnability. However, the gold standard is an unvalidated embedding-similarity proxy, and several statistical details of the evaluation are not fully addressed. These issues are fixable but need to be resolved before the central claim can be fully accepted.","major_comments":[{"comment":"The gold standard is defined as the average ConceptNet Numberbatch cosine similarity between human guesses at masked target words and the true target word. This operationalizes 'contextual informativeness' as guessability in embedding space, but the paper never validates this proxy against an independent human rating of contextual informativeness. The necessary data already exist in Section 5 and Appendix D: 200 contexts from Kapelner et al. (2018) are re-annotated under the cloze schema while original Likert-scale informativeness ratings are available for the same contexts. The authors should report the correlation between the two gold standards on these 200 contexts. If the correlation is high, construct validity is supported; if low, the central result reduces to 'Gemini matches human guesses in ConceptNet space' rather than measuring contextual informativeness. The shared embedding metric makes this especially consequential: both human gold scores and all model scores are computed as cosine similarity to the target in the same embedding space, so systematic biases in that metric can inflate agreement without reflecting true semantic support. I also note that no inter-annotator agreement is reported for the child-directed dataset; the paper should report agreement among the three annotators.","section":"§3.1, §5, Appendix D"},{"comment":"The reported p-values for Spearman's rho assume that the 765 word-instances are independent observations. However, these instances are nested in 153 stories, each with five target words, so instances from the same story are likely correlated. The p-values in Table 1 (e.g., 3.39e-49 for Gemini) are therefore optimistic, and the comparison between Gemini (0.4983) and RoBERTa-mult (0.4601) may not be statistically significant once clustering is accounted for. The authors should use a cluster-robust or bootstrap procedure that resamples whole stories, or fit a mixed-effects model with story as a random effect, to provide valid significance estimates. The same issue applies to Table 2 for the Kapelner re-annotation.","section":"§4.4, Table 1"},{"comment":"The semantic-similarity baselines are reported only for the best-performing configuration selected from a set of hyperparameters tested on the same evaluation data: Appendix C states that 'we initially experiment with multiple thresholds as well as context window sizes, including only the best performing in the results.' Selecting the best baseline configuration on the test set is a form of test-set fitting and makes the baseline numbers optimistic upper bounds; it does not invalidate the main conclusion, since the proposed methods still outperform these optimistic baselines, but it weakens the claim that the baselines are representative. The authors should either report results for all configurations in the main text, or use a validation split for hyperparameter selection, or clearly disclose that the baselines were tuned on the test set.","section":"§4.2, Appendix C"},{"comment":"The generalization experiment is based on only 200 re-annotated contexts, each annotated by two annotators, and no inter-annotator agreement is reported for this re-annotation either. More importantly, the comparison between models on this subset is not accompanied by any test of whether the difference between Gemini (rho=0.3908) and the strongest baseline (Nam et al.+WordNet, rho=0.3660) is significant. Given the small sample size, the claim that Gemini 'outperforms all baselines' on adult-directed text should be supported by a confidence interval or a significance test that accounts for the nesting of instances in contexts and for annotator variability.","section":"§5, Table 2"}],"minor_comments":[{"comment":"The word 'exasperated' in 'This issue is exasperated' should be 'exacerbated.'","section":"§1"},{"comment":"The headers 'N-significance' and 'r-significance' are ambiguous; 'p-value for Spearman's rho' and 'p-value for Pearson's r' would be clearer.","section":"§4.3, Tables 1 and 2"},{"comment":"The reproducibility details for Gemini are incomplete: the paper should report the exact model version, decoding parameters (temperature, max tokens), and full prompt template, since these affect the predictions.","section":"Appendix B"},{"comment":"The formal definition includes Ti as part of the input, but it is not clear whether the model is given the target word string (e.g., its spelling) or only the instruction to ignore other blanks; the experiments appear to mask the target, so the definition should be aligned with the implementation.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is solidly within scope for a CS/CL venue and the core empirical finding is likely of interest. The main risk is construct validity: the gold standard is an embedding-similarity proxy that is never checked against an independent rating, and both human and model scores share the same metric. The nested-data issue is also important. Both are fixable with additional analyses that the authors already have the data for, so I recommend major revision rather than rejection. I would also encourage the editor to ask the authors to make the code for reproducing all tables available, since the dataset is already public."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper's real contributions are the child-directed dataset (765 target-word instances across 153 generated stories, annotated with a cloze schema) and the finding that an LLM-based scorer (Gemini) correlates with human guesses at 0.4983 Spearman, beating strong baselines and generalizing to adult-directed text. That's worth having: the task definition is clear, the baselines are reasonable, and the code and data are released.\n\nThe soft spots are real but mostly addressable. The biggest is the gold standard. The paper defines contextual informativeness as the average ConceptNet Numberbatch cosine similarity between annotators' guesses and the target. That's a guessability proxy, and it's never validated against an independent human rating of informativeness. The paper does not report inter-annotator agreement, so we don't know how reliable the guesses are. Most tellingly, the paper re-annotates 200 contexts from Kapelner et al., which already have Likert-scale informativeness ratings, but it never reports the direct correlation between the two gold standards. That is a cheap, obvious check, and its absence weakens the central claim. The shared embedding metric makes this worse: both the human gold and all model scores use the same cosine similarity to the target, so any bias in that metric is shared and can inflate agreement.\n\nTwo smaller statistical points: the 765 instances are nested in 153 stories, but the significance tests treat them as independent; and the reported baselines are the best-performing thresholds/window sizes selected on the test set. Neither is fatal—the effect sizes are large enough that the main rankings likely survive—but both should be discussed.\n\nWho should read this: people working on evaluating generated children's content, cloze-based metrics, or vocabulary learning. The paper deserves a serious referee. I'd send it out, but ask the authors to validate the proxy against the Kapelner Likert labels, report annotator agreement, and rerun with clustered tests. If the proxy validation goes the wrong way, the claim should be reframed as 'the model predicts human cloze-guess similarity,' not 'contextual informativeness.' As is, the result is a solid empirical baseline but not yet a validated measure.","headline":"Useful dataset and a plausible LLM scorer, but the gold-standard proxy is unvalidated and the authors already hold the data to check it.","tokens_in":16792,"tokens_out":4250,"would_cite":true,"duration_ms":36638,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a masked-word-guessing language model scores how well a children's story explains a target vocabulary word, correlating with human judgments at a Spearman correlation of 0.4983.","keywords":["contextual informativeness","child-directed text","vocabulary learning","cloze task","masked language models","large language models","semantic similarity","story generation"],"falsifier":"Compare the cloze-based gold standard with a direct rating-scale judgment of contextual informativeness on the same stories: if passages where annotators can produce a close synonym score much lower than passages that merely mention words commonly associated with the target, the embedding proxy misorders contextual support and the reported correlation is with the wrong target.","tokens_in":15837,"feed_emoji":"📖","tokens_out":23214,"duration_ms":185554,"temperature":0.7,"pith_summary":"The paper introduces a new task: measuring contextual informativeness in children's stories, i.e., how much the surrounding text clarifies the meaning of a target vocabulary word. To support the task, the authors annotate 153 LLM-generated children's stories by masking all target words and asking human readers to guess them, scoring each guess by its ConceptNet embedding similarity to the true word. They then propose automating the same procedure: replace target words with masks, have a language model guess them, and score the guesses with the same embedding similarity. Their Gemini-based method reaches Spearman's $\\rho = 0.4983$ with the human gold standard, beating the strongest baseline at $\\rho = 0.3534$, and the same approach also outperforms baselines on adult-directed text. If the finding holds, story generators and vocabulary interventions can rank candidate stories by how well they support target vocabulary words.","feed_headline":"Scores story contexts at 0.50 with a word-guessing LLM","feed_subtitle":"Outperforms all baselines at ranking how much a story explains a target word, on child and adult text.","key_machinery":"The central object is the modified cloze task with embedding-similarity scoring. Both the human gold standard and the automated model score share the same operation: replace target words with blanks, produce a guess for each blank, and compute the cosine similarity between the guess and the true target word using ConceptNet Numberbatch 19.08 embeddings, averaged over guesses (for humans, three annotators per story). The automated methods instantiate the guessing step with RoBERTa, a masked language model, and Gemini, a general-purpose LLM prompted to ignore other blanks and guess only the masked target; RoBERTa combines multiple occurrences of a word by lemmatizing predictions and summing probabilities per lemma. This machinery makes the model's prediction task identical to the annotator's task, so the same similarity metric measures both and the model's performance is directly comparable to the human gold standard.","core_discovery":"The paper claims that contextual informativeness of a target word in a passage can be measured by cloze-style prediction: mask every occurrence of the target, ask a model to fill the blank, and take the ConceptNet Numberbatch cosine similarity between the model's predicted word and the true target as the informativeness score; a higher similarity means the context lets the predictor narrow the semantic space. Defined this way, the task has a concrete gold standard: human annotators perform the same cloze guesses, and the averaged similarity of their guesses to the target forms the score. On the new child-directed dataset of 765 target words in 153 stories, Gemini's predictions correlate with the human scores at Spearman's $\\rho = 0.4983$, compared with $\\rho = 0.3534$ for the strongest baseline; RoBERTa-mult reaches $\\rho = 0.4601$. Re-annotating 200 contexts from an existing adult-directed dataset with the same schema, the method again beats all baselines at $\\rho = 0.3908$. The paper's conclusion is that a pretrained language model can approximate human judgments of contextual support well enough to be useful in evaluating and generating educational stories, and that the approach transfers across text domains.","pith_inferences":["A testable extension, which the paper flags for future work, is to compare automatic scores with children's actual vocabulary gains from the same stories; that would test whether informativeness as scored here predicts learning.","Because the gold standard averages only three annotators, measuring inter-annotator agreement would provide an upper bound on how well any automatic method can correlate with it.","The method inherits the coverage of ConceptNet Numberbatch, so validating the same scoring on rare child-directed words or in other languages is a direct test of how far it generalizes beyond the reported domains."],"forward_implications":["Generated children's stories can be automatically screened for how well they support each target vocabulary word, without requiring new human ratings for every story.","The same LLM-based scorer transfers to adult-directed text, so a single method can be applied across child and adult reading materials.","The new dataset gives the field a shared benchmark of 765 target-word instances with human cloze-based informativeness scores.","Because both the LLM and RoBERTa beat all embedding-similarity baselines, replacing literal context similarity with masked-word prediction is what drives the improvement."],"supporting_citations":[{"why":"Supplies the 180 LLM-generated children's stories and the target-word setup that the new dataset annotates.","marker":"Valentini et al., 2023"},{"why":"Defines the earlier contextual-informativeness task and contributes the adult-directed texts used in the generalization experiment.","marker":"Kapelner et al., 2018"},{"why":"Provides the attention-based model and adult-trained baseline whose poor transfer to child-directed data motivates the new approach.","marker":"Nam et al., 2022"},{"why":"Provides the ConceptNet Numberbatch 19.08 embeddings used to compute both the gold-standard scores and the automated similarity scores.","marker":"Speer et al., 2017"},{"why":"Underlies the RoBERTa-mult approach that predicts masked targets and combines per-lemma probabilities.","marker":"Liu et al., 2019b"},{"why":"Supplies the Gemini model whose prompted masked-word guessing achieves the highest correlations.","marker":"Gemini-Team et al., 2024"},{"why":"Defines the cloze task that the paper's annotation schema adapts into a partial-credit, multi-target scoring procedure.","marker":"Taylor, 1953"}],"fun_headline_variants":["LLM scores context informativeness at 0.50 Spearman","Word-guessing LLM beats all baselines at context scoring","Cloze LLM measures word context, 0.50 Spearman vs 0.35 baseline","LLM cloze test beats baselines at ranking story context informativeness","New LLM method hits 0.50 correlation on word context quality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the average cosine similarity between human annotators' guesses of a masked word and the true target word, computed in ConceptNet Numberbatch space, actually measures the contextual support a child receives.","fun_headline_variants_meta":{"raw":{"variants":["LLM scores context informativeness at 0.50 Spearman","Word-guessing LLM beats all baselines at context scoring","Cloze LLM measures word context, 0.50 Spearman vs 0.35 baseline","LLM cloze test beats baselines at ranking story context informativeness","New LLM method hits 0.50 correlation on word context quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001381,"raw_usage":{"total_tokens":5586,"prompt_tokens":932,"completion_tokens":4654,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":4555}},"tokens_in":548,"tokens_out":4654,"duration_ms":29751,"temperature":1.0,"reasoning_tokens":4555,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:26:44.924762+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the cloze-based gold standard with a direct rating-scale judgment of contextual informativeness on the same stories: if passages where annotators can produce a close synonym score much lower than passages that merely mention words commonly associated with the target, the embedding proxy misorders contextual support and the reported correlation is with the wrong target.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 180 LLM-generated children's stories and the target-word setup that the new dataset annotates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the earlier contextual-informativeness task and contributes the adult-directed texts used in the generalization experiment."},{"cited_title":"An Attention-Based Model for Predicting Contextual Informativeness and Curriculum Learning Applications","cited_arxiv_id":"2204.09885","evidence_quote":"Provides the attention-based model and adult-trained baseline whose poor transfer to child-directed data motivates the new approach."},{"cited_title":"cloze procedure","cited_arxiv_id":null,"evidence_quote":"Defines the cloze task that the paper's annotation schema adapts into a partial-credit, multi-target scoring procedure."}],"review_version":1}