{"id":"faebb91b-1954-44ed-9e3b-d28f2a2aa852","arxiv_id":"2412.10105","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MALAMUTE is a 116k-prompt cloze-style dataset derived from 71 university textbooks in three languages, used to probe language models' fine-grained subject knowledge.","lead":"Researchers built a large multilingual quiz dataset from university textbooks, where each question is a sentence with a key term blanked out. The dataset lets educators and developers test whether language models actually know specific course concepts rather than broad topics.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-subdomain prompt quality is unmeasured; reported knowledge gaps may reflect prompt ambiguity rather than model knowledge.","rationale":"The reader's weakest assumption—that the automated pipeline yields unambiguous, specific cloze prompts—is indeed the most load-bearing condition for the paper's central claim. The paper's own QC numbers (92.9%/93.9% concept fidelity on a 495-prompt sample) directly acknowledge that a non-trivial fraction of prompts are not fully specific, but the analysis never examines where those failures occur. Because the dataset's advertised advantage is fine-grained subdomain evaluation, per-subdomain quality is what matters, and the 9-per-subdomain sample cannot estimate it. I also note the Spanish/Polish QC gap, which further weakens cross-lingual comparisons. The best-prompt selection issue raised by the reader is real but secondary; it inflates absolute scores, whereas prompt ambiguity can change relative rankings. My proposed test is a concrete, reproducible way to decide whether the granularity conclusions survive. If the test shows stable rankings, the paper's claims stand; if not, the benchmark's comparative results need qualification. This does not change the reader's CONDITIONAL verdict—the dataset is a solid contribution, but the evaluation claims need this additional evidence before being taken at face value. I therefore mark the verdict as unchanged.","tokens_in":36212,"tokens_out":2951,"duration_ms":31310,"concrete_test":"Draw a new stratified sample of 50 prompts per subdomain for English (N≈2,750), plus similarly sized samples for Spanish and Polish, and have annotators rate specificity and grammaticality using the same rubric as Appendix C. Compute per-subdomain and per-language quality rates. Then re-run the Llama 3.1 405B and GPT-4o evaluations on only the prompts rated unambiguous in each subdomain. If the ordering of subdomains (e.g., Biology vs. Calculus 3) changes, or if the gap between high- and low-scoring subdomains narrows by more than a few points, the granularity-based conclusions are not robust. This is feasible because the code and data are released.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central granularity claim depends on uniform prompt quality across subdomains. The paper's quality control (Appendix C) samples only 9 prompts per subdomain, all English, and reports aggregate fidelity rates of 92.9% (paragraph) and 93.9% (sentence). With n=9 per subdomain, a single ambiguous prompt shifts the subdomain estimate by ~11 percentage points, so the true per-subdomain ambiguous-prompt rate is essentially unknown. If the ~6–7% ambiguous prompts cluster in particular subdomains—especially those with many multi-word terms or near-synonyms, such as Calculus 3, Marketing, or Business Law—the reported ~50% scores could be depressed relative to cleaner subdomains like Biology or American Government. The Spanish and Polish portions receive no human QC at all, yet the paper reports large cross-lingual gaps. Consequently, the headline finding 'significant subject-specific knowledge gaps' is not yet separated from 'subdomain-dependent prompt ambiguity.' The dataset itself may be valuable, but the comparative conclusions drawn from it are only as sound as the prompt-quality assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MALAMUTE, a multilingual cloze-style probing dataset constructed by extracting glossary-term definitions from 71 OpenStax textbooks and masking each term, producing sentence-level and paragraph-level prompts in English, Spanish, and Polish. The authors report 116,887 prompts organized into 8 domains, 55 subdomains, and 33,361 concepts, and they evaluate five masked and eight causal language models, reporting overall accuracy and subdomain-level breakdowns. The central claim is that MALAMUTE is the first education-based cloze-style benchmark and that current models, despite decent overall scores, show large subject-specific and language-specific knowledge gaps.","tokens_in":36390,"tokens_out":7950,"duration_ms":77882,"significance":"If the quality assumptions hold, MALAMUTE would be a valuable community resource: it is the first cloze-style benchmark tied to university-level educational content, it is template-free at the probe level, it provides both sentence-level and paragraph-level variants, and it ships with code and data. The evaluation is a clean measurement rather than a fitted model, so circularity concerns are limited; using the dataset's concept list as the MLM candidate set is a scoring choice rather than a derivation. The main value is as a granular diagnostic tool for educational deployment. The strength of the comparative conclusions, however, depends on prompt quality being uniform across subdomains and languages, and the current evidence for that uniformity is weak.","major_comments":[{"comment":"Table 3's #Concepts column sums to 50,008 (42,172 for English alone), not the 33,361 distinct concepts claimed in the Abstract and Section 3.4. Likewise, the #Prompts column sums to 117,340 (100,711 English), not 116,887 (100,258 English). The manuscript should define whether the table reports unique concepts/prompts or textbook-specific occurrences, and the totals should be reconciled. As written, the headline dataset statistics are internally inconsistent.","section":"Section 3.4 / Table 3"},{"comment":"The quality-control study samples 9 prompts per subdomain and is English-only. With n=9, a single ambiguous prompt changes the estimated specificity rate for that subdomain by 11 percentage points, so the aggregate 92.9%/93.9% specificity figures do not establish that subdomain-level scores are comparable. If the roughly 6-7% ambiguous prompts cluster in subdomains such as Calculus 3, Marketing, or Business Law, the reported knowledge gaps in Section 5.4 could be artifacts of prompt ambiguity rather than model knowledge. The authors should provide per-subdomain QC results with confidence intervals, expand the QC sample, or explicitly qualify subdomain comparisons as exploratory. The absence of any human QC for the Spanish and Polish portions also weakens the cross-lingual comparisons in Section 5.1.","section":"Appendix C / Section 5.4"},{"comment":"The filtration heuristics in Appendix B remove prompts starting with 'this/that/we/also', prompts under five words, prompts referencing figures, prompts with action verbs, and parenthesized labels. These rules are content-sensitive: math definitions frequently start with 'we', and business or computer-science terms often carry parenthetical acronyms. The paper reports no per-subdomain removal counts and no comparison of filtered versus retained prompt difficulty. Without such analysis, subdomain-level performance differences in Section 5.4 may reflect differential filtration rather than model knowledge. The authors should report filtration rates per subdomain and test whether the main subdomain rankings are robust to alternative filter settings.","section":"Appendix B"},{"comment":"The Abstract and Introduction describe the probes as 'expert-written, peer-reviewed.' Section 3.2 shows that the prompts are extracted automatically from OpenStax textbook prose: the textbook authors selected the glossary terms, and the peer review applies to the textbooks, not to the masked prompts themselves. This wording overstates the human curation of the probes and should be corrected, for example to 'based on expert-written, peer-reviewed textbook definitions.'","section":"Abstract / Section 1"},{"comment":"The reporting of CLM results is internally inconsistent. Table 4's note says causal LM results are shown as [prompt 1, prompt 2, prompt 3], but the GPT-4 family and Llama-3.1-405B cells contain single numbers, and Section 4.2 states that these models were evaluated with only the in-context prompt selected after a pilot on smaller models. Since Section 5.2 shows that prompt choice can change a subdomain score by 18 points (Spanish Business Statistics: 23.0% vs. 5.0%), the single-number presentation obscures the uncertainty in the headline model rankings. The authors should report which prompts contribute to each number and give per-prompt or variance information for all models.","section":"Table 4 / Section 4.2"}],"minor_comments":[{"comment":"Appendix A.1 is incomplete: it displays only 'pti' and does not specify the actual instruction text for Prompt 1, unlike Prompts 2 and 3.","section":"Appendix A.1"},{"comment":"The reference list contains placeholder links for Kalo (2022) and Nayak (2023); these should be replaced with full bibliographic entries.","section":"References"},{"comment":"There are several typos, including 'Managarial Accounting' (Tables 5-7), 'Entrepeneurship' (Table 2), and 'IW A' (Table 2).","section":"Tables 2, 5-7"},{"comment":"The column headers 'Words (Pg.)' and 'Words (Sent.)' should be spelled out or defined in the caption for clarity.","section":"Table 3"},{"comment":"The phrase 'establishing a new higher lower-bound of knowledge' is confusing and should be reworded for precision.","section":"Section 1"}],"recommendation":"major_revision","confidential_remarks":"This is a potentially useful resource paper, but the internal arithmetic inconsistency in Table 3 and the thin QC evidence for subdomain-level comparisons need to be addressed before the comparative findings can be trusted. I do not see grounds for rejection if these issues are fixed with additional analysis or appropriately qualified claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Best thing about this paper is the artifact, not the argument. MALAMUTE is the first cloze benchmark aimed at education with a domain-subdomain-concept hierarchy, template-free prompts, sentence- and paragraph-level variants, and three languages. That is a real gap in the ecosystem, and pulling it from OpenStax is a sensible, reproducible move. The pipeline is documented in detail, the release includes code and data, and the evaluation spans a reasonable set of MLMs and CLMs. If I needed a curriculum-aligned knowledge probe next year, I would look here first.\n\nThe soft spots are all in the evaluation layer. The QC sample is 495 English prompts, 9 per subdomain, with no Spanish or Polish annotation at all. The paper reports 92.9% paragraph-level specificity overall, but with n=9 per subdomain a single ambiguous prompt moves the subdomain estimate by 11 points. The reported gaps—Calculus 3, Marketing, Business Law at ~50% versus 80% for Biology—could partly be an artifact of prompt ambiguity clustering in those subdomains. The stress-test note lands; this is the load-bearing weakness. Also, the \"best prompt\" selection for large CLMs is a form of selection bias, and the absence of error bars makes the comparative claims look firmer than they are. The authors acknowledge the apples-to-oranges MLM/CLM comparison, which is honest.\n\nOne smaller quibble: the abstract says \"expert-written, peer-reviewed probes,\" but the probes are extracted from textbook paragraphs; what is expert-vetted are the textbook terms. That is a labeling overreach, not a fatal flaw.\n\nThe dataset deserves a serious referee. The artifact is solid, the construction is transparent, and the concerns are addressable: expand QC per subdomain, add a held-out annotation sample for Spanish and Polish, and report confidence intervals or at least per-subdomain sample sizes. Who is this for: anyone building educational QA or knowledge-probing benchmarks, and people studying LM curriculum knowledge. I would expect heavy revision but not rejection on the dataset itself.","headline":"MALAMUTE is a genuinely useful new resource, but its headline knowledge-gap numbers lean on a thin QC sample.","tokens_in":36901,"tokens_out":1696,"would_cite":true,"duration_ms":18648,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MALAMUTE, the first education-based multilingual cloze-style dataset, extracts 116,887 probes from 71 university textbooks and shows that current language models have large subject-specific knowledge gaps.","keywords":["cloze-style probing","education-based benchmark","multilingual evaluation","knowledge probing","template-free prompts","language model evaluation","textbook-derived dataset","subject-specific knowledge gaps"],"falsifier":"Have three domain-expert annotators label a stratified random sample of, say, 2,000 prompts from each language and subdomain for whether the masked term is uniquely inferable, and compare the ambiguity rate across subdomains; if ambiguity varies strongly, the reported subdomain ranking of models could change after excluding ambiguous prompts.","tokens_in":35989,"feed_emoji":"🎓","tokens_out":8182,"duration_ms":77052,"temperature":0.7,"pith_summary":"This paper builds a benchmark that tests whether language models know the specific concepts taught in university courses, not just broad general knowledge. It introduces MALAMUTE, a cloze-style dataset of 116,887 prompts mined from 71 university-level open textbooks in English, Spanish, and Polish, organized into 8 domains, 55 subdomains, and 33,361 concepts. The authors argue that this fine granularity matters for classrooms: a model can look strong on averages while failing whole courses. Their evaluations show exactly this pattern, with the strongest model tested scoring 67.0% overall on English paragraph-level prompts but dropping to roughly 50% on Calculus 3, Marketing, and Business Law. The paper's point is that education-based, template-free, concept-level probing is a sharper tool for deciding whether a model is safe to use with students.","feed_headline":"116K textbook probes reveal LLMs' course-level blind spots","feed_subtitle":"First education-based cloze benchmark spans 71 textbooks and 3 languages; top models fall to ~50% on calculus, marketing, and business law.","key_machinery":"The dataset is itself the machinery. It rests on the hierarchy built into the source open textbook library: each book belongs to one of eight domains, each domain splits into subdomains (roughly one college course each), and each subdomain's glossary terms define concepts tied to specific textbook sections. For every concept term, the pipeline scrapes the paragraph containing the term, creates a paragraph-level prompt by masking all occurrences of the term and hiding other occurrences as [HIDDEN], and creates a sentence-level prompt by splitting the paragraph and keeping only sentences that contain the term. Regular-expression and part-of-speech filters remove prompts with verbs like 'describe' or 'solve', references to figures, digit labels, and short or deictic openings; the surviving 116,887 prompts form the probe set. Evaluation uses top-k accuracy over an entity list for masked language models and best sub-span accuracy across three prompt designs for causal language models.","core_discovery":"MALAMUTE is claimed to be the first education-based cloze-style knowledge probing dataset, and the first probing dataset to pair sentence-level and paragraph-level template-free prompts for the same concepts. Each prompt is made by taking a glossary term from the source textbooks' editorial structure and replacing it with [MASK]; paragraph-level prompts also hide other occurrences of the same term as [HIDDEN]. The paper reports that on this dataset current masked and causal language models show substantial knowledge gaps at subdomain level even when aggregate scores look acceptable, and that paragraph context consistently raises accuracy relative to sentence context across nearly all subdomains and model types, implying a higher estimated lower bound of knowledge than sentence-only probes give.","pith_inferences":["A testable extension: use MALAMUTE-style extraction on the same textbooks' end-of-chapter exercises or lecture slides to see whether cloze probes predict performance on application-style questions, not just definitional recall.","The roughly 6-7% of prompts judged not to convey the intended concept could be concentrated in particular subdomains; weighting subdomain scores by per-concept ambiguity would make the granular comparisons more robust.","Because the quality-control sample covered only English prompts, the Spanish and Polish subdomain scores rest on the assumption that the extraction pipeline behaves identically across languages; a multilingual annotation sample would test that.","The dataset's entity-list-based top-k evaluation may underestimate masked models, since it restricts predictions to textbook terms that appear as labels anywhere in the collection; richer answer surfaces would change ranks."],"forward_implications":["If MALAMUTE is representative of university curricula, then a model's aggregate benchmark score is not enough to certify classroom readiness; subdomain-level scores must be reported.","Teachers and developers can use the domain-subdomain-concept hierarchy to identify specific courses a model can or cannot support before deployment.","The consistent sentence-versus-paragraph gap implies knowledge estimates rise with context, so future benchmarks should evaluate at both granularities to approximate a model's true lower bound.","The English-Spanish-Polish performance gap means deployment in multilingual classrooms will need language-specific validation, not just multilingual model labels.","Because prompts are generated automatically from an expanding open textbook library, the dataset can be extended and updated as textbooks are revised."],"supporting_citations":[{"why":"Establishes cloze-style probing as a measure of a model's lower bound of factual knowledge and provides the LAMA dataset that MALAMUTE positions against.","marker":"Petroni et al., 2019"},{"why":"Supplies the masked-LM evaluation protocol (top-k accuracy over entity ranking) and the BioLAMA biomedical cloze benchmark that motivates domain-specific probing.","marker":"Sung et al., 2021"},{"why":"Shows that template-free probes can reveal a higher lower bound of knowledge than template-based ones, the design choice at the core of MALAMUTE.","marker":"Shaier et al., 2024a"},{"why":"Source of all 71 textbooks and of the domain-subdomain-concept hierarchy and glossary terms from which prompts are extracted.","marker":"OpenStax, 2012"},{"why":"Contributes the multilingual entity-ranking evaluation framework that MALAMUTE adapts for multilingual masked-model scoring.","marker":"Kassner et al., 2021"},{"why":"Provides the best sub-span accuracy metric used to score causal language model generations.","marker":"Kandpal et al., 2023"},{"why":"Co-supplies the best sub-span accuracy metric and the long-tail knowledge framing that supports the paper's granular-gap analysis.","marker":"Mallen et al., 2023"},{"why":"Defines the BERT and mBERT models whose masked predictions are among the evaluated baselines.","marker":"Devlin et al., 2019"}],"fun_headline_variants":["First education cloze dataset: 116K textbook probes expose LLM blind spots","Template-free, multilingual probes: 116K textbook prompts reveal LLM gaps","New benchmark: 116K course-level probes show LLMs flunk subdomains","LLMs pass broad tests, fail calculus: 116K probes show gaps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument leans on the assumption that the automated extraction and filtering pipeline, checked on a 495-prompt English sample, produces prompts that unambiguously point to the intended concept in every subdomain and every language, even though that sample is about 0.4% of the dataset.","fun_headline_variants_meta":{"raw":{"variants":["First education cloze dataset: 116K textbook probes expose LLM blind spots","Template-free, multilingual probes: 116K textbook prompts reveal LLM gaps","New benchmark: 116K course-level probes show LLMs flunk subdomains","LLMs pass broad tests, fail calculus: 116K probes show gaps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001299,"raw_usage":{"total_tokens":5312,"prompt_tokens":967,"completion_tokens":4345,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":4259}},"tokens_in":583,"tokens_out":4345,"duration_ms":32866,"temperature":1.0,"reasoning_tokens":4259,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:19:45.157083+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have three domain-expert annotators label a stratified random sample of, say, 2,000 prompts from each language and subdomain for whether the masked term is uniquely inferable, and compare the ambiguity rate across subdomains; if ambiguity varies strongly, the reported subdomain ranking of models could change after excluding ambiguous prompts.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of all 71 textbooks and of the domain-subdomain-concept hierarchy and glossary terms from which prompts are extracted."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Co-supplies the best sub-span accuracy metric and the long-tail knowledge framing that supports the paper's granular-gap analysis."}],"review_version":1}