{"id":"a26af28d-4e43-445f-857f-e9b76549222e","arxiv_id":"2501.14877","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new expert-annotated benchmark of students' handwritten math responses shows current vision language models perform poorly, especially on correctness and error-detection questions.","lead":"This paper introduces DrawEduMath, a dataset of 2,030 images of real students' handwritten math work annotated by teachers with descriptions and question-answer pairs. It evaluates four vision language models on these questions and finds they still struggle, especially at judging whether student work is correct.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synthetic/teacher QA ranking similarity rests on a single LLM judge with n=12 validation; other metrics contradict it","rationale":"DrawEduMath is a useful dataset: the images are real student work, teacher annotations are detailed, and the finding that current VLMs struggle on handwritten math is credible. The central claim that synthetic QA can proxy teacher QA for model ranking, however, is not yet established. The paper's evidence for this is Table 7, where the model ordering is identical under the Mixtral judge for both QA sets. But the Mixtral judge's superiority over other metrics was demonstrated on only 500 responses, with aggregate Spearman computed over 12 points, and the paper's own BERTScore/ROUGE-L columns give a different order on teacher QA. This internal inconsistency is a red flag that the 'similar rankings' result may depend on the specific judge used. The reader's weakest assumption captures this accurately. A stronger evaluation—an alternative judge, larger human sample with independent raters, and confidence intervals—would settle it. No change to the verdict is needed: conditional acceptance remains appropriate, with the condition being that the synthetic-QA ranking claim be re-verified before downstream use.","tokens_in":18244,"tokens_out":7443,"duration_ms":62359,"concrete_test":"Recompute the teacher-QA rankings using an alternative judge (e.g., GPT-4o as judge or a normalized exact-match/ROUGE variant after answer canonicalization), and/or collect human annotations on a larger stratified sample (at least 200 teacher-QA responses per model, rated by two independent annotators, preferably teachers). Then compute the rank correlation between synthetic-QA and teacher-QA model orderings under each judge. If the alternative judge or larger human sample yields a different teacher-QA ordering (e.g., Llama not last, or GPT-4o/Claude swap), the abstract's 'similar rankings' claim is a Mixtral-judge artifact. Also bootstrap the 12-point judge-human Spearman to report a confidence interval.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 7's central comparison—synthetic and teacher QA yield similar model rankings—is based entirely on the Mixtral 8x22B judge columns. That judge was validated on only 500 model responses (Section 7), and its claimed agreement with humans is a Spearman rho=0.801 over just 12 aggregate points (4 models x 3 QA sets); per-cell human samples are n=62 (synthetic) and n=63 (teacher), far too small to support a stable ranking. The paper reports no separate judge-human agreement for teacher QA, and no confidence intervals for the 12-point Spearman. Critically, the two reference-based metrics the paper itself reports contradict the LLM judge on teacher QA: BERTScore (0.785) and ROUGE-L (0.253) rank Llama 3.2-11B highest among the four models, while the LLM judge and human sample rank it lowest. Thus the claim that synthetic QA can yield similar model rankings as teacher QA is only as strong as the Mixtral judge; if that judge is biased on long, teacher-written answers, the headline finding is an artifact. With only four models, any small judge bias can reorder the ranking, and no significance test or bootstrap is provided.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DrawEduMath, a dataset of 2,030 images of K-12 students' handwritten math responses, annotated by expert teachers with detailed descriptions and 11,661 QA pairs, plus 44,362 synthetic QA pairs generated from the descriptions by Claude 3.5 and GPT-4o. The authors evaluate four VLMs (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3.2-11B Vision) on both QA sets using BERTScore, ROUGE-L, and an LLM (Mixtral 8x22B) judge, and validate the automatic metrics on a sample of 500 model responses with human annotations. The main findings are that all evaluated models perform far below a practically usable level on teacher QA, with closed models outperforming Llama, and that synthetic QA yields similar model rankings to teacher QA.","tokens_in":18462,"tokens_out":7402,"duration_ms":60643,"significance":"The dataset is a valuable new resource: it contains real student handwriting in naturalistic settings, teacher annotations with a second revision round, PII removal, and a public release. The evaluation is external to model training, so the benchmark is not circular. If the ranking-similarity claim holds, it would justify using synthetic QA for scalable model evaluation in this domain. However, the evidence for that claim is currently weak because it rests on a single LLM judge with limited and pooled validation, and the reference metrics contradict the LLM judge on teacher QA. With the ranking claim either strengthened or substantially qualified, the paper would be a solid contribution.","major_comments":[{"comment":"The conclusion that synthetic QA yields similar model rankings to teacher QA is not supported by the evidence presented. The LLM judge's agreement with humans was estimated on only 500 model responses pooled across 4 models and 3 QA sets, yielding a Spearman correlation of 0.801 over 12 aggregate points; per-cell human samples are n=62 for synthetic and n=63 for teacher, or about 16 per model, which is too small to establish stable per-QA-set rankings. The paper does not report judge-human agreement separately for teacher QA, nor confidence intervals or a bootstrap for the 12-point Spearman. Given that only four models are compared, a small judge bias can reorder the top ranks, so the ranking-similarity claim in the abstract and Section 7 is not robust.","section":"Section 7, 'Assessing Our Automatic Metrics' and Table 7"},{"comment":"The reference-based metrics contradict the LLM judge on teacher QA. For teacher-written QA, BERTScore and ROUGE-L rank Llama 3.2-11B highest (0.785 and 0.253, respectively), whereas the LLM judge and the human sample rank it lowest (0.296 and 0.127). The LLM judge also reverses the top two models between synthetic QA (GPT-4o 0.723 vs. Claude 3.5 0.715) and teacher QA (Claude 3.5 0.657 vs. GPT-4o 0.628). The paper should report ranking similarity per QA set with uncertainty and reconcile these contradictions before claiming that synthetic QAs yield similar rankings.","section":"Section 7, Table 7"},{"comment":"The prompt-selection procedure may distort model rankings. The authors state that they iterated over three possible prompts for each model 'on samples of data' and selected the best-performing one, but they do not specify whether these samples were disjoint from the evaluated QA pairs or whether the selection was done on a held-out set. If the same data were used for both prompt selection and final evaluation, the reported performance is optimistically biased. Please clarify the split and report the chosen prompts.","section":"Section 7, 'Experimental Setup'"}],"minor_comments":[{"comment":"'multimodel' should be 'multimodal'.","section":"Section 2, first paragraph"},{"comment":"The phrase 'both QA sets rank models similarity' should be 'similarly'; moreover, the claim is too strong given the top-two reversal noted in the major comments.","section":"Section 7, 'Results and Findings'"},{"comment":"The caption does not clearly indicate which of the two numbers per model corresponds to synthetic versus teacher QA, and the typesetting of the symbols after 'combined' and 'by teachers' appears to be missing; please repair the table.","section":"Table 8"},{"comment":"The reported QA quality assessment relies on only 100 pairs and shows notable annotator disagreement (e.g., Evaluator A found zero unanswerable questions while Evaluator B found nine); reporting inter-annotator agreement, such as Cohen's kappa, would strengthen this part.","section":"Section 4, 'Quality Assessment of Synthetic QA'"},{"comment":"The human evaluation groups 'Maybe' with 'Yes' and 'Neither' with 'Quite different'; since this binarization is used to validate the LLM judge, a sensitivity analysis under alternative groupings would be informative.","section":"Appendix D.2"}],"recommendation":"major_revision","confidential_remarks":"The dataset is a strong asset, and the annotation pipeline is careful. The main risk is the headline claim about synthetic QA ranking similarity, which depends entirely on the LLM judge. If the authors can provide per-QA-set validation, confidence intervals, and reconcile the contradictions with reference metrics, I would support publication. If not, the claim should be removed or strongly qualified. The benchmark contribution alone is likely sufficient for the journal, but the current framing overstates the ranking evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: DrawEduMath is a genuinely useful new benchmark—real students' handwritten responses to K-12 math problems, with expert teacher descriptions and QA pairs. The finding that current VLMs are far from reliable at judging this kind of work is credible and practically important. The paper deserves a serious referee.\n\nWhat's new and good: The dataset construction is careful. Teachers were paid well, descriptions were revised in a second round, and the QA pairs cover pedagogical aspects (errors, strategies) that most visual math benchmarks ignore. The taxonomy of question types is useful—correctness/errors being hardest is a nice, actionable result. Releasing the data is a real service.\n\nNow the soft spots, in proportion. The main weakness is the claim that synthetic QA yields similar model rankings as teacher QA. That claim rests entirely on the Mixtral 8x22B judge. The judge's agreement with humans was validated on 500 responses, but the Spearman correlation is over just 12 aggregate points (4 models x 3 QA sets), with no confidence interval. The per-cell human samples are tiny (n=62 and 63). More troublingly, the paper's own reference-based metrics contradict the judge on teacher QA: BERTScore and ROUGE-L both rank Llama 3.2-11B highest, while the LLM judge and the human sample rank it lowest. That doesn't necessarily mean the judge is wrong—reference metrics are poor on long teacher answers, as the paper acknowledges—but it means the 'similar rankings' claim is only as strong as the judge's validity, and that validity is not established for teacher QA. The top-2 also flips between GPT-4o and Claude across synthetic and teacher sets. With four models, any small judge bias can reorder things, and no bootstraps or significance tests are provided.\n\nAlso, synthetic QA quality is checked on only 100 pairs, and the generation is derived from the very descriptions that define the gold answers, so the synthetic set is a bit self-referential. Not fatal, but it should be acknowledged more explicitly.\n\nWho is this for? Researchers building or evaluating VLMs for educational applications, and anyone needing a realistic handwriting-heavy math benchmark. The dataset itself deserves to be published. For peer review, I'd send it out, but ask the authors to either soften the synthetic-QA ranking claim to 'similar broad ordering' with appropriate caveats, or add stronger validation of the judge and confidence intervals around the rankings. The benchmark is solid; the secondary claim needs to match its evidence.","headline":"DrawEduMath is a valuable benchmark, but its central ranking claim leans on a single, thinly validated LLM judge; the dataset deserves publication, the claim needs tempering.","tokens_in":19001,"tokens_out":4404,"would_cite":true,"duration_ms":65423,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Vision-language models still cannot reliably interpret students' handwritten math work, and synthetic questions can substitute for teacher-written ones when ranking models.","keywords":["DrawEduMath","handwritten math","vision language models","visual question answering","K-12 math education","teacher annotations","synthetic QA","benchmark evaluation"],"falsifier":"Take the full set of teacher-written QA pairs, or a much larger stratified sample such as 2,000 responses per model, and have human evaluators score each VLM answer; then recompute model rankings and the Spearman correlation between synthetic-based and teacher-based rankings. If human-judged rankings diverge from Mixtral-judged rankings, or if synthetic and teacher rankings no longer agree, the paper's two central findings would fail.","tokens_in":18046,"feed_emoji":"✏️","tokens_out":6481,"duration_ms":51782,"temperature":0.7,"pith_summary":"DrawEduMath is a benchmark built to test whether vision-language models can do what a teacher does when looking at a student's handwritten math work: see what is actually on the page, understand the strategy, and spot errors. The paper introduces 2,030 images of real K-12 students' handwritten responses, each paired with expert teacher descriptions and 11,661 teacher-written question-answer pairs, plus 44,362 synthetic QA pairs generated from those descriptions. Evaluating four vision-language models, the authors find that even the strongest closed models answer only about seven in ten synthetic questions correctly by an LLM judge, and noticeably fewer teacher-written ones, while an open-source model lags far behind. A second finding is that synthetic QA pairs, despite being noisy, rank the models in nearly the same order as teacher-written QA. If these results hold, current vision-language models are not yet reliable enough to interpret students' handwritten work automatically, and synthetically generated questions could be a scalable proxy for expert-written ones when comparing models rather than measuring absolute accuracy.","feed_headline":"Vision models still can't reliably read students' math work","feed_subtitle":"A teacher-annotated benchmark of 2,030 handwritten responses shows wide gaps, yet synthetic questions can rank models similarly.","key_machinery":"The load-bearing object is the benchmark itself, structured as a chain from classroom artifact to evaluable question. Teachers' free-form descriptions of each image are the expert ground truth; from these, an LM pipeline extracts atomic facts, called facets, and rewrites them as close-ended QA pairs, producing 44,362 synthetic questions. A seven-category taxonomy separates higher-level understanding of math and correctness-and-error questions from purely visual or low-level questions, so model scores can be broken down by what is being tested. Evaluation relies on three automatic metrics, ROUGE-L, BERTScore, and an LLM judge based on Mixtral 8x22B, with the LLM judge validated on a 500-response human sample showing Spearman correlation 0.801 over 12 aggregate points. The facet-to-QA mechanism is what makes the benchmark scalable, while the taxonomy and the human-validated judge are what make the scores interpretable.","core_discovery":"The paper's central claim is that current vision-language models, including state-of-the-art commercial ones, leave substantial room for improvement on the task of interpreting students' handwritten math responses, and that a scalable pipeline can produce useful evaluation questions without full teacher effort. The benchmark DrawEduMath pairs each student-response image with a problem statement and teacher annotations; teachers described every image and wrote questions that target pedagogical concerns such as problem-solving strategy, composition of diagrams, labels, and correctness. From the teacher descriptions, language models generated additional QA pairs by decomposing each description into atomic facets and rewriting each facet as a self-contained question and answer. On these questions, even the best model reaches 0.657 LLM-judged correctness on teacher-written QA and 0.715 on synthetic QA, while the open-source Llama model scores roughly half that. Questions about correctness and errors are consistently the hardest for all models. Despite the noise in synthetic QA, the authors find that the relative ordering of models is similar whether teacher-written or synthetic questions are used, suggesting the synthetic set is informative for model ranking.","pith_inferences":["If synthetic QA can rank models reliably, an implicit next step is to generate QA from teacher descriptions at much larger scale and use it for model selection during development, not just for final evaluation.","A deployment consequence the paper leaves implicit is that automated feedback systems on learning platforms should keep a human in the loop for the correctness-and-error category, since that is both the most pedagogically important and the least reliable for current models.","The taxonomy suggests a testable extension: weighting model scores by pedagogical importance, such as correctness and strategy questions rather than paper type, would likely widen the measured gap between models.","Because the ranking claim depends on the LLM judge, a natural extension is to test whether the synthetic-versus-teacher ranking agreement persists under human evaluation on a much larger sample; if it does not, the claim would need to be restricted to automatic-judge settings."],"forward_implications":["If the central finding holds, VLM-based tools for K-12 classrooms should not yet be trusted to give feedback on handwritten work, especially on correctness-and-error questions, where all evaluated models perform worst.","The similarity between synthetic and teacher-written rankings implies that benchmark creators can use LM-generated QA from expert descriptions to compare models cheaply, reserving teacher-written QA for final validation.","The observed failure mode, models giving the mathematically correct answer even when the student's own response is wrong, points to a specific capability gap: reading a student's work as evidence rather than solving the problem independently.","Dark or low-quality images that human annotators could still read caused additional model errors, suggesting that robustness to image quality is a bottleneck for real-world deployment.","The released dataset, with teacher descriptions, QA pairs, and metadata, provides a reusable resource for measuring progress on naturalistic handwritten math understanding."],"supporting_citations":[{"why":"Supplies the caption-to-VQA decomposition method, facets turned into QA pairs, that the synthetic QA pipeline builds on.","marker":"Changpinyo et al., 2022"},{"why":"GSM8k serves as the standard curated math benchmark contrasted with real handwritten student work.","marker":"Cobbe et al., 2021"},{"why":"The MATH benchmark is the other standard curated math benchmark the paper contrasts with messy classroom images.","marker":"Hendrycks et al."},{"why":"MathVista represents prior visual math benchmarks with standardized or typed diagrams, against which DrawEduMath's handwritten images are positioned.","marker":"Lu et al., 2024"},{"why":"Mixtral 8x22B is the LLM judge that produces the paper's main model rankings, and its validation is the key to the ranking claim.","marker":"Jiang et al., 2024"},{"why":"Whisper transcribes teachers' spoken descriptions, a major annotation channel that made the dataset feasible.","marker":"Radford et al., 2023"},{"why":"ROUGE-L is one of the automatic metrics whose correlation with human judgments is compared with the LLM judge.","marker":"Lin, 2004"},{"why":"BERTScore is the other automatic metric compared against human evaluation in validating the scoring approach.","marker":"Zhang et al., 2020"}],"fun_headline_variants":["AI still stumbles on kids' handwritten math","Handwritten math trips up even top vision-language models","New benchmark shows VLMs lag on student math work","Synthetic questions rank VLMs like teacher-written ones","DrawEduMath: 2,000+ images reveal AI's math vision gaps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported rankings depend on Mixtral 8x22B as an automatic judge for all QA pairs, but its agreement with humans was checked on only 500 sampled responses; if its similarity ratings are systematically biased on the unvalidated majority, the rankings in the paper could reflect the judge's preferences rather than genuine model ability.","fun_headline_variants_meta":{"raw":{"variants":["AI still stumbles on kids' handwritten math","Handwritten math trips up even top vision-language models","New benchmark shows VLMs lag on student math work","Synthetic questions rank VLMs like teacher-written ones","DrawEduMath: 2,000+ images reveal AI's math vision gaps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000257,"raw_usage":{"total_tokens":1606,"prompt_tokens":1000,"completion_tokens":606,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":616,"completion_tokens_details":{"reasoning_tokens":524}},"tokens_in":616,"tokens_out":606,"duration_ms":6922,"temperature":1.0,"reasoning_tokens":524,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:49:58.884728+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the full set of teacher-written QA pairs, or a much larger stratified sample such as 2,000 responses per model, and have human evaluators score each VLM answer; then recompute model rankings and the Spearman correlation between synthetic-based and teacher-based rankings. If human-judged rankings diverge from Mixtral-judged rankings, or if synthetic and teacher rankings no longer agree, the paper's two central findings would fail.","supporting_citations":[],"review_version":1}