Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

DrawEduMath: Evaluating Vision Language Models with Expert-Annotated Students' Hand-Drawn Math Images

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Vision-language models still cannot reliably interpret students' handwritten math work, and synthetic questions can substitute for teacher-written ones when ranking models.

desk verdict DrawEduMath is a valuable benchmark, but its central ranking claim leans on a single, thinly validated LLM judge; the dataset deserves publication, the claim needs tempering. read the letter →

arxiv 2501.14877 v1 pith:YRXD3RIU submitted 2025-01-24 cs.CL cs.CV

Sami Baral , Li Lucy , Ryan Knight , Alice Ng , Luca Soldaini , Neil T. Heffernan , Kyle Lo This is my paper · ORCID
classification cs.CLcs.CV
keywords DrawEduMathhandwrittenvisionlanguagemodelsvisualquestionansweringK-12educationteacherannotationssyntheticQAbenchmarkevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DrawEduMath is a benchmark built to test whether vision-language models can do what a teacher does when looking at a student's handwritten math work: see what is actually on the page, understand the strategy, and spot errors. The paper introduces 2,030 images of real K-12 students' handwritten responses, each paired with expert teacher descriptions and 11,661 teacher-written question-answer pairs, plus 44,362 synthetic QA pairs generated from those descriptions. Evaluating four vision-language models, the authors find that even the strongest closed models answer only about seven in ten synthetic questions correctly by an LLM judge, and noticeably fewer teacher-written ones, while an open-source model lags far behind. A second finding is that synthetic QA pairs, despite being noisy, rank the models in nearly the same order as teacher-written QA. If these results hold, current vision-language models are not yet reliable enough to interpret students' handwritten work automatically, and synthetically generated questions could be a scalable proxy for expert-written ones when comparing models rather than measuring absolute accuracy.

What carries the argument

The load-bearing object is the benchmark itself, structured as a chain from classroom artifact to evaluable question. Teachers' free-form descriptions of each image are the expert ground truth; from these, an LM pipeline extracts atomic facts, called facets, and rewrites them as close-ended QA pairs, producing 44,362 synthetic questions. A seven-category taxonomy separates higher-level understanding of math and correctness-and-error questions from purely visual or low-level questions, so model scores can be broken down by what is being tested. Evaluation relies on three automatic metrics, ROUGE-L, BERTScore, and an LLM judge based on Mixtral 8x22B, with the LLM judge validated on a 500-response human sample showing Spearman correlation 0.801 over 12 aggregate points. The facet-to-QA mechanism is what makes the benchmark scalable, while the taxonomy and the human-validated judge are what make the scores interpretable.

What would settle it

Take the full set of teacher-written QA pairs, or a much larger stratified sample such as 2,000 responses per model, and have human evaluators score each VLM answer; then recompute model rankings and the Spearman correlation between synthetic-based and teacher-based rankings. If human-judged rankings diverge from Mixtral-judged rankings, or if synthetic and teacher rankings no longer agree, the paper's two central findings would fail.

Watch

Extended reading notes

Core claim

The paper's central claim is that current vision-language models, including state-of-the-art commercial ones, leave substantial room for improvement on the task of interpreting students' handwritten math responses, and that a scalable pipeline can produce useful evaluation questions without full teacher effort. The benchmark DrawEduMath pairs each student-response image with a problem statement and teacher annotations; teachers described every image and wrote questions that target pedagogical concerns such as problem-solving strategy, composition of diagrams, labels, and correctness. From the teacher descriptions, language models generated additional QA pairs by decomposing each description into atomic facets and rewriting each facet as a self-contained question and answer. On these questions, even the best model reaches 0.657 LLM-judged correctness on teacher-written QA and 0.715 on synthetic QA, while the open-source Llama model scores roughly half that. Questions about correctness and errors are consistently the hardest for all models. Despite the noise in synthetic QA, the authors find that the relative ordering of models is similar whether teacher-written or synthetic questions are used, suggesting the synthetic set is informative for model ranking.

Load-bearing premise

The reported rankings depend on Mixtral 8x22B as an automatic judge for all QA pairs, but its agreement with humans was checked on only 500 sampled responses; if its similarity ratings are systematically biased on the unvalidated majority, the rankings in the paper could reflect the judge's preferences rather than genuine model ability.

Editorial extensions

If this is right

  • If the central finding holds, VLM-based tools for K-12 classrooms should not yet be trusted to give feedback on handwritten work, especially on correctness-and-error questions, where all evaluated models perform worst.
  • The similarity between synthetic and teacher-written rankings implies that benchmark creators can use LM-generated QA from expert descriptions to compare models cheaply, reserving teacher-written QA for final validation.
  • The observed failure mode, models giving the mathematically correct answer even when the student's own response is wrong, points to a specific capability gap: reading a student's work as evidence rather than solving the problem independently.
  • Dark or low-quality images that human annotators could still read caused additional model errors, suggesting that robustness to image quality is a bottleneck for real-world deployment.
  • The released dataset, with teacher descriptions, QA pairs, and metadata, provides a reusable resource for measuring progress on naturalistic handwritten math understanding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If synthetic QA can rank models reliably, an implicit next step is to generate QA from teacher descriptions at much larger scale and use it for model selection during development, not just for final evaluation.
  • A deployment consequence the paper leaves implicit is that automated feedback systems on learning platforms should keep a human in the loop for the correctness-and-error category, since that is both the most pedagogically important and the least reliable for current models.
  • The taxonomy suggests a testable extension: weighting model scores by pedagogical importance, such as correctness and strategy questions rather than paper type, would likely widen the measured gap between models.
  • Because the ranking claim depends on the LLM judge, a natural extension is to test whether the synthetic-versus-teacher ranking agreement persists under human evaluation on a much larger sample; if it does not, the claim would need to be restricted to automatic-judge settings.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces DrawEduMath, a dataset of 2,030 images of K-12 students' handwritten math responses, annotated by expert teachers with detailed descriptions and 11,661 QA pairs, plus 44,362 synthetic QA pairs generated from the descriptions by Claude 3.5 and GPT-4o. The authors evaluate four VLMs (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3.2-11B Vision) on both QA sets using BERTScore, ROUGE-L, and an LLM (Mixtral 8x22B) judge, and validate the automatic metrics on a sample of 500 model responses with human annotations. The main findings are that all evaluated models perform far below a practically usable level on teacher QA, with closed models outperforming Llama, and that synthetic QA yields similar model rankings to teacher QA.

Significance. The dataset is a valuable new resource: it contains real student handwriting in naturalistic settings, teacher annotations with a second revision round, PII removal, and a public release. The evaluation is external to model training, so the benchmark is not circular. If the ranking-similarity claim holds, it would justify using synthetic QA for scalable model evaluation in this domain. However, the evidence for that claim is currently weak because it rests on a single LLM judge with limited and pooled validation, and the reference metrics contradict the LLM judge on teacher QA. With the ranking claim either strengthened or substantially qualified, the paper would be a solid contribution.

major comments (3)
  1. [Section 7, 'Assessing Our Automatic Metrics' and Table 7] The conclusion that synthetic QA yields similar model rankings to teacher QA is not supported by the evidence presented. The LLM judge's agreement with humans was estimated on only 500 model responses pooled across 4 models and 3 QA sets, yielding a Spearman correlation of 0.801 over 12 aggregate points; per-cell human samples are n=62 for synthetic and n=63 for teacher, or about 16 per model, which is too small to establish stable per-QA-set rankings. The paper does not report judge-human agreement separately for teacher QA, nor confidence intervals or a bootstrap for the 12-point Spearman. Given that only four models are compared, a small judge bias can reorder the top ranks, so the ranking-similarity claim in the abstract and Section 7 is not robust.
  2. [Section 7, Table 7] The reference-based metrics contradict the LLM judge on teacher QA. For teacher-written QA, BERTScore and ROUGE-L rank Llama 3.2-11B highest (0.785 and 0.253, respectively), whereas the LLM judge and the human sample rank it lowest (0.296 and 0.127). The LLM judge also reverses the top two models between synthetic QA (GPT-4o 0.723 vs. Claude 3.5 0.715) and teacher QA (Claude 3.5 0.657 vs. GPT-4o 0.628). The paper should report ranking similarity per QA set with uncertainty and reconcile these contradictions before claiming that synthetic QAs yield similar rankings.
  3. [Section 7, 'Experimental Setup'] The prompt-selection procedure may distort model rankings. The authors state that they iterated over three possible prompts for each model 'on samples of data' and selected the best-performing one, but they do not specify whether these samples were disjoint from the evaluated QA pairs or whether the selection was done on a held-out set. If the same data were used for both prompt selection and final evaluation, the reported performance is optimistically biased. Please clarify the split and report the chosen prompts.
minor comments (5)
  1. [Section 2, first paragraph] 'multimodel' should be 'multimodal'.
  2. [Section 7, 'Results and Findings'] The phrase 'both QA sets rank models similarity' should be 'similarly'; moreover, the claim is too strong given the top-two reversal noted in the major comments.
  3. [Table 8] The caption does not clearly indicate which of the two numbers per model corresponds to synthetic versus teacher QA, and the typesetting of the symbols after 'combined' and 'by teachers' appears to be missing; please repair the table.
  4. [Section 4, 'Quality Assessment of Synthetic QA'] The reported QA quality assessment relies on only 100 pairs and shows notable annotator disagreement (e.g., Evaluator A found zero unanswerable questions while Evaluator B found nine); reporting inter-annotator agreement, such as Cohen's kappa, would strengthen this part.
  5. [Appendix D.2] The human evaluation groups 'Maybe' with 'Yes' and 'Neither' with 'Quite different'; since this binarization is used to validate the LLM judge, a sensitivity analysis under alternative groupings would be informative.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DrawEduMath's evaluation chain is externally grounded in human teacher annotations, not in the models being evaluated.

full rationale

The central evaluation loop is not circular. Gold answers come from external human teachers (11,661 teacher-written QA pairs) or from synthetic QA pairs generated by LMs from teacher-written image descriptions. The VLMs under evaluation are frozen and see only the question and the student's handwritten image; they never see the teacher descriptions or the synthetic QA derivation pipeline. The synthetic QA construction, following Changpinyo et al., is a dataset-generation strategy rather than a fitted parameter or a hidden input to the models, so measuring agreement between model answers and these gold answers is a legitimate external evaluation. The use of Mixtral 8x22B as an automatic judge was validated against human judgments on 500 sampled responses with a reported Spearman correlation of 0.801 over 12 aggregate points; this is a limited validation that raises external-validity and robustness concerns, but it is an independent human check, not a definitional reduction. Similarly, the paper's prompt selection on a sample of data is an overfitting risk, not circularity. No load-bearing claim depends on a self-citation or on an author-imported uniqueness theorem; the authors' earlier works appear only as related work or context. The finding that synthetic QAs yield similar model rankings to teacher QAs is an empirical result about the judge and the models, not a consequence of the evaluation being constructed from its own output.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The benchmark's central claims rest on several domain assumptions about annotation accuracy, representativeness, and the validity of an LLM judge. The hand-chosen binarization and sampling choices are not standard free parameters in a derivation, but they shape the reported results. No new physical or conceptual entities are introduced.

free parameters (3)
  • LLM judge binarization threshold = Likert 1-2 as incorrect, 3-4 as correct
    Used in Section 7 to convert Mixtral's 4-point similarity rating into a binary correctness score; affects all reported accuracies and the correlation with human judgments, but the cut point is arbitrary.
  • Grouping of 'Maybe' with 'Yes' in human evaluation = 'Maybe, but could be better' counted as correct
    In Appendix D, 'Maybe' is binarized together with 'Yes' as correct, which inflates human-validated accuracy and could mask ranking differences if 'Maybe' were treated as incorrect.
  • Images sampled per problem = 15
    Section 3.1 randomly samples 15 images per problem from roughly 300; this balances the dataset but the choice of 15 is arbitrary and affects the diversity of student responses represented.
assumptions (4)
  • domain assumption Teacher annotations accurately describe the student responses in the images.
    Section 3.2-3.3 use teacher descriptions and QA as gold standard; any description errors propagate to all downstream QA and model evaluation.
  • domain assumption The 188 problems and sampled images are representative of real K-12 handwritten math work.
    Section 3.1 samples from ASSISTments platform with OER problems; generalizing to all classrooms assumes this sample is representative, which the paper itself notes is U.S.-centric Common Core content.
  • domain assumption Mixtral 8x22B similarity judgments are a valid substitute for human judgment on all QA pairs.
    Section 7 relies on Mixtral for full-benchmark evaluation after validation on 500 samples; the validity on the remaining 56k QA pairs is unverified.
  • domain assumption LM-generated QA pairs (Claude, GPT-4o) derived from teacher descriptions are answerable and correct for the image.
    Section 4 generates 44,362 synthetic QA pairs from descriptions; quality was checked on only 100 pairs by two annotators with notable disagreement, yet the full set is used as a benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DrawEduMath: Evaluating Vision Language Models with Expert-Annotated Students' Hand-Drawn Math Images." pith.science (2026). https://pith.science/paper/YRXD3RIU

@misc{pith2026250114877,
  author       = {Pith},
  title        = {Pith review of: DrawEduMath: Evaluating Vision Language Models with Expert-Annotated Students' Hand-Drawn Math Images},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YRXD3RIU}},
  note         = {Machine review of arXiv:2501.14877}
}
read the original abstract

In real-world settings, vision language models (VLMs) should robustly handle naturalistic, noisy visual content as well as domain-specific language and concepts. For example, K-12 educators using digital learning platforms may need to examine and provide feedback across many images of students' math work. To assess the potential of VLMs to support educators in settings like this one, we introduce DrawEduMath, an English-language dataset of 2,030 images of students' handwritten responses to K-12 math problems. Teachers provided detailed annotations, including free-form descriptions of each image and 11,661 question-answer (QA) pairs. These annotations capture a wealth of pedagogical insights, ranging from students' problem-solving strategies to the composition of their drawings, diagrams, and writing. We evaluate VLMs on teachers' QA pairs, as well as 44,362 synthetic QA pairs derived from teachers' descriptions using language models (LMs). We show that even state-of-the-art VLMs leave much room for improvement on DrawEduMath questions. We also find that synthetic QAs, though imperfect, can yield similar model rankings as teacher-written QAs. We release DrawEduMath to support the evaluation of VLMs' abilities to reason mathematically over images gathered with educational contexts in mind.

Figures

Figures reproduced from arXiv: 2501.14877 by the authors.

Figure 1
Figure 1. Each image in our dataset is a concatenation of a math problem on the left with a student response on [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. For some annotators, their recorded descrip [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Examples of teachers’ answers to a question asking about possible errors in students’ responses to math [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: A screenshot of our recording website, where teachers would view an image from our dataset and either [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: A screenshot of the interface teachers used to write answers to teacher-written questions about students’ [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Prompt for decomposing teacher-written captions for images into atomic facets. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Prompt for converting atomic facets to QA pairs. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Prompt for categorizing questions into question types. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Prompt used with VLMs for answering question about the student’s handwritten response. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Prompt used for comparing model-generated answer with teacher-provided answer about student [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MIDAL: A Dataset of Math Image Descriptions for Accessible Learning

    cs.CV 2026-08 unverdicted novelty 6.0 of 10

    MIDAL provides 2,020 described math images to train vision-language models for accessible math image descriptions and improved math reasoning.

  2. Automated Grading of Students' Handwritten Graphs: A Comparison of Meta-Learning and Vision-Large Language Models

    cs.LG 2025-07 conditional novelty 5.0 of 10

    The best meta-learning models reach 56.9% in 2-way grading and the best vision-language models reach 50.0% in 3-way grading of handwritten economics graphs, both near chance levels.

Reference graph

Works this paper leans on

51 extracted references · 30 canonical work pages · cited by 2 Pith papers

  1. [1]

    John R Anderson, Albert T Corbett, Kenneth R Koedinger, and Ray Pelletier. 1995. Cognitive tutors: Lessons learned. The journal of the learning sciences, 4(2):167--207

    work page 1995

  2. [2]

    AI Anthropic. 2024. The C laude 3 model family: O pus, S onnet, H aiku. Claude-3 M odel C ard , 1

    work page 2024

  3. [3]

    Sami Baral, Anthony Botelho, Abhishek Santhanam, Ashish Gurung, Li Cheng, and Neil Heffernan. 2023. Auto-scoring student responses with images in mathematics. International Educational Data Mining Society

    work page 2023

  4. [4]

    Sami Baral, Anthony F Botelho, John A Erickson, Priyanka Benachamardi, and Neil T Heffernan. 2021. Improving automated scoring of student open responses in mathematics. International Educational Data Mining Society

    work page 2021

  5. [5]

    Anthony Botelho, Sami Baral, John A Erickson, Priyanka Benachamardi, and Neil T Heffernan. 2023. Leveraging natural language processing to support automated assessment and feedback for student open responses in mathematics. Journal of computer assisted learning, 39(3):823--840

    work page 2023

  6. [6]

    Soravit Changpinyo, Doron Kukliansy, Idan Szpektor, Xi Chen, Nan Ding, and Radu Soricut. 2022. https://doi.org/10.18653/v1/2022.naacl-main.142 All you may need for VQA are image captions . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1947--1963, Sea...

    doi:10.18653/v1/2022.naacl-main.142 2022

  7. [7]

    Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric Xing, and Liang Lin. 2021. https://doi.org/10.18653/v1/2021.findings-acl.46 G eo QA : A geometric question answering benchmark towards multimodal numerical reasoning . In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 513--523, Online. Association fo...

    doi:10.18653/v1/2021.findings-acl.46 2021

  8. [8]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . Preprint, arXiv:2110.14168

    arXiv 2021

Show all 51 references
  1. [9]

    Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. 2017. https://doi.org/10.1109/IJCNN.2017.7966217 Emnist: Extending mnist to handwritten letters . In 2017 International Joint Conference on Neural Networks (IJCNN), pages 2921--2926

    2017

  2. [10]

    Smith, Hanna Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Christopher Callison-...

    2024

  3. [11]

    David Ebert. 2014. Graphing projects with desmos. The Mathematics Teacher, 108(5):388--391

    2014

  4. [12]

    Bill & Melinda Gates Foundation . 2024. https://usprogram.gatesfoundation.org/news-and-insights/articles/ai-powered-innovations-in-mathematics-teaching-and-learning-rfi Ai-powered innovations in mathematics teaching and learning: Request for information . Accessed: 2024-09-19

    2024

  5. [13]

    Philippe Gervais, Asya Fadeeva, and Andrii Maksai. 2024. https://arxiv.org/abs/2404.10690 Mathwriting: A dataset for handwritten mathematical expression recognition . Preprint, arXiv:2404.10690

    2024 arXiv

  6. [14]

    Google . 2023. https://blog.google/outreach-initiatives/education/google-learnlm-gemini-generative-ai/ Google learnlm and gemini: How google’s generative ai is transforming learning . Accessed: 2024-09-19

    2023

  7. [15]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6325--6334. IE...

    2017

  8. [16]

    Ashish Gurung, Sami Baral, Morgan P Lee, Adam C Sales, Aaron Haim, Kirk P Vanacore, Andrew A McReynolds, Hilary Kreisberg, Cristina Heffernan, and Neil T Heffernan. 2023. How common are common wrong answers? crowdsourcing remediation at scale. In Proceedings of the Tenth ACM C...

    2023

  9. [17]

    Abram Handler, Matthew Denny, Hanna Wallach, and Brendan O ' Connor. 2016. https://doi.org/10.18653/v1/W16-5615 Bag of what? simple noun phrase extraction for text analysis . In Proceedings of the First Workshop on NLP and Computational Social Science , pages 114--124, Austin,...

    2016 doi

  10. [18]

    Neil T Heffernan and Cristina Lindquist Heffernan. 2014. The assistments ecosystem: Building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching. International Journal of Artificial Intelligence in Education, 2...

    2014

  11. [19]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track ...

  12. [20]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024 arXiv

  13. [21]

    Khan Academy . 2024. https://blog.khanacademy.org/why-were-deeply-invested-in-making-ai-better-at-math-tutoring-and-what-weve-been-up-to-lately/ Why we’re deeply invested in making ai better at math tutoring (and what we’ve been up to lately) . Accessed: 2024-09-19

    2024

  14. [22]

    Ren \'e F Kizilcec and Hansol Lee. 2022. Algorithmic fairness in education. In The ethics of artificial intelligence in education, pages 174--202. Routledge

    2022

  15. [23]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  16. [24]

    Liwicki and H

    M. Liwicki and H. Bunke. 2005. https://doi.org/10.1109/ICDAR.2005.132 Iam-ondb - an on-line english sentence database acquired from handwritten text on a whiteboard . In Eighth International Conference on Document Analysis and Recognition (ICDAR'05), pages 956--961 Vol. 2

    2005 doi

  17. [25]

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. https://openreview.net/forum?id=KUNzEQMWU7 Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts . I...

    2024

  18. [26]

    Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. 2021. https://doi.org/10.18653/v1/2021.acl-long.528 I nter- GPS : Interpretable geometry problem solving with formal language and symbolic reasoning . In Proceedings of the 59th Annual ...

    2021 doi

  19. [27]

    fairness

    Michael Madaio, Su Lin Blodgett, Elijah Mayfield, and Ezekiel Dixon-Rom \'a n. 2022. Beyond “fairness”: Structural (in) justice lenses on ai for education. In The ethics of artificial intelligence in education, pages 203--239. Routledge

    2022

  20. [28]

    Rizwaan Malik, Dorna Abdi, Rose Wang, and Dorottya Demszky. 2024. Scaling high-leverage curriculum scaffolding in middle-school mathematics. In Proceedings of the Eleventh ACM Conference on Learning@ Scale, pages 476--480

    2024

  21. [29]

    U-V Marti. 2002. The iam-database: an english sentence database for offline handwriting recognition. International Journal on Document Analysis and Recognition, 5:39--46

    2002

  22. [30]

    Matelsky, Felipe Parodi, Tony Liu, Richard D

    Jordan K. Matelsky, Felipe Parodi, Tony Liu, Richard D. Lange, and Konrad P. Kording. 2023. https://arxiv.org/abs/2308.02439 A large language model-assisted education tool to provide feedback on open-ended responses . Preprint, arXiv:2308.02439

    2023 arXiv

  23. [31]

    Minesh Mathew, Lluis Gomez, Dimosthenis Karatzas, and CV Jawahar. 2021 a . Asking questions on handwritten document collections. International Journal on Document Analysis and Recognition (IJDAR), 24(3):235--249

    2021

  24. [32]

    Minesh Mathew, Dimosthenis Karatzas, and C.V. Jawahar. 2021 b . Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2200--2209

    2021

  25. [33]

    Meta AI . 2024. https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/ Llama 3.2: Revolutionizing edge ai and vision with open, customizable models . Accessed: Oct 10, 2024

    2024

  26. [34]

    Microsoft News Center . 2024. https://news.microsoft.com/source/features/ai/khan-academy-and-microsoft-partner-to-expand-access-to-ai-tools/ Khan academy and microsoft partner to expand access to ai tools . Accessed: 2024-09-19

    2024

  27. [35]

    Harold Mouchere, Christian Viard-Gaudin, Dae Hwan Kim, Jin Hyung Kim, and Utpal Garain. 2011. https://doi.org/10.1109/ICDAR.2011.297 Crohme2011: Competition on recognition of online handwritten mathematical expressions . In 2011 International Conference on Document Analysis an...

    2011 doi

  28. [36]

    Laura K Nelson. 2020. Computational grounded theory: A methodological framework. Sociological Methods & Research, 49(1):3--42

    2020

  29. [37]

    Zachary A Pardos and Shreya Bhandari. 2024. Chatgpt-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills. Plos one, 19(5):e0304013

    2024

  30. [38]

    Jordi Pont-Tuset, Jasper R. R. Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. 2019. https://api.semanticscholar.org/CorpusID:208857532 Connecting vision and language with localized narratives . In European Conference on Computer Vision

    2019

  31. [39]

    Andrew Porter, Jennifer McMaken, Jun Hwang, and Rui Yang. 2011. Common core standards: The new us intended curriculum. Educational researcher, 40(3):103--116

    2011

  32. [40]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. https://arxiv.org/abs/2212.04356 Robust speech recognition via large-scale weak supervision . Preprint, arXiv:2212.04356

    2022 arXiv

  33. [41]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. 2023. https://proceedings.mlr.press/v202/radford23a.html Robust speech recognition via large-scale weak supervision . In Proceedings of the 40th International Conference on Machine Lear...

    2023

  34. [42]

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arX...

    2024 arXiv

  35. [43]

    Nils Reimers and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/D19-1410 Sentence- BERT : Sentence embeddings using S iamese BERT -networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference...

    2019 doi

  36. [44]

    Sherry Ruan, Jiayu He, Rui Ying, Jonathan Burkle, Dunia Hakim, Anna Wang, Yufeng Yin, Lily Zhou, Qianyao Xu, Abdallah AbuHashem, et al. 2020. Supporting children's math learning with feedback-augmented narrative technology. In Proceedings of the interaction design and children...

    2020

  37. [45]

    Hava E Vidergor and Paz Ben-Amram. 2020. Khan academy effectiveness: The case of math secondary students' perceptions. Computers & Education, 157:103985

    2020

  38. [46]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024. MMMU : A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conferen...

    2024

  39. [47]

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, and Hongsheng Li. 2024. https://arxiv.org/abs/2403.14624 Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European ...

    2024 arXiv

  40. [48]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In International Conference on Learning Representations

    2020

  41. [49]

    Shusen Zhou, Qingcai Chen, and Xiaolong Wang. 2010. https://doi.org/10.1145/1815330.1815359 Hit-or3c: an opening recognition corpus for chinese characters . In Proceedings of the 9th IAPR International Workshop on Document Analysis Systems, DAS '10, page 223–230, New York, NY,...

    2010

  42. [50]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  43. [51]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.