Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

An Exploratory Study of ML Sketches and Visual Code Assistants

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that popular vision-capable language models, given only a photograph of an informal whiteboard sketch of a machine-learning workflow, can reliably reconstruct the notebook's high-level outline (about 79% accuracy) while…

desk verdict First real user study of sketch-to-code for ML notebooks, with useful qualitative findings, but the abstract overstates a non-significant correlation and the LLM-judge validation is thinner than the paper admits. read the letter →

arxiv 2412.13386 v1 pith:54QJVSGJ submitted 2024-12-17 cs.SE cs.AIcs.HC

Luís F. Gomes , Vincent J. Hellendoorn , Jonathan Aldrich , Rui Abreu This is my paper · ORCID
classification cs.SEcs.AIcs.HC
keywords sketch-to-codevisualcodeassistantswhiteboardsketchingmachinelearningworkflowsmultimodalLLMsLLM-as-a-judgeJupyternotebooksdatascience
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the informal whiteboard sketches that data scientists already draw can be used directly as input for code generation. The authors built a prototype that sends a photograph of a sketch to a vision-capable language model and assembles the returned steps into a Jupyter notebook, then tested it with 19 data scientists working on three realistic ML tasks. The central result is that current models capture the skeleton of the intended notebook—roughly 70–80% of high-level steps are recognized—but only about 25–40% of implementation details match the sketch. That finding matters because it suggests a feasible near-term role for visual code assistants: generating a useful first draft that a developer corrects, rather than fully finished code. The study also reports a positive association between sketching time and outline quality, and participants identify education, prototyping, and meetings as the most promising settings.

What carries the argument

The carrying mechanism is a two-stage pipeline built around a vision language model. In the generation stage, the sketch image is sent to the model with a fixed prompt asking for a JSON list of steps, each containing Python code and a short markdown explanation; the prototype then assembles those steps into notebook cells. In the evaluation stage, a separate model acts as an automated judge: given the generated notebook, the original task description, and hand-written True/False grading questions, it produces outline and instantiation scores. Outline accuracy counts how many high-level subtasks from the sketch appear in the notebook, while instantiation accuracy counts how many specific details are implemented correctly; because instantiation is a sub-specification of outline, the two metrics are nested. A small human-graded subsample was used to validate the judge, reporting 92.6% outline judgment accuracy and 78.3% instantiation judgment accuracy.

What would settle it

Have two independent human raters grade all 19 generated notebooks against the original sketches with the same subtask rubric, without knowing which model produced them; if human outline and instantiation scores disagree with the automated judge by much more than the reported validation accuracy (92.6% and 78.3%), the 79% and 36% headline numbers would need revision.

Watch

Extended reading notes

Core claim

The paper's core discovery is that popular vision-capable LLMs can turn an informal whiteboard sketch of an ML pipeline into a notebook skeleton that is largely correct in structure but weak in detail. On the 19 participant sketches, the average generated notebook scored 79% outline accuracy and 36% instantiation accuracy, with outline measuring the presence of the major subtasks and instantiation measuring correct parameters and implementation choices. In a retrospective comparison, the three tested models—GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro—performed similarly, with no statistically significant differences, all landing in the 68–79% outline range and 26–36% instantiation range. The paper also finds that each extra minute of sketching predicts a 2.4-percentage-point increase in outline accuracy, though the result falls just short of conventional significance, and that participants changed a median of 19.5 lines in a 40-line final notebook, indicating that the generated skeleton supplied roughly half the final content. Qualitative results show that text annotations are the most common sketch element across all subtasks and that users want explainable, iteratively refinable assistants.

Load-bearing premise

The whole accuracy measurement depends on trusting the automated judge's True/False scores to reflect how well the generated notebook matches what the participant sketched, and that trust rests on a small hand-checked subsample with a judge from the same model family as the generator.

Editorial extensions

If this is right

  • Sketch-to-code can seed a coding session with a structurally correct notebook skeleton, letting developers edit a draft instead of starting from a blank file.
  • Because instantiation accuracy is low, users should expect to correct file names, exact parameters, and specific function choices by hand.
  • Longer sketching is associated with better outline quality, so tools could encourage users to add more detail when they want a more faithful skeleton.
  • The three tested vision models are roughly interchangeable for this task, so tool builders can choose among them on cost, latency, or availability rather than sketch-understanding ability.
  • The same generation-and-judge pipeline could be applied to other diagram-driven specification formats, since the model demonstrably follows sketched ordering and annotations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 36% instantiation ceiling may be set as much by what sketches omit as by model capability; the paper notes sketches often lack file names and exact parameters, so an explicit measure of sketch information sufficiency would separate the two.
  • A natural next experiment is a checklist prompt that asks users to include data source, split ratio, encoding, model choice, and plot types; if outline accuracy rises well above 79%, the bottleneck is partly promptable user input.
  • The judge validation rests on a small subsample and a judge drawn from the same model family as the generator; a blind cross-family judge with a larger human-graded set would harden the headline numbers.
  • Neighboring domains such as database schema design or API orchestration share the same dependency-and-ordering structure, so the outline/instantiation gap likely reappears there and could be tested with the same two-metric setup.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper reports an exploratory user study (n=19) in which data scientists sketch an ML workflow on a whiteboard, a VSCode plugin ("Visual Code Assistant") uses GPT-4o to convert a photo of the sketch into a Jupyter notebook, and the participant then edits the notebook toward a solution. The authors analyze sketching patterns (RQ1), evaluate the quality of the generated code using an LLM-as-judge built on GPT-4 (RQ2), and collect interview and survey data on user perceptions (RQ3). The headline quantitative results are an outline accuracy of 79% and an instantiation accuracy of 36% for sketch-only generation, a positive association between sketch time and outline accuracy, and a three-model comparison (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro) with no significant differences. The paper also manually verifies the final submitted notebooks and reports that 11/19 participants produced fully correct solutions.

Significance. If the quantitative claims were supported, this would be a useful first benchmark for sketch-to-code for ML workflows and an interesting exploration of a new interaction paradigm for code assistants. The paper's strengths include a reproducible prototype, openly available artifacts (sketches, notebooks, annotations, surveys on Zenodo), a qualitative analysis grounded in think-aloud protocols and interviews, and a manual verification of final submitted notebooks. However, the central accuracy numbers and the sketch-time correlation rest on an LLM-judge whose validation is, for the outline metric, too imbalanced to discriminate the judge from an always-True classifier. The same-model family for generator and judge compounds this concern. The qualitative findings and the feasibility demonstration (participants could, with editing, produce working solutions) are likely to stand, but the specific quantitative conclusions about sketch-only code quality are not yet established.

major comments (3)
  1. [Section IV-E3 (Judge Validation)] The validation of the outline judge is uninformative because of extreme class imbalance. The subsample contains 64 positive and only 4 negative outline judgments; a judge that answers 'True' to every item would achieve 64/68 = 94.1% accuracy and 64/68 = 94.1% precision, both above the reported 92.6% and 92.2%. Thus the reported agreement values cannot distinguish the LLM judge from a degenerate always-True classifier. Additionally, the reported 'precision of 92.2% (59/64)' is internally inconsistent with a 4-negative subsample if precision is TP/(TP+FP); the value appears to be recall. Because the headline outline accuracy (79%) and the sketch-time regression (Table III) are computed from these judge scores, the current evidence does not support the quantitative claims for outline quality. Please re-validate with a balanced set of items (e.g., oversampling negative outline judgments) or provide a manual grading of all 19 notebooks for the outline and instantiation metrics.
  2. [Abstract and Section V-B (Table III)] The abstract and Finding 3 claim a positive correlation between sketch time and outline accuracy, but the regression coefficient for Sketch Time has p = 0.105 after Benjamini-Hochberg correction, which the paper itself notes 'lies slightly outside the standard confidence intervals (0.1)'. At the paper's own threshold (p < 0.1), this result is not statistically significant. The claim in the abstract of 'a positive correlation' is therefore an overstatement; the appropriate wording is a suggestive trend that did not reach significance in this small sample. The same issue affects the summary in Section VII that 'longer sketch times yield more accurate notebooks.'
  3. [Section IV-E3 and Section V-B (Manual Verification)] The generator (GPT-4o) and the judge (GPT-4) come from the same model family, and the paper's assertion that 'the risk of bias is slim' is not a measurement. A True-biased judge would inflate both outline and instantiation accuracy and could also create a spurious positive relationship between sketch time and outline accuracy if longer sketches yield more plausible-looking code. The manual verification of submitted notebooks (11/19 fully correct) evaluates human-edited final solutions, not the sketch-only generated notebooks, so it does not corroborate the 79%/36% figures. Please provide additional evidence that the judge scores track sketch fidelity rather than code plausibility, for example by scoring deliberately corrupted or sketch-divergent notebooks, or by using an independent judge from a different model family.
minor comments (4)
  1. [Section V-B] The reported maximum coding time of '373' minutes is clearly a typo for 37 minutes (the text immediately discusses the 37-minute outlier); please correct it.
  2. [Abstract and Section I] The Abstract refers to 'Artifical Intelligence' in the introduction; please fix the spelling of 'Artificial' throughout.
  3. [Section V-B] The sentence 'sketching reduced the amount of written lines of code by 49% on average' is not obviously derived from the preceding numbers (median 19.5 lines changed, median solution 40 lines); please clarify the computation or rephrase.
  4. [Section IV-E3] The statement that p = 0.105 'lies slightly outside the standard confidence intervals (0.1)' conflates a p-value threshold with a confidence interval; please rephrase to avoid this statistical terminology error.

Circularity Check

0 steps flagged · score 2.0 of 10

No constructional circularity; the same-family LLM judge is a measurement-validity threat, not a derivation loop.

full rationale

This paper is an empirical user study rather than a derivation chain, so most circularity patterns do not apply. The headline numbers (79% outline, 36% instantiation) are averages of scores produced by a GPT-4 judge applied to GPT-4o-generated notebooks; the judge is given only the notebook, the task description, and hand-written True/False grading criteria, and the paper validates the judge against human grading on a subsample and manually verifies the final submitted notebooks. No parameter is fitted to a subset of the data and then renamed as a prediction, and no uniqueness theorem or ansatz is imported from prior work. The only self-referential element is that the generator and judge come from the same LLM family, which the paper itself flags as a possible bias; this is a genuine measurement-validity threat, but it does not make the reported accuracy equal to an input by construction, because the judge's criteria are externally authored and the paper provides some independent human grounding. A related-work citation to d'Amorim et al. overlaps with one author but is not load-bearing for any result. The judge-validation subsample is small and the paper does not report the full confusion matrix, which weakens confidence, but those are robustness concerns rather than exhibited circular reductions. Accordingly, no circular step can be identified with the specificity required by the analysis protocol, and the score reflects only the minor self-referential design and non-load-bearing self-citation.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central claims rest on a small, non-random sample (n=19), a hand-authored grading rubric, and regression models fitted to the same data. The stated assumptions about participants' task comprehension and the proxies for sketching proficiency and experience are acknowledged in the paper. No new theoretical entities are introduced; the 'Visual Code Assistant' is a prototype built from an existing API.

free parameters (3)
  • Outline regression coefficients (intercept, sketch time, last sketching, experience) = 55.9498, 2.4678, 0.0112, 0.0310
    Fitted to n=19 participants to estimate the relationship between sketch time and outline accuracy; the sketch-time coefficient has p=0.105, not significant at conventional levels.
  • Instantiation regression coefficients = 24.5364, 1.4536, 0.0266, -0.4327
    Fitted to the same 19 participants; adjusted R-squared is -0.025, so the model has no explanatory power.
  • Hand-authored True/False grading questions for Outline and Instantiation metrics = Non-numeric, task-specific criteria
    These questions define what counts as a correct outline or implementation detail; they are constructed by the authors and used to prompt the judge, so the accuracy metrics depend on this subjective instrument.
assumptions (6)
  • domain assumption Participants correctly comprehend and execute the provided tasks in their sketches
    Stated in Section IV-E2 as a key assumption. If participants misunderstood the task, the sketch and generated code would not reflect their true intent.
  • domain assumption LLM-as-a-judge provides a valid proxy for human grading
    Validated on a subsample (92.6% outline, 78.3% instantiation human agreement), but the judge remains a model with possible systematic bias.
  • domain assumption Sketching time is a proxy for sketch quality and detail
    Acknowledged in Section VI as true on average but not for all participants.
  • domain assumption Years of programming experience is a proxy for DS domain knowledge
    Acknowledged in Section VI; may not hold for all participants.
  • domain assumption GPT-4 judge scores are not biased by being from the same model family as the generator
    The authors assert this risk is slim because the judge does not see model names, but no empirical test is provided.
  • standard math Linear regression assumptions (linearity, independence, homoscedasticity) hold for the 19-participant sample
    Standard assumptions; small sample and one outlier (37-minute coding session) make them fragile.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Exploratory Study of ML Sketches and Visual Code Assistants." pith.science (2026). https://pith.science/paper/54QJVSGJ

@misc{pith2026241213386,
  author       = {Pith},
  title        = {Pith review of: An Exploratory Study of ML Sketches and Visual Code Assistants},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/54QJVSGJ}},
  note         = {Machine review of arXiv:2412.13386}
}
read the original abstract

This paper explores the integration of Visual Code Assistants in Integrated Development Environments (IDEs). In Software Engineering, whiteboard sketching is often the initial step before coding, serving as a crucial collaboration tool for developers. Previous studies have investigated patterns in SE sketches and how they are used in practice, yet methods for directly using these sketches for code generation remain limited. The emergence of visually-equipped large language models presents an opportunity to bridge this gap, which is the focus of our research. In this paper, we built a first prototype of a Visual Code Assistant to get user feedback regarding in-IDE sketch-to-code tools. We conduct an experiment with 19 data scientists, most of whom regularly sketch as part of their job. We investigate developers' mental models by analyzing patterns commonly observed in their sketches when developing an ML workflow. Analysis indicates that diagrams were the preferred organizational component (52.6%), often accompanied by lists (42.1%) and numbered points (36.8%). Our tool converts their sketches into a Python notebook by querying an LLM. We use an LLM-as-judge setup to score the quality of the generated code, finding that even brief sketching can effectively generate useful code outlines. We also find a positive correlation between sketch time and the quality of the generated code. We conclude the study by conducting extensive interviews to assess the tool's usefulness, explore potential use cases, and understand developers' needs. As noted by participants, promising applications for these assistants include education, prototyping, and collaborative settings. Our findings signal promise for the next generation of Code Assistants to integrate visual information, both to improve code generation and to better leverage developers' existing sketching practices.

Figures

Figures reproduced from arXiv: 2412.13386 by the authors.

Figure 1
Figure 1. Example of usage of our prototype. Left editor tab: The whiteboard sketch uploaded by the participant to be used by the code assistant to generate [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Overview of our benchmarking using LLM-as-a-Judge. To evaluate a Jupyter Notebook generated from a sketch (1), the Judge has access to knowledge [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison of top visual LLMs on the 19 notebooks. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HiLDe: Intentional Code Generation via Human-in-the-Loop Decoding

    cs.HC 2025-05 conditional novelty 7.0 of 10

    HiLDe, a code completion UI that exposes and lets users override the LLM's token-level choices, reduced security vulnerabilities in generated code compared to a baseline assistant in a within-subjects study of 18 programmers.

Reference graph

Works this paper leans on

37 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    The collaborative software process,

    L. Williams and R. R. Kessler, “The collaborative software process,” in International Conference on Software Engineering 2000 , 2000

    work page 2000

  2. [2]

    Software design sketching with calico,

    N. Mangano, A. Baker, M. Dempsey, E. Navarro, and A. van der Hoek, “Software design sketching with calico,” in Proceedings of the 25th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’10. New York, NY , USA: Association for Computing Machinery, 2010, p. 23–32. [Online]. Available: https://doi.org/10.1145/1858996.1859003

    arXiv

  3. [3]

    Let’s go to the whiteboard: how and why software developers use drawings,

    M. Cherubini, G. Venolia, R. DeLine, and A. J. Ko, “Let’s go to the whiteboard: how and why software developers use drawings,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , ser. CHI ’07. New York, NY , USA: Association for Computing Machinery, 2007, p. 557–566. [Online]. Available: https://doi.org/10.1145/1240624.1240714

    arXiv 2007

  4. [4]

    Visual thinking in action: Visualizations as used on whiteboards,

    J. Walny, S. Carpendale, N. Henry Riche, G. Venolia, and P. Fawcett, “Visual thinking in action: Visualizations as used on whiteboards,” IEEE Transactions on Visualization and Computer Graphics , vol. 17, no. 12, pp. 2508–2517, 2011

    work page 2011

  5. [5]

    Sketches and diagrams in practice,

    S. Baltes and S. Diehl, “Sketches and diagrams in practice,” in Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering , ser. FSE 2014. New York, NY , USA: Association for Computing Machinery, 2014, p. 530–541. [Online]. Available: https://doi.org/10.1145/2635868.2635891

    arXiv 2014

  6. [6]

    Maintaining mental models: a study of developer work habits,

    T. D. LaToza, G. Venolia, and R. DeLine, “Maintaining mental models: a study of developer work habits,” in Proceedings of the 28th International Conference on Software Engineering , ser. ICSE ’06. New York, NY , USA: Association for Computing Machinery, 2006, p. 492–501. [Online]. Available: https://doi.org/10.1145/1134285.1134355

    arXiv 2006

  7. [7]

    Gemini: A family of highly capable multimodal models,

    G. Team et al. , “Gemini: A family of highly capable multimodal models,” 2024

    work page 2024

  8. [8]

    Gpt-4 technical report,

    OpenAI, “Gpt-4 technical report,” 2023

    2023

Show all 37 references
  1. [9]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter,...

    2021

  2. [10]

    Deepseek-coder: When the large language model meets programming – the rise of code intelligence,

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . K. Li, F. Luo, Y . Xiong, and W. Liang, “Deepseek-coder: When the large language model meets programming – the rise of code intelligence,” 2024. [Online]. Available: https://arxiv.org/abs/2401.14196

    2024 arXiv

  3. [11]

    Code llama: Open foundation models for code,

    B. Rozi `ere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. C. Ferrer, A. Grattafiori, W. Xiong, A. D ´efossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier...

    2024 arXiv

  4. [12]

    Design2code: How far are we from automating front-end engineering?

    C. Si, Y . Zhang, Z. Yang, R. Liu, and D. Yang, “Design2code: How far are we from automating front-end engineering?” 2024. [Online]. Available: https://arxiv.org/abs/2403.03163

    2024 arXiv

  5. [13]

    Forecasting the future of artificial intelligence with machine learning-based link prediction in an exponentially growing knowledge network,

    M. Krenn, L. Buffoni, B. Coutinho, S. Eppel, J. G. Foster, A. Gritsevskiy, H. Lee, Y . Lu, J. P. Moutinho, N. Sanjabi, R. Sonthalia, N. M. Tran, F. Valente, Y . Xie, R. Yu, and M. Kopp, “Forecasting the future of artificial intelligence with machine learning-based link predict...

    2023 doi

  6. [14]

    K. A. Ericsson and H. A. Simon, Protocol Analysis: Verbal Reports as Data . The MIT Press, Apr. 1993. [Online]. Available: http://dx.doi.org/10.7551/mitpress/5657.001.0001

    1993 doi

  7. [15]

    Follow that sketch: Lifecycles of diagrams and sketches in software development,

    J. Walny, J. Haber, M. D ¨ork, J. Sillito, and S. Carpendale, “Follow that sketch: Lifecycles of diagrams and sketches in software development,” in 2011 6th International Workshop on Visualizing Software for Under- standing and Analysis (VISSOFT) , 2011, pp. 1–8

    2011

  8. [16]

    Linking sketches and diagrams to source code artifacts,

    S. Baltes, P. Schmitz, and S. Diehl, “Linking sketches and diagrams to source code artifacts,” in Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, ser. FSE 2014. New York, NY , USA: Association for Computing Machinery, 2014, p...

    2014

  9. [17]

    Round-Trip Sketches: Supporting the Lifecycle of Software Development Sketches from Analog to Digital and Back ,

    S. Baltes, F. Hollerich, and S. Diehl, “ Round-Trip Sketches: Supporting the Lifecycle of Software Development Sketches from Analog to Digital and Back ,” in 2017 IEEE Working Conference on Software Visualization (VISSOFT) . Los Alamitos, CA, USA: IEEE Computer Society, Sep. 2...

    2017 doi

  10. [18]

    Let’s Go to the Whiteboard (Again): Perceptions From Software Architects on Whiteboard Architecture Meetings ,

    E. Santana de Almeida, I. Ahmed, and A. van der Hoek, “ Let’s Go to the Whiteboard (Again): Perceptions From Software Architects on Whiteboard Architecture Meetings ,” IEEE Transactions on Software Engineering , vol. 49, no. 10, pp. 4773–4795, Oct

  11. [19]

    pix2code: Generating code from a graphical user interface screenshot,

    T. Beltramelli, “pix2code: Generating code from a graphical user interface screenshot,” in Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems , ser. EICS ’18. New York, NY , USA: Association for Computing Machinery, 2018. [Online]. Available: ...

    2018

  12. [20]

    Learning to infer graphics programs from hand-drawn images,

    K. Ellis, D. Ritchie, A. Solar-Lezama, and J. B. Tenenbaum, “Learning to infer graphics programs from hand-drawn images,” in Proceedings of the 32nd International Conference on Neural Information Processing Systems, ser. NIPS’18. Red Hook, NY , USA: Curran Associates Inc., 201...

    2018

  13. [21]

    Psdoodle: fast app screen search via partial screen doodle,

    S. Mohian and C. Csallner, “Psdoodle: fast app screen search via partial screen doodle,” in Proceedings of the 9th IEEE/ACM International Conference on Mobile Software Engineering and Systems, ser. MOBILESoft ’22. New York, NY , USA: Association for Computing Machinery, 2022, ...

    2022

  14. [22]

    Dlpaper2code: Auto-generation of code from deep learning research papers,

    A. Sethi, A. Sankaran, N. Panwar, S. Khare, and S. Mani, “Dlpaper2code: Auto-generation of code from deep learning research papers,” Proceedings of the AAAI Conference on Artificial Intelligence , vol. 32, no. 1, Apr. 2018. [Online]. Available: http://dx.doi.org/10.1609/ aaai....

    2018

  15. [23]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts,

    P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao, “Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts,” 2024. [Online]. Available: https://arxiv.org/abs/2310.02255

    2024 arXiv

  16. [24]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi,

    X. Yue, Y . Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y . Sun, C. Wei, B. Yu, R. Yuan, R. Sun, M. Yin, B. Zheng, Z. Yang, Y . Liu, W. Huang, H. Sun, Y . Su, and W. Chen, “Mmmu: A massive multi-discipline multimodal understanding and reasoning benc...

    2024 arXiv

  17. [25]

    Benchmarking generative models on computational thinking tests in elementary visual programming,

    V .-A. P ˘adurean and A. Singla, “Benchmarking generative models on computational thinking tests in elementary visual programming,” 2024. [Online]. Available: https://arxiv.org/abs/2406.09891

    2024 arXiv

  18. [26]

    Plot2code: A comprehensive benchmark for evaluating multi-modal large language models in code generation from scientific plots,

    C. Wu, Y . Ge, Q. Guo, J. Wang, Z. Liang, Z. Lu, Y . Shan, and P. Luo, “Plot2code: A comprehensive benchmark for evaluating multi-modal large language models in code generation from scientific plots,” 2024. [Online]. Available: https://arxiv.org/abs/2405.07990

    2024 arXiv

  19. [27]

    Mmcode: Evaluating multi- modal code large language models with visually rich programming problems,

    K. Li, Y . Tian, Q. Hu, Z. Luo, and J. Ma, “Mmcode: Evaluating multi- modal code large language models with visually rich programming problems,” 2024. [Online]. Available: https://arxiv.org/abs/2404.09486

    2024 arXiv

  20. [28]

    Visual sketching: From image sketches to code,

    M. d’Amorim, R. Abreu, and C. Mello, “Visual sketching: From image sketches to code,” in 2020 IEEE/ACM 42nd International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) , 2020, pp. 101–104

    2020

  21. [29]

    Wildvision arena: Benchmarking multimodal llms in the wild,

    Y . Lu, D. Jiang, W. Chen, W. Wang, Y . Choi, and B. Y . Lin, “Wildvision arena: Benchmarking multimodal llms in the wild,” February 2024. [Online]. Available: https://huggingface.co/spaces/ WildVision/vision-arena/

    2024

  22. [30]

    Opencompass: A universal evaluation platform for foundation models,

    O. Contributors, “Opencompass: A universal evaluation platform for foundation models,” https://github.com/open-compass/opencompass, 2023

    2023

  23. [31]

    Selecting em- pirical methods for software engineering research,

    S. Easterbrook, J. Singer, M.-A. Storey, and D. Damian, “Selecting em- pirical methods for software engineering research,” Guide to advanced empirical software engineering , pp. 285–311, 2008

    2008

  24. [32]

    Software engineering data collection for field studies,

    J. Singer, S. E. Sim, and T. C. Lethbridge, “Software engineering data collection for field studies,” Guide to advanced empirical software engineering, pp. 9–34, 2008

    2008

  25. [33]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging llm-as-a-judge with mt-bench and chatbot arena,”

  26. [34]

    Chatbot arena: An open platform for evaluating llms by human preference,

    W.-L. Chiang, L. Zheng, Y . Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, and I. Stoica, “Chatbot arena: An open platform for evaluating llms by human preference,” 2024

    2024

  27. [35]

    Available: https://arxiv.org/abs/2306.05685

    [Online]. Available: https://arxiv.org/abs/2306.05685

    arXiv

  28. [37]

    On the limitations of fine-tuned judge models for llm evaluation,

    H. Huang, Y . Qu, H. Zhou, J. Liu, M. Yang, B. Xu, and T. Zhao, “On the limitations of fine-tuned judge models for llm evaluation,” 2024. [Online]. Available: https://arxiv.org/abs/2403.02839

    2024 arXiv

  29. [2023]

    Available: https://doi.ieeecomputersociety.org/10.1109/ TSE.2023.3314410

    [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/ TSE.2023.3314410

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.