Pith. sign in

REVIEW 3 major objections 7 minor 126 references

Multilingual Multimodal Software Developer for Code Generation

T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 7-billion-parameter model trained to read UML diagrams and flowcharts alongside text outperforms much larger models, including a 72-billion-parameter one, on a new 10-language code-generation benchmark.

desk verdict Solid two-stage SFT recipe for diagram-conditioned code gen, undermined by an in-distribution 300-problem benchmark and a noisy headline comparison to a 72B model. read the letter →

arxiv 2507.08719 v1 pith:MN6Z7RKN submitted 2025-07-11 cs.CL cs.AIcs.SE

Linzheng Chai , Jian Yang , Shukai Liu , Wei Zhang , Liran Wang , Ke Jin , Tao Sun , Congnan Liu
show 7 more authors
Chenchen Zhang Hualei Zhu Jiaheng Liu Xianjie Wu Ge Zhang Tianyu Liu Zhoujun Li
This is my paper · ORCID
classification cs.CLcs.AIcs.SE
keywords multimodalcodegenerationvisualworkflowUMLdiagramsflowchartsinstructiontuningmultilingualevaluationM2EVALbenchmarkPass@1
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that code generation gets better when the model can see the software design: UML diagrams and flowcharts, which the authors call Visual Workflow, carry information that text prompts alone do not. To test the idea, the authors built M2C-INSTRUCT, a 13.1-million-sample instruction-tuning corpus, and M2EVAL, a new benchmark of 300 problems across 10 programming languages in which key details are deliberately placed only inside the diagram. Their 7-billion-parameter model, M2-Coder, reaches 25.3 average Pass@1 — the share of generated programs that pass all test cases — beating every similarly sized model and exceeding the 72-billion-parameter Qwen2.5-VL at 24.7. The result matters because software teams routinely communicate through diagrams, and a small open model that can read them would turn design intent directly into working code.

What carries the argument

The load-bearing object is the Visual Workflow: UML class and sequence diagrams and Mermaid-rendered flowcharts generated from code by an LLM, then manually refined and re-embedded into problems. Two mechanisms carry the argument. The first is the diagram-necessity construction principle: when a text problem is converted into a multimodal problem, information is removed from the prompt and re-encoded inside the diagram, so the image is the unique carrier of facts the code must reflect. The second is the two-stage instruction-tuning recipe: stage 1 aligns the model with rendered code images and OCR-heavy inputs at scale, and stage 2, with the vision tower frozen, sharpens diagram comprehension and instruction following on shorter, higher-quality problems. M2EVAL applies the same necessity principle in reverse, stripping prompts until they are unsolvable without the diagram, and scores model outputs by executing them against test cases in ten languages.

What would settle it

Re-run M2EVAL's 30 problems with the same content but diagrams re-rendered in a different style, layout, or drawing tool, or replaced by hand-drawn versions. If M2-Coder-7B's Pass@1 falls sharply while GPT-4o's holds, the gains come from style matching rather than diagram reading. As a second check, give text-only models such as Qwen2.5-Coder-32B the diagram content rewritten as plain text inside the prompt: if they then match or beat M2-Coder, the model's advantage is the extra information the diagram carries, not the visual channel itself.

Watch

Extended reading notes

Core claim

The central claim is that large-scale multimodal instruction tuning on visual workflow data turns a standard vision-language model into a multilingual code generator that uses design diagrams. Starting from Qwen2-VL-7B, the authors fine-tune in two stages: first on 12.9 million cross-modal problems, in which source code is rendered as syntax-highlighted images alongside diagram problems, then on 168 thousand high-quality diagram problems whose essential information is encoded only in the image. On M2EVAL, the resulting M2-Coder-7B scores 25.3 average Pass@1 across ten programming languages, above every other open-weight model evaluated up to 400 billion parameters, including Qwen2.5-VL-72B at 24.7, and close to proprietary leaders such as GPT-4o at 49.7. The paper also shows that text-only code models score 0.0 when the diagram is withheld, evidence that the diagrams are genuinely load-bearing, and that the two-stage recipe beats either stage alone (10.0 and 18.0 individually, 25.3 combined).

Load-bearing premise

The benchmark that measures success, M2EVAL, is built with the same LLM-generated Mermaid/PlantUML diagram pipeline as the training data, so the reported Pass@1 numbers could reflect mastery of synthetic diagram styles rather than of real-world design documents; with only 30 unique concepts, each diagram is also reused across ten languages.

Editorial extensions

If this is right

  • A 7-billion-parameter open-weight model can post 25.3 average Pass@1 on multilingual multimodal code generation, above most models an order of magnitude larger, making diagram-reading a small-model capability rather than a frontier-model one.
  • Even the strongest evaluated model, GPT-4o, reaches only about 50 percent, so M2EVAL leaves clear headroom; the paper names precise visual capture, instruction following, and design-pattern knowledge as the open bottlenecks.
  • Text-only code models score 0.0 when the diagram is withheld, so software workflows that communicate through diagrams will need multimodal components rather than text-only code LLMs.
  • Design-pattern problems are the hardest category for every model, and strictly typed languages (C#, Scala, Swift) trail scripting languages (Python, PHP, JavaScript), pointing to where data and evaluation effort should go next.
  • The dataset, not the base model, drives most of the gain: stage-2 fine-tuning alone moves Qwen2-VL-Instruct from 12.0 to 16.7, and the full two-stage pipeline reaches 25.3.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because M2EVAL and M2C-INSTRUCT share the same LLM-assisted diagram pipeline, I would expect Pass@1 to drop on hand-drawn or professionally authored diagrams; a re-rendered dev set would quantify how much of the result is genuine visual understanding.
  • The diagram-necessity trick is portable: the same strip-text-and-re-encode-in-image recipe could be applied to UI mockups, database schemas, or API specifications to build multimodal training data for other software-engineering tasks.
  • Stage-1 data volume keeps helping, so the 13.1-million-instance budget is not exhausted; a cheaper question worth testing is whether a much smaller, diagram-only corpus could capture most of the gain, since stage 2 alone already contributes a large share.
  • If the gains transfer across rendering styles, then code-from-diagram generation could be plugged into industrial design tools that already export Mermaid or PlantUML, giving the approach a natural deployment path.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes M2-Coder, a 7B-parameter vision-language model for code generation from UML diagrams and flowcharts ('visual workflows'), trained in two stages on a newly constructed 13.1M-instance instruction-tuning corpus M2C-Instruct. It also introduces M2Eval, a 300-problem benchmark covering 10 programming languages built from 30 concepts, each with one LLM-generated diagram shared across languages. On M2Eval, M2-Coder-7B reports a 25.3 average Pass@1, higher than all evaluated models of comparable size and nominally higher than Qwen2.5-VL-72B (24.7). The authors argue this demonstrates the value of large-scale multimodal instruction tuning and exposes remaining weaknesses of current LMMs in visual understanding and instruction following.

Significance. The resources are potentially valuable: M2C-Instruct is large and multilingual, M2Eval is execution-based with human-verified translations and test cases, and the two-stage SFT recipe gives a substantial improvement over the Qwen2-VL-7B base (25.3 vs 12.0). The paper also reports detailed annotation protocols, payment information, and plans to release code and data, which support reproducibility. However, the central comparative claim against larger models is not statistically supported (a 2-problem difference out of 300), and the benchmark/training distribution overlap means the advantage over other models may partly reflect style matching rather than general multimodal coding ability. With added uncertainty quantification and a robustness analysis against diagram style, the contribution would be a useful step for multimodal code generation.

major comments (3)
  1. [§4.2, Table 4, Figure 17] The headline result that M2-Coder-7B outperforms Qwen2.5-VL-72B rests on a 0.6-point gap (25.3 vs 24.7 average Pass@1), which in Figure 17 corresponds to 76 versus 74 correct (problem, language) pairs out of 300. With a single greedy decoding run and no confidence intervals, the standard error of a 25% pass rate on 300 items is about 2.5 points, so this difference is not distinguishable from sampling noise. The authors should report multiple runs or a paired significance test (e.g., McNemar's test on the 300 items) before claiming superiority, or explicitly soften the claim to 'comparable within noise.' As written, contribution bullet 3 ('competitiveness with larger 70B+ LMMs') is not supported by this comparison, although the larger gaps against 7B baselines (e.g., Qwen2-VL-7B at 12.0) are more robust.
  2. [§2.2, §3.2, Appendices C.2, D.2] M2EVAL and the diagram subset of M2C-INSTRUCT are generated by the same LLM pipeline: Qwen2.5-Coder produces PlantUML/Mermaid diagrams from a code problem and then rewrites the prompt so that key details appear only in the diagram. Because M2-Coder is trained on M2C-INSTRUCT, it is evaluated on the same synthetic diagram style that it saw during training, while all comparison models encounter that style for the first time. This distributional overlap is a plausible source of M2-Coder's advantage beyond actual multimodal coding skill. The paper should (i) explicitly discuss this overlap, (ii) measure sensitivity to diagram style, e.g., by testing on human-created or out-of-pipeline diagrams, and (iii) report whether the performance gap persists when comparison models are given additional diagram examples or when M2-Coder is evaluated on a held-out style. Without such analysis, the comparative results in Table 4 are open to an overfitting explanation.
  3. [§2.3, Figure 17] The effective visual diversity of M2EVAL is 30 unique diagrams (one per concept, shared across the 10 language versions), so the 300-problem count overstates the independent sample size for the visual component. Per-language Pass@1 scores are computed on 30 items, giving standard errors around 8-9 percentage points; the per-language differences in Table 4 (e.g., C# 26.7 vs 16.7) are therefore not meaningful without uncertainty estimates. The authors should report confidence intervals or cluster the analysis by the 30 concepts (e.g., bootstrap by concept) when comparing models.
minor comments (7)
  1. [Abstract and throughout] The naming is inconsistent: the abstract and contributions use 'MM-Coder', 'MMc-Instruct', and 'MMEval', while the body and tables use M2-Coder, M2C-INSTRUCT, and M2EVAL. Please unify the terminology.
  2. [Table 3] The M2EVAL row lists '#Train 13.1M' and a checkmark under 'Instruct', which conflates the 300-problem evaluation benchmark with the 13.1M-instance M2C-INSTRUCT training corpus. M2EVAL should be listed as evaluation-only, with the training-corpus statistics attributed to M2C-INSTRUCT.
  3. [§4.1] The evaluation protocol mixes greedy decoding for 'ordinary' models with temperature 0.6 sampling for 'thinking' models, and no seed averaging is reported for the sampling-based results. This makes the Pass@1 values for thinking models (e.g., Doubao1.5-thinking-pro, QVQ, Kimi) not directly comparable to greedy scores; please either use a uniform protocol or report both sets of numbers and justify the choice.
  4. [§4.2 and §2.2] The 0.0 scores for the text-only models DeepSeek-V3 and Qwen2.5-Coder are expected by construction: §2.2 states that the curation process removes from the prompt all details that are moved into the diagram, making the prompt alone insufficient. This should not be presented as evidence that diagrams are generally necessary for code generation in real-world settings.
  5. [Table 5] The ablation table omits the score for Qwen2-VL-Base without any fine-tuning (model ③ shows dashes). Without this baseline, the ablation cannot separate the effect of stage-1 SFT from the base model's own capability or inability to follow the instruction format; please add the missing baseline or explain why it cannot be evaluated.
  6. [Appendix D.1 and Figure 6] There is a typo ('Futhermore') in Appendix D.1, and Figure 6's 'data efficiency' analysis lacks a description of how many checkpoints were used and whether the trend is based on a single run. Please clarify the experimental design behind this figure.
  7. [Evaluation methodology] The paper does not report a contamination check between M2Eval and the training corpora of the evaluated models. Since M2C-INSTRUCT is derived from GitHub code and LLM-generated problems, the possibility that M2Eval's prototype problems or solutions appear in pretraining corpora should be addressed, e.g., by a fuzzy-match or near-duplicate analysis.

Circularity Check

1 steps flagged · score 4.0 of 10

One peripheral claim ('diagrams are necessary') is true by construction, but the central training/evaluation results are not circular.

  1. self definitional [Section 2.2 (Problem Design) and Section 4.2 (Main Results)]
    "We revise the problem prompts by removing information redundant with the diagrams, making the prompt alone insufficient for a correct solution. ... The results show that DeepSeek-V3 and Qwen2.5-Coder cannot solve the problems without diagrams. This proves that diagrams are necessary in M2EVAL(we paid special attention to this during annotation)."

    The paper presents the 0% text-only result as empirical proof that diagrams are necessary, but this outcome is guaranteed by the benchmark construction itself: the curation protocol deliberately removed information required for a correct solution from the prompt and moved it into the diagram. The phrase 'making the prompt alone insufficient for a correct solution' makes the later claim 'this proves that diagrams are necessary' a restatement of the design choice rather than an independent finding. No model evaluation is needed to establish that a text-only model cannot solve a problem whose prompt was intentionally made unsolvable without the diagram.

full rationale

The central training-and-evaluation loop is not circular: M2-Coder is trained on M2C-INSTRUCT and scored on M2EVAL by executing generated code against held-out test cases, so the Pass@1 numbers do not reduce to a fitted parameter or to the training objective. The one clear self-definitional step is the 'diagrams are necessary' claim in Section 4.2, which follows by construction from the prompt-curation rule in Section 2.2; however, this is a peripheral observation rather than the paper's main contribution. A related but non-circular concern is that M2EVAL was built with the same LLM-based diagram-generation recipe as M2C-INSTRUCT (Sections 2.2 and D.2), making the benchmark in-distribution for the trained model; this is a benchmark-validity and generalization risk, not a logical reduction of the reported scores. The central claims of improved multimodal code generation and competitive performance remain independently measured by execution-based evaluation, so the overall circularity is moderate rather than severe.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the representativeness of LLM-generated diagrams and the correctness of the self-built benchmark. No numeric free parameters are fit, but the data-generation protocol embeds several hand-chosen design decisions, including the diagram tools, the removal strategy, and the filtering thresholds.

assumptions (4)
  • domain assumption UML diagrams and flowcharts are an appropriate representation of visual design intent for code generation.
    Section 1 introduces Visual Workflow as the input modality; if real-world design artifacts are not well captured by these diagram types, the claimed practical utility weakens.
  • domain assumption Qwen2.5-Coder-generated diagrams and multimodal problems, filtered by Mermaid rendering, are high-quality enough for training and evaluation.
    Section 3.2 and Appendix D.2 rely on the LLM to produce diagrams and incomplete problems; the entire approach depends on the generator's quality.
  • ad hoc to paper Removing problem details from the prompt and placing them in the diagram makes the prompt alone insufficient.
    Section 2.2 and Appendix C.2 enforce this by construction, which guarantees text-only models score 0% but is not an independent property of natural problems.
  • domain assumption Canonical solutions and test cases in M2EVAL are correct and sufficient.
    Appendix C.3.4 asserts a 100% pass rate for canonical solutions, but the test cases are authored by the same team and limited to an average of 9 per problem.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multilingual Multimodal Software Developer for Code Generation." pith.science (2026). https://pith.science/paper/MN6Z7RKN

@misc{pith2026250708719,
  author       = {Pith},
  title        = {Pith review of: Multilingual Multimodal Software Developer for Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MN6Z7RKN}},
  note         = {Machine review of arXiv:2507.08719}
}
read the original abstract

The rapid advancement of Large Language Models (LLMs) has significantly improved code generation, yet most models remain text-only, neglecting crucial visual aids like diagrams and flowcharts used in real-world software development. To bridge this gap, we introduce MM-Coder, a Multilingual Multimodal software developer. MM-Coder integrates visual design inputs-Unified Modeling Language (UML) diagrams and flowcharts (termed Visual Workflow)-with textual instructions to enhance code generation accuracy and architectural alignment. To enable this, we developed MMc-Instruct, a diverse multimodal instruction-tuning dataset including visual-workflow-based code generation, allowing MM-Coder to synthesize textual and graphical information like human developers, distinct from prior work on narrow tasks. Furthermore, we introduce MMEval, a new benchmark for evaluating multimodal code generation, addressing existing text-only limitations. Our evaluations using MMEval highlight significant remaining challenges for models in precise visual information capture, instruction following, and advanced programming knowledge. Our work aims to revolutionize industrial programming by enabling LLMs to interpret and implement complex specifications conveyed through both text and visual designs.

Figures

Figures reproduced from arXiv: 2507.08719 by the authors.

Figure 1
Figure 1. Overview of M2 -CODER, M2C-INSTRUCT and M2EVAL. We constructed M2C-INSTRUCT, an instruction tuning dataset with over 13.1 million instances, to enhance the multilingual multimodal programming capabilities of M2 -CODER. For evaluating M2 -CODER, we curate M2EVAL, a multi￾modal benchmark for code evaluation across 10 programming languages. system architecture/design patterns/process workflows in [PITH_FULL_IMAGE:figu… view at source ↗
Figure 2
Figure 2. The curation process for M2EVAL: (1) designing Python prototype problems grounded in core programming concepts; (2) transforming these into multimodal problems by incorporating diagrams and refining prompts; and (3) translating the problems into multiple programming languages. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. PL distribution of M2C￾INSTRUCT. 159-1978px (height) and 338-2081px (width). All 300 problems have solutions, averaging 326 tokens (max 826). Each solution is evaluated against an average of 9 test cases. In [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (31 more)
Figure 4
Figure 4. Figure 4: The M2C-INSTRUCT data construction pipeline. Stage-1 data generates Cross-Modal Problems by converting code to images. Stage-2 data generates Diagram Problems via a two-step process: (1) creating diagrams from problems/solutions, and (2) formulating multimodal problems…
Figure 6
Figure 6. Figure 6: Ablation study on stage-1 1 data efficiency. Ablation experiment for M2 -CODER fine-tuning. We perform an ablation study on M2 -CODER and present the results in [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Distribution of GPT￾4o’s Errors in Python. Error Analysis. To gain deeper insights into model capabilities, we conducted an error analysis on GPT-4o’s responses to 11 prob￾lems in Python. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: A case study for M2EVAL showcasing common error patterns in multimodal code gener￾ation. It highlights failures on specific challenges (complex sorting, diagram-to-code with naming conventions). Programming Language Matters. Extending the analysis of language-specific …
Figure 9
Figure 9. Figure 9: Performance differences between each PLs on M [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Programming language distribution in M2C-INSTRUCT stage 1 data M2C-INSTRUCT Stage 2. D.2 M2C-INSTRUCT Construction Process This section provides a detailed account of the specific methodologies and procedures used for data synthesis and quality control during the cons…
Figure 11
Figure 11. Figure 11: Programming language distribution of M2C-INSTRUCT Stage 2. Source Data Preparation. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Examples of rendered code images via pygments. [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: Examples of diagram code images via mermaid. [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]
Figure 14
Figure 14. Figure 14: Prompt template for step 1 of constructing diagram-type data in M [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]
Figure 15
Figure 15. Figure 15: Prompt template for step 2 of constructing diagram-type data in M [PITH_FULL_IMAGE:figures/full_fig_p027_15.png]
Figure 16
Figure 16. Figure 16: Performance comparison of models across task types. 1 [PITH_FULL_IMAGE:figures/full_fig_p029_16.png]
Figure 17
Figure 17. Figure 17: shows the model’s performance on different problems (a score of 10 indicates that the code in all languages passes the tests). It can be seen that on some simple problems (e.g., problems 1, 2, 16, and 19), most models perform well in all languages. However, on some co…
Figure 18
Figure 18. Figure 18: Kotlin error cases from GPT-4o on M2EVAL. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_18.png]
Figure 19
Figure 19. Figure 19: Ruby error cases from GPT-4o on M2EVAL. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_19.png]
Figure 20
Figure 20. Figure 20: Cross-modal example 1 from M2C-INSTRUCT. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_20.png]
Figure 21
Figure 21. Figure 21: Cross-modal example 2 from M2C-INSTRUCT. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_21.png]
Figure 22
Figure 22. Figure 22: Cross-modal example 3 from M2C-INSTRUCT. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_22.png]
Figure 23
Figure 23. Figure 23: Diagram example 1 from M2C-INSTRUCT. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_23.png]
Figure 24
Figure 24. Figure 24: Diagram example 2 from M2C-INSTRUCT. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_24.png]
Figure 25
Figure 25. Figure 25: Diagram example 3 from M2C-INSTRUCT. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_25.png]
Figure 26
Figure 26. Figure 26: Diagram example 4 from M2C-INSTRUCT. 38 [PITH_FULL_IMAGE:figures/full_fig_p038_26.png]
Figure 27
Figure 27. Figure 27: Diagram example 5 from M2C-INSTRUCT. 39 [PITH_FULL_IMAGE:figures/full_fig_p039_27.png]
Figure 28
Figure 28. Figure 28: Example 1 from M2EVAL. 40 [PITH_FULL_IMAGE:figures/full_fig_p040_28.png]
Figure 29
Figure 29. Figure 29: Example 2 from M2EVAL. 41 [PITH_FULL_IMAGE:figures/full_fig_p041_29.png]
Figure 30
Figure 30. Figure 30: Example 3 from M2EVAL. 42 [PITH_FULL_IMAGE:figures/full_fig_p042_30.png]
Figure 31
Figure 31. Figure 31: Example 4 from M2EVAL. 43 [PITH_FULL_IMAGE:figures/full_fig_p043_31.png]
Figure 32
Figure 32. Figure 32: Example 5 from M2EVAL. 44 [PITH_FULL_IMAGE:figures/full_fig_p044_32.png]
Figure 33
Figure 33. Figure 33: Example 6 from M2EVAL. 45 [PITH_FULL_IMAGE:figures/full_fig_p045_33.png]
Figure 34
Figure 34. Figure 34: Example 7 from M2EVAL. 46 [PITH_FULL_IMAGE:figures/full_fig_p046_34.png]
Figure 35
Figure 35. Figure 35: Example 8 from M2EVAL. 47 [PITH_FULL_IMAGE:figures/full_fig_p047_35.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

126 extracted references · 9 canonical work pages

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

    arXiv 2024

  2. [2]

    Introducing meta llama 3: The most capable openly available llm to date

    Meta AI. Introducing meta llama 3: The most capable openly available llm to date. https: //ai.meta.com/blog/meta-llama-3/ , apr 2024

    2024

  3. [3]

    Aider llm leaderboards

    Aider Team. Aider llm leaderboards

  4. [4]

    SantaCoder: Don’t reach for the stars! arXiv preprint arXiv:2301.03988, 2023

    Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Car- los Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. SantaCoder: Don’t reach for the stars! arXiv preprint arXiv:2301.03988, 2023

    arXiv 2023

  5. [5]

    The claude 3 model family: Opus, sonnet, haiku

    Anthropic. The claude 3 model family: Opus, sonnet, haiku. Technical report, Anthropic, 2024

    2024

  6. [6]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425–2433, 2015

    2015

  7. [7]

    Multi-lingual evaluation of code generation models

    Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Gi- aquinto, Haifeng Qian, Murali Krishna Ramanathan, and Ramesh Nallapati. Multi-lingual evaluation of cod...

    2023

  8. [8]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

Show all 126 references
  1. [9]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Si...

    2023 arXiv

  2. [10]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023

    2023 arXiv

  3. [11]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Han...

    2025 arXiv

  4. [12]

    Codeplan: Repository-level coding using llms and planning

    Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, Arun Iyer, Suresh Parthasarathy, Sriram Rajamani, B Ashok, and Shashank Shet. Codeplan: Repository-level coding using llms and planning. Proceedings of the ACM on Software Engineering , 1(FSE):675–698, 2024

    2024

  5. [13]

    Multipl-e: A scalable and polyglot approach to benchmarking neural code generation

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multipl-e: A scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on S...

    2023

  6. [14]

    Mceval: Massively multilingual code evaluation

    Linzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin, Ke Jin, Jiaheng Liu, Tao Sun, Ge Zhang, Changyu Ren, Hongcheng Guo, et al. Mceval: Massively multilingual code evaluation. arXiv preprint arXiv:2406.07436, 2024

    2024 arXiv

  7. [15]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  8. [16]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024

    2024 arXiv

  9. [17]

    Gemini 2.5 pro preview: even better coding performance, 2025

    Google Deepmind. Gemini 2.5 pro preview: even better coding performance, 2025

    2025

  10. [18]

    R2c2-coder: Enhancing and benchmarking real-world repository-level code completion abilities of code large language models

    Ken Deng, Jiaheng Liu, He Zhu, Congnan Liu, Jingxin Li, Jiakai Wang, Peng Zhao, Chenchen Zhang, Yanan Wu, Xueqiao Yin, et al. R2c2-coder: Enhancing and benchmarking real-world repository-level code completion abilities of code large language models. arXiv preprint arXiv:2406.0...

    2024 arXiv

  11. [19]

    BERT: pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Lan...

    2019

  12. [20]

    Codebert: A pre-trained model for programming and natural languages

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. Codebert: A pre-trained model for programming and natural languages. In Trevor Cohn, Yulan He, and Yang Liu, editors, Findings of the Associati...

    2020

  13. [21]

    Gemma: Open models based on gemini research and technology

    Google Gemma Team. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  14. [22]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6904–6913, 2017

    2017

  15. [23]

    Deepseek-coder: When the large language model meets programming – the rise of code intelligence

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming – the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024

    2024 arXiv

  16. [24]

    Chartllama: A multimodal llm for chart understanding and generation

    Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, and Hanwang Zhang. Chartllama: A multimodal llm for chart understanding and generation. arXiv preprint arXiv:2311.16483, 2023

    2023 arXiv

  17. [25]

    On distribution shift in learning-based bug detectors

    Jingxuan He, Luca Beurer-Kellner, and Martin Vechev. On distribution shift in learning-based bug detectors. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Lear...

    2022

  18. [26]

    Olympicarena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai

    Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyumanshan Ye, Ethan Chern, Yixin Ye, et al. Olympicarena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai. arXiv preprint arXiv:2406.12753, 2024

    2024 arXiv

  19. [27]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Ji- ajun Zhang, Bowen Yu, Kai Dang, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024. 11

    2024 arXiv

  20. [28]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  21. [29]

    Livecodebench: Holistic and contami- nation free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contami- nation free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  22. [30]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  23. [31]

    Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    2023 arXiv

  24. [32]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems...

    2023

  25. [33]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  26. [34]

    Mmcode: Evaluating multi- modal code large language models with visually rich programming problems

    Kaixin Li, Yuchen Tian, Qisheng Hu, Ziyang Luo, and Jing Ma. Mmcode: Evaluating multi- modal code large language models with visually rich programming problems. arXiv preprint arXiv:2404.09486, 2024

    2024 arXiv

  27. [35]

    Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161, abs/2305.06161, 2023

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, ...

    2023 arXiv

  28. [36]

    Towards real-world writing assistance: A Chinese character checking benchmark with faked and misspelled characters

    Yinghui Li, Zishan Xu, Shaoshen Chen, Haojing Huang, Yangning Li, Shirong Ma, Yong Jiang, Zhongli Li, Qingyu Zhou, Hai-Tao Zheng, and Ying Shen. Towards real-world writing assistance: A Chinese character checking benchmark with faked and misspelled characters. In Lun-Wei Ku, A...

    2024

  29. [37]

    Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, ...

    2022 arXiv

  30. [38]

    Quixbugs: a multi- lingual program repair benchmark set based on the quixey challenge

    Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama. Quixbugs: a multi- lingual program repair benchmark set based on the quixey challenge. In Proceedings Compan- ion of the 2017 ACM SIGPLAN international conference on systems, programming, languages, and applicat...

    2017

  31. [39]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  32. [40]

    Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    2024

  33. [41]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023

    2023 arXiv

  34. [42]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. arXiv preprint arXiv:2305.01210, abs/2305.01210, 2023

    2023 arXiv

  35. [43]

    Repobench: Benchmarking repository-level code auto-completion systems

    Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXiv:2306.03091, 2023

    2023 arXiv

  36. [44]

    Logomotion: Visually grounded code generation for content-aware animation

    Vivian Liu, Rubaiat Habib Kazi, Li-Yi Wei, Matthew Fisher, Timothy Langlois, Seth Walker, and Lydia Chilton. Logomotion: Visually grounded code generation for content-aware animation. arXiv preprint arXiv:2405.07065, 2024

    2024 arXiv

  37. [45]

    Visualagentbench: Towards large multimodal models as visual foundation agents

    Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long Iong, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, et al. Visualagentbench: Towards large multimodal models as visual foundation agents. arXiv preprint arXiv:2408.06327, 2024

    2024 arXiv

  38. [46]

    Code genera- tion from flowcharts with texts: A benchmark dataset and an approach

    Zejie Liu, Xiaoyu Hu, Deyu Zhou, Lin Li, Xu Zhang, and Yanzheng Xiang. Code genera- tion from flowcharts with texts: A benchmark dataset and an approach. In Findings of the Association for Computational Linguistics: EMNLP 2022 , pages 6069–6077, 2022

    2022

  39. [47]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  40. [48]

    Starcoder 2 and the stack v2: The next generation

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173, 2024

    2024 arXiv

  41. [49]

    WizardCoder: Empowering code large language models with evol-instruct

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. WizardCoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568, 2023

    2023 arXiv

  42. [50]

    Llama 4, leading intelligence., 2025

    Meta. Llama 4, leading intelligence., 2025

    2025

  43. [51]

    Robocodex: Multimodal code generation for robotic behavior synthesis

    Yao Mu, Junting Chen, Qinglong Zhang, Shoufa Chen, Qiaojun Yu, Chongjian Ge, Runjian Chen, Zhixuan Liang, Mengkang Hu, Chaofan Tao, et al. Robocodex: Multimodal code generation for robotic behavior synthesis. arXiv preprint arXiv:2402.16117, 2024

    2024 arXiv

  44. [52]

    OctoPack: Instruction tuning code large language models

    Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. OctoPack: Instruction tuning code large language models. arXiv preprint arXiv:2308.07124, abs/2308.07124, 2023

    2023 arXiv

  45. [53]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  46. [54]

    Introducing gpt-4.5, 2025

    OpenAI. Introducing gpt-4.5, 2025

    2025

  47. [55]

    Runbugrun – an executable dataset for automated program repair

    Julian Aron Prenner and Romain Robbes. Runbugrun – an executable dataset for automated program repair. arXiv preprint arXiv:2304.01102, 2023. 13

    2023 arXiv

  48. [56]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , p...

    2021

  49. [57]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. OpenAI blog, 2018

    2018

  50. [58]

    Code llama: Open foundation models for code

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  51. [59]

    Bloom: A 176b-parameter open-access multilingual language model

    Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022

    2022 arXiv

  52. [60]

    Seed-thinking-v1.5: Advancing superb reasoning models with reinforcement learning

    ByteDance Seed, Yufeng Yuan, Yu Yue, Mingxuan Wang, Xiaochen Zuo, Jiaze Chen, Lin Yan, Wenyuan Xu, Chi Zhang, Xin Liu, et al. Seed-thinking-v1.5: Advancing superb reasoning models with reinforcement learning. arXiv preprint arXiv:2504.13914, 2025

    2025

  53. [61]

    Visual cot: Unleashing chain-of-thought reasoning in multi-modal language models

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Unleashing chain-of-thought reasoning in multi-modal language models. arXiv preprint arXiv:2403.16999, 2024

    2024 arXiv

  54. [62]

    Chartmimic: Evaluating lmm’s cross-modal reasoning capability via chart-to-code generation

    Chufan Shi, Cheng Yang, Yaxin Liu, Bo Shui, Junjie Wang, Mohan Jing, Linran Xu, Xinyu Zhu, Siheng Li, Yuxiang Zhang, et al. Chartmimic: Evaluating lmm’s cross-modal reasoning capability via chart-to-code generation. arXiv preprint arXiv:2406.09961, 2024

    2024 arXiv

  55. [63]

    Design2code: How far are we from automating front-end engineering? arXiv preprint arXiv:2403.03163, 2024

    Chenglei Si, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. Design2code: How far are we from automating front-end engineering? arXiv preprint arXiv:2403.03163, 2024

    2024 arXiv

  56. [64]

    Flowvqa: Mapping multimodal logic in visual question answering with flowcharts

    Shubhankar Singh, Purvi Chaurasia, Yerram Varun, Pranshu Pandya, Vatsal Gupta, Vivek Gupta, and Dan Roth. Flowvqa: Mapping multimodal logic in visual question answering with flowcharts. arXiv preprint arXiv:2406.19237, 2024

    2024 arXiv

  57. [65]

    UniCoder: Scaling code large language model via universal code

    Tao Sun, Linzheng Chai, Jian Yang, Yuwei Yin, Hongcheng Guo, Jiaheng Liu, Bing Wang, Liqun Yang, and Zhoujun Li. UniCoder: Scaling code large language model via universal code. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of ...

    2024

  58. [66]

    Mtvqa: Benchmarking multilingual text-centric visual question answering

    Jingqun Tang, Qi Liu, Yongjie Ye, Jinghui Lu, Shu Wei, Chunhui Lin, Wanqing Li, Mohamad Fitri Faiz Bin Mahmood, Hao Feng, Zhen Zhao, et al. Mtvqa: Benchmarking multilingual text-centric visual question answering. arXiv preprint arXiv:2405.11985, 2024

    2024 arXiv

  59. [67]

    Flowchartqa: the first large-scale benchmark for reasoning over flowcharts

    Simon Tannert, Marcelo G Feighelstein, Jasmina Bogojeska, Joseph Shtok, Assaf Arbelle, Peter WJ Staar, Anika Schumann, Jonas Kuhn, and Leonid Karlinsky. Flowchartqa: the first large-scale benchmark for reasoning over flowcharts. In Proceedings of the 1st Workshop on Linguistic...

    2023

  60. [68]

    Gemma 3 technical report

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025

    2025 arXiv

  61. [69]

    Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, Dehao Zhang, Dikang Du, Dongliang Wang, Enming Yuan, Enzhe Lu, Fang Li, Flood Sung, Guangda Wei, Guokun Lai, Han Zhu, Hao Ding, Hao Hu, Hao Yan...

    2025

  62. [70]

    Qvq: To see the world with wisdom, December 2024

    Qwen Team. Qvq: To see the world with wisdom, December 2024

    2024

  63. [71]

    Debugbench: Evaluating debugging capability of large language models

    Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Debugbench: Evaluating debugging capability of large language models. arXiv preprint arXiv:2401.04621, 2024

    2024 arXiv

  64. [72]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  65. [73]

    Solving olympiad geometry without human demonstrations

    Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476–482, 2024

    2024

  66. [74]

    Code-vision: Evaluating multimodal llms logic understanding and code generation capabilities

    Hanbin Wang, Xiaoxuan Zhou, Zhipeng Xu, Keyuan Cheng, Yuxin Zuo, Kai Tian, Jingwei Song, Junting Lu, Wenhui Hu, and Xueyang Liu. Code-vision: Evaluating multimodal llms logic understanding and code generation capabilities. arXiv preprint arXiv:2502.11829, 2025

    2025 arXiv

  67. [75]

    Delving into commit-issue correlation to enhance commit message generation models

    Liran Wang, Xunzhu Tang, Yichen He, Changyu Ren, Shuhua Shi, Chaoran Yan, and Zhoujun Li. Delving into commit-issue correlation to enhance commit message generation models. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) , pages 710–722. ...

    2023

  68. [76]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  69. [77]

    Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859, 2021

    2021 arXiv

  70. [78]

    Execution-based evaluation for open-domain code generation

    Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig. Execution-based evaluation for open-domain code generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1271–1290, 2023

    2023

  71. [79]

    Charxiv: Charting gaps in realistic chart understanding in multimodal llms

    Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, et al. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. arXiv preprint arXiv:2406.18521, 2024

    2024 arXiv

  72. [80]

    Magicoder: Source code is all you need

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code is all you need. arXiv preprint arXiv:2312.02120, abs/2312.02120, 2023

    2023 arXiv

  73. [81]

    Plot2code: A comprehensive benchmark for evaluating multi-modal large language models in code generation from scientific plots

    Chengyue Wu, Yixiao Ge, Qiushan Guo, Jiahao Wang, Zhixuan Liang, Zeyu Lu, Ying Shan, and Ping Luo. Plot2code: A comprehensive benchmark for evaluating multi-modal large language models in code generation from scientific plots. arXiv preprint arXiv:2405.07990, 2024

    2024 arXiv

  74. [82]

    Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding, 2024

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang,...

    2024

  75. [83]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972, 2024. 15

    2024 arXiv

  76. [84]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  77. [85]

    Swe-bench multimodal: Do ai systems generalize to visual software domains? arXiv preprint arXiv:2410.03859, 2024

    John Yang, Carlos E Jimenez, Alex L Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R Narasimhan, et al. Swe-bench multimodal: Do ai systems generalize to visual software domains? arXiv preprint arXiv:2410.03859, 2024

    arXiv

  78. [86]

    Matplotagent: Method and evaluation for llm-based agentic scientific data visualization

    Zhiyu Yang, Zihan Zhou, Shuo Wang, Xin Cong, Xu Han, Yukun Yan, Zhenghao Liu, Zhixing Tan, Pengyuan Liu, Dong Yu, et al. Matplotagent: Method and evaluation for llm-based agentic scientific data visualization. arXiv preprint arXiv:2402.11453, 2024

    2024 arXiv

  79. [87]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  80. [88]

    mplug-owl: Modularization empowers large language models with multimodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023

    2023 arXiv

  81. [89]

    Web2code: A large-scale webpage-to-code dataset and evaluation framework for multimodal llms

    Sukmin Yun, Haokun Lin, Rusiru Thushara, Mohammad Qazim Bhat, Yongxin Wang, Zutao Jiang, Mingkai Deng, Jinhong Wang, Tianhua Tao, Junbo Li, et al. Web2code: A large-scale webpage-to-code dataset and evaluation framework for multimodal llms. arXiv preprint arXiv:2406.20098, 2024

    2024 arXiv

  82. [90]

    Humaneval-v: Evaluating visual understanding and reasoning abilities of large multimodal models through coding tasks

    Fengji Zhang, Linquan Wu, Huiyu Bai, Guancheng Lin, Xiao Li, Xiao Yu, Yue Wang, Bei Chen, and Jacky Keung. Humaneval-v: Evaluating visual understanding and reasoning abilities of large multimodal models through coding tasks. arXiv preprint arXiv:2410.12381, 2024

    arXiv

  83. [91]

    Codev: Issue resolving with visual data

    Linhao Zhang, Daoguang Zan, Quanshun Yang, Zhirong Huang, Dong Chen, Bo Shen, Tianyu Liu, Yongshun Gong, Pengjie Huang, Xudong Lu, et al. Codev: Issue resolving with visual data. arXiv preprint arXiv:2412.17315, 2024

    2024 arXiv

  84. [92]

    Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output. arXiv preprint arXiv:2407.03320, 2024

    2024 arXiv

  85. [93]

    A critical review of large language model on software engineering: An example from chatgpt and automated program repair

    Quanjun Zhang, Tongke Zhang, Juan Zhai, Chunrong Fang, Bowen Yu, Weisong Sun, and Zhenyu Chen. A critical review of large language model on software engineering: An example from chatgpt and automated program repair. arXiv preprint arXiv:2310.08879, 2023

    arXiv

  86. [94]

    Mavis: Mathematical visual instruction tuning

    Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Yichi Zhang, Ziyu Guo, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, Shanghang Zhang, et al. Mavis: Mathematical visual instruction tuning. arXiv preprint arXiv:2407.08739, 2024

    2024 arXiv

  87. [95]

    Multimodal self-instruct: Syn- thetic abstract image and visual reasoning instruction using language model

    Wenqi Zhang, Zhenglin Cheng, Yuanyu He, Mengna Wang, Yongliang Shen, Zeqi Tan, Guiyang Hou, Mingqian He, Yanna Ma, Weiming Lu, et al. Multimodal self-instruct: Syn- thetic abstract image and visual reasoning instruction using language model. arXiv preprint arXiv:2407.07053, 2024

    2024 arXiv

  88. [96]

    Pybench: Evaluating llm agent on various real-world coding tasks

    Yaolun Zhang, Yinxu Pan, Yudong Wang, Jie Cai, Zhi Zheng, Guoyang Zeng, and Zhiyuan Liu. Pybench: Evaluating llm agent on various real-world coding tasks. arXiv preprint arXiv:2407.16732, 2024

    2024 arXiv

  89. [97]

    Chartcoder: Advancing multimodal large language model for chart-to-code generation

    Xuanle Zhao, Xianzhen Luo, Qi Shi, Chi Chen, Shuo Wang, Wanxiang Che, Zhiyuan Liu, and Maosong Sun. Chartcoder: Advancing multimodal large language model for chart-to-code generation. arXiv preprint arXiv:2501.06598, 2025

    2025 arXiv

  90. [98]

    Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568, abs...

    2023 arXiv

  91. [99]

    Opencodeinterpreter: Integrating code generation with execution and refinement

    Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinterpreter: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402.14658, 2024

    2024 arXiv

  92. [100]

    Llamafactory: Unified efficient fine-tuning of 100+ language models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 3: System...

    2024

  93. [101]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learni...

    2024

  94. [102]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In The Twelfth International Conference on Learning Representations, 2023

    2023

  95. [103]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025

    2025 arXiv

  96. [104]

    Bigcodebench: Bench- marking code generation with diverse function calls and complex instructions

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Bench- marking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15...

    2024 arXiv

  97. [105]

    long thinking,

    Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hong- sheng Li, and Yu Liu. Mova: Adapting mixture of vision experts to multimodal context. arXiv preprint arXiv:2404.13046, 2024. 17 Appendix A Limitations 19 B Potential societal impacts 19 B.1 Pos...

    2024 arXiv

  98. [106]

    Identify and highlight the key information from the problem and solution to include in the Mermaid diagram, such as main processes, detail args, steps, decision points, and outcomes

  99. [107]

    For example, you can use **flowcharts, sequence diagrams, or class diagrams .etc** depending on the complexity and nature of the problem and solution

    Use Mermaid syntax to visually represent the flow of the problem and solution. For example, you can use **flowcharts, sequence diagrams, or class diagrams .etc** depending on the complexity and nature of the problem and solution

  100. [108]

    Place the Mermaid code inside ```mermaid```

  101. [109]

    Receive arrays A and B of length n

    Description in the diagram should be natural, here is an example: flowchart TD Start([Start]) --> Input[("Receive arrays A and B of length n")] Input --> Init[("Initialize result array C with size 2n-1")] Init --> Loop["Iterate i from 0 to 2n-2"] Loop --> InnerLoop["For each i...

  102. [110]

    Figure 14: Prompt template for step 1 of constructing diagram-type data in M 2C-I NSTRUCT

    Most Importantly, **ensuring all text is described within quotes** to avoid syntax errors. Figure 14: Prompt template for step 1 of constructing diagram-type data in M 2C-I NSTRUCT . 26 Prompt Template For Step 2 ### Please gain inspiration from the following Problem, Solution...

  103. [111]

    xxx is detailed in the provided diagram

    [Incomplete Problem]: - This problem is incomplete, and can not be solved with only your generated [Incomplete Problem], some key information is only provided by the diagram. - Replace some steps or details with "xxx is detailed in the provided diagram" or "xxx could be found ...

  104. [112]

    - Don’t generate the main or check function

    [Solution]: - Offer a comprehensive, **correct** solution that accurately addresses the [Problem] you provided. - Don’t generate the main or check function. Figure 15: Prompt template for step 2 of constructing diagram-type data in M 2C-I NSTRUCT . 27 E Experimental Details E....

    2025

  105. [113]

    For optimization, we employ AdamW [47] as the optimizer, with a batch size of 1024 and a maximum sequence length of 2048

    Input data is formatted using the qwen2_vl template, with sequences truncated or padded to a maximum length of 2048 tokens. For optimization, we employ AdamW [47] as the optimizer, with a batch size of 1024 and a maximum sequence length of 2048. The model is trained for 1.0 ep...

  106. [114]

    thinking

    Input data is formatted using the qwen2_vl template, with sequences truncated or padded to a maximum length of 6000 tokens. For optimization, we employ AdamW as the optimizer, with a global batch size of 1024 and a maximum sequence length of 6000. The model is trained for 2.0 ...

    2025

  107. [115]

    If the hard drive is full, return an error message

    **Create File**: Add a file to the file system. If the hard drive is full, return an error message

  108. [116]

    **Delete File**: Remove a file from the file system

  109. [117]

    **List Files**: Return a list of all files in the file system along with their sizes

  110. [118]

    File size must be a positive number

    **Check Capacity**: Return the remaining capacity of the hard drive. ### Example Response Here is an improved version of your code with these considerations: class File { constructor(public name: string, public size: number) { if (size <= 0) { throw new Error("File size must b...

  111. [119]

    **Factory Function**: Create a factory function ‘createRecordFactory‘ that takes a ‘type‘ parameter and returns a constructor function for that type of record

  112. [120]

    **Constructor Function**: The constructor function should initialize the object with properties based on the type

  113. [121]

    For example, ‘User‘ records should have a ‘greet‘ method, and ‘Product‘ records should have a ‘displayPrice‘ method

    **Methods**: Each type of record should have specific methods. For example, ‘User‘ records should have a ‘greet‘ method, and ‘Product‘ records should have a ‘displayPrice‘ method

  114. [122]

    ### Example Usage

    **Edge Cases**: Handle cases where the type is not recognized by returning a generic ‘Record‘ type with a default method. ### Example Usage ... Response ### Explanation and Code Review The provided solution effectively meets the requirements by creating a factory function that...

  115. [123]

    - It uses a switch statement to determine which constructor to return

    **Factory Function ( `createRecordFactory`)**: - This function takes a `type` parameter and returns a constructor function for the specified type. - It uses a switch statement to determine which constructor to return

  116. [124]

    - **Product**: Initializes with `name` and `price` properties and has a `displayPrice` method

    **Constructor Functions**: - **User**: Initializes with `name` and `email` properties and has a `greet` method. - **Product**: Initializes with `name` and `price` properties and has a `displayPrice` method. - **Record**: A generic constructor for unrecognized types with an `in...

  117. [125]

    **Prototypes**: - Methods are added to the prototypes of the constructors to ensure they are shared across all instances, which is a good practice for memory efficiency

  118. [126]

    ‘python def fix_application_names(apps: list) -> list: pass “‘ ... Response def fix_application_names(apps: list) -> list: corrected_apps = [] for app in apps: if app ==

    **Edge Cases**: - If the `type` is not recognized, the factory returns a `Record` constructor, providing a default method. Figure 22: Cross-modal example 3 from M 2C-INSTRUCT . 34 M2C-I NSTRUCT Diagram Example 1 Problem You are tasked with creating a TypeScript class method th...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.