Pith. sign in

REVIEW 3 major objections 4 minor 7 cited by

VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VisCodex claims that merging a coding LLM into a vision-language backbone via task vectors lets open-source models generate code from images and screenshots, reaching the strongest results among open MLLMs and approaching GPT-4o on its new

desk verdict Promising resource paper, but the supplied full text is unreadable and the SOTA claim rests entirely on the authors' own benchmark. read the letter →

arxiv 2508.09945 v1 pith:E6NRZPWI submitted 2025-08-13 cs.CL cs.AIcs.CV

Lingjie Jiang , Shaohan Huang , Xun Wu , Yixia Li , Dongdong Zhang , Furu Wei This is my paper
classification cs.CLcs.AIcs.CV
keywords multimodalcodegenerationvision-languagemodelstask-vectormodelmerginglargelanguagecodingdatasetvisualprogrammingbenchmarkHTMLLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a vision-language model can gain strong multimodal code generation by task-vector merging with a coding LLM, instead of expensive joint training from scratch. It introduces VisCodex, trained on the 598k-sample Multimodal Coding Dataset spanning HTML pages, chart-code pairs, image-augmented StackOverflow QA, and algorithmic problems. To measure the result, it introduces InfiBench-V, a benchmark of visually rich programming questions, and reports that VisCodex is the best open-source MLLM on that benchmark and comes close to GPT-4o. If this is right, the merge strategy plus curated data gives open models a practical path to vision-grounded coding ability.

What carries the argument

The central object is the task vector: the difference between the weights of a coding-focused model and a vision-language backbone, $\tau = \theta_{\text{coder}} - \theta_{\text{vlm}}$, added back to the backbone with a scaling factor, $\theta_{\text{merged}} = \theta_{\text{vlm}} + \lambda\tau$. This single operation is what is supposed to inject coding ability while keeping visual understanding, and it is paired with the 598k-sample Multimodal Coding Dataset to adapt the merged model to visually grounded code generation.

What would settle it

Run VisCodex on an independently written set of visual coding problems that were not derived from MCD or InfiBench-V, and compare its score against the same open MLLMs and GPT-4o; as a companion check, measure exact and near-duplicate overlap between MCD training samples and InfiBench-V test questions. If VisCodex's advantage shrinks on the independent set, or if substantial overlap exists, the central state-of-the-art claim fails.

Watch

Extended reading notes

Core claim

The central claim is that task-vector model merging can transfer coding expertise into a vision-language backbone without destroying visual comprehension. The authors build VisCodex by adding a scaled version of the weight difference between a strong coding LLM and a vision-language model back into the vision-language model, then training the merged model on MCD, 598k multimodal code examples. They also construct InfiBench-V, a benchmark of visually rich, real-world programming questions requiring joint understanding of text and images. On InfiBench-V, VisCodex reports state-of-the-art performance among open-source MLLMs and performance approaching proprietary GPT-4o, which the paper present

Load-bearing premise

The load-bearing premise is that InfiBench-V fairly measures real-world visually rich programming ability; because the same group built the model, the training data, and the benchmark, a benchmark that leaks MCD content, shares the authors' data-generation style, or rewards their training distribution could make the rankings look stronger than they truly are.

Editorial extensions

If this is right

  • If the central claim holds, task-vector merging is a cheap recipe for giving vision-language models code-generation ability without retraining a large model from scratch.
  • The MCD dataset, with its HTML, chart-code, StackOverflow QA, and algorithmic components, becomes a reusable training resource for other multimodal code models.
  • Open-source MLLMs could close much of the gap to proprietary models on visually rich programming, at least on benchmarks shaped like InfiBench-V.
  • InfiBench-V would serve as a new evaluation target for future work on visual code reasoning, giving the field a shared measure of text-plus-image programming ability.
  • The merged model is claimed to preserve visual comprehension while gaining coding skill, implying that the two capabilities can coexist in one weight set rather than trading off.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the headline performance claim rests entirely on a benchmark built by the same authors who built the model and the training data, so the ranking could be inflated by stylistic or data overlap between MCD and InfiBench-V; an independently constructed visual coding benchmark would be the real test.
  • My inference: the task-vector mechanism may generalize beyond coding, meaning the same merge recipe could inject other specialized skills such as math reasoning, tool use, or agentic planning into a vision-language backbone, but each transfer would need its own validation.
  • My inference: the 598k sample size is a natural ablation variable; a reader should ask whether the gains come from the merge itself or from the sheer volume of curated multimodal code data, since a smaller MCD would separate those effects.
  • My inference: if the merge works as claimed, it suggests that visual understanding and code generation are somewhat modular capabilities inside large models, with the task vector capturing a consistent coding direction that can be moved between model instances.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes VisCodex, a unified multimodal code generation model obtained by merging a vision-language backbone with a coding LLM via task vectors. The authors introduce MCD, a 598k-sample dataset spanning HTML, chart-image, StackOverflow QA, and algorithmic problems, and InfiBench-V, a new benchmark for visually rich programming. The Abstract claims VisCodex achieves state-of-the-art performance among open-source MLLMs and approaches proprietary models such as GPT-4o. The supplied full text is severely corrupted (mojibake): nearly all method details, equations, tables, and baseline numbers are undecodable. I could not verify the model-merging formulation, the dataset construction, the benchmark design, or the experimental claims.

Significance. If the empirical claims hold, the contribution would be useful: a low-cost way to inject coding skill into an MLLM through task-vector merging, plus a large dataset and a new benchmark. The scale of MCD (598k samples) and the proposed InfiBench-V could be of community value. However, as submitted, the manuscript does not permit independent verification. The evaluation relies entirely on a benchmark introduced by the same authors, and no external benchmark comparison or contamination analysis is decodable. Therefore the significance of the claimed SOTA and near-GPT-4o results is currently unassessable.

major comments (3)
  1. [Full text (Sections 1–6, all tables)] The supplied file is mojibake from the first section onward: equations, table headers, row labels, and prose are undecodable. For example, Section 4 and the table fragments consist of sequences such as '�����' with no readable labels. This makes it impossible to check the task-vector formula, the merging coefficient, the MCD construction, the InfiBench-V design, or the baseline numbers. The authors must resubmit a readable manuscript before substantive review can proceed.
  2. [Abstract; InfiBench-V introduction] The central SOTA claim is stated only against InfiBench-V, a benchmark introduced in the same paper. Since MCD was also built by the same authors and used for training, overlap or contamination between MCD and InfiBench-V is a genuine risk that is not addressed in any decodable part of the manuscript. This is not an accusation; it is a verification requirement. The paper should report results on independently established benchmarks (e.g., Design2Code, Chart2Code, HumanEval-V) and provide a contamination analysis, such as n-gram overlap statistics and manual inspection, between MCD and InfiBench-V. Without this, the headline comparison to GPT-4o is not externally grounded.
  3. [Task-vector merging (Sections 2–3, undecodable)] The decodable abstract and fragments do not report the task-vector merging coefficient λ or the mixture proportions of the MCD components. These are free parameters that directly control the method, and the claim 'highlighting the effectiveness of our model merging strategy' depends on them. The paper should include a sensitivity analysis over λ and the component ratios, and ideally show that the chosen values are not overfit to InfiBench-V. If such ablations exist in the full text, they are unreadable in the supplied version.
minor comments (4)
  1. [Throughout] Fix the PDF/HTML encoding. Currently section headings, equations, tables, and the bibliography are unreadable, which blocks any detailed review.
  2. [Abstract / availability] Please add a footnote or link with code, dataset, and benchmark release information, and specify licenses. None of this is visible in the decodable text.
  3. [Tables] The numeric table entries appear without readable row/column headers. Even after re-encoding, ensure every table has clear model names, benchmark names, and metric definitions.
  4. [Experiments] Report variance or error bars and the number of runs where applicable. The decodable portions give only point values, which is insufficient for comparing models of this class.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found in the decodable derivation chain.

full rationale

The paper's derivation chain—task-vector merging of a coding LLM into a vision-language backbone, training on the newly introduced MCD, and evaluation on the newly proposed InfiBench-V—does not, in the decodable text, contain a step where a predicted quantity is defined in terms of the fitted input, a fitted parameter renamed as a prediction, or a load-bearing self-citation. The abstract's SOTA claim is evaluated on InfiBench-V, a benchmark introduced by the same authors, and no contamination analysis or external benchmark cross-check is reported; that is a legitimate concern about benchmark validity and generalizability, but it is not equation-level circularity. The full text is encoding-corrupted, so no further internal equations or ablations are decodable to check; absence of decodable detail is a verification limitation, not evidence of circularity. Under the stated hard rules, circularity requires exhibiting a specific reduction or renamed fit, and none can be quoted from this manuscript. Therefore the appropriate finding is no significant circularity (score 0).

Assumptions & free parameters 2 free parameters · 3 assumptions · 2 invented entities

Because the body is unreadable, the ledger records only premises recoverable from the abstract. The central claim rests on (i) the linear-composability premise of task-vector merging, (ii) the validity and non-contamination of the new InfiBench-V benchmark, and (iii) tuning knobs (merging scale, data mixture) whose values are not stated. The new dataset and benchmark are the key invented artifacts, with no external validation advertised.

free parameters (2)
  • task-vector merging coefficient lambda = not stated in abstract
    Task-vector merging requires a scalar mixing weight; its value controls the trade-off between preserved vision capability and injected coding skill, and is typically tuned on a validation set. The abstract does not report it.
  • MCD component mixture proportions = not stated in abstract
    The 598k samples span four components (HTML, chart-image code, StackOverflow QA, algorithmic problems); the balance among them is a design choice that affects downstream performance and is not disclosed in the abstract.
assumptions (3)
  • domain assumption Skills encoded in fine-tuned parameter deltas are linearly composable in weight space
    The method rests on the task-vector premise that adding a coding model's parameter delta to a vision backbone transfers coding skill while retaining vision ability ('preserving both visual comprehension and advanced coding skills', Abstract). This is a known but approximation-heavy assumption.
  • domain assumption InfiBench-V is a valid, unbiased measure of real-world multimodal programming ability
    The headline SOTA claim is measured, as far as the abstract shows, only on this new benchmark; the rubric's validity and question difficulty are assumed rather than externally established.
  • domain assumption No test contamination between MCD training data and InfiBench-V evaluation questions
    A 598k-sample dataset is collected and a benchmark is built, presumably from disjoint sources, but no contamination analysis is mentioned in the abstract. Contamination would inflate the reported scores.
invented entities (2)
  • MCD (Multimodal Coding Dataset)
    purpose: 598k training samples spanning HTML code, chart image-code pairs, image-augmented StackOverflow QA, and algorithmic problems
    The dataset is the training substrate; no release URL, hash, or external usage is cited in the abstract, so others cannot yet probe or benchmark against it.
  • InfiBench-V
    purpose: Evaluation benchmark for visually-rich, real-world programming questions
    The yardstick for the SOTA claim; no external validation, rubric release, or cross-check against pre-existing multimodal code benchmarks is described in the abstract.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models." pith.science (2026). https://pith.science/paper/E6NRZPWI

@misc{pith2026250809945,
  author       = {Pith},
  title        = {Pith review of: VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E6NRZPWI}},
  note         = {Machine review of arXiv:2508.09945}
}
read the original abstract

Multimodal large language models (MLLMs) have significantly advanced the integration of visual and textual understanding. However, their ability to generate code from multimodal inputs remains limited. In this work, we introduce VisCodex, a unified framework that seamlessly merges vision and coding language models to empower MLLMs with strong multimodal code generation abilities. Leveraging a task vector-based model merging technique, we integrate a state-of-the-art coding LLM into a strong vision-language backbone, while preserving both visual comprehension and advanced coding skills. To support training and evaluation, we introduce the Multimodal Coding Dataset (MCD), a large-scale and diverse collection of 598k samples, including high-quality HTML code, chart image-code pairs, image-augmented StackOverflow QA, and algorithmic problems. Furthermore, we propose InfiBench-V, a novel and challenging benchmark specifically designed to assess models on visually-rich, real-world programming questions that demand a nuanced understanding of both textual and visual contexts. Extensive experiments show that VisCodex achieves state-of-the-art performance among open-source MLLMs and approaches proprietary models like GPT-4o, highlighting the effectiveness of our model merging strategy and new datasets.

Discussion (0). Sign in to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Visual-ERM: Reward Modeling for Visual Equivalence

    cs.CV 2026-03 unverdicted novelty 7.0 of 10

    Visual-ERM is a new multimodal reward model that supplies fine-grained visual feedback for training vision-language models on chart-to-code, table, and SVG tasks, yielding measurable gains over prior rewards.

  2. Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generation

    cs.SE 2026-08 conditional novelty 6.0 of 10

    On a new 1,440-screenshot fill-in-the-blank benchmark, five frontier multimodal models default to repeated UI patterns instead of visual deviations, with mean bias rates from 69.8% on card widths to 80.2% on font sizes.

  3. Deep Agentic Search for Repository-Level Code Question Answering: An Empirical Study

    cs.SE 2026-08 conditional novelty 6.0 of 10

    On SWE-QA, semantic search beat deep agentic search in accuracy (65.2% vs 46.2% Pass) and in cost per correct answer, with 41.8% of deep agent failures attributed to planner-to-subagent coordination breakdowns.

  4. CharTide: Data-Centric Chart-to-Code Generation via Tri-Perspective Tuning and Inquiry-Driven Evolution

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    CharTide decouples chart-to-code data into three perspectives and uses inquiry-driven RL with atomic QA verification to let smaller VLMs surpass GPT-4o on chart-to-code tasks.

  5. CharTide: Data-Centric Chart-to-Code Generation via Tri-Perspective Tuning and Inquiry-Driven Evolution

    cs.CV 2026-04 conditional novelty 6.0 of 10

    A 7B/8B model trained with decoupled tri-perspective SFT and QA-verified RL matches GPT-4o and approaches GPT-5 on chart-to-code generation benchmarks.

  6. Neural Change Prediction: Relating Software Changes to Their Effects and Vice Versa

    cs.SE 2026-06 unverdicted novelty 5.0 of 10

    Neural Change Prediction generates mutation data to train bidirectional models linking code changes to behavioral effects for any executable program.

  7. Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence

    cs.CL 2026-06 unverdicted novelty 3.0 of 10

    A structured survey of multimodal code intelligence that formulates the field by code roles and organizes work into four domains while proposing verification-centered research directions.

Reference graph

Works this paper leans on

66 extracted references · 25 canonical work pages · cited by 6 Pith papers

  1. [1]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425--2433, 2015

    work page 2015

  2. [2]

    Openflamingo: An open-source framework for training large autoregressive vision-language models

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390 , 2023

    arXiv 2023

  3. [3]

    Opencodereasoning: Advancing data distillation for competitive coding, 2025

    Wasi Uddin Ahmad, Sean Narenthiran, Somshubra Majumdar, Aleksander Ficek, Siddhartha Jain, Jocelyn Huang, Vahid Noroozi, and Boris Ginsburg. Opencodereasoning: Advancing data distillation for competitive coding, 2025

    work page 2025

  4. [4]

    Evolutionary optimization of model merging recipes

    Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. Nature Machine Intelligence , 7(2):195--204, 2025

    work page 2025

  5. [5]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966 , 2023

    arXiv 2023

  6. [6]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. a...

    arXiv 2025

  7. [7]

    pix2code: Generating code from a graphical user interface screenshot

    Tony Beltramelli. pix2code: Generating code from a graphical user interface screenshot. In Proceedings of the ACM SIGCHI symposium on engineering interactive computing systems , pages 1--6, 2018

    work page 2018

  8. [8]

    Layer swapping for zero-shot cross-lingual transfer in large language models

    Lucas Bandarkar, Benjamin Muller, Pritish Yuvraj, Rui Hou, Nayan Singhal, Hongjiang Lv, and Bing Liu. Layer swapping for zero-shot cross-lingual transfer in large language models. arXiv preprint arXiv:2410.01335 , 2024

    arXiv 2024

Show all 66 references
  1. [9]

    Pali-x: On scaling up a multilingual vision and language model

    Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. Pali-x: On scaling up a multilingual vision and language model. arXiv preprint arXiv:2305.18565 , 2023

    2023 arXiv

  2. [10]

    Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023) , 2(3):6, 2023

    2023

  3. [11]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021

    2021 arXiv

  4. [12]

    Dualfocus: Integrating macro and micro perspectives in multi-modal large language models

    Yuhang Cao, Pan Zhang, Xiaoyi Dong, Dahua Lin, and Jiaqi Wang. Dualfocus: Integrating macro and micro perspectives in multi-modal large language models. arXiv preprint arXiv:2402.14767 , 2024

    2024 arXiv

  5. [13]

    Bring reason to vision: Understanding perception and reasoning through model merging

    Shiqi Chen, Jinghan Zhang, Tongyao Zhu, Wei Liu, Siyang Gao, Miao Xiong, Manling Li, and Junxian He. Bring reason to vision: Understanding perception and reasoning through model merging. arXiv preprint arXiv:2505.05464 , 2025

    2025 arXiv

  6. [14]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    2025

  7. [15]

    An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  8. [16]

    Enhancing chat language models by scaling high-quality instructional conversations

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233 , 2023

    2023 arXiv

  9. [17]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints , pages arXiv--2407, 2024

    2024

  10. [18]

    Instructblip: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500 , 2023

    2023 arXiv

  11. [19]

    Palm-e: An embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 , 2023

    2023 arXiv

  12. [20]

    Webcode2m: A real-world dataset for code generation from webpage designs

    Yi Gui, Zhen Li, Yao Wan, Yemin Shi, Hongyu Zhang, Bohua Chen, Yi Su, Dongping Chen, Siyuan Wu, Xing Zhou, et al. Webcode2m: A real-world dataset for code generation from webpage designs. In Proceedings of the ACM on Web Conference 2025 , pages 1834--1845, 2025

    2025

  13. [21]

    Merrill, Tatsunori Hashimoto, Yejin Choi, Jenia Jitsev, Reinhard Heckel, Maheswaran Sathiamoorthy, Alexandros G

    Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, Wanj...

    2025

  14. [22]

    LeetCode Solutions , 2023

    Eric Hartford. LeetCode Solutions , 2023. Accessed: 2025-02-11

    2023

  15. [23]

    Language is not all you need: Aligning perception with language models

    Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning perception with language models. Advances in Neural Information Processing Systems , 36, 2024

    2024

  16. [24]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186 , 2024

    2024 arXiv

  17. [25]

    Chartllama: A multimodal llm for chart understanding and generation

    Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, and Hanwang Zhang. Chartllama: A multimodal llm for chart understanding and generation. arXiv preprint arXiv:2311.16483 , 2023

    2023 arXiv

  18. [26]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089 , 2022

    2022 arXiv

  19. [27]

    Dataless knowledge fusion by merging weights of language models

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. arXiv preprint arXiv:2212.09849 , 2022

    2022 arXiv

  20. [28]

    Codeforces Dataset , 2022

    Jur1cek. Codeforces Dataset , 2022. Accessed: 2025-02-11

    2022

  21. [29]

    Pythonic Data Structures and Algorithms , 2018

    Keon . Pythonic Data Structures and Algorithms , 2018. Accessed: 2025-02-11

    2018

  22. [30]

    Competition-level code generation with alphacode

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R \'e mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal,...

    2022 arXiv

  23. [31]

    Taco: Topics in algorithmic code generation dataset

    Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset. arXiv preprint arXiv:2312.14852 , 2023

    2023 arXiv

  24. [32]

    Infibench: Evaluating the question-answering capabilities of code large language models

    Linyi Li, Shijie Geng, Zhenwen Li, Yibo He, Hao Yu, Ziyue Hua, Guanghan Ning, Siwei Wang, Tao Xie, and Hongxia Yang. Infibench: Evaluating the question-answering capabilities of code large language models. Advances in Neural Information Processing Systems , 37:128668--128698, 2024

    2024

  25. [33]

    Transferring textual preferences to vision-language understanding through model merging, 2025

    Chen-An Li, Tzu-Han Lin, Yun-Nung Chen, and Hung yi Lee. Transferring textual preferences to vision-language understanding through model merging, 2025

    2025

  26. [34]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    2024

  27. [35]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning , pages 19730--19742. PMLR, 2023

    2023

  28. [36]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023

    2023

  29. [37]

    Mmcode: Evaluating multi-modal code large language models with visually rich programming problems

    Kaixin Li, Yuchen Tian, Qisheng Hu, Ziyang Luo, and Jing Ma. Mmcode: Evaluating multi-modal code large language models with visually rich programming problems. arXiv preprint arXiv:2404.09486 , 2024

    2024 arXiv

  30. [38]

    Unlocking the conversion of web screenshots into html code with the websight dataset

    Hugo Lauren c on, L \'e o Tronchon, and Victor Sanh. Unlocking the conversion of web screenshots into html code with the websight dataset. arXiv preprint arXiv:2403.09029 , 2024

    2024 arXiv

  31. [39]

    Mimic-it: Multi-modal in-context instruction tuning

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425 , 2023

    2023 arXiv

  32. [40]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 , 2024

    2024 arXiv

  33. [41]

    Detecting near-duplicates for web crawling

    Gurmeet Singh Manku, Arvind Jain, and Anish Das Sarma. Detecting near-duplicates for web crawling. In Proceedings of the 16th international conference on World Wide Web , pages 141--150, 2007

    2007

  34. [42]

    Merging models with fisher-weighted averaging

    Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems , 35:17703--17716, 2022

    2022

  35. [43]

    Kosmos-g: Generating images in context with multimodal large language models

    Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei. Kosmos-g: Generating images in context with multimodal large language models. arXiv preprint arXiv:2310.02992 , 2023

    2023 arXiv

  36. [44]

    Kosmos-2: Grounding multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824 , 2023

    2023 arXiv

  37. [45]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing , 568:127063, 2024

    2024

  38. [46]

    Chartmimic: Evaluating lmm's cross-modal reasoning capability via chart-to-code generation

    Chufan Shi, Cheng Yang, Yaxin Liu, Bo Shui, Junjie Wang, Mohan Jing, Linran Xu, Xinyu Zhu, Siheng Li, Yuxiang Zhang, et al. Chartmimic: Evaluating lmm's cross-modal reasoning capability via chart-to-code generation. arXiv preprint arXiv:2406.09961 , 2024

    2024 arXiv

  39. [47]

    Design2code: How far are we from automating front-end engineering? arXiv e-prints , pages arXiv--2403, 2024

    Chenglei Si, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. Design2code: How far are we from automating front-end engineering? arXiv e-prints , pages arXiv--2403, 2024

    2024

  40. [48]

    Python Algorithms , 2023

    The Algorithms . Python Algorithms , 2023. Accessed: 2025-02-11

    2023

  41. [49]

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's pe...

    2024 arXiv

  42. [50]

    Plot2code: A comprehensive benchmark for evaluating multi-modal large language models in code generation from scientific plots

    Chengyue Wu, Yixiao Ge, Qiushan Guo, Jiahao Wang, Zhixuan Liang, Zeyu Lu, Ying Shan, and Ping Luo. Plot2code: A comprehensive benchmark for evaluating multi-modal large language models in code generation from scientific plots. arXiv preprint arXiv:2405.07990 , 2024

    2024 arXiv

  43. [51]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing infe...

    2022

  44. [52]

    Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding, 2025

    Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding, 2025

    2025

  45. [53]

    Wizardlm: Empowering large language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244 , 2023

    2023 arXiv

  46. [54]

    Xing, Xiaodan Liang, and Zhiqiang Shen

    Sukmin Yun, Haokun Lin, Rusiru Thushara, Mohammad Qazim Bhat, Yongxin Wang, Zutao Jiang, Mingkai Deng, Jinhong Wang, Tianhua Tao, Junbo Li, Haonan Li, Preslav Nakov, Timothy Baldwin, Zhengzhong Liu, Eric P. Xing, Xiaodan Liang, and Zhiqiang Shen. Web2code: A large-scale webpag...

    2024 arXiv

  47. [55]

    Ties-merging: Resolving interference when merging models

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems , 36:7093--7115, 2023

    2023

  48. [56]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023

    2023

  49. [57]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 , 2023

    2023 arXiv

  50. [58]

    Chartcoder: Advancing multimodal large language model for chart-to-code generation, 2025

    Xuanle Zhao, Xianzhen Luo, Qi Shi, Chi Chen, Shuo Wang, Wanxiang Che, Zhiyuan Liu, and Maosong Sun. Chartcoder: Advancing multimodal large language model for chart-to-code generation, 2025

    2025

  51. [59]

    Chartcoder: Advancing multimodal large language model for chart-to-code generation

    Xuanle Zhao, Xianzhen Luo, Qi Shi, Chi Chen, Shuo Wang, Wanxiang Che, Zhiyuan Liu, and Maosong Sun. Chartcoder: Advancing multimodal large language model for chart-to-code generation. arXiv preprint arXiv:2501.06598 , 2025

    2025 arXiv

  52. [60]

    Advancing multimodal large language models in chart question answering with visualization-referenced instruction tuning

    Xingchen Zeng, Haichuan Lin, Yilin Ye, and Wei Zeng. Advancing multimodal large language models in chart question answering with visualization-referenced instruction tuning. IEEE Transactions on Visualization and Computer Graphics , 2024

    2024

  53. [61]

    Remedy: Recipe merging dynamics in large vision-language models

    Didi Zhu, Yibing Song, Tao Shen, Ziyu Zhao, Jinluan Yang, Min Zhang, and Chao Wu. Remedy: Recipe merging dynamics in large vision-language models. In The Thirteenth International Conference on Learning Representations , 2025

    2025

  54. [62]

    Humaneval-v: Evaluating visual understanding and reasoning abilities of large multimodal models through coding tasks

    Fengji Zhang, Linquan Wu, Huiyu Bai, Guancheng Lin, Xiao Li, Xiao Yu, Yue Wang, Bei Chen, and Jacky Keung. Humaneval-v: Evaluating visual understanding and reasoning abilities of large multimodal models through coding tasks. arXiv preprint arXiv:2410.12381 , 2024

    2024 arXiv

  55. [63]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479 , 2025

    2025 arXiv

  56. [64]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li,...

    2025

  57. [65]

    Mm-llms: Recent advances in multimodal large language models, 2024

    Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. Mm-llms: Recent advances in multimodal large language models, 2024

    2024

  58. [66]

    Codev: Issue resolving with visual data

    Linhao Zhang, Daoguang Zan, Quanshun Yang, Zhirong Huang, Dong Chen, Bo Shen, Tianyu Liu, Yongshun Gong, Pengjie Huang, Xudong Lu, et al. Codev: Issue resolving with visual data. arXiv preprint arXiv:2412.17315 , 2024

    2024 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.