Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

MMCircuitEval: A Comprehensive Multimodal Circuit-Focused Benchmark for Evaluating LLMs

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MMCircuitEval is a 3,614-question multimodal benchmark spanning digital and analog circuit design, and its results show current models are weakest at back-end design and computation-heavy questions.

desk verdict A genuinely useful multimodal circuit QA benchmark with a plausible but under-validated scoring function; the dataset deserves adoption, the rankings should be read with caution. read the letter →

arxiv 2507.19525 v1 pith:YOCWNIXQ submitted 2025-07-20 cs.LG cs.AI

Chenchen Zhao , Zhengyuan Shi , Xiangyu Wen , Chengjie Liu , Yi Liu , Yunhao Zhou , Yuxiang Zhao , Hefei Feng
show 14 more authors
Yinan Zhu Gwok-Waa Wan Xin Cheng Weiyu Chen Yongqi Fu Chujie Chen Chenhao Xue Guangyu Sun Ying Wang Yibo Lin Jun Yang Ning Xu Xi Wang Qiang Xu
This is my paper
classification cs.LGcs.AI
keywords multimodallargelanguagemodelsbenchmarkelectronicdesignautomationcircuitback-endquestionansweringchain-of-thoughtmodelevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents MMCircuitEval, a benchmark of 3,614 expert-reviewed question-answer pairs for testing multimodal large language models on circuit design. The authors claim it is the first such benchmark to span the major stages of the electronic design automation workflow, covering general knowledge, design specification, front-end design, and back-end design across both digital and analog circuits. Every question is labeled by design stage, circuit type, tested ability (knowledge, comprehension, reasoning, computation), and difficulty, so model performance can be diagnosed along each axis. Evaluations of a broad set of models show that current systems are weakest on back-end design questions and on computation-heavy problems, which the paper takes as evidence that circuit-specific training data and image-processing strategies are the main bottlenecks. If the benchmark measures what it claims, it gives the field a reusable instrument for tracking progress in AI-assisted chip design.

What carries the argument

The load-bearing object is the benchmark itself: a curated set of 3,614 question-answer pairs organized along four axes—design stage (general knowledge, specification, front-end, back-end), circuit type (digital versus analog), tested ability (knowledge, comprehension, reasoning, computation), and difficulty (easy, medium, hard). Answers are scored by a weighted composite that doubles the weight of a GPT-4-turbo preference rating and singly weights BLEU-4, average ROUGE, and embedding cosine similarity, with models required to provide explanations. That composite is the machinery that turns free-form model outputs into the accuracy numbers underlying every comparison in the paper.

What would settle it

Have several circuit engineers independently score a random sample of model answers as correct or incorrect, then compare their human rankings against the benchmark's composite score. If the human ordering of models differs substantially from the composite ordering, or if a masked re-scoring by GPT-4-turbo changes when model identity is revealed, the paper's reported model rankings would not be reproducible.

Watch

Extended reading notes

Core claim

The paper's central claim is that MMCircuitEval is a valid, reusable instrument for measuring how well multimodal large language models handle circuit-design questions across the full EDA workflow, and that the measurements it reports reveal a consistent weakness profile in current models. Across 3,614 questions, the strongest tested model answers about 69 percent correctly, most open-weights models stay below 50 percent, back-end design questions trail other stages by roughly 12 to 22 percentage points, and computation questions are the hardest category for nearly every model family. The paper also reports that models with image encoders often score lower on multimodal questions than text-only models that receive automatically generated image captions, and that a small circuit-specialized model can outperform much larger general-purpose ones. Taken together, these results support the paper's conclusion that circuit-specific training data and image-processing strategies, rather than raw model scale, are the main levers for progress.

Load-bearing premise

The evaluation assumes the weighted mix of text-similarity scores and GPT-4-turbo preference faithfully reflects whether an answer is correct for circuit problems, with validation limited to a manual check of 100 questions per stage and no inter-annotator agreement or exact-match comparison.

Editorial extensions

If this is right

  • Adding high-quality back-end design and layout data to training corpora becomes the direct route to raising model scores, since back-end accuracy lags every other stage by 12 to 22 points.
  • A circuit-specialized small model beating much larger general-purpose models implies that domain-specific fine-tuning can offset raw scale in EDA tasks.
  • Because captioned text-only models can outperform models with native visual encoders on multimodal questions, circuit images currently hurt most multimodal models more than they help.
  • Chain-of-thought prompting helps most on front-end reasoning and little on knowledge retrieval or computation, so test-time compute is best spent on reasoning-heavy questions.
  • The per-question labels for ability and difficulty convert the benchmark from a single leaderboard into a diagnostic tool for per-skill model comparison.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct validity check would mask model identity in the GPT preference scorer; if scores change depending on whether the evaluated model comes from the same family, the evaluator has a self-preference that needs correction.
  • The ability and difficulty labels suggest a natural curriculum: circuit-specialized training could order data from knowledge to computation and easy to hard.
  • The benchmark's format could be extended to interactive or multi-turn design scenarios where the model must request missing datasheet or netlist information before answering.
  • Because the dataset contains real-world datasheets and netlists, it could be used to synthesize additional training pairs, attacking the data scarcity that the paper identifies as the main bottleneck.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces MMCircuitEval, a benchmark of 3,614 multimodal question-answer pairs for evaluating LLMs in circuit design, organized by EDA stage (general knowledge, specification, front-end, back-end), circuit type, tested ability, question type, and difficulty. The authors describe an expert-reviewed curation pipeline, evaluate a wide range of proprietary and open-source LLMs using a weighted combination of BLEU, ROUGE, embedding cosine similarity, and GPT-4-turbo preference, and report that models perform worst on back-end design and computation tasks, with GPT-family models and ChipExpert leading. The paper also reports CoT prompting experiments and discusses data collection and limitations.

Significance. If the benchmark and its evaluation protocol are sound, MMCircuitEval would be a useful and much-needed resource: it is the first multimodal circuit-focused benchmark spanning multiple EDA stages, it has a substantial and categorically rich question set, and it is publicly released. The strengths of the paper are the scale and diversity of the dataset, the expert-review curation procedure, the fine-grained metadata, and the broad coverage of tested models. However, the central empirical claims — that back-end design is hardest, that computation is the weakest ability, and that certain model families lead — are all computed through a scorer whose validation is informal. The evaluator validation, model comparisons without uncertainty estimates, and the admitted family bias of the GPT-based scorer are load-bearing gaps that must be addressed before the reported performance gaps and rankings can be accepted.

major comments (4)
  1. [§III-C, §IV-B] The proposed evaluator is not adequately validated for the claims built on it. The manual check in §IV-B samples 100 questions per stage and reports only that testers were "generally positive" about aggregate scores; there is no inter-annotator agreement, no error analysis, and no comparison against exact-match correctness. This matters because 1,220 of the 3,614 items (738 single-answer choice, 86 multi-answer choice, and 396 fill-in-the-blank) have well-defined ground-truth answers for which exact-match scoring is objective. Since all headline results — the stage ordering, the ability gaps, and the model rankings — are computed through the weighted similarity/GPT-preference score, a systematic bias in the scorer could reorder models or exaggerate gaps. Please validate the evaluator on the closed-form subset against exact-match accuracy, report agreement statistics and error patterns, and show that rankings are stable when the GPT-preference component is ablated.
  2. [§III-C, §V] The double weight assigned to GPT-4-turbo preference creates a circularity risk that is acknowledged but never quantified. The paper states in Limitations that the evaluator "may favor models in the same family (e.g., models in the GPT series)", yet GPT-family models are among the top performers in Table III. Because the weight is a free parameter, the reported superiority of GPT-4v/GPT-4o over other models could be partly an artifact of the scorer rather than of circuit competence. Please report rank correlations or rank changes when GPT preference is down-weighted or removed, and ideally calibrate the metric weights against human correctness labels rather than setting GPT preference to 2 by construction.
  3. [Table III, Table IV, Table V] All reported accuracies are single point estimates without confidence intervals, error bars, or significance tests. Differences such as GPT-4v at 69.4% versus GPT-4o at 68.0%, or the stage-level declines of 12.0–21.8% reported in §IV-C, may be within sampling noise. This is especially important for the fine-grained conclusions about stage ordering and ability ordering, which are based on a single run over 3,614 items. Please report bootstrap confidence intervals or per-item variance, and for the CoT experiment in Table V (100 random questions per stage) provide uncertainty estimates or a paired test before claiming improvement.
  4. [§III-B, §III-D] The claim that GPT-generated questions "exhibit minimal overlap with the training corpora of existing foundation models" is asserted without any supporting check. This is load-bearing for fair horizontal comparison because GPT-generated questions are used to evaluate GPT-family models; if overlap is substantial, memorization could inflate their scores. Please describe the method used to test overlap (e.g., n-gram or embedding-based inspection, training-data cutoff analysis) or soften the claim to a statement of intent rather than a demonstrated property.
minor comments (4)
  1. [Author affiliations] The affiliation "School of Intergrated Circuits, Southeast University" should read "Integrated"; also, "LlaMa" is spelled inconsistently (e.g., LlaMa3.2 vs Llama 3) across the text and tables.
  2. [§IV-B] The description of the evaluator validation should specify how the 100 questions per stage were selected, how many testers were involved, their domain expertise, and what instruction they received; "generally positive" is too vague to support the conclusion that the metric is effective.
  3. [Table III] The reference for LlaMa3.2-Vision-Instruct-90B is cited as [33], which is the Llama 3 herd paper; please provide the specific vision-model reference or clarify that the model is a derivative of that release.
  4. [Table II] The source column lists "Synthesis" for the MMCircuitEval row, which is not defined; if it refers to paraphrase-based augmentation of existing questions, the terminology should be explained in the text.

Circularity Check

1 steps flagged · score 4.0 of 10

Mild partial circularity: the GPT-lead ranking is computed with a double-weighted GPT-4-turbo judge, a same-family bias the paper itself concedes; the benchmark dataset itself is independently grounded.

  1. self definitional [Section III-C (MMCircuitEval evaluation) and Section V (Limitations)]
    "GPT preference. In MMCircuitEval, we leverage GPT-4-turbo [26], proficient in text processing, for overall correctness rating from a well-trained expert's perspective. We assign a weight of 2 for GPT preference and a weight of 1 for others in the MMCircuitEval evaluation. ... This may result in two issues: the evaluation may not be 100% accurate; it may favor models in the same family (e.g., models in the GPT series in this paper)."

    GPT-family models top the leaderboard (GPT-4v 69.4%) via a score in which GPT-4-turbo preference carries double weight (2) versus BLEU/ROUGE/cosine (1 each). For those models, 40% of the measured score is GPT-4-turbo judging other GPT-family outputs, and Section V concedes the judge 'may favor models in the same family.' Hence the headline GPT advantage is partly an input to the measurement (a same-family judge) rather than an independent circuit-competence finding. The circularity is partial, not total: BLEU/ROUGE/cosine supply 60% of the weight, and the abstract's secondary findings (back-end design and complex computations are hardest) are cross-model patterns that are less exposed to this family bias.

full rationale

The benchmark construction is self-contained and independent: 3,614 QA pairs from textbooks, datasheets, RTL codes, netlists, and online sources, each curated and manually reviewed, so the dataset itself is not circular. The circularity concern is confined to the evaluation layer. Section III-C defines correctness as a weighted mixture in which GPT-4-turbo preference receives weight 2, and Section V explicitly admits the judge 'may favor models in the same family (e.g., models in the GPT series in this paper).' Because the models at the top of Table III are from the GPT family, a substantial share of their reported scores is GPT-4-turbo's own judgment of GPT-family outputs; that is a partial self-referentiality, though not a by-construction identity, since the objective metrics still carry 60% of the weight and the stage/ability gaps (back-end and computation hardest) are reported across all model categories. The ChipExpert highlight involves a self-citation ([25], with overlapping authors), but ChipExpert's score is directly measured in this paper, so the citation is not the load-bearing evidence for the 'targeted training helps' argument. The evaluator validation in Section IV-B is informal ('testers are generally positive,' with no inter-annotator agreement, error analysis, or exact-match anchor on the 33.7% closed-form items); that is a correctness-risk gap rather than a circular step, because the evaluator is not fitted to the results. Overall, no central derivation is forced by its own inputs; an admitted evaluator bias and minor same-group citations justify a moderate score.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

No physical entities or fitted scientific parameters are introduced. The central claims rest on assumptions about data correctness, scoring validity, judge neutrality, and data contamination. These assumptions are stated where relevant, and the scoring and bias ones are partly acknowledged as limitations.

free parameters (1)
  • GPT-preference metric weight = 2 vs 1 for other metrics
    The paper assigns a weight of 2 to GPT-4-turbo preference and 1 to BLEU, ROUGE, and embedding similarity (Section III-C). This hand-chosen weighting directly affects every reported accuracy score, and no sensitivity analysis is provided.
assumptions (5)
  • domain assumption The expert-reviewed answers in the dataset are correct ground truth.
    Correctness of 3,614 QA pairs is asserted after manual review; no inter-annotator agreement or external audit is reported (Section III-B2).
  • domain assumption The weighted metric (BLEU, ROUGE, embedding similarity, GPT-4-turbo preference) approximates answer correctness for circuit questions.
    Scores are similarity-based rather than exact-match; validation in Section IV-B is informal and covers only a small sample.
  • domain assumption GPT-4-turbo can serve as an unbiased judge of circuit answer correctness.
    The judge is from the same family as several top-scoring models, and the paper acknowledges possible same-family bias in Section V.
  • domain assumption GPT-generated questions have minimal overlap with the training corpora of evaluated models.
    This fairness assumption is stated in Section III-D without a contamination measurement or cited method.
  • domain assumption The GPT-4o-assigned ability labels are accurate after manual correction.
    Labels for knowledge, comprehension, reasoning, and computation are used for all analysis but residual error is not quantified (Section III-B2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of MMCircuitEval: A Comprehensive Multimodal Circuit-Focused Benchmark for Evaluating LLMs." pith.science (2026). https://pith.science/paper/YOCWNIXQ

@misc{pith2026250719525,
  author       = {Pith},
  title        = {Pith review of: MMCircuitEval: A Comprehensive Multimodal Circuit-Focused Benchmark for Evaluating LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YOCWNIXQ}},
  note         = {Machine review of arXiv:2507.19525}
}
read the original abstract

The emergence of multimodal large language models (MLLMs) presents promising opportunities for automation and enhancement in Electronic Design Automation (EDA). However, comprehensively evaluating these models in circuit design remains challenging due to the narrow scope of existing benchmarks. To bridge this gap, we introduce MMCircuitEval, the first multimodal benchmark specifically designed to assess MLLM performance comprehensively across diverse EDA tasks. MMCircuitEval comprises 3614 meticulously curated question-answer (QA) pairs spanning digital and analog circuits across critical EDA stages - ranging from general knowledge and specifications to front-end and back-end design. Derived from textbooks, technical question banks, datasheets, and real-world documentation, each QA pair undergoes rigorous expert review for accuracy and relevance. Our benchmark uniquely categorizes questions by design stage, circuit type, tested abilities (knowledge, comprehension, reasoning, computation), and difficulty level, enabling detailed analysis of model capabilities and limitations. Extensive evaluations reveal significant performance gaps among existing LLMs, particularly in back-end design and complex computations, highlighting the critical need for targeted training datasets and modeling approaches. MMCircuitEval provides a foundational resource for advancing MLLMs in EDA, facilitating their integration into real-world circuit design workflows. Our benchmark is available at https://github.com/cure-lab/MMCircuitEval.

Figures

Figures reproduced from arXiv: 2507.19525 by the authors.

Figure 1
Figure 1. The overview of MMCircuitEval and sampled questions. Better viewed with zoom-in. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. CircuChain: Disentangling Competence and Compliance in LLM Circuit Analysis

    cs.SE 2026-01 unverdicted novelty 7.0 of 10

    Stronger LLMs show near-perfect physical reasoning in circuits but violate explicit sign and polarity instructions in trap setups, while weaker models follow instructions better but reason less accurately.

Reference graph

Works this paper leans on

54 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    Chipnemo: Domain-adapted llms for chip design,

    M. Liu, T.-D. Ene, R. Kirby et al. , “Chipnemo: Domain-adapted llms for chip design,” arXiv preprint arXiv:2311.00176 , 2023

    arXiv 2023

  2. [2]

    SemiKong: Curating, Training, and Evaluating A Semiconductor Industry-Specific Large Language Model

    C. Nguyen, W. Nguyen, A. Suzuki et al., “Semikong: Curating, training, and evaluating a semiconductor industry-specific large language model,” arXiv preprint arXiv:2411.13802 , 2024

    work page Pith review arXiv 2024

  3. [3]

    Openllm-rtl: Open dataset and benchmark for llm-aided design rtl generation,

    S. Liu, Y . Lu, W. Fang, M. Li, and Z. Xie, “Openllm-rtl: Open dataset and benchmark for llm-aided design rtl generation,” in Proceedings of the 43rd IEEE/ACM International Conference on Computer-Aided Design, 2024, pp. 1–9

    work page 2024

  4. [4]

    Rtllm: An open-source benchmark for design rtl generation with large language model,

    Y . Lu, S. Liu, Q. Zhang, and Z. Xie, “Rtllm: An open-source benchmark for design rtl generation with large language model,” in Asia and South Pacific Design Automation Conference (ASP-DAC) , 2024

    work page 2024

  5. [5]

    Autobench: Automatic testbench generation and evaluation using llms for hdl design,

    R. Qiu, G. L. Zhang, R. Drechsler et al. , “Autobench: Automatic testbench generation and evaluation using llms for hdl design,” in International Symposium on Machine Learning for CAD , 2024

    work page 2024

  6. [6]

    Verilogeval: Evaluating large language models for verilog code generation,

    M. Liu, N. Pinckney, B. Khailany et al., “Verilogeval: Evaluating large language models for verilog code generation,” in 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD) , 2023

    work page 2023

  7. [7]

    Deepcircuitx: A comprehensive repository-level dataset for rtl code understanding, generation, and ppa analysis,

    Z. Li, C. Xu, Z. Shi, Z. Peng, Y . Liu, Y . Zhou, L. Zhou, C. Ma, J. Zhong, X. Wang et al., “Deepcircuitx: A comprehensive repository-level dataset for rtl code understanding, generation, and ppa analysis,” arXiv preprint arXiv:2502.18297, 2025

    arXiv 2025

  8. [8]

    EDA Corpus: A Large Language Model Dataset for Enhanced Interaction with OpenROAD

    B.-Y . Wu, U. Sharma, S. R. D. Kankipati, A. Yadav, B. K. George, S. R. Guntupalli, A. Rovinski, and V . A. Chhabria, “Eda corpus: A large language model dataset for enhanced interaction with openroad,” arXiv preprint arXiv:2405.06676 , 2024

    work page Pith review arXiv 2024

Show all 54 references
  1. [9]

    Customized retrieval augmented generation and benchmarking for eda tool documentation qa,

    Y . Pu, Z. He, T. Qiu, H. Wu, and B. Yu, “Customized retrieval augmented generation and benchmarking for eda tool documentation qa,” arXiv preprint arXiv:2407.15353, 2024

    2024 arXiv

  2. [10]

    The dawn of ai-native eda: Promises and challenges of large circuit models,

    L. Chen, Y . Chen, Z. Chu et al. , “The dawn of ai-native eda: Promises and challenges of large circuit models,” arXiv preprint arXiv:2403.07257, 2024

    2024 arXiv

  3. [11]

    Deepgate: Learning neural representations of logic gates,

    M. Li, S. Khan, Z. Shi, N. Wang, H. Yu, and Q. Xu, “Deepgate: Learning neural representations of logic gates,” in Proceedings of the 59th ACM/IEEE Design Automation Conference , 2022, pp. 667–672

    2022

  4. [12]

    Deepgate2: Functionality-aware circuit representation learning,

    Z. Shi, H. Pan, S. Khan, M. Li, Y . Liu, J. Huang, H.-L. Zhen, M. Yuan, Z. Chu, and Q. Xu, “Deepgate2: Functionality-aware circuit representation learning,” in 2023 IEEE/ACM International Conference on Computer Aided Design . IEEE, 2023

    2023

  5. [13]

    Deep- gate3: Towards scalable circuit representation learning,

    Z. Shi, Z. Zheng, S. Khan, J. Zhong, M. Li, and Q. Xu, “Deep- gate3: Towards scalable circuit representation learning,” arXiv preprint arXiv:2407.11095, 2024

    2024 arXiv

  6. [14]

    Deepgate4: Efficient and effective representation learning for circuit design at scale,

    Z. Zheng, S. Huang, J. Zhong, Z. Shi, G. Dai, N. Xu, and Q. Xu, “Deepgate4: Efficient and effective representation learning for circuit design at scale,” arXiv preprint arXiv:2502.01681 , 2025

    2025 arXiv

  7. [15]

    Au- tochip: Automating hdl generation using llm feedback,

    S. Thakur, J. Blocklove, H. Pearce, B. Tan, S. Garg, and R. Karri, “Au- tochip: Automating hdl generation using llm feedback,” arXiv preprint arXiv:2311.04887, 2023

    2023 arXiv

  8. [16]

    Rtl- coder: Fully open-source and efficient llm-assisted rtl code generation technique,

    S. Liu, W. Fang, Y . Lu, J. Wang, Q. Zhang, H. Zhang, and Z. Xie, “Rtl- coder: Fully open-source and efficient llm-assisted rtl code generation technique,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , 2024

    2024

  9. [17]

    Verigen: A large language model for verilog code generation,

    S. Thakur, B. Ahmad, H. Pearce et al., “Verigen: A large language model for verilog code generation,” ACM Transactions on Design Automation of Electronic Systems , vol. 29, no. 3, pp. 1–31, 2024

    2024

  10. [18]

    Ampagent: An llm-based multi-agent system for multi-stage amplifier schematic design from literature for process and performance porting,

    C. Liu, W. Chen, A. Peng, Y . Du, L. Du, and J. Yang, “Ampagent: An llm-based multi-agent system for multi-stage amplifier schematic design from literature for process and performance porting,” arXiv preprint arXiv:2409.14739, 2024

    2024 arXiv

  11. [19]

    Chateda: A large language model powered autonomous agent for eda,

    H. Wu, Z. He, X. Zhang, X. Yao, S. Zheng, H. Zheng, and B. Yu, “Chateda: A large language model powered autonomous agent for eda,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2024

    2024

  12. [20]

    Openroad: Toward a self-driving, open-source digital layout implementation tool chain,

    T. Ajayi and D. Blaauw, “Openroad: Toward a self-driving, open-source digital layout implementation tool chain,” in Proceedings of Government Microcircuit Applications and Critical Technology Conference , 2019

    2019

  13. [21]

    Chipgpt: How far are we from natural language hardware design,

    K. Chang, Y . Wang, H. Ren, M. Wang, S. Liang, Y . Han, H. Li, and X. Li, “Chipgpt: How far are we from natural language hardware design,” arXiv preprint arXiv:2305.14019 , 2023

    2023

  14. [22]

    Seed-bench: Benchmarking multimodal llms with generative comprehension,

    B. Li, R. Wang, G. Wang et al., “Seed-bench: Benchmarking multimodal llms with generative comprehension,” arXiv preprint arXiv:2307.16125, 2023

    2023 arXiv

  15. [23]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi,

    X. Yue, Y . Ni, K. Zhang et al. , “Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 9556–9567

    2024

  16. [24]

    Mm- bench: Is your multi-modal model an all-around player?

    Y . Liu, H. Duan, Y . Zhang, B. Li, S. Zhang, W. Zhao et al. , “Mm- bench: Is your multi-modal model an all-around player?” in European Conference on Computer Vision . Springer, 2025, pp. 216–233

    2025

  17. [25]

    Chipexpert: The open-source integrated-circuit-design-specific large language model,

    N. Xu, Z. Zhang, L. Qi et al. , “Chipexpert: The open-source integrated-circuit-design-specific large language model,” arXiv preprint arXiv:2408.00804, 2024

    2024 arXiv

  18. [26]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  19. [27]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in International conference on machine learning , 2022

    2022

  20. [28]

    Qwen-vl: A frontier large vision-language model with versatile abilities,

    J. Bai, S. Bai, S. Yang et al., “Qwen-vl: A frontier large vision-language model with versatile abilities,” arXiv preprint arXiv:2308.12966 , 2023

    2023 arXiv

  21. [29]

    Internlm-xcomposer: A vision- language large model for advanced text-image comprehension and composition,

    P. Zhang, X. Dong, B. Wang et al. , “Internlm-xcomposer: A vision- language large model for advanced text-image comprehension and composition,” arXiv preprint arXiv:2309.15112 , 2023

    2023 arXiv

  22. [30]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,

    Z. Chen, J. Wu, W. Wang et al., “Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 24 185–24 198

    2024

  23. [31]

    Instructblip: Towards general-purpose vision- language models with instruction tuning,

    W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi, “Instructblip: Towards general-purpose vision- language models with instruction tuning,” 2023

    2023

  24. [32]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” in International conference on machine learning , 2023

    2023

  25. [33]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  26. [34]

    Minicpm-v: A gpt-4v level mllm on your phone,

    Y . Yao, T. Yu, A. Zhang, C. Wang, J. Cui, H. Zhu, T. Cai, H. Li, W. Zhao, Z. He et al., “Minicpm-v: A gpt-4v level mllm on your phone,” arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  27. [35]

    Yi: Open foundation models by 01.ai,

    A. Young, B. Chen et al., “Yi: Open foundation models by 01.ai,” 2024

    2024

  28. [36]

    Kosmos-2: Grounding multimodal large language models to the world,

    Z. Peng, W. Wang, L. Dong et al., “Kosmos-2: Grounding multimodal large language models to the world,” arXiv preprint arXiv:2306.14824 , 2023

    2023 arXiv

  29. [37]

    Gpt-4o system card,

    OpenAI, “Gpt-4o system card,” arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  30. [38]

    Reka core, flash, and edge: A series of powerful multimodal language models,

    R. Team, A. Ormazabal, C. Zheng et al. , “Reka core, flash, and edge: A series of powerful multimodal language models,” arXiv preprint arXiv:2404.12387, 2024

    2024 arXiv

  31. [39]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang et al., “Training language models to follow instructions with human feedback,” Advances in neural information processing systems, vol. 35, pp. 27 730–27 744, 2022

    2022

  32. [40]

    Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models,

    D. Dai, C. Deng, C. Zhao et al., “Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models,” arXiv preprint arXiv:2401.06066, 2024

    2024 arXiv

  33. [41]

    Deepseek llm: Scaling open-source language models with longtermism,

    X. Bi, D. Chen, G. Chen et al. , “Deepseek llm: Scaling open-source language models with longtermism,” arXiv preprint arXiv:2401.02954 , 2024

    2024 arXiv

  34. [42]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models,

    Z. Shao, P. Wang, Q. Zhu et al. , “Deepseekmath: Pushing the limits of mathematical reasoning in open language models,” arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  35. [43]

    Deepseek-v2: A strong, econom- ical, and efficient mixture-of-experts language model,

    A. Liu, B. Feng, B. Wang et al. , “Deepseek-v2: A strong, econom- ical, and efficient mixture-of-experts language model,” arXiv preprint arXiv:2405.04434, 2024

    2024 arXiv

  36. [44]

    Qwen2 technical report,

    A. Yang, B. Yang, B. Hui et al., “Qwen2 technical report,” arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  37. [45]

    Internlm: A multilingual language model with progressively enhanced capabilities,

    I. Team, “Internlm: A multilingual language model with progressively enhanced capabilities,” 2023

    2023

  38. [46]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288 , 2023

    2023 arXiv

  39. [47]

    Minicpm: Unveiling the potential of small language models with scalable training strategies,

    S. Hu, Y . Tu, X. Han et al. , “Minicpm: Unveiling the potential of small language models with scalable training strategies,” arXiv preprint arXiv:2404.06395, 2024

    2024 arXiv

  40. [48]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools,

    T. GLM, A. Zeng, B. Xu et al. , “Chatglm: A family of large language models from glm-130b to glm-4 all tools,” arXiv preprint arXiv:2406.12793, 2024

    2024 arXiv

  41. [49]

    Gemini: a family of highly capable multimodal models,

    G. Team, R. Anil, S. Borgeaud et al. , “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805 , 2023

    2023 arXiv

  42. [50]

    Gemini 1.5: Unlocking mul- timodal understanding across millions of tokens of context,

    G. Team, P. Georgiev, V . I. Lei et al. , “Gemini 1.5: Unlocking mul- timodal understanding across millions of tokens of context,” arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  43. [51]

    Introducing the next generation of claude,

    C. Team, “Introducing the next generation of claude,” 2024. [Online]. Available: https://www.anthropic.com/news/claude-3-family

    2024

  44. [52]

    Hotpotqa: A dataset for diverse, explain- able multi-hop question answering,

    Z. Yang, P. Qi, S. Zhang et al., “Hotpotqa: A dataset for diverse, explain- able multi-hop question answering,” arXiv preprint arXiv:1809.09600 , 2018

    2018 arXiv

  45. [53]

    Squad: 100,000+ questions for machine comprehension of text,

    P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “Squad: 100,000+ questions for machine comprehension of text,” arXiv preprint arXiv:1606.05250, 2016

    2016 arXiv

  46. [54]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder et al. , “Language models are few-shot learners,” Advances in neural information processing systems , vol. 33, pp. 1877–1901, 2020

    1901

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.