Pith. sign in

REVIEW 4 major objections 3 minor 45 references

EdgeMM: Multi-Core CPU with Heterogeneous AI-Extension and Activation-aware Weight Pruning for Multimodal LLMs at Edge

T0 review · 4 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A multi-core CPU extended with systolic-array and compute-in-memory coprocessors claims 2.84x speedup over a laptop RTX 3060 for edge multimodal LLM inference.

desk verdict The architecture is a real integration of systolic-array and digital-CIM coprocessors for MLLMs, but the headline efficiency numbers are internally inconsistent and the memory system is unauditable, so the 2.84x claim cannot be taken at face value. read the letter →

arxiv 2505.10782 v1 pith:57EJANUG submitted 2025-05-16 cs.AR

Kangbo Bai , Le Ye , Ru Huang , Tianyu Jia This is my paper · ORCID
classification cs.AR
keywords multimodallargelanguagemodeledgeinferencemulti-coreCPUsystolicarraydigitalcompute-in-memoryactivation-awareweightpruningbandwidthmanagementRISC-VISAextension
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EdgeMM argues that the place for multimodal LLM inference at the edge is not a GPU but a multi-core CPU densely extended with two specialized coprocessors: compute-centric systolic arrays for the GEMM-heavy vision encoder and prefill phase, and memory-centric digital compute-in-memory (CIM) macros for the GEMV-heavy decoding phase. On top of this heterogeneous substrate, the paper proposes dynamic activation-aware weight pruning that skips near-zero activation channels and their corresponding weight rows, reducing DRAM traffic, and token-length-driven bandwidth management that shifts memory budget between core types as output length grows. The claimed result is 2.84x higher throughput than a laptop RTX 3060 GPU on representative multimodal LLMs, reaching 138 tokens/s with roughly 112 mW chip power at 22nm. If correct, this makes privacy-preserving, real-time multimodal AI feasible on power-constrained devices without ASIC compiler effort, because the accelerators are exposed as CPU ISA extensions.

What carries the argument

The load-bearing mechanism is a heterogeneous coprocessor pair driven by extended RISC-V instructions: a systolic-array coprocessor (compute-centric, GEMM-optimized, $ ext{L}_{SA}=2R+C+M-3$ cycles) and a digital CIM coprocessor (memory-centric, GEMV-optimized, $ ext{L}_{CIM}=M W+1$ cycles) that keeps weights in SRAM and broadcasts activations bit-serially. Around these sits the hardware pruner implementing Algorithm 1: per-core activation vectors are Top-k selected, the maximum value masks channels below a threshold $t=16$, and the resulting index list drives the address generator so only non-pruned weight rows are fetched from DRAM and multiplied in the CIM array. Bandwidth management is the third piece: each cluster gets a memory-access budget $B$ enforced by performance-monitoring counters in the DMA, and the $B_c:B_m$ ratio is adjusted with output token length, switching to stream-based batch processing when the budget ratio saturates.

What would settle it

Measure the EdgeMM CIM macro and systolic array on a fabricated testchip or calibrated silicon-validated macros, then replay the same SPHINX-Tiny and KarmaVLM workloads with measured cycle counts, power, and DRAM traffic in the system simulator; if the measured GEMV throughput or DRAM reduction falls materially below the model, the claimed 2.84x speedup and 138 tokens/s do not survive.

Watch

Extended reading notes

Core claim

The paper's central claim is that the two phases of multimodal LLM inference have opposite hardware bottlenecks—compute-bound GEMM in the encoder and prefill, memory-bound GEMV in decoding—so a single homogeneous accelerator wastes most of its silicon in the phase it was not built for. EdgeMM therefore pairs CC-cores, whose systolic-array coprocessor keeps weights stationary and streams activations, with MC-cores, whose digital CIM macro keeps weights in SRAM and broadcasts activations bit-serially, cutting load/store traffic. The authors further claim that activation vectors inside the gated-MLP are channel-sparse with layer-dependent outliers, and that their layer-wise dynamic Top-k pruning can drop many weight rows at DRAM access time with minimal accuracy loss, cutting decoding latency by 42%. Together with bandwidth budgeting that throttles CC-clusters when output tokens lengthen, EdgeMM reports 2.84x speedup over a laptop RTX 3060, 138 tokens/s, and about 112 mW power at 22nm.

Load-bearing premise

The reported speedup rests on the paper's own simulation and CIM-model tools faithfully representing both the EdgeMM chip and the laptop RTX 3060 baseline; if those models are optimistic, the 2.84x result collapses.

Editorial extensions

If this is right

  • Edge devices could run multimodal LLMs on CPU-class hardware with ISA-visible coprocessors, inheriting the existing RISC-V compiler stack and avoiding bus-attached ASIC control latency.
  • Because the pruner cuts DRAM traffic by skipping weight rows, decoding throughput scales with activation sparsity rather than only with raw memory bandwidth.
  • The reported 42% decoding-latency reduction from dynamic Top-k pruning suggests similar gains for other gated-MLP small LLMs with channel-sparse activations.
  • Token-length-driven bandwidth allocation keeps the encoder/prefill and decoding pipeline balanced, and batch processing recovers throughput when output sequences grow past the budget range.
  • At 112 mW and 138 tokens/s, a sub-watt, 22nm-class chip can plausibly serve real-time MLLM tasks if the simulation assumptions hold in silicon.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension not tested in the paper is combining activation-aware row pruning with weight quantization and structured sparsity; the digital CIM's bit-serial multiply logic suggests that activation bit-width, not just active channels, could also be pruned on the fly.
  • The dynamic Top-k policy is validated by cosine similarity and one VQA score on two small MLLMs; applying it to larger MLLMs or to models with different FFN dimensions would show whether the layer-depth trend generalizes.
  • Because the architecture exposes coprocessors through ISA extensions, the same cluster graph could be retargeted to other token-processing workloads, such as speech encoders or streaming sensor fusion, both of which share the GEMM-then-GEMV phase split.
  • The reported 2.84x GPU comparison assumes a laptop GPU under specific bandwidth conditions; a fairer test would measure against a mobile GPU with comparable thermal and power envelope, which may change the speedup sign.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes EdgeMM, a multi-core RISC-V CPU with heterogeneous AI extensions for edge multimodal LLM inference: compute-centric cores with a systolic-array GEMM coprocessor and memory-centric cores with a digital compute-in-memory GEMV coprocessor. It also introduces layer-wise dynamic top-k activation-aware weight pruning and token-length-driven DRAM bandwidth management. The headline claim, from the abstract, introduction, and conclusion, is a 2.84x performance speedup over a laptop RTX 3060 GPU for representative MLLMs such as SPHINX-Tiny and Karmavlm, at 138 tokens/s and 112mW in commercial 22nm technology. The paper reports a 42% latency reduction from the pruning scheme and a 2.14x throughput improvement from bandwidth management at certain output token lengths.

Significance. If the quantitative claims were fully supported, the paper would make a solid systems contribution: it identifies the GEMM/GEMV phase imbalance in edge MLLMs, maps those phases to heterogeneous in-core coprocessors, and couples the design with a dynamic activation-channel pruning scheme. A 22nm RTL implementation with place-and-route power and area numbers is a concrete artifact. However, the significance is conditional: the central speedup and efficiency claims rest on an internally inconsistent set of efficiency numbers and on an evaluation pipeline whose memory-system assumptions are not reported. The architecture-level ideas are credible and worth pursuing, but the paper as written does not yet provide an auditable quantitative case.

major comments (4)
  1. [Section I, Section V-C, Table II, Conclusion] The reported energy-efficiency numbers are internally inconsistent. The contribution bullet in Section I reports 0.217 token/J, while Section V-C and the conclusion report 0.28 token/J. From the stated throughput and power of 138 tokens/s and 112mW, the efficiency is 1232 tokens/J (0.812 mJ/token). These three values cannot all be true. This is a load-bearing inconsistency because the 2.84x speedup claim depends on the same simulation/power report; the authors must reconcile the methodology that produced these numbers.
  2. [Section V-A] The memory system behind the headline throughput is not specified. For SPHINX-Tiny, whose decoder is TinyLlama-1.1B, reading the full weight set at BF16 requires roughly 2.2GB per decoding token; at 138 tokens/s that is about 304GB/s of sustained weight traffic, or about 176GB/s even with the reported 42% pruning. The paper reports no DRAM bandwidth, no CIM macro capacity, no on-chip weight-residency fraction, and no external memory configuration. Without these parameters, the 138 tokens/s and the resulting 2.84x speedup are not auditable. Please add a memory hierarchy specification and a bandwidth/roofline analysis.
  3. [Section III-B, Equations (2)-(3)] The cycle counts for the systolic array and CIM coprocessors, L_SA = 2R+C+M-3 and L_CIM = M*W+1, are asserted without derivation or validation. These equations directly determine the modeled performance and hence the comparison with the RTX 3060. Please provide a derivation, or better, a validation table showing modeled cycles against RTL-simulated cycles for representative matrix shapes and data types.
  4. [Section IV-A and Section V-C, Figure 12] The generality of the activation-aware pruning scheme is not established. The threshold t=16 is fixed, the first layer is skipped because, as the text states, pruning it causes significant accuracy degradation, and the profiling is performed on only two MLLMs. Accuracy impact is reported only as cosine similarity and a qualitative 'minimal score reduction in VQA' with no end-task numerical scores. The claimed 42% latency reduction and the additional speedup in Table II therefore need a sensitivity analysis over t, a justification for skipping the first layer, and end-task accuracy numbers on a broader set of models.
minor comments (3)
  1. [Table II] The comparison with the RTX 3060 Laptop is underspecified: please report the GPU power, software stack, batch size, output token length, and precision used for the GPU baseline, and state whether the GPU number is measured or taken from a prior source.
  2. [Section IV-A] The name 'activation-aware weight pruning' is potentially misleading because the scheme prunes weight rows corresponding to low-magnitude activation channels; a term such as 'dynamic activation-channel row pruning' would be more precise.
  3. [Figure 3(b) and Section V-C] Figure 3(b) should define the x-axis and the color/height scale in the caption; in Section V-C, the sentence 'with pruning ratio of a randomly chosen core' should clarify whether the curve is a single core or an average over cores and why that is representative.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found; the central claims are empirical simulation results rather than derivations from their own inputs.

full rationale

No circular step meets the evidentiary bar of the review. The paper is an empirical architecture study: EdgeMM is implemented in RTL and post-P&R flow, and its speedup, throughput, and pruning benefits are reported from simulation. The activation-aware pruning scheme in Algorithm 1 uses a threshold t=16 and skips the first decoder layer after the authors observe accuracy degradation; these are hyperparameter choices evaluated with cosine similarity and VQA scores, not quantities defined as the target outcome, so the 42% latency reduction is a measured result rather than a prediction forced by construction. The 2.84x comparison against RTX 3060 is a simulator-reported benchmark; while the simulator is in-house and unaudited, and the reported efficiency numbers are internally inconsistent (138 tokens/s at 112mW implies roughly 1232 tokens/J, not the 0.217 token/J in Section I or 0.28 token/J in Table II), those are correctness, unit-consistency, and auditability concerns, not circularity. The paper does not rely on load-bearing self-citation: Snitch [43] is third-party open source, and the pruning references [44], [45] are external works. There is no imported uniqueness theorem, no ansatz smuggled in by citation, and no equation that reduces to its own input. Accordingly, the derivation chain is self-contained even though its empirical support is incomplete.

Assumptions & free parameters 5 free parameters · 5 assumptions · 2 invented entities

The central performance claims rest on a set of hand-chosen thresholds (t=16, l_e=36, l_b=131, bandwidth ratios), an unvalidated cycle model, and an in-house simulator plus CIM IP that are not publicly available. The pruning efficacy is partly determined by the chosen t and by the post hoc skip-first-layer rule. These are the main burden items that a reader must accept to trust the 2.84x result.

free parameters (5)
  • pruning threshold t = 16
    Sec. IV-A: a channel is considered negligible when its magnitude is t times below the max channel; t is fixed at 16 without sensitivity analysis.
  • expected output token length l_e for bandwidth balance = 36
    Sec. V-C and Fig. 13: l_e is set to 36 as the point where CC and MC cluster latencies balance; the value is workload-specific.
  • batch-start token length l_b = 131
    Sec. V-C: when output length reaches l_b=131, bandwidth allocation saturates and batch decoding is introduced; the value is chosen for the evaluated workload.
  • bandwidth budget ratio B_c:B_m = 1:3 to 1:7
    Sec. IV-B and Fig. 13: the CC-to-MC bandwidth budget is adjusted dynamically over a range, but the schedule is heuristic and not derived from a closed-form model.
  • systolic array dimensions R x C and CIM macro dimensions M x N = not stated in text
    Fig. 10 shows design configurations, but exact array sizes and CIM tile sizes that determine cycle counts, area, and bandwidth are not given in the text.
assumptions (5)
  • domain assumption Activation vectors in the FFN of MLLMs are channel-sparse with outliers such that top-k pruning preserves output quality.
    Sec. II-B and Fig. 3 profile SPHINX-Tiny; the assumption is used to justify the pruning scheme in Sec. IV-A and is not proven across all MLLMs.
  • domain assumption FFN weight matrices dominate DRAM access and KV cache is negligible for short-context edge MLLMs.
    Sec. II-B and Fig. 2 profile two MLLMs with about 300 input tokens; this may not hold for long-context or streaming workloads.
  • domain assumption The in-house 22nm digital CIM macro and the in-house simulator accurately model the speed, energy, and area of the full EdgeMM system.
    Sec. V-A states the performance is evaluated by RTL simulation and an in-house simulator, but no silicon measurements or independent validation are provided.
  • standard math The cycle count equations L_SA = 2R+C+M-3 and L_CIM = M*W+1 correctly model the coprocessors.
    Sec. III-B, equations (2) and (3), are asserted without derivation or validation against RTL cycle counts.
  • domain assumption The Snitch cluster with SIMD cores is a fair baseline and the RISC-V extension inherits the compiler stack without significant overhead.
    Sec. I and Sec. V-A use the open-source Snitch cluster as baseline, but software overhead and compiler integration are not quantified.
invented entities (2)
  • In-house 22nm digital CIM macro IP
    purpose: Memory-centric coprocessor for GEMV that computes inside SRAM banks to reduce weight load and store traffic.
    Sec. III-B and Sec. V-A describe the macro, but no fabricated chip measurements are reported; its cycle and energy behavior is simulated only.
  • Hardware activation-aware pruner
    purpose: Selects top-k activation channels and generates addresses of weight rows to skip pruned DRAM reads during GEMV.
    Sec. IV-A and Fig. 8 present this unit; it is evaluated only inside the authors' simulator and has no standalone measured silicon evidence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EdgeMM: Multi-Core CPU with Heterogeneous AI-Extension and Activation-aware Weight Pruning for Multimodal LLMs at Edge." pith.science (2026). https://pith.science/paper/57EJANUG

@misc{pith2026250510782,
  author       = {Pith},
  title        = {Pith review of: EdgeMM: Multi-Core CPU with Heterogeneous AI-Extension and Activation-aware Weight Pruning for Multimodal LLMs at Edge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/57EJANUG}},
  note         = {Machine review of arXiv:2505.10782}
}
read the original abstract

Emerging multimodal LLMs (MLLMs) exhibit strong cross-modality perception and reasoning capabilities and hold great potential for various applications at edge. However, MLLMs typically consist of a compute-intensive modality encoder and a memory-bound LLM decoder, leading to distinct bottlenecks for hardware designs. In this work, we present a multi-core CPU solution with heterogeneous AI extensions, which are based on either the compute-centric systolic array or memory-centric digital compute-in-memory (CIM) co-processors. In addition, dynamic activation-aware weight pruning and bandwidth management are developed to enhance bandwidth efficiency and core utilization, improving overall performance. We implemented our solution using commercial 22nm technology. For representative MLLMs, our evaluations show EdgeMM can achieve 2.84x performance speedup compared to laptop 3060 GPU.

Figures

Figures reproduced from arXiv: 2505.10782 by the authors.

Figure 1
Figure 1. MLLMs and hardware architecture design choices. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Workload analysis of two MLLMs: (a) latency breakdown, (b) model [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Gated-MLP and the activation vector sparsity in FFN. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Compute-centric core with Systolic Array. [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 9
Figure 9. Figure 9: Pipeline of bandwidth and workload management. [PITH_FULL_IMAGE:figures/full_fig_p005_9.png]
Figure 11
Figure 11. Figure 11: Perf. improvements of homo. and hetero. designs [PITH_FULL_IMAGE:figures/full_fig_p006_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 22 canonical work pages

  1. [1]

    A survey on multimodal large language models for autonomous driving,

    C. Cuiet al., “A survey on multimodal large language models for autonomous driving,” inIEEE/CVF Winter Conference on Applications of Computer Vision, pp. 958–979, 2024

    work page 2024

  2. [2]

    Aligning cyber space with physical world: A com- prehensive survey on embodied ai,

    Y . Liu, W. Chen, Y . Bai, J. Luo, X. Song, K. Jiang, Z. Li, G. Zhao, J. Lin, G. Li,et al., “Aligning cyber space with physical world: A com- prehensive survey on embodied ai,”arXiv preprint arXiv:2407.06886, 2024

    arXiv 2024

  3. [3]

    Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices,

    X. Chu, L. Qiao, X. Lin, S. Xu, Y . Yang, Y . Hu, F. Wei, X. Zhang, B. Zhang, X. Wei,et al., “Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices,”arXiv preprint arXiv:2312.16886, 2023

    arXiv 2023

  4. [4]

    Sam4mllm: Enhance multi-modal large language model for referring expression segmentation,

    Y .-C. Chen, W.-H. Li, C. Sun, Y .-C. F. Wang, and C.-S. Chen, “Sam4mllm: Enhance multi-modal large language model for referring expression segmentation,”arXiv preprint arXiv:2409.10542, 2024

    arXiv 2024

  5. [5]

    Lmdrive: Closed-loop end-to-end driving with large language models,

    H. Shaoet al., “Lmdrive: Closed-loop end-to-end driving with large language models,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15120–15130, 2024

    work page 2024

  6. [6]

    A 17–95.6 tops/w deep learning inference accelerator with per-vector scaled 4-bit quantization for transformers in 5nm,

    B. Kelleret al., “A 17–95.6 tops/w deep learning inference accelerator with per-vector scaled 4-bit quantization for transformers in 5nm,” in 2022 IEEE Symposium on VLSI Technology and Circuits (VLSI), 2022

    work page 2022

  7. [7]

    22.9 a 12nm 18.1tflops/w sparse transformer processor with entropy-based early exit, mixed-precision predication and fine- grained power management,

    T. Tambeet al., “22.9 a 12nm 18.1tflops/w sparse transformer processor with entropy-based early exit, mixed-precision predication and fine- grained power management,” inIEEE International Solid-State Circuits Conference (ISSCC), 2023

    work page 2023

  8. [8]

    Hetegen: Efficient heterogeneous parallel inference for large language models on resource-constrained devices,

    Z. XUANLEI, B. Jia, H. Zhou, Z. Liu, S. Cheng, and Y . You, “Hetegen: Efficient heterogeneous parallel inference for large language models on resource-constrained devices,” inMachine Learning and Systems (P. Gibbons, G. Pekhimenko, and C. D. Sa, eds.), vol. 6, pp. 162–172, 2024

    work page 2024

Show all 45 references
  1. [9]

    A survey on deep learning hardware accelerators for heterogeneous hpc platforms,

    C. Silvano, D. Ielmini, F. Ferrandi, L. Fiorin, S. Curzel, L. Benini, F. Conti, A. Garofalo, C. Zambelli, E. Calore,et al., “A survey on deep learning hardware accelerators for heterogeneous hpc platforms,”arXiv preprint arXiv:2306.15552, 2023

    2023 arXiv

  2. [10]

    Blueface: Integrating an accelerator into the core’s pipeline through algorithm-interface co- design for real-time socs,

    Z. Jiang, N. Fisher, N. Guan, and Z. Dong, “Blueface: Integrating an accelerator into the core’s pipeline through algorithm-interface co- design for real-time socs,” in2023 60th ACM/IEEE Design Automation Conference (DAC), pp. 1–6, 2023

    2023

  3. [11]

    Online inhttps://www.intel.com/content/www/us/en/products/docs/ accelerator-engines/advanced-matrix-extensions/overview.html

  4. [12]

    Sapphire rapids: The next-generation intel xeon scal- able processor,

    N. Nassifet al., “Sapphire rapids: The next-generation intel xeon scal- able processor,” inIEEE International Solid-State Circuits Conference (ISSCC), pp. 44–46, 2022

    2022

  5. [13]

    Intel accelerators ecosystem: An soc-oriented perspec- tive,

    Y . Yuanet al., “Intel accelerators ecosystem: An soc-oriented perspec- tive,” in2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pp. 848–862, 2024

    2024

  6. [14]

    Online inhttps://developer.arm.com/documentation/ddi0616/latest/

  7. [15]

    Online inhttps://github.com/T-head-Semi/riscv-matrix-extension-spec

  8. [16]

    Generative multimodal models are in-context learners,

    Q. Sun, Y . Cui, X. Zhang, F. Zhang, Q. Yu, Y . Wang, Y . Rao, J. Liu, T. Huang, and X. Wang, “Generative multimodal models are in-context learners,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14398–14409, 2024

    2024

  9. [17]

    Eva-clip: Improved training techniques for clip at scale,

    Q. Sun, Y . Fang, L. Wu, X. Wang, and Y . Cao, “Eva-clip: Improved training techniques for clip at scale,”arXiv preprint arXiv:2303.15389, 2023

    2023 arXiv

  10. [18]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” Advances in neural information processing systems, vol. 36, 2024

    2024

  11. [19]

    Learning transferable visual models from natural lan- guage supervision,

    A. Radfordet al., “Learning transferable visual models from natural lan- guage supervision,” inInternational Conference on Machine Learning (ICML), pp. 8748–8763, 2021

    2021

  12. [20]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023,

    W.-L. Chiang, Z. Li, Z. Lin, Y . Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y . Zhuang, J. E. Gonzalez,et al., “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023,”URL https://lmsys. org/blog/2023-03-30-vicuna, vol. 3, no. 5, 2023

    2023

  13. [21]

    Mobilevlm v2: Faster and stronger baseline for vision language model,

    X. Chu, L. Qiao, X. Zhang, S. Xu, F. Wei, Y . Yang, X. Sun, Y . Hu, X. Lin, B. Zhang,et al., “Mobilevlm v2: Faster and stronger baseline for vision language model,”arXiv preprint arXiv:2402.03766, 2024

    2024 arXiv

  14. [22]

    Tinygpt-v: Efficient multimodal large language model via small backbones,

    Z. Yuan, Z. Li, and L. Sun, “Tinygpt-v: Efficient multimodal large language model via small backbones,”arXiv preprint arXiv:2312.16862, 2023

    2023 arXiv

  15. [23]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” inInternational Conference on Machine Learning (ICML), pp. 19730–19742, 2023

    2023

  16. [24]

    Phi-2: The surprising power of small language models,

    M. Javaheripi, S. Bubeck, M. Abdin, J. Aneja, S. Bubeck, C. C. T. Mendes, W. Chen, A. Del Giorno, R. Eldan, S. Gopi,et al., “Phi-2: The surprising power of small language models,”Microsoft Research Blog, 2023

    2023

  17. [25]

    Sphinx-x: Scaling data and parameters for a family of multi-modal large language models,

    P. Gao, R. Zhang, C. Liu, L. Qiu, S. Huang, W. Lin, S. Zhao, S. Geng, Z. Lin, P. Jin,et al., “Sphinx-x: Scaling data and parameters for a family of multi-modal large language models,”arXiv preprint arXiv:2402.05935, 2024

    2024 arXiv

  18. [26]

    Dinov2: Learning robust visual features without supervision,

    M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby,et al., “Dinov2: Learning robust visual features without supervision,”arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  19. [27]

    Tinyllama: An open-source small language model,

    P. Zhang, G. Zeng, T. Wang, and W. Lu, “Tinyllama: An open-source small language model,”arXiv preprint arXiv:2401.02385, 2024

    2024 arXiv

  20. [28]

    Deepseek-vl: towards real-world vision-language understanding,

    H. Lu, W. Liu, B. Zhang, B. Wang, K. Dong, B. Liu, J. Sun, T. Ren, Z. Li, Y . Sun,et al., “Deepseek-vl: towards real-world vision-language understanding,”arXiv preprint arXiv:2403.05525, 2024

    2024 arXiv

  21. [29]

    Sigmoid loss for language image pre-training,

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” inIEEE/CVF International Conference on Computer Vision (ICCV), pp. 11975–11986, 2023

    2023

  22. [30]

    https://github.com/ thomas-yanxin/KarmaVLM, 2024

    Online inKarmavlm: A family of high efficiency and powerful visual language model. https://github.com/ thomas-yanxin/KarmaVLM, 2024

    2024

  23. [31]

    Qwen technical report,

    J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, X. Deng, Y . Fan, W. Ge, Y . Han, F. Huang,et al., “Qwen technical report,”arXiv preprint arXiv:2309.16609, 2023

    2023 arXiv

  24. [32]

    Gemini: a family of highly capable multimodal models,

    G. Team, R. Anil, S. Borgeaud, Y . Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth,et al., “Gemini: a family of highly capable multimodal models,”arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  25. [33]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat,et al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  26. [34]

    Improved baselines with visual instruction tuning,

    H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 26296–26306, 2024

    2024

  27. [35]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering,

    Y . Goyal, T. Khot, A. Agrawal, D. Summers-Stay, D. Batra, and D. Parikh, “Making the v in vqa matter: Elevating the role of image understanding in visual question answering,” 2019

    2019

  28. [36]

    Learn to explain: Multimodal reason- ing via thought chains for science question answering,

    P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan, “Learn to explain: Multimodal reason- ing via thought chains for science question answering,”ArXiv, vol. abs/2209.09513, 2022

    2022 arXiv

  29. [37]

    Seed-bench: Benchmarking multimodal llms with generative comprehension,

    B. Li, R. Wang, G. Wang, Y . Ge, Y . Ge, and Y . Shan, “Seed-bench: Benchmarking multimodal llms with generative comprehension,”ArXiv, vol. abs/2307.16125, 2023

    2023 arXiv

  30. [38]

    Mmbench: Is your multi-modal model an all-around player?,

    Y . Liu, H. Duan, Y . Zhang, B. Li, S. Zhang, W. Zhao, Y . Yuan, J. Wang, C. He, Z. Liu, K. Chen, and D. Lin, “Mmbench: Is your multi-modal model an all-around player?,” inEuropean Conference on Computer Vision, 2023

    2023

  31. [39]

    Full stack optimization of transformer inference: a survey,

    S. Kim, C. Hooper, T. Wattanawong, M. Kang, R. Yan, H. Genc, G. Dinh, Q. Huang, K. Keutzer, M. W. Mahoney,et al., “Full stack optimization of transformer inference: a survey,”arXiv preprint arXiv:2302.14017, 2023

    2023 arXiv

  32. [40]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale,et al., “Llama 2: Open foundation and fine-tuned chat models,”arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  33. [41]

    Mistral 7b,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier,et al., “Mistral 7b,”arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  34. [42]

    H2o: Heavy- hitter oracle for efficient generative inference of large language models,

    Z. A. Zhang, Y . Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y . Tian, C. R ´e, C. W. Barrett, Z. Wang, and B. Chen, “H2o: Heavy- hitter oracle for efficient generative inference of large language models,” ArXiv, vol. abs/2306.14048, 2023

    2023 arXiv

  35. [43]

    Snitch: A tiny pseudo dual-issue processor for area and energy efficient execution of floating- point intensive workloads,

    F. Zaruba, F. Schuiki, T. Hoefler, and L. Benini, “Snitch: A tiny pseudo dual-issue processor for area and energy efficient execution of floating- point intensive workloads,”IEEE Transactions on Computers, 2020

    2020

  36. [44]

    A simple and effective pruning approach for large language models,

    M. Sun, Z. Liu, A. Bair, and J. Z. Kolter, “A simple and effective pruning approach for large language models,”arXiv preprint arXiv:2306.11695, 2023

    2023 arXiv

  37. [45]

    Cats: Contextually-aware thresholding for sparsity in large language models,

    J.-Y . Lee, D. Lee, G. Zhang, M. Tiwari, and A. Mirhoseini, “Cats: Contextually-aware thresholding for sparsity in large language models,” arXiv preprint arXiv:2404.08763, 2024

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.