Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

EventGPT: Event Stream Understanding with Multimodal Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read EventGPT is presented as the first multimodal large language model built for event-camera streams, with a three-stage training scheme that aligns sparse asynchronous pixel-change signals to language and outperforms image-based multimodal…

desk verdict First event-stream chat MLLM with two useful new datasets, but the quantitative superiority claim rests on an in-distribution, self-scored benchmark with untrained baselines and test-set hyperparameter tuning. read the letter →

arxiv 2412.00832 v1 pith:F4CQL3AP submitted 2024-12-01 cs.CV

Shaoyu Liu , Jianing Li , Guanghui Zhao , Yunjian Zhang , Xin Meng , Fei Richard Yu , Xiangyang Ji , Ming Li This is my paper · ORCID
classification cs.CV
keywords eventcamerastreamunderstandingmultimodallargelanguagemodelspatio-temporalaggregationcross-modalalignmentinstructiontuningevent-textdatasetvisualquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a multimodal large language model can be made to understand event-camera streams—not just static images—by progressively aligning the event representation with language in three training stages. It introduces EventGPT, built from an event encoder, a spatio-temporal aggregator, a linear projector, an event-language adapter, and an LLM. Because event streams are sparse brightness-change signals rather than dense images, the authors first warm up the projector on ordinary image-text data, then align event features to language with a synthetic million-sample dataset, and finally fine-tune the whole model on real-world instruction data. On their own benchmark, EventGPT outperforms existing image-based multimodal LLMs on detailed captioning, complex reasoning, and visual question answering for event scenes. The paper also releases the two event-text datasets it builds.

What carries the argument

The central mechanism is a two-stage cross-modal projector driven by a three-stage training schedule. The spatio-temporal aggregator divides the event stream into $T$ temporal bins, encodes each bin, applies pooling along the temporal and spatial dimensions, and concatenates the pooled features into $\hat{Z} \in \mathbb{R}^{(T+S)\times D}$, capturing motion cues that a single static event frame lacks. The event-language adapter then maps these fused features into the LLM's embedding space. Its role is to make sparse event data expressible in the language space already anchored by the LLM, so the model can describe scenes, answer questions, and reason rather than merely classify.

What would settle it

Run the same three metrics with human raters on a held-out real-event dataset whose question-answer pairs are written by event-camera experts after the fact, keeping EventGPT frozen, and compare against an image-based MLLM that has been fine-tuned on the same event-text training data; if EventGPT does not beat that model, the reported superiority is an artifact of the benchmark.

Watch

Extended reading notes

Core claim

The central claim is that event-stream understanding can be brought to LLMs by treating the event stream as a sequence of temporal windows, encoding each window, pooling spatial and temporal dimensions separately, and fusing the result before projecting it into language space. The authors call this model EventGPT and present it as the first multimodal large language model designed specifically for event streams. The load-bearing device is the three-stage schedule: freeze most modules while training only the projector on image-text pairs, then freeze the encoder, projector, and LLM while training the spatio-temporal aggregator and event-language adapter on synthetic event-text data, then unfreeze everything for instruction tuning on real-world data. This progression, the paper argues, narrows the domain gap between asynchronous event data and natural language and yields superior generation quality, descriptive accuracy, and reasoning compared with state-of-the-art image-oriented multimodal LLMs evaluated on the same event-frame inputs.

Load-bearing premise

The reported gains are measured on test splits drawn from the same data distributions used in training, with comparison models that never received event-specific training; if those gains do not survive human-scored or out-of-distribution evaluation, the claimed superiority would not hold.

Editorial extensions

If this is right

  • EventGPT extends multimodal LLM capabilities to event cameras, enabling scene summarization, reasoning, and visual question answering in low-light and high-dynamic-range scenarios.
  • The text generated by EventGPT can be piped into open-set detection and segmentation heads, yielding object-level reasoning from event data.
  • The two released datasets—one million synthetic samples and 120,000 instruction samples—provide a training and evaluation foundation for future event-language models.
  • The paper reports that a temporal window count of five gives the best results, with performance degrading when windows are too few or too many.
  • The three-stage alignment paradigm is presented as a reusable recipe for bridging large cross-modal gaps beyond the event-domain setting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same three-stage recipe could transfer to other asynchronous sensors, such as LiDAR, radar, or neuromorphic audio, because the first stage establishes language grounding and the second stage adapts a specialized encoder.
  • A credible next test would be human-scored evaluation on real outdoor event data with out-of-distribution scenes; the paper's current numbers do not yet establish transfer to unseen distributions because the test questions come from the same sources as the training data.
  • Because EventGPT inherits the LLM's world knowledge, its descriptions may be plausible but not always grounded; an object-presence probe on synthetic event scenes with known ground truth could quantify the rate of ungrounded claims.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces EventGPT, an MLLM for event stream understanding, with an event encoder, a spatio-temporal aggregator, a linear projector, an event-language adapter, and an LLM. Training proceeds in three stages: visual-language alignment on LLaVA-Pre-train, event-language alignment on a synthetic N-ImageNet-Chat dataset (about 1M samples), and instruction fine-tuning on an Event-Chat dataset (about 120k samples). The authors also release two event-text datasets and a benchmark with GPT-based scoring on Detail Captioning, Complex Reasoning, and VQA. The central claim is that EventGPT surpasses previous state-of-the-art MLLMs in generation quality, descriptive accuracy, and reasoning capability for event streams.

Significance. If the evaluation were properly controlled, this would be a useful contribution: EventGPT appears to be one of the first attempts to adapt an LLM to event streams, and the release of large event-text datasets could benefit the neuromorphic and multimodal communities. The three-stage training paradigm is a reasonable, if incremental, recipe for cross-modal alignment. However, the current evidence does not establish the paper's headline claim of superiority over existing MLLMs, because the benchmark is in-distribution for EventGPT and out-of-distribution for all baselines, the automatic judge is unspecified, and key hyperparameters are selected on the test set. These are load-bearing issues, not presentation details, and they affect Table 1, which is the only quantitative support for the abstract claim.

major comments (4)
  1. [§5.1, §5.2, Table 1] The quantitative comparison is not a valid test of the claim that EventGPT 'surpasses previous state-of-the-art MLLMs.' The test splits of N-ImageNet-Chat and Event-Chat are derived from the same datasets used in Stages 2 and 3 of training, while the baseline MLLMs (LLaVA, Qwen2-VL, Intern2VL, DeepSeek-VL) are evaluated on event frames without any event-domain training. Thus Table 1 conflates 'trained on the test distribution' with 'better event understanding.' To support the claim, the authors need either event-adapted baselines (e.g., fine-tuned on the same Event-Chat training data), an external held-out event benchmark, or at least a matched protocol where all models see the same event representation and receive equivalent fine-tuning.
  2. [§5.1, Evaluation Metrics; Table 1] The scoring procedure is underspecified and potentially circular. The paper says responses are 'quantitatively scored on a 1-5 scale' and that the benchmark is 'inspired by Qwen2-72B-Instruct,' but it does not state which model judges the outputs, whether the judge is the same model that generated the training data, or whether any human validation or inter-annotator agreement was performed. Since EventGPT is fine-tuned on GPT-generated text, an automatic judge that prefers that response style could inflate EventGPT's scores. The authors should report the exact judge, prompt, sample size, human correlation, and variance or confidence intervals.
  3. [§5.3, Table 3] The temporal window number Nw is selected by maximizing performance on Event-Chat, and Table 1 then reports EventGPT results using the selected Nw=5. This means the test set has influenced model selection, so the reported numbers in Table 1 are not unbiased estimates of generalization. The selection should be done on a validation split, or the paper should report both validation-selected and test-selected results with an explicit statement that the test set was used only once.
  4. [§5.1, 'zero-shot question-answering setting'] The evaluation is described as zero-shot, but EventGPT is fine-tuned on Event-Chat, which contains the same tasks (captioning, VQA, reasoning) and the same data distribution as the test set. Calling this zero-shot is misleading. The authors should either rephrase the evaluation as in-distribution, or provide a genuinely zero-shot setting (e.g., held-out event domains or tasks) and compare against baselines under the same conditions.
minor comments (5)
  1. [§4.2, Eqs. (4)–(6)] The text states that average pooling is applied along the temporal and spatial dimensions, but Eq. (6) concatenates 'max-pooled representations' Zmax_T and Zmax_S. Please harmonize the notation and text so the reader knows which pooling operation is actually used.
  2. [§3.1, §3.2, Fig. 3] The dataset sizes are stated inconsistently: Section 3.1 says N-ImageNet-Chat has over 1,000,000 instances, N-ImageNet-Instruction has 69,000 entries, and Event-Chat has 59,000 entries, while the abstract and Fig. 3 refer to '120,000 instruction samples' for Event-Chat. Please clarify how the 69k and 59k numbers combine into 120k, and define which split is used for training versus testing.
  3. [§5.3, Table 2] The ablation gains are small (roughly 1–3% relative improvement) and are reported without error bars, significance tests, or multiple runs. Given the 1–5 scoring scale and high baseline scores (e.g., 4.15 VQA), the paper should show that these differences are not within judge noise.
  4. [§4.2, Eq. (3)] The definition of Z uses separate temporal bins Et, but the relationship between the time window T, the temporal window number Nw, and the event tensor representation is not defined precisely. Please state how Nw is used to construct the tensor in Eq. (3).
  5. [§1, Abstract] The claim 'the first MLLM for event stream understanding' should be softened to 'to the best of our knowledge' in the abstract and introduction, and it would be helpful to cite or discuss the closest existing works (e.g., EventCLIP, EventBind, ExACT) more explicitly to support the novelty statement.

Circularity Check

2 steps flagged · score 5.0 of 10

EventGPT's headline Event-Chat superiority is partly a fitted quantity: the temporal-window hyperparameter Nw is selected on the Event-Chat test set, and the same test set is then reported as evidence in Table 1.

  1. fitted input called prediction [Section 5.3, Table 3 and 'Influence of Temporal Aggregation Size'; used in Section 5.2, Table 1]
    "The quantitative results in Table 3 reveal that our framework is robust to a wide range of temporal aggregation sizes. Furthermore, a temporal window number of 5 yields the best overall performance. Accordingly, we set Nw = 5 in the spatio-temporal aggregator by default."

    Nw is selected by maximizing Event-Chat test scores in Table 3, and the same Event-Chat test scores are then reported in Table 1 as the headline evidence that EventGPT 'surpasses previous state-of-the-art MLLMs.' The benchmark is based on test sets of N-ImageNet-Chat and Event-Chat, the same datasets used for training in Stages 2 and 3, so the Event-Chat test score is not an independent prediction; it is a quantity partly optimized on the test set. The comparison is further skewed because the baselines receive event frames but no event-domain training, conflating 'trained on the test distribution' with 'better event understanding.'

  2. other [Section 5.1, 'Evaluation Metrics' and Section 4.3, 'Instruction Tuning']
    "Our benchmark, specifically designed for text generation from event streams, is based on the test sets of N-ImageNet-Chat and Event-Chat datasets... Trained on our high-quality Event-Chat question-answering dataset, this stage further enhances the model's understanding and generation capabilities for event stream data."

    The benchmark test sets are drawn from the very datasets used in the model's training stages, and the reference QA pairs in those datasets are produced by the same GPT-assisted generation pipeline described in Section 5.1. Measuring 'event understanding' on this in-distribution, AI-generated benchmark reduces the superiority claim largely to 'the model performs well on its own training distribution,' especially since the baselines have no event-specific training. This is a partial circularity in the evaluation protocol rather than in the model's architecture, but it directly weakens the paper's central claim of surpassing prior MLLMs.

full rationale

The paper's central derivation is not circular in the sense of an equation reducing to its inputs: EventGPT's architecture, three-stage training, and dataset construction are real, and no load-bearing argument rests on a self-citation or an imported uniqueness theorem. The main circular step is empirical: the temporal-window hyperparameter Nw is tuned on the Event-Chat test set (Table 3), and the resulting Event-Chat scores are then reported as evidence of superiority in Table 1, making that portion of the headline result statistically forced. Additionally, the benchmark is built from the same N-ImageNet-Chat and Event-Chat datasets used for training, and the reference answers are generated by the same GPT-assisted pipeline, so the comparison against untrained RGB-image baselines conflates in-distribution familiarity with event understanding. These evaluation-protocol issues are substantial, but they do not make the model's derivation itself circular; the architecture and training pipeline stand as independent contributions. Score 5 reflects one concrete fitted-input-called-prediction step plus a benchmark that is partly self-referential, without reaching a full 6+ where the central claim reduces entirely to a fit or a self-citation chain.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the validity of the self-constructed benchmark and the alignment of synthetic event frames with GPT-generated text. No new physical entities are introduced; the datasets N-ImageNet-Chat and Event-Chat are new resources but are not yet publicly available.

free parameters (1)
  • Temporal window number Nw = 5
    Selected because it yields the best scores on the Event-Chat test set (Table 3), which is hyperparameter tuning on the test set.
assumptions (4)
  • domain assumption N-ImageNet event frames preserve the semantics of their aligned ImageNet images, so GPT-generated captions for ImageNet images are valid text for the corresponding event frames.
    The N-ImageNet-Chat dataset is built on this alignment (Section 3.1). If the event simulation loses semantic content, the synthetic captions are wrong.
  • domain assumption Event frames rendered as images provide sufficient information for language understanding; the event encoder initialized with OpenCLIP ViT can extract useful features from these frames.
    The whole pipeline feeds event frames as images to the encoder (Section 5.2).
  • ad hoc to paper GPT-based scoring of generated text (inspired by Qwen2-72B-Instruct) is a valid measure of descriptive accuracy and reasoning for event-based QA.
    The evaluation metrics DC, CR, and VQA all rely on this scoring (Section 5.1).
  • standard math Standard average pooling and concatenation operations work as described.
    Used in the spatio-temporal aggregator (Section 4.2). The text also refers to max pooling without defining it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EventGPT: Event Stream Understanding with Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/F4CQL3AP

@misc{pith2026241200832,
  author       = {Pith},
  title        = {Pith review of: EventGPT: Event Stream Understanding with Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F4CQL3AP}},
  note         = {Machine review of arXiv:2412.00832}
}
read the original abstract

Event cameras record visual information as asynchronous pixel change streams, excelling at scene perception under unsatisfactory lighting or high-dynamic conditions. Existing multimodal large language models (MLLMs) concentrate on natural RGB images, failing in scenarios where event data fits better. In this paper, we introduce EventGPT, the first MLLM for event stream understanding, to the best of our knowledge, marking a pioneering attempt to integrate large language models (LLMs) with event stream comprehension. To mitigate the huge domain gaps, we develop a three-stage optimization paradigm to gradually equip a pre-trained LLM with the capability of understanding event-based scenes. Our EventGPT comprises an event encoder, followed by a spatio-temporal aggregator, a linear projector, an event-language adapter, and an LLM. Firstly, RGB image-text pairs generated by GPT are leveraged to warm up the linear projector, referring to LLaVA, as the gap between natural image and language modalities is relatively smaller. Secondly, we construct a synthetic yet large dataset, N-ImageNet-Chat, consisting of event frames and corresponding texts to enable the use of the spatio-temporal aggregator and to train the event-language adapter, thereby aligning event features more closely with the language space. Finally, we gather an instruction dataset, Event-Chat, which contains extensive real-world data to fine-tune the entire model, further enhancing its generalization ability. We construct a comprehensive benchmark, and experiments show that EventGPT surpasses previous state-of-the-art MLLMs in generation quality, descriptive accuracy, and reasoning capability.

Figures

Figures reproduced from arXiv: 2412.00832 by the authors.

Figure 1
Figure 1. Our EventGPT is the first multimodal large language model tailored for event stream understanding, including scene summarization, reasoning, and question answering, with a great potential for downstream tasks. plication potential in scenarios under imperfect lighting conditions or including high dynamic objects [7, 12, 47]. The relevant tasks on event data, e.g. object detection [8, 10, 16, 39], tracking [9, 25, 33,… view at source ↗
Figure 2
Figure 2. Overview of our framework. The event encoder transforms raw event tensors into high-dimensional features, which are then [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Data distribution across three progressive training stages [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Instruction examples from the Event-Chat dataset for [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Our three-stage pipeline consists of (1) training a linear [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Representative visualization comparison between EventGPT and other open-source MLLMs models under extreme lighting vari [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Representative comparison of EventGPT and other open-source multimodal language models (MLLMs) in high-speed motion [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Object detection [22] using events and text from our EventGPT. Note that, our EventGPT enables the generation of high-quality text for the open-set object detection task. EventGPT: A car is an object that is typically used for personal transportation on the road. There…
Figure 9
Figure 9. Figure 9: Instance segmentation [31] leveraging events and text generated by our EventGPT. Our EventGPT is capable of produc￾ing high-quality text descriptions for open-set instance segmenta￾tion tasks. gence. Instance Segmentation. To validate EventGPT’s scal￾ability on complex…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Event-Priori-Based Vision-Language Model for Efficient Visual Understanding

    cs.CV 2025-06 conditional novelty 6.0 of 10

    EP-VLM uses event-camera motion data to sparsify image patches before a vision-language model processes them, cutting FLOPs by about half with a small accuracy drop.

Reference graph

Works this paper leans on

51 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [48]

    E- clip: Towards label-efficient event-based open-world under- standing by clip

    Jiazhou Zhou, Xu Zheng, Yuanhuiyi Lyu, and Lin Wang. E- clip: Towards label-efficient event-based open-world under- standing by clip. arXiv e-prints, pages arXiv–2308, 2023. 2, 3

    work page 2023

  2. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NIPS, 35: 23716–23736, 2022. 3

    work page 2022

  3. [2]

    The (r) evolution of multi- modal large language models: A survey

    Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Marcella Cornia, and Rita Cucchiara. The (r) evolution of multi- modal large language models: A survey. arXiv preprint arXiv:2402.12451, 2024. 3

    arXiv 2024

  4. [3]

    First Place Solution to the CVPR'2023 AQTC Challenge: A Function-Interaction Centric Approach with Spatiotemporal Visual-Language Alignment

    Tom Tongjia Chen, Hongshan Yu, Zhengeng Yang, Ming Li, Zechuan Li, Jingwen Wang, Wei Miao, Wei Sun, and Chen Chen. First place solution to the cvpr’2023 aqtc challenge: A function-interaction centric approach with spatiotemporal visual-language alignment. arXiv preprint arXiv:2306.13380, 2023. 3

    work page Pith review arXiv 2023

  5. [4]

    Dress: Instructing large vision-language models to align and interact with humans via natural lan- guage feedback

    Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. Dress: Instructing large vision-language models to align and interact with humans via natural lan- guage feedback. In CVPR, pages 14239–14250, 2024. 3

    work page 2024

  6. [5]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In CVPR, pages 24185–24198, 2024. 6

    work page 2024

  7. [6]

    Reproducible scal- ing laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuh- mann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scal- ing laws for contrastive language-image learning. In CVPR, pages 2818–2829, 2023. 2

    work page 2023

  8. [7]

    Event-based vision: A survey

    Guillermo Gallego, Tobi Delbr ¨uck, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, J ¨org Conradt, Kostas Daniilidis, et al. Event-based vision: A survey. IEEE TPAMI, 44(1):154–180,

    work page

Show all 51 references
  1. [8]

    Low-latency auto- motive vision with event cameras

    Daniel Gehrig and Davide Scaramuzza. Low-latency auto- motive vision with event cameras. Nature, 629(8014):1034– 1040, 2024. 1

    2024

  2. [9]

    Eklt: Asynchronous photometric feature tracking using events and frames

    Daniel Gehrig, Henri Rebecq, Guillermo Gallego, and Da- vide Scaramuzza. Eklt: Asynchronous photometric feature tracking using events and frames. IJCV, 128(3):601–618,

  3. [10]

    Recurrent vision transformers for object detection with event cameras

    Mathias Gehrig and Davide Scaramuzza. Recurrent vision transformers for object detection with event cameras. In CVPR, pages 13884–13893, 2023. 1

    2023

  4. [11]

    Dsec: A stereo event camera dataset for driving scenarios

    Mathias Gehrig, Willem Aarents, Daniel Gehrig, and Davide Scaramuzza. Dsec: A stereo event camera dataset for driving scenarios. IEEE RAL, 6(3):4947–4954, 2021. 4, 5

    2021

  5. [12]

    Event-based simultaneous localization and mapping: A com- prehensive survey

    Kunping Huang, Sen Zhang, Jing Zhang, and Dacheng Tao. Event-based simultaneous localization and mapping: A com- prehensive survey. arXiv preprint arXiv:2304.09793, 2023. 1

    2023 arXiv

  6. [13]

    Img-diff: Contrastive data synthesis for multimodal large language models

    Qirui Jiao, Daoyuan Chen, Yilun Huang, Yaliang Li, and Ying Shen. Img-diff: Contrastive data synthesis for multimodal large language models. arXiv preprint arXiv:2408.04594, 2024. 3

    2024 arXiv

  7. [14]

    Real-time 3d reconstruction and 6-dof tracking with an event camera

    Hanme Kim, Stefan Leutenegger, and Andrew J Davison. Real-time 3d reconstruction and 6-dof tracking with an event camera. In ECCV, pages 349–364. Springer, 2016. 1

    2016

  8. [15]

    N-imagenet: Towards robust, fine-grained object recognition with event cameras

    Junho Kim, Jaehyeok Bae, Gangin Park, Dongsu Zhang, and Young Min Kim. N-imagenet: Towards robust, fine-grained object recognition with event cameras. InICCV, pages 2146– 2156, 2021. 3, 5

    2021

  9. [16]

    Sodformer: Streaming object detection with transformer using events and frames

    Dianze Li, Yonghong Tian, and Jianing Li. Sodformer: Streaming object detection with transformer using events and frames. IEEE TPAMI, 45(11):14020–14037, 2023. 1

    2023

  10. [17]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, pages 12888–12900. PMLR, 2022. 3

    2022

  11. [18]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pages 19730–19742. PMLR, 2023. 3

    2023

  12. [19]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In CVPR, pages 26689–26699, 2024. 3

    2024

  13. [20]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, pages 26296–26306, 2024. 1

    2024

  14. [21]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NIPS, 36, 2024. 1, 3, 6, 7

    2024

  15. [22]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 8

    2023 arXiv

  16. [23]

    Deepseek-vl: towards real-world vision- language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision- language understanding. arXiv preprint arXiv:2403.05525,

    arXiv

  17. [24]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 6

    2023 arXiv

  18. [25]

    Data-driven feature tracking for event cameras

    Nico Messikommer, Carter Fang, Mathias Gehrig, and Da- vide Scaramuzza. Data-driven feature tracking for event cameras. In CVPR, pages 5642–5651, 2023. 1

    2023

  19. [26]

    Esl: Event-based structured light

    Manasi Muglikar, Guillermo Gallego, and Davide Scara- muzza. Esl: Event-based structured light. In 3DV, pages 1165–1174. IEEE, 2021. 1

    2021

  20. [27]

    Cross-modal attention congruence regularization for vision-language relation align- ment

    Rohan Pandey, Rulin Shao, Paul Pu Liang, Ruslan Salakhut- dinov, and Louis-Philippe Morency. Cross-modal attention congruence regularization for vision-language relation align- ment. arXiv preprint arXiv:2212.10549, 2022. 3

    2022 arXiv

  21. [28]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, pages 8748–8763. PMLR, 2021. 2

    2021

  22. [29]

    Emvs: Event-based multi-view stereo—3d 9 reconstruction with an event camera in real-time

    Henri Rebecq, Guillermo Gallego, Elias Mueggler, and Da- vide Scaramuzza. Emvs: Event-based multi-view stereo—3d 9 reconstruction with an event camera in real-time. IJCV, 126 (12):1394–1414, 2018. 1

    2018

  23. [30]

    Events-to-video: Bringing modern computer vision to event cameras

    Henri Rebecq, Ren ´e Ranftl, Vladlen Koltun, and Davide Scaramuzza. Events-to-video: Bringing modern computer vision to event cameras. CVPR, 2019. 4, 5

    2019

  24. [31]

    Grounded sam: Assembling open-world models for diverse visual tasks

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159 ,

    arXiv

  25. [32]

    Aligning and prompting everything all at once for univer- sal visual perception

    Yunhang Shen, Chaoyou Fu, Peixian Chen, Mengdan Zhang, Ke Li, Xing Sun, Yunsheng Wu, Shaohui Lin, and Rongrong Ji. Aligning and prompting everything all at once for univer- sal visual perception. In CVPR, pages 13193–13203, 2024. 1, 3

    2024

  26. [33]

    Blinktrack: Feature tracking over 100 fps via events and im- ages

    Yichen Shen, Yijin Li, Shuo Chen, Guanglin Li, Zhaoyang Huang, Hujun Bao, Zhaopeng Cui, and Guofeng Zhang. Blinktrack: Feature tracking over 100 fps via events and im- ages. arXiv preprint arXiv:2409.17981, 2024. 1

    2024 arXiv

  27. [34]

    Flava: A foundational language and vision alignment model

    Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guil- laume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In CVPR, pages 15638–15650, 2022. 3

    2022

  28. [35]

    Cloud-device collaborative learning for multimodal large language models

    Guanqun Wang, Jiaming Liu, Chenxuan Li, Yuan Zhang, Junpeng Ma, Xinyu Wei, Kevin Zhang, Maurice Chong, Renrui Zhang, Yijiang Liu, et al. Cloud-device collaborative learning for multimodal large language models. In CVPR, pages 12646–12655, 2024. 3

    2024

  29. [36]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 3, 6

    2024 arXiv

  30. [37]

    Next-gpt: Any-to-any multimodal llm

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519, 2023. 3

    2023 arXiv

  31. [38]

    Eventclip: Adapting clip for event-based object recognition

    Ziyi Wu, Xudong Liu, and Igor Gilitschenski. Eventclip: Adapting clip for event-based object recognition. arXiv preprint arXiv:2306.06354, 2023. 2, 3

    2023 arXiv

  32. [39]

    Leod: Label-efficient object detection for event cameras

    Ziyi Wu, Mathias Gehrig, Qing Lyu, Xudong Liu, and Igor Gilitschenski. Leod: Label-efficient object detection for event cameras. In CVPR, pages 16933–16943, 2024. 1

    2024

  33. [40]

    xgen-mm (blip-3): A family of open large multimodal models

    Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yu- tong Dai, Michael S Ryoo, et al. xgen-mm (blip-3): A family of open large multimodal models. arXiv preprint arXiv:2408.08872, 2024. 1

    2024

  34. [41]

    A survey on multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549, 2023. 3

    2023 arXiv

  35. [42]

    Eventps: Real-time photometric stereo using an event camera

    Bohan Yu, Jieji Ren, Jin Han, Feishi Wang, Jinxiu Liang, and Boxin Shi. Eventps: Real-time photometric stereo using an event camera. In CVPR, pages 9602–9611, 2024. 1

    2024

  36. [43]

    Anygpt: Unified multimodal llm with dis- crete sequence modeling

    Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. Anygpt: Unified multimodal llm with dis- crete sequence modeling. arXiv preprint arXiv:2402.12226,

    arXiv

  37. [44]

    Mm-llms: Recent ad- vances in multimodal large language models

    Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. Mm-llms: Recent ad- vances in multimodal large language models. arXiv preprint arXiv:2401.13601, 2024. 3

    2024 arXiv

  38. [45]

    Spiking transform- ers for event-based single object tracking

    Jiqing Zhang, Bo Dong, Haiwei Zhang, Jianchuan Ding, Fe- lix Heide, Baocai Yin, and Xin Yang. Spiking transform- ers for event-based single object tracking. In CVPR, pages 8801–8810, 2022. 1

    2022

  39. [46]

    Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding

    Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng Yan. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding. arXiv preprint arXiv:2406.19389, 2024. 3

    2024 arXiv

  40. [47]

    Deep learning for event-based vision: A comprehensive survey and bench- marks

    Xu Zheng, Yexin Liu, Yunfan Lu, Tongyan Hua, Tianbo Pan, Weiming Zhang, Dacheng Tao, and Lin Wang. Deep learning for event-based vision: A comprehensive survey and bench- marks. arxiv 2023. arXiv preprint arXiv:2302.08890, 2020. 1

    2023 arXiv

  41. [49]

    Ex- act: Language-guided conceptual reasoning and uncertainty estimation for event-based action recognition and more

    Jiazhou Zhou, Xu Zheng, Yuanhuiyi Lyu, and Lin Wang. Ex- act: Language-guided conceptual reasoning and uncertainty estimation for event-based action recognition and more. In CVPR, pages 18633–18643, 2024. 3

    2024

  42. [50]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1, 3

    2023 arXiv

  43. [51]

    Vl-gpt: A generative pre-trained transformer for vision and language understanding and generation

    Jinguo Zhu, Xiaohan Ding, Yixiao Ge, Yuying Ge, Sijie Zhao, Hengshuang Zhao, Xiaohua Wang, and Ying Shan. Vl-gpt: A generative pre-trained transformer for vision and language understanding and generation. arXiv preprint arXiv:2312.09251, 2023. 3 10

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.