Pith. sign in

REVIEW 4 major objections 5 minor 71 references

Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MDST++ claims that converting video frames into synthetic events and processing them with spiking transformers removes background scene bias and improves audio-visual zero-shot classification on VGGSound, UCF101, and ActivityNet.

desk verdict Coherent architecture, but the EGM threshold ablation is internally inconsistent and the abstract overstates gains; send to review with a demand for code and corrected tables. read the letter →

arxiv 2505.19938 v1 pith:BZUEX7FX submitted 2025-05-26 cs.CV

Wenrui Li , Penghong Wang , Xingtao Wang , Wangmeng Zuo , Xiaopeng Fan , Yonghong Tian This is my paper
classification cs.CV
keywords audio-visualzero-shotlearninggeneralizedsyntheticeventsspikingneuralnetworksmotiondecouplingbackgroundscenebiastransformerdynamicLIFthreshold
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that background scene bias, rather than missing information, is the main obstacle to audio-visual zero-shot learning, and that converting RGB frames into sparse synthetic events removes that bias while preserving the motion details needed to recognize unseen classes. To do this it proposes MDST++, a dual-stream spiking transformer in which one stream extracts context from audio and video with a recurrent joint learning unit and the other extracts motion from synthetic event streams with spiking neural networks. A dynamic threshold adjusts LIF neuron firing according to how much scene semantics and motion are present, and a multi-stage timestep shrinkage cuts inference cost while retaining multi-scale temporal features. The paper reports that MDST++ outperforms previous methods on VGGSound, UCF101, and ActivityNet, and that adding motion and multi-timescale information raises HM and ZSL accuracy by 26.2% and 39.9%. A sympathetic reader would care because this offers a concrete mechanism for making zero-shot video classifiers rely on what an action is rather than where it happens.

What carries the argument

The load-bearing object is the Event Generation Model together with the spiking stream that consumes it. The EGM triggers an event when the log-brightness change at a pixel exceeds a contrast threshold $C$, written as $\Delta L(\mathbf{u},t_k)=L(\mathbf{u},t_k)-L(\mathbf{u},t_k-\Delta t_k)\ge p_k C$, producing an event stream $E_v=\{e_k\}_{k=1}^N$ of pixel locations, timestamps, and polarities. A Discrepancy Analysis Block converts the difference between visual events and audio features into audio motion cues. The spiking stream uses Leaky Integrate-and-Fire neurons whose thresholds are updated each timestep as $V^t_{th}=(\varphi+\omega)V^{t-1}_{th}$, where $\varphi$ measures scene semantic richness and $\omega$ measures motion entropy, so firing is suppressed when the scene is semantically rich or the motion is noisy. MDST++ wraps this stream in spiking self-attention and divides it into stages with shrinking timesteps $T_1>T_2>\dots>T_m$, so later layers run faster while preserving temporal information.

What would settle it

Train MDST++ on UCF-GZSL with the EGM threshold $C$ fixed by a dataset-independent rule, such as the median contrast across training frames, and compare the result with the tuned thresholds in Table VII. If accuracy falls back to the level of the no-MIM ablation, the reported gains come from threshold tuning rather than from event-based motion decoupling; if it holds, the claim that events themselves remove background bias is supported.

Watch

Extended reading notes

Core claim

On its own terms, the central claim is that actions can be represented for zero-shot transfer by decoupling what the scene is from what moves, with the motion side encoded as synthetic events. The Event Generation Model converts RGB frames to event streams using a log-intensity threshold, the events are processed by SNNs, and the LIF neuron thresholds are adjusted dynamically using the entropy of semantic context and event motion. A Discrepancy Analysis Block derives audio motion cues from the difference between visual events and audio features, and MDST++ adds spiking self-attention and a multi-stage timestep shrinkage strategy to capture long-range and multi-scale temporal dependencies. The paper reports consistent gains over earlier methods: on UCF-GZSL, MDST++ reaches HM 33.38 and ZSL 33.81 versus 31.36 and 31.53 for its predecessor MDST, and it also improves the best harmonic means on VGGSound-GZSL and ActivityNet-GZSL. These results are interpreted as evidence that the event representation suppresses background scene information while retaining fine motion cues such as player, ball, and audience movement.

Load-bearing premise

The load-bearing premise is that a single log-intensity threshold $C$ in the Event Generation Model turns RGB frames into event streams that keep enough motion detail for zero-shot transfer while suppressing the background scene information that biases classifiers; if that event representation cannot carry the semantic content needed to match unseen class labels, the motion-decoupling contribution collapses.

Editorial extensions

If this is right

  • If the event stream suppresses background scene information as claimed, classifiers built this way should be less fooled by videos whose setting matches a different action than the one being performed.
  • The multi-stage timestep shrinkage lets later spiking layers run at fewer timesteps, so deployment latency drops while multi-scale temporal information is retained.
  • The dynamic LIF threshold makes performance largely insensitive to the choice of EGM threshold $C$, because the network adapts its firing rate to the amount of scene semantics and motion noise present.
  • Both streams are necessary: removing the motion stream (W/o MIM) drops unseen-class accuracy, while the full audio-visual model beats either the audio-only or visual-only variant on UCF-GZSL.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run is to feed the same EGM event streams to a conventional non-spiking transformer under identical fusion and losses; the paper's MLP-versus-SNN ablation suggests the spiking stream helps, but a modern transformer baseline would isolate whether the gain comes from events or from spiking coding.
  • If event-based motion decoupling is the right mechanism, the same EGM preprocessing could be applied to other video tasks with scene-action confounds, such as compositional action recognition or text-to-video retrieval where scene and action labels are separated.
  • The EGM threshold $C$ is tuned per dataset in Table VII, so the paper's claim implies that the dynamic threshold block absorbs much of that sensitivity; a stronger statement would be to learn $C$ itself as a parameter and remove the free choice entirely.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MDST++, a dual-stream spiking transformer for audio-visual zero-shot learning (ZSL) and generalized ZSL. The architecture decouples contextual semantic information from dynamic motion information by converting RGB frames into synthetic events via an Event Generation Model (EGM), processing events with spiking neural networks and a spiking self-attention module, and introducing a multi-stage timestep shrinkage strategy plus dynamic LIF neuron thresholds. The paper reports state-of-the-art results on VGGSound, UCF101, and ActivityNet, and includes component ablations, parameter analysis, and qualitative visualizations.

Significance. If the results are reliable, the paper addresses a genuine problem—background scene bias in audio-visual ZSL—with a novel combination of synthetic events, spiking attention, and dynamic thresholds, while also providing an architectural extension over the authors' prior ACM MM 2023 work. The paper does not ship code or machine-checked proofs, and the empirical evidence as presented is not reproducible from the text. The significance of the claimed contribution is high, but the reported numbers contain internal inconsistencies that currently prevent the paper from supporting its conclusions.

major comments (4)
  1. [Table VII and §IV-D6] The four threshold rows in Table VII are numerically identical, to four decimal places, to the first four rows of Table VI (the loss-function ablation). This cannot arise from two independent ablations, and it invalidates the EGM threshold study. The text in §IV-D6 states that 'little difference in performance between different EGM thresholds' exists, but the HM values in those rows range from 13.14 to 24.29, and the MDST++ row reaches 33.38, which is more than a factor of two larger. The authors must clarify how these tables were produced and correct or rerun the experiments.
  2. [§III-B1, Eq. (5)] The event generation condition is written as ΔL(u,t_k) ≥ p_k C, with p_k ∈ {−1,1}. For p_k = −1, this inequality is triggered whenever ΔL ≥ −C, which includes nearly all positive brightness changes, so the condition does not implement the standard contrast-event model described in the text. The correct condition should be p_k · ΔL ≥ C, or equivalently ΔL ≥ C for positive polarity and ΔL ≤ −C for negative polarity. As written, Eq. (5) leaves the EGM's behavior underspecified, which is a load-bearing flaw because the method's core motion-capture mechanism relies on this formula.
  3. [Abstract and Table I] The abstract claims that incorporating motion and multi-timescale information improves HM and ZSL accuracy by 26.2% and 39.9%. The relative improvements in Table I for MDST++ versus MDST on UCF-GZSL are HM (33.38 vs 31.36, +6.4%) and ZSL (33.81 vs 31.53, +7.2%), and no other table row reproduces the abstract's percentages. The paper must explicitly state which comparison yields 26.2% and 39.9%, or correct the abstract.
  4. [Table I and §IV-D6 / Fig. 9] The main comparison in Table I is reported without error bars, number of seeds, or statistical significance tests. Additionally, hyperparameters such as the EGM threshold (Table VII) and the average timestep (Fig. 9) are selected based on target-dataset accuracy. Because the evaluation protocol appears to use the same data for hyperparameter selection and final reporting, the 'consistent superiority' claim is not statistically supported, especially for margins of only ~1.5–2 points on some metrics. The authors should report multi-run statistics and use a validation split for model selection.
minor comments (5)
  1. [Table VI and §III-G] Table VI includes a row labeled 'W/o L_n+L_c', but the loss L_c is never defined; the defined losses are L_n, L_p, and L_r. This makes the loss ablation uninterpretable as written and should be corrected.
  2. [§IV-B] The running text reports MDST ZSL scores of 29.34 (UCF-GZSL) and 11.94 (ActivityNet-GZSL), which do not match Table I's values of 31.53 and 12.55, respectively. These inconsistencies should be reconciled.
  3. [§V] The conclusion notes that removing the MIM causes a 'substantial performance drop' and attributes this to 'inconsistencies in data distribution between the SNN and RJLU outputs,' which is an acknowledged limitation of the current architecture. This should be discussed more prominently, as it qualifies the decoupling claim.
  4. [Figure 9] The axis labels and legend text in Figure 9 appear corrupted (e.g., strings like '/uni0000002b/uni00000030'), making the figure unreadable. The authors should regenerate the figure with proper text.
  5. [§III-B2, Eq. (6)] The Discrepancy Analysis Block defines E_a = a_t + (1 − exp(−||E_v − a_t||_2^2 / β_a)), which is not a discrepancy in the usual sense but rather a gated addition. The naming and formulation should be justified or clarified.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation is present: the benchmark claims rest on external comparisons, and the duplicated Table VII is an integrity concern rather than a reduction.

full rationale

The paper's load-bearing claim is empirical: Table I reports that MDST++ achieves higher HM/ZSL than prior methods on VGGSound, UCF101, and ActivityNet under fixed splits. That claim is not derived from the method's equations; it is a measured comparison against external baselines and can be falsified independently. The architectural components (EGM Eq. 5, RJLU Eqs. 1-3, DAB Eq. 6, SpikeFormer Eqs. 7-8, timestep shrinkage Eq. 9, dynamic threshold Eq. 10) are presented as design choices and are validated by ablations, not by construction. The paper does not invoke a uniqueness theorem or any external result that it itself must establish; its citations to prior MDST [10] and related spiking-transformer work are used for provenance and baseline comparison, not to define away the reported gains. The most serious evidentiary problem is Table VII: the 'Different Threshold in EGM' rows (28.71/8.52/13.14/10.11, etc.) are numerically identical to the loss-ablation rows of Table VI, and the text's assertion that different thresholds make 'little difference' is contradicted by the table's own spread; this is a data-integrity issue that undermines the threshold-robustness ablation, but it does not make the benchmark result equivalent to the method's inputs. Hyperparameter selection on target-data accuracy (Table VII, Fig. 9) similarly raises a generalization-risk concern, not a definitional circularity. No step in the claimed derivation chain reduces by construction to its own inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central claim depends on a set of domain assumptions about event generation, pre-trained features, and an ad hoc definition of audio motion. The key free parameters are the EGM threshold and the timestep schedule, both tuned on target datasets, which weakens the claim that the architecture is generally superior rather than well-tuned for these benchmarks.

free parameters (4)
  • EGM threshold C = reported as '-8< x< 8' to '-15< x< 15' in Table VII
    Controls which brightness changes become events; tuned per dataset without a principled criterion, and the sign convention conflicts with Eq. (5).
  • Average timestep T_avg = tested from 5 to 12 (Fig. 9)
    The multi-stage schedule is selected by accuracy on the UCF-GZSL dataset; exact T_1...T_m values are not given.
  • beta_a in DAB = learnable, no stated range
    Scales the discrepancy between visual events and audio features in Eq. (6); no initialization or constraints are reported.
  • Dynamic threshold factors = phi in (0,1), omega unconstrained
    Eq. (10) multiplies previous threshold by (phi+omega); if omega is negative, thresholds can become negative, yet no clipping is described.
assumptions (4)
  • standard math LIF neuron dynamics and surrogate gradient training are valid for this task
    Invoked in Section III-C and II-C; standard in SNN literature.
  • domain assumption SeLaVi pre-trained features provide sufficient audio-visual semantics
    Section III-A1 uses SeLaVi [55] as frozen feature extractor; no analysis of failure cases from these features.
  • domain assumption Synthetic events from EGM faithfully approximate event-camera data
    Section III-B1 relies on event generation model adapted from ESim [41]; no validation on real events.
  • ad hoc to paper Audio motion can be modeled as the discrepancy between visual events and audio features
    Eq. (6) defines audio motion without external evidence that this quantity corresponds to motion in sound.
invented entities (1)
  • audio motion cues (DAB output)
    purpose: To give the audio stream a motion-like representation for fusion with visual events
    The paper postulates that the difference between visual event features and audio features captures audio motion, but provides no external or physical validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning." pith.science (2026). https://pith.science/paper/BZUEX7FX

@misc{pith2026250519938,
  author       = {Pith},
  title        = {Pith review of: Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BZUEX7FX}},
  note         = {Machine review of arXiv:2505.19938}
}
read the original abstract

Audio-visual zero-shot learning (ZSL) has been extensively researched for its capability to classify video data from unseen classes during training. Nevertheless, current methodologies often struggle with background scene biases and inadequate motion detail. This paper proposes a novel dual-stream Multi-Timescale Motion-Decoupled Spiking Transformer (MDST++), which decouples contextual semantic information and sparse dynamic motion information. The recurrent joint learning unit is proposed to extract contextual semantic information and capture joint knowledge across various modalities to understand the environment of actions. By converting RGB images to events, our method captures motion information more accurately and mitigates background scene biases. Moreover, we introduce a discrepancy analysis block to model audio motion information. To enhance the robustness of SNNs in extracting temporal and motion cues, we dynamically adjust the threshold of Leaky Integrate-and-Fire neurons based on global motion and contextual semantic information. Our experiments validate the effectiveness of MDST++, demonstrating their consistent superiority over state-of-the-art methods on mainstream benchmarks. Additionally, incorporating motion and multi-timescale information significantly improves HM and ZSL accuracy by 26.2\% and 39.9\%.

Figures

Figures reproduced from arXiv: 2505.19938 by the authors.

Figure 1
Figure 1. To mitigate background bias and highlight the differences among [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The MDST architecture combines visual, audio, and textual features (represented by blue, green, and pink lines, respectively) using a two-stream [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. An illustration of a LIF neuron. The membrane potential [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Main differences between MDST and MDST++. In MDST, a simple SNN is built using three linear layers with LIF neurons. In MDST++, self-attention [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: The illustration of dynamic threshold block, which adaptively modifies [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The illustration of the fusion block effectively integrates the semantic [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: The comparison of different spiking thresholds on UCF-GZSL. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 9
Figure 9. Figure 9: Ablation study on different average timestep of MDST and MDST++ [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Qualitative comparison of ablation study. The correctly matched [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

71 extracted references · 63 canonical work pages

  1. [1]

    Contrastive masked autoencoders are stronger vision learn- ers,

    Z. Huang, X. Jin, C. Lu, Q. Hou, M.-M. Cheng, D. Fu, X. Shen, and J. Feng, “Contrastive masked autoencoders are stronger vision learn- ers,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 4, pp. 2506–2517, 2024

    work page 2024

  2. [2]

    Pyramid constrained self-attention network for fast video salient object detection,

    Y . Gu, L. Wang, Z. Wang, Y . Liu, M.-M. Cheng, and S.-P. Lu, “Pyramid constrained self-attention network for fast video salient object detection,” vol. 34, pp. 10 869–10 876, Apr. 2020

    work page 2020

  3. [3]

    Cap4video: What can auxiliary captions do for text-video retrieval?

    W. Wu, H. Luo, B. Fang, J. Wang, and W. Ouyang, “Cap4video: What can auxiliary captions do for text-video retrieval?” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, pp. 10 704–10 713

    work page 2023

  4. [4]

    Isomer: Isomerous transformer for zero-shot video object segmentation,

    Y . Yuan, Y . Wang, L. Wang, X. Zhao, H. Lu, Y . Wang, W. Su, and L. Zhang, “Isomer: Isomerous transformer for zero-shot video object segmentation,” inProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023, pp. 966–976

    work page 2023

  5. [5]

    Avgzslnet: Audio-visual generalized zero-shot learning by reconstructing label fea- tures from multi-modal embeddings,

    P. Mazumder, P. Singh, K. K. Parida, and V . P. Namboodiri, “Avgzslnet: Audio-visual generalized zero-shot learning by reconstructing label fea- tures from multi-modal embeddings,” inProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2021

    work page 2021

  6. [6]

    Audio-visual generalised zero-shot learning with cross-modal attention and language,

    O.-B. Mercea, L. Riesch, A. Koepke, and Z. Akata, “Audio-visual generalised zero-shot learning with cross-modal attention and language,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    work page 2022

  7. [7]

    Temporal and cross-modal attention for audio-visual zero-shot learning,

    O.-B. Mercea, T. Hummel, A. Koepke, and Z. Akata, “Temporal and cross-modal attention for audio-visual zero-shot learning,”European Conference on Computer Vision (ECCV), 2022

    work page 2022

  8. [8]

    Enhancing unsupervised video representation learning by decoupling the scene and the motion,

    J. Wang, Y . Gao, K. Li, J. Hu, X. Jiang, X. Guo, R. Ji, and X. Sun, “Enhancing unsupervised video representation learning by decoupling the scene and the motion,” inProceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2021

    work page 2021

Show all 71 references
  1. [9]

    Spiking tucker fusion trans- former for audio-visual zero-shot learning,

    W. Li, P. Wang, R. Xiong, and X. Fan, “Spiking tucker fusion trans- former for audio-visual zero-shot learning,”IEEE Transactions on Image Processing, vol. 33, pp. 4840–4852, 2024

    2024

  2. [10]

    Motion- decoupled spiking transformer for audio-visual zero-shot learning,

    W. Li, X.-L. Zhao, Z. Ma, X. Wang, X. Fan, and Y . Tian, “Motion- decoupled spiking transformer for audio-visual zero-shot learning,” in Proceedings of the 31st ACM International Conference on Multimedia, 2023, p. 3994–4002

    2023

  3. [11]

    Conv2former: A simple transformer-style convnet for visual recognition,

    Q. Hou, C.-Z. Lu, M.-M. Cheng, and J. Feng, “Conv2former: A simple transformer-style convnet for visual recognition,”IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1–10, 2024

    2024

  4. [12]

    Bidi- rectional cross-modal knowledge exploration for video recognition with pre-trained vision-language models,

    W. Wu, X. Wang, H. Luo, J. Wang, Y . Yang, and W. Ouyang, “Bidi- rectional cross-modal knowledge exploration for video recognition with pre-trained vision-language models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, p...

    2023

  5. [13]

    Box2mask: Box-supervised instance segmentation via level-set evolution,

    W. Li, W. Liu, J. Zhu, M. Cui, R. Y . X. Hua, and L. Zhang, “Box2mask: Box-supervised instance segmentation via level-set evolution,”IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1–17, 2024

    2024

  6. [14]

    The devil is in the crack orientation: A new perspective for crack detection,

    Z. Chen, J. Zhang, Z. Lai, G. Zhu, Z. Liu, J. Chen, and J. Li, “The devil is in the crack orientation: A new perspective for crack detection,” inProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023, pp. 6653–6663. 14

    2023

  7. [15]

    Offline and online optical flow enhancement for deep video compression,

    C. Tang, X. Sheng, Z. Li, H. Zhang, L. Li, and D. Liu, “Offline and online optical flow enhancement for deep video compression,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 6, 2024, pp. 5118–5126

    2024

  8. [16]

    Ustc-td: A test dataset and benchmark for image and video coding in 2020s,

    Z. Li, J. Liao, C. Tang, H. Zhang, Y . Li, Y . Bian, X. Sheng, X. Feng, Y . Li, C. Gaoet al., “Ustc-td: A test dataset and benchmark for image and video coding in 2020s,”IEEE Transactions on Multimedia, 2024

    2024

  9. [17]

    Object segmentation- assisted inter prediction for versatile video coding,

    Z. Li, Z. Yuan, L. Li, D. Liu, X. Tang, and F. Wu, “Object segmentation- assisted inter prediction for versatile video coding,”IEEE Transactions on Broadcasting, 2024

    2024

  10. [18]

    Geometry-aware guided loss for deep crack recognition,

    Z. Chen, J. Zhang, Z. Lai, J. Chen, Z. Liu, and J. Li, “Geometry-aware guided loss for deep crack recognition,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 4703–4712

    2022

  11. [19]

    Latent embedding feedback and discriminative features for zero-shot classifi- cation,

    S. Narayan, A. Gupta, F. S. Khan, C. G. Snoek, and L. Shao, “Latent embedding feedback and discriminative features for zero-shot classifi- cation,” inEuropean Conference on Computer Vision (ECCV), 2020

    2020

  12. [20]

    Gener- alized zero-and few-shot learning via aligned variational autoencoders,

    E. Schonfeld, S. Ebrahimi, S. Sinha, T. Darrell, and Z. Akata, “Gener- alized zero-and few-shot learning via aligned variational autoencoders,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019

    2019

  13. [21]

    Generalized zero- shot learning via synthesized examples,

    V . K. Verma, G. Arora, A. Mishra, and P. Rai, “Generalized zero- shot learning via synthesized examples,” inProceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2018, pp. 4281–4289

    2018

  14. [22]

    Feature generating networks for zero-shot learning,

    Y . Xian, T. Lorenz, B. Schiele, and Z. Akata, “Feature generating networks for zero-shot learning,” inProceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2018

    2018

  15. [23]

    A gener- ative adversarial approach for zero-shot learning from noisy texts,

    Y . Zhu, M. Elhoseiny, B. Liu, X. Peng, and A. Elgammal, “A gener- ative adversarial approach for zero-shot learning from noisy texts,” in Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2018

    2018

  16. [24]

    Bootstrapping audio-visual video segmentation by strengthening audio cues,

    T. Chen, Z. Tan, T. Gong, Q. Chu, Y . Wu, B. Liu, N. Yu, L. Lu, and J. Ye, “Bootstrapping audio-visual video segmentation by strengthening audio cues,”IEEE Transactions on Circuits and Systems for Video Technology, pp. 1–1, 2024

    2024

  17. [25]

    Learning affective features with a hybrid deep model for audio–visual emotion recognition,

    S. Zhang, S. Zhang, T. Huang, W. Gao, and Q. Tian, “Learning affective features with a hybrid deep model for audio–visual emotion recognition,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 28, no. 10, pp. 3030–3043, 2018

    2018

  18. [26]

    Audio-visual temporal forgery de- tection using embedding-level fusion and multi-dimensional contrastive loss,

    M. Liu, J. Wang, X. Qian, and H. Li, “Audio-visual temporal forgery de- tection using embedding-level fusion and multi-dimensional contrastive loss,”IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 8, pp. 6937–6948, 2024

    2024

  19. [27]

    Multimodal imbalance-aware gradient modulation for weakly-supervised audio-visual video parsing,

    J. Fu, J. Gao, B.-K. Bao, and C. Xu, “Multimodal imbalance-aware gradient modulation for weakly-supervised audio-visual video parsing,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 6, pp. 4843–4856, 2024

    2024

  20. [28]

    Question-aware global-local video understanding network for audio-visual question answering,

    Z. Chen, L. Wang, P. Wang, and P. Gao, “Question-aware global-local video understanding network for audio-visual question answering,”IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 5, pp. 4109–4119, 2024

    2024

  21. [29]

    Zero-shot audio classification via semantic embeddings,

    H. Xie and T. Virtanen, “Zero-shot audio classification via semantic embeddings,”IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP), 2021

    2021

  22. [30]

    Activitynet: A large-scale video benchmark for human activity under- standing,

    F. Caba Heilbron, V . Escorcia, B. Ghanem, and J. Carlos Niebles, “Activitynet: A large-scale video benchmark for human activity under- standing,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015

    2015

  23. [31]

    The multivehicle stereo event camera dataset: An event camera dataset for 3d perception,

    A. Z. Zhu, D. Thakur, T. ¨Ozaslan, B. Pfrommer, V . Kumar, and K. Daniilidis, “The multivehicle stereo event camera dataset: An event camera dataset for 3d perception,”IEEE Robotics and Automation Letters, vol. 3, no. 3, pp. 2032–2039, 2018

    2018

  24. [32]

    Events-to-video: Bringing modern computer vision to event cameras,

    Rebecq, Henri and Ranftl, Rene and Koltun, Vladlen and Scaramuzza, Davide, “Events-to-video: Bringing modern computer vision to event cameras,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019

    2019

  25. [33]

    High speed and high dynamic range video with an event cam- era,

    Rebecq, Henri and Ranftl, Ren ´e and Koltun, Vladlen and Scaramuzza, Davide, “High speed and high dynamic range video with an event cam- era,”IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2021

    2021

  26. [34]

    A controlled-delay event camera framework for on-line robotics,

    A. Glover, V . Vasco, and C. Bartolozzi, “A controlled-delay event camera framework for on-line robotics,” in2018 IEEE International Conference on Robotics and Automation (ICRA), 2018, pp. 2178–2183

    2018

  27. [35]

    Exploring event camera-based odome- try for planetary robots,

    F. Mahlknecht, D. Gehrig, J. Nash, F. M. Rockenbauer, B. Morrell, J. Delaune, and D. Scaramuzza, “Exploring event camera-based odome- try for planetary robots,”IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 8651–8658, 2022

    2022

  28. [36]

    Emergent visual sensors for autonomous vehicles,

    Y . Li, J. Moreau, and J. Ibanez-Guzman, “Emergent visual sensors for autonomous vehicles,”IEEE Transactions on Intelligent Transportation Systems, vol. 24, no. 5, pp. 4716–4737, 2023

    2023

  29. [37]

    Dsec: A stereo event camera dataset for driving scenarios,

    M. Gehrig, W. Aarents, D. Gehrig, and D. Scaramuzza, “Dsec: A stereo event camera dataset for driving scenarios,”IEEE Robotics and Automation Letters, vol. 6, no. 3, pp. 4947–4954, 2021

    2021

  30. [38]

    High frame rate video reconstruction based on an event camera,

    L. Pan, R. Hartley, C. Scheerlinck, M. Liu, X. Yu, and Y . Dai, “High frame rate video reconstruction based on an event camera,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 5, pp. 2519–2533, 2022

    2022

  31. [39]

    Eventcap: Monocular 3d capture of high-speed human motions using an event camera,

    L. Xu, W. Xu, V . Golyanik, M. Habermann, L. Fang, and C. Theobalt, “Eventcap: Monocular 3d capture of high-speed human motions using an event camera,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020

    2020

  32. [40]

    Towards a framework for end-to-end control of a simulated vehicle with spiking neural networks,

    J. Kaiser, J. C. Vasquez Tieck, C. Hubschneider, P. Wolf, M. Weber, M. Hoff, A. Friedrich, K. Wojtasik, A. Roennau, R. Kohlhaas, R. Dill- mann, and J. M. Z¨ollner, “Towards a framework for end-to-end control of a simulated vehicle with spiking neural networks,” inIEEE Internat...

    2016

  33. [41]

    Esim: an open event camera simulator,

    H. Rebecq, D. Gehrig, and D. Scaramuzza, “Esim: an open event camera simulator,” inProceedings of The 2nd Conference on Robot Learning, ser. Proceedings of Machine Learning Research (PMLR), 2018

    2018

  34. [42]

    Deep residual learning in spiking neural networks,

    W. Fang, Z. Yu, Y . Chen, T. Huang, T. Masquelier, and Y . Tian, “Deep residual learning in spiking neural networks,” inAdvances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  35. [43]

    Neuron-based spiking transmission and reasoning network for robust image-text retrieval,

    W. Li, Z. Ma, L.-J. Deng, X. Fan, and Y . Tian, “Neuron-based spiking transmission and reasoning network for robust image-text retrieval,” IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2022

    2022

  36. [44]

    Spikemba: Multi-modal spiking saliency mamba for temporal video grounding,

    W. Li, X. Hong, R. Xiong, and X. Fan, “Spikemba: Multi-modal spiking saliency mamba for temporal video grounding,” 2024

    2024

  37. [45]

    Spikformer: When spiking neural network meets transformer,

    Z. Zhou, Y . Zhu, C. He, Y . Wang, S. Yan, Y . Tian, and L. Yuan, “Spikformer: When spiking neural network meets transformer,” 2022. [Online]. Available: https://arxiv.org/abs/2209.15425

    2022 arXiv

  38. [46]

    Learning optical flow from continuous spike streams,

    R. Zhao, R. Xiong, J. Zhao, Z. Yu, X. Fan, and T. Huang, “Learning optical flow from continuous spike streams,” in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022...

    2022

  39. [47]

    Mrdflow: Unsupervised optical flow estimation network with multi-scale recurrent decoder,

    R. Zhao, R. Xiong, Z. Ding, X. Fan, J. Zhang, and T. Huang, “Mrdflow: Unsupervised optical flow estimation network with multi-scale recurrent decoder,”IEEE Transactions on Circuits and Systems for Video Tech- nology, vol. 32, no. 7, pp. 4639–4652, 2022

    2022

  40. [48]

    Progressive tandem learning for pattern recognition with deep spiking neural networks,

    J. Wu, C. Xu, X. Han, D. Zhou, M. Zhang, H. Li, and K. C. Tan, “Progressive tandem learning for pattern recognition with deep spiking neural networks,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 11, pp. 7824–7840, 2022

    2022

  41. [49]

    A hybrid neural coding approach for pattern recognition with spiking neural networks,

    X. Chen, Q. Yang, J. Wu, H. Li, and K. Tan, “A hybrid neural coding approach for pattern recognition with spiking neural networks,”IEEE Transactions on Pattern Analysis Machine Intelligence, vol. 46, no. 05, pp. 3064–3078, 2024

    2024

  42. [50]

    Neuron-based spiking transmission and reasoning network for robust image-text retrieval,

    W. Li, Z. Ma, L.-J. Deng, X. Fan, and Y . Tian, “Neuron-based spiking transmission and reasoning network for robust image-text retrieval,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 7, pp. 3516–3528, 2023

    2023

  43. [51]

    Multi-scale spiking pyramid wireless communication framework for food recognition,

    W. Li, J. Li, M. Ma, X. Hong, and X. Fan, “Multi-scale spiking pyramid wireless communication framework for food recognition,”IEEE Transactions on Multimedia, pp. 1–13, 2024

    2024

  44. [52]

    Modality-fusion spiking transformer network for audio-visual zero-shot learning,

    W. Li, Z. Ma, L.-J. Deng, H. Man, and X. Fan, “Modality-fusion spiking transformer network for audio-visual zero-shot learning,” in2023 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2023, pp. 426–431

    2023

  45. [53]

    Evolving spiking neural network controllers for autonomous robots,

    H. Hagras, A. Pounds-Cornish, M. Colley, V . Callaghan, and G. Clarke, “Evolving spiking neural network controllers for autonomous robots,” inIEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA ’04. 2004, vol. 5, 2004, pp. 4620–4626 V ol.5

    2004

  46. [54]

    A hybrid rein- forcement learning approach with a spiking actor network for efficient robotic arm target reaching,

    K. M. Oikonomou, I. Kansizoglou, and A. Gasteratos, “A hybrid rein- forcement learning approach with a spiking actor network for efficient robotic arm target reaching,”IEEE Robotics and Automation Letters, vol. 8, no. 5, pp. 3007–3014, 2023. 15

    2023

  47. [55]

    Labelling unla- belled videos from scratch with multi-modal self-supervision,

    Y . Asano, M. Patrick, C. Rupprecht, and A. Vedaldi, “Labelling unla- belled videos from scratch with multi-modal self-supervision,”Advances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  48. [56]

    Training deep spiking neural networks using backpropagation,

    J. H. Lee, T. Delbruck, and M. Pfeiffer, “Training deep spiking neural networks using backpropagation,”Frontiers in Neuroscience, 2016

    2016

  49. [57]

    Ucf101: A dataset of 101 human actions classes from videos in the wild,

    K. Soomro, A. R. Zamir, and M. Shah, “Ucf101: A dataset of 101 human actions classes from videos in the wild,”arXiv preprint arXiv:1212.0402, 2012

    2012 arXiv

  50. [58]

    Vggsound: A large-scale audio-visual dataset,

    H. Chen, W. Xie, A. Vedaldi, and A. Zisserman, “Vggsound: A large-scale audio-visual dataset,” inIEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020

    2020

  51. [59]

    Evaluation of out- put embeddings for fine-grained image classification,

    Z. Akata, S. Reed, D. Walter, H. Lee, and B. Schiele, “Evaluation of out- put embeddings for fine-grained image classification,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015

    2015

  52. [60]

    Devise: A deep visual-semantic embedding model,

    A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, and T. Mikolov, “Devise: A deep visual-semantic embedding model,” Advances in Neural Information Processing Systems (NeurIPS), 2013

    2013

  53. [61]

    Attribute proto- type network for zero-shot learning,

    W. Xu, Y . Xian, J. Wang, B. Schiele, and Z. Akata, “Attribute proto- type network for zero-shot learning,”Advances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  54. [62]

    f-vaegan-d2: A feature generating framework for any-shot learning,

    Y . Xian, S. Sharma, B. Schiele, and Z. Akata, “f-vaegan-d2: A feature generating framework for any-shot learning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019

    2019

  55. [63]

    Coordinated joint multimodal embeddings for generalized audio-visual zero-shot classifi- cation and retrieval of videos,

    K. Parida, N. Matiyali, T. Guha, and G. Sharma, “Coordinated joint multimodal embeddings for generalized audio-visual zero-shot classifi- cation and retrieval of videos,” inProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2020

    2020

  56. [64]

    Hyperbolic audio-visual zero-shot learning,

    J. Hong, Z. Hayder, J. Han, P. Fang, M. Harandi, and L. Petersson, “Hyperbolic audio-visual zero-shot learning,” in2023 IEEE/CVF Inter- national Conference on Computer Vision (ICCV), 2023, pp. 7839–7849

    2023

  57. [65]

    Label-embedding for image classification,

    Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid, “Label-embedding for image classification,”IEEE transactions on pattern analysis and machine intelligence, vol. 38, no. 7, pp. 1425–1438, 2015

    2015

  58. [66]

    Coordinated joint multimodal embeddings for generalized audio-visual zero-shot classifi- cation and retrieval of videos,

    K. Parida, N. Matiyali, T. Guha, and G. Sharma, “Coordinated joint multimodal embeddings for generalized audio-visual zero-shot classifi- cation and retrieval of videos,” inProceedings of the IEEE/CVF winter conference on applications of computer vision, 2020, pp. 3251–3260

    2020

  59. [67]

    Learning spatiotemporal features with 3d convolutional networks,

    D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning spatiotemporal features with 3d convolutional networks,” in2015 IEEE International Conference on Computer Vision (ICCV), 2015, pp. 4489– 4497

    2015

  60. [68]

    Large-scale video classification with convolutional neural networks,

    A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei, “Large-scale video classification with convolutional neural networks,” in2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1725–1732

    2014

  61. [69]

    Cnn architectures for large-scale audio classification,

    S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, M. Slaney, R. J. Weiss, and K. Wilson, “Cnn architectures for large-scale audio classification,” in2017 IEEE International Conference on Acoustics, ...

    2017

  62. [70]

    Youtube-8m: A large-scale video classi- fication benchmark,

    S. Abu-El-Haija, N. Kothari, J. Lee, P. Natsev, G. Toderici, B. Varadara- jan, and S. Vijayanarasimhan, “Youtube-8m: A large-scale video classi- fication benchmark,” 2016. Wenrui Lireceived the B.S. degree from the School of Information and Software Engineering, University of ...

    2016

  63. [2020]

    degree with the School of Computer Science, Harbin Institute of Technology, Harbin, China

    He is currently pursuing the Ph.D. degree with the School of Computer Science, Harbin Institute of Technology, Harbin, China. His main research inter- ests include wireless sensor networks, joint source- channel coding, and computer vision. Xingtao Wangobtained his B.S. degree...

    2016

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.