Pith. sign in

REVIEW 3 major objections 4 minor 111 references

SMM Transformer shows that an all-spiking multimodal Transformer can match ANN baselines on vision, captioning, and retrieval tasks while replacing dense softmax attention with channel-wise spike co-activation, cutting estimated attention a

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 23:59 UTC pith:BNJTDNWU

load-bearing objection SMM Transformer is a plausible SNN architecture with an honest energy model, but the captioning numbers look wrong and the missing code/hyperparameters block verification. the 3 major comments →

arxiv 2608.01622 v1 pith:BNJTDNWU submitted 2026-08-03 cs.NE cs.MM

SMM Transformer: Leveraging Spiking Neural Networks for Multimodal Tasks

Xiubo Liang , Jinxing Han , Yuke Li , Haoqi Zhu , Yu Zhao , Hongzhi Wang This is my paper
classification cs.NE cs.MM
keywords spiking neural networksmultimodal learningTransformerenergy-efficient attentionspike-driven token mixingmixture of expertsimage captioningcross-modal retrieval
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper's thesis is that the compute bottleneck of Transformer-style multimodal models—dense pairwise attention—can be replaced by spiking, event-driven computation without giving up accuracy. It builds a full SNN-based Transformer, SMM Transformer, around three redesigned parts: a trainable multi-branch spiking neuron (PLMP), a training rule for it (P-STBP), an attention substitute that operates on binary spikes (SMSA), and a spiking expert router for fusing modalities (SMoE). On ImageNet, ADE20K, COCO captioning, and Flickr30K/COCO retrieval, the model lands within a few points of ANN baselines. Under a standard MAC/AC arithmetic model, SMSA lowers estimated operator-level energy of the attention block by up to 97%, while whole-model profiling gives a more modest but consistent saving. If the estimates hold on real hardware, this is a concrete path toward cheaper vision–language Transformers.

Core claim

The central discovery is that dense softmax attention is not required for competitive multimodal accuracy in an SNN setting. SMSA replaces the N×N token-pair similarity matrix with channel-wise spike co-activation aggregated over tokens, binarization into a spike gate, column-wise masking of value spikes, and a small spiking-MLP self-compensation branch. This keeps the Transformer macro-architecture—residual blocks, patch and word embeddings, and standard pre-training objectives such as image-text contrast, masked language modeling, and image-text matching—while making every internal tensor binary and event-driven. The paper reports that this design reaches 83.2% ImageNet top-1 accuracy and

What carries the argument

PLMP (Parallel LIF with Multistage Learnable Parameters) is the trainable spiking unit: K parallel LIF branches, each with learnable leakage αk and threshold Vth,k, whose internal spikes are summed and passed through a fixed-threshold read-out gate that emits a single binary spike. P-STBP is the tailored spatio-temporal backpropagation rule that pushes gradients through the branch dynamics, the internal spike-count aggregation, the read-out gate, and the learnable leakage and threshold parameters. SMSA (Spiking MLP Self-Attention) is the attention substitute: it computes token-aggregated channel co-activation on binary Q and K spikes, binarizes the result into a gate, masks V column-wise, an

Load-bearing premise

The paper's energy estimate assumes that arithmetic energy, scaled down by the measured fraction of active spikes, is the dominant cost and that memory access and scheduling can be ignored; if those costs dominate on real hardware, the claimed 97% and 21.6% savings will not materialize.

What would settle it

Measure the same SMSA and vanilla softmax attention blocks on a neuromorphic chip or ASIC that reports per-operation energy including memory access and scheduling, at equal accuracy and sequence length; if the measured SMSA-vs-softmax energy ratio is far above 3% (i.e., the saving is well below 97%), the central efficiency claim is falsified. A simpler proxy is to scale sequence length and record end-to-end latency: SMSA should grow roughly linearly while dense attention grows quadratically.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • On ImageNet, SMM-Base reaches 83.2% top-1 accuracy (vs. 83.6% for ViT-Base and 85.2% for BEiT-Base) and 49.8 mIoU on ADE20K, well above the PSSD spiking baseline.
  • On COCO captioning, SMM-Transformer-Large posts BLEU-4 of 45.33 and CIDEr of 128.72, competitive with ANN captioners while carrying an estimated 22.1 mJ per inference.
  • On cross-modal retrieval, SMM-Transformer-Large reaches COCO R@1 of 75.3 (text) and 59.3 (image) and Flickr30K R@1 of 93.9/83.7, close to ALBEF-Base and above several fusion-encoder baselines.
  • The SMSA attention module is estimated to consume about 3% of the operator-level arithmetic energy of vanilla softmax attention (a 97% cut); whole-model FLOPs fall 25.1% and estimated energy per pair falls 21.6%.
  • Ablations show PLMP with K=3 branches and the SMSA self-compensation branch each contribute consistent gains, and shared SMSA before SMoE improves cross-modal alignment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because SMSA avoids the N×N attention matrix, its per-token cost should stay roughly linear in sequence length, so the efficiency advantage over dense softmax attention is predicted to grow on longer multimodal sequences—a direct, testable consequence of the design.
  • Editorial inference: the 97% figure counts arithmetic only and explicitly excludes memory access, scheduling, and hardware-specific effects; a real-chip measurement would likely show a smaller but still positive saving, especially in memory-bound regimes.
  • Editorial inference: the PLMP recipe—learnable leakage and thresholds with a binary coincidence read-out—generalizes beyond Transformers to any deep SNN stack, and P-STBP is the mechanism that makes such trainable neuronal dynamics stable.
  • Editorial inference: the stagewise pre-training with frozen experts suggests a modular route to adding new modalities, such as audio or video, by training only a new expert block while reusing the shared SMSA pathway.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes SMM Transformer, an SNN-based multimodal Transformer with three new components: PLMP, a parallel multi-branch LIF neuron with learnable leakage and thresholds; P-STBP, a spatio-temporal backpropagation scheme for PLMP; SMSA, a spike-driven token-mixing module that avoids dense N×N softmax attention; and SMoE, a spike-based mixture-of-experts for modality-aware fusion. The paper reports competitive accuracy with ANN baselines on COCO and Flickr30K retrieval, COCO captioning, ImageNet classification, and ADE20K segmentation, and claims up to 97% operator-level attention energy reduction and a 21.6% whole-model energy saving under a MAC/AC arithmetic model.

Significance. If the experimental claims held, SMM Transformer would be a meaningful step toward energy-efficient multimodal Transformers: it combines learnable spiking neurons, a sparse attention-inspired mixing mechanism, and modality-expert routing, and it provides controlled ablations for each component. The PLMP/P-STBP formalism is clearly written and the ablation structure gives partial credit to the design choices. However, the central evidence is not currently credible: the captioning results are internally inconsistent with the paper's own ablations and with established ANN baselines, and the energy analysis omits a load-bearing parameter (the number of simulation steps) that can reverse the claimed savings. The paper also does not supply the training details or code needed to check any of the headline numbers. The ideas are worth further study, but the manuscript in its present form does not establish the claimed accuracy-efficiency trade-off.

major comments (3)
  1. [Sec. 4.1.2, Tables 2 and 5] The captioning results are internally inconsistent and implausibly strong. Table 2 reports SMM-Large with BLEU-4=45.33, METEOR=34.25, and ROUGE-L=61.18, while the strongest ANN baseline (PureT) has BLEU-4=41.4, METEOR=30.1, and ROUGE-L=60.4. A spiking model with binary activations beating the best ANN by >10% relative on BLEU-4 is remarkable, yet CIDEr and SPICE (128.72 and 21.87) remain below PureT (136.0 and 24.2), an unusual pattern not explained in the text. More damagingly, the paper's own ablation in Table 5 reports PLMP K=3 achieving BLEU-4=36.60 on captioning, far below the 45.33 in Table 2. No training setting is given that would reconcile these numbers. Because the abstract's accuracy claim rests on this table, the manuscript needs either a rigorous explanation of the discrepancy or corrected experiments.
  2. [Sec. 4.3, Tables 7–9] The energy comparison is incomplete because the SNN simulation step count T is never specified. Table 7 charges vanilla self-attention with one pass of dense MACs (E_MAC multiplied by operation counts), while SMSA is charged E_AC·T·R·ops, where T multiplies every term. If T>1, the SMSA cost grows linearly in T, and the claimed 97% attention-block reduction could disappear or even invert. The text says the comparison is made 'under the same hidden size, sequence length, and simulation-step setting,' but no T value appears anywhere in the paper, and there are no training or inference-time simulation steps reported. The whole-model estimates in Table 9 also explicitly exclude memory access and scheduling overhead, yet Table 2's caption labels the energy column as 'measured power.' The central efficiency claim cannot be assessed without T, a clear statement of the energy model's inclusion/ex
  3. [Sec. 4, Appendix A] The manuscript does not provide enough information to reproduce any of the experiments. Missing are the number of simulation steps T, the specific surrogate function forms (Eq. 22 defines φ_br and φ_ro only symbolically), the model sizes and layer counts for Base and Large variants on each task, the training hyperparameters (optimizer, learning rate, batch size, epochs, warmup, data augmentation, etc.), and the hardware/software environment. Given that the headline accuracy and energy numbers are nonstandard, these details are essential. The authors should release code or, at minimum, a complete configuration table before the results can be trusted.
minor comments (4)
  1. [Sec. 3.3, Eqs. (15)–(16)] Equations (15) and (16) define an REINFORCE-style RL loss and gradient (r(y), greedy decoding score, etc.) but are never connected to SMSA, PLMP, or any objective in the paper. They appear to be an editorial leftover. Either remove them or integrate them into a captioning training subsection.
  2. [Table 2 caption] The caption says 'We report standard captioning metrics and measured power,' but Section 4.3 and the subsequent text clearly describe estimated arithmetic energy excluding memory access and scheduling overhead. The caption should say 'estimated energy under the MAC/AC model.'
  3. [Sec. 4.1.1, Table 1] The caption states that ALBEF uses a fusion encoder to rerank top-k candidates, but the row labels and the comparison text do not clarify whether SMM's retrieval pathway is directly comparable to ALBEF or to the dual-encoder baselines (UNITER, ViLT). A short sentence in the text distinguishing the two retrieval modes would help.
  4. [Figure 1] The qualitative examples are selected to highlight SMM's outputs. While anecdotal evidence is acceptable, the examples should be labeled as cherry-picked; they carry no statistical weight and should not be used in the text to assert that SMM 'provides more accurate descriptions' (Figure 1 caption).

Circularity Check

0 steps flagged

No significant circularity: the energy estimate is explicit post-hoc accounting, and the accuracy claims are external benchmark comparisons.

full rationale

I walked the paper's derivation chain. PLMP (Eqs. 1-4), P-STBP (Eqs. 5-8, 22-24), SMSA (Eqs. 9-14), and SMoE (Eqs. 17-18) are architectural definitions and standard gradient derivations; none derive a target result from its own inputs. The only quantitative efficiency claim is the Sec. 4.3 MAC/AC estimate, which is explicitly described as an arithmetic-cost estimate using measured non-zero spike ratios from the trained model (Table 8). That is a post-hoc energy accounting, not a prediction from fitted parameters, and the paper discloses that memory access, scheduling, and hardware-specific effects are excluded ('This estimate reflects attention-block arithmetic cost only and does not include memory access, scheduling overhead, or hardware-specific kernel effects'). The accuracy results are comparisons against external ANN baselines on COCO, Flickr30K, ImageNet, and ADE20K; they are empirical claims, not reductions. I found no load-bearing self-citations: STBP, Spikformer, Spike-driven Transformer, and other cited works are external. The suspiciously high captioning metrics in Table 2 and the misplaced RL equations (Eqs. 15-16) are reproducibility/quality concerns, not circularity. Therefore the paper is not circular; the appropriate score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 0 invented entities

The central claims rest on hand-chosen architectural hyperparameters (K, initializations, simulation steps) and on the validity of the MAC/AC energy model with measured spike ratios. No new physical entities are proposed. The model's learnable parameters are trained, not treated as free constants.

free parameters (4)
  • K (number of PLMP branches) = 3
    Chosen by hand based on ablation (Table 10, Appendix B); K=3 used as default in all main experiments.
  • Initial leakage factor and thresholds (alpha_init, Vth_init) = alpha=0.5; Vth=0.3/0.6/0.9
    Initializations of learnable leakage factors and thresholds for the K=3 branches, reported in Table 5 and used in all main experiments.
  • SNN simulation steps T = not reported
    T appears in the energy accounting (Table 7) and is required to compute the claimed energy savings, but the value is never given, making the efficiency numbers non-reproducible.
  • Model architecture hyperparameters (Base/Large) = not reported
    Hidden size, number of layers, heads, patch size, and feed-forward dimensions for SMM-Transformer-Base/Large are not specified despite being labeled in Tables 1-3.
axioms (3)
  • domain assumption Surrogate gradients phi_br and phi_ro exist and yield stable training with P-STBP
    P-STBP relies on unspecified surrogate derivatives (Appendix A, Eq. 22); the paper gives no functional form, and training stability is only asserted empirically.
  • domain assumption MAC/AC energy model with EMAC=4.6pJ, EAC=0.9pJ at 45nm is an appropriate proxy for real energy
    Sec. 4.3 uses this standard model to estimate energy; the 97% and 21.6% savings depend on this model and exclude memory/scheduling overhead.
  • domain assumption Measured spike ratios (Table 8) on the trained model generalize to the energy estimate
    Energy estimates scale AC cost by these ratios; ratios are model-specific and could differ on new hardware or inputs.

pith-pipeline@v1.3.0-daily-deepseek · 15047 in / 13556 out tokens · 131702 ms · 2026-08-04T23:59:39.102397+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of SMM Transformer: Leveraging Spiking Neural Networks for Multimodal Tasks." pith.science (2026). https://pith.science/paper/BNJTDNWU

@misc{pith2026260801622,
  author       = {Pith},
  title        = {Pith review of: SMM Transformer: Leveraging Spiking Neural Networks for Multimodal Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BNJTDNWU}},
  note         = {Machine review of arXiv:2608.01622}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Spiking Neural Networks (SNNs) enable event-driven computation with sparse activations, but building multimodal Transformers on SNNs is hindered by unstable training in deep spiking stacks and the mismatch between dense softmax attention and spike-based communication. We propose SMM Transformer, an SNN-based multimodal Transformer framework that combines (i)PLMP, a Parallel LIF with Multistage Learnable Parameters neuron and a tailored P-STBP algorithm for stable deep SNN training, (ii) SMSA, an attention-inspired spike-driven token-mixing module that replaces dense pairwise softmax attention with channel-wise spike co-activation and self-compensation, and (iii)SMoE, a spiking mixture-of-experts module for modality-aware fusion. Across visual and multimodal benchmarks, SMM Transformer achieves competitive accuracy compared to ANN baselines. Under a standard MAC/AC arithmetic model, SMSA reduces the estimated operator-level compute energy of the attention module by up to 97%, while whole-model profiling shows more moderate but consistent efficiency gains.

Figures

Figures reproduced from arXiv: 2608.01622 by Haoqi Zhu, Hongzhi Wang, Jinxing Han, Xiubo Liang, Yuke Li, Yu Zhao.

Figure 1
Figure 1. Figure 1: Comparison of image captioning results from different models in the image shows that our SMM Transformer provides more accurate descriptions. Additionally, it captures color features well. patible with sparse spike communication. Experimental re￾sults demonstrate that SMM Transformer achieves accuracy comparable to traditional ANNs in both visual tasks and multimodal tasks. We also designed ablation experi… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of SMM Transformer. The model parameters are shared across image-text contrastive learning, masked language modeling, and image-text matching pre-training tasks. circumvent spike non-differentiability, such as SpikeProp (Bohte et al., 2000) and STBP (Wu et al., 2018). Subsequent advances extend network depth (e.g., SEW-ResNet (Fang et al., 2021)) and integrate spiking blocks within Trans￾former fr… view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of the proposed components. (a) SMSA first maps Query, Key, and Value into PLMP read-out spikes, then performs spike co-activation, column-wise aggregation, value masking, and self-compensation without constructing a dense N × N attention matrix. (b) P-STBP propagates gradients through both the temporal branch dynamics and the fixed-threshold PLMP read-out gate. 3. Method As summarized in [PI… view at source ↗
Figure 4
Figure 4. Figure 4: Illustration of stagewise pre-training strategy. 3.4. SMoE In this section, we introduce SMoE, a spike-driven mixture￾of-experts framework that dynamically selects modality￾specific experts. SMoE is designed to preserve the repre￾sentation benefit of expert routing while keeping the fusion pathway compatible with sparse spiking computation. Its structure is y = X K k=1 Rk(XA) · Ek(XA), (17) where XA is the… view at source ↗
Figure 5
Figure 5. Figure 5: Training curves for PLMP variants with different temporal parameter initializations [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

111 extracted references · 51 canonical work pages

  1. [1]

    Z., and Shah, M

    Aafaq, N., Mian, A., Liu, W., Gilani, S. Z., and Shah, M. Video description: A survey of methods, datasets, and evaluation metrics. ACM Computing Surveys (CSUR), 52 0 (6): 0 1--37, 2019

    2019

  2. [2]

    Bottom-up and top-down attention for image captioning and visual question answering

    Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 6077--6086, 2018

    2018

  3. [3]

    M., Kok, J

    Bohte, S. M., Kok, J. N., and La Poutr \'e , J. A. Spikeprop: backpropagation for networks of spiking neurons. In ESANN, volume 48, pp.\ 419--424. Bruges, 2000

    2000

  4. [4]

    and Dan, Y

    Caporale, N. and Dan, Y. Spike timing--dependent plasticity: a hebbian learning rule. Annu. Rev. Neurosci., 31: 0 25--46, 2008

    2008

  5. [5]

    Extracting training data from large language models

    Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp.\ 2633--2650, 2021

    2021

  6. [6]

    Deep reinforcement learning with spiking q-learning, 2024

    Chen, D., Peng, P., Huang, T., and Tian, Y. Deep reinforcement learning with spiking q-learning, 2024

    2024

  7. [7]

    and Triggs, B

    Dalal, N. and Triggs, B. Histograms of oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05), volume 1, pp.\ 886--893. Ieee, 2005

    2005

  8. [8]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp.\ 4171--4186, 2019

    2019

  9. [10]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv e-prints, pp.\ arXiv--2407, 2024

    2024

  10. [11]

    Deep residual learning in spiking neural networks

    Fang, W., Yu, Z., Chen, Y., Huang, T., Masquelier, T., and Tian, Y. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems, 34: 0 21056--21069, 2021

    2021

  11. [12]

    Large-scale neuromorphic computing systems

    Furber, S. Large-scale neuromorphic computing systems. Journal of neural engineering, 13 0 (5): 0 051001, 2016

    2016

  12. [13]

    J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y

    Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial networks, 2014

    2014

  13. [14]

    Hebb, D. O. The organization of behavior: A neuropsychological theory. Psychology press, 2005

    2005

  14. [15]

    Image captioning: Transforming objects into words

    Herdade, S., Kappeler, A., Boakye, K., and Soares, J. Image captioning: Transforming objects into words. Advances in neural information processing systems, 32, 2019

    2019

  15. [16]

    Hybrid macro/micro level backpropagation for training deep spiking neural networks

    Jin, Y., Zhang, W., and Li, P. Hybrid macro/micro level backpropagation for training deep spiking neural networks. Advances in neural information processing systems, 31, 2018

    2018

  16. [17]

    Synaptic plasticity dynamics for deep continuous local learning (decolle)

    Kaiser, J., Mostafa, H., and Neftci, E. Synaptic plasticity dynamics for deep continuous local learning (decolle). Frontiers in Neuroscience, 14: 0 515306, 2020

    2020

  17. [18]

    u chemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., G \

    Kasneci, E., Se ler, K., K \"u chemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., G \"u nnemann, S., H \"u llermeier, E., et al. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences, 103: 0 102274, 2023

    2023

  18. [19]

    K., Sain, A., Chowdhury, P

    Koley, S., Bhunia, A. K., Sain, A., Chowdhury, P. N., Xiang, T., and Song, Y.-Z. Text-to-image diffusion models are great sketch-photo matchmakers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 16826--16837, 2024

    2024

  19. [21]

    Multi-head hashing with orthogonal decomposition for cross-modal retrieval

    Liu, W., Li, J., Wu, Z., Xu, J., and Yang, B. Multi-head hashing with orthogonal decomposition for cross-modal retrieval. In International Conference on Multimedia Modeling, pp.\ 170--183. Springer, 2024

    2024

  20. [22]

    Lowe, D. G. Object recognition from local scale-invariant features. In Proceedings of the seventh IEEE international conference on computer vision, volume 2, pp.\ 1150--1157. Ieee, 1999

    1999

  21. [23]

    Physics for neuromorphic computing

    Markovi \'c , D., Mizrahi, A., Querlioz, D., and Grollier, J. Physics for neuromorphic computing. Nature Reviews Physics, 2 0 (9): 0 499--510, 2020

    2020

  22. [24]

    X-linear attention networks for image captioning

    Pan, Y., Yao, T., Li, Y., and Mei, T. X-linear attention networks for image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 10971--10980, 2020

    2020

  23. [26]

    Exploring models and data for image question answering

    Ren, M., Kiros, R., and Zemel, R. Exploring models and data for image question answering. Advances in neural information processing systems, 28, 2015

    2015

  24. [27]

    Towards spike-based machine intelligence with neuromorphic computing

    Roy, K., Jaiswal, A., and Panda, P. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575 0 (7784): 0 607--617, 2019

    2019

  25. [28]

    Going deeper in spiking neural networks: Vgg and residual architectures

    Sengupta, A., Ye, Y., Wang, R., and Roy, K. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in neuroscience, 13: 0 425055, 2019

    2019

  26. [29]

    Knowledge-enhanced dual-stream zero-shot composed image retrieval

    Suo, Y., Ma, F., Zhu, L., and Yang, Y. Knowledge-enhanced dual-stream zero-shot composed image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 26951--26962, 2024

    2024

  27. [30]

    R., Masquelier, T., and Maida, A

    Tavanaei, A., Ghodrati, M., Kheradpisheh, S. R., Masquelier, T., and Maida, A. Deep learning in spiking neural networks. Neural networks, 111: 0 47--63, 2019

    2019

  28. [31]

    T., Malliaras, G., and Salleo, A

    van De Burgt, Y., Melianas, A., Keene, S. T., Malliaras, G., and Salleo, A. Organic electronics for neuromorphic computing. Nature Electronics, 1 0 (7): 0 386--397, 2018

    2018

  29. [32]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  30. [33]

    Wang, T., Li, F., Zhu, L., Li, J., Zhang, Z., and Shen, H. T. Cross-modal retrieval: a systematic review of methods and future directions. Proceedings of the IEEE, 2025

    2025

  31. [34]

    End-to-end transformer based model for image captioning

    Wang, Y., Xu, J., and Sun, Y. End-to-end transformer based model for image captioning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.\ 2585--2594, 2022

    2022

  32. [36]

    Spatio-temporal backpropagation for training high-performance spiking neural networks

    Wu, Y., Deng, L., Li, G., and Shi, L. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience, 12: 0 323875, 2018

    work page 2018

  33. [37]

    Video graph transformer for video question answering

    Xiao, J., Zhou, P., Chua, T.-S., and Yan, S. Video graph transformer for video question answering. In European Conference on Computer Vision, pp.\ 39--58. Springer, 2022

    work page 2022

  34. [38]

    Image-text retrieval with main semantics consistency

    Xie, Y., Wang, Y., Xie, Y., Tan, X., Li, J., Li, X., Peng, W., Tang, M., and Fang, M. Image-text retrieval with main semantics consistency. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp.\ 2629--2638, 2024

    work page 2024

  35. [39]

    Msr-vtt: A large video description dataset for bridging video and language

    Xu, J., Mei, T., Yao, T., and Rui, Y. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 5288--5296, 2016

    work page 2016

  36. [40]

    Show, attend and tell: Neural image caption generation with visual attention

    Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., and Bengio, Y. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pp.\ 2048--2057. PMLR, 2015

    work page 2048

  37. [41]

    Dense captioning with joint inference and visual context

    Yang, L., Tang, K., Yang, J., and Li, L.-J. Dense captioning with joint inference and visual context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 2193--2202, 2017

    work page 2017

  38. [42]

    Stacked attention networks for image question answering

    Yang, Z., He, X., Gao, J., Deng, L., and Smola, A. Stacked attention networks for image question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 21--29, 2016

    work page 2016

  39. [43]

    Spike-driven transformer

    Yao, M., Hu, J., Zhou, Z., Yuan, L., Tian, Y., Xu, B., and Li, G. Spike-driven transformer. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  40. [44]

    Yu, Q., Ma, C., Song, S., Zhang, G., Dang, J., and Tan, K. C. Constructing accurate and efficient deep spiking neural networks with double-threshold and augmented schemes. IEEE Transactions on Neural Networks and Learning Systems, 33 0 (4): 0 1714--1726, 2021

    work page 2021

  41. [45]

    Meacap: Memory-augmented zero-shot image captioning

    Zeng, Z., Xie, Y., Zhang, H., Chen, C., Chen, B., and Wang, Z. Meacap: Memory-augmented zero-shot image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 14100--14110, 2024

    work page 2024

  42. [46]

    and Li, P

    Zhang, W. and Li, P. Spike-train level backpropagation for training deep recurrent spiking neural networks. Advances in neural information processing systems, 32, 2019

    work page 2019

  43. [48]

    Nature , volume=

    Towards spike-based machine intelligence with neuromorphic computing , author=. Nature , volume=. 2019 , publisher=

    work page 2019

  44. [49]

    Neural networks , volume=

    Deep learning in spiking neural networks , author=. Neural networks , volume=. 2019 , publisher=

    2019

  45. [50]

    arXiv e-prints , pages=

    The llama 3 herd of models , author=. arXiv e-prints , pages=

  46. [51]

    Frontiers in neuroscience , volume=

    Spatio-temporal backpropagation for training high-performance spiking neural networks , author=. Frontiers in neuroscience , volume=. 2018 , publisher=

    work page 2018

  47. [52]

    2005 , publisher=

    The organization of behavior: A neuropsychological theory , author=. 2005 , publisher=

    2005

  48. [53]

    Spike timing--dependent plasticity: a Hebbian learning rule , author=. Annu. Rev. Neurosci. , volume=. 2008 , publisher=

    work page 2008

  49. [54]

    Frontiers in neuroscience , volume=

    Going deeper in spiking neural networks: VGG and residual architectures , author=. Frontiers in neuroscience , volume=. 2019 , publisher=

    work page 2019

  50. [55]

    International Journal of Computer Vision , volume=

    Spiking deep convolutional neural networks for energy-efficient object recognition , author=. International Journal of Computer Vision , volume=. 2015 , publisher=

    work page 2015

  51. [56]

    IEEE Transactions on Neural Networks and Learning Systems , volume=

    Constructing accurate and efficient deep spiking neural networks with double-threshold and augmented schemes , author=. IEEE Transactions on Neural Networks and Learning Systems , volume=. 2021 , publisher=

    work page 2021

  52. [57]

    , author=

    SpikeProp: backpropagation for networks of spiking neurons. , author=. ESANN , volume=. 2000 , organization=

    work page 2000

  53. [58]

    Frontiers in neuroscience , volume=

    Training deep spiking neural networks using backpropagation , author=. Frontiers in neuroscience , volume=. 2016 , publisher=

    2016

  54. [59]

    Advances in neural information processing systems , volume=

    Hybrid macro/micro level backpropagation for training deep spiking neural networks , author=. Advances in neural information processing systems , volume=

    work page

  55. [60]

    Frontiers in Neuroscience , volume=

    Synaptic plasticity dynamics for deep continuous local learning (DECOLLE) , author=. Frontiers in Neuroscience , volume=. 2020 , publisher=

    work page 2020

  56. [61]

    arXiv preprint arXiv:2005.01807 , year=

    Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation , author=. arXiv preprint arXiv:2005.01807 , year=

    Pith/arXiv arXiv 2005

  57. [62]

    Advances in neural information processing systems , volume=

    Slayer: Spike layer error reassignment in time , author=. Advances in neural information processing systems , volume=

  58. [63]

    Advances in neural information processing systems , volume=

    Spike-train level backpropagation for training deep recurrent spiking neural networks , author=. Advances in neural information processing systems , volume=

    work page

  59. [64]

    Advances in Neural Information Processing Systems , volume=

    Deep residual learning in spiking neural networks , author=. Advances in Neural Information Processing Systems , volume=

    work page

  60. [65]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

  61. [66]

    arXiv preprint arXiv:2209.15425 , year=

    Spikformer: When spiking neural network meets transformer , author=. arXiv preprint arXiv:2209.15425 , year=

    Pith/arXiv arXiv

  62. [67]

    Advances in Neural Information Processing Systems , volume=

    Spike-driven transformer , author=. Advances in Neural Information Processing Systems , volume=

    work page

  63. [68]

    Proceedings of the seventh IEEE international conference on computer vision , volume=

    Object recognition from local scale-invariant features , author=. Proceedings of the seventh IEEE international conference on computer vision , volume=. 1999 , organization=

    work page 1999

  64. [69]

    2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05) , volume=

    Histograms of oriented gradients for human detection , author=. 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05) , volume=. 2005 , organization=

    2005

  65. [70]

    International conference on machine learning , pages=

    Show, attend and tell: Neural image caption generation with visual attention , author=. International conference on machine learning , pages=. 2015 , organization=

    work page 2015

  66. [71]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    Bottom-up and top-down attention for image captioning and visual question answering , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    work page

  67. [72]

    Advances in neural information processing systems , volume=

    Image captioning: Transforming objects into words , author=. Advances in neural information processing systems , volume=

    work page

  68. [73]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    X-linear attention networks for image captioning , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    work page

  69. [74]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    End-to-end transformer based model for image captioning , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    work page

  70. [75]

    Proceedings of the IEEE international conference on computer vision , pages=

    Towards diverse and natural image descriptions via a conditional gan , author=. Proceedings of the IEEE international conference on computer vision , pages=

    work page

  71. [76]

    Proceedings of the IEEE international conference on computer vision , pages=

    Improved image captioning via policy gradient optimization of spider , author=. Proceedings of the IEEE international conference on computer vision , pages=

    work page

  72. [77]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    Dense captioning with joint inference and visual context , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    work page

  73. [78]

    arXiv preprint arXiv:1510.08829 , year=

    Spiking deep networks with LIF neurons , author=. arXiv preprint arXiv:1510.08829 , year=

    Pith/arXiv arXiv

  74. [79]

    Frontiers in neuroscience , volume=

    Conversion of continuous-valued deep networks to efficient event-driven networks for image classification , author=. Frontiers in neuroscience , volume=. 2017 , publisher=

    work page 2017

  75. [80]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Direct training for spiking neural networks: Faster, larger, better , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  76. [81]

    Frontiers in neuroscience , volume=

    Enabling spike-based backpropagation for training deep neural network architectures , author=. Frontiers in neuroscience , volume=. 2020 , publisher=

    work page 2020

  77. [82]

    arXiv preprint arXiv:2008.03658 , year=

    Diet-snn: Direct input encoding with leakage and threshold optimization in deep spiking neural networks , author=. arXiv preprint arXiv:2008.03658 , year=

    Pith/arXiv arXiv 2008

  78. [83]

    arXiv preprint arXiv:2210.06386 , year=

    Multi-level firing with spiking ds-resnet: Enabling better and deeper directly-trained spiking neural networks , author=. arXiv preprint arXiv:2210.06386 , year=

    Pith/arXiv arXiv

  79. [84]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    Incorporating learnable membrane time constant to enhance learning of spiking neural networks , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

  80. [85]

    Advances in Neural Information Processing Systems , volume=

    LTMD: learning improvement of spiking neural networks with learnable thresholding neurons and moderate dropout , author=. Advances in Neural Information Processing Systems , volume=

    work page

Showing first 80 references.