Pith. sign in

REVIEW 4 major objections 6 minor 67 references

A hybrid spiking-transformer tracker with sparse-coding adapters sets new state-of-the-art results across four RGB-event tracking benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

ISTASTrack fuses RGB and event features through bidirectional ISTA-unrolled adapters between an ANN ViT and an SNN SpikingFormer, reporting state-of-the-art benchmark scores.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Plausible hybrid ANN-SNN tracker with a useful energy story, but the headline SOTA table does not match the paper's own ablations, so the central claim is currently unreproducible. the 4 major comments →

arxiv 2509.09977 v2 pith:OWC72WPQ submitted 2025-09-12 cs.CV

ISTASTrack: Bridging ANN and SNN via ISTA Adapter for RGB-Event Tracking

Siying Liu , Zikai Wang , Hanle Zheng , Yifan Hu , Xilin Wang , Qingkai Yang , Jibin Wu , Hao Guo
show 1 more author
Lei Deng
This is my paper
classification cs.CV
keywords RGB-event trackingspiking neural networkhybrid ANN-SNNiterative shrinkage-thresholdingalgorithm unfoldingtransformer trackersparse representationtemporal fusion
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes ISTASTrack, a tracker that pairs a vision transformer on RGB frames with a spiking transformer on event-camera data. It claims this is the first transformer-based ANN-SNN hybrid for RGB-Event tracking, and that the key to making the two very different feature types work together is a set of 'ISTA adapters' obtained by unrolling the iterative shrinkage-thresholding algorithm for sparse coding. The adapters let each branch read and refine the other's features in both directions, at both attention and MLP stages, while a temporal downsampling attention module compresses the SNN's multi-step outputs into the ANN's single-step format. On FE240hz, VisEvent, COESOT, and FELT, ISTASTrack reports the best success-rate and precision scores among compared trackers, alongside an estimated energy cost roughly 35-40 percent lower than an equivalent ANN-ANN variant. The paper's broader point is that treating cross-modal fusion as a sparse-coding problem gives an interpretable, lightweight route to bridging ANN and SNN representations.

Core claim

The central claim is that bidirectional feature interaction between an ANN transformer and an SNN transformer, formulated as a shared sparse-code problem and solved by unrolled ISTA steps, lets a hybrid network match and exceed the accuracy of ANN-only trackers while cutting estimated energy use. The authors argue that because RGB and event features are assumed to share the same sparse code, the ISTA adapter can project each branch into the other's latent space, and iterate this projection across the first four encoder layers. The temporal downsampling attention module then aligns the event branch's T time steps with the RGB branch's single step before synthesis. Under this design, the 8-lay

What carries the argument

The ISTA adapter: a module formed by unrolling one iteration of the iterative shrinkage-thresholding algorithm, with soft-thresholding using learnable thresholds and dictionary and transpose-dictionary linear layers. Four such adapters are inserted per encoder layer—two directions (RGB-to-event, event-to-RGB) at both the MSA and MLP stages—so the sparse code is progressively refined as features move through the transformer. The TDA module uses adaptive pooling and a sigmoid weighting over the T time steps to collapse multi-step SNN sparse codes to a single-step representation.

Load-bearing premise

The comparison is fair: most baselines are retrained under the authors' protocol, but two are taken from their published papers, and the best hyperparameters (N=4, T=3) are chosen on the same test sets; if those protocol differences are not equivalent, the state-of-the-art margins are not meaningful.

What would settle it

Retrain CrossEI and SNNPTrack under the exact protocol used for the other baselines (same data splits, loss, epochs) and compare success rate on COESOT or VisEvent: if either matches or exceeds ISTASTrack's score, the claimed state of the art is an artifact of protocol mixing rather than the adapter design.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, transformer-based ANN-SNN hybrids are a viable route to state-of-the-art RGB-event tracking, not just a low-energy compromise.
  • The bidirectional, stage-wise adapter design outperforms unidirectional prompt-based fusion on the tested benchmarks.
  • Early-layer interaction (first 4 layers) beats late fusion; adding adapters beyond four layers can hurt accuracy.
  • The SNN branch's 8 layers and 3 time steps deliver accuracy comparable to a 12-layer ANN branch, at estimated energy savings of roughly 40 percent.
  • A lightweight adapter (0.32M parameters, about 2 percent of the total) is sufficient to bridge the ANN-SNN gap, suggesting the fusion mechanism rather than extra capacity drives the gains.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The shared-sparse-code assumption, if it holds generally, makes ISTA adapters a candidate template for fusing ANN and SNN features in other event-based tasks such as detection, action recognition, or deblurring, where similar hybrid branches already exist.
  • Because energy estimates rely on 45nm 32-bit MAC/AC costs and assumed firing rates, the efficiency margin may change on real neuromorphic hardware; a direct chip-level measurement would be needed to confirm it.
  • A testable extension: with SNN states allowed to persist across frames—the paper names this as a limitation—the TDA module could attend over history, potentially improving long-term tracking on FELT further.
  • The largest reported gains over the strongest baseline are around 0.2-0.4 success-rate points; whether that margin survives identical training protocols for the two baselines whose published scores are used is an open empirical question the paper does not settle.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes ISTASTrack, a hybrid ANN-SNN tracker for RGB-event visual object tracking. The architecture uses a ViT-based ANN branch for RGB and a SpikingFormer-based SNN branch for event streams, with bidirectional ISTA adapters inserted at multiple transformer layers to transfer features between branches. A temporal downsampling attention (TDA) module aggregates multi-step SNN features into single-step ANN representations. The authors report state-of-the-art results on FE240hz, VisEvent, COESOT, and FELT, along with energy-efficiency estimates showing lower estimated energy than a dual-ANN counterpart. The derivation of the adapter from sparse representation theory via ISTA unfolding is a central methodological claim. The paper includes extensive ablations over adapter direction, depth, placement, and time steps, and the code is publicly released.

Significance. If the results are reproducible, the contribution is notable: it is among the first transformer-based ANN-SNN hybrids for RGB-event tracking, and the proposed adapter is lightweight (0.32M parameters), which is a practically relevant property. The paper gives explicit credit to reproducible artifacts: the code is public, the comparisons cover many recent baselines, and the ablations are fairly comprehensive. The energy-estimation methodology follows a widely used convention for SNN cost. However, the strength of the claim rests on the consistency and fairness of the reported benchmark numbers, and on whether the ISTA derivation is genuinely enforced by the training objective. The main value would be as a practical hybrid architecture with competitive accuracy and low estimated cost, rather than as a strict validation of the sparse-coding model.

major comments (4)
  1. [Table I vs. Table IV] The headline SOTA numbers in Table I are internally inconsistent with the full-model ablation in Table IV. Table I reports ISTASTrack as FE240hz 64.7/92.2, VisEvent 67.3/84.6, COESOT 75.7/87.1, FELT 55.2/65.8 (SR/PR). Table IV reports the bidirectional-adapter model (the full model, with N=4 and T=3 according to Section IV-A) as FE240hz 64.59/92.83, VisEvent 66.46/84.34, COESOT 75.74/86.82, FELT 54.93/65.75. On VisEvent the SR gap is 0.84, larger than the reported advantage over BAT (0.2); on FE240hz and COESOT the PR gaps are 0.63 and 0.28. These are not rounding errors. Table VI also does not resolve the discrepancy: T=1 gives VisEvent 67.26/84.61 and FELT 55.18/65.75, close to Table I, while Table IV uses T=3. The FE240hz PR 92.2 matches neither T=1 (91.8) nor T=3 (92.83). The authors must identify which configuration produced Table I and report the exact same numbers in the ablation
  2. [Sec. IV-C.1, Table I] The benchmark comparison mixes retrained baselines with published numbers from other papers. The text states that nearly all networks are retrained on the four datasets using the authors' settings, while CrossEI and SNNPTrack are reported as published. Since the authors' training protocol may not match each baseline's best configuration, and since no per-method hyperparameter tuning or validation protocol is described, the small margins over BAT (e.g., 0.2 SR on VisEvent) cannot be interpreted as meaningful improvements. The authors should either use each baseline's official code/checkpoints under its own protocol, or report variances and a common tuning budget. This is especially important because the claimed SOTA margins are small.
  3. [Sec. III-C.2, Eq. (4) and Eq. (12)] The sparse-coding derivation is not enforced by the training objective. Eq. (4) defines a LASSO problem for the sparse code, but the adapters in Eq. (7) are trained by backpropagation through the tracking loss in Eq. (12); there is no reconstruction term, no sparsity penalty, and no constraint that the learned dictionaries satisfy the ISTA fixed-point equations. Moreover, Eq. (6) explicitly treats P and D^T as independent matrices, which breaks the ISTA update's reliance on D^T. As a result, the adapters behave as learned residual cross-modal modules, and the claim that the method is 'derived from sparse representation theory' and 'interpretable' overstates the link. The authors should either enforce the model-based objective (e.g., add a sparse-coding auxiliary loss or a reconstruction term) or reframe the ISTA connection as architectural motivation only.
  4. [Sec. IV-A, Sec. IV-E] The hyperparameters N (number of adapter layers) and T (number of time steps) are selected on the same test datasets used to report SOTA results. Section IV-A fixes N=4 and T=3 based on the ablations in Tables V and VI, but those ablations are evaluated on FE240hz, VisEvent, COESOT, and FELT themselves, not on a held-out validation split. Selecting hyperparameters on the test set inflates the reported accuracy and makes the SOTA claim optimistic. The authors should use a validation split or nested evaluation, and report mean and variance over multiple runs, especially given the small margins in Table I.
minor comments (6)
  1. [Sec. III-A heading] The heading reads 'Overview of ISTATrack'; should be 'ISTASTrack' to match the paper title.
  2. [Eq. (6)] The sentence 'with a threshold θ=θ' is circular and likely a typo; it should define the threshold parameter, e.g., θ>0.
  3. [Eq. (7) and Sec. III-D] The notation D_T for the TDA module conflicts with the dictionary notation D (e.g., D_I, D_E). Using the same letter for a temporal downsampling operator and for dictionaries is confusing; rename one of them.
  4. [Sec. IV-A] The 'Experimental Settings' subsection has no body text; it jumps directly to 'Datasets'. Either fill in the settings or merge the heading with the previous section.
  5. [Fig. 7] The label 'FEL T' in the radar chart appears to be a typo; should be 'FELT'.
  6. [References [14] and [45]] In Sec. III-C.3, the text cites [14] for the inspiration from bidirectional adapters, but [14] is 'Bidirectional Multi-Scale Implicit Neural Representations for Image Deraining,' which is unrelated to multimodal tracking. The relevant reference appears to be [45] (BAT). Please correct the citation.

Circularity Check

0 steps flagged

No significant circularity: the ISTA adapter is constructed by unrolling a stated sparse-coding objective, and the shared-code relation is an explicit assumption rather than a hidden reuse of the tracking result.

full rationale

The paper's central derivation chain—ISTA adapters from sparse-representation theory—is not circular. Eq. (4) poses a LASSO reconstruction problem for each modality's features under its own dictionary, and Eq. (5) synthesizes the translated feature using a second dictionary. The key relation, Eq. (2), is explicitly stated as an assumption: 'we assume that both feature embeddings have similar structure and thus share a common sparse code' (Sec. III-C). This is an ansatz, not a conclusion obtained from the tracking result, and it is not smuggled in via citation. The unrolled ISTA updates (Eqs. (6)-(7)) follow the standard, independently documented algorithm-unfolding procedure [27]-[29]. The energy estimate in Eq. (11) is self-contained once per-operation energies are taken from Horowitz [67]; the citation to [66] is a methodological pointer, not the basis for the SOTA numerical claim. The SOTA evaluation is against external benchmarks (FE240hz, VisEvent, COESOT, FELT) and does not reduce to a fitted input: baselines are either retrained under the stated protocol or explicitly marked as published (CrossEI, SNNPTrack). Self-citations ([49], [51], [66]) appear only in background and efficiency-estimation passages and are not load-bearing. The paper also states a genuine limitation in Sec. V ('the potential of SNNs for processing long sequences has not been fully explored ... resets SNN states after each inference step'), which is a limitation, not circularity. Separately, there are reproducibility concerns—e.g., the ISTASTrack row in Tab. I does not exactly match any configuration in Tabs. IV/VI, and the stated default T=3 conflicts with Tab. VI showing T=1 better on VisEvent and FELT—but these are correctness/external-validity issues, not evidence that any prediction is equivalent to its input by construction. No circular step can be exhibited.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 1 invented entities

The paper's central tracking result depends mainly on standard ML assumptions and a fair benchmark protocol. The mathematical derivation of the ISTA adapter rests on a shared-code postulate and on untied P/D matrices, neither of which is validated or enforced. Hyperparameters T and N are selected on the test sets, which is a form of free-parameter fitting to the benchmarks.

free parameters (4)
  • Number of ISTA adapter layers N = 4
    Selected as optimal in Tab. V ablation on the test benchmarks; not derived.
  • Number of SNN time steps T = 3
    Default chosen in Sec. IV-A from Tab. VI trade-off; not derived.
  • Loss weights lambda_1, lambda_2, lambda_3 = 2, 5, 1
    Copied from OSTrack [6], not tuned or justified by sensitivity analysis.
  • Learnable ISTA thresholds theta_k = learned per element
    Free threshold vector in Eq. (6)-(7), not tied to the sparsity penalty lambda.
axioms (5)
  • ad hoc to paper RGB and event transformer features share a common sparse code a_I = a_E' (Eq. 2)
    Postulated to derive cross-modal transfer; never verified or enforced by the training objective.
  • ad hoc to paper P and D^T can be treated as independent matrices in the ISTA update (Sec. III-C.2)
    The text sets P=D^T then says they are independent; this breaks the formal link to ISTA for Eq. (4).
  • domain assumption Event streams can be summarized as T stacked event frames aligned with RGB frames
    Standard in prior RGB-event work but discards asynchronous event timing.
  • domain assumption Energy can be estimated from MAC/AC counts at 4.6/0.9 pJ with firing rates (Eq. 11)
    Standard estimation from [64]-[67], not a hardware measurement.
  • domain assumption Pretrained OSTrack and SpikingFormer weights transfer to the tracking task after fine-tuning
    Used in Sec. IV-A initialization; supported only indirectly by final accuracy.
invented entities (1)
  • shared sparse code a_{I->E} / a_{E->I} no independent evidence
    purpose: Latent code assumed identical for RGB and event features to justify bidirectional adapters
    Eqs. (2)-(3) introduce this shared code; no external or internal validation is provided beyond tracking accuracy.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of ISTASTrack: Bridging ANN and SNN via ISTA Adapter for RGB-Event Tracking." pith.science (2026). https://pith.science/paper/OWC72WPQ

@misc{pith2026250909977,
  author       = {Pith},
  title        = {Pith review of: ISTASTrack: Bridging ANN and SNN via ISTA Adapter for RGB-Event Tracking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OWC72WPQ}},
  note         = {Machine review of arXiv:2509.09977}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

RGB-Event tracking has become a promising trend in visual object tracking to leverage the complementary strengths of both RGB images and dynamic spike events for improved performance. However, existing artificial neural networks (ANNs) struggle to fully exploit the sparse and asynchronous nature of event streams. Recent efforts toward hybrid architectures combining ANNs and spiking neural networks (SNNs) have emerged as a promising solution in RGB-Event perception, yet effectively fusing features across heterogeneous paradigms remains a challenge. In this work, we propose ISTASTrack, the first transformer-based \textbf{A}NN-\textbf{S}NN hybrid \textbf{Track}er equipped with \textbf{ISTA} adapters for RGB-Event tracking. The two-branch model employs a vision transformer to extract spatial context from RGB inputs and a spiking transformer to capture spatio-temporal dynamics from event streams. To bridge the modality and paradigm gap between ANN and SNN features, we systematically design a model-based ISTA adapter for bidirectional feature interaction between the two branches, derived from sparse representation theory by unfolding the iterative shrinkage thresholding algorithm. Additionally, we incorporate a temporal downsampling attention module within the adapter to align multi-step SNN features with single-step ANN features in the latent space, improving temporal fusion. Experimental results on RGB-Event tracking benchmarks, such as FE240hz, VisEvent, COESOT, and FELT, have demonstrated that ISTASTrack achieves state-of-the-art performance while maintaining high energy efficiency, highlighting the effectiveness and practicality of hybrid ANN-SNN designs for robust visual tracking. The code is publicly available at https://github.com/lsying009/ISTASTrack.git.

Figures

Figures reproduced from arXiv: 2509.09977 by Hanle Zheng, Hao Guo, Jibin Wu, Lei Deng, Qingkai Yang, Siying Liu, Xilin Wang, Yifan Hu, Zikai Wang.

Figure 1
Figure 1. Figure 1: ANN and hybrid ANN-SNN architecture with different fusion [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the ISTATrack architecture. The hybrid network incorporates a vision transformer branch for RGB inputs and a spiking transformer [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: ISTA adapters for bidirectional feature adaptation and fusion. (a) ISTA adapters [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Temporal downsampling attention (TDA) module [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative comparison of tracking results across challenging scenarios, including low light, overexposure, small objects, similar objects, fast motion, [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Tracking results on challenging long sequences with similar objects and occlusions. Green boxes and arrows indicate ground truth (GT) targets, and [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

67 extracted references · 6 linked inside Pith

  1. [1]

    Event-based Vision: A Survey,

    G. Gallego, T. Delbruck, G. Orchard, C. Bartolozzi, B. Taba, A. Censi, S. Leutenegger, A. Davison, J. Conradt, K. Daniilidis, and D. Scara- muzza, “Event-based Vision: A Survey,”IEEE Trans. Pattern Anal. Mach. Intell., pp. 1–1, 2020

    2020

  2. [2]

    DSEC: A Stereo Event Camera Dataset for Driving Scenarios,

    M. Gehrig, W. Aarents, D. Gehrig, and D. Scaramuzza, “DSEC: A Stereo Event Camera Dataset for Driving Scenarios,”IEEE Robot. Autom. Lett., vol. 6, no. 3, pp. 4947–4954, Jul 2021

    2021

  3. [3]

    Combined frame- and event-based detection and tracking,

    H. Liu, D. P. Moeys, G. Das, D. Neil, S.-C. Liu, and T. Delbr ¨uck, “Combined frame- and event-based detection and tracking,” in2016 IEEE International Symposium on Circuits and Systems (ISCAS), May 2016, pp. 2511–2514

    2016

  4. [4]

    HARDVS: Revisiting Human Activity Recognition with Dynamic Vision Sensors,

    X. Wang, Z. Wu, B. Jiang, Z. Bao, L. Zhu, G. Li, Y . Wang, and Y . Tian, “HARDVS: Revisiting Human Activity Recognition with Dynamic Vision Sensors,”Proc. AAAI Conf. Artif. Intell., vol. 38, no. 6, pp. 5615– 5623, Mar 2024

    2024

  5. [5]

    Learning Discrim- inative Model Prediction for Tracking,

    G. Bhat, M. Danelljan, L. Van Gool, and R. Timofte, “Learning Discrim- inative Model Prediction for Tracking,” in2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Oct 2019, pp. 6181– 6190

    2019

  6. [6]

    Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework,

    B. Ye, H. Chang, B. Ma, S. Shan, and X. Chen, “Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework,” in Computer Vision – ECCV 2022, S. Avidan, G. Brostow, M. Ciss´e, G. M. Farinella, and T. Hassner, Eds. Springer Nature Switzerland, 2022, vol. 13682, pp. 341–357

    2022

  7. [7]

    Revisiting Color-Event based Tracking: A Unified Network, Dataset, and Metric,

    C. Tang, X. Wang, J. Huang, B. Jiang, L. Zhu, J. Zhang, Y . Wang, and Y . Tian, “Revisiting Color-Event based Tracking: A Unified Network, Dataset, and Metric,” Jan 2024. [Online]. Available: http://arxiv.org/abs/2211.11010

    Pith/arXiv arXiv 2024

  8. [8]

    Object Tracking by Jointly Exploiting Frame and Event Domain,

    J. Zhang, X. Yang, Y . Fu, X. Wei, B. Yin, and B. Dong, “Object Tracking by Jointly Exploiting Frame and Event Domain,” in2021 IEEE/CVF International Conference on Computer Vision (ICCV), Oct 2021, pp. 13 023–13 032

    2021

  9. [9]

    Frame-Event Alignment and Fusion Network for High Frame Rate Tracking,

    J. Zhang, Y . Wang, W. Liu, M. Li, J. Bai, B. Yin, and X. Yang, “Frame-Event Alignment and Fusion Network for High Frame Rate Tracking,” in2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Jun 2023, pp. 9781–9790

    2023

  10. [10]

    VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows,

    X. Wang, J. Li, L. Zhu, Z. Zhang, Z. Chen, X. Li, Y . Wang, Y . Tian, and F. Wu, “VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows,”IEEE Trans. Cybern., vol. 54, no. 3, pp. 1997–2010, Mar 2024

    1997

  11. [11]

    CrossEI: Boosting Motion- Oriented Object Tracking With an Event Camera,

    Z. Chen, J. Wu, W. Dong, L. Li, and G. Shi, “CrossEI: Boosting Motion- Oriented Object Tracking With an Event Camera,”IEEE Trans. Image Process., vol. 34, pp. 73–84, 2025

    2025

  12. [12]

    EMTrack: Efficient Multimodal Object Tracking,

    C. Liu, Z. Guan, S. Lai, Y . Liu, H. Lu, and D. Wang, “EMTrack: Efficient Multimodal Object Tracking,”IEEE Trans. Circuits Syst. Video Technol., pp. 1–1, 2024. JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. XX, SEPTEMBER 2025 14 TABLE VII COMPARISON OF COMPUTATIONAL COSTS ACROSS DIFFERENT EVENT BACKBONE ARCHITECTURES AND OUR PROPOSED MODULE. “ANN-SNN” DENOTES...

    2024

  13. [13]

    Single-Model and Any-Modality for Video Object Tracking,

    Z. Wu, J. Zheng, X. Ren, F.-A. Vasluianu, C. Ma, D. P. Paudel, L. Van Gool, and R. Timofte, “Single-Model and Any-Modality for Video Object Tracking,” in2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Jun 2024, pp. 19 156– 19 166

    2024

  14. [14]

    Bidirectional Multi-Scale Implicit Neural Representations for Image Deraining,

    X. Chen, J. Pan, and J. Dong, “Bidirectional Multi-Scale Implicit Neural Representations for Image Deraining,” Apr 2024. [Online]. Available: http://arxiv.org/abs/2404.01547

    Pith/arXiv arXiv 2024

  15. [15]

    Event-Driven Video Restoration With Spiking-Convolutional Architecture,

    C. Cao, X. Fu, Y . Zhu, Z. Sun, and Z.-J. Zha, “Event-Driven Video Restoration With Spiking-Convolutional Architecture,”IEEE Trans. Neural Netw. Learn. Syst., pp. 1–15, 2023

    2023

  16. [16]

    Motion- Oriented Hybrid Spiking Neural Networks for Event-Based Motion Deblurring,

    Z. Liu, J. Wu, G. Shi, W. Yang, W. Dong, and Q. Zhao, “Motion- Oriented Hybrid Spiking Neural Networks for Event-Based Motion Deblurring,”IEEE Trans. Circuits Syst. Video Technol., vol. 34, no. 5, pp. 3742–3754, May 2024

    2024

  17. [17]

    ReSpike: Residual Frames-based Hybrid Spiking Neural Networks for Efficient Action Recognition,

    S. Xiao, Y . Li, Y . Kim, D. Lee, and P. Panda, “ReSpike: Residual Frames-based Hybrid Spiking Neural Networks for Efficient Action Recognition,” Sep 2024. [Online]. Available: http://arxiv.org/abs/2409. 01564

    2024

  18. [18]

    SSTFormer: Bridging Spiking Neural Network and Memory Support Transformer for Frame-Event based Recognition,

    X. Wang, Z. Wu, Y . Rong, L. Zhu, B. Jiang, J. Tang, and Y . Tian, “SSTFormer: Bridging Spiking Neural Network and Memory Support Transformer for Frame-Event based Recognition,” Aug 2023. [Online]. Available: http://arxiv.org/abs/2308.04369

    Pith/arXiv arXiv 2023

  19. [19]

    HDI-Former: Hybrid Dynamic Interaction ANN-SNN Transformer for Object Detection Using Frames and Events,

    D. Li, J. Li, X. Liu, Z. Zhou, X. Fan, and Y . Tian, “HDI-Former: Hybrid Dynamic Interaction ANN-SNN Transformer for Object Detection Using Frames and Events,” Nov 2024

    2024

  20. [20]

    Reliable object tracking by multimodal hybrid feature extraction and transformer-based fusion,

    H. Sun, R. Liu, W. Cai, J. Wang, Y . Wang, H. Tang, Y . Cui, D. Yao, and D. Guo, “Reliable object tracking by multimodal hybrid feature extraction and transformer-based fusion,”Neural Networks, vol. 178, p. 106493, Oct 2024

    2024

  21. [21]

    SNNPTrack: Spiking Neural Network Based Prompt for High-Accuracy RGBE Tracking,

    Y . Ji, Q. Zhao, Y . Liang, and J. Wu, “SNNPTrack: Spiking Neural Network Based Prompt for High-Accuracy RGBE Tracking,” inICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Apr 2025, pp. 1–5

    2025

  22. [22]

    A Hybrid ANN- SNN Architecture for Low-Power and Low-Latency Visual Perception,

    A. Aydin, M. Gehrig, D. Gehrig, and D. Scaramuzza, “A Hybrid ANN- SNN Architecture for Low-Power and Low-Latency Visual Perception,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 5701–5711

    2024

  23. [23]

    Elad,Sparse and Redundant Representations: From Theory to Applications in Signal and Image Processing

    M. Elad,Sparse and Redundant Representations: From Theory to Applications in Signal and Image Processing. Springer, 2010

    2010

  24. [24]

    Deep Convolutional Neural Network for Multi-Modal Image Restoration and Fusion,

    X. Deng and P. L. Dragotti, “Deep Convolutional Neural Network for Multi-Modal Image Restoration and Fusion,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, no. 10, pp. 3333–3348, Oct 2021

    2021

  25. [25]

    Designing CNNs for Multimodal Image Restoration and Fusion via Unfolding the Method of Multipliers,

    I. Marivani, E. Tsiligianni, B. Cornelis, and N. Deligiannis, “Designing CNNs for Multimodal Image Restoration and Fusion via Unfolding the Method of Multipliers,”IEEE Trans. Circuits Syst. Video Technol., vol. 32, no. 9, pp. 5830–5845, Sep 2022

    2022

  26. [26]

    Learning Sparse and Discriminative Multimodal Feature Codes for Finger Recognition,

    S. Li, B. Zhang, L. Fei, S. Zhao, and Y . Zhou, “Learning Sparse and Discriminative Multimodal Feature Codes for Finger Recognition,”IEEE Trans. Multimed., vol. 25, pp. 805–815, 2023

    2023

  27. [27]

    An iterative thresholding algorithm for linear inverse problems with a sparsity constraint,

    I. Daubechies, M. Defrise, and C. De Mol, “An iterative thresholding algorithm for linear inverse problems with a sparsity constraint,”Comm. Pure Appl. Math., vol. 57, no. 11, pp. 1413–1457, Nov 2004

    2004

  28. [28]

    Learning Fast Approximations of Sparse Coding,

    K. Gregor and Y . LeCun, “Learning Fast Approximations of Sparse Coding,” inProc. 27th Int. Conf. Mach. Learn., 2010, pp. 399–406

    2010

  29. [29]

    Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing,

    V . Monga, Y . Li, and Y . C. Eldar, “Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing,”IEEE Signal Process. Mag., vol. 38, no. 2, pp. 18–44, 2021

    2021

  30. [30]

    Fully-Convolutional Siamese Networks for Object Tracking,

    L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. S. Torr, “Fully-Convolutional Siamese Networks for Object Tracking,” in Computer Vision – ECCV 2016 Workshops, G. Hua and H. J ´egou, Eds. Springer International Publishing, 2016, pp. 850–865

    2016

  31. [31]

    High Performance Visual Tracking with Siamese Region Proposal Network,

    B. Li, J. Yan, W. Wu, Z. Zhu, and X. Hu, “High Performance Visual Tracking with Siamese Region Proposal Network,” in2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Jun 2018, pp. 8971–8980

    2018

  32. [32]

    ATOM: Accurate Tracking by Overlap Maximization,

    M. Danelljan, G. Bhat, F. S. Khan, and M. Felsberg, “ATOM: Accurate Tracking by Overlap Maximization,” in2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Jun 2019, pp. 4655–4664

    2019

  33. [33]

    Probabilistic Regression for Visual Tracking,

    M. Danelljan, L. Van Gool, and R. Timofte, “Probabilistic Regression for Visual Tracking,” in2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Jun 2020, pp. 7181–7190

    2020

  34. [34]

    Learning Target Candidate Association to Keep Track of What Not to Track,

    C. Mayer, M. Danelljan, D. Pani Paudel, and L. Van Gool, “Learning Target Candidate Association to Keep Track of What Not to Track,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Oct 2021, pp. 13 424–13 434

    2021

  35. [35]

    Learning Spatio-Temporal Transformer for Visual Tracking,

    B. Yan, H. Peng, J. Fu, D. Wang, and H. Lu, “Learning Spatio-Temporal Transformer for Visual Tracking,” in2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Oct 2021, pp. 10 428– 10 437

    2021

  36. [36]

    SwinTrack: A Simple and Strong Baseline for Transformer Tracking,

    X. Chen, B. Yan, J. Zhu, D. Wang, X. Yang, and H. Lu, “SwinTrack: A Simple and Strong Baseline for Transformer Tracking,” in2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Jun 2021, pp. 8122–8131

    2021

  37. [37]

    AiATrack: Attention in Attention for Transformer Visual Tracking,

    S. Gao, C. Zhou, C. Ma, X. Wang, and J. Yuan, “AiATrack: Attention in Attention for Transformer Visual Tracking,” inComputer Vision – ECCV 2022, S. Avidan, G. Brostow, M. Ciss ´e, G. M. Farinella, and T. Hassner, Eds. Springer Nature Switzerland, 2022, vol. 13682, pp. 146–164

    2022

  38. [38]

    MixFormer: End-to- End Tracking with Iterative Mixed Attention,

    Y . Cui, C. Jiang, L. Wang, and G. Wu, “MixFormer: End-to- End Tracking with Iterative Mixed Attention,” Mar 2022. [Online]. Available: http://arxiv.org/abs/2203.11082

    Pith/arXiv arXiv 2022

  39. [39]

    Event-Assisted Object Tracking on High-Speed Drones in Harsh Illumination Environment,

    Y . Han, X. Yu, H. Luan, and J. Suo, “Event-Assisted Object Tracking on High-Speed Drones in Harsh Illumination Environment,”Drones, vol. 8, no. 1, p. 22, Jan 2024

    2024

  40. [40]

    Long-term Frame-Event Visual Tracking: Benchmark Dataset and Baseline,

    X. Wang, J. Huang, S. Wang, C. Tang, B. Jiang, Y . Tian, J. Tang, and B. Luo, “Long-term Frame-Event Visual Tracking: Benchmark Dataset and Baseline,” Apr 2024. [Online]. Available: http://arxiv.org/abs/2403.05839

    Pith/arXiv arXiv 2024

  41. [41]

    CRSOT: Cross-Resolution Object Tracking using Unaligned Frame and Event Cameras,

    Y . Zhu, X. Wang, C. Li, B. Jiang, L. Zhu, Z. Huang, Y . Tian, and J. Tang, “CRSOT: Cross-Resolution Object Tracking using Unaligned Frame and Event Cameras,” Jan 2024. [Online]. Available: http://arxiv.org/abs/2401.02826

    Pith/arXiv arXiv 2024

  42. [42]

    TENet: Targetness entanglement incorporating with multi-scale pooling and mutually-guided fusion for RGB-E object tracking,

    P. Shao, T. Xu, Z. Tang, L. Li, X.-J. Wu, and J. Kittler, “TENet: Targetness entanglement incorporating with multi-scale pooling and mutually-guided fusion for RGB-E object tracking,”Neural Networks, vol. 183, p. 106948, Mar 2025

    2025

  43. [43]

    Visual Prompt Multi- Modal Tracking,

    J. Zhu, S. Lai, X. Chen, D. Wang, and H. Lu, “Visual Prompt Multi- Modal Tracking,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 9516–9526

    2023

  44. [44]

    OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning,

    L. Hong, S. Yan, R. Zhang, W. Li, X. Zhou, P. Guo, K. Jiang, Y . Chen, J. Li, Z. Chen, and W. Zhang, “OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 19 079–19 091

    2024

  45. [45]

    Bi-directional Adapter for Multimodal Tracking,

    B. Cao, J. Guo, P. Zhu, and Q. Hu, “Bi-directional Adapter for Multimodal Tracking,”Proc. AAAI Conf. Artif. Intell., vol. 38, no. 2, pp. 927–935, Mar 2024

    2024

  46. [46]

    Com- JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. XX, SEPTEMBER 2025 15 plementary Learning Systems,

    R. C. O’Reilly, R. Bhattacharyya, M. D. Howard, and N. Ketz, “Com- JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. XX, SEPTEMBER 2025 15 plementary Learning Systems,”Cognitive Science, vol. 38, no. 6, pp. 1229–1248, Aug 2014

    2025

  47. [47]

    What Learning Systems do Intelligent Agents Need? Complementary Learning Systems Theory Updated,

    D. Kumaran, D. Hassabis, and J. L. McClelland, “What Learning Systems do Intelligent Agents Need? Complementary Learning Systems Theory Updated,”Trends in Cognitive Sciences, vol. 20, no. 7, pp. 512– 534, Jul 2016

    2016

  48. [48]

    A framework for the general design and computation of hybrid neural networks,

    R. Zhao, Z. Yang, H. Zheng, Y . Wu, F. Liu, Z. Wu, L. Li, F. Chen, S. Song, J. Zhu, W. Zhang, H. Huang, M. Xu, K. Sheng, Q. Yin, J. Pei, G. Li, Y . Zhang, M. Zhao, and L. Shi, “A framework for the general design and computation of hybrid neural networks,”Nat Commun, vol. 13, no. 1, p. 3427, Jun 2022

    2022

  49. [49]

    Advancing brain-inspired computing with hybrid neural networks,

    F. Liu, H. Zheng, S. Ma, W. Zhang, X. Liu, Y . Chua, L. Shi, and R. Zhao, “Advancing brain-inspired computing with hybrid neural networks,” Natl. Sci. Rev., vol. 11, no. 5, p. nwae066, Apr 2024

    2024

  50. [50]

    Hybrid neural networks for continual learning inspired by corticohippocampal circuits,

    Q. Shi, F. Liu, H. Li, G. Li, L. Shi, and R. Zhao, “Hybrid neural networks for continual learning inspired by corticohippocampal circuits,” Nat Commun, vol. 16, no. 1, p. 1272, Feb 2025

    2025

  51. [51]

    Adaptive spatiotemporal neural networks through complementary hy- bridization,

    Y . Wu, B. Shi, Z. Zheng, H. Zheng, F. Yu, X. Liu, G. Luo, and L. Deng, “Adaptive spatiotemporal neural networks through complementary hy- bridization,”Nat Commun, vol. 15, no. 1, p. 7355, Aug 2024

    2024

  52. [52]

    Spiking Transformers for Event-based Single Object Tracking,

    J. Zhang, B. Dong, H. Zhang, J. Ding, F. Heide, B. Yin, and X. Yang, “Spiking Transformers for Event-based Single Object Tracking,” in2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Jun 2022, pp. 8791–8800

    2022

  53. [53]

    Chasing Day and Night: Towards Robust and Efficient All-Day Object Detection Guided by an Event Camera,

    J. Cao, X. Zheng, Y . Lyu, J. Wang, R. Xu, and L. Wang, “Chasing Day and Night: Towards Robust and Efficient All-Day Object Detection Guided by an Event Camera,” in2024 IEEE International Conference on Robotics and Automation (ICRA), May 2024, pp. 9026–9032

    2024

  54. [54]

    Message-passing algo- rithms for compressed sensing,

    D. L. Donoho, A. Maleki, and A. Montanari, “Message-passing algo- rithms for compressed sensing,”Proc. Natl. Acad. Sci., vol. 106, no. 45, pp. 18 914–18 919, Nov 2009

    2009

  55. [55]

    Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers,

    S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers,”Found. Trends Mach. Learn., vol. 3, no. 1, pp. 1–122, Jan 2011

    2011

  56. [56]

    Deep unfolding of a proximal interior point method for image restoration,

    C. Bertocchi, E. Chouzenoux, M.-C. Corbineau, J.-C. Pesquet, and M. Prato, “Deep unfolding of a proximal interior point method for image restoration,”Inverse Problems, vol. 36, no. 3, p. 034005, Feb 2020

    2020

  57. [57]

    Deep Generalized Unfolding Networks for Image Restoration,

    C. Mou, Q. Wang, and J. Zhang, “Deep Generalized Unfolding Networks for Image Restoration,” in2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Jun 2022, pp. 17 378– 17 389

    2022

  58. [58]

    Event Enhanced High- Quality Image Recovery,

    B. Wang, J. He, L. Yu, G.-S. Xia, and W. Yang, “Event Enhanced High- Quality Image Recovery,” inEuropean Conference on Computer Vision. Springer, 2020

    2020

  59. [59]

    Deep Coupled ISTA Network for Multi- Modal Image Super-Resolution,

    X. Deng and P. L. Dragotti, “Deep Coupled ISTA Network for Multi- Modal Image Super-Resolution,”IEEE Trans. on Image Process., vol. 29, no. 10, pp. 1683–1698, Oct 2020

    2020

  60. [60]

    Learning to Super-Resolve Blurry Images With Events,

    L. Yu, B. Wang, X. Zhang, H. Zhang, W. Yang, J. Liu, and G.-S. Xia, “Learning to Super-Resolve Blurry Images With Events,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 8, pp. 10 027–10 043, Aug 2023

    2023

  61. [61]

    Sensing Diversity and Sparsity Models for Event Generation and Video Reconstruction from Events,

    S. Liu and P. L. Dragotti, “Sensing Diversity and Sparsity Models for Event Generation and Video Reconstruction from Events,”IEEE Trans. Pattern Anal. Mach. Intell., pp. 1–16, 2023

    2023

  62. [62]

    Interpretable Multi- Modal Image Registration Network Based on Disentangled Convolu- tional Sparse Coding,

    X. Deng, E. Liu, S. Li, Y . Duan, and M. Xu, “Interpretable Multi- Modal Image Registration Network Based on Disentangled Convolu- tional Sparse Coding,”IEEE Trans. Image Process., vol. 32, pp. 1078– 1091, 2023

    2023

  63. [63]

    Revisiting Convolutional Sparse Coding for Image Denoising: From a Multi-Scale Perspective,

    J. Xu, X. Deng, and M. Xu, “Revisiting Convolutional Sparse Coding for Image Denoising: From a Multi-Scale Perspective,”IEEE Signal Process. Lett., vol. 29, pp. 1202–1206, 2022

    2022

  64. [64]

    Spikingformer: Spike-driven Residual Learning for Transformer- based Spiking Neural Network,

    C. Zhou, L. Yu, Z. Zhou, Z. Ma, H. Zhang, H. Zhou, and Y . Tian, “Spikingformer: Spike-driven Residual Learning for Transformer- based Spiking Neural Network,” May 2023. [Online]. Available: http://arxiv.org/abs/2304.11954

    arXiv 2023

  65. [65]

    Deep Directly-Trained Spiking Neural Networks for Object Detection,

    Q. Su, Y . Chou, Y . Hu, J. Li, S. Mei, Z. Zhang, and G. Li, “Deep Directly-Trained Spiking Neural Networks for Object Detection,” in 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Oct 2023, pp. 6532–6542

    2023

  66. [66]

    Advancing Spiking Neural Networks towards Deep Residual Learning,

    Y . Hu, L. Deng, Y . Wu, M. Yao, and G. Li, “Advancing Spiking Neural Networks towards Deep Residual Learning,” Mar 2023

    2023

  67. [67]

    1.1 Computing’s energy problem (and what we can do about it),

    M. Horowitz, “1.1 Computing’s energy problem (and what we can do about it),” in2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), Feb 2014, pp. 10–14

    2014

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.