Pith. sign in

REVIEW 4 major objections 5 minor 66 references

Improving vision-language alignment with graph spiking hybrid Networks

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Panoptic-segmentation mask tokens, encoded by a graph attention and spiking neural hybrid, improve vision-language alignment on VQA, visual entailment, NLVR2, and image-text retrieval.

desk verdict A genuinely new GAT+SNN hybrid for VLP with competitive numbers, but the method is under-specified and the reporting is sloppy; the Table VII 'contradiction' in the stress test is a misread of the columns. read the letter →

arxiv 2501.19069 v2 pith:Z7PBSYAL submitted 2025-01-31 cs.CV cs.AI

Siyu Zhang , Wenzhe Liu , Yeming Chen , Yiming Wu , Heming Zheng , Cheng Cheng This is my paper · ORCID
classification cs.CVcs.AI
keywords vision-languagepre-trainingpanopticsegmentationgraphattentionnetworkspikingneuralcontrastivelearningvisualquestionansweringimage-textretrievalsemanticrepresentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to close the semantic gap between images and text by changing how images are tokenized and encoded. It claims that panoptic segmentation masks — object-level regions that include both countable 'thing' and background 'stuff' classes — are better visual tokens than detector bounding boxes or fixed patches, because they preserve object boundaries and context without irrelevant background. On top of those mask tokens, the paper builds a Graph Spiking Hybrid Network (GSHN) that runs a graph attention network for continuous semantic features and a spiking neural network for discrete spike-based features, fusing the two through a learned weight ratio and a semantic memory unit. Contrastive learning reorders similarly matched graph nodes into a spike stream, and a pre-training task called Spiked Text Learning (STL) supervises the discrete encoder using text features. If the reported experiments are right, this hybrid discrete-continuous visual encoder delivers competitive or state-of-the-art results on VQAv2, VQA-CP v2, GQA, SNLI-VE, NLVR2, and image-text retrieval.

What carries the argument

The machinery is the Graph Spiking Hybrid Network (GSHN), a dual encoder whose input nodes are mask instances from panoptic segmentation. A graph attention network (GAT) computes continuous node features by aggregating neighbors with learned attention weights; independently, a leaky integrate-and-fire (LIF) spiking neural network samples those features into binary spike sequences over a time window, and a trainable semantic memory unit selects and combines spike activations into discrete features. A squeeze-excitation-style weight ratio, computed from the continuous output, rescales the discrete features before they are added to the continuous features, and the sum is fed to the multimodal Transformer. The STL pre-training task masks a small fraction of SNN outputs and predicts them from text features, which is what ties the discrete encoder to language.

What would settle it

Run the GSHN pipeline with several different adjacency definitions for the same masks, for example a fully connected graph, a graph with edges only between spatially adjacent masks, and a graph with random edges, and compare VQAv2 or retrieval scores; if accuracy barely changes, the graph attention encoder is not doing the semantic-relation work the paper claims, and if it changes a lot, the missing specification makes the published numbers ambiguous.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a vision-language model can be improved by treating an image as a graph of panoptic mask regions and encoding those regions twice: once through a GAT that produces continuous node embeddings, and once through an SNN whose binary spike outputs are combined with a semantic memory unit to produce discrete semantic features. The two representations are fused by an information weight ratio derived from the continuous output, and the resulting image features are aligned with text through ITM, MLM, contrastive learning, and the STL pre-training task. The paper reports that this design reaches 73.39% on VQAv2 test-dev, 89.22% on SNLI-VE test, 84.85% on NLVR2 test-P, and 82.25% on MSCOCO 1K text-retrieval R@1, outperforming the box- and grid-based variants in its own ablations and matching or exceeding several published baselines.

Load-bearing premise

The paper assumes the connections between segmented objects can be defined well enough for the graph encoder to be meaningful, but it never specifies how those connections are computed from the panoptic masks.

Editorial extensions

If this is right

  • If GSHN is right, panoptic mask tokens are a viable replacement for detector boxes and patch grids as the visual input of vision-language pre-training models, with fine-grained boundaries and thing/stuff categories contributing to alignment.
  • The hybrid design implies that discrete spike-based features and continuous graph features carry complementary information, so models that fuse both should outperform either alone, which the paper's ablations support.
  • The contrastive reordering trick turns a batch of graph nodes into an ordered spike stream, meaning the SNN's temporal dimension is used for grouping similar semantics rather than for processing video, extending SNN use to static-image vision-language tasks.
  • The STL pre-training objective gives a text-supervised training signal directly to the discrete encoder, analogous to masked language modeling for spike outputs, which should improve discrete semantic encoding on any vision-language task that benefits from sparse visual features.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One extension the paper leaves implicit is that the method's ceiling likely depends on segmentation quality, since panoptic masks are produced by an upstream segmenter; using a stronger segmenter or making the mask encoder trainable end-to-end could improve results further.
  • The graph adjacency matrix is never specified, so a natural test is to vary how edges are defined, such as a full graph, k-nearest neighbors by mask overlap, or spatial adjacency; the sensitivity of results to this choice would reveal whether the GAT is learning relations or mostly passing features through.
  • The contrastive reordering assumes ITM matching scores are a good proxy for node-level semantic similarity; replacing ITM scores with direct feature-space nearest neighbors, or with mask-to-text grounding scores, could test that assumption and possibly improve the spike stream.
  • The STL task could be transferred to other discrete visual tokenizers, not just spike outputs, giving a text-supervised masked-token objective outside the SNN setting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a vision-language pre-training architecture called GSHN, which combines panoptic-segmentation mask tokens, a GAT-based continuous visual encoder, an SNN-based discrete semantic encoder, a semantic memory unit, contrastive learning over graph nodes, and a new pre-training task called Spiked Text Learning (STL). The authors report results on VQA, visual entailment, NLVR2, and image-text retrieval, claiming competitive or state-of-the-art performance on several benchmarks, including VQAv2 test-dev 73.39%, SNLI-VE test 89.22%, NLVR2 test-P 84.85%, and MSCOCO 1K TR R@1 82.25%. The paper also includes ablations on the information weight ratio, embedding manner, batch size, pre-training tasks, time window size, and semantic memory unit.

Significance. If the proposed hybrid continuous-discrete encoding pipeline worked as described and the reported results were reproducible, the work would be of interest to the vision-language and neuromorphic computing communities by demonstrating a practical use of SNNs in multimodal pre-training. The experimental scope is broad, covering four mainstream VL tasks with standard benchmarks, and the panoptic-segmentation visual tokenization is a plausibly useful idea. However, the paper currently contains a direct internal contradiction in its central ablation, under-specified core components, and a mislabeled loss function, so the significance of the claimed contribution cannot be assessed from the manuscript as written. No code or implementation details are provided to support reproducibility.

major comments (4)
  1. [§IV-C, Table VII] The ablation table directly contradicts the text and undermines the central architectural claim. In Table VII, the GAT+GAT row reports VQAv2 test-dev accuracy of 82.13%, while the GSHN row reports 73.39%; the text immediately below states that 'GSHN outperformed GAT and GAT+GAT'. Since GAT+GAT is a non-hybrid continuous baseline, a correct table would require GSHN to be higher, not lower. If the table is accurate, the paper's own evidence refutes the claim that the SNN/GAT fusion improves over the continuous-only model on the headline VQA benchmark. If the table is mislabeled or the numbers are transposed, the experimental record needs a correction. Either way, the central performance claim cannot be accepted from the manuscript as written.
  2. [§III-C, Eqs. (3)–(7)] The adjacency matrix A that defines the graph for the GAT encoder is never specified. Section III-C introduces G = (X, A) and defines attention weights α_ij over neighboring nodes j ∈ N_i, but the manuscript does not state how A is constructed from the panoptic masks, how N_i is determined, or whether A is learned, thresholded, or fixed. Every GAT output F_GAT in Eqs. (3)–(7) depends on A and N_i, so without this information the continuous semantic encoder is underspecified and not reproducible from the paper.
  3. [§III-E, Eq. (15)] Eq. (15) is presented as the focal loss for the STL task, but the formula is exactly binary cross-entropy summed over tokens and classes: −(1/N) Σ Σ [y log(ŷ) + (1−y) log(1−ŷ)]. There are no focal-weighting terms such as (1−p_t)^γ or α-balancing. The text says focal loss is used to reduce the weight of easy samples, but the displayed loss does not implement this. Either the formula or the description must be corrected.
  4. [§III-D] The contrastive learning input reordering is operationally underspecified. The text says that node features are reordered by the highest ITM matching scores and that a retrieve function recalls similar samples, but it does not define how ITM scores are computed for individual graph nodes, how many samples are grouped, what the positive/negative pair construction is, or how the reordered 'visual stream' is aligned with the SNN time dimension. The claim that ITM matching scores identify semantically similar graph nodes is an assumption that is never validated, so the CL-based hybrid transmission mechanism is not fully specified.
minor comments (5)
  1. [Section III title] The section title contains a typo: 'Hybird' should be 'Hybrid'.
  2. [§IV-C, Table VII] Table VII uses the column header 'Parameter (M), Time (h)' but the Time column is not part of the parameter count; please separate the two quantities or rename the columns to avoid confusion.
  3. [§III-C, Eq. (10)] The notation 'Acc_b_{i=1}' in Eq. (10) is undefined; please define the accumulation operation clearly, including how the batch index i maps to the time window T.
  4. [§IV-C, Table V] Table V reports GAT at 61.08 VQAv2 test-dev, whereas Table VII reports GAT at 66.93 VQAv2 test-dev under what appears to be the same single-GAT setting; please clarify the difference in configuration or correct the inconsistency.
  5. [§III-E, Eq. (14)] The masking rule 'SNN_prob=0.05 = STL(S, others, W)' is not explained clearly; the reader cannot tell what 'others' refers to or how the 5% mask probability is applied to the 3000-dimensional spike output.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: GSHN's empirical comparisons and ablations are self-contained; the self-citations are not load-bearing.

full rationale

The paper's central claim is empirical: GSHN is trained on standard VLP corpora (VG and MSCOCO) and evaluated on held-out VQA, VE, NLVR2, and image-text retrieval benchmarks. The claimed improvements are supported by comparisons against external baselines and by ablations (Tables II-VI and VIII-XIII), not by a derivation from assumed outputs. The GAT and SNN encoders are defined by standard equations (Eqs. 3-13) with learned parameters; the STL pre-training task is a self-supervised prediction of masked SNN outputs from text features, so its training objective is not the downstream metric. The only self-citations are [31] (STBP-tdBN training rule) and [47] (LOIS baseline); [31] is cited as a training technique and [47] as a comparison method, and neither carries the paper's conclusion. The under-specified adjacency matrix A in Section III-C and the apparent contradiction between Table VII and the text in Section IV-C are serious reproducibility and correctness concerns, but they are not cases where a prediction is equivalent to an input by construction. No step reduces to its own input, so no significant circularity is found.

Assumptions & free parameters 5 free parameters · 6 assumptions · 1 invented entities

The model is an empirical system whose behavior depends on several hand-set hyperparameters and on under-specified modeling assumptions. The largest assumptions are that panoptic masks are the correct semantic tokens, that the graph adjacency can encode visual relations (though it is never defined), and that ITM-based reordering identifies semantically similar nodes. No new physical or mathematical entity with external falsifiable handle is introduced beyond the semantic memory unit, which is a model component.

free parameters (5)
  • batch_size = 16
    Selected from a sweep on downstream test sets (Table IV); large batches add noise, small batches waste SNN training, per the paper.
  • SNN_time_window_T = unspecified (Fig. 3 tests 1, 5, 10, 20)
    Time window determines spike sequence length in Eq. (10); final value used for main results is not stated.
  • semantic_memory_capacity = 3000
    Capacity of semantic memory unit M and length of spike sequence S; set in Section IV-A without sensitivity analysis.
  • STL_mask_probability = 0.05
    5% chance to mask SNN output in Eq. (14); chosen ad hoc, no sensitivity study.
  • Mask R-CNN NMS and confidence thresholds = 0.5 and 0.6
    Detector thresholds set in Section IV-A; affect the quality of mask tokens.
assumptions (6)
  • domain assumption Panoptic segmentation masks (Mask R-CNN + DeepLab) partition an image into semantic units suitable for vision-language alignment.
    Section III-B builds all visual tokens from these masks; segmentation errors propagate to tokens.
  • domain assumption A graph of mask instances with attention weights captures local and global visual relations.
    Section III-C assumes GAT on mask graphs models context; adjacency matrix A is never defined.
  • domain assumption LIF neurons with tanh surrogate gradients can be trained end-to-end via STBP-tdBN and preserve needed semantics in binary spikes.
    Section III-C Eqs. (8)-(10); no analysis of information loss from binarization.
  • ad hoc to paper ITM matching scores identify semantically similar graph nodes for contrastive reordering.
    Section III-D uses ITM scores to reorder batch; no validation of similarity claim.
  • ad hoc to paper Text features are sufficient supervision for predicting masked spike outputs (STL).
    Section III-E introduces STL; only ablation evidence, no analysis of why text can supervise spikes.
  • domain assumption Standard VL benchmarks (VQAv2, GQA, SNLI-VE, NLVR2, Flickr30K, MSCOCO) measure alignment quality.
    Section IV uses them as ground truth; potential biases not discussed.
invented entities (1)
  • Semantic memory unit M (dimension 3000)
    purpose: Selects and combines basic semantic features from SNN spike outputs into discrete visual semantics (Eq. 11).
    Introduced in Section III-D; ablation in Fig. 4 shows it helps, but no external or theoretical evidence supports its necessity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving vision-language alignment with graph spiking hybrid Networks." pith.science (2026). https://pith.science/paper/Z7PBSYAL

@misc{pith2026250119069,
  author       = {Pith},
  title        = {Pith review of: Improving vision-language alignment with graph spiking hybrid Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z7PBSYAL}},
  note         = {Machine review of arXiv:2501.19069}
}
read the original abstract

To bridge the semantic gap between vision and language (VL), it is necessary to develop a good alignment strategy, which includes handling semantic diversity, abstract representation of visual information, and generalization ability of models. Recent works use detector-based bounding boxes or patches with regular partitions to represent visual semantics. While current paradigms have made strides, they are still insufficient for fully capturing the nuanced contextual relations among various objects. This paper proposes a comprehensive visual semantic representation module, necessitating the utilization of panoptic segmentation to generate coherent fine-grained semantic features. Furthermore, we propose a novel Graph Spiking Hybrid Network (GSHN) that integrates the complementary advantages of Spiking Neural Networks (SNNs) and Graph Attention Networks (GATs) to encode visual semantic information. Intriguingly, the model not only encodes the discrete and continuous latent variables of instances but also adeptly captures both local and global contextual features, thereby significantly enhancing the richness and diversity of semantic representations. Leveraging the spatiotemporal properties inherent in SNNs, we employ contrastive learning (CL) to enhance the similarity-based representation of embeddings. This strategy alleviates the computational overhead of the model and enriches meaningful visual representations by constructing positive and negative sample pairs. We design an innovative pre-training method, Spiked Text Learning (STL), which uses text features to improve the encoding ability of discrete semantics. Experiments show that the proposed GSHN exhibits promising results on multiple VL downstream tasks.

Figures

Figures reproduced from arXiv: 2501.19069 by the authors.

Figure 1
Figure 1. Overview of the GSHN architecture. We use panoptic segmentation to optimize the fine-grained image semantic representations. We also adopt [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Information transmission diagram. We encode GAT-based concrete (solid green pathway) and SNN-based discrete (solid blue pathway) visual semantic [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Performance testing for time window sizes on the VQAv2 dataset. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Performance comparison of the proposed GSHN with and without [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 59 canonical work pages

  1. [1]

    Positional attention guided transformer-like architecture for visual question answering,

    A. Mao, Z. Yang, K. Lin, J. Xuan, and Y . J. Liu, “Positional attention guided transformer-like architecture for visual question answering,” IEEE Trans. Multimedia , vol. 25, pp. 6997–7009, 2023

    work page 2023

  2. [2]

    AlignVE: Visual entailment recognition based on align- ment relations,

    B. Cao et al. , “AlignVE: Visual entailment recognition based on align- ment relations,” IEEE Trans. Multimedia , 2022

    work page 2022

  3. [3]

    Feature First: Advancing image-text retrieval through improved visual features,

    D. Wu et al. , “Feature First: Advancing image-text retrieval through improved visual features,” IEEE Trans. Multimedia , vol. 26, pp. 3827–3841, 2024

    work page 2024

  4. [4]

    Explicit cross-modal representation learning for visual commonsense reasoning,

    X. Zhang, F. Zhang, and C. Xu, “Explicit cross-modal representation learning for visual commonsense reasoning,” IEEE Trans. Multimedia , vol. 24, pp. 2986–2997, 2022

    work page 2022

  5. [5]

    UNITER: Universal Image-text representation learning,

    Y . C. Chen et al. , “UNITER: Universal Image-text representation learning,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV) , pp. 104–120, 2020

    work page 2020

  6. [6]

    Bottom-up and top-down attention for image captioning and visual question answering,

    P. Anderson et al. , “Bottom-up and top-down attention for image captioning and visual question answering,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , pp. 6077–6086, 2018

    work page 2018

  7. [7]

    In de- fense of grid features for visual question answering,

    H. Jiang, I. Misra, M. Rohrbach, E. Learned-Miller, and X. Chen, “In de- fense of grid features for visual question answering,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , pp.10267–10276, 2020

    work page 2020

  8. [8]

    Pixel-BERT: Align- ing image pixels with text by deep multi-modal transformers,

    Z. Huang, Z. Zeng, B. Liu, D. Fu, and J. Fu, “Pixel-BERT: Align- ing image pixels with text by deep multi-modal transformers,” 2020, arXiv:2004.00849. [Online]. Available: https://arxiv.org/abs/2004.00849

    arXiv 2020

Show all 66 references
  1. [9]

    ViLT: Vision-and-language transformer without convolution or region supervision,

    W. Kim, B. Son, and I. Kim, “ViLT: Vision-and-language transformer without convolution or region supervision,” in Proc. Int. Conf. Mach. Learn. PMLR , pp. 5583–5594, 2021

    2021

  2. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy et al. , “An image is worth 16x16 words: Transformers for image recognition at scale,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2021

    2021

  3. [11]

    Panoptic scene graph generation,

    J. Yang et al. , “Panoptic scene graph generation,” in Proc. Eur . Conf. Comput. Vis. (ECCV) , pp. 178–196, 2022

    2022

  4. [12]

    Seeing out of the box: end-to-end pre-training for vision-language representation learning,

    Z. Huang, Z. Zeng, Y . Huang, B. Liu, D. Fu, and J. Fu, “Seeing out of the box: end-to-end pre-training for vision-language representation learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pp.12976–12985, 2021

    2021

  5. [13]

    Probing inter-modality: Visual parsing with self-attention for vision-and-language pre-training,

    H. Xue et al., “Probing inter-modality: Visual parsing with self-attention for vision-and-language pre-training,” in Proc. AAAI Conf. Artif. Intell. , vol. 34, pp. 4514–4528, 2021

    2021

  6. [14]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu et al. , “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proc. IEEE/CVF Inter . Conf. Comput. Vis. (ICCV) , pp. 10012–10022, 2021

    2021

  7. [15]

    LXMERT: Learning cross-modality encoder representations from transformers,

    H. Tan and M. Bansal, “LXMERT: Learning cross-modality encoder representations from transformers,” in Proc. Conf. Empirical Methods Natural Lang. Process. (EMNLP) , 2019

    2019

  8. [16]

    Ernie-vil: Knowledge enhanced vision-language represen- tations through scene graphs,

    F. Yu et al. , “Ernie-vil: Knowledge enhanced vision-language represen- tations through scene graphs,” in Proc. AAAI Conf. Artif. Intell. , vol. 35, no. 4, pp. 3208–3216, 2021

    2021

  9. [17]

    12-in-1: Mul- titask vision and language representation learning,

    J. Lu, V . Goswami, M. Rohrbach, D. Parikh, and S. Lee, “12-in-1: Mul- titask vision and language representation learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , pp. 10437–10446, 2020

    2020

  10. [18]

    Large-scale adversarial training for vision-and-language representation learning,

    Z. Gan, Y . C. Chen, L. Li, C. Zhu, Y . Cheng, and J. Liu, “Large-scale adversarial training for vision-and-language representation learning,” in Proc. Eur . Conf. Comput. Vis. (ECCV) , vol. 33, pp. 6616–6628, 2020. 12

    2020

  11. [19]

    SemVLP: Vision-language pre-training by aligning seman- tics at multiple levels,

    C. Li et al., “SemVLP: Vision-language pre-training by aligning seman- tics at multiple levels,” 2021, arXiv: 2103.07829. [Online]. Available: http://arxiv.org/abs/2103.07829

    2021 arXiv

  12. [20]

    Scaling up dynamic graph representation learning via spiking neural networks,

    J. Li et al. , “Scaling up dynamic graph representation learning via spiking neural networks,” in Proc. AAAI Conf. Artif. Intell. , vol. 37, no. 7, pp. 8588–8596, 2023

    2023

  13. [21]

    Spiking-YOLO: Spiking neural network for energy-efficient object detection

    S. Kim, S. Park, B. Na, and S. Yoon, “Spiking-YOLO: Spiking neural network for energy-efficient object detection”, in Proc. AAAI Conf. Artif. Intell., vol. 34, no. 7, pp.11270–11277, 2020

    2020

  14. [22]

    Enabling spike-based backpropagation for training deep neural network architectures,

    C. Lee et al. , “Enabling spike-based backpropagation for training deep neural network architectures,” Frontiers in neuroscience , 14: 497482, 2020

    2020

  15. [23]

    Spiking deep residual networks,

    Y . Hu, H. Tang, and G. Pan, “Spiking deep residual networks,” IEEE Trans. Neural Networks Learn. Syst. , vol. 34, no. 8, pp. 5200–5205, Aug. 2021

    2021

  16. [24]

    Spiking graph convolutional networks,

    Z. Zhu et al. , “Spiking graph convolutional networks,” 2022, arXiv: 2205.02767. [Online]. Available: http://arxiv.org/abs/2205.02767

    2022 arXiv

  17. [25]

    Spiking transformers for event-based single object tracking,

    J. Zhang et al. , “Spiking transformers for event-based single object tracking,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 8801-8810, 2022

    2022

  18. [26]

    Masked spiking transformer,

    Z. Wang et al., “Masked spiking transformer,” in Proc. IEEE/CVF Inter . Conf. Comput. Vis. (ICCV) , pp. 1761–1771, 2023

    2023

  19. [27]

    Spikformer: When spiking neural network meets transformer,

    Z. Zhou et al. , “Spikformer: When spiking neural network meets transformer,” 2022, arXiv: 2209.15425 . [Online]. Available: http://arxiv.org/abs/2209.15425

    2022 arXiv

  20. [28]

    Convolutional feature masking for joint object and stuff segmentation,

    J. Dai, K. He, and J. Sun, “Convolutional feature masking for joint object and stuff segmentation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , pp. 3992–4000, 2015

    2015

  21. [29]

    Graph attention networks,

    P. Veli ˇckovi´c et al. , “Graph attention networks,” 2017, arXiv: 1710.10903. [Online]. Available: http://arxiv.org/abs/1710.10903

    2017 arXiv

  22. [30]

    Neuronal dynamics: from single neurons to networks and models of cognition

    W. Gerstner, W. M. Kistler, R. Naud, and L. Paninski, “Neuronal dynamics: from single neurons to networks and models of cognition”, Cambridge University Press , 2014

    2014

  23. [31]

    Going deeper with directly-trained larger spiking neural networks

    H. Zheng, Y . Wu, L. Deng, Y . Hu, and G. Li, “Going deeper with directly-trained larger spiking neural networks”, in Proc. AAAI Conf. Artif. Intell. , vol. 35, no. 12, pp. 11062–11070, 2021

    2021

  24. [32]

    Bert: pre-training of deep bidirectional transformers for language understanding

    J. Devlin, M. Chang, K. Lee, and K. Toutanova, “Bert: pre-training of deep bidirectional transformers for language understanding”, in Proc. Annual Meeting of the ACL , pp. 4171–4186, 2019

    2019

  25. [33]

    Uniter: universal image-text representation learning

    Y . Chen et al., “Uniter: universal image-text representation learning”, in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) , pp.104–120, 2020

    2020

  26. [34]

    Align before fuse: vision and language representation learning with momentum distillation

    J. Li et al. , “Align before fuse: vision and language representation learning with momentum distillation”, in Proc. Adv. Neural Inf. Process. Syst., vol. 34, pp. 9694–9705, 2021

    2021

  27. [35]

    Imagenet: a large-scale hierarchical image database

    J. Deng, W. Dong, R. Socher, L. Li, K. Li, and F. Li, “Imagenet: a large-scale hierarchical image database”, in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. , pp. 248–255, 2009

    2009

  28. [36]

    DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs,

    L. C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 40, no. 4, pp. 834–848, Apr. 2018

    2018

  29. [37]

    Mask R-CNN,

    K. He, G. Gkioxari, P. Doll ´ar, and R. Girshick, “Mask R-CNN,” in Proc. IEEE/CVF Inter . Conf. Comput. Vis. (ICCV) , 2017, pp. 2980–2988

    2017

  30. [38]

    Visual genome: connecting language and vision using crowdsourced dense image annotations,

    R. Krishna et al., “Visual genome: connecting language and vision using crowdsourced dense image annotations,” in Proc. Int. J. Comput. Vis. , vol. 123, pp. 32–73, 2017

    2017

  31. [39]

    Microsoft COCO: common objects in context

    T. Lin et al. , “Microsoft COCO: common objects in context”, in Proc. Eur . Conf. Comput. Vis., pp. 740–755, 2014

    2014

  32. [40]

    Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,

    B. A. Plummer et al. , “Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV) , pp. 2641–2649, 2015

    2015

  33. [41]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering,

    Y . Goyal et al., “Making the v in vqa matter: Elevating the role of image understanding in visual question answering,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , pp. 6904-6913, 2017

    2017

  34. [42]

    Dual attention and question categorization-based visual question answering,

    A. Mishra, A. Anand, and P. Guha, “Dual attention and question categorization-based visual question answering,” IEEE Trans. Artif. Intell., vol. 4, no. 1, pp. 81–91, 2023

    2023

  35. [43]

    Multitask learning for visual question answering,

    J. Ma, J. Liu, Q. Lin, B. Wu, Y . Wang, and Y . You, “Multitask learning for visual question answering,” IEEE Trans. Neural Networks Learn. Syst., vol. 34, no. 3, pp. 1380–1394, 2023

    2023

  36. [44]

    Bilinear graph networks for visual question answering,

    D. Guo, C. Xu, and D. Tao, “Bilinear graph networks for visual question answering,” IEEE Trans. Neural Networks Learn. Syst. , vol. 34, no. 2, pp. 1023–1034, 2023

    2023

  37. [45]

    Oscar: Object-semantics aligned pre-training for vision language tasks,

    X. Li et al. , “Oscar: Object-semantics aligned pre-training for vision language tasks,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS) , pp. 121–137, 2020

    2020

  38. [46]

    Position-guided text prompt for vision-language pre-training,

    J. Wang, P. Zhou, M. Z. Shou, and S. Yan, “Position-guided text prompt for vision-language pre-training,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , pp. 23242–23251, 2023

    2023

  39. [47]

    LOIS: Looking out of instance semantics for visual question answering,

    S. Zhang, Y . Chen, Y . Sun, F. Wang, H. Shi, and H. Wang, “LOIS: Looking out of instance semantics for visual question answering,” IEEE Trans. Multimedia, vol. 26, pp. 6202–6214, 2024

    2024

  40. [48]

    Answer again: Improving VQA with cascaded-answering model,

    L. Peng et al., “Answer again: Improving VQA with cascaded-answering model,” IEEE Trans. Knowl. Data Eng. , vol. 34, no. 4, pp. 1644–1655, Apr. 2022

    2022

  41. [49]

    MRANet: Improving VQA via multi-modal relation attention network,

    L. Peng, Y . Yang, Z. Wang, Z. Huang, and H. T. Shen, “MRANet: Improving VQA via multi-modal relation attention network,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 44, no. 1, pp. 318–329, Jan. 2022

    2022

  42. [50]

    LPF: a language-prior feedback objec- tive function for de-biased visual question answering,

    Z. Liang, H. Hu, and J. Zhu, “LPF: a language-prior feedback objec- tive function for de-biased visual question answering,” in Proc. 44th ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1955–1959, 2021

    1955

  43. [51]

    Greedy gradient ensemble for robust visual question answering,

    X. Han, S. Wang, C. Su, Q. Huang, and Q. Tian, “Greedy gradient ensemble for robust visual question answering,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV) , pp. 1584–1593, 2021

    2021

  44. [52]

    Coun- terfactual samples synthesizing for robust visual question answering,

    L. Chen, X. Yan, J. Xiao, H. Zhang, S. Pu, and Y . Zhuang, “Coun- terfactual samples synthesizing for robust visual question answering,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , pp.10800–10809, 2020

    2020

  45. [53]

    Lyrics: Boosting fine-grained language-vision align- ment and comprehension via semantic-aware visual objects,

    J. Lu et al. , “Lyrics: Boosting fine-grained language-vision align- ment and comprehension via semantic-aware visual objects,” 2023, arXiv:2312.05278. [Online]. Available: http://arxiv.org/abs/2312.05278

    2023 arXiv

  46. [54]

    Learning by abstraction: The neural state machine,

    D. Hudson and C. D. Manning, “Learning by abstraction: The neural state machine,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS) , vol. 32, 2019

    2019

  47. [55]

    MDETR-modulated detection for end-to-end multimodal understand- ing,

    A. Kamath, M. Singh, Y . LeCun, G. Synnaeve, I. Misra, and N. Carion, “MDETR-modulated detection for end-to-end multimodal understand- ing,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV) , pp. 1780–1790, 2021

    2021

  48. [56]

    Cumo: Scaling multimodal LLM with co-upcycled mixture-of-experts,

    J. Li et al. , “Cumo: Scaling multimodal LLM with co-upcycled mixture-of-experts,” 2024, arXiv: 2405.05949 . [Online]. Available: http://arxiv.org/abs/2405.05949

    2024 arXiv

  49. [57]

    VinVL: Revisiting visual representations in vision language models,

    P. Zhang et al. , “VinVL: Revisiting visual representations in vision language models,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , pp. 5575–5584, 2021

    2021

  50. [58]

    An empirical study of training end-to-end vision-and language transformers,

    Z. Y . Dou et al. , “An empirical study of training end-to-end vision-and language transformers,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , pp.18166–18176, 2022

    2022

  51. [59]

    SimVLM: Simple visual language model pretraining with weak supervision,

    Z. Wang, J. Yu, A. W. Yu, Z. Dai, Y . Tsvetkov, and Y . Cao, “SimVLM: Simple visual language model pretraining with weak supervision,” in Proc. Int. Conf. Learn. Represent. (ICLR) , 2022

    2022

  52. [60]

    J. Yu, Z. Wang, V . Vasudevan, L. Yeung, M. Seyedhosseini, and Y . Wu, “Coca: contrastive captioners are image-text foundation models, 2022, arXiv:2205.01917. [Online]. Available: http://arxiv.org/abs/2205.01917

    2022 arXiv

  53. [61]

    OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework,

    P. Wang et al. , “OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework,” in Proc. Int. Conf. Mach. Learn. (ICML) , pp. 23318–23340, 2022

    2022

  54. [62]

    Learning the best pooling strategy for visual semantic embedding,

    J. Chen, H. Hu, H. Wu, Y . Jiang, and C. Wang, “Learning the best pooling strategy for visual semantic embedding,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , pp. 15789–15798, 2021

    2021

  55. [63]

    ViSTA: Vision and scene text aggregation for cross-modal retrieval,

    M. Cheng et al. , “ViSTA: Vision and scene text aggregation for cross-modal retrieval,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , pp. 5184–5193, 2022

    2022

  56. [64]

    Implicit differentiable outlier detection enables robust deep multimodal analysis,

    Z. Wang, S. Medya, and S. N. Ravi, “Implicit differentiable outlier detection enables robust deep multimodal analysis,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS) , 36:13854–13872, 2023

    2023

  57. [65]

    Multi-grained vision language pretrain- ing: Aligning texts with visual concepts,

    Y . Zeng, X. Zhang, and H. Li, “Multi-grained vision language pretrain- ing: Aligning texts with visual concepts,” in Proc. Int. Conf. Mach. Learn. PMLR , pp. 25994–26009, 2022

    2022

  58. [66]

    Diet-snn: A low-latency spiking neural net- work with direct input encoding and leakage and threshold optimiza- tion,

    N. Rathi and K. Roy, “Diet-snn: A low-latency spiking neural net- work with direct input encoding and leakage and threshold optimiza- tion,” IEEE Trans. Neural Networks Learn. Syst. , vol. 34, no. 6, pp. 3174–3182, 2021

    2021

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.