Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Learning Musical Representations for Music Performance Question Answering

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Amuse, built around early cross-modal interaction and temporally aligned rhythm and source encoders, achieves the highest reported accuracy on both Music AVQA benchmarks.

desk verdict A well-engineered music AVQA model with a genuine architectural contribution, but an underspecified pretraining split could undermine the SOTA comparison and must be clarified. read the letter →

arxiv 2502.06710 v1 pith:5GEXQBUV submitted 2025-02-10 cs.CV cs.MMcs.SDeess.AS

Xingjian Diao , Chunhui Zhang , Tingxuan Wu , Ming Cheng , Zhongyu Ouyang , Weiyi Wu , Jiang Gui This is my paper · ORCID
classification cs.CVcs.MMcs.SDeess.AS
keywords audio-visualquestionansweringmusicperformanceunderstandingmultimodalearlyfusioncross-modaladaptersrhythmannotationsourceseparationtemporalalignmentYOLOregionsofinterest
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Amuse, a system for answering questions about music performance videos, and claims it outperforms prior audio-visual QA methods on both Music AVQA and Music AVQA-v2, reaching 83.52% and 85.16% overall accuracy. The motivating problem is that music performances have dense, continuous audio and tightly coupled sound and image, which general-purpose audio-visual QA models handle poorly. Amuse's answer is threefold: let vision, audio, and question tokens interact from early layers through cross-modal adapters; make musical characteristics explicit by annotating rhythm and instrument sources along time and training predictors on them; and add a lightweight detector's region features for instruments and performers as a shortcut. A sympathetic reader would take the paper's central claim to be that these three music-specific additions, not a larger backbone, are what lift accuracy.

What carries the argument

The load-bearing machinery is a multimodal interactive encoder: three transformers (Swin-V2 for video, HTS-AT for audio, and a language transformer for questions) connected by cross-modal adapters that compute attention between tokens of different modalities and inject the attended outputs into each other's next blocks. Around this backbone, Amuse adds per-modality rhythm and source encoders trained with MSE loss on automatically generated labels (beat-tracking BPM changes over six-second segments at a 25% threshold, and instrument-presence counts from Universal Source Separation), frozen after pretraining. A YOLOv8 detector supplies musical regions of interest as a local visual shortcut. The adapters enable early fusion, the temporal encoders tie musical characteristics to the time axis, and the RoI highway gives local visual evidence for visual and counting questions.

What would settle it

Shuffle or randomize the rhythm and source labels while keeping the model architecture and training protocol identical; if Music AVQA-v2 accuracy does not fall materially below the reported 85.16%, the claimed contribution of temporal musical annotation is not real. A second check is to have human annotators mark rhythm changes and instrument presence on a subset and measure agreement with the automatic labels.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a QA model built specifically for music performance can beat general audio-visual QA baselines by treating rhythm, instrument source, and temporal alignment as first-class supervision signals. Concretely, Amuse combines a multimodal interactive encoder with a rhythm encoder, a music source encoder, and a YOLOv8-based musical regions-of-interest highway. The experiments report the highest average accuracy on both Music AVQA test sets, and the ablations attribute most of the gain to the multimodal interactive encoder (removing it drops overall accuracy by 28.79 percentage points), with smaller but consistent contributions from the rhythm, source, and RoI modules.

Load-bearing premise

Everything rests on the automatically generated rhythm and instrument-source labels being true enough to the music: if the 25% BPM-change threshold and the source-separation counts do not match what humans or the questions actually care about, the temporal encoders are supervising the model with noise.

Editorial extensions

If this is right

  • Audio-visual QA in music settings can be improved without larger models, by fusing modalities early and adding temporally aligned musical features.
  • Counting questions benefit substantially from the rhythm and source encoders, with ablation drops of up to 5.55% on audio-visual counting.
  • Visual-region features from a light detector are what drive visual and audio-visual existential and counting accuracy, with a 13.40% drop when removed.
  • The released rhythm and source annotations can serve as supervision for future music AVQA models and may be reused as pretraining signals.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the temporal annotations are indeed what drive the gain, the same recipe should transfer to other dense-audio domains such as speech, sports, or ambient music by swapping the source-separation categories; the paper does not test this.
  • Editorial inference: the ablation suggests the audio branch of the source encoder matters most for temporal and audio-related questions, so a natural next experiment is to feed the model pretrained separated instrument stems rather than count labels.
  • Editorial inference: the 25% BPM-change threshold and six-second window are hand-picked; sweeping these hyperparameters against human rhythm judgments would reveal how much of the reported gain depends on label quality.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Amuse, a framework for audio-visual question answering (AVQA) on music performance videos. Amuse combines (i) three interactive transformers for vision, audio, and language with cross-modal adapters for early fusion, (ii) rhythm and music-source encoders pretrained with automatically generated temporal labels and then frozen, and (iii) a YOLOv8-based musical region-of-interest highway. The authors release rhythm/source annotations for the Music AVQA v1 and v2 datasets and report state-of-the-art overall accuracies of 83.52% and 85.16%, with ablations showing the contribution of each component.

Significance. If the reported results hold, Amuse would be a meaningful step for music-specific AVQA: the early-interaction encoder design is plausible, the temporal alignment of rhythm and source features is a sensible inductive bias for music, and the released annotations could support further research. The paper includes ablations for each proposed module and makes code available. The main significance is currently qualified by two concerns: the pretraining stage for the frozen encoders is not explicitly restricted to the training split, and all central numerical comparisons are reported without variance estimates or significance tests. These issues directly affect the state-of-the-art claim, so the significance can only be assessed after they are resolved.

major comments (3)
  1. [§4.1, §3.2] The pretraining stage for the rhythm and source encoders is not stated to be restricted to the training split. Section 4.1 says only that pretraining 'exclusively utilizes the rhythm and music source annotations,' and Section 3.2 describes an annotation pipeline that is fully automatic and applicable to every clip in the dataset. If the frozen encoders were pretrained on audio and visual features from test videos, the comparison against baselines trained only on labeled training data would be transductive and potentially inflated. Because the reported gains over prior work (e.g., +2.33 overall on v1, +9.72 on v2) are large enough that this effect could matter, the authors must specify which split(s) were used for pretraining and, if the full dataset was used, rerun the pretraining on the training split only and report the resulting test accuracies.
  2. [§4.2, Tables 2 and 3] All Amuse numbers are described as averages over three runs, but no standard deviations, confidence intervals, or significance tests are reported, and baseline numbers appear to be taken from prior papers without associated variance. Several margins are small: on Music AVQA v1, the overall gap over LSTTA is 2.33 points and Amuse is actually below LAST-Att on audio counting (84.61 vs. 85.71). Without error bars or a significance test, the claim that Amuse 'demonstrates impressive performance across all categories' is not statistically supported. Please report per-run results and variance for Amuse and, where possible, for rerun baselines, or at least state clearly which comparisons are not statistically distinguishable.
  3. [§3.2, Eqs. (4)–(6)] The automatically generated rhythm and source annotations are not validated. The 25% BPM threshold and the 6-second segment length are chosen by hand, and the Universal Source Separation outputs are used without reporting agreement with human judgment or with the QA question content. Since these labels are the supervision signal for the pretrained encoders and the ablations attribute substantial gains to rhythm and source modules (Table 4), noisy or misaligned labels could either overstate or understate the contribution of these components. The authors should provide annotation-quality evidence, for example agreement with a small human-annotated subset or a comparison of pretraining on predicted vs. gold labels where feasible.
minor comments (5)
  1. [§4.2, Table 2] The text says Amuse shows 'impressive performance across all categories,' but Table 2 shows LAST-Att is better on audio counting on v1 (85.71 vs. 84.61) and several baseline entries for average columns are missing. Please qualify the claim as 'overall average' and fill or footnote the missing entries.
  2. [§4.4] The sentence 'its final feature mappings exhibit approximate linearity' is not supported by the attention-importance analysis presented in that section; either provide a derivation or remove the claim.
  3. [§3.2] The source annotation procedure says 'Counts track the occurrences' without specifying which instrument classes are tracked, how counts are aggregated into labels, or what prediction target is used for the source predictor. Please specify the exact vocabulary and labeling format, since the released annotations are a stated contribution.
  4. [Figure 5] The importance scores in Figure 5 lack details on how attention outputs are aggregated and normalized across layers and runs. Please state the procedure and, ideally, report variance over the three runs.
  5. [Throughout] There are several wording and formatting issues: 'Music A VQA' is inconsistently spaced, 'convolutedly correlate' in Section 3.2 is awkward, and 'approximately linearity' in Section 4.4 should be 'approximately linear.' Please copyedit.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: external annotations and independent QA benchmarks; potential pretraining-split leakage is a fairness concern, not a circular derivation.

full rationale

The paper's claimed derivation is self-contained with respect to its inputs. The rhythm and source annotations (Eqs. 4-6 and the Universal Source Separation step in Section 3.2) are produced by an external beat tracker and an external universal source separation model, not from the QA answers or from Amuse's own predictions; the pretraining loss (Eq. 10) is supervised by these externally derived annotations. The final QA accuracies (Tables 2-3) are evaluated on the standard Music AVQA and Music AVQA-v2 test sets against independently published baselines, and the ablations (Table 4) remove modules rather than reusing fitted values as evidence. The only self-citation (Diao et al., 2023) appears in a related-work enumeration and is not load-bearing. The skeptical concern that rhythm/source pretraining may have used test-set audio and visual frames is a possible data-contamination or unfair-comparison issue, but it is not circular: the pretraining targets do not encode the QA answer labels, and the final evaluation remains independent of those targets. No circular step meeting the required burden of proof is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on pretrained encoders and automatic annotation tools rather than on fitted physical parameters. The main hand-set choices are the annotation threshold and segment length; no new entities are introduced.

free parameters (2)
  • rhythm change threshold multiplier = 0.25 x mean BPM
    Equation 5 sets the threshold at 25% of average BPM. This value is chosen by hand and directly determines which audio segments are labeled as rhythm changes.
  • annotation segment length = 6 seconds
    Equation 4 divides 60-second clips into 6-second segments for both rhythm and source annotation, affecting label granularity and temporal alignment.
assumptions (4)
  • domain assumption Pretrained Swin-V2, HTS-Audio Transformer, YOLOv8, and Universal Source Separation models transfer effectively to Music AVQA data.
    The framework builds on these pretrained models; if their representations fail on musical content, the claimed gains would vanish. Invoked in Sections 3.1 and 3.3.
  • domain assumption Beat tracking on 6-second segments provides reliable BPM values representative of perceived rhythm changes.
    Used in Eqs. 4-6; no validation against human rhythm annotations is reported.
  • domain assumption Universal Source Separation correctly identifies which instruments are present in each audio segment.
    Used in the source annotation step in Section 3.2; no accuracy numbers for the separation on music performance audio are provided.
  • domain assumption Cross-modal attention is an effective early-fusion mechanism for this task and does not introduce harmful noise.
    Table 4 ablations suggest it helps, but the architecture choice itself is assumed rather than proven across diverse settings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Musical Representations for Music Performance Question Answering." pith.science (2026). https://pith.science/paper/5GEXQBUV

@misc{pith2026250206710,
  author       = {Pith},
  title        = {Pith review of: Learning Musical Representations for Music Performance Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5GEXQBUV}},
  note         = {Machine review of arXiv:2502.06710}
}
read the original abstract

Music performances are representative scenarios for audio-visual modeling. Unlike common scenarios with sparse audio, music performances continuously involve dense audio signals throughout. While existing multimodal learning methods on the audio-video QA demonstrate impressive capabilities in general scenarios, they are incapable of dealing with fundamental problems within the music performances: they underexplore the interaction between the multimodal signals in performance and fail to consider the distinctive characteristics of instruments and music. Therefore, existing methods tend to answer questions regarding musical performances inaccurately. To bridge the above research gaps, (i) given the intricate multimodal interconnectivity inherent to music data, our primary backbone is designed to incorporate multimodal interactions within the context of music; (ii) to enable the model to learn music characteristics, we annotate and release rhythmic and music sources in the current music datasets; (iii) for time-aware audio-visual modeling, we align the model's music predictions with the temporal dimension. Our experiments show state-of-the-art effects on the Music AVQA datasets. Our code is available at https://github.com/xid32/Amuse.

Figures

Figures reproduced from arXiv: 2502.06710 by the authors.

Figure 1
Figure 1. Sparse audio in general videos vs. continuous [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Amuse framework integrates (a) and (b) combining multimodal interactive and musical-specialized representations for answering: (a) multimodal interactive encoder’s audio, vision, and question modules are interconnected via adapters that perform cross-modal attention. (b) Source/Rhythm encoders extract and encode musical-specific characteristics such as rhythm and sound sources. Universal Encoder incorporates pretrai… view at source ↗
Figure 3
Figure 3. Visual frames and audio spectrograms are [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Demonstration of audio-visual temporal and counting QA. We show examples that our model cor￾rectly handles audio-visual temporal and counting ques￾tions, while SoTA models LAVisH and DG-SCT fail. Audio Question Answering. For the ques￾tions specifically related to audi…
Figure 5
Figure 5. Figure 5: Distribution of importance scores for different [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learning

    cs.CV 2025-09 conditional novelty 4.0 of 10

    OpenVision 2 shows that a caption-only generative objective can match contrastive learning for multimodal vision encoders at lower training cost, scaling to 1B parameters.

Reference graph

Works this paper leans on

43 extracted references · 20 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Andrea Agostinelli, Timo I Denk, Zal \'a n Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. 2023. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325

    arXiv 2023

  4. [4]

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In International Conference on Computer Vision

    work page 2015

  5. [5]

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. 2020. Vggsound: A large-scale audio-visual dataset. In International Conference on Acoustics, Speech and Signal Processing

    2020

  6. [6]

    Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2022. Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection. In International Conference on Acoustics, Speech and Signal Processing

    work page 2022

  7. [7]

    Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre D \'e fossez. 2024. Simple and controllable music generation. In Advances in Neural Information Processing Systems

    work page 2024

  8. [8]

    Shuqi Dai, Huiran Yu, and Roger B Dannenberg. 2022. What is missing in deep music generation? a study of repetition and structure in popular music. arXiv preprint arXiv:2209.00182

    work page Pith review arXiv 2022

Show all 43 references
  1. [9]

    Xingjian Diao, Ming Cheng, and Shitong Cheng. 2023. Av-maskenhancer: Enhancing video representations through audio-visual masked autoencoder. In International Conference on Tools with Artificial Intelligence

    2023

  2. [10]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In Internation...

    2021

  3. [11]

    Haoyi Duan, Yan Xia, Zhou Mingze, Li Tang, Jieming Zhu, and Zhou Zhao. 2024. Cross-modal prompts: Adapting large pre-trained models for audio-visual downstream tasks. In Advances in Neural Information Processing Systems

    2024

  4. [12]

    Fayek and Justin Johnson

    Haytham M. Fayek and Justin Johnson. 2020. Temporal reasoning via audio question answering. Transactions on Audio, Speech, and Language Processing

    2020

  5. [13]

    Noa Garcia, Mayu Otani, Chenhui Chu, and Yuta Nakashima. 2020. Knowit vqa: Answering knowledge-based questions about videos. In AAAI Conference on Artificial Intelligence

    2020

  6. [14]

    Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, et al. 2024. Audiogpt: Understanding and generating speech, music, sound, and talking head. In AAAI Conference on Artificial Intelligence

    2024

  7. [15]

    Diederick P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations

    2015

  8. [16]

    Plumbley

    Qiuqiang Kong, Ke Chen, Haohe Liu, Xingjian Du, Taylor Berg-Kirkpatrick, Shlomo Dubnov, and Mark D. Plumbley. 2023. Universal source separation with weakly labelled data. arXiv preprint arXiv:2305.07447

    2023 arXiv

  9. [17]

    Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. 2018. Tvqa: Localized, compositional video question answering. In Conference on Empirical Methods in Natural Language Processing

    2018

  10. [18]

    Guangyao Li, Wenxuan Hou, and Di Hu. 2023 a . Progressive spatio-temporal perception for audio-visual question answering. In International Conference on Multimedia

    2023

  11. [19]

    Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. 2022. Learning to answer questions in dynamic audio-visual scenarios. In Conference on Computer Vision and Pattern Recognition

    2022

  12. [20]

    Guangyao Li, Yixin Xu, and Di Hu. 2023 b . Multi-scale attention for audio question answering. In Annual Conference of the International Speech Communication Association

    2023

  13. [21]

    Yan-Bo Lin, Yi-Lin Sung, Jie Lei, Mohit Bansal, and Gedas Bertasius. 2023. Vision transformers are parameter-efficient audio-visual learners. In Conference on Computer Vision and Pattern Recognition

    2023

  14. [22]

    Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, and Tuomas Virtanen. 2022. Clotho-aqa: A crowdsourced dataset for audio question answering. In European Signal Processing Conference

    2022

  15. [23]

    Hongye Liu, Xianhai Xie, Yang Gao, and Zhou Yu. 2023. Parameter-efficient transfer learning for audio-visual-language tasks. In International Conference on Multimedia

    2023

  16. [24]

    Xiulong Liu, Zhikang Dong, and Peng Zhang. 2024. Tackling data bias in music-avqa: Crafting a balanced dataset for unbiased question-answering. In Winter Conference on Applications of Computer Vision

    2024

  17. [25]

    Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. 2022. Swin transformer v2: Scaling up capacity and resolution. In Conference on Computer Vision and Pattern Recognition

    2022

  18. [26]

    Peiling Lu, Xin Xu, Chenfei Kang, Botao Yu, Chengyi Xing, Xu Tan, and Jiang Bian. 2023. Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110

    2023 arXiv

  19. [27]

    Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. 2011. Multimodal deep learning. In International Conference on Machine Learning

    2011

  20. [28]

    Sahithya Ravi, Aditya Chinchure, Leonid Sigal, Renjie Liao, and Vered Shwartz. 2023. Vlc-bert: visual question answering with contextualized commonsense knowledge. In Winter Conference on Applications of Computer Vision

    2023

  21. [29]

    Kuniaki Saito, Kihyuk Sohn, Chen-Yu Lee, and Yoshitaka Ushiku. 2024. Unsupervised llm adaptation for question answering. arXiv preprint arXiv:2402.12170

    2024 arXiv

  22. [30]

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2024. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems

    2024

  23. [31]

    Salakhutdinov

    Nitish Srivastava and Russ R. Salakhutdinov. 2012. Multimodal learning with deep boltzmann machines. In Advances in Neural Information Processing Systems

    2012

  24. [32]

    Parthasaarathy Sudarsanam and Tuomas Virtanen. 2023. Attention-based methods for audio question answering. In European Signal Processing Conference

    2023

  25. [33]

    Yapeng Tian, Dingzeyu Li, and Chenliang Xu. 2020. Unified multisensory perception: Weakly-supervised audio-visual video parsing. In European Conference on Computer Vision

    2020

  26. [34]

    Ultralytics. 2023. Yolov8. https://github.com/ultralytics/ultralytics. Accessed: 2023-05-26

    2023

  27. [35]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems

    2017

  28. [36]

    Ziyu Wang, Hao Li, Di Huang, and Amir M. Rahmani. 2024. Healthq: Unveiling questioning capabilities of llm chains in healthcare conversations. arXiv preprint arXiv:2409.19487

    2024 arXiv

  29. [37]

    Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. 2022. Avqa: A dataset for audio-visual question answering on videos. In International Conference on Multimedia

    2022

  30. [38]

    Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. 2024. Self-chained image-language model for video localization and question answering. In Advances in Neural Information Processing Systems

    2024

  31. [39]

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. 2019. Activitynet-qa: A dataset for understanding complex web videos via question answering. In AAAI Conference on Artificial Intelligence

    2019

  32. [40]

    Heeseung Yun, Youngjae Yu, Wonsuk Yang, Kangil Lee, and Gunhee Kim. 2021. Pano-avqa: Grounded audio-visual question answering on 360deg videos. In International Conference on Computer Vision

    2021

  33. [41]

    Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba. 2019. The sound of motions. In International Conference on Computer Vision

    2019

  34. [42]

    Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. 2018. The sound of pixels. In European Conference on Computer Vision

    2018

  35. [43]

    Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. Toolqa: A dataset for llm question answering with external tools. Advances in Neural Information Processing Systems

    2023

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.