Pith. sign in

REVIEW 3 major objections 5 minor 70 references

Query-centric Audio-Visual Cognition Network for Moment Retrieval, Segmentation and Step-Captioning

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a query-centric audio-visual network, which aligns audio and video before filtering the fused representation with the text query, achieves state-of-the-art results on HIREST moment retrieval and segmentation and…

desk verdict A solid incremental contribution to HIREST with a real but fixable gap: the paper never explains how Whisper transcriptions become the Nv-length audio tensor, so the central audio-visual synergy claim is underdetermined as written. read the letter →

arxiv 2412.13543 v1 pith:DZBM52HB submitted 2024-12-18 cs.CV cs.AIcs.CL

Yunbin Tu , Liang Li , Li Su , Qingming Huang This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords momentretrievalsegmentationstep-captioningaudio-visualrepresentationquery-centricmulti-tasklearningHIRESTcontrastive
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to establish that a query-centric audio-visual representation—built by first aligning visual and audio at both global and local levels and only then letting the text query filter the fused content—is a stronger foundation for the three HIREST tasks than the direct element-wise fusion used by the Joint baseline. On the HIREST benchmark, the proposed QUAG network reports the best moment-retrieval results (Recall@0.5 IoU of 72.54, Recall@0.7 IoU of 38.86), the best moment-segmentation results (Recall@0.5 IoU of 39.27, Precision@0.5 IoU of 31.68), and competitive step-captioning (CIDEr of 25.44, entailment of 40.10) against task-specific models. The authors also show the same representation transfers to query-based video summarization on TVSum, where QUAG reaches an average top-5 mAP of 87.0, on par with a task-specific baseline. If the claim holds, it suggests that explicitly structuring modality hierarchies—shallow audio-visual perception before deep query cognition—matters more than the capacity of the fusion operation itself.

What carries the argument

The central object is the query-centric audio-visual representation $\tilde{R}_m$, produced by two modules. Modality-synergistic perception first applies an InfoNCE contrastive loss to the mean-pooled visual and audio features so the two modalities share an embedding space, then exchanges local token-level information through multi-head cross-attention and fuses the two streams with a linear layer. Query-centric cognition then concatenates the text query with the audio-visual representation, projects the result to a temporal attention vector $A_{te}$ and a channel attention vector $A_{ch}$ through sigmoid-gated linear layers, and multiplies their element-wise product into the audio-visual representation; a self-attended version of the query-augmented representation is added back. This filtered, query-centric representation is fed to a transformer encoder and the prediction heads. The load-bearing property is that the query acts as a multiplicative temporal-channel gate on audio-visual content that has already been brought into alignment, rather than being fused with the visual and audio streams from the start.

What would settle it

Replace the audio stream with a constant vector repeated on every frame and retrain QUAG on HIREST; if moment-retrieval Recall@0.5 stays within one point of 72.54, the claimed audio-visual synergy is not what drives the result.

Watch

Extended reading notes

Core claim

The paper's discovery is that replacing undifferentiated multimodal fusion with a two-stage, shallow-to-deep processing pipeline improves all three HIREST tasks. In the modality-synergistic perception stage, an InfoNCE contrastive loss pulls global visual and audio features into a shared space, then multi-head cross-attention exchanges local token-level information between the two modalities before a linear fusion produces an audio-visual representation. In the query-centric cognition stage, the text query is concatenated with that representation, projected to temporal and channel attention maps via sigmoid-gated linear layers, and the element-wise product of these maps filters the audio-visual representation; the filtered output is then combined with a self-attended version of the query-augmented representation. QUAG reports state-of-the-art numbers on HIREST moment retrieval and moment segmentation, and improves step-captioning over the Joint baseline on every metric, with particularly large gains on CIDEr (from 21.19 to 25.44) and entailment (from 35.97 to 40.10). The authors argue this validates that association relations across modalities are best modeled before the query filters content, rather than fused indiscriminately with it.

Load-bearing premise

The paper assumes the audio representation lines up with the video frame by frame, but it never describes how the transcribed speech is laid onto the frame timeline; if each frame just receives the same sentence embedding, the audio-visual interaction has no frame-level timing to work with.

Editorial extensions

If this is right

  • Moment retrieval on HIREST improves over the Joint baseline by 2.2 points at Recall@0.5 IoU and 4.2 points at Recall@0.7 IoU, and it also exceeds the strong task-specific baselines such as UVCOM and QD-DETR.
  • Moment segmentation improves by 3.0 points in Recall@0.5 IoU and 3.1 points in Precision@0.5 IoU over the Joint baseline, indicating more accurate step-boundary localization.
  • Step-captioning benefits from the query-centric representation: CIDEr rises from 21.19 to 25.44, SPICE from 3.02 to 4.49, and entailment from 35.97 to 40.10 compared with Joint.
  • The same query-centric audio-visual representation transfers to query-based video summarization on TVSum, reaching an average top-5 mAP of 87.0 that is on par with the task-specific TR-DETR (87.1).
  • Ablations show that the two modules complement each other: with both MSP and QC2, CIDEr improves 20.1 percent and SPICE 48.7 percent over the Joint baseline, whereas each module alone gives smaller or inconsistent gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper never specifies how Whisper transcriptions are mapped to the Nv frame positions, the audio stream may be a single sentence embedding repeated across frames; if that is the case, the MSP cross-attention cannot be doing frame-level audio-visual alignment, and the gains would come mostly from visual features and the query gate. A direct ablation would replace the audio stream with a
  • The QC2 temporal-channel gate is essentially a learned multiplicative mask; the same mask can be applied to other query-conditioned video tasks such as highlight detection, dense video captioning, or weakly supervised moment grounding, and compared with additive fusion to see whether the gating form, rather than the specific encoders, carries the benefit.
  • On TVSum the features are clip-level I3D and PANN rather than the EVA-CLIP, Whisper, and MiniLM features used for HIREST, so the reported generalization may reflect the gating structure rather than the audio-visual encoders; testing QUAG with identical feature extractors on both datasets would separate those factors.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes QUAG, a query-centric audio-visual cognition network for three HIREST subtasks: moment retrieval, moment segmentation, and step-captioning. The architecture introduces a Modality-Synergistic Perception (MSP) module that applies global contrastive alignment (InfoNCE) and local cross-attention between visual and audio features, followed by a Query-Centric Cognition (QC2) module that performs temporal-channel filtering of the audio-visual representation using the text query. The resulting query-centric representation is fed into a multi-modal encoder and task heads. Experiments on the HIREST dataset report improvements over the Joint baseline and task-specific methods on moment retrieval and segmentation, competitive step-captioning performance, and additional experiments on TVSum for query-based video summarization.

Significance. If the reported results are statistically reliable, QUAG is a reasonable step toward unified multi-task video understanding, and the shallow-to-deep framing is intuitively appealing. The paper ships a public code repository, which strengthens reproducibility. However, the empirical support is weakened by the lack of error bars or significance tests, the small margin over strong baselines on moment retrieval, and a critical underspecification of how audio is temporally aligned with video. The proposed audio-visual synergy mechanism is the main novelty, so the missing alignment detail is a substantial obstacle to validating the central claim.

major comments (3)
  1. [Multi-modal Input Embedding / Modality-Synergistic Perception, Eq. (3)] The paper states that the audio representation Ra has length Nv equal to the video representation, but it never specifies how Whisper transcription, a variable-length token stream, is converted into Nv frame-aligned vectors. If the entire transcription is encoded by MiniLM into a single sentence vector and broadcast across all Nv positions, then in Eq. (3), MHCA(Rv, Ra, Ra) provides identical key and value vectors at every temporal position, so the claimed 'local fine-grained interaction' degenerates to a global audio bias. The InfoNCE loss in Eq. (2) is computed on mean-pooled vectors and cannot recover temporal audio structure. Because the paper attributes QUAG's gains over Joint to audio-visual synergy and to temporal-channel filtration, this missing alignment is load-bearing. Please describe the exact alignment procedure (e.g., Whisper word timestamps, sliding windows, or a learned temporal projection), or, if the implementation broadcasts a pooled embedding, justify why MSP still provides temporal audio information and provide an ablation with a properly aligned audio representation.
  2. [Tables 1-3, Performance Comparison on HIREST] No error bars, confidence intervals, or significance tests are reported for any of the HIREST results. The test set contains 1,391 video-query pairs, and the margins over the strongest task-specific baselines are narrow; for example, in Table 1 the gap to UVCOM is +0.42 R@0.5 and +0.33 R@0.7. The claim that QUAG achieves state-of-the-art results is therefore not robustly supported by the reported numbers. Please report the mean and standard deviation over at least three training seeds, and, if possible, a paired bootstrap test or significance test against the strongest baseline.
  3. [Supplementary, Study on the Trade-off Parameter λ, Eq. (15)] The supplementary reports step-captioning metrics for λ values 0, 0.0001, ..., 0.0006 and selects λ=0.0003, but it does not state whether these evaluations were performed on the validation split or the test split. If Table 6 reports test-set numbers for each λ, then the final test results are optimistically biased because the hyperparameter was selected directly on the test set. Please clarify the selection split; if λ was selected on the test set, re-run the evaluation with λ chosen on the validation split and update the main results accordingly.
minor comments (5)
  1. [Methodology, Eq. (10)] In the definition of Pstep, the variable bt is used in the formula, but the following sentence says 'where Wt ∈ RD×1 and be ∈ R1'; this should be bt, not be.
  2. [Abstract and Introduction] The paper claims 'SOTA results on HIREST' without qualification, but the experiments only address three of the four HIREST tasks; video retrieval is described in the task definition but never evaluated in the paper. Please qualify the claim to the three evaluated subtasks.
  3. [Supplementary, Ablation Study for Query-Centric Cognition (QC2)] The supplementary says 'The ablation study on Ate and Ach is shown in Table 4 of the main paper', but Table 4 reports the MSP/QC2 module ablation, not the Ate/Ach ablation; the Ate/Ach results actually appear in Table 7 of the supplementary. The cross-reference should be corrected.
  4. [Figure 2] The overview figure is dense and the small font sizes make the MSP and QC2 data flows difficult to follow; consider enlarging the figure or presenting the two modules in separate panels.
  5. [Multi-modal Input Embedding] The phrase 'the audio information helps perceptive the main objects' is ungrammatical; also, the query representation is described as Rt ∈ RD after projection, but the query consists of Nt tokens and the token-to-vector reduction is not described. Please clarify how the token-level query is pooled or projected.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: QUAG's MSP and QC2 components are ordinary trainable modules, and the reported HIREST gains are empirical results against external baselines rather than consequences of construction.

full rationale

The paper's derivation chain is self-contained. MSP combines an InfoNCE contrastive loss (Eq. 2) and multi-head cross-attention (Eq. 3) over visual and audio representations, while QC2 computes temporal and channel gates from the query and audio-visual representation (Eqs. 5-7) and adds a query-aware residual (Eq. 8). These are standard differentiable layers trained against the task losses (Eqs. 12-14) on the official HIREST split; no task label enters the definition of the representation in a way that would make the reported test metrics true by construction. The hyperparameter lambda is selected empirically and reported in the supplementary material, which is ordinary hyperparameter tuning rather than a fitted quantity renamed as a prediction. Self-citations to prior work by the same authors appear only in related work and are not load-bearing: no uniqueness theorem, ansatz, or definitional equivalence is imported from those citations. The underspecified mapping from Whisper transcription to the Nv-frame audio tensor is a reproducibility and correctness concern, not a circularity: even if the audio representation were broadcast, the model would still be trained and evaluated against external labels and baselines. The comparisons against QD-DETR, TR-DETR, UVCOM, Joint, and other externally implemented models provide independent empirical grounding for the central SOTA claim.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper relies on the HIREST task definition and pre-trained extractors as external inputs. The main implicit assumptions are the validity of the shallow-to-deep design principle, the feasibility of frame-aligned audio embeddings, and the non-interference of the three tasks under round-robin training. The principal free parameter is the contrastive-loss weight lambda, tuned on the benchmark itself.

free parameters (3)
  • lambda (trade-off for contrastive loss) = 0.0003 (HIREST), 0.3 (TVSum)
    Chosen by evaluating step-captioning metrics in Supplementary Table 6; the paper states it is set empirically on the HIREST dataset.
  • temperature tau in InfoNCE = not reported
    Temperature for the contrastive loss in Eq. (2); no value is given, so replication must guess or tune it.
  • hidden size, batch size, learning rate = 768, 5, 1e-5 (HIREST); 256, 4, 1e-3 (TVSum)
    Standard training hyperparameters; they are part of the method but not central to the claim.
assumptions (4)
  • domain assumption The shallow-to-deep hierarchy of perception and cognition is a valid design principle for multimodal video understanding.
    Introduced in the Introduction and used to structure MSP (shallow) and QC2 (deep); no empirical or theoretical evidence is provided that this ordering is necessary.
  • domain assumption Embedding the speech transcription with MiniLM yields an audio representation that can be aligned frame-by-frame with the video representation of length Nv.
    The Multi-modal Input Embedding section assumes Ra has length Nv; the paper does not describe how transcription segments are projected onto the frame timeline.
  • domain assumption The round-robin multi-task training with a single shared representation will not cause tasks to interfere destructively.
    Training uses Eq. (15) with one task per iteration; no task-interference analysis or regularization is provided.
  • domain assumption Standard evaluation metrics, such as R@IoU, METEOR, and CIDEr, are accepted proxies for the quality of moment localization, segmentation, and step captioning.
    The paper relies on these metrics without discussing their limitations for step-level evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Query-centric Audio-Visual Cognition Network for Moment Retrieval, Segmentation and Step-Captioning." pith.science (2026). https://pith.science/paper/DZBM52HB

@misc{pith2026241213543,
  author       = {Pith},
  title        = {Pith review of: Query-centric Audio-Visual Cognition Network for Moment Retrieval, Segmentation and Step-Captioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DZBM52HB}},
  note         = {Machine review of arXiv:2412.13543}
}
read the original abstract

Video has emerged as a favored multimedia format on the internet. To better gain video contents, a new topic HIREST is presented, including video retrieval, moment retrieval, moment segmentation, and step-captioning. The pioneering work chooses the pre-trained CLIP-based model for video retrieval, and leverages it as a feature extractor for other three challenging tasks solved in a multi-task learning paradigm. Nevertheless, this work struggles to learn the comprehensive cognition of user-preferred content, due to disregarding the hierarchies and association relations across modalities. In this paper, guided by the shallow-to-deep principle, we propose a query-centric audio-visual cognition (QUAG) network to construct a reliable multi-modal representation for moment retrieval, segmentation and step-captioning. Specifically, we first design the modality-synergistic perception to obtain rich audio-visual content, by modeling global contrastive alignment and local fine-grained interaction between visual and audio modalities. Then, we devise the query-centric cognition that uses the deep-level query to perform the temporal-channel filtration on the shallow-level audio-visual representation. This can cognize user-preferred content and thus attain a query-centric audio-visual representation for three tasks. Extensive experiments show QUAG achieves the SOTA results on HIREST. Further, we test QUAG on the query-based video summarization task and verify its good generalization.

Figures

Figures reproduced from arXiv: 2412.13543 by the authors.

Figure 1
Figure 1. The illustrative example consisting of moment [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overview of our method. Based on the principle of shallow-to-deep, we propose a query-centric audio-visual [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Given a text query “How to create a brunch menu”, and the ground-truth annotations, we compare the predicted and [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4-6
Figure 4-6. Figure 4-6: These cases include text queries, ground-truth [PITH_FULL_IMAGE:figures/full_fig_p010_4-6.png]
Figure 4
Figure 4. Figure 4: Given a text query “How to use a fire pit”, and the ground-truth annotations, we compare the predicted and generated [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Given a text query “How to clean resin”, and the ground-truth annotations, we compare the predicted and generated [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Given a text query “How to make a brooch”, and the ground-truth annotations, we compare the predicted and generated [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Given a text query “How to make glazed beans”, and the ground-truth annotations, we show a failure case of our [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 61 canonical work pages

  1. [1]

    Anderson, P.; Fernando, B.; Johnson, M.; and Gould, S. 2016. Spice: Semantic propositional image caption evaluation. In ECCV, 382--398

    work page 2016

  2. [2]

    Banerjee, S.; and Lavie, A. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In ACL, 65--72

    work page 2005

  3. [3]

    Carreira, J.; and Zisserman, A. 2017. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 6299--6308

    work page 2017

  4. [4]

    Cho, J.; Lei, J.; Tan, H.; and Bansal, M. 2021. Unifying vision-and-language tasks via text generation. In ICML, 1931--1942

    work page 2021

  5. [5]

    Cong, G.; Li, L.; Liu, Z.; Tu, Y.; Qin, W.; Zhang, S.; Yan, C.; Wang, W.; and Jiang, B. 2022. Ls-gan: iterative language-based image manipulation via long and short term consistency reasoning. In ACM MM, 4496--4504

    work page 2022

  6. [6]

    Cong, G.; Li, L.; Qi, Y.; Zha, Z.-J.; Wu, Q.; Wang, W.; Jiang, B.; Yang, M.-H.; and Huang, Q. 2023. Learning to dub movies via hierarchical prosody models. In CVPR, 14687--14697

    work page 2023

  7. [7]

    Dong, J.; Chen, X.; Zhang, M.; Yang, X.; Chen, S.; Li, X.; and Wang, X. 2022. Partially relevant video retrieval. In ACM MM, 246--257

    work page 2022

  8. [8]

    Fang, Y.; Wang, W.; Xie, B.; Sun, Q.; Wu, L.; Wang, X.; Huang, T.; Wang, X.; and Cao, Y. 2023. Eva: Exploring the limits of masked visual representation learning at scale. In CVPR, 19358--19369

    work page 2023

Show all 70 references
  1. [9]

    F.; Ellis, D

    Gemmeke, J. F.; Ellis, D. P.; Freedman, D.; Jansen, A.; Lawrence, W.; Moore, R. C.; Plakal, M.; and Ritter, M. 2017. Audio set: An ontology and human-labeled dataset for audio events. In ICASSP, 776--780

    2017

  2. [10]

    Iashin, V.; and Rahtu, E. 2020. A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformer. In BMVC

    2020

  3. [11]

    Kay, W.; Carreira, J.; Simonyan, K.; Zhang, B.; Hillier, C.; Vijayanarasimhan, S.; Viola, F.; Green, T.; Back, T.; Natsev, P.; et al. 2017. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950

    2017 arXiv

  4. [12]

    B.; Moon, J.; Choi, J.; and Kim, S

    Kim, M.; Kim, H. B.; Moon, J.; Choi, J.; and Kim, S. T. 2024. Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval. In CVPR, 13894--13904

    2024

  5. [13]

    C.; Lo, W.-Y.; et al

    Kirillov, A.; Mintun, E.; Ravi, N.; Mao, H.; Rolland, C.; Gustafson, L.; Xiao, T.; Whitehead, S.; Berg, A. C.; Lo, W.-Y.; et al. 2023. Segment anything. In ICCV, 4015--4026

    2023

  6. [14]

    Kong, Q.; Cao, Y.; Iqbal, T.; Wang, Y.; Wang, W.; and Plumbley, M. D. 2020. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28: 2880--2894

    2020

  7. [15]

    Krishna, R.; Hata, K.; Ren, F.; Fei-Fei, L.; and Carlos Niebles, J. 2017. Dense-captioning events in videos. In ICCV, 706--715

    2017

  8. [16]

    L.; and Bansal, M

    Lei, J.; Berg, T. L.; and Bansal, M. 2021. Detecting moments and highlights in videos via natural language queries. NeurIPS, 34: 11846--11858

    2021

  9. [17]

    L.; and Bansal, M

    Lei, J.; Yu, L.; Berg, T. L.; and Bansal, M. 2020. Tvr: A large-scale dataset for video-subtitle moment retrieval. In ECCV, 447--463

    2020

  10. [18]

    Li, H.; Yang, S.; Zhang, Y.; Tao, D.; and Yu, Z. 2023. Progressive Feature Mining and External Knowledge-Assisted Text-Pedestrian Image Retrieval. arXiv preprint arXiv:2308.11994

    2023 arXiv

  11. [19]

    Li, L.; Chen, Y.-C.; Cheng, Y.; Gan, Z.; Yu, L.; and Liu, J. 2020. HERO: Hierarchical Encoder for Video+ Language Omni-representation Pre-training. In EMNLP, 2046--2065

    2020

  12. [20]

    Li, L.; Gao, X.; Deng, J.; Tu, Y.; Zha, Z.-J.; and Huang, Q. 2022. Long short-term relation transformer with global gating for video captioning. IEEE Transactions on Image Processing, 31: 2726--2738

    2022

  13. [21]

    Li, Q.; Su, L.; Zhao, J.; Xia, L.; Cai, H.; Cheng, S.; Tang, H.; Wang, J.; and Yin, D. 2024. Text-Video Retrieval via Multi-Modal Hypergraph Networks. In WSDM, 369--377

    2024

  14. [22]

    Lin, C.-Y. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, 74--81

    2004

  15. [23]

    Lin, K.; Li, L.; Lin, C.-C.; Ahmed, F.; Gan, Z.; Liu, Z.; Lu, Y.; and Wang, L. 2022. Swinbert: End-to-end transformers with sparse attention for video captioning. In CVPR, 17949--17958

    2022

  16. [24]

    Liu, X.; Li, L.; Wang, S.; Zha, Z.-J.; Li, Z.; Tian, Q.; and Huang, Q. 2022 a . Entity-enhanced adaptive reconstruction network for weakly supervised referring expression grounding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3): 3003--3018

    2022

  17. [25]

    Liu, Y.; Li, S.; Wu, Y.; Chen, C.-W.; Shan, Y.; and Qie, X. 2022 b . Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection. In CVPR, 3042--3051

    2022

  18. [26]

    Loshchilov, I.; and Hutter, F. 2018. Decoupled Weight Decay Regularization. In ICLR

    2018

  19. [27]

    Moon, W.; Hyun, S.; Park, S.; Park, D.; and Heo, J.-P. 2023. Query-dependent video representation for moment retrieval and highlight detection. In CVPR, 23023--23033

    2023

  20. [28]

    Narasimhan, M.; Nagrani, A.; Sun, C.; Rubinstein, M.; Darrell, T.; Rohrbach, A.; and Schmid, C. 2022. Tl; dw? summarizing instructional videos with task relevance and cross-modal saliency. In ECCV, 540--557

    2022

  21. [29]

    Oord, A. v. d.; Li, Y.; and Vinyals, O. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748

    2018 arXiv

  22. [30]

    a ckstr \

    Parikh, A.; T \"a ckstr \"o m, O.; Das, D.; and Uszkoreit, J. 2016. A Decomposable Attention Model for Natural Language Inference. In EMNLP, 2249--2255

    2016

  23. [31]

    Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. 2019. Pytorch: An imperative style, high-performance deep learning library. NeurIPS, 32

    2019

  24. [32]

    E.; Neumann, M.; Iyyer, M.; Gardner, M.; Clark, C.; Lee, K.; and Zettlemoyer, L

    Peters, M. E.; Neumann, M.; Iyyer, M.; Gardner, M.; Clark, C.; Lee, K.; and Zettlemoyer, L. 2018. Deep Contextualized Word Representations. In NAACL-HLT, 2227--2237

    2018

  25. [33]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In ICML, 8748--8763

    2021

  26. [34]

    W.; Xu, T.; Brockman, G.; McLeavey, C.; and Sutskever, I

    Radford, A.; Kim, J. W.; Xu, T.; Brockman, G.; McLeavey, C.; and Sutskever, I. 2023. Robust speech recognition via large-scale weak supervision. In ICML, 28492--28518

    2023

  27. [35]

    Reimers, N.; and Gurevych, I. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In EMNLP-IJCNLP, 3982--3992

    2019

  28. [36]

    Sharghi, A.; Gong, B.; and Shah, M. 2016. Query-focused extractive video summarization. In ECCV, 3--19

    2016

  29. [37]

    S.; and Gong, B

    Sharghi, A.; Laurel, J. S.; and Gong, B. 2017. Query-focused video summarization: Dataset, evaluation, and a memory network based approach. In CVPR, 4788--4797

    2017

  30. [38]

    Song, Y.; Vallmitjana, J.; Stent, A.; and Jaimes, A. 2015. Tvsum: Summarizing web videos using titles. In CVPR, 5179--5187

    2015

  31. [39]

    Sun, H.; Zhou, M.; Chen, W.; and Xie, W. 2024. Tr-detr: Task-reciprocal transformer for joint moment retrieval and highlight detection. In AAAI, volume 38, 4998--5007

    2024

  32. [40]

    Sun, X.; Wang, X.; Gao, J.; Liu, Q.; and Zhou, X. 2022. You need to read again: Multi-granularity perception network for moment retrieval in videos. In SIGIR, 1022--1032

    2022

  33. [41]

    Tacca, M. C. 2011. Commonalities between perception and cognition. Frontiers in psychology, 2: 358

    2011

  34. [42]

    Tang, M.; Wang, Z.; Liu, Z.; Rao, F.; Li, D.; and Li, X. 2021. Clip4caption: Clip for video caption. In ACM MM, 4858--4862

    2021

  35. [43]

    Tang, W.; Li, L.; Liu, X.; Jin, L.; Tang, J.; and Li, Z. 2024. Context Disentangling and Prototype Inheriting for Robust Visual Grounding. IEEE Transactions on Pattern Analysis & Machine Intelligence, 46(05): 3213--3229

    2024

  36. [44]

    Tu, Y.; Li, L.; Su, L.; Gao, S.; Yan, C.; Zha, Z.-J.; Yu, Z.; and Huang, Q. 2022. I2 Transformer: Intra-and inter-relation embedding transformer for TV show captioning. IEEE Transactions on Image Processing, 31: 3565--3577

    2022

  37. [45]

    Tu, Y.; Li, L.; Su, L.; Yan, C.; and Huang, Q. 2024 a . Distractors-Immune Representation Learning with Cross-modal Contrastive Regularization for Change Captioning. In ECCV, 311--328

    2024

  38. [46]

    Tu, Y.; Li, L.; Su, L.; Zha, Z.-J.; and Huang, Q. 2024 b . SMART: Syntax-Calibrated Multi-Aspect Relation Transformer for Change Captioning. IEEE Transactions on Pattern Analysis & Machine Intelligence, 46(07): 4926--4943

    2024

  39. [47]

    Tu, Y.; Li, L.; Su, L.; Zha, Z.-J.; Yan, C.; and Huang, Q. 2023 a . Self-supervised cross-view representation reconstruction for change captioning. In ICCV, 2805--2815

    2023

  40. [48]

    Tu, Y.; Li, L.; Su, L.; Zha, Z.-J.; Yan, C.; and Huang, Q. 2024 c . Context-aware Difference Distilling for Multi-change Captioning. In ACL, 7941--7956

    2024

  41. [49]

    Tu, Y.; Zhou, C.; Guo, J.; Li, H.; Gao, S.; and Yu, Z. 2023 b . Relation-aware attention for video captioning via graph learning. Pattern Recognition, 136: 109204

    2023

  42. [50]

    N.; Kaiser, .; and Polosukhin, I

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention is all you need. In NeurIPS, 5998--6008

    2017

  43. [51]

    Vedantam, R.; Lawrence Zitnick, C.; and Parikh, D. 2015. Cider: Consensus-based image description evaluation. In CVPR, 4566--4575

    2015

  44. [52]

    Wang, L.; Huang, X.; Yu, Z.; Peng, H.; Gao, S.; Mao, C.; Huang, Y.; Dong, L.; and Yu, P. S. 2024 a . Zero-Shot Text Normalization via Cross-Lingual Knowledge Distillation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32: 4631--4646

    2024

  45. [53]

    Wang, L.; Yu, Z.; Gao, S.; Mao, C.; and Huang, Y. 2024 b . DETS: End-to-End Single-Stage Text-to-Speech Via Hierarchical Diffusion Gan Models. In ICASSP, 10916--10920

    2024

  46. [54]

    Wang, M.; Li, H.; Zhang, Y.; Li, J.; Xie, M.; and Tao, D. 2024 c . Dual-task Mutual Reinforcing Embedded Joint Video Paragraph Retrieval and Grounding. arXiv preprint arXiv:2411.17481

    2024 arXiv

  47. [55]

    Wu, W.; Luo, H.; Fang, B.; Wang, J.; and Ouyang, W. 2023. Cap4video: What can auxiliary captions do for text-video retrieval? In CVPR, 10704--10713

    2023

  48. [56]

    Xiao, Y.; Luo, Z.; Liu, Y.; Ma, Y.; Bian, H.; Ji, Y.; Yang, Y.; and Li, X. 2024. Bridging the gap: A unified video comprehension framework for moment retrieval and highlight detection. In CVPR, 18709--18719

    2024

  49. [57]

    Xie, M.; Wang, M.; Li, H.; Zhang, Y.; Tao, D.; and Yu, Z. 2024. Phrase Decoupling Cross-Modal Hierarchical Matching and Progressive Position Correction for Visual Grounding. arXiv preprint arXiv:2410.23570

    2024 arXiv

  50. [58]

    Xiong, B.; and Grauman, K. 2014. Detecting snap points in egocentric video with a web photo prior. In ECCV, 282--298

    2014

  51. [59]

    H.; Miech, A.; Pont-Tuset, J.; Laptev, I.; Sivic, J.; and Schmid, C

    Yang, A.; Nagrani, A.; Seo, P. H.; Miech, A.; Pont-Tuset, J.; Laptev, I.; Sivic, J.; and Schmid, C. 2023. Vid2seq: Large-scale pretraining of a visual language model for dense video captioning. In CVPR, 10714--10726

    2023

  52. [60]

    Yang, Y.; Zhuang, Y.; and Pan, Y. 2021. Multiple knowledge representation for big data artificial intelligence: framework, applications, and case studies. Frontiers of Information Technology & Electronic Engineering, 22(12): 1551--1558

    2021

  53. [61]

    Ye, Q.; Shen, X.; Gao, Y.; Wang, Z.; Bi, Q.; Li, P.; and Yang, G. 2021. Temporal cue guided video highlight detection with low-rank audio-visual fusion. In ICCV, 7950--7959

    2021

  54. [62]

    Yue, S.; Tu, Y.; Li, L.; Gao, S.; and Yu, Z. 2024. Multi-grained Representation Aggregating Transformer with Gating Cycle for Change Captioning. ACM Transactions on Multimedia Computing, Communications and Applications, 20(10): 321:1--321:23

    2024

  55. [63]

    Yue, S.; Tu, Y.; Li, L.; Yang, Y.; Gao, S.; and Yu, Z. 2023. I3n: Intra-and inter-representation interaction network for change captioning. IEEE Transactions on Multimedia, 25: 8828--8841

    2023

  56. [64]

    Zala, A.; Cho, J.; Kottur, S.; Chen, X.; Oguz, B.; Mehdad, Y.; and Bansal, M. 2023. Hierarchical video-moment retrieval and step-captioning. In CVPR, 23056--23065

    2023

  57. [65]

    Zha, Z.-J.; Liu, D.; Zhang, H.; Zhang, Y.; and Wu, F. 2019. Context-aware visual policy network for fine-grained image captioning. IEEE transactions on pattern analysis and machine intelligence, 44(2): 710--722

    2019

  58. [66]

    Zhang, B.; Li, L.; Wang, S.; Cai, S.; Zha, Z.-J.; Tian, Q.; and Huang, Q. 2024. Inductive State-Relabeling Adversarial Active Learning With Heuristic Clique Rescaling. IEEE Transactions on Pattern Analysis & Machine Intelligence, 46(12): 9780--9796

    2024

  59. [67]

    Zhang, D.; Zhang, H.; Tang, J.; Hua, X.-S.; and Sun, Q. 2020. Causal intervention for weakly-supervised semantic segmentation. NeurIPS, 33: 655--666

    2020

  60. [68]

    Zhou, L.; Xu, C.; and Corso, J. 2018. Towards automatic learning of procedures from web instructional videos. In AAAI, volume 32

    2018

  61. [69]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  62. [70]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.