Pith. sign in

REVIEW 4 major objections 6 minor 68 references

PR-DETR: Injecting Position and Relation Prior for Dense Video Captioning

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PR-DETR claims that injecting data-derived position and relation priors into a transformer-based dense video captioner improves both event localization and caption quality over implicit-learning baselines.

desk verdict A solid DETR-based dense video captioning paper with genuinely useful priors, but the ActivityNet results don't support the 'simultaneous improvement' headline. read the letter →

arxiv 2506.16082 v1 pith:E4MSYXAS submitted 2025-06-19 cs.CV

Yizhe Li , Sanping Zhou , Zheng Qin , Le Wang This is my paper · ORCID
classification cs.CV
keywords densevideocaptioningeventlocalizationdetectiontransformerposition-anchoredqueryrelationpriorencoderunderstandingdecoder
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Dense video captioning asks a model to find every event in an untrimmed video and describe each one in language. The paper argues that current transformer-based systems learn event locations and event relations purely implicitly, which demands large training sets and leaves the model with ambiguous search regions and inconsistent captions. PR-DETR tries to establish that two data-derived priors fix this: position-anchored queries seeded from the clustering centers of real event locations, and a relation mask built from overlap-aware distances between predicted event boundaries and injected into decoder self-attention. On ActivityNet Captions and YouCook2, the paper reports that both priors improve localization and captioning over strong end-to-end baselines with the same visual features, and that combining them gives the best result.

What carries the argument

Two components carry the argument. The position-anchored query turns the global statistics of event locations into a starting point: k-means centroids of ground-truth event centers and durations are projected through sinusoidal positional encoding into N event slots, K slot-attention iterations aggregate frame features into event-centric embeddings, and an MLP predicts offsets that turn centroids into scene-specific anchors $P = \{(c_n, d_n)\}_{n=1}^N$, supervised by a Hungarian-matched generalized-IoU proposal loss. The relation-enhanced decoder then makes event interaction explicit: at each layer it computes a pairwise overlap-aware distance $r(p_i, p_j) = (\log(\beta/d_i + 1), \log(d_i/d_j))$ from current event boundaries, embeds it, and maps it through a $1\times1$ convolution to a relation mask $\tilde{R} \in \mathbb{R}^{N\times N\times M}$ that is added to $QK^\top/\sqrt{D}$ before softmax in self-attention. Static anchors $P_{sta} = P$ keep the position prior fixed during decoding while the event anchors $P$ update layer by layer, so the mask tells the decoder which queries are temporally connected and should exchange semantic context.

What would settle it

Compute the rank correlation (e.g., Kendall's tau) between the pairwise location-correlation metric of Eq. (1) and the caption-similarity scores used in the paper's Fig. 3(b) on the validation sets; if the correlation is near zero or negative, the relation prior's motivation collapses. A second decisive check is to replace the learned relation mask with a randomly permuted mask and measure whether the reported CIDEr and SODAc gains survive; if they do, the explicit relation encoding is not carrying the claimed signal.

Watch

Extended reading notes

Core claim

The central claim is that event locations and event semantics in a long video are not arbitrary: long events tend to sit near the video center, and events that are temporally connected tend to share semantic content. A dense video captioner can exploit these regularities directly instead of learning them from scratch. Position-anchored queries are initialized at k-means centroids of ground-truth event boundaries, refined by slot-attention feature aggregation into scene-specific event anchors, and supervised by a proposal loss. A relation-enhanced decoder computes pairwise overlap-aware distances between anchors, encodes them into an attention mask, and adds this mask to self-attention so that interactions between temporally connected events are guided explicitly. With both priors, the model predicts more precise event boundaries and more coherent captions; the ablations attribute localization gains mainly to the position prior and captioning gains mainly to the relation prior.

Load-bearing premise

The load-bearing premise is that events that are temporally close or overlapping really do have semantically related captions; the paper supports this with one scatter plot and no statistical test, so if that correlation is weak or dataset-specific, the relation mask could stop helping.

Editorial extensions

If this is right

  • Event localization should improve on structured videos, because seeded queries eliminate implausible proposals; the paper reports higher precision and F1 on both YouCook2 (F1 27.29 to 28.87) and ActivityNet Captions (F1 54.91 to 55.59) against the same-feature baseline.
  • Caption coherence should improve, because the relation mask strengthens interactions among events sharing context; on YouCook2 CIDEr rises from 29.20 to 37.30 and SODAc from 4.95 to 5.43.
  • Training-data demand should drop, because competitive results are reached without pretraining on millions of extra videos; the injected priors carry part of the learning burden.
  • Robustness to varied event distributions should rise; on a mixed ActivityNet-YouCook2 training set, the method improves SODAc from 4.87 to 5.33 and F1 from 26.69 to 27.91 over the base model.
  • The two priors are complementary rather than redundant; ablations show the position prior mainly helps localization, the relation prior mainly helps captioning, and their combination outperforms either alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the paper motivates the relation prior with a scatter plot rather than a quantified correlation, so a natural extension is to measure the rank correlation between the location metric and caption similarity per dataset; the prediction would be that the relation mask's benefit grows with that correlation.
  • Inference: because the position prior is fit to training-set event clusters, out-of-distribution videos with atypical event layouts are a stress test; one would expect the anchor initialization to help less or even bias localization on such data.
  • Inference: the relation mask is computed from anchors that are noisy early in decoding, so gating or annealing the mask as anchors refine is a plausible improvement the paper does not explore.
  • Inference: the same recipe—clustering-based position seeding plus a relation mask over outputs—could transfer to other set-prediction tasks with structured targets, such as temporal action detection or dense paragraph captioning, wherever output coherence matters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. PR-DETR proposes two explicit priors for DETR-style dense video captioning: (1) position-anchored queries, initialized from k-means clustering of ground-truth event locations and refined through slot-attention feature aggregation into video-specific event anchors; and (2) an event relation encoder that computes an overlap-aware pairwise distance between predicted event anchors and converts it into an attention mask used in decoder self-attention. The model is trained end-to-end with a proposal loss supervising the anchor generation. On ActivityNet Captions and YouCook2, the paper reports improved localization (precision/F1) and captioning (BLEU4, METEOR, CIDEr, SODA_c) over PDVC and CM2 baselines reproduced with CLIP features, with component ablations on YouCook2 validating the two priors. The paper frames these results as evidence that injecting position and relation priors improves event localization and caption quality simultaneously.

Significance. If substantiated, the paper would provide a clean empirical demonstration that explicit position and relation priors can reduce the optimization burden of DETR-like dense video captioning and improve both localization and caption coherence. The main strengths are the simple, interpretable priors; the consistent CLIP-feature reproduction of PDVC and CM2; and the YouCook2 ablations that attribute gains to each component. However, the absence of ActivityNet ablations, missing error bars, and the SODA_c regression on ActivityNet mean the general claim of simultaneous improvement on both datasets is not yet established. The significance is therefore moderate: the idea is plausible and useful, but the evidence is incomplete.

major comments (4)
  1. [Table I (Section IV-B)] The caption-quality claim on ActivityNet is not fully supported: PR-DETR's SODA_c (6.13) is below CM2's (6.18), and the other caption deltas (BLEU4 +0.20, METEOR +0.17, CIDEr +0.15) are small and likely within run-to-run variation. Since no error bars or significance tests are reported anywhere (Tables I-III and Fig. 5), the paper has not demonstrated consistent caption improvement on ActivityNet, which weakens the abstract's claim of simultaneous improvement. Please report multiple seeds with standard deviations and a significance test for the key comparisons.
  2. [Section IV-C (Tables III-VII)] All component ablations are conducted only on the YouCook2 validation set. Without a Base/P/R/P+R ablation on ActivityNet, it is impossible to verify that the position and relation priors, rather than the additional feature-aggregator and relation-encoder parameters, are responsible for the gains on that dataset. Please include the same ablation on ActivityNet, or at least a Base versus full PR-DETR comparison there.
  3. [Section III-A and Eq. (8)] The relation prior's motivation rests on the claim that temporally connected events have more semantically related captions, which is supported only by a scatter plot (Fig. 3(b)) without a correlation coefficient or statistical test. Since Eq. (8) directly encodes this assumption, please quantify the correlation (e.g., Pearson or Spearman) on both datasets and show that the observed association is not dataset-specific.
  4. [Section IV-A (Datasets)] The statement that training uses a YouTube-accessible subset with 'approximately 7% less' than the original video count creates a potential confound when comparing with PDVC† and CM2. Please state explicitly whether these reproduced baselines were trained and evaluated on exactly the same video subset, and provide the exact train/validation/test video counts for each dataset and method.
minor comments (6)
  1. [Section III-E] The section heading 'Predction Heads' contains a typo and should read 'Prediction Heads'.
  2. [Section III-D, Eq. (8)] The sentence 'The overlap-aware distance of two temporally connected events is 0, whereas overlapping or distant events are computed as close to 1' is inconsistent with Eq. (8), where overlapping events give beta > 0 and hence a positive logarithmic value; please clarify what 'temporally connected' means and how the formula maps to the described ranges.
  3. [Fig. 3(b)] Fig. 3(b) lacks axis labels and a legend; adding them, along with a fitted line and correlation coefficient, would make the claimed relationship between location correlation and semantic similarity visible.
  4. [Table I] In Table I, the best ActivityNet SODA_c value is CM2's 6.18, not PR-DETR's 6.13; the bold formatting should be corrected to avoid misleading the reader.
  5. [Section IV-C, Table VI] The statement 'too large or too small lambda_prop leads to performance degradation' is not strictly true for all metrics (METEOR at lambda_prop=2 is 6.59, higher than at lambda_prop=1); please qualify the conclusion with respect to the primary metric or explain the discrepancy.
  6. [Section IV-C, Table V] Table V's first row (no checkmarks) should be labeled as Base+R to match Table III, making the component-wise comparison easier to follow.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the position and relation priors are learned model components, and no reported prediction reduces to a fitted input by construction.

full rationale

This is an empirical architecture paper rather than a derivation chain, and I found no step in which a claimed prediction is equivalent to an input by construction. The position prior is a set of k-means centroids computed from training ground-truth event locations; these centroids only initialize scene-specific queries, and the final event locations are produced by learned offsets conditioned on video features. The relation prior is an overlap-aware distance matrix computed from predicted event anchors during decoding and injected into self-attention via Eq. (8)-(11); caption and localization outputs still depend on learned attention, cross-attention, and prediction heads, so no metric is algebraically forced by the prior. The ablation in Table III shows that adding and removing the components changes measured performance, which is evidence that the components contribute empirically rather than being definitionally identical to the outputs. The paper's self-citation (Ref. [43]) appears only in a related-work string and is not load-bearing. Concerns about the unquantified location-semantics correlation in Fig. 3(b) and the ActivityNet SODA_c shortfall versus CM2 are evidence-strength and robustness issues, not circularity. The burden for this kind of paper is low, and the honest finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

PR-DETR introduces no new physical or conceptual entities. The free parameters are standard hyperparameters tuned on validation. The main data-derived quantity, k-means event centroids, is a legitimate use of training labels and is not a free parameter in the model itself.

free parameters (3)
  • Number of queries N = 10 for ActivityNet, 100 for YouCook2
    Sets the number of event slots and anchors. Chosen per dataset without justification.
  • Feature aggregator iterations K = 3
    Tuned on YouCook2 validation (Table IV); K=3 gives best CIDEr.
  • Proposal loss weight lambda_prop = 1
    Tuned on YouCook2 validation (Table VI); lambda_prop=1 yields best performance.
assumptions (4)
  • domain assumption Ground-truth event locations in the training set are representative of the test set distribution for k-means clustering.
    Used to initialize position anchors. If test videos have a different event distribution, the prior could mislead the model.
  • domain assumption Temporally adjacent or overlapping events have semantically related captions.
    Motivates the overlap-aware relation prior. Fig. 3(b) shows a scatter plot but no correlation coefficient or test.
  • domain assumption CLIP ViT-L/14 features at 1 FPS are sufficient for event localization and captioning.
    All experiments rely on these features; the results may not transfer to other feature extractors.
  • standard math The PDVC-style DETR architecture is a valid base model for dense video captioning.
    The paper builds entirely on PDVC and does not provide a derivation of the transformer properties.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PR-DETR: Injecting Position and Relation Prior for Dense Video Captioning." pith.science (2026). https://pith.science/paper/E4MSYXAS

@misc{pith2026250616082,
  author       = {Pith},
  title        = {Pith review of: PR-DETR: Injecting Position and Relation Prior for Dense Video Captioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E4MSYXAS}},
  note         = {Machine review of arXiv:2506.16082}
}
read the original abstract

Dense video captioning is a challenging task that aims to localize and caption multiple events in an untrimmed video. Recent studies mainly follow the transformer-based architecture to jointly perform the two sub-tasks, i.e., event localization and caption generation, in an end-to-end manner. Based on the general philosophy of detection transformer, these methods implicitly learn the event locations and event semantics, which requires a large amount of training data and limits the model's performance in practice. In this paper, we propose a novel dense video captioning framework, named PR-DETR, which injects the explicit position and relation prior into the detection transformer to improve the localization accuracy and caption quality, simultaneously. On the one hand, we first generate a set of position-anchored queries to provide the scene-specific position and semantic information about potential events as position prior, which serves as the initial event search regions to eliminate the implausible event proposals. On the other hand, we further design an event relation encoder to explicitly calculate the relationship between event boundaries as relation prior to guide the event interaction to improve the semantic coherence of the captions. Extensive ablation studies are conducted to verify the effectiveness of the position and relation prior. Experimental results also show the competitive performance of our method on ActivityNet Captions and YouCook2 datasets.

Figures

Figures reproduced from arXiv: 2506.16082 by the authors.

Figure 1
Figure 1. Illustration of the event locations and captions in different scenarios [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a) Visualization of query distribution with and without position prior. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Statistical analysis on dense video captioning datasets. (a) The ground [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Illustration of the proposed PR-DETR. We first obtain the video representations from the input video in feature extraction and encoding. Next, we [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Performance comparison between PDVC and our PR-DETR on the [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Visualization of dense video captioning results of PDVC and our PR-DETR on ActivityNet Captions and YouCook2 validation set. Lines in different [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 46 canonical work pages

  1. [1]

    Swinbert: End-to-end transformers with sparse attention for video captioning,

    K. Lin, L. Li, C.-C. Lin, F. Ahmed, Z. Gan, Z. Liu, Y . Lu, and L. Wang, “Swinbert: End-to-end transformers with sparse attention for video captioning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 17 949–17 958

    work page 2022

  2. [2]

    Univl: A unified video and language pre-training model for multimodal understanding and generation,

    H. Luo, L. Ji, B. Shi, H. Huang, N. Duan, T. Li, J. Li, T. Bharti, and M. Zhou, “Univl: A unified video and language pre-training model for multimodal understanding and generation,” 2020

    work page 2020

  3. [3]

    End-to-end generative pretraining for multimodal video captioning,

    P. H. Seo, A. Nagrani, A. Arnab, and C. Schmid, “End-to-end generative pretraining for multimodal video captioning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 17 959–17 968. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 10

    work page 2022

  4. [4]

    Memory- attended recurrent network for video captioning,

    W. Pei, J. Zhang, X. Wang, L. Ke, X. Shen, and Y .-W. Tai, “Memory- attended recurrent network for video captioning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 8347–8356

    work page 2019

  5. [5]

    Sports video captioning via attentive motion representation and group relationship modeling,

    M. Qi, Y . Wang, A. Li, and J. Luo, “Sports video captioning via attentive motion representation and group relationship modeling,” IEEE Transactions on Circuits and Systems for Video Technology , vol. 30, no. 8, pp. 2617–2633, 2019

    work page 2019

  6. [6]

    Reconstruction network for video captioning,

    B. Wang, L. Ma, W. Zhang, and W. Liu, “Reconstruction network for video captioning,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 7622–7631

    work page 2018

  7. [7]

    Concept-aware video captioning: Describing videos with effective prior information,

    B. Yang, M. Cao, and Y . Zou, “Concept-aware video captioning: Describing videos with effective prior information,” IEEE Transactions on Image Processing , vol. 32, pp. 5366–5378, 2023

    work page 2023

  8. [8]

    Dense- captioning events in videos,

    R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. Carlos Niebles, “Dense- captioning events in videos,” in Proceedings of the IEEE international conference on computer vision , 2017, pp. 706–715

    work page 2017

Show all 68 references
  1. [9]

    Jointly localizing and describing events for dense video captioning,

    Y . Li, T. Yao, Y . Pan, H. Chao, and T. Mei, “Jointly localizing and describing events for dense video captioning,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 7492–7500

    2018

  2. [10]

    End-to- end dense video captioning with parallel decoding,

    T. Wang, R. Zhang, Z. Lu, F. Zheng, R. Cheng, and P. Luo, “End-to- end dense video captioning with parallel decoding,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 6847–6857

    2021

  3. [11]

    Vid2seq: Large-scale pretraining of a visual language model for dense video captioning,

    A. Yang, A. Nagrani, P. H. Seo, A. Miech, J. Pont-Tuset, I. Laptev, J. Sivic, and C. Schmid, “Vid2seq: Large-scale pretraining of a visual language model for dense video captioning,” in CVPR, 2023

    2023

  4. [12]

    Cap4video: What can auxiliary captions do for text-video retrieval?

    W. Wu, H. Luo, B. Fang, J. Wang, and W. Ouyang, “Cap4video: What can auxiliary captions do for text-video retrieval?” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2023, pp. 10 704–10 713

    2023

  5. [13]

    Multi-event video-text retrieval,

    G. Zhang, J. Ren, J. Gu, and V . Tresp, “Multi-event video-text retrieval,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 22 113–22 123

    2023

  6. [14]

    Exploiting unlabeled videos for video-text retrieval via pseudo-supervised learning,

    Y . Lu, R. Quan, L. Zhu, and Y . Yang, “Exploiting unlabeled videos for video-text retrieval via pseudo-supervised learning,” IEEE Transactions on Image Processing , 2024

    2024

  7. [15]

    Video recap: Recursive captioning of hour-long videos,

    M. M. Islam, N. Ho, X. Yang, T. Nagarajan, L. Torresani, and G. Bertasius, “Video recap: Recursive captioning of hour-long videos,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 18 198–18 208

    2024

  8. [16]

    Vidchapters- 7m: Video chapters at scale,

    A. Yang, A. Nagrani, I. Laptev, J. Sivic, and C. Schmid, “Vidchapters- 7m: Video chapters at scale,” Advances in Neural Information Process- ing Systems, vol. 36, pp. 49 428–49 444, 2023

    2023

  9. [17]

    Hierarchical representation network with auxiliary tasks for video captioning and video question answering,

    L. Gao, Y . Lei, P. Zeng, J. Song, M. Wang, and H. T. Shen, “Hierarchical representation network with auxiliary tasks for video captioning and video question answering,” IEEE Transactions on Image Processing , vol. 31, pp. 202–215, 2021

    2021

  10. [18]

    Multi-modal dense video captioning,

    V . Iashin and E. Rahtu, “Multi-modal dense video captioning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops , 2020, pp. 958–959

    2020

  11. [19]

    A better use of audio-visual cues: Dense video captioning with bi-modal transformer,

    ——, “A better use of audio-visual cues: Dense video captioning with bi-modal transformer,” arXiv preprint arXiv:2005.08271 , 2020

    2005 arXiv

  12. [20]

    Hierarchical context encoding for events caption- ing in videos,

    D. Yang and C. Yuan, “Hierarchical context encoding for events caption- ing in videos,” in 2018 25th IEEE International Conference on Image Processing (ICIP). IEEE, 2018, pp. 1288–1292

    2018

  13. [21]

    End-to-end object detection with transformers,

    N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in European conference on computer vision . Springer, 2020, pp. 213– 229

    2020

  14. [22]

    Do you remember? dense video captioning with cross-modal memory retrieval,

    M. Kim, H. B. Kim, J. Moon, J. Choi, and S. T. Kim, “Do you remember? dense video captioning with cross-modal memory retrieval,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 13 894–13 904

    2024

  15. [23]

    Parallel pathway dense video captioning with deformable transformer,

    W. Choi, J. Chen, and J. Yoon, “Parallel pathway dense video captioning with deformable transformer,” IEEE Access , vol. 10, pp. 129 899– 129 910, 2022

    2022

  16. [24]

    Dibs: Enhancing dense video captioning with unlabeled videos via pseudo boundary enrichment and online refinement,

    H. Wu, H. Liu, Y . Qiao, and X. Sun, “Dibs: Enhancing dense video captioning with unlabeled videos via pseudo boundary enrichment and online refinement,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 18 699–18 708

    2024

  17. [25]

    Towards automatic learning of procedures from web instructional videos,

    L. Zhou, C. Xu, and J. Corso, “Towards automatic learning of procedures from web instructional videos,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 32, no. 1, 2018

    2018

  18. [26]

    Bidirectional attentive fusion with context gating for dense video captioning,

    J. Wang, W. Jiang, L. Ma, W. Liu, and Y . Xu, “Bidirectional attentive fusion with context gating for dense video captioning,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 7190–7198

    2018

  19. [28]

    Sketch, ground, and refine: Top-down dense video captioning,

    C. Deng, S. Chen, D. Chen, Y . He, and Q. Wu, “Sketch, ground, and refine: Top-down dense video captioning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021, pp. 234–243

    2021

  20. [29]

    iperceive: Applying common- sense reasoning to multi-modal dense video captioning and video question answering,

    A. Chadha, G. Arora, and N. Kaloty, “iperceive: Applying common- sense reasoning to multi-modal dense video captioning and video question answering,” arXiv preprint arXiv:2011.07735 , 2020

    2011 arXiv

  21. [30]

    Towards bridging event captioner and sentence localizer for weakly supervised dense event captioning,

    S. Chen and Y .-G. Jiang, “Towards bridging event captioner and sentence localizer for weakly supervised dense event captioning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, 2021, pp. 8425–8435

    2021

  22. [31]

    Streamlined dense video captioning,

    J. Mun, L. Yang, Z. Ren, N. Xu, and B. Han, “Streamlined dense video captioning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2019, pp. 6588–6597

    2019

  23. [32]

    Watch, listen and tell: Multi- modal weakly supervised dense event captioning,

    T. Rahman, B. Xu, and L. Sigal, “Watch, listen and tell: Multi- modal weakly supervised dense event captioning,” in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 8908–8917

    2019

  24. [33]

    Dense procedure captioning in narrated instructional videos,

    B. Shi, L. Ji, Y . Liang, N. Duan, P. Chen, Z. Niu, and M. Zhou, “Dense procedure captioning in narrated instructional videos,” in Proceedings of the 57th annual meeting of the association for computational linguistics , 2019, pp. 6382–6391

    2019

  25. [34]

    Retrieval- augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschel et al. , “Retrieval- augmented generation for knowledge-intensive nlp tasks,” Advances in Neural Information Processing Systems , vol. 33, pp. 9459–9474, 2020

    2020

  26. [35]

    Streaming dense video captioning,

    X. Zhou, A. Arnab, S. Buch, S. Yan, A. Myers, X. Xiong, A. Nagrani, and C. Schmid, “Streaming dense video captioning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 18 243–18 252

    2024

  27. [36]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929 , 2020

    2010 arXiv

  28. [37]

    Learning texture transformer network for image super-resolution,

    F. Yang, H. Yang, J. Fu, H. Lu, and B. Guo, “Learning texture transformer network for image super-resolution,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 5791–5800

    2020

  29. [38]

    Siamese-detr for generic multi-object tracking,

    Q. Liu, Y . Li, Y . Jiang, and Y . Fu, “Siamese-detr for generic multi-object tracking,” IEEE Transactions on Image Processing , 2024

    2024

  30. [39]

    Depth anything: Unleashing the power of large-scale unlabeled data,

    L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao, “Depth anything: Unleashing the power of large-scale unlabeled data,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 10 371–10 381

    2024

  31. [40]

    Spectralgpt: Spectral remote sensing foun- dation model,

    D. Hong, B. Zhang, X. Li, Y . Li, C. Li, J. Yao, N. Yokoya, H. Li, P. Ghamisi, X. Jia et al. , “Spectralgpt: Spectral remote sensing foun- dation model,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  32. [41]

    Segment anything in medical images,

    J. Ma, Y . He, F. Li, L. Han, C. You, and B. Wang, “Segment anything in medical images,” Nature Communications, vol. 15, no. 1, p. 654, 2024

    2024

  33. [42]

    Point to set similarity based deep feature learning for person re-identification,

    S. Zhou, J. Wang, J. Wang, Y . Gong, and N. Zheng, “Point to set similarity based deep feature learning for person re-identification,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 3741–3750

    2017

  34. [43]

    Visual-linguistic feature align- ment with semantic and kinematic guidance for referring multi-object tracking,

    Y . Li, S. Zhou, Z. Qin, and L. Wang, “Visual-linguistic feature align- ment with semantic and kinematic guidance for referring multi-object tracking,” IEEE Transactions on Multimedia , 2025

    2025

  35. [44]

    Explainability enhanced object detection transformer with feature disentanglement,

    W. Yu, R. Liu, D. Chen, and Q. Hu, “Explainability enhanced object detection transformer with feature disentanglement,” IEEE Transactions on Image Processing , 2024

    2024

  36. [45]

    Deformable detr: Deformable transformers for end-to-end object detection,

    X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, “Deformable detr: Deformable transformers for end-to-end object detection,”arXiv preprint arXiv:2010.04159, 2020

    2010 arXiv

  37. [46]

    Fast temporal activity proposals for efficient detection of human actions in untrimmed videos,

    F. C. Heilbron, J. C. Niebles, and B. Ghanem, “Fast temporal activity proposals for efficient detection of human actions in untrimmed videos,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1914–1923

    2016

  38. [47]

    Turn tap: Temporal unit regression network for temporal action proposals,

    J. Gao, Z. Yang, K. Chen, C. Sun, and R. Nevatia, “Turn tap: Temporal unit regression network for temporal action proposals,” inProceedings of the IEEE international conference on computer vision , 2017, pp. 3628– 3636. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 11

    2017

  39. [48]

    Daps: Deep action proposals for action understanding,

    V . Escorcia, F. Caba Heilbron, J. C. Niebles, and B. Ghanem, “Daps: Deep action proposals for action understanding,” in Computer Vision– ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14 . Springer, 2016, pp. 768–784

    2016

  40. [49]

    Bmn: Boundary-matching network for temporal action proposal generation,

    T. Lin, X. Liu, X. Li, E. Ding, and S. Wen, “Bmn: Boundary-matching network for temporal action proposal generation,” in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 3889–3898

    2019

  41. [50]

    Bsn: Boundary sensitive network for temporal action proposal generation,

    T. Lin, X. Zhao, H. Su, C. Wang, and M. Yang, “Bsn: Boundary sensitive network for temporal action proposal generation,” in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 3–19

    2018

  42. [51]

    Temporal action detection with structured segment networks,

    Y . Zhao, Y . Xiong, L. Wang, Z. Wu, X. Tang, and D. Lin, “Temporal action detection with structured segment networks,” in Proceedings of the IEEE international conference on computer vision , 2017, pp. 2914– 2923

    2017

  43. [52]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Pro- ceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technolo...

    2019

  44. [53]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763

    2021

  45. [54]

    Attention is all you need,

    A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems, 2017

    2017

  46. [55]

    The hungarian method for the assignment problem,

    H. W. Kuhn, “The hungarian method for the assignment problem,” Naval research logistics quarterly, vol. 2, no. 1-2, pp. 83–97, 1955

    1955

  47. [56]

    Generalized intersection over union: A metric and a loss for bounding box regression,

    H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese, “Generalized intersection over union: A metric and a loss for bounding box regression,” in Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , 2019, pp. 658–666

    2019

  48. [57]

    Focal loss for dense object detection,

    T.-Y . Ross and G. Doll ´ar, “Focal loss for dense object detection,” in proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2980–2988

    2017

  49. [58]

    Video-llama: An instruction-tuned audio-visual language model for video understanding,

    H. Zhang, X. Li, and L. Bing, “Video-llama: An instruction-tuned audio-visual language model for video understanding,” arXiv preprint arXiv:2306.02858, 2023

    2023 arXiv

  50. [59]

    Videochat: Chat-centric video understanding,

    K. Li, Y . He, Y . Wang, Y . Li, W. Wang, P. Luo, Y . Wang, L. Wang, and Y . Qiao, “Videochat: Chat-centric video understanding,” arXiv preprint arXiv:2305.06355, 2023

    2023 arXiv

  51. [60]

    Timechat: A time-sensitive multimodal large language model for long video understanding,

    S. Ren, L. Yao, S. Li, X. Sun, and L. Hou, “Timechat: A time-sensitive multimodal large language model for long video understanding,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 14 313–14 323

    2024

  52. [61]

    End- to-end dense video captioning as sequence generation,

    W. Zhu, B. Pang, A. V . Thapliyal, W. Y . Wang, and R. Soricut, “End- to-end dense video captioning as sequence generation,” arXiv preprint arXiv:2204.08121, 2022

    2022 arXiv

  53. [62]

    Event-centric hier- archical representation for dense video captioning,

    T. Wang, H. Zheng, M. Yu, Q. Tian, and H. Hu, “Event-centric hier- archical representation for dense video captioning,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 5, pp. 1890– 1900, 2020

    1900

  54. [63]

    End-to-end dense video captioning with masked transformer,

    L. Zhou, Y . Zhou, J. J. Corso, R. Socher, and C. Xiong, “End-to-end dense video captioning with masked transformer,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 8739–8748

    2018

  55. [64]

    Dense-captioning events in videos: Sysu submission to activitynet challenge 2020,

    T. Wang, H. Zheng, and M. Yu, “Dense-captioning events in videos: Sysu submission to activitynet challenge 2020,” arXiv preprint arXiv:2006.11693, 2020

    2020 arXiv

  56. [65]

    Cider: Consensus- based image description evaluation,

    R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus- based image description evaluation,” in Proceedings of the IEEE confer- ence on computer vision and pattern recognition , 2015, pp. 4566–4575

    2015

  57. [66]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 2002, pp. 311–318

    2002

  58. [67]

    Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,” in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , 2005, pp. 65–72

    2005

  59. [68]

    Soda: Story oriented dense video captioning evaluation framework,

    S. Fujita, T. Hirao, H. Kamigaito, M. Okumura, and M. Nagata, “Soda: Story oriented dense video captioning evaluation framework,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16 . Springer, 2020, pp. 517–531

    2020

  60. [69]

    Move forward and tell: A progressive generator of video descriptions,

    Y . Xiong, B. Dai, and D. Lin, “Move forward and tell: A progressive generator of video descriptions,” in Proceedings of the European Con- ference on Computer Vision (ECCV) , 2018, pp. 468–483

    2018

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.