Pith. sign in

REVIEW 4 major objections 8 minor 35 references

Exploiting Temporal Relationships in Video Moment Localization with Natural Language

T0 review · 4 major / 8 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Splitting temporal sentences into main event, context, and signal improves video moment localization.

desk verdict Missing inference mapping makes the SOTA claim unverifiable; the decomposition idea is real but the paper as submitted is not reproducible. read the letter →

arxiv 1908.03846 v1 pith:3SFH55AP submitted 2019-08-11 cs.MM cs.CVcs.IR

Songyang Zhang , Jinsong Su , Jiebo Luo This is my paper · ORCID
classification cs.MMcs.CVcs.IR
keywords videomomentlocalizationtemporallanguagetreeattentionnetworkcross-modalretrievalreasoningensemblelearningnaturalgrounding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that existing video moment localization models underperform when the query describes multiple events linked by temporal words such as before, after, while, and then. It proposes TCMN, a Temporal Compositional Modular Network that first decomposes the sentence into three phrase-level descriptions and then matches each to video segments through separate modules. On the TEMPO benchmark, TCMN reports the best average R@1, R@5, and mIoU, beating strong baselines including MLLC. The core message is that explicitly modeling temporal relationships inside the query, rather than encoding the whole sentence at once, helps localize the correct moment.

What carries the argument

The tree attention network is the central mechanism: a Tree-LSTM encodes each node of the parse tree, and an attention layer conditioned on node labels and word embeddings produces three phrase embeddings d_m, d_c, and d_s for the main event, context event, and temporal signal. This soft parsing feeds two complementary modules, one for visual similarity and one for location similarity, and the four-stream late fusion handles cases where the main and context events rely on different visual modalities.

What would settle it

Retrain TCMN on TEMPO-HL using manually annotated gold parse trees instead of the automatic parser; if performance does not improve (or if automatically parsed trees already match gold), the parser is not the bottleneck. Conversely, deliberately corrupting parse trees should cause a measurable drop if the decomposition is truly load-bearing.

Watch

Extended reading notes

Core claim

TCMN learns to softly decompose a temporal query into a main event, a context event, and a temporal signal by attending over the nodes of a constituency parse tree, then computes matching scores with two modules: a temporal localization module that aligns the event phrase embeddings with video segments and location features, and a temporal relationship module that aligns the signal phrase embedding with location pairs. An ensemble of four models, each trained on a different RGB/optical-flow combination for the two events, is late-fused into the final score. The paper claims this design achieves state-of-the-art performance on both template-based (TEMPO-TL) and human-annotated (TEMPO-HL) temporal language benchmarks, with the relationship module and the tree-structured attention providing clear gains in ablations.

Load-bearing premise

The gains depend on the off-the-shelf parser producing parse trees whose node labels and structure cleanly separate the main event, context event, and temporal signal, and on the attention learning to pick the right nodes from the final localization losses alone.

Editorial extensions

If this is right

  • If the claim holds, moment localization systems should stop encoding the query as a single vector and instead explicitly separate event descriptions from ordering signals.
  • The success of the relationship module suggests that purely linguistic cues like before and after are better matched against location features than against visual content, which may generalize to other spatio-temporal reasoning tasks.
  • The ensemble over RGB and optical-flow pairs indicates that a single visual backbone may be insufficient when a sentence mentions events with different motion or appearance characteristics.
  • The method's compatibility with simple sentences, despite being designed for complex temporal language, means a single model can cover both query types without data augmentation tricks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One could test whether the same tree-attention decomposition transfers to other video-and-language benchmarks beyond TEMPO, such as tasks where queries contain causal or conditional conjunctions rather than purely temporal ones.
  • The reliance on an off-the-shelf parser suggests a natural ablation: replacing automatic parse trees with gold or human-annotated trees should quantify how much of the gain depends on parser quality.
  • The temporal signal attention appears to focus strongly on single words (before, after, while, then); a simpler word-level classifier might achieve similar gains, so the marginal value of the full tree structure is a testable question.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper addresses video moment localization with natural-language queries that express temporal relationships between events. It proposes a Temporal Compositional Modular Network (TCMN) that uses a Tree-LSTM over a constituent parse of the query, with a tree attention network to softly decompose the sentence into embeddings for the main event, the context event, and the temporal signal. Two scoring modules are introduced: a temporal localization module matching the event embeddings to visual segment features, and a temporal relationship module matching the temporal-signal embedding to location features. An ensemble of four models, corresponding to the four combinations of RGB/flow features for main and context events, is combined by late fusion. Experiments on the TEMPO dataset report state-of-the-art average R@1, R@5, and mIoU on TEMPO-TL and TEMPO-HL, together with ablations on the proposed components.

Significance. If the empirical claims are reproducible, the paper makes a useful contribution to moment localization with temporal language: it is one of the first attempts to explicitly decompose a temporal query into main event, context event, and temporal signal, and it introduces a modality-combination ensemble that is well motivated by the observation that events in a query may rely on different visual channels. The paper also includes a parameter-matched ablation, attention visualizations, and an honest discussion of cases where the model underperforms on simple sentences. However, the central empirical claim is currently difficult to verify because the paper does not specify how the pairwise matching scores are converted into a single moment prediction, and the ensemble weights are left unspecified. These omissions undermine reproducibility and make the reported state-of-the-art claim uncheckable as written.

major comments (4)
  1. [Sections 3.2-3.4, Eqs. (3)-(5)] The model defines a matching score s_ij for every pair of candidate segments (i,j), and the losses in Eq. (5) supervise the main-event and context-event segments. However, the paper never specifies how these pairwise scores are reduced to a single moment prediction for evaluation. The reported R@1, R@5, and mIoU in Tables 1 and 2 assume a ranking of moments, but no inference procedure is given: the paper does not state which pair is selected, how a composite span is formed, or how a single segment score is derived from s_ij. Section 4.2.3 discusses 'the prediction of A then B is the union of A and B' and 'A while B is the intersection', which implies a pair-to-span construction, yet that construction is never defined. Without this inference mapping, the quantitative results cannot be reproduced, and the comparison with prior methods may not measure the same ranking object (e.g., main-event segments versus composite moments). This is the load-bearing gap for the paper's central claim.
  2. [Section 3.5, Eq. (6)] The ensemble weights gamma_n in Eq. (6) are never specified. The final scores used for all reported results are the late-fusion combination of four independently trained models, and the values of gamma_n directly determine the reported metrics. The paper should state whether the weights are uniform, chosen on a validation split, or learned, and should provide the actual values used. Without this, the ensemble contribution and the state-of-the-art claim are incompletely specified.
  3. [Tables 1-3 and Section 4.2] All reported results appear to come from single runs, with no standard deviations, confidence intervals, or significance tests. This is a concern for the central claim because several individual comparisons are close or go against the proposed method: on the DiDeMo category, Ensemble TCMN has lower mIoU than MLLC+conTEF in both Table 1 (41.03 vs. 41.20) and Table 3 (41.03 vs. 42.45), and in Table 3 it is also below TGN and MAN. The overall state-of-the-art claim rests on average improvements over complex-sentence categories; without variance estimates, the reader cannot judge whether these differences are meaningful.
  4. [Tables 1-2 (captions) and Section 4.2] TMN [17] is listed as 'based on our implementation', but no code or detailed reimplementation settings are provided. Since TMN is a close comparison point (a temporal modular network with compositional structure), an unverified reimplementation raises the risk that the comparison is not faithful. The authors should either release the reimplementation, provide full hyperparameter and training details, or use the original authors' reported numbers where available.
minor comments (8)
  1. [Eq. (2)] The attention weight in the summation is written as alpha_n^j, but the superscript j is not defined in the preceding softmax expression; the notation should be made consistent (e.g., alpha_n^j = softmax_j(...)).
  2. [Eq. (2)] There are typographical errors in Eq. (2): 'so f tmax' and 'embeddinд' should be 'softmax' and 'embedding'.
  3. [Section 3.4] In Eq. (5), N is introduced as the set of all possible segments, but is then used as the normalization constant in 1/N; using |N| for the cardinality would be clearer.
  4. [Tables 1 and 2] The label 'Frequeny Prior' should be 'Frequency Prior'.
  5. [Section 4.2.2] The phrase 'the state-of-art methods' appears in the text and should be 'state-of-the-art'.
  6. [Reference [23]] Reference [23] contains an encoding artifact ('SchlÃijter') and should be typeset correctly as 'Schlüter'.
  7. [Section 3.5] The paper does not state whether the ensemble weights gamma_n are shared across all queries or are query-dependent; this should be clarified.
  8. [Figure 6] In the main text the attention colors are described as 'red, yellow and orange', while the caption orders them as 'red, orange and yellow'; the description should be consistent for all three attention types.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TCMN is trained end-to-end on ground-truth event segments and evaluated against an external benchmark; the central SOTA claim does not reduce to its inputs.

full rationale

The paper's derivation chain is empirical rather than deductive, and no load-bearing step reduces to its own input. The tree attention network (Eq. 2) learns soft decompositions from parse-tree node embeddings and labels, and the localization/relationship modules are trained with margin ranking losses (Eq. 5) that supervise the main and context event ground-truth segments. The evaluation metric (R@1, R@5, mIoU on TEMPO) is computed against held-out annotations, not against quantities fitted in training. The late fusion of Eq. 6 combines independently trained modality streams and does not reintroduce the evaluation labels. The only overlapping-author citation is [4] (Jiebo Luo is a co-author), but it appears in the related-work survey of cross-gated attention and is not used to justify any architectural choice or result; it is not load-bearing. The parsing-structure premise rests on external work [5], and the parser is an external tool [15]. A genuine reproducibility concern exists: Section 3 defines pairwise matching scores s_ij (Eqs. 3-4) and Section 4.2.3 informally describes how temporal relations map to segment unions/intersections, but the paper never specifies the exact inference procedure that converts s_ij into the single ranked moment used in Tables 1-2. That gap affects verifiability and correctness risk, but it is not circularity: no equation or fitted parameter is defined in terms of the reported R@1/mIoU numbers. Accordingly, the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim is empirical and depends on conventional deep-learning choices rather than on a small set of fitted constants. The main axiom is the reliability of the parser's tree structure for temporal language; the main modeling assumptions are that temporal signals are location-only and that mean-pooled VGG/flow features suffice. Hyperparameters are set by hand with no sensitivity analysis, and ensemble weights are not reported.

free parameters (3)
  • Margins M_m and M_c = 0.1
    Set in Section 3.6; no sensitivity analysis reported.
  • Loss weight lambda = 1
    Set in Section 3.6; no sensitivity analysis reported.
  • Ensemble weights gamma_n = not reported
    Eq. 6 requires weights summing to one; how they are learned or chosen is not specified.
assumptions (4)
  • domain assumption Parsed constituent trees from Berkeley Neural Parser reliably identify temporal structures (SBAR-TMP, PP-TMP) in temporal language.
    The tree attention network in Section 3.1 relies on these labels and tree structure to decompose the query.
  • ad hoc to paper Temporal signals such as 'before' and 'after' are not visually grounded and should be matched only to location features.
    This motivates the separate relationship module in Section 3.3; if temporal words carry visual cues, the design would miss them.
  • domain assumption Pretrained VGG fc7 and optical-flow features, mean-pooled over segments, are sufficient to distinguish the event pairs in TEMPO.
    Section 3.2 uses these features for all segment matching.
  • ad hoc to paper For single-event sentences, using the entire video as the context event is a valid default.
    Section 3.4 states this choice when context ground truth is unavailable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploiting Temporal Relationships in Video Moment Localization with Natural Language." pith.science (2026). https://pith.science/paper/3SFH55AP

@misc{pith2026190803846,
  author       = {Pith},
  title        = {Pith review of: Exploiting Temporal Relationships in Video Moment Localization with Natural Language},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3SFH55AP}},
  note         = {Machine review of arXiv:1908.03846}
}
read the original abstract

We address the problem of video moment localization with natural language, i.e. localizing a video segment described by a natural language sentence. While most prior work focuses on grounding the query as a whole, temporal dependencies and reasoning between events within the text are not fully considered. In this paper, we propose a novel Temporal Compositional Modular Network (TCMN) where a tree attention network first automatically decomposes a sentence into three descriptions with respect to the main event, context event and temporal signal. Two modules are then utilized to measure the visual similarity and location similarity between each segment and the decomposed descriptions. Moreover, since the main event and context event may rely on different modalities (RGB or optical flow), we use late fusion to form an ensemble of four models, where each model is independently trained by one combination of the visual input. Experiments show that our model outperforms the state-of-the-art methods on the TEMPO dataset.

Figures

Figures reproduced from arXiv: 1908.03846 by the authors.

Figure 1
Figure 1. The pipeline of our proposed model kitten paws at before the bottle is dropped is composed of a main event, kitten paws at the bottle, a context event the bottle is dropped, and their temporal ordering before. Localizing a single event de￾scription has been explored by recent work [3, 4, 7–10, 17–19, 29], where most of them focus on elaborating the interaction between words and frames. However, the description that … view at source ↗
Figure 2
Figure 2. Two typical structures of temporal language. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the tree attention network. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Illustration of the temporal relationship module. [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Visualization of attention values in TCMN. Node attention values that are less than [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 28 canonical work pages

  1. [17]

    Bingbin Liu, Serena Yeung, Edward Chou, De-An Huang, Li Fei-Fei, and Juan Car- los Niebles. 2018. Temporal Modular Networks for Retrieving Complex Compo- sitional Activities in Videos. In ECCV

    work page 2018

  2. [1]

    David S Brée, Allel Feddag, and Ian Pratt. 1993. Towards a formalization of the semantics of some temporal prepositions. Time & Society 2, 2 (1993), 219–240

    work page 1993

  3. [2]

    David S Brée and Ruud A Smit. 1986. Temporal relations. Journal of Semantics 5, 4 (1986), 345–384

    work page 1986

  4. [3]

    Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jie, and Tat-Seng Chua. 2018. Temporally Grounding Natural Sentence in Video. In EMNLP

    work page 2018

  5. [4]

    Jingyuan Chen, Lin Ma, Xinpeng Chen, Zequn Jie, and Jiebo Luo. 2019. Localizing Natural Language in Videos. In AAAI

    2019

  6. [5]

    Leon RA Derczynski. 2017. Automatically ordering events and times in text . Springer

    work page 2017

  7. [6]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805 (2018)

    arXiv 2018

  8. [7]

    Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. 2017. TALL: Temporal Activity Localization via Language Query. In ICCV

    work page 2017

Show all 35 references
  1. [8]

    Runzhou Ge, Jiyang Gao, Kan Chen, and Ram Nevatia. 2019. MAC: Mining Activity Concepts for Language-based Temporal Localization. In W ACV

    2019

  2. [9]

    Dongliang He, Xiang Zhao, Jizhou Huang, Fu Li, Xiao Liu, and Shilei Wen. 2019. Read, Watch, and Move: Reinforcement Learning for Temporally Grounding Natural Language Descriptions in Videos. In AAAI

    2019

  3. [10]

    Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2017. Localizing Moments in Video With Natural Language. In ICCV

    2017

  4. [11]

    Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2018. Localizing Moments in Video with Temporal Language.. In EMNLP

    2018

  5. [12]

    Janet Hitzeman. 1995. Semantic Partition and the Ambiguity of Sentences Containing Temporal Adverbials. Natural Language Semantics 5 (09 1995). https://doi.org/10.1023/A:1008221528793

    1995 doi

  6. [13]

    Ronghang Hu, Marcus Rohrbach, Jacob Andreas, Trevor Darrell, and Kate Saenko

  7. [14]

    Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic opti- mization. In ICLR

    2015

  8. [15]

    Nikita Kitaev and Dan Klein. 2018. Constituency Parsing with a Self-Attentive Encoder. In ACL

    2018

  9. [16]

    Junwei Liang, Lu Jiang, Liangliang Cao, Li-Jia Li, and Alexander G Hauptmann

  10. [18]

    Meng Liu, Xiang Wang, Liqiang Nie, Xiangnan He, Baoquan Chen, and Tat-Seng Chua. 2018. Attentive moment retrieval in videos. In SIGIR

    2018

  11. [19]

    Meng Liu, Xiang Wang, Liqiang Nie, Qi Tian, Baoquan Chen, and Tat-Seng Chua

  12. [20]

    Xihui Liu, Zihao Wang, Jing Shao, Xiaogang Wang, and Hongsheng Li. 2019. Improving Referring Expression Grounding with Cross-modal Attention-guided Erasing. CVPR

    2019

  13. [21]

    Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In EMNLP

    2014

  14. [22]

    In ACM MM

    Cross-modal Moment Localization in Videos. In ACM MM

  15. [23]

    Norbert SchlÃijter. 2002. Temporal specification of the present perfect: a corpus- based study. Language and Computers (01 2002), 307–315

    2002

  16. [24]

    Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. ICLR

    2015

  17. [25]

    Hans Reichenbach. 2005. The tenses of verbs. The language of time: A reader (2005), 71–78

    2005

  18. [26]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems . 5998–6008

    2017

  19. [27]

    Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. 2016. Temporal segment networks: Towards good practices for deep action recognition. In ECCV

    2016

  20. [28]

    Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved Semantic Representations From Tree-Structured Long Short-Term Memory Net- works. In ACL

    2015

  21. [29]

    Plummer, Leonid Sigal, Stan Sclaroff, and Kate Saenko

    Huijuan Xu, Kun He, Bryan A. Plummer, Leonid Sigal, Stan Sclaroff, and Kate Saenko. 2019. Multilevel Language and Vision Integration for Text-to-Clip Re- trieval.. In AAAI

    2019

  22. [30]

    Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. 2018. MAttNet: Modular Attention Network for Referring Ex- pression Comprehension. In CVPR

    2018

  23. [31]

    Xiang Xiang, Ye Tian, Austin Reiter, Gregory D Hager, and Trac D Tran. 2018. S3d: Stacking segmental p3d for action quality assessment. In 2018 25th IEEE International Conference on Image Processing (ICIP) . IEEE, 928–932

    2018

  24. [32]

    Hanwang Zhang, Yulei Niu, and Shih-Fu Chang. 2018. Grounding referring expressions in images by variational context. In CVPR

    2018

  25. [34]

    Da Zhang, Xiyang Dai, Xin Wang, Yuan-Fang Wang, and Larry S. Davis. 2019. MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment. In CVPR

    2019

  26. [2017]

    Modeling relationships in referential expressions with compositional modular networks. In CVPR

  27. [2018]

    In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    Focal visual-text attention for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 6135–6143

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.