{"id":"49c7b5be-3ae3-4839-8e9a-dd1c374cdc45","arxiv_id":"1908.03846","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A temporal compositional modular network that decomposes queries into main event, context event, and temporal signal improves video moment localization on TEMPO.","lead":"This paper introduces a neural network that splits a natural-language video query into a main event, a context event, and a temporal word, then matches each piece against video segments. It reports state-of-the-art results on the TEMPO benchmark, which tests temporal language such as 'before' and 'after'.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pairwise scores in Eqs. 3-6 are never reduced to a single-moment prediction, so the reported R@1/mIoU gains are not reproducible and the SOTA claim is unverifiable.","rationale":"The reader identified the parser-based decomposition as the weakest assumption. That is a genuine risk: TEMPO-HL contains anaphora and omission, and Berkeley Neural Parser output on such sentences can be noisy; a bad tree could misalign d_m, d_c, and d_s. However, the soft tree attention is trained end-to-end and may compensate for parser noise, and this risk affects robustness/generalization rather than the internal validity of the reported tables. The missing inference aggregation affects the headline result directly: Eqs. 3-6 produce a pairwise score, but the evaluation is single-moment, and the paper never connects them. This is not a matter of style; without the mapping, the reported R@1/mIoU cannot be reproduced, and the comparison to MLLC may be comparing different prediction objects. The ablation study, parameter-count check, and attention visualizations are useful supporting evidence, but they do not supply the missing protocol. Since this is a fixable omission rather than a demonstrated error, I retain the reader's CONDITIONAL verdict and would make the inference-protocol specification and re-run an explicit acceptance condition.","tokens_in":12458,"tokens_out":8198,"duration_ms":90758,"concrete_test":"The single check is to require a precise, implementable statement of the inference protocol and re-run the TEMPO-HL evaluation with it. Concretely: (1) state how s_ij is converted to a per-query score (e.g., score_i = max_j s_ij, sum_j, or pair-level max); (2) state how the selected pair (i,j) becomes a predicted moment for before/after/while/then (e.g., intersection, union, or main-event span); (3) state which ground-truth span is used for each category. Then reproduce Table 2's Ensemble TCMN row under this documented protocol and re-run MLLC+conTEF under the same protocol. If the reported numbers change by more than a small margin, or if no protocol can be supplied, the state-of-the-art claim should be withdrawn pending correction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 defines a pairwise matching score s_ij (Eqs. 3-6) for candidate segments i and j, with losses supervising the main and context event segments. Yet Tables 1 and 2 report R@1, R@5, and mIoU for a single localized moment per query. Nowhere does the paper specify how s_ij is aggregated into a single segment ranking or how the predicted span is formed from the selected pair. The Section 4.2.3 discussion makes the gap concrete: it says 'the prediction of A then B is the union of A and B' and 'A while B is the intersection', implying a pair-to-span construction that is never defined in the model. If, for example, inference scores segment i by max_j s_ij, the method ranks main-event candidates only, whereas MLLC's published protocol may rank the composite moment; the comparison in Table 2 would then not measure the same object. This is more load-bearing than the parser concern: even a perfect tree attention network would not make the central empirical claim checkable without this inference mapping.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses video moment localization with natural-language queries that express temporal relationships between events. It proposes a Temporal Compositional Modular Network (TCMN) that uses a Tree-LSTM over a constituent parse of the query, with a tree attention network to softly decompose the sentence into embeddings for the main event, the context event, and the temporal signal. Two scoring modules are introduced: a temporal localization module matching the event embeddings to visual segment features, and a temporal relationship module matching the temporal-signal embedding to location features. An ensemble of four models, corresponding to the four combinations of RGB/flow features for main and context events, is combined by late fusion. Experiments on the TEMPO dataset report state-of-the-art average R@1, R@5, and mIoU on TEMPO-TL and TEMPO-HL, together with ablations on the proposed components.","tokens_in":12703,"tokens_out":4409,"duration_ms":46347,"significance":"If the empirical claims are reproducible, the paper makes a useful contribution to moment localization with temporal language: it is one of the first attempts to explicitly decompose a temporal query into main event, context event, and temporal signal, and it introduces a modality-combination ensemble that is well motivated by the observation that events in a query may rely on different visual channels. The paper also includes a parameter-matched ablation, attention visualizations, and an honest discussion of cases where the model underperforms on simple sentences. However, the central empirical claim is currently difficult to verify because the paper does not specify how the pairwise matching scores are converted into a single moment prediction, and the ensemble weights are left unspecified. These omissions undermine reproducibility and make the reported state-of-the-art claim uncheckable as written.","major_comments":[{"comment":"The model defines a matching score s_ij for every pair of candidate segments (i,j), and the losses in Eq. (5) supervise the main-event and context-event segments. However, the paper never specifies how these pairwise scores are reduced to a single moment prediction for evaluation. The reported R@1, R@5, and mIoU in Tables 1 and 2 assume a ranking of moments, but no inference procedure is given: the paper does not state which pair is selected, how a composite span is formed, or how a single segment score is derived from s_ij. Section 4.2.3 discusses 'the prediction of A then B is the union of A and B' and 'A while B is the intersection', which implies a pair-to-span construction, yet that construction is never defined. Without this inference mapping, the quantitative results cannot be reproduced, and the comparison with prior methods may not measure the same ranking object (e.g., main-event segments versus composite moments). This is the load-bearing gap for the paper's central claim.","section":"Sections 3.2-3.4, Eqs. (3)-(5)"},{"comment":"The ensemble weights gamma_n in Eq. (6) are never specified. The final scores used for all reported results are the late-fusion combination of four independently trained models, and the values of gamma_n directly determine the reported metrics. The paper should state whether the weights are uniform, chosen on a validation split, or learned, and should provide the actual values used. Without this, the ensemble contribution and the state-of-the-art claim are incompletely specified.","section":"Section 3.5, Eq. (6)"},{"comment":"All reported results appear to come from single runs, with no standard deviations, confidence intervals, or significance tests. This is a concern for the central claim because several individual comparisons are close or go against the proposed method: on the DiDeMo category, Ensemble TCMN has lower mIoU than MLLC+conTEF in both Table 1 (41.03 vs. 41.20) and Table 3 (41.03 vs. 42.45), and in Table 3 it is also below TGN and MAN. The overall state-of-the-art claim rests on average improvements over complex-sentence categories; without variance estimates, the reader cannot judge whether these differences are meaningful.","section":"Tables 1-3 and Section 4.2"},{"comment":"TMN [17] is listed as 'based on our implementation', but no code or detailed reimplementation settings are provided. Since TMN is a close comparison point (a temporal modular network with compositional structure), an unverified reimplementation raises the risk that the comparison is not faithful. The authors should either release the reimplementation, provide full hyperparameter and training details, or use the original authors' reported numbers where available.","section":"Tables 1-2 (captions) and Section 4.2"}],"minor_comments":[{"comment":"The attention weight in the summation is written as alpha_n^j, but the superscript j is not defined in the preceding softmax expression; the notation should be made consistent (e.g., alpha_n^j = softmax_j(...)).","section":"Eq. (2)"},{"comment":"There are typographical errors in Eq. (2): 'so f tmax' and 'embeddinд' should be 'softmax' and 'embedding'.","section":"Eq. (2)"},{"comment":"In Eq. (5), N is introduced as the set of all possible segments, but is then used as the normalization constant in 1/N; using |N| for the cardinality would be clearer.","section":"Section 3.4"},{"comment":"The label 'Frequeny Prior' should be 'Frequency Prior'.","section":"Tables 1 and 2"},{"comment":"The phrase 'the state-of-art methods' appears in the text and should be 'state-of-the-art'.","section":"Section 4.2.2"},{"comment":"Reference [23] contains an encoding artifact ('SchlÃijter') and should be typeset correctly as 'Schlüter'.","section":"Reference [23]"},{"comment":"The paper does not state whether the ensemble weights gamma_n are shared across all queries or are query-dependent; this should be clarified.","section":"Section 3.5"},{"comment":"In the main text the attention colors are described as 'red, yellow and orange', while the caption orders them as 'red, orange and yellow'; the description should be consistent for all three attention types.","section":"Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The missing inference procedure from pairwise scores to a single moment is a serious but fixable gap: the authors can add a precise description of how s_ij is converted into a ranked list of moments and how the composite span is constructed. I would also urge the editor to require the authors to specify the ensemble weights and to provide variance estimates or significance tests for the main comparisons, given that some DiDeMo numbers are below prior methods. The reimplementation of TMN without code is a secondary fairness concern. No issues of scientific integrity or citation behavior are apparent."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a credible systems attempt at a real subproblem: localizing video moments from sentences with temporal relations. The new bit is the tree attention network that softly decomposes a query into main event, context event, and temporal signal embeddings, plus a late fusion over four modality combinations. The ablation in Table 4 suggests the relationship module contributes, and the attention visualizations are a nice touch. So there is a genuine idea here, not a repackaging.\n\nBut the stress-test note is correct, and it's more serious than the reader's parser worry. Section 3 defines a pairwise score s_ij for segment pair (i,j), and the losses supervise main and context event segments. Yet Tables 1 and 2 report R@1, R@5, and mIoU for one localized moment per query. Nowhere does the paper say how s_ij is reduced to a single segment ranking or a predicted span. Section 4.2.3 says \"the prediction of A then B is the union of A and B\" and \"A while B is the intersection,\" which implies a pair-to-span construction that is never defined. If inference scores segment i by max_j s_ij, you are ranking main-event candidates; if you rank composite spans by some other aggregate, you are measuring something different from MLLC. Either way, the reported gains are not reproducible, and the SOTA claim is unverifiable as written.\n\nOther soft spots, in order: no error bars or significance tests; TMN is reimplemented without code; the ensemble weights gamma_n in Eq. 6 are not specified (are they learned? grid-searched? equal?). Those are fixable. The parser fragility is real but secondary: the tree attention is soft, so it can partially recover from bad parses, and the ablation shows the tree attention helps over sequential attention on complex sentences.\n\nOverall: the architecture is coherent and the components are well-motivated, but the empirical core has a hole. I would not cite the numbers until the inference is specified and code or detailed hyperparameters are released. That said, this deserves a serious referee: with the inference mapping clarified and error bars added, it would be a solid systems contribution to a specialized venue.\n\nFor peer review: send it, but expect heavy revision.\n\nBest.","headline":"Missing inference mapping makes the SOTA claim unverifiable; the decomposition idea is real but the paper as submitted is not reproducible.","tokens_in":13177,"tokens_out":3752,"would_cite":false,"duration_ms":35241,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Splitting temporal sentences into main event, context, and signal improves video moment localization.","keywords":["video moment localization","temporal language","tree attention network","cross-modal retrieval","temporal reasoning","ensemble learning","natural language grounding"],"falsifier":"Retrain TCMN on TEMPO-HL using manually annotated gold parse trees instead of the automatic parser; if performance does not improve (or if automatically parsed trees already match gold), the parser is not the bottleneck. Conversely, deliberately corrupting parse trees should cause a measurable drop if the decomposition is truly load-bearing.","tokens_in":12248,"feed_emoji":"🎬","tokens_out":2639,"duration_ms":28233,"temperature":0.7,"pith_summary":"The paper argues that existing video moment localization models underperform when the query describes multiple events linked by temporal words such as before, after, while, and then. It proposes TCMN, a Temporal Compositional Modular Network that first decomposes the sentence into three phrase-level descriptions and then matches each to video segments through separate modules. On the TEMPO benchmark, TCMN reports the best average R@1, R@5, and mIoU, beating strong baselines including MLLC. The core message is that explicitly modeling temporal relationships inside the query, rather than encoding the whole sentence at once, helps localize the correct moment.","feed_headline":"Tree attention splits temporal queries to find the right video moment","feed_subtitle":"Decomposing sentences into main event, context, and signal beats prior models on the TEMPO benchmark.","key_machinery":"The tree attention network is the central mechanism: a Tree-LSTM encodes each node of the parse tree, and an attention layer conditioned on node labels and word embeddings produces three phrase embeddings d_m, d_c, and d_s for the main event, context event, and temporal signal. This soft parsing feeds two complementary modules, one for visual similarity and one for location similarity, and the four-stream late fusion handles cases where the main and context events rely on different visual modalities.","core_discovery":"TCMN learns to softly decompose a temporal query into a main event, a context event, and a temporal signal by attending over the nodes of a constituency parse tree, then computes matching scores with two modules: a temporal localization module that aligns the event phrase embeddings with video segments and location features, and a temporal relationship module that aligns the signal phrase embedding with location pairs. An ensemble of four models, each trained on a different RGB/optical-flow combination for the two events, is late-fused into the final score. The paper claims this design achieves state-of-the-art performance on both template-based (TEMPO-TL) and human-annotated (TEMPO-HL) temporal language benchmarks, with the relationship module and the tree-structured attention providing clear gains in ablations.","pith_inferences":["One could test whether the same tree-attention decomposition transfers to other video-and-language benchmarks beyond TEMPO, such as tasks where queries contain causal or conditional conjunctions rather than purely temporal ones.","The reliance on an off-the-shelf parser suggests a natural ablation: replacing automatic parse trees with gold or human-annotated trees should quantify how much of the gain depends on parser quality.","The temporal signal attention appears to focus strongly on single words (before, after, while, then); a simpler word-level classifier might achieve similar gains, so the marginal value of the full tree structure is a testable question."],"forward_implications":["If the claim holds, moment localization systems should stop encoding the query as a single vector and instead explicitly separate event descriptions from ordering signals.","The success of the relationship module suggests that purely linguistic cues like before and after are better matched against location features than against visual content, which may generalize to other spatio-temporal reasoning tasks.","The ensemble over RGB and optical-flow pairs indicates that a single visual backbone may be insufficient when a sentence mentions events with different motion or appearance characteristics.","The method's compatibility with simple sentences, despite being designed for complex temporal language, means a single model can cover both query types without data augmentation tricks."],"supporting_citations":[{"why":"Introduces the TEMPO dataset and the MLLC baseline, which is the primary comparison point and the source of evaluation metrics.","marker":"[11]"},{"why":"Provides the TALL baseline with temporal end features, used as a comparison in the tables.","marker":"[7]"},{"why":"Defines the DiDeMo dataset and the MCN model, the source of the underlying videos and a baseline for simple sentences.","marker":"[10]"},{"why":"Presents TMN, a modular network that also uses parsing information, and serves as a key baseline for complex sentences.","marker":"[17]"},{"why":"Establishes that temporal signals often appear in SBAR-TMP or PP-TMP subtrees, the structural observation motivating the tree attention network.","marker":"[5]"},{"why":"Supplies the constituency parser used to generate the parse trees that the tree attention network operates on.","marker":"[15]"},{"why":"Provides the Tree-LSTM architecture that encodes the parse tree nodes into phrase representations.","marker":"[25]"},{"why":"Introduces the compositional modular network paradigm that TCMN adapts from image referring-expression comprehension to video.","marker":"[13]"}],"fun_headline_variants":["Tree attention decomposes temporal queries to find video moments","Main, context, signal: tree attention improves moment grounding","Decomposing sentences into events lifts TEMPO localization","Tree-query split plus late fusion beats state of the art on TEMPO"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gains depend on the off-the-shelf parser producing parse trees whose node labels and structure cleanly separate the main event, context event, and temporal signal, and on the attention learning to pick the right nodes from the final localization losses alone.","fun_headline_variants_meta":{"raw":{"variants":["Tree attention decomposes temporal queries to find video moments","Main, context, signal: tree attention improves moment grounding","Decomposing sentences into events lifts TEMPO localization","Tree-query split plus late fusion beats state of the art on TEMPO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000175,"raw_usage":{"total_tokens":1238,"prompt_tokens":850,"completion_tokens":388,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":319}},"tokens_in":466,"tokens_out":388,"duration_ms":4510,"temperature":1.0,"reasoning_tokens":319,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:00:18.364884+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain TCMN on TEMPO-HL using manually annotated gold parse trees instead of the automatic parser; if performance does not improve (or if automatically parsed trees already match gold), the parser is not the bottleneck. Conversely, deliberately corrupting parse trees should cause a measurable drop if the decomposition is truly load-bearing.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the TEMPO dataset and the MLLC baseline, which is the primary comparison point and the source of evaluation metrics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the TALL baseline with temporal end features, used as a comparison in the tables."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the DiDeMo dataset and the MCN model, the source of the underlying videos and a baseline for simple sentences."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Presents TMN, a modular network that also uses parsing information, and serves as a key baseline for complex sentences."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that temporal signals often appear in SBAR-TMP or PP-TMP subtrees, the structural observation motivating the tree attention network."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the constituency parser used to generate the parse trees that the tree attention network operates on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Tree-LSTM architecture that encodes the parse tree nodes into phrase representations."}],"review_version":1}