{"id":"2c6b7a37-49c7-4773-97fd-540100fa84cf","arxiv_id":"2412.09202","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A temporal action localization model that decouples classification and localization across feature pyramid layers and fuses instant, local, and global temporal features achieves state-of-the-art results on five benchmarks.","lead":"This paper presents a new model for spotting actions in videos, using different layers of the video's own feature representation for classifying the action and for locating its start and end. It reports the best published scores on five standard action-detection benchmarks and releases code and pretrained models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The MultiTHUMOS SOTA margins (6.4–7.4 mAP over TriDet) are an order of magnitude larger than the same modules deliver on THUMOS14 (0.6–1.7 mAP); if this gap is an evaluation or feature mismatch, the 'five benchmarks' claim fails.","rationale":"I chose the MultiTHUMOS evaluation concern over the reader's weakest_assumption because the universal feature-utility gradient of CLTDR is actually supported by the paper's own ablations (Table 9 and supplementary Table 13) and, even if the mechanism is not fully universal, it would not invalidate SOTA on the other four benchmarks. The MultiTHUMOS margins, in contrast, are internally inconsistent in magnitude with the same method's THUMOS14 gains and, if caused by protocol mismatch, directly falsify the five-benchmark claim. The reader did flag MultiTHUMOS as an unexplained outlier in the rationale but did not make it the weakest assumption, which is why I mark agreement as partial. The existing CONDITIONAL verdict remains appropriate; adding a specific reproducibility condition on MultiTHUMOS sharpens it without changing the overall assessment.","tokens_in":20566,"tokens_out":7024,"duration_ms":68632,"concrete_test":"Reproduce the MultiTHUMOS comparison under an identical protocol: download or re-extract the same I3D and VideoMAEv2 features used by the baselines (same stride and feature files), run the released CLTDR-GMG code and official TriDet/ActionFormer code with identical training splits, epochs, batch size, and the official MultiTHUMOS evaluation script, then compute average mAP over tIoU [0.1:0.1:0.9]. If the CLTDR-GMG vs TriDet gap shrinks to roughly 1–2 mAP, the MultiTHUMOS SOTA claim is not supported; if it remains above 5 mAP, the result is likely genuine and this concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is state-of-the-art performance on five benchmarks. The MultiTHUMOS leg of that claim is the least secure. On THUMOS14, with the same CLTDR and GMG modules and the same base Trident head, CLTDR-GMG beats TriDet by 0.6, 1.7, and 1.3 average mAP for I3D, VideoMAEv2, and InterVideo2 features, respectively. On MultiTHUMOS, the reported margins over TriDet are 6.4 (I3D RGB), 6.5 (I3D RGB+Flow), and 7.4 (VideoMAEv2) average mAP. A change in classification/regression feature fusion that moves THUMOS14 by about one point does not, by itself, explain a six-to-seven point jump on a sibling benchmark using the same backbones and loss. The likely explanation is a protocol mismatch: the paper states feature extraction for MultiTHUMOS is consistent with THUMOS14, yet Table 2's first block uses I3D RGB-only features, and no evidence is given that the quoted TriDet and ActionFormer numbers were produced with identical feature files, temporal strides, NMS, and evaluation scripts. If the MultiTHUMOS margin is an artifact of evaluation protocol, the 'five benchmarks' SOTA claim reduces to four, and the headline claim fails as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CLTDR-GMG, a one-stage temporal action localization model. GMG is an encoder module that aggregates instant, local, and global temporal information using pointwise/depthwise convolutions and an FFT-based global filter with a gating mechanism. CLTDR is a decoder that, at each pyramid layer, combines the current layer with the next higher layer's semantically coarse feature for classification and with the next lower layer's finer feature for regression, then refines both outputs through a RefineHead. The method is evaluated on THUMOS14, MultiTHUMOS, EPIC-KITCHENS-100, ActivityNet-1.3, and HACS, and the authors report state-of-the-art average mAP on all five benchmarks, with ablations on THUMOS14 using InterVideo2-6B features.","tokens_in":20830,"tokens_out":8191,"duration_ms":74389,"significance":"If substantiated, the paper makes a useful empirical contribution: it shows a convolutional/FFT encoder can replace self-attention in a TAL decoder at lower parameter cost (Table 8), and it provides a clean study of cross-layer feature selection for task decoupling. The release of code and pre-trained models, the plug-and-play experiments with ActionFormer/TriDet encoders/decoders (Supplementary E.1–E.2), and the error diagnosis are strengths. However, the headline claim rests on small margins on most benchmarks and on a single large MultiTHUMOS margin that is not yet explained; the core design assumption is only ablated on one dataset. The contribution is therefore promising but needs verification before the SOTA claim can be accepted.","major_comments":[{"comment":"Table 2 and the 'MultiTHUMOS' paragraph: the gains over TriDet are 6.4, 6.5, and 7.4 average mAP for I3D RGB, I3D RGB+Flow, and VideoMAEv2 features, respectively, whereas the same modules on THUMOS14 (Table 1) improve over TriDet by only 0.6, 1.7, and 1.3 average mAP. The manuscript does not establish that the quoted TriDet and ActionFormer rows for MultiTHUMOS were produced with identical feature files, temporal strides, NMS, and evaluation scripts; the statement in Supplementary Section D that feature extraction is 'consistent with THUMOS14' concerns the authors' own features and does not address baseline parity. Please provide a protocol-matched comparison (e.g., rerun TriDet/ActionFormer under the exact same pipeline), report per-threshold numbers, and either explain the outlier or remove MultiTHUMOS from the SOTA claim.","section":"MultiTHUMOS (Table 2)"},{"comment":"The CLTDR design in Eqs. (5)–(8) assumes a universal feature-utility gradient: higher-layer features always help classification and lower-layer features always help regression. The only ablation for this choice, Table 9 on THUMOS14 with InterVideo2-6B, does not fully support that universality: adding both higher and lower layers can slightly degrade performance (e.g., regression mAP 73.2 with all three sources versus 74.3 for the best two-source configuration), and the beneficial direction is not shown to transfer to other datasets/features. Please add ablations in at least one additional dataset/feature setting and report variance, or revise the claim to describe CLTDR as a dataset-tuned design rather than a universal principle.","section":"Ablation of CLTDR (Table 9)"},{"comment":"The margins that support the SOTA claim are small on several benchmarks: 0.6 average mAP on THUMOS14 with I3D (Table 1), 0.3 on ActivityNet-1.3 (Table 4), and 0.4 on HACS with I3D (Table 5). No standard deviations or multiple seeds are reported anywhere, and for the largest claimed gains (MultiTHUMOS) the protocol mismatch described above applies. Please report mean±std over at least three seeds for the proposed method and for the closest baselines, and state whether the comparison rows are from the authors' runs or published numbers.","section":"Tables 1, 4, 5 and Implementation Details"},{"comment":"Several comparisons are based on the authors' own re-implementations (Table 1 marks TemporalMaxer § and TriDet § with ∓; Supplementary Tables 12–13 are entirely re-implementations), but the main tables do not indicate which rows on MultiTHUMOS, EPIC-KITCHENS-100, ActivityNet, and HACS are re-implemented, nor are the re-implementation details (hyper-parameters, feature preprocessing, NMS) provided. To make the comparisons reproducible and fair, please mark all re-implemented rows and provide configuration files or a link to the code with the exact evaluation protocol.","section":"Experimental protocol"}],"minor_comments":[{"comment":"The FFT formula uses N in the exponent but the summation is over T, and the range '0 < u < T−1' should be '0 ≤ u < T'; please fix the notation.","section":"Eq. (1)"},{"comment":"The term ReLU(flocal)⊗FC(x) is labeled finstant and ReLU(fglobal)⊗Conv(x) is labeled flocal; these labels appear swapped relative to the text, please check.","section":"Eq. (4)"},{"comment":"The text says the method achieves 39.2% with SlowFast features, while Table 5 reports 39.3; please align the numbers.","section":"HACS results"},{"comment":"Table 14 header contains 'PIC-KITCHENS-100', which should be 'EPIC-KITCHENS-100', and Supplementary Section C contains the typo 'respecviely'.","section":"Supplementary tables and text"},{"comment":"The phrase 'This can be attribute to' should be 'This can be attributed to'.","section":"Supplementary Section E.1"}],"recommendation":"major_revision","confidential_remarks":"I see no citation or authorship concerns. The main risk is the MultiTHUMOS protocol: if the authors cannot substantiate the large margins, the 'five benchmarks' claim should be reduced. I would not require new datasets if the authors instead temper the claim and provide the missing protocol and variance information; major revision is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Plainly: this is a solid empirical architecture paper, not a conceptual breakthrough, and the MultiTHUMOS leg of the five-benchmark SOTA claim needs verification before you should quote it. The core novelty is the cross-layer task decoupling with attention-weighted fusion of higher/lower pyramid features, plus a refinement head, and a gated multi-granularity encoder using FFT for global context. The ablations are genuinely informative: they compare fusion strategies, granularity branches, gate vs no gate, and even show the GMG encoder as a drop-in replacement for ActionFormer and TriDet encoders with modest gains. That kind of evidence is worth crediting.\n\nThe soft spots are mostly about benchmarking hygiene. There are no error bars or multiple seeds anywhere. The THUMOS14 win over TriDet with I3D features is 0.6 mAP, which is easily within run-to-run noise. More importantly, the MultiTHUMOS margins are 6.4 to 7.4 mAP over TriDet while the same modules only move THUMOS14 by about a point. That is a red flag. The supplementary states feature extraction for MultiTHUMOS is consistent with THUMOS14, but the baselines quoted in Table 2 may not have been run under identical settings. There is no evidence that the TriDet and ActionFormer numbers come from the same feature files, temporal strides, NMS, or evaluation script. If the MultiTHUMOS margin is an artifact of protocol mismatch, the 'five benchmarks' claim collapses to four. The authors need to release the exact evaluation code and baseline configs, and ideally rerun the baselines themselves.\n\nThe ablation in Table 9 also shows that adding both higher and lower pyramid features can slightly hurt, so the universal feature-utility gradient assumed by CLTDR is not fully supported. That is a design weakness, but the overall module still works on the datasets tested. The CLTDR premise may not transfer to other action distributions without tuning.\n\nWho should read this: anyone working on one-stage TAL architectures, especially those interested in efficient alternatives to self-attention for temporal encoding. The GMG module is lightweight and the plug-and-play results make it a useful building block. The paper deserves a serious referee: the claims are empirically checkable, the ablations are well designed, and the potential protocol issue on MultiTHUMOS is exactly what reviewers should push on. I would send it to peer review with a request for clarification and reproducibility. I would not cite the five-benchmark SOTA claim without first checking the MultiTHUMOS setup.","headline":"Solid TAL architecture paper with a suspicious MultiTHUMOS SOTA margin that needs verification before the five-benchmark claim is trusted.","tokens_in":21399,"tokens_out":3841,"would_cite":true,"duration_ms":34363,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Temporal action localization improves when classification and localization are fed features from different pyramid layers, and the paper reports state-of-the-art mAP on five benchmarks with this design.","keywords":["temporal action localization","cross-layer task decoupling","feature pyramid","gated multi-granularity","FFT global filter","action classification","boundary regression","video understanding"],"falsifier":"Run the CLTDR ablation on a dataset of semantically defined actions (e.g., meetings or assembly steps) using the same features: if classification improves more when fed the lower-layer fine feature (or regression improves more with the higher-layer coarse feature), or if replacing the cross-layer input with a same-layer feature matches the reported gains, the claimed feature-utility gradient fails. A cheaper check is the paper's own Table 9, which already shows that adding both higher and lower features can slightly reduce mAP.","tokens_in":20329,"feed_emoji":"🎯","tokens_out":5940,"duration_ms":52896,"temperature":0.7,"pith_summary":"The paper tries to establish that temporal action localization improves when the classification and localization heads stop sharing the same input features. It proposes CLTDR, which feeds each pyramid layer's classification head with the semantically stronger feature from the next-higher layer and its regression head with the boundary-richer feature from the next-lower layer, gated by attention weights. A refinement head then fuses both cross-layer features to align the two outputs. Together with a lightweight Gated Multi-Granularity encoder that captures instant, local, and global temporal context via an FFT-based global filter, the method reports state-of-the-art average mAP on THUMOS14, MultiTHUMOS, EPIC-KITCHENS-100, ActivityNet-1.3, and HACS. If correct, this shows that feature-level task decoupling, not just parameter-level head separation, is a reliable source of TAL gains.","feed_headline":"Cross-layer task decoupling sets new action-localization records","feed_subtitle":"Classification gets coarse high-level features, localization gets fine low-level details, lifting mAP on all five datasets.","key_machinery":"The CLTDR decoder is the central mechanism: at each intermediate pyramid layer $l$, a transposed-convolution-upsampled feature $P^{l+1}$ is gated by an attention weight $W_{l+1}$ and added to $P^l$ to form the classification feature $f^l_{cls}$, while a downsampled $P^{l-1}$ is gated by $W_{l-1}$ and added to form the regression feature $f^l_{reg}$ (Equations 5-8). The RefineHead then fuses $P^{l-1}$, $P^l$, and $P^{l+1}$ into $f^l_c$ and predicts a classification scaling vector $R^l_{co}$ and boundary offsets $R^l_{so}$ and $R^l_{eo}$ used to adjust the coarse predictions. The GMG encoder is the supporting object: a three-branch module where a fully-connected branch captures instant information, a 1D depth-wise convolution captures local context, and an FFT branch with a learnable global filter $W_\\varphi$ captures global dependencies, with a ReLU-gated fusion to suppress redundancy.","core_discovery":"The central claim is that the conventional practice of giving classification and localization heads identical input features is a bottleneck, and that the fix is to route each task to the pyramid layer whose feature scale suits it. For classification, the paper combines each layer's feature with the temporally coarser, semantically richer feature from the layer above, selected by a learned attention weight; for regression, it combines with the temporally finer, boundary-detailed feature from the layer below. The decoupled outputs are then refined by a RefineHead that fuses higher and lower layer features and produces multiplicative classification adjustments and additive boundary offsets. On five benchmarks the full system reports top average mAP, and ablations attribute the gains to both the cross-layer routing and the multi-granularity encoder. The paper further claims the FFT-based global branch matches vanilla self-attention accuracy at roughly one-third the parameters.","pith_inferences":["The universal feature-utility gradient (higher = semantic, lower = boundary) is assumed per layer; a natural test would be to swap the routing on datasets where boundaries are semantically defined, and the paper's own Table 9 shows that adding both cross-layer inputs can slightly reduce mAP, suggesting the two routes carry partially redundant information.","The gains from the FFT global branch suggest that other token-mixing operators (e.g., Mamba-style state spaces) could be plugged into GMG, though the paper does not test them.","The method's sensitivity to feature quality (I3D vs VideoMAEv2 vs InterVideo2-6B) indicates that part of the reported SOTA is inherited from stronger pretrained features; a controlled comparison with fixed features would isolate the architectural contribution.","Since RefineHead only sees the immediate neighbors of a layer, using a wider or learnable cross-layer span might further improve long-range boundary consistency, but the paper does not explore it."],"forward_implications":["Classification and localization heads in TAL no longer need to consume identical features; feature-level decoupling becomes a transferable design choice.","On THUMOS14, CLTDR-GMG reports average mAP 69.9% with I3D features, 71.8% with VideoMAEv2, and 74.3% with InterVideo2-6B, each above prior state of the art; on MultiTHUMOS the advantage over TriDet grows to roughly 6-7 average mAP points.","The GMG encoder can be dropped into other TAL decoders: it improves ActionFormer's decoder by 0.8 and TriDet's by 0.4 average mAP under the paper's settings, while the CLTDR decoder improves ActionFormer, TemporalMaxer, TriDet, and ActionMamba encoders by 0.6-0.8.","An FFT-based global filter with a learnable filter achieves accuracy comparable to vanilla self-attention at 9.5M vs 26.1M parameters in the GMG comparison.","Because CLTDR is applied only to intermediate pyramid layers, adding more layers eventually hurts; the paper observes performance peaks at 6 layers on three datasets and 7 layers on two."],"supporting_citations":[{"why":"Supplies the Trident-head with relative boundary modeling that CLTDR reuses, the SGP encoder baseline, and the main comparison target on multiple datasets.","marker":"Shi et al. 2023"},{"why":"ActionFormer, a central baseline using shared-input decoupled heads and local self-attention; CLTDR and GMG are benchmarked against it and its decoder is replaced in plug-in experiments.","marker":"Zhang, Wu, and Li 2022"},{"why":"TemporalMaxer is a recent one-stage baseline that CLTDR-GMG outperforms, and its encoder is used in the CLTDR plug-in study.","marker":"Tang, Kim, and Sohn 2023"},{"why":"InternVideo2-6B features are the strongest input features used for the THUMOS14 experiments where the largest gains are reported.","marker":"Wang et al. 2024"},{"why":"VideoMAEv2 features are used for THUMOS14 and MultiTHUMOS experiments, providing a mid-strength feature comparison.","marker":"Wang et al. 2023"},{"why":"Prior work using Fourier transforms for token mixing; the GMG global branch extends this idea with a learned filter and inverse transform.","marker":"Lee-Thorp et al. 2022"},{"why":"TSCODE is the object-detection feature-level task decoupling approach that CLTDR extends to temporal action localization with attention-based selection and a refinement head.","marker":"Zhuang et al. 2023"},{"why":"Varifocal loss, used to supervise the refined classification output and handle sample imbalance.","marker":"Zhang et al. 2021"},{"why":"TSP R(2+1)D features are used for the ActivityNet-1.3 experiments.","marker":"Alwassel, Giancola, and Ghanem 2021"},{"why":"IoU-based regression loss used to supervise boundary predictions.","marker":"Rezatofighi et al. 2019"}],"fun_headline_variants":["Task-specific features boost temporal action localization","Decoupling classification and localization in videos","Cross-layer fusion lifts action localization mAP","Multi-granularity gating refines video action detection","New TAL method tops five benchmarks via task decoupling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that at every pyramid layer, the immediately higher layer's coarser feature always helps classification and the immediately lower layer's finer feature always helps localization; if that ordering is wrong for some action distribution, the cross-layer routing can hurt rather than help.","fun_headline_variants_meta":{"raw":{"variants":["Task-specific features boost temporal action localization","Decoupling classification and localization in videos","Cross-layer fusion lifts action localization mAP","Multi-granularity gating refines video action detection","New TAL method tops five benchmarks via task decoupling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1376,"prompt_tokens":957,"completion_tokens":419,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":348}},"tokens_in":573,"tokens_out":419,"duration_ms":4566,"temperature":1.0,"reasoning_tokens":348,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:13:19.047374+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the CLTDR ablation on a dataset of semantically defined actions (e.g., meetings or assembly steps) using the same features: if classification improves more when fed the lower-layer fine feature (or regression improves more with the higher-layer coarse feature), or if replacing the cross-layer input with a same-layer feature matches the reported gains, the claimed feature-utility gradient fails. A cheaper check is the paper's own Table 9, which already shows that adding both higher and lower features can slightly reduce mAP.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Trident-head with relative boundary modeling that CLTDR reuses, the SGP encoder baseline, and the main comparison target on multiple datasets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ActionFormer, a central baseline using shared-input decoupled heads and local self-attention; CLTDR and GMG are benchmarked against it and its decoder is replaced in plug-in experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"VideoMAEv2 features are used for THUMOS14 and MultiTHUMOS experiments, providing a mid-strength feature comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TSP R(2+1)D features are used for the ActivityNet-1.3 experiments."},{"cited_title":"D.; and Savarese, S","cited_arxiv_id":null,"evidence_quote":"IoU-based regression loss used to supervise boundary predictions."}],"review_version":1}