{"id":"5d478b08-fbcd-4f64-9245-bc1f595b5047","arxiv_id":"2511.13039","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A multi-grained network that recognizes seen actions with a supervised classifier and unseen actions via video-level coarse filtering plus proposal-level matching achieves state-of-the-art open-vocabulary temporal action localization.","lead":"MGCA-Net improves open-vocabulary temporal action localization by splitting action recognition into a seen-category classifier and a coarse-to-fine zero-shot classifier for unseen categories. It reports state-of-the-art results on THUMOS'14 and ActivityNet-1.3, with the largest gains on unseen action categories.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fine-grained novel classification relies on an untested base-to-novel transfer of the projection layer φ_proj; if it overfits to base categories, the reported ZS-TAL gains (1.2 avg on THUMOS, 0.3 on ActivityNet) would disappear.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: the coarse-to-fine classifier's only learned component is trained on base categories and must generalize to novel categories. This is the most direct threat to the central claim because the ZS-TAL SOTA margin over Ti-FAD is small, and the fine-grained assignment stage is what pushes MGCA-Net ahead. The reader's verdict of CONDITIONAL is appropriate; additional evidence or a dedicated transfer experiment would be needed to raise confidence. My proposed test would settle whether the concern actually lands, without requiring a change to the verdict.","tokens_in":18635,"tokens_out":15619,"duration_ms":140815,"concrete_test":"Hold out a random half of the base categories as pseudo-novel. Train MGCA-Net with C_novel = the remaining base categories (as in Sec. III-F), then evaluate the fine-classification step on the held-out categories. Compare top-1 assignment accuracy and the resulting ZS-TAL mAP against the in-distribution setting (C_novel = all base categories). If accuracy on held-out categories is substantially below (e.g., >10% relative drop), the transfer assumption is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The coarse-to-fine classifier's only learned component, φ_proj (Sec. III-F), is trained with C_novel set equal to C_base, because only base annotations exist. At inference, φ_proj is applied to proposal features and matched against text features of novel categories. This assumes the linear alignment learned among base categories transfers to unseen categories. The paper provides no evidence for this: the ablation in Table IV only contrasts the full coarse-to-fine pipeline against coarser variants within the same category split, so it cannot detect transfer failure. If φ_proj overfits to base-specific visual/textual alignment (e.g., by exploiting dataset-specific biases), novel proposals would be matched to the wrong coarse categories, directly reducing mAP novel. The ZS-TAL margins over Ti-FAD are thin (48.0 vs 46.8 on THUMOS 75/25; 35.0 vs 34.7 on ActivityNet), so a moderate transfer failure would erase the claimed SOTA advantage. The paper's own Limitations section flags a related generalization risk for the localizer, but the projection-layer transfer is the more direct assumption for the novel-classification path. Moreover, Eq. 5 in Sec. III-F writes S_np = F_np·F_coarse^T without explicitly showing φ_proj, despite the text stating φ_proj is used; this ambiguity obscures whether and how the projection is applied, which is exactly the assumption at risk.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"MGCA-Net addresses open-vocabulary temporal action localization by combining a category-agnostic ActionFormer localizer with an action presence predictor, a conventional base-action classifier, and a coarse-to-fine classifier that uses frozen CLIP image/text features. Novel categories are handled by computing video-level coarse categories via MIL over image-text similarities and then assigning each proposal to one of the coarse categories using a learned projection layer trained with contrastive loss on base categories. The paper reports state-of-the-art OV-TAL and ZS-TAL results on THUMOS'14 and ActivityNet-1.3 under 75/25 and 50/50 category splits, with ablations of each module.","tokens_in":19026,"tokens_out":5501,"duration_ms":50434,"significance":"If the empirical claims hold, the multi-granularity decoupling of base and novel action prediction and the training-free coarse category stage are a useful contribution to OV-TAL. The paper evaluates both OV-TAL and ZS-TAL, does not require prompt tuning, and releases code. However, the evidence is weakened by missing variance information, hyperparameter selection on the benchmark itself, and an untested base-to-novel transfer assumption in the projection layer, so the strength of the claimed state-of-the-art advantage is not yet fully established.","major_comments":[{"comment":"The paper states 'we perform 10 random splits to make it statistically robust' but reports only averaged mAP without standard deviations or per-split results. Several SOTA margins are small: Table II, 75%-25% ActivityNet average mAP is 35.0 for MGCA-Net versus 34.7 for Ti-FAD, and the 50%-50% comparison is 32.2 versus 32.0. Without a measure of spread or a significance test, the 'state-of-the-art' claim is not statistically supported. Please report standard deviations or per-split results and, if possible, paired significance tests.","section":"Sec. IV-A and Tables I-II"},{"comment":"Hyperparameters λ_retain, λ_base, N_coarse, N_neg, and the T_img/8 pooling size in Eq. 3 are selected via ablations on THUMOS'14 using the same test benchmark on which final numbers are reported. This makes the reported gains optimistic. Please either use a validation split for hyperparameter selection, or report sensitivity across the 10 random splits and state which choices were fixed before evaluation.","section":"Sec. IV-A and Tables VI-IX"},{"comment":"The projection layer φ_proj is the only learned component of the coarse-to-fine classifier, and it is trained with C_novel set equal to C_base because only base annotations exist. At inference it must transfer to unseen novel categories. The paper provides no direct evidence for this transfer; Table IV only compares variants of the coarse/fine pipeline within the same category split, so it cannot detect transfer failure. Moreover, Eq. 5 writes S_np = F_np · F_coarse^T without showing φ_proj, so it is unclear whether F_np is the raw image feature or the projected feature. Please clarify the exact computation and add an experiment that measures base-to-novel transfer of φ_proj, e.g., train on a subset of base categories and evaluate on held-out base categories, or compare with and without the projection on novel categories.","section":"Sec. III-F, Eq. 5"},{"comment":"The ground-truth generation pseudo-code is ambiguous. In the inner loop, for every temporal position i it appends one entry per annotation (either the ground-truth interval or (-1,-1)) to P_gt, so P_gt is a set of T_fpn × N_a entries, yet line 15 indexes P_gt_i as if it were a single interval associated with position i. This makes the action-presence training target undefined. Please correct the algorithm or replace it with a clear textual definition, e.g., assign the interval with maximum tIoU for overlapping cases.","section":"Sec. III-D, Algorithm 1"},{"comment":"The ZS-TAL protocol is underspecified. The text says MGCA-Net 'removes the conventional classifier and action presence predictor and only uses the coarse-to-fine classifier,' but it does not describe how proposals are filtered or how thresholds are set in this mode. Since Table II is a central result, this inference protocol must be defined precisely for reproducibility.","section":"Sec. IV-C"}],"minor_comments":[{"comment":"The dimensional relationship is unclear: F_np is defined as a mean of image features F_img, while F_coarse is selected from text features F_text. If D_img and D_text differ, Eq. 5 is not well-formed without the projection layer. Please state the dimensions explicitly and show where φ_proj is applied.","section":"Sec. III-F, Eq. 4-5"},{"comment":"The Limitations section acknowledges that unannotated action instances are treated as negatives during localizer training. This is a genuine limitation and should be discussed in relation to novel-category recall; please clarify whether the baseline methods share this issue and how it affects the comparison.","section":"Sec. IV-F"},{"comment":"The definition of coarse categories via 'top-N_coarse values' is imprecise when scores are tied or when fewer than N_coarse categories pass a meaningful threshold. Please specify tie handling and the minimum score threshold, if any.","section":"Sec. III-F, Eq. 3"},{"comment":"The paper reports averaging over 10 random splits but does not give the random seeds or the exact split generation procedure. This information is needed for reproducibility, especially since category splits drive the OV/ZS evaluation.","section":"Sec. IV-A"},{"comment":"The baseline description says the method with the conventional classifier, action presence predictor, and coarse-to-fine classifier removed 'uses a single template for action categories and predicts action categories based on the zero-shot capability of VLMs.' It is unclear whether proposals come from the localizer alone and how they are thresholded; please clarify or add a caption-level description.","section":"Fig. 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the multi-grained idea is worth considering, but the empirical evidence needs substantial strengthening before the SOTA claim can be accepted. The missing error bars and benchmark-based hyperparameter selection are particularly important because several reported margins are small. The projection-layer transfer issue is also central and should be addressed with a targeted experiment rather than only an ablation within the same category split."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a competent architecture paper that combines known pieces into a new SOTA for OV-TAL/ZS-TAL on two standard benchmarks. The novel-category gains are real but modest, and the ZS-TAL margins over Ti-FAD are thin. Worth a serious referee, but the authors need to clean up a few ambiguities and provide error bars before I'd trust the exact numbers.\n\nWhat's new: the specific combination—supervised base classifier plus action-presence predictor plus video-level-to-proposal-level coarse-to-fine zero-shot classifier—hasn't appeared before in the OV-TAL literature, and the ablations (Tables III–V) give credit where it's due: each module buys something, and the tIoU-based action-presence ground truth is a sensible choice that beats classification-score alternatives. The paper is also honest about its main known weakness: the Limitations section acknowledges that unannotated instances are treated as negatives during localizer training, biasing localization toward annotated actions.\n\nSoft spots. First, the transfer assumption in the novel-action path is real and under-tested. The projection layer φ_proj is trained with C_novel set equal to C_base (Sec. III-F), and the ablation in Table IV only varies the coarse-to-fine pipeline within the same split, so it can't detect a base-to-novel transfer failure. This matters because the ZS-TAL averages are close to Ti-FAD: 48.0 vs 46.8 on THUMOS 75/25, 35.0 vs 34.7 on ActivityNet. A moderate degradation in transfer would erase the SOTA claim. The stress-test note is fair, though not fatal. Second, Eq. 5 writes S_np = F_np · F_coarse^T without showing φ_proj, even though the text says the projection is applied. That needs fixing. Third, Algorithm 1 is genuinely confusing: it builds P_gt as a set with N_a entries per temporal position, then indexes P_gt_i as if it were a single interval. Either the pseudocode is wrong or the text underspecifies it. Fourth, no error bars are reported despite averaging over 10 random splits, and hyperparameters λ_retain, λ_base, N_coarse, N_neg are tuned on the same benchmarks. That's common in this line of work, but the lack of variance reporting is a real weakness. The code URL is given but no commit hash, so reproducibility is not yet verifiable.\n\nWho it's for: anyone working on OV-TAL or zero-shot TAL. The architecture is a useful baseline and the ablation story is informative. It deserves to go through peer review, with revisions. I'd send it.","headline":"Solid OV-TAL/ZS-TAL architecture paper with genuine but modest gains; the transfer assumption in the novel-action path is the main soft spot.","tokens_in":19449,"tokens_out":2254,"would_cite":true,"duration_ms":19412,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MGCA-Net decouples base and novel action recognition by granularity, reporting that supervising base actions at snippet level and novel actions via a coarse-to-fine video-to-proposal classifier raises novel-category localization by more tha","keywords":["open-vocabulary temporal action localization","zero-shot temporal action localization","multi-grained category awareness","coarse-to-fine classification","action presence prediction","vision-language models","contrastive learning","video understanding"],"falsifier":"Measure the recall of the coarse-category step: for videos containing novel actions, check how often the true category appears among the top-N coarse categories selected by the multi-instance learning step. If this recall is far below the reported novel accuracy, or if replacing the learned projection layer with raw vision-language similarity produces the same novel accuracy, then the coarse-to-fine classifier is not the source of the reported gains.","tokens_in":18553,"feed_emoji":"🎬","tokens_out":7348,"duration_ms":59054,"temperature":0.7,"pith_summary":"Open-vocabulary temporal action localization asks a model to find and label actions in untrimmed video, including categories it was never trained on. Existing methods classify proposals at a single granularity, which the authors argue hurts both familiar and unfamiliar actions. MGCA-Net splits the problem: a supervised classifier handles base (seen) actions at snippet granularity, while a coarse-to-fine classifier handles novel (unseen) actions by first deciding which categories occur anywhere in the video and then assigning each proposal one of those categories. The paper reports that this split improves both base and novel action localization, with novel-category gains of over six points on THUMOS'14 and over three points on ActivityNet-1.3. If the results hold, the insight is that decoupling category perception by granularity makes better use of a vision-language model's zero-shot ability than classifying every proposal against the full novel vocabulary at once.","feed_headline":"Two-stage labels for novel actions lift open-vocabulary localization","feed_subtitle":"Separating seen and unseen actions lets one network localize both, with the largest gains on categories never seen in training","key_machinery":"The load-bearing component is the coarse-to-fine classifier for novel actions, gated by the action-presence predictor. For each video, the classifier computes image-text similarities from a frozen vision-language model, aggregates them by multi-instance learning over the top eighth of frames, and keeps the top-N categories as coarse categories. Each novel proposal's feature is the mean of frame image features inside its temporal interval, passed through a two-layer projection layer, and the proposal is assigned the coarse category with the highest similarity. The projection layer is the only learned part of the novel path; it is trained with a contrastive loss whose positive text feature is","core_discovery":"The central claim is that multi-grained category awareness is the mechanism that makes open-vocabulary localization work. MGCA-Net uses a category-agnostic localizer to propose candidate segments. An action-presence score and a conventional classifier then divide proposals into base and novel groups: proposals with high action presence and a confident base label become base instances; proposals with high action presence but no confident base label become novel proposals. The novel path operates in two steps. At video granularity, a multi-instance learning step over per-frame image-text similarities selects the coarse categories present in the video. At proposal granularity, averaged image fe","pith_inferences":["My inference: the coarse stage sets a hard upper bound on novel recall; if a true novel category is missing from the top-N coarse categories, the fine stage cannot recover it. A direct test is to measure coarse-category recall against an oracle that always includes the true categories.","My inference: because only the projection layer is trained, the method points to a cheap adaptation recipe—keep the vision-language model frozen and train only the gating and projection heads. The same recipe could be tried on other video grounding tasks with minimal retraining.","My inference: the paper's stated limitation that unannotated action instances are treated as negatives during localizer training means the reported gap may depend on annotation completeness; on datasets with denser novel-action annotations, the advantage of the two-stage design could narrow or widen.","My inference: the improvement at high tIoU thresholds suggests the two-stage assignment mostly reduces category-label noise on novel proposals; a finer error analysis separating temporal misalignment from category confusion would test this directly."],"forward_implications":["A model can recognize actions it was never annotated for, as long as the action category can be described in text and a frozen vision-language model can relate that text to video frames.","Base and novel actions no longer compete in the same classifier, reducing the tendency to misclassify novel proposals as base actions and improving base-category accuracy at the same time.","The novel path is mostly training-free, so adapting to a new set of novel categories at inference requires only new text descriptions, not re-training.","The same architecture transfers to zero-shot temporal action localization by removing the conventional classifier and action-presence predictor, with the largest gains at high temporal intersection-over-union thresholds.","The design suggests that granularity, not just feature quality, is a first-order choice for open-vocabulary video tasks."],"fun_headline_variants":["Coarse-to-fine category awareness beats single-granularity in OV-TAL","Separating seen and unseen actions improves open-vocabulary localization","Two granularities for seen and unseen actions lift open-vocab localization","Zero-shot action localization gets a multi-grained boost","MGCA-Net multi-grained labels push open-vocab action localization to SOTA"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The novel-action path assumes that the contrastive alignment learned on base categories transfers to unseen categories, and that the localizer, trained with unannotated novel instances treated as negatives, still recalls novel action segments at all.","fun_headline_variants_meta":{"raw":{"variants":["Coarse-to-fine category awareness beats single-granularity in OV-TAL","Separating seen and unseen actions improves open-vocabulary localization","Two granularities for seen and unseen actions lift open-vocab localization","Zero-shot action localization gets a multi-grained boost","MGCA-Net multi-grained labels push open-vocab action localization to SOTA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001211,"raw_usage":{"total_tokens":4843,"prompt_tokens":787,"completion_tokens":4056,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":3960}},"tokens_in":531,"tokens_out":4056,"duration_ms":26041,"temperature":1.0,"reasoning_tokens":3960,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T21:54:25.326487+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the recall of the coarse-category step: for videos containing novel actions, check how often the true category appears among the top-N coarse categories selected by the multi-instance learning step. If this recall is far below the reported novel accuracy, or if replacing the learned projection layer with raw vision-language similarity produces the same novel accuracy, then the coarse-to-fine classifier is not the source of the reported gains.","supporting_citations":[],"review_version":1}