{"id":"f6d7ad03-f394-49f3-81dd-c14846d5a33c","arxiv_id":"2412.12791","paper_version":2,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A complementary masking approach trains a mask generator with positive and negative masked captioning losses to implicitly align event locations with captions under weak supervision.","lead":"The paper proposes a weakly supervised dense video captioning method that learns to localize and describe events using only video-level captions. It introduces complementary masking, where positive and negative masks on video frames produce captions that together describe the whole video, simplifying event localization.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Masked-captioning losses may not be what drives localization; a caption-shuffling control is needed to confirm the implicit-alignment mechanism.","rationale":"The reader's weakest_assumption identifies the same broad area, but this stress-test sharpens it: the issue is not only the absence of temporal supervision, but also that the caption text itself is present in the input prompt, which can decouple the captioning loss from the mask. The ablations in Table 3 and the localization comparison in Table 7 are genuine empirical support, and the released code is a plus. However, neither experiment rules out the alternative explanation that localization is driven mainly by the diversity loss and by the fixed chronological ordering of event embeddings, with the captioning losses providing only a weak alignment signal. The proposed caption-shuffling control is cheap, uses the existing codebase, and directly tests the causal role of the location-caption alignment; if it fails, the central novelty of the paper would need to be substantially restated. Therefore the paper should be accepted only on the condition that this control is run and reported, or the claim is weakened accordingly.","tokens_in":15709,"tokens_out":9732,"duration_ms":104289,"concrete_test":"Run the localizing-stage training on ActivityNet Captions with the caption-to-mask assignment randomly permuted per video: for each event index i, use a randomly chosen ground-truth caption C_{pi(i)} as the positive target and the remaining captions as the negative target, keeping the full pipeline, hyperparameters, and inference procedure otherwise identical. If SODA/CIDEr and the localization F1 of Table 7 remain within a small margin of the reported values, the implicit alignment is not the mechanism and the central claim fails; if they collapse, the alignment signal is load-bearing and the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that the positive/negative masked-captioning losses (Eqs. 12 and 15) are sufficient to train the Gaussian mask predictor to locate events without temporal annotations. The weak point is that these losses may not provide the alignment signal attributed to them. First, in Eqs. 11 and 14 the full target caption(s) are included in the prompt, so a pretrained GPT-2 decoder can predict most caption tokens from the text prefix alone; the gradient reaching the mask generator through the visual prefix is concentrated in the first few tokens and does not strongly constrain mask boundaries. Second, the training objective is compatible with degenerate solutions: a mask that is too wide but still covers the event keeps the positive loss low, and the diversity loss only encourages masks to spread apart, not to match true event boundaries. The localization numbers in Table 7 show the overall pipeline works, but they do not isolate whether localization is produced by the caption-alignment signal or by the combination of the frozen order of event embeddings, the diversity loss, and the temporal encoder's inductive bias. This is the load-bearing assumption behind the claimed implicit location-caption alignment.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a weakly-supervised dense video captioning (WSDVC) method that learns temporal event localization through complementary masking. A dual-mode captioning module generates full-video captions (providing the number of events) and masked-video captions, while a mask generation module produces differentiable Gaussian masks. Positive and negative masked-captioning losses and a diversity loss train the mask generator without temporal annotations. Experiments on ActivityNet Captions, YouCook2, and ViTT report consistent improvements over weakly-supervised baselines and results competitive with fully-supervised methods. The paper includes ablations for each component, an analysis of mask types, model size, hyperparameters, and event overlap robustness, and it releases code.","tokens_in":15922,"tokens_out":8771,"duration_ms":85217,"significance":"If the claimed mechanism holds, the work offers a practical simplification of WSDVC: it removes external proposal detectors and random proposal sampling while achieving state-of-the-art weakly-supervised performance and competitive fully-supervised results. The manuscript is unusually transparent: it reruns baseline methods on new datasets, ablates each loss and mask type, and provides code. The main weakness is that the core mechanism—that complementary captioning losses are what drive accurate localization—is not directly verified; this is the key risk to the paper's central claim and should be addressed before publication.","major_comments":[{"comment":"The central claim that the complementary masked-captioning losses produce implicit location-caption alignment is not isolated from alternative training signals. Table 3 ablates the positive and negative captioning losses and the diversity loss against composite captioning metrics, but the localization metrics from Table 7 are not reported for these ablations. Consequently, the strong localization performance in Table 7 could be driven by the diversity loss, the learned event queries, or the temporal encoder's inductive bias rather than by the complementary captioning objective. A control that breaks the correspondence between masks and captions (e.g., training with a random permutation of the captions assigned to the masks, or with captions from a different video) would test the mechanism directly. This is a load-bearing point for the paper's stated paradigm and should be addressed with additional experiments or with localization-metric ablations.","section":""},{"comment":"The event embeddings e_i are described only as a 'randomly initialized learnable embedding' for each caption proposal. It is not specified whether these embeddings are globally shared queries, per-video queries, or initialized from caption text. If they are globally shared and ordered, the positive captioning loss may simply align each query to the i-th event in the fixed temporal order of the ground-truth captions, meaning the reported alignment is partly a consequence of the order rather than of learned semantic correspondence. The authors should clarify the initialization and discuss the implications for videos whose caption order does not follow the temporal order.","section":""},{"comment":"The prompts for both positive and negative masked captioning contain the full target captions. Because the decoder is a pretrained language model, a large fraction of the caption tokens can be predicted from the text prefix, and the gradient signal reaching the mask generator through the masked video features is concentrated in the first token(s). The paper does not quantify this effect. The authors should either report an analysis of gradient magnitudes or run a control in which the caption is removed from the prompt (using only '[MASK] 1 event:') to show that the mask can still be trained. This would strengthen the claim that the visual masking signal, rather than the language prior, drives localization.","section":""}],"minor_comments":[{"comment":"There are minor grammatical errors, e.g., 'the temporal location of event' and 'an dual-mode'. These should be corrected.","section":""},{"comment":"The notation T_{n,i} appears in the parameter list but not in the summation, and the denominator N is not explicitly defined in the equation. Please clarify the token-counting conventions.","section":""},{"comment":"The main text states that scores are averaged over tIoU thresholds 0.3, 0.5, 0.7, and 0.9, but Table 1's caption does not mention this. Add this information to the table caption for clarity.","section":""},{"comment":"The model sizes reported for GPT2 are inconsistent: Table 4 lists 'GPT2-Base' as 104.62M, while Supplementary Table 6 lists 'GPT2' as 163.89M. Please reconcile the numbers and clarify which configuration is used in each table.","section":""},{"comment":"The paper does not report standard deviations or significance tests for any of the main results. For a method claiming consistent improvements, it would be helpful to report variance across runs or at least state that differences are stable across seeds.","section":""},{"comment":"The reimplementation of WSDEC and PWS-DVC on YouCook2 and ViTT lacks implementation details. Please provide hyperparameters, number of epochs, and any preprocessing shared with the proposed method to support reproducibility.","section":""}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong empirical contribution with released code and careful ablations. The main concern is the lack of a direct test of the proposed implicit-alignment mechanism; adding the control experiments suggested in the major comments would substantially strengthen the contribution. The paper fits the scope of the venue, and the writing is generally clear but needs a pass for grammar and notation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the complementary masking training paradigm: predict Gaussian masks for events, then train positive and negative masked captioning losses so that captions generated from the masked and inverse-masked video complement each other. That is a real departure from the reconstruction and distillation approaches in the cited weakly supervised work, and it removes the proposal machinery entirely. The experiments back it up: three datasets, multiple metrics, ablations for every component, re-run baselines, a model-size comparison, overlap analysis, and released code. The gains over prior weakly supervised methods are consistent, and the method also lands close to fully supervised baselines. This is careful empirical work.\n\nThe soft spot is the localization mechanism. In training, the full target captions are already present in the prompt text, so a GPT-2 decoder can largely copy from the prompt; the gradient reaching the mask generator through the visual prefix is concentrated in the first few tokens and may not strongly constrain mask boundaries. The paper does not include a shuffled-caption control (e.g., corrupting or permuting the prompt captions) to confirm that the masks actually respond to visual-caption alignment rather than a combination of event-embedding order, the diversity loss, and the temporal encoder's inductive bias. The overlap analysis in Table 8 is a partial response, and the ablations show the positive masked captioning loss is essential, so the mechanism is not empty. But it remains an empirical bet.\n\nMinor issues: no variance estimates, and no theoretical guarantee for the alignment. These are minor and fairly standard for this subfield.\n\nThe stress-test concern does not sink the paper. It points to a missing control that should be easy to run and would strengthen the causal story. The central empirical claim—that complementary masking outperforms existing weakly supervised methods without proposals—holds up. This paper is for researchers working on dense video captioning and weakly supervised temporal grounding. It deserves a serious referee and, in my view, acceptance with a request for the shuffled-caption control and variance reporting in revision.","headline":"A solid, well-evaluated WSDVC method with a genuinely new masking trick, though the localization mechanism is less clean than the title implies.","tokens_in":16405,"tokens_out":3660,"would_cite":true,"duration_ms":37317,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that complementary masking lets a captioning model learn event locations implicitly from video-level captions alone, removing the need for temporal annotations and complex proposal pipelines.","keywords":["Weakly-supervised dense video captioning","Complementary masking","Implicit location-caption alignment","Gaussian mask","Event localization","Temporal grounding","Dual-mode captioning","Video captioning"],"falsifier":"Retrain the model on ActivityNet Captions with each video's sentence annotations randomly permuted across events: if predicted masks still localize the true events about as well as in the intact training run, the captions cannot be supplying the localization signal. As a cheaper check, measure the CIDEr gap between captions generated from a predicted mask and from a random mask of the same width on the validation set — a near-zero gap would mean the masks are not aligned to caption content.","tokens_in":15555,"feed_emoji":"🎬","tokens_out":10592,"duration_ms":87973,"temperature":0.7,"pith_summary":"Dense video captioning normally requires knowing where each event starts and ends in a video; this paper claims those temporal boundaries can be learned implicitly from caption text alone. The trick is complementary masking: for each event the model predicts a soft Gaussian mask, then must write that event's caption from the masked video and all other captions from the inverse mask, so the two half-descriptions have to fit together into the full video description. That pairing pushes each mask onto the region whose visual content actually supports its caption. On ActivityNet Captions the method beats every weakly supervised baseline and lands close to fully supervised systems, and the authors report the first weakly supervised results on YouCook2 and ViTT. If the claim holds, the elaborate event-proposal machinery of prior weakly supervised methods — predefined proposals, external detectors, heavy inference-time sampling — is unnecessary.","feed_headline":"Masked captioning locates video events, no timestamps needed","feed_subtitle":"Complementary masking beats weakly supervised baselines and rivals fully supervised dense captioning.","key_machinery":"Complementary masking with differentiable Gaussian masks is the load-bearing mechanism. For each event, a Transformer decoder over learnable event embeddings and temporally encoded CLIP frame embeddings predicts a center $\\mu_i$ and width $\\sigma_i$, which define a Gaussian mask $M_i(t) = \\exp\\big(-(t/N_v - \\mu_i)^2 / (2(\\sigma_i/\\tau)^2)\\big)$ over the frame dimension. The positive captioning loss $L(\\hat{v}_b, \\hat{r})$ requires that the caption written from the masked video be exactly that event's caption, the negative loss $L(\\check{v}_b, \\check{r})$ requires the inverse mask to yield all other captions, and a cosine-similarity diversity loss keeps the masks apart; the two losses sum so that writing the full description forces each mask to sit on the region that visually supports its caption. The dual-mode captioning module supplies the supporting bookkeeping: a full-video mode with a '[FULL] N events' prompt fixes the event count, and masked modes with '[MASK] 1 event' / '[MASK] N−1 events' prompts structure the complementary pairs. This mechanism replaces the explicit proposal-and-matching loop of earlier weakly supervised systems.","core_discovery":"The central discovery is that an event's location and its caption can be aligned with no temporal supervision at all, using only the requirement that captions generated from a masked video and from its inverse mask be complementary. A mask generation module predicts a Gaussian center and width for each event from a learned event embedding; positive masking keeps only that region and must reproduce the event's caption, while negative masking keeps everything else and must reproduce the remaining captions. Because caption accuracy depends on which region the mask reveals, gradient flows from the two captioning losses reshape the masks toward the true event segments. With CLIP features the full model reaches a SODA score of 6.08 and CIDEr of 33.42 on ActivityNet Captions, surpassing all weakly supervised comparators and a fully supervised Vid2Seq baseline; temporal localization recall (53.72) trails fully supervised PDVC (55.42) by a small margin, suggesting implicit alignment recovers most of what explicit boundary supervision provides.","pith_inferences":["The same complementary-masking loss should apply to other weakly supervised grounding tasks — temporal sentence grounding, moment retrieval, or weakly supervised action localization — wherever a captioning head can be made sensitive to which region it sees; the paper only demonstrates it for dense video captioning.","The paper's inference still uses a refinement stage that re-generates captions from the predicted masks; an untested consequence of the training scheme is that the masks alone might be precise enough to serve as final timestamps, making inference a single forward pass.","Because nothing ties the mask to a true boundary, the mechanism's precision is bounded by the caption model's sensitivity to content: on datasets where captions are predictable from global video cues, the masks should drift to coarse or arbitrary regions while caption scores stay high — a failure mode the overlap analysis only partially probes.","The explicit complementary structure suggests a stronger test than the paper runs: if captions are randomly permuted across events during training and localization accuracy survives, then the alignment signal is not coming from the caption–region pairing."],"forward_implications":["Event localization in WSDVC can be trained without any proposal generator: the mask module is learned purely from captioning losses, so inference needs only a single pass through the mask predictor rather than sampling many candidate segments.","The event count is recovered from the full-video captioning mode at inference, letting the pipeline handle videos of unknown content without a separate counting or proposal stage.","The Gaussian (soft) mask is essential to the mechanism: replacing it with a hard binary mask collapses CIDEr from 33.42 to 16.96, because the hard mask blocks gradients from reaching the predicted location.","Both masked captioning directions matter — dropping the positive loss drops CIDEr to 20.22, while dropping the negative loss costs about 3 points — so complementarity, not just masking, is what drives alignment.","The same recipe transfers to other datasets: the model reports the first weakly supervised results on YouCook2 and ViTT and outperforms the reimplemented baselines there."],"supporting_citations":[{"why":"Defines the dense video captioning task and the detect-then-describe paradigm the paper contrasts with its implicit alignment.","marker":"(Krishna et al. 2017)"},{"why":"Introduces the weakly supervised dense video captioning problem and the iterative segment reconstruction approach this method abandons.","marker":"(Duan et al. 2018)"},{"why":"A reconstruction-line baseline whose concept-learner sentence localizer the paper must outperform.","marker":"(Chen and Jiang 2021)"},{"why":"PWS-DVC, a pretraining-enhanced weakly supervised baseline compared with and without extra datasets.","marker":"(Choi, Chen, and Yoon 2023)"},{"why":"The distillation-and-matching baseline that relies on external proposal detectors, the design the paper removes.","marker":"(Wu et al. 2021)"},{"why":"Source of the differentiable Gaussian mask construction that the mask generation module adapts for event proposals.","marker":"(Zheng et al. 2022a,b)"},{"why":"Vid2Seq, the fully supervised model whose SODA and CIDEr scores the paper matches without temporal supervision.","marker":"(Yang et al. 2023)"},{"why":"Supplies the frozen CLIP ViT-L/14 frame encoder whose features drive both captioning and localization.","marker":"(Radford et al. 2021)"},{"why":"Defines the SODA metric used to report the headline overall scores.","marker":"(Fujita et al. 2020)"}],"fun_headline_variants":["Complementary masks locate video events without timestamps","No timestamps? No problem: complementary masks align captions to events","Implicit alignment: masking makes captions find their own events","Complementary masks align captions to events without any temporal labels","No event boundaries? Complementary masking still localizes and describes events"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The mask predictor is trained only through captioning losses, with nothing ever comparing a predicted mask to a true event segment, so the entire method rests on the bet that caption accuracy is sensitive enough to which region of the video the mask reveals to push each Gaussian onto the real event.","fun_headline_variants_meta":{"raw":{"variants":["Complementary masks locate video events without timestamps","No timestamps? No problem: complementary masks align captions to events","Implicit alignment: masking makes captions find their own events","Complementary masks align captions to events without any temporal labels","No event boundaries? Complementary masking still localizes and describes events"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00062,"raw_usage":{"total_tokens":2885,"prompt_tokens":965,"completion_tokens":1920,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":1835}},"tokens_in":581,"tokens_out":1920,"duration_ms":12633,"temperature":1.0,"reasoning_tokens":1835,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:43:01.915742+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the model on ActivityNet Captions with each video's sentence annotations randomly permuted across events: if predicted masks still localize the true events about as well as in the intact training run, the captions cannot be supplying the localization signal. As a cheaper check, measure the CIDEr gap between captions generated from a predicted mask and from a random mask of the same width on the validation set — a near-zero gap would mean the masks are not aligned to caption content.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the weakly supervised dense video captioning problem and the iterative segment reconstruction approach this method abandons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A reconstruction-line baseline whose concept-learner sentence localizer the paper must outperform."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PWS-DVC, a pretraining-enhanced weakly supervised baseline compared with and without extra datasets."},{"cited_title":"H.; Miech, A.; Pont-Tuset, J.; Laptev, I.; Sivic, J.; and Schmid, C","cited_arxiv_id":null,"evidence_quote":"Vid2Seq, the fully supervised model whose SODA and CIDEr scores the paper matches without temporal supervision."}],"review_version":1}