{"id":"03211fc3-6742-4d5a-94c7-534ebb38a4c0","arxiv_id":"2411.17481","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"DMR-JRG jointly learns video paragraph retrieval and weakly supervised sentence-level grounding through mutually reinforcing retrieval and grounding branches.","lead":"This paper defines a new task, video paragraph retrieval and grounding (VPRG), where a paragraph query must both retrieve the right video and locate the time interval for each sentence. The authors propose a dual-branch network that does retrieval and grounding together under weak supervision, and report gains over sentence-query baselines on three datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed outperformance over VSRG baselines is confounded: DMR-JRG uses full paragraph queries while all baselines use single-sentence queries, so the gains may reflect extra query context rather than the proposed architecture.","rationale":"The reader's weakest-assumption analysis focused on BTSE-TD's chronological-order premise. That is a real approximation, and Eqs. (18)–(19) go further: they use subtractive suppression 1 − ΣP_j, which can become negative when predicted score maps overlap, implicitly assuming events are both temporally ordered and non-overlapping. However, on datasets like ActivityNet Captions, paragraph sentences are usually assembled in temporal order, so the BTSE-TD assumption is less likely to invalidate the whole method than the evaluation confound. The strongest claim includes 'more accurately than existing VSRG methods,' and that claim is supported only by comparisons against single-sentence-query baselines. Since paragraph queries give the proposed method access to inter-sentence context that the baselines cannot use, the comparison tests query format as much as architecture. This is directly load-bearing: if a trivial mean-pooling paragraph adaptation of JSG matches DMR-JRG, the paper's main empirical contribution disappears. The TaCoS 'JSG*' row being numerically identical to the Charades-STA row is a concrete symptom that the baseline table needs verification. The conditionality of the reader's verdict is therefore appropriate, and no verdict change is needed; the requested baselines, code release, and corrected TaCoS numbers are the right conditions for acceptance.","tokens_in":23327,"tokens_out":10959,"duration_ms":113419,"concrete_test":"Run a matched paragraph-query baseline: adapt the released JSG (or SCN) model to accept a paragraph by pooling its per-sentence retrieval similarities using both mean and max pooling, while keeping its per-sentence grounding score maps; evaluate on the same splits and metrics as Tables II–III. If DMR-JRG does not beat both pooling variants at R@10/R@100 for IoU 0.3/0.5/0.7, the central outperformance claim fails. Separately, re-run JSG* on TaCoS from the released code to confirm whether the Table III baseline row is correct.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that DMR-JRG 'effectively solves VPRG' and does so 'more accurately than existing VSRG methods' rests on Tables II–III. However, every baseline (MCN, CAL, XML, HMAN, ReLoCLNet, MS-SL, JSG) is a VSRG method designed for single-sentence queries, while DMR-JRG is evaluated with paragraph queries containing multiple sentences. This is not a matched comparison: paragraph queries provide strictly more information for both retrieval and grounding, so the reported gains conflate the method's contribution with the task's richer query format. The TaCoS table compounds the problem: the 'JSG* [31]' row in Table III (7.23, 28.71, 5.67, 22.50, 3.28, 12.34) is identical to JSG's Charades-STA row in Table II, so the claimed TaCoS improvements are computed against a baseline value that appears erroneously replicated. Without a paragraph-query baseline (e.g., JSG or SCN with sentence-level scores pooled over the paragraph), the empirical superiority claim is not load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines a new task, Video Paragraph Retrieval and Grounding (VPRG), and proposes DMR-JRG, a weakly supervised dual-branch method. The retrieval branch aligns global paragraph and video features via InfoNCE and triplet losses; the grounding branch uses local, global, and temporal consistency modules (VTC-LD, VTFA-GD, BTSE-TD), with a GRRM module distilling grounding scores back into retrieval scores. Training requires only paragraph-video correspondence, not temporal boundary labels. The method is evaluated on ActivityNet Captions, Charades-STA, and TaCoS against VSRG baselines, with ablations and hyperparameter studies. The code is released.","tokens_in":23667,"tokens_out":4082,"duration_ms":37361,"significance":"If the central empirical claim were cleanly supported, this would be a useful contribution: it defines a sensible new task that combines paragraph-level retrieval with sentence-level grounding, it operates under a weak-supervision setting that reduces annotation cost, and it openly provides code. The ablation study (Table IV) and hyperparameter analyses (Tables V, Fig. 7-8) are thorough and internally consistent. However, the headline comparisons in Tables II-III are confounded by an unfair query-setting mismatch, and the TaCoS baseline row appears to be a verbatim copy of the Charades-STA row. The self-training nature of BTSE-TD (Eqs. 18-21) and GRRM (Eq. 17) also deserves closer scrutiny. Because the empirical superiority claim is the paper's third stated contribution, these issues are load-bearing for the current form of the paper.","major_comments":[{"comment":"The comparison is confounded: DMR-JRG uses full paragraph queries (§III.A) while every baseline (MCN, CAL, XML, HMAN, ReLoCLNet, MS-SL, JSG) is a VSRG method that takes a single-sentence query. The paper acknowledges this mismatch in §IV.C and asserts 'relative fairness,' but paragraph queries contain strictly more information: sentences in a paragraph can mutually disambiguate events and constrain the temporal layout, which helps both retrieval and grounding. The reported gains therefore conflate the query-format advantage with the proposed architecture. The claim that DMR-JRG 'significantly outperforms several latest VSRG methods' (Sec. I, contribution 3) is not established without paragraph-query baselines (e.g., JSG or SCN with sentence-level scores pooled over the paragraph) or a sentence-query variant of DMR-JRG.","section":"§IV.C, Tables II-III"},{"comment":"The row labeled 'JSG* [31]' on TaCoS lists values 7.23, 28.71, 5.67, 22.50, 3.28, 12.34, which are exactly the values reported for JSG on Charades-STA in Table II. Since TaCoS and Charades-STA differ substantially in video count, paragraph length, and domain, identical results are implausible and indicate a replication/copy error. As a result, all TaCoS improvement percentages in §IV.C (e.g., 11.51%, 16.74%) are computed against an invalid baseline, and the TaCoS superiority claim is unsupported until JSG is rerun on TaCoS and the table is corrected.","section":"Table III"},{"comment":"BTSE-TD generates soft labels gt^f and gt^r by taking argmax over P^f_m and P^r_m, which are themselves predictions of the same network (Eqs. 18-19), and then uses those labels to supervise the main score map P_m via binary cross-entropy (Eqs. 20-21). This is a self-training loop with no ground-truth anchor; its correctness hinges on the assumption stated in §III.G that 'sentences describing these events in text paragraphs follow a certain logical sequence.' The assumption is not verified on the experimental datasets, and ActivityNet Captions in particular does not guarantee chronological caption order. If a paragraph is not temporally ordered, the pseudo-labels will be systematically wrong and the BCE losses will train P_m with incorrect supervision. The paper should either provide evidence that paragraphs in the three datasets are consistently ordered, or include a diagnostic showing that BTSE-TD improves over a version trained without this self-supervision.","section":"§III.G, Eqs. (18)-(21)"},{"comment":"The GRRM loss forces the retrieval score Sr(·) to match the grounding score Sg(·), which is itself a product of the same network. The paper states that grounding scores are used as 'pseudo-labels' to reinforce retrieval, but it does not show that Sg is more reliable than Sr before distillation, nor does it evaluate the retrieval branch in the setting where the correspondence between videos and paragraphs is truly unknown at training time. The claim in Sec. I that the method works 'when the correspondence between videos and paragraphs is unknown' is only demonstrated in the standard scenario where correspondence is known during training and removed at inference. The paper should clarify the intended meaning of 'correspondence unknown' and, if the stronger claim is intended, provide an experiment that trains without paragraph-video pairing, or at least justify why the current training setting suffices.","section":"§III.F, Eq. (17)"}],"minor_comments":[{"comment":"The method name is typed inconsistently: the first contribution bullet reads 'Dual-task Mutual Reinforcing Embedded Joint Video Paragraph Retrieval and Grounding (DMP-JRG)', while the rest of the paper uses DMR-JRG.","section":"Sec. I, contribution list"},{"comment":"The index term 'Video Paragraph Retrival' contains a misspelling; it should read 'Retrieval'.","section":"Index Terms"},{"comment":"The text says 'E1 and E1 are two Transformer encoders' but the two encoders should be E1 and E2, based on Eq. (1).","section":"§III.B, Eq. (1)"},{"comment":"The inference section refers to 'TSVT-TD', which appears to be a typo for the BTSE-TD module.","section":"§III.H.2"},{"comment":"The hyperparameter section repeatedly writes 'ActiveNet Captions'; the correct dataset name is 'ActivityNet Captions'.","section":"§IV.E"},{"comment":"The caption reads 'It comprises there core parts'; 'there' should be 'three'.","section":"Fig. 3 caption"},{"comment":"The sentence 'our approach shows significant improvements in the VSRG task' should refer to the VPRG task, since that is the task being evaluated.","section":"§IV.C (ActivityNet discussion)"}],"recommendation":"major_revision","confidential_remarks":"The identical TaCoS and Charades-STA baseline numbers in Tables II-III are a serious rigor concern: it is either a transcription error or an indication that the baseline was not actually run on TaCoS. The authors should be asked to provide the raw logs or a corrected table. The novelty claim of being the 'first attempt' at VPRG should also be checked against PRVG [30], which addresses dense video grounding with paragraph-level queries, even though it does not perform corpus-level retrieval. Overall, the proposed task and framework are reasonable, but the empirical validation in the current form does not yet support the stated claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the VPRG task is a natural extension of VSRG, and the authors put together a clean multi-task architecture with sensible components. But the empirical case, as written, is not persuasive: every baseline runs on single-sentence queries while DMR-JRG gets full paragraphs, the TaCoS baseline numbers are copied from Charades-STA, and the BTSE-TD module looks like it has a training loop that never updates the forward/reverse predictors. I'd send it to review, but only with a request for major revision.\n\nWhat's new: the paper is the first (to my knowledge) to formulate video-paragraph retrieval as a joint retrieval-and-grounding task with only video-paragraph correspondence during training. That's a real, usable setting. The framework is coherent: InfoNCE/triplet retrieval branch, 2D-TAN-style local grounding, global contrastive alignment, and a temporal synchronization block. The ablation in Table IV shows monotonic gains as modules are added, which is at least internally consistent.\n\nSoft spots. First, the comparison is confounded. The stress-test is right: Tables II-III compare against VSRG methods that consume one sentence, while DMR-JRG consumes the whole paragraph. Paragraphs contain more information for both retrieval and localization, so the margin over JSG may just be extra query context. The paper acknowledges this in Sec. IV-A and then dismisses it as 'relative fairness,' which is not convincing. A proper paragraph-query baseline is missing. Second, the JSG* row in Table III is byte-for-byte identical to the Charades-STA JSG row from Table II. That's an error or a data leak; either way it invalidates the TaCoS improvement claim until corrected. Third, reading Sec. III-G carefully: the soft labels for the main score map come from argmax of P_f and P_r, which are produced by independent, unshared prediction layers. The BCE losses (20)-(21) are computed with these as targets for P_m, so gradients flow only to P_m, not to the forward/reverse predictors. Those predictors then stay at their initial values. The improvements from BTSE-TD in the ablation are hard to explain unless the paper has an unstated training signal. Fourth, the chronological-order assumption (Sec. III-G) is explicitly stated, and the results may degrade on out-of-order paragraphs—the paper acknowledges generalization limits in the conclusion.\n\nOverall: the task definition is worth taking seriously, but the empirical claims need a matched baseline, a fixed TaCoS table, and a clarification (or correction) of the BTSE-TD training. As is, I wouldn't trust the specific numbers, but the architecture and problem are worth a round of careful revision.","headline":"VPRG is a sensible new task, but the reported gains are confounded by query format and a suspect baseline row, and the temporal-synchronization module has a training-loop problem as written.","tokens_in":24156,"tokens_out":4221,"would_cite":false,"duration_ms":59139,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper defines VPRG, a task that jointly retrieves a video for a paragraph query and localizes each sentence's event, and claims DMR-JRG solves it without temporal labels.","keywords":["video paragraph retrieval","video paragraph grounding","weakly supervised grounding","cross-modal retrieval","contrastive learning","temporal grounding","dual-task learning","multi-dimensional congruity"],"falsifier":"Randomly shuffle the sentence order of test paragraphs in ActivityNet Captions before feeding them to a trained DMR-JRG model and compare grounding accuracy at IoU thresholds 0.3, 0.5, and 0.7 with the unshuffled results; a sharp drop, particularly relative to an ablation without the bidirectional synchronization module, would show that the chronological-order assumption carries the reported gains.","tokens_in":23148,"feed_emoji":"🎬","tokens_out":13923,"duration_ms":107875,"temperature":0.7,"pith_summary":"The paper introduces a task it calls Video Paragraph Retrieval and Grounding (VPRG): given a paragraph narrating several events, a system must find the right video in a corpus and mark the time span of each sentence's event. The authors argue that existing paragraph-grounding methods need temporal start/end labels and assume the matching video is already known, which is unrealistic. They propose DMR-JRG, a two-branch network in which retrieval and grounding reinforce each other: retrieval aligns whole paragraphs with whole videos, grounding refines the alignment sentence by sentence, and grounding scores are fed back to improve retrieval. The model is trained only on paragraph-video correspondences and does not need temporal labels. If the reported results hold, this would remove a major annotation bottleneck for paragraph-level video understanding.","feed_headline":"New model finds right video and times each sentence, no labels needed","feed_subtitle":"Both tasks reinforce each other: timestamp labels are unnecessary and pairs are only needed during training.","key_machinery":"The load-bearing mechanism is the mutual-reinforcement loop between a coarse and a fine cross-modal feature space. The retrieval branch (Video-Paragraph Cross Modal Retrieval) constructs the coarse space with class tokens and InfoNCE plus triplet losses. The grounding branch constructs the fine space with three modules: Visual-Textual Consistency on Local Dimension aligns sentence features to candidate moments via 2D-TAN-style proposals and masked-language reconstruction; Visual-Textual Feature Alignment on Global Dimension fuses top candidate-moment features and aligns a fused class token to the paragraph token; and Bidirectional Temporal Synchronization of Events on Temporal Dimension runs forward and reverse branches whose predictions become soft pseudo-labels supervising the main score map. The Grounding Reinforcement Retrieval Module closes the loop by using grounding scores as pseudo-labels for retrieval scores with an MSE loss.","core_discovery":"The paper's central claim is that VPRG can be solved under weak supervision by treating retrieval and grounding as mutually reinforcing tasks. This is, by the authors' account, the first attempt at the combined task. The retrieval branch uses class tokens and contrastive losses to build a coarse paragraph-video feature space; the grounding branch builds a fine-grained space through local sentence-video consistency, global class-token alignment, and bidirectional temporal synchronization, with the forward and reverse synchronization branches producing pseudo-labels that train the main score maps. A grounding-reinforcement-retrieval module regresses retrieval scores toward grounding scores, completing the mutual-reinforcement loop. The paper reports that DMR-JRG outperforms existing sentence-based retrieval-and-grounding methods on ActivityNet Captions, Charades-STA, and TaCoS, and interprets the gains as evidence that paragraph context plus multi-dimensional consistency helps both tasks.","pith_inferences":["A natural next test is to compare DMR-JRG against fully supervised VPG methods under a shared protocol; the paper only compares with VSRG baselines, so the gap between weak and full supervision is not yet measured.","If the chronological-order assumption transfers, the bidirectional synchronization idea could be applied to other ordered-alignment problems, such as aligning instruction text to procedural videos or narrated slides to recordings.","The method's reliance on paragraph-video pairs during training still assumes one paragraph maps to one video in the batch; extending it to partially relevant or multi-paragraph queries would require reformulating the contrastive negatives."],"forward_implications":["A single weakly supervised model can retrieve a video from a paragraph query and localize every sentence's event, using only paragraph-video correspondence during training.","Paragraph-level queries can beat sentence-level retrieval-and-grounding systems on the same datasets, since the extra context improves both retrieval and grounding.","Grounding quality transfers to retrieval: the pseudo-label feedback from grounding to retrieval (GRRM) improves retrieval scores in the ablations.","Bidirectional chronological synchronization contributes more than either direction alone, indicating that exploiting sentence order is useful for grounding.","The approach remains effective on a visually homogeneous dataset (TaCoS), where distinguishing events requires language as much as vision."],"supporting_citations":[{"why":"Defines VSRG and gives the sentence-query baseline whose evaluation protocol and metrics the paper uses for comparison.","marker":"[31]"},{"why":"Supplies the weakly supervised masked-language reconstruction and rank/reward loss that VTC-LD adapts into the local-dimension grounding module.","marker":"[49]"},{"why":"Provides the candidate-moment construction and the Temporal Adjacent Network that generate the 2D score maps.","marker":"[32]"},{"why":"Gives the InfoNCE contrastive objective used to align paragraph and video class-token features in retrieval and global alignment.","marker":"[57]"},{"why":"Gives the triplet ranking loss used alongside InfoNCE for cross-modal retrieval and global alignment.","marker":"[58]"}],"fun_headline_variants":["Mutual reinforcement lets one model retrieve videos and ground sentences","No timestamp labels: retrieval and grounding reinforce each other","Mutual reinforcement for video retrieval and grounding without labels","Retrieval and grounding reinforce each other, skipping timestamp labels","One model does video retrieval and paragraph grounding with no temporal supervision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the sentences of a paragraph describe events in the same chronological order in which they occur in the video; if a paragraph is not ordered that way, the forward and reverse synchronization branches generate wrong pseudo-labels and the grounding score maps are trained with incorrect supervision.","fun_headline_variants_meta":{"raw":{"variants":["Mutual reinforcement lets one model retrieve videos and ground sentences","No timestamp labels: retrieval and grounding reinforce each other","Mutual reinforcement for video retrieval and grounding without labels","Retrieval and grounding reinforce each other, skipping timestamp labels","One model does video retrieval and paragraph grounding with no temporal supervision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000638,"raw_usage":{"total_tokens":2947,"prompt_tokens":961,"completion_tokens":1986,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":1905}},"tokens_in":577,"tokens_out":1986,"duration_ms":14961,"temperature":1.0,"reasoning_tokens":1905,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:04:28.651387+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Randomly shuffle the sentence order of test paragraphs in ActivityNet Captions before feeding them to a trained DMR-JRG model and compare grounding accuracy at IoU thresholds 0.3, 0.5, and 0.7 with the unshuffled results; a sharp drop, particularly relative to an ablation without the bidirectional synchronization module, would show that the chronological-order assumption carries the reported gains.","supporting_citations":[{"cited_title":"Joint searching and grounding: Multi-granularity video content retrieval,","cited_arxiv_id":null,"evidence_quote":"Defines VSRG and gives the sentence-query baseline whose evaluation protocol and metrics the paper uses for comparison."},{"cited_title":"Weakly-supervised video moment retrieval via semantic completion network,","cited_arxiv_id":null,"evidence_quote":"Supplies the weakly supervised masked-language reconstruction and rank/reward loss that VTC-LD adapts into the local-dimension grounding module."},{"cited_title":"Learning 2d temporal adjacent networks for moment localization with natural language,","cited_arxiv_id":null,"evidence_quote":"Provides the candidate-moment construction and the Temporal Adjacent Network that generate the 2D score maps."},{"cited_title":"Momentum contrast for unsupervised visual representation learning,","cited_arxiv_id":null,"evidence_quote":"Gives the InfoNCE contrastive objective used to align paragraph and video class-token features in retrieval and global alignment."},{"cited_title":"Facenet: A unified embed- ding for face recognition and clustering,","cited_arxiv_id":null,"evidence_quote":"Gives the triplet ranking loss used alongside InfoNCE for cross-modal retrieval and global alignment."}],"review_version":1}