{"id":"cf3b98a3-c02a-433b-ab90-5a84f48971fd","arxiv_id":"2412.09601","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"A training reformulation that turns timestamp prediction into iterative offset refinement, with an auxiliary L1 loss, improves Video LLM temporal grounding.","lead":"TimeRefine improves how video-language AI models locate the right moments in a video by having them first guess roughly and then refine the guess multiple times, plus a training signal that penalizes bigger timing errors more. This makes a plug-and-play upgrade that raises temporal grounding accuracy on two standard benchmarks without changing the underlying model architecture.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The refinement training data never contains the model's own predictions: offsets are trained on GT-plus-Gaussian-noise, so the claimed self-correction may be an artifact of the denoising curriculum and L1 head rather than a learned refinement process.","rationale":"TimeRefine is a well-executed empirical paper: the ablations in Tables 2-5 isolate the refinement task, the number of steps, noise scale, loss type, and decoding strategy, and gains are consistent across two datasets and two backbone models. The central claim, however, is that the model 'self-corrects errors' through iterative offset prediction. This mechanism rests on the training data generation in Eqs. 3-4. A close reading shows that each training step's rough prediction s_k is GT plus independently sampled Gaussian noise, and the next rough prediction s_{k+1} is another independent sample, not the result of applying the model's previous offset to its own previous prediction. Under teacher forcing, the model learns to emit a coarse-to-fine sequence of increasingly denoised GT estimates, but it is never trained on its own errors. The offset tokens, whose targets are the sampled noises, provide supervision that is conditionally independent of the video/query; the only learnable signal is the sequence of segment tokens. At inference, the model autoregressively generates its own initial segment and then offsets. Whether the learned offset function transfers depends on whether the model's self-generated initial predictions look like GT plus Gaussian noise. This is closely related to the reader's weakest assumption, but it is more severe: even if the Gaussian distribution matched, the training procedure never conditions the next refinement step on the model's own previous output, so the network cannot be said to learn self-correction in the usual sense. That said, the method could still be valuable as a training-time curriculum, and the L1 auxiliary head is a clean, well-ablated addition. The paper's empirical claims about mIoU improvements are plausible even if the stated mechanism is imprecise. A direct correlation analysis of predicted offsets vs. initial errors would settle whether true self-correction is learned; an ablation using the model's own errors as training noise would test the Gaussian-noise assumption. If either test fails, the paper's narrative should be revised, but the headline numbers may stand.","tokens_in":16912,"tokens_out":9965,"duration_ms":94287,"concrete_test":"Using the released model, decode a held-out set from Charades-STA or ActivityNet and record the initial predicted segment (s0,e0), the predicted offsets (o0^s,o0^e), and the final prediction. Compute the initial error d = GT - (s0,e0) and the predicted correction c = (o0^s,o0^e). If the model has learned self-correction, c should correlate positively with d (e.g., Spearman rho > 0) and |c| should grow with |d|. If c is near zero and uncorrelated with d, the refinement sequence is not correcting the model's own errors, and the gains should be attributed to the curriculum and L1 head. Alternatively, re-train with s_k generated from a base model's actual errors on the training set instead of Gaussian noise; if performance does not drop, the Gaussian-noise assumption is not load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eqs. 3-4 generate each training 'rough prediction' (s_k, e_k) by adding Gaussian noise to the ground truth, and the target offset is exactly that noise. The next rough prediction (s_{k+1}, e_{k+1}) is independently re-sampled from GT with a smaller noise scale; it is never computed by applying the model's previous offset to its previous prediction. Under teacher forcing, the model is trained to output a coarse-to-fine sequence of GT-corrupted segments, but it is never trained on its own initial predictions or on corrections of those predictions. At inference, the model generates its own s_0 and then predicts offsets; this train/test mismatch is the load-bearing assumption. Concretely, the offset target o_k is random noise centered at 0 conditional on the video/query (though it equals GT - s_k, where s_k is itself a random perturbation). If the model's test-time rough predictions have a different error structure (e.g., systematic bias or correlated start/end errors), the learned offset function may not transfer. The reported gains (Tables 1-5) could therefore come from the multi-step denoising curriculum and the auxiliary L1 head, rather than from a learned self-correction mechanism. The paper does not provide evidence that the model's predicted offsets correlate with its own errors at inference.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes TimeRefine, a training-objective modification for Video-LLM temporal grounding. Instead of directly generating start and end timestamps, the model generates a coarse segment and then predicts offsets to the target, repeating this over K refinement steps; an auxiliary L1 head on the <refine> token is added during training and discarded at inference. On ActivityNet Captions and Charades-STA, applying TimeRefine to VTimeLLM improves mIoU from 30.4 to 34.0 and from 31.2 to 36.2, and applying it to VTG-LLM improves Charades-STA mIoU from 34.4 to 35.6. The paper also reports ablations of the refinement target, number of steps, noise schedule, auxiliary loss type, and decoding strategy.","tokens_in":17217,"tokens_out":9755,"duration_ms":85491,"significance":"The empirical contribution is clearly presented and well ablated: the authors isolate offset prediction, K=4 with the {5,3,1,0} schedule, and the L1 auxiliary loss, and they report consistent gains across two base models and datasets under matched training data. The plug-and-play nature of the method and the promise of code release increase its value. The main reservation is interpretational: during training the rough segments are sampled from ground truth plus Gaussian noise, and the model is never conditioned on its own predictions, so the 'self-correction' mechanism is not directly supported. The reported gains may still hold as a coarse-to-fine denoising curriculum with auxiliary supervision, but the paper should test this before the self-correction claim is accepted.","major_comments":[{"comment":"The central 'self-correct' claim is not supported by the training procedure. The model never sees its own rough prediction during training: (s_k,e_k) are obtained by adding Gaussian noise to the ground-truth timestamps, and the target offsets are exactly the sampled noise. At inference, the model's initial segment comes from its own autoregressive generation, whose error distribution may differ from the zero-mean, isotropic, truncated Gaussian used in training. The reported gains could therefore come from the denoising curriculum and the auxiliary L1 head rather than from learning to correct the model's own errors. Please provide (a) a quantitative comparison of the test-time error distribution of the model's first generated segment with the training noise schedule, and (b) an experiment in which refinement is trained on the model's own generated rough segments (e.g., scheduled sampling or iterative self-training), or, at minimum, reframe the contribution as coarse-to-fine/denoising supervision rather than self-correction.","section":"Sec. 3.2.2 (Eqs. 3-4)"},{"comment":"There is no control that isolates sequential refinement from the extra supervision tokens. A variant that predicts K segments in parallel, or predicts a coarse segment and the target without using the offsets as conditioning, would test whether the improvement is due to the refinement mechanism or merely to having more regression targets. The IoU-prediction variant in Table 2, which also improves over no refinement, suggests that additional feedback alone can help. This does not invalidate the empirical gains, but it is necessary to support the paper's interpretation that the model learns to self-correct through iterative refinement.","section":"Sec. 4.4 (Tables 2-5)"},{"comment":"The indexing of the refinement sequence is inconsistent. Equation (2) indexes tuples from 0 to K, giving K+1 tuples, but Eq. (3) defines K Gaussian distributions for k=1..K, and Sec. 3.4.1 sets K=4 with four variance values. The construction of the initial tuple (s0,e0,o^s0,o^e0) is never defined, and it is unclear whether the zero-noise final step is part of the target sequence. Please fix the indexing so that Eq. (3)-(4) and the K=4 schedule are consistent and reproducible.","section":"Sec. 3.2.2 (Eq. 2) and Sec. 3.4.1"}],"minor_comments":[{"comment":"The sentence 's = s_k + o^s_k, e_i = e_k + o^e_k' contains a typo: e_i should be e_k, and the superscripted offset notation in the tuple is inconsistent with the subscripted notation used in the following equations.","section":"Sec. 3.2.2, after Eq. (2)"},{"comment":"The text calls sigma^2 = {5,3,1,0} a set of standard deviations, while Eq. (3) defines N(0,sigma^2_k); please use consistent notation for variances versus standard deviations.","section":"Sec. 3.4.1 and Table 3"},{"comment":"The example shows only two refinement steps (the second <refine> yields the final 20s to 25s), whereas the default setting is K=4; please label the figure as illustrative or extend it to match the default schedule.","section":"Figure 1"},{"comment":"The L1 term 1/(2|S|)|Ŝ - S| should specify the norm and the summation over segments and endpoints; as written, the normalization is ambiguous when |S| > 1.","section":"Eq. (6)"},{"comment":"The sentence 'We apply Gaussian noise with a standard deviation of 5 seconds' for the IoU/offset comparison should state the full schedule {5,0}, matching Table 3's K=2 row; otherwise Tables 2 and 3 are hard to reconcile.","section":"Sec. 4.4.1"},{"comment":"The phrase 'discarded during inference ˙For the prediction' contains a stray punctuation mark before 'For'.","section":"Sec. 3.3"},{"comment":"The claim of zero-shot evaluation for VTG-LLM-TIMEREFINE on Charades-STA should be stated more carefully, and the paper should clarify whether the VTimeLLM-TIMEREFINE model is evaluated on the same splits used for training, since the zero-shot terminology could be misleading.","section":"Sec. 4.2 and Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution, but the central mechanism claim is stronger than the evidence. If the authors can provide the model-generated rough-prediction experiment and fix the indexing, I would be willing to accept; otherwise, the contribution should be reframed as a coarse-to-fine denoising curriculum with auxiliary L1 supervision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here’s my take on TimeRefine. It’s a clean, well-ablated method that gives real gains on temporal grounding for video LLMs. The idea is simple: instead of predicting start/end timestamps directly, the model produces a coarse segment and then iteratively predicts offsets to the target, with a noise schedule on the training sequence, plus an auxiliary L1 head. The numbers hold up: 3.6 and 5.0 mIoU improvements over VTimeLLM on ActivityNet and Charades-STA, and 1.2 over VTG-LLM on Charades-STA, with the same training data and settings. The ablations in Tables 2–5 justify each design choice: offset prediction beats IoU and direct prediction, K=4 with sigma^2={5,3,1,0} is best, and L1 helps over CE alone. This is a solid engineering contribution, and the plug-and-play claim is credible since it works on two base models.\n\nThe main soft spot is the train/test mismatch in the “self-correction” narrative. During training, rough predictions are generated as GT plus Gaussian noise; the model is never shown its own predictions as inputs to later refinement steps. At inference, it autoregressively generates its own initial segment and offsets. If the model’s initial predictions have a different error structure than zero-mean Gaussian (e.g., systematic bias or correlated start/end errors), the learned offsets may not transfer. The paper doesn’t test this. The gains could come from the denoising curriculum and the L1 head rather than from true self-correction. This is not fatal—the method works—but the central narrative is oversold. A simple fix would be to analyze the distribution of the model’s s_0 at test time, or to train with scheduled sampling where some steps use the model’s own outputs.\n\nAlso minor: no error bars or multiple seeds, so the 1.2% gain on VTG-LLM is within noise. Code is promised but not released yet, and there are a few typos. But none of this undermines the core empirical result.\n\nWho’s it for? Anyone working on video LLMs or temporal grounding. It’s a useful trick to add to your toolbox. I’d send it to reviewers—the empirical work is real, and the mismatch concern is addressable in revision rather than a desk-reject issue. I’d want reviewers to push on the self-correction mechanism and ask for uncertainty estimates.","headline":"Solid, well-ablated engineering contribution to temporal grounding in video LLMs; the “self-correction” story is weaker than the numbers because the training sequence never uses the model’s own predictions.","tokens_in":773,"tokens_out":920,"would_cite":true,"duration_ms":46354,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TimeRefine claims that reformulating video temporal grounding as iterative offset refinement—rough segment followed by repeated corrections, trained with an L1 auxiliary head—yields 3.6% and 5.0% mIoU gains over VTimeLLM on ActivityNet…","keywords":["temporal grounding","video large language models","iterative refinement","offset prediction","L1 auxiliary loss","coarse-to-fine","ActivityNet Captions","Charades-STA"],"falsifier":"Record the first-round segment predictions TimeRefine makes on a held-out set, compare the empirical distribution of their start/end errors to the $\\mathcal{N}(0,\\sigma^2)$ training noise used to generate refinement sequences, and check whether refinement gains persist when the two distributions differ substantially; a test set with systematically biased first guesses (for example, all target events placed near one end of the video) would expose a mismatch that the fixed symmetric noise never simulates.","tokens_in":16724,"feed_emoji":"⏱️","tokens_out":6886,"duration_ms":56561,"temperature":0.7,"pith_summary":"TimeRefine argues that Video LLMs fail at temporal grounding largely because the standard training target is mismatched to the regression nature of timestamps: cross-entropy loss treats a prediction of 21s and 100s as equally wrong when the ground truth is 20s, and only two answer tokens carry localization signal. The paper's solution is to reformulate grounding as an iterative refinement task in which the model first predicts a rough segment and then predicts offsets that move that segment toward the target, repeating this correction several times. Training examples are produced by adding Gaussian noise of decreasing variance (schedule $\\sigma^2 = \\{5,3,1,0\\}$ seconds over $K=4$ steps) to ground-truth timestamps, and an auxiliary L1 head on the refinement token supplies distance-aware supervision. With the same training data and settings, TimeRefine reports 3.6% and 5.0% mIoU improvements over VTimeLLM on ActivityNet Captions and Charades-STA, and 1.2% over VTG-LLM on Charades-STA. If correct, the work shows that localization ability in Video LLMs can be bought by changing what the model is asked to predict, rather than by building new visual or temporal encoders.","feed_headline":"Video LLMs localize better by refining timestamps step by step","feed_subtitle":"Coarse prediction plus repeated offset corrections lifts mIoU by up to 5 points on ActivityNet and Charades-STA.","key_machinery":"The carrying object is the iterative time refinement sequence with its control-token grammar, paired with the auxiliary L1 regression head attached to the <refine> token. The sequence turns one sparse timestamp prediction into several rounds of segment-and-offset predictions, so the model receives dense, distance-aware supervision and practices self-correction; the decreasing noise schedule $\\sigma^2=\\{5,3,1,0\\}$ simulates a coarse-to-fine path from rough guess to exact target. The auxiliary head, a single linear layer on the hidden state of <refine>, provides the gradient that cross-entropy cannot, namely a penalty proportional to how far the predicted segment lies from the ground truth, and it can be removed after training without losing accuracy.","core_discovery":"The central discovery is that a coarse-to-fine temporal refinement sequence is a better learning target for LLM-based temporal grounding than direct timestamp prediction. For a target segment $S=(s,e)$, the method converts the ground truth into the sequence $S' = ((s_0,e_0,o^s_0,o^e_0),\\ldots,(s_K,e_K,o^s_K,o^e_K))$ with $s=s_k+o^s_k$ and $e=e_k+o^e_k$, serialized with control tokens <seg_start>, <offset>, <refine>, and <seg_end>. The rough guesses $s_k,e_k$ are synthesized in training by sampling offsets from zero-mean Gaussians with variances $\\{5,3,1,0\\}$ seconds at the four steps, so the model is asked to correct perturbations that shrink toward the ground truth. At every <refine> token, a lightweight linear head predicts the current segment and is trained with an L1 loss, while the LLM's ordinary cross-entropy loss is retained. At inference the auxiliary head is discarded and the final prediction is read from the last refinement step, $s_K+o^s_K$ and $e_K+o^e_K$. The paper reports that this scheme improves VTimeLLM by 3.6 mIoU on ActivityNet Captions and 5.0 mIoU on Charades-STA, and VTG-LLM by 1.2 mIoU on Charades-STA.","pith_inferences":["An unstated extension is to replace the fixed Gaussian noise schedule with noise matched to the model's measured first-round error distribution; the paper's ablation already shows fixed noise beats duration-normalized noise, but it never tests error-matched noise.","The offset-refinement pattern is a generic remedy for any continuous output an LLM must generate as tokens, so the same trick may transfer to bounding-box regression, audio event onsets, or 3D localization tasks that suffer from the same cross-entropy blindness.","A testable prediction is that refinement gains are concentrated on segments whose first rough guess is near but not exactly on target; for very short or highly ambiguous events, extra refinement tokens might add compute without improving IoU.","One could also train with fewer refinement steps and, at inference, feed the model's own last output back as the next rough guess, turning the learned correction behavior into an open-ended self-improvement loop."],"forward_implications":["On ActivityNet Captions, adding TimeRefine to VTimeLLM-7B is reported to lift R@0.3 from 44.0 to 48.0, R@0.5 from 27.8 to 33.6, R@0.7 from 14.3 to 17.5, and mIoU from 30.4 to 34.0.","On Charades-STA, VTimeLLM-TIMEREFINE is reported to lift mIoU from 31.2 to 36.2, and VTG-LLM-TIMEREFINE from 34.4 to 35.6.","The auxiliary head need not exist at test time: decoding from the last refinement step, from the auxiliary head, or from both yields essentially the same mIoU, so the method remains compatible with standard next-token generation.","The refinement objective also improves dense video captioning metrics on ActivityNet Captions, suggesting temporal understanding gains transfer to related tasks.","Because only the supervised target and loss are changed, the method can be layered onto any LLM-based temporal grounding model without modifying its architecture."],"supporting_citations":[{"why":"Supplies the VTimeLLM baseline, its stage-one checkpoint, and its stage-two and stage-three training data for the direct comparison on ActivityNet Captions and Charades-STA.","marker":"Huang et al. [2024a]"},{"why":"Supplies the VTG-LLM baseline and its training data and hyperparameters, which TimeRefine reuses to demonstrate plug-and-play integration.","marker":"Guo et al. [2024a]"},{"why":"Provides the denoising-diffusion inspiration for generating training refinement sequences with progressively decreasing Gaussian noise.","marker":"Ho et al. [2020]"},{"why":"Reinforces the progressive-denoising idea that shapes the K=4 noise schedule used to synthesize rough-to-fine training pairs.","marker":"Dhariwal and Nichol [2021]"},{"why":"Defines the ActivityNet Captions dataset used for temporal grounding evaluation and the supplementary dense captioning evaluation.","marker":"Krishna et al. [2017]"},{"why":"Defines the Charades-STA dataset used for the main temporal grounding evaluation and the VTG-LLM comparison.","marker":"Gao et al. [2017]"}],"fun_headline_variants":["Coarse-to-fine refinement boosts video temporal grounding accuracy","TimeRefine: stepwise timestamp refinement improves video LLM grounding","Refining timestamps stepwise lifts temporal grounding mIoU by 5 points","Video LLMs get better at time localization via iterative offset refinement","Iterative offset refinement sharpens video temporal grounding in LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that Gaussian noise added to ground-truth timestamps during training produces rough predictions whose error pattern matches the model's own initial mistakes at test time, so the offset corrections learned under that synthetic noise will transfer to real errors.","fun_headline_variants_meta":{"raw":{"variants":["Coarse-to-fine refinement boosts video temporal grounding accuracy","TimeRefine: stepwise timestamp refinement improves video LLM grounding","Refining timestamps stepwise lifts temporal grounding mIoU by 5 points","Video LLMs get better at time localization via iterative offset refinement","Iterative offset refinement sharpens video temporal grounding in LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000416,"raw_usage":{"total_tokens":2221,"prompt_tokens":1093,"completion_tokens":1128,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":709,"completion_tokens_details":{"reasoning_tokens":1038}},"tokens_in":709,"tokens_out":1128,"duration_ms":7853,"temperature":1.0,"reasoning_tokens":1038,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:52:08.483968+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record the first-round segment predictions TimeRefine makes on a held-out set, compare the empirical distribution of their start/end errors to the $\\mathcal{N}(0,\\sigma^2)$ training noise used to generate refinement sequences, and check whether refinement gains persist when the two distributions differ substantially; a test set with systematically biased first guesses (for example, all target events placed near one end of the video) would expose a mismatch that the fixed symmetric noise never simulates.","supporting_citations":[],"review_version":1}