{"id":"fa05cc6a-5017-4055-82f2-ff32ef2fa175","arxiv_id":"2505.24282","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Generating LLM-written start/end descriptions of actions and converting rigid boundary labels into similarity-weighted probability targets yields consistent but small gains in language-driven action localization across three datasets.","lead":"Researchers used a large language model to generate sentence descriptions of how an action starts and ends, then fed those descriptions into video models to make action-boundary predictions more consistent. Tests on three benchmark datasets with five base models show small but consistent gains, though no code or error bars are provided.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Boundary probability module may be silently inactive: τ=0.8 on raw cosine-minus-distance scores (Eqs. 7–9) is likely above achievable CLIP similarity values, so the claimed soft semantic supervision may degenerate to hard binary labels or empty probability scores.","rationale":"The reader's weakest assumption is about pseudo-boundary quality relative to human annotations. That is a real conceptual risk, but a more immediate and checkable weak point is whether Section 3.5's probability model can ever fire as written. The threshold τ = 0.8 is imposed on a raw cosine-minus-distance score, not on a normalized score; cosine similarities from CLIP-style encoders are typically in a range where 0.8 is an extreme value, and the distance term only lowers the score. The paper's own hyperparameter discussion says τ = 1.0 'filters out low-confidence frames,' which implies the authors observed scores above 0.5, but no distribution is reported. If the threshold zeroes out the boundary probabilities for most samples, Eq. 10 degenerates to a binary segmentation label, and the claimed semantic soft supervision in the abstract and conclusion is not what drives the results. Additionally, s′ and e′ are independent maxima with no ordering constraint, so Eq. 10 can be undefined when s′ > e′. This is an internal consistency check, not a challenge from outside consensus: it can be settled by a few lines of instrumentation. The proposed test would also indirectly quantify the reader's concern, because |s′ − τ_s| measures whether pseudo-boundaries are close to human annotations. My recommended verdict remains conditional: release the diagnostic and code, and depending on the outcome, either the mechanism is confirmed or the explanation must be revised. Thus no change to the reader's verdict is needed.","tokens_in":14501,"tokens_out":9704,"duration_ms":136567,"concrete_test":"Instrument Eqs. 5–10 on the released code with the LLaMa3-8B expansions and one base model (e.g., QD-DETR) on the Charades-STA or QVHighlights training split. Report: (1) the histogram of max_i S_s(i) and max_i S_e(i) over all video-query pairs; (2) the fraction of pairs with at least one frame above τ = 0.8; (3) the distribution of |s′ − τ_s| and the fraction of pairs with s′ > e′. If the ≥τ fraction is below about 10% or s′ > e′ occurs nontrivially, Eqs. 8–10 are not producing the described soft semantic probabilities; rerun the Table 2 ablation under that condition (e.g., with explicit binary labels) to determine whether the 'Probability' gain survives and what it actually measures.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.5 is the mechanism behind the second contribution and the 'probability scores' in the abstract. Eq. 7 defines S_s(i) = sim(F_v,i,F_s) − dis(i,s′); Eq. 8 zeroes every S_s(i) < τ; Eq. 9 min-max normalizes only surviving positive scores. With τ = 0.8, survival requires sim − dis ≥ 0.8. sim is explicitly a cosine similarity between CLIP-derived video tokens and an LLM-expanded query embedding; raw CLIP cosine values are typically far below 0.8, and dis is never negative. If no frame of a clip passes the threshold, p_s (and similarly p_e) is all zero, and Eq. 10 collapses to a hard binary label (1 inside [s′, e′], 0 outside). The paper reports no statistics on S_s; the ablation in Sec. 4.4 and Table 4 credit this module with part of the gain, but if the soft labels are empty, that gain cannot come from semantically graded boundary supervision. The pseudo-boundary pair (s′, e′) is also unconstrained: independent argmaxes in Eq. 6 can yield s′ > e′, in which case the three-branch piecewise function in Eq. 10 is undefined. Because the paper's headline claim is that model-agnostic soft boundary supervision improves all tested systems, this degenerate-mode risk is load-bearing and directly checkable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two plug-in modules for language-driven temporal action localization: (1) LLM-powered query expansion, which uses LLaMa3-8B to generate textual descriptions of action start and end boundaries, and (2) a query-guided temporal modeling module plus a boundary probability modeling module that converts rigid boundary annotations into soft probability labels via pseudo-boundary generation and semantic-similarity/temporal-distance scoring. The modules are inserted into five existing DETR-style base models (QD-DETR, Eatr, TaskWeave, UVCOM, CG-DETR) and evaluated on QVHighlights, Charades-STA, and TACoS, reporting consistent improvements across all 45 metric/dataset/base-model combinations in Table 1. The paper also provides component ablations, hyperparameter sensitivity studies, robustness to boundary-noise perturbations, a query-quality user study, and qualitative visualizations.","tokens_in":14673,"tokens_out":6228,"duration_ms":81986,"significance":"If the empirical claims hold, the method offers a simple, model-agnostic recipe for improving boundary prediction using LLM-generated text and learned soft supervision, which could be practically useful and easy to adopt. The evaluation is broad (five models, three datasets, 45 positive deltas) and the ablation structure is clean, including a useful robustness experiment to boundary annotation noise. However, the absolute gains are often small (many below 1 point), the comparisons are single-run without statistical significance or released code, and the mechanism underlying the probability-modeling module is not demonstrated to actually produce soft labels under the stated threshold. The paper is therefore valuable as an incremental contribution, but its central 'seamless and universally beneficial' claim needs stronger support.","major_comments":[{"comment":"The threshold τ=0.8 in Eq. (8) is applied to scores S_s(i)=sim(F_{v,i}, F_s)-dis(i,s'), where sim is a cosine similarity and dis≥0. Since any cosine similarity is at most 1, S_s(i)<1 for every frame, and for CLIP-style features typical near-boundary cosine similarities are far below 0.8. The paper does not report the fraction of frames or training samples for which S_s(i) actually exceeds τ. More tellingly, Fig. 5 shows that τ=1.0—a value at which no frame can survive the threshold because S_s(i)<1—degrades performance only modestly. This is consistent with the 'soft' probability supervision being almost always empty, so Eq. (10) degenerates to hard binary labels on [s',e']. The observed gains would then come from the pseudo-boundary shift, not from graded probability supervision. Please report survival statistics for S_s and S_e, and add a control that uses hard binary labels on the pseudo-boundary interval, to isolate the contribution of graded probabilities.","section":"3.5, Eqs. (7)-(9), Fig. 5"},{"comment":"The pseudo-boundaries s' and e' are computed by independent argmaxes in Eq. (6) and are never constrained to satisfy s'≤e'. If s'>e', the piecewise definition in Eq. (10) is undefined for frames between e' and s', and the first and third branches overlap on that interval. This is a correctness gap in the training loss and could silently occur when the start query is semantically more similar to frames after the annotated start boundary or vice versa. The authors should impose an ordering constraint (e.g., clip s' to at most e') or otherwise handle this case, and report how often s'>e' occurs in practice.","section":"3.5, Eqs. (5)-(6), (10)"},{"comment":"The headline claim that 'all methods integrated with our modules consistently achieve better performance on all three datasets' rests on single-run comparisons without standard deviations, significance tests, or released code for the reproduced baselines. Several deltas are small (e.g., QD-DETR R1@0.7 on QVHighlights +0.71, TaskWeave mAP on TACoS +0.71) and within typical run-to-run variation for DETR-style training. Because this is the paper's central model-agnostic claim, the authors should provide multiple seeds or bootstrap confidence intervals, and ideally release the expanded queries, code, and configuration files to enable verification of the reproduced baselines and the reported gains.","section":"4.3, Table 1"},{"comment":"The user study in Sec. 4.5 measures whether the LLM-generated start/end descriptions are relevant to the original query and consistent across similar queries, but it does not test whether those descriptions actually match the visual boundary motion in the specific video being localized. The failure case in Sec. 4.6.2 ('person starts laughing') shows that the premise fails for subtle motions. Since the mechanism of the method depends on the expanded query being visually grounded at the true boundary, the paper should acknowledge this limitation explicitly and, if possible, evaluate grounding directly (e.g., by checking whether the generated text retrieves the annotated boundary frame better than the original query).","section":"4.5, 4.6.2"}],"minor_comments":[{"comment":"Typo: 'moduels' should be 'modules' in the sentence 'all the methods integrated with our moduels consistently achieve better performance'.","section":"4.3"},{"comment":"The row for LLaMa2-13B appears as 'LLaMa2-13B Touvron et al. (2023)' and the R1@0.7 value is printed as '38.3136.22' without a separating space; please fix the formatting and the duplicated citation.","section":"Table 3"},{"comment":"The text describing τ=1.0 says it 'filters out low-confidence frames' but does not state the stronger fact that with τ=1.0 the soft probabilities in Eq. (10) are necessarily empty because S_s(i)=sim-dis<1 for all i; this makes τ=1.0 a hard-label control and should be discussed explicitly.","section":"4.4, Fig. 5"},{"comment":"The notation F_s and F_e denotes query features, but it is not specified whether these are pooled sentence-level embeddings or token-level features; clarify how the multi-token query representation is reduced to a single vector for cosine similarity.","section":"3.5, Eq. (5)"},{"comment":"The hyperparameters a and b are fixed to 1 in the main experiments; the sensitivity analysis in Fig. 4 covers this, but the paper should state whether the same values are used for all five base models and all three datasets, since the 'off-the-shelf' claim implies no per-model tuning.","section":"3.4, Eq. (4)"},{"comment":"The dataset name 'Qvhighlights' is capitalized inconsistently (QVHighlights in the abstract and elsewhere); please unify the spelling throughout.","section":"4.1"},{"comment":"Typo: 'Data Availibility' should be 'Data Availability'.","section":"Data Availability"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central claim is plausible but currently thinner than it appears: the 45/45 positive deltas are single-run and the probability-modeling mechanism may be degenerate under τ=0.8. I would urge the editor to require the authors to provide threshold-survival statistics, a hard-label control, multiple seeds, and code/data release for the baselines before this can be accepted. The paper is not circular in the sense of evaluating on generated labels, but the supervision target in Eqs. (5)-(10) is produced from the same frozen CLIP-style features used by the base model, so a control using only the pseudo-boundary hard labels is essential to separate the contribution of the soft-labels claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. Using LLM-generated start/end descriptions as expanded queries is a real departure from prior boundary-uncertainty work (Otani, Pan, Huang), and the reported 45/45 positive deltas across five base models and three datasets are directionally consistent. But the boundary probability module, the second headline contribution, may be silently off: at tau=0.8, Eq. 7's cosine-minus-distance scores will almost never reach threshold with raw CLIP-style features, so the soft supervision (Eqs. 8-9) collapses to hard binary labels at the pseudo boundaries. The paper gives no statistics on these scores, and the ablation in Table 4 cannot distinguish real soft supervision from pseudo-boundary label replacement. This is directly checkable and is the load-bearing question for the second contribution.\n\nWhat the paper does well: the query expansion idea is clean, the prompts are sensible, and integrating into five DETR variants is a useful engineering effort. The ablation study separates the two modules, and the noise-perturbation experiment (Fig. 6) supports the robustness claim. The user study, while measuring text quality rather than video matching, is a fair attempt to assess generation consistency. The cited related work is appropriate, and the authors are honest about the failure case for subtle motions.\n\nSoft spots, in proportion:\n- No standard deviations or significance tests anywhere; all baselines are the authors' own reproductions without released code. With deltas of 0.3-1.5 points, variance could change the story.\n- Hyperparameters a, b, tau are selected on the same dataset (Charades-STA) used for the main evaluations, so reported gains there are partially optimistic.\n- One TACoS cell (CG-DETR mAP 33.28) is an outlier relative to all other TACoS mAP values (~20-23) and looks like a misprint.\n- 'Any existing models' overstates the tested scope: five DETR-based methods, not proposal-based or span-based models.\n- The user study rates text quality, not whether generated boundary descriptions align with visually distinguishable moments in particular videos. The failure case shows they don't always.\n\nOverall: the first contribution (query expansion) is likely sound and useful; the second (probabilistic supervision) is plausible but unproven, and the threshold issue could mean the mechanism is not what the authors think. The paper deserves serious peer review and a major-revision request: release code, multi-seed runs, report score distributions, and correct the table. If the module is indeed inactive, the method remains an incremental improvement, but the probabilistic-supervision claim needs to be withdrawn or reworked.","headline":"A well-scoped plug-in with a novel LLM-query-expansion idea, but the soft-boundary module may be silently discrete at its chosen threshold, and the evidence lacks variance controls.","tokens_in":15351,"tokens_out":3480,"would_cite":false,"duration_ms":43718,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding LLM-written descriptions of an action's start and end, together with probability-based boundary supervision, improves boundary prediction across five base models and three video-moment-retrieval benchmarks.","keywords":["language-driven action localization","video moment retrieval","boundary uncertainty","query expansion","large language models","boundary probability modeling","temporal grounding"],"falsifier":"Take the videos where pseudo-boundaries differ from the original annotations and ask independent annotators to mark the action start/end again. If the pseudo-boundary is not closer to the median re-annotation than the original boundary is, the core premise fails. A cheaper probe: on actions with visually subtle starts, such as \"person starts laughing,\" check whether the pseudo-boundary lands on the true transition frame or on ordinary body motion.","tokens_in":14161,"feed_emoji":"🎬","tokens_out":8463,"duration_ms":86022,"temperature":0.7,"pith_summary":"Language-driven action localization faces a problem this paper attacks directly: the text query names the action but rarely says how the action starts or ends, so different annotators mark different frames as boundaries and the training signal is inconsistent. The paper proposes a two-part, model-agnostic recipe: prompt a large language model to write separate sentences describing the start and end of the action, then use those expanded queries to convert rigid boundary annotations into soft probability scores during training. The start/end sentences are fused into the video representation through local and global attention branches, and the probability scores are built from each frame's visual similarity to the boundary text minus its temporal distance to the annotated boundary. Integrated into five existing base models across QVHighlights, Charades-STA, and TACoS, the additions are reported to improve every evaluated metric, giving positive deltas in all 45 comparisons in the main table. The paper's claim is that this turns a coarse query and noisy boundaries into more consistent, more accurate boundary predictions without changing the base model.","feed_headline":"LLM boundary cues lift every base model on all three datasets","feed_subtitle":"Expanded start/end text plus probability supervision improve boundary prediction across five video-grounding models","key_machinery":"The machinery is a pair of plug-in modules around any existing video-query transformer. The first, LLM-powered query expansion, prompts LLaMa3-8B with a template such as \"describe the beginning and ending process in one sentence of the following action\", plus a constraint against inventing objects, to produce a start query $Q_s$ and an end query $Q_e$. The second, query-guided temporal modeling, lets the video features attend to $Q_s$, $Q$, and $Q_e$ in a local branch and to their concatenation in a global branch, then fuses the results into an enhanced video feature. The third, boundary probability modeling, computes a per-frame score $S^p_s(i) = \\mathrm{sim}(F_{v,i}, F_s) - \\mathrm{dis}(i, \\tau_s)$ (and analogously for the end), takes the argmax as a pseudo-boundary, keeps only frames scoring above threshold 0.8, min-max normalizes the survivors into start/end probability curves, and sets the interior frames to probability 1. The base model is then trained with its original loss plus a cross-entropy boundary loss between its predicted frame probabilities and these soft targets.","core_discovery":"The central claim is that boundary uncertainty in language-driven action localization can be reduced without modifying the base model, by supplying the model with textual descriptions of the action start and end and by supervising it with probability scores instead of hard boundary timestamps. The paper argues that the original query under-specifies boundaries: for the same query \"person eats sandwich\", one annotator may mark \"place hand on food plate\" as the start while another marks \"put food into the mouth\". An LLM prompted with the action name generates canonical start/end sentences, and the boundary probability module scores every frame as $\\mathrm{sim}(F_{v,i}, F_s) - \\mathrm{dis}(i, \\tau_s)$ for the start, and analogously for the end, takes the argmax as a pseudo-boundary, thresholds and min-max normalizes the scores into probabilities, and trains the base model with an added cross-entropy boundary loss. The authors report that all five base models improve on all metrics on all three datasets, and that the gains hold under perturbed boundary annotations. The method is presented as an off-the-shelf plug-in: only the text inputs and the training supervision change, not the architecture.","pith_inferences":["A natural next experiment is to compare the pseudo-boundaries against multiple independent human boundary annotations; this would test whether the LLM's canonical start/end text is closer to the consensus boundary than a single annotator's timestamp is.","The same soft-supervision trick could transfer to other temporally subjective tasks such as highlight detection, phase segmentation in surgery or cooking, and weakly supervised action segmentation, wherever boundary definitions vary across annotators.","Because the visual features are 2-second clip-level CLIP features, the method may reach a ceiling on micro-actions with sub-second transitions; dense or motion-focused features would be a direct stress test.","The ablation that swaps start and end queries suggests the method may be somewhat insensitive to the exact content of the expanded text; a sharper test would be to compare LLM-generated boundary text against generic action descriptions to isolate what information actually drives the gain."],"forward_implications":["Any existing moment-retrieval or temporal-grounding model can adopt the recipe by changing only the text query and the training loss; no architectural re-design is needed.","Models trained with LLM-expanded boundary queries become more robust to annotation noise, so datasets with subjective boundary definitions become less hazardous to train on.","Predictions become more consistent across videos with similar queries, because the LLM supplies a canonical description of the start/end motion (e.g., \"reaching out to touch the door\") that anchors the boundary.","The choice of LLM has little effect on final performance, and even swapped start/end queries cause only a slight drop, suggesting the main benefit comes from the presence of extra boundary text and soft supervision rather than from any single generator.","The gains appear across all three datasets, including TACoS with dense multi-query annotations, supporting the claim that the modules are generally useful rather than dataset-specific."],"supporting_citations":[{"why":"Defines the QVHighlights dataset and the DETR-based moment-retrieval architecture that serves as the main base model.","marker":"Lei et al. (2021)"},{"why":"Provides QD-DETR, the base model used for the ablation study and the noise-robustness experiments.","marker":"Moon et al. (2023b)"},{"why":"Supplies LLaMa3-8B, the LLM that generates the expanded start and end boundary queries.","marker":"Dubey et al. (2024)"},{"why":"Provides the CLIP text and visual encoders whose features feed both the query expansion and the similarity-based boundary scores.","marker":"Radford et al. (2021)"},{"why":"Supplies SlowFast clip-level video features that are concatenated with CLIP features for the visual representation.","marker":"Feichtenhofer et al. (2019)"},{"why":"Contributes the Charades-STA dataset used for the main comparisons and ablations.","marker":"Gao et al. (2017)"},{"why":"Contributes the TACoS dataset, the third benchmark in the cross-dataset evaluation.","marker":"Regneri et al. (2013)"},{"why":"Documents the boundary annotation uncertainty that motivates the paper's soft-supervision design.","marker":"Otani et al. (2020)"},{"why":"Provides the Gaussian-smoothing soft-label baseline that the boundary probability modeling module is compared against.","marker":"Pan et al. (2022)"},{"why":"Prior work on elastic moment bounding for boundary uncertainty that the paper contrasts with its LLM-based approach.","marker":"Huang et al. (2022)"}],"fun_headline_variants":["LLM-crafted boundary cues improve all tested action localization models","LLM-generated start/end text lifts every video grounding baseline","Soft boundaries from LLM-expanded queries improve all five models","LLM wrote boundary cues: every model wins on every dataset","No architecture change: LLM boundary cues boost all five models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the frame whose visual similarity to the LLM's start/end description, discounted by temporal distance to the human annotation, is a more reliable boundary than the human annotation itself.","fun_headline_variants_meta":{"raw":{"variants":["LLM-crafted boundary cues improve all tested action localization models","LLM-generated start/end text lifts every video grounding baseline","Soft boundaries from LLM-expanded queries improve all five models","LLM wrote boundary cues: every model wins on every dataset","No architecture change: LLM boundary cues boost all five models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000934,"raw_usage":{"total_tokens":4016,"prompt_tokens":981,"completion_tokens":3035,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":2949}},"tokens_in":597,"tokens_out":3035,"duration_ms":26263,"temperature":1.0,"reasoning_tokens":2949,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:28:08.448946+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the videos where pseudo-boundaries differ from the original annotations and ask independent annotators to mark the action start/end again. If the pseudo-boundary is not closer to the median re-annotation than the original boundary is, the core premise fails. A cheaper probe: on actions with visually subtle starts, such as \"person starts laughing,\" check whether the pseudo-boundary lands on the true transition frame or on ordinary body motion.","supporting_citations":[{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"Defines the QVHighlights dataset and the DETR-based moment-retrieval architecture that serves as the main base model."},{"cited_title":"ArXiv preprint","cited_arxiv_id":null,"evidence_quote":"Supplies LLaMa3-8B, the LLM that generates the expanded start and end boundary queries."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"Provides the CLIP text and visual encoders whose features feed both the query expansion and the similarity-based boundary scores."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"Supplies SlowFast clip-level video features that are concatenated with CLIP features for the visual representation."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"Contributes the Charades-STA dataset used for the main comparisons and ablations."},{"cited_title":"TACL pp 25--36","cited_arxiv_id":null,"evidence_quote":"Contributes the TACoS dataset, the third benchmark in the cross-dataset evaluation."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"Documents the boundary annotation uncertainty that motivates the paper's soft-supervision design."},{"cited_title":"IEEE Transactions on Neural Networks and Learning Systems","cited_arxiv_id":null,"evidence_quote":"Provides the Gaussian-smoothing soft-label baseline that the boundary probability modeling module is compared against."}],"review_version":1}