{"id":"0eeb96d5-c808-4e9e-b52f-423b4e439fea","arxiv_id":"2412.10720","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Injecting self-attention and a transformer encoder into LVLMs is reported to improve video captioning metrics on MSVD and MSR-VTT, but the losses, dataset, and code are not described.","lead":"This paper proposes adding a Causal-Temporal Reasoning Module, built from self-attention and a transformer encoder, to video captioning models, and reports improved scores on MSVD and MSR-VTT. The work is relevant to anyone tracking how close vision-language models are to understanding causal and temporal structure in video.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 2's 'causal attention' is unmasked full self-attention, and the auxiliary losses/dataset that could make it causal are undefined; the reported gains cannot be traced to a causal-temporal mechanism.","rationale":"The paper's central claim is that CTRM explicitly encodes causal dependencies and temporal consistency, and that this explains the Table I improvements. The most load-bearing assumption is that Eq. 2 is a causal attention mechanism and that Eqs. 7 and 9 define a training procedure with causal-temporal supervision. Neither holds as written. Eq. 2 is unmasked full self-attention: for any query at time i, the softmax normalizes over all T keys, including future frames, so the 'causal attention matrix' is not causal. Eq. 4 is a standard transformer encoder. Thus the proposed module is mathematically equivalent to a generic self-attention/transformer block unless the undefined Lcausal and Ltemporal alter the training objective. But those losses are not given anywhere in the paper, the curated causal-temporal dataset is not described, and no code or data are released. The reported CIDEr improvements are therefore uninterpretable: they could come from added capacity, from an unspecified training set, or from different evaluation splits. This is more than a reproducibility complaint; it is a mismatch between the stated mechanism and the equations. The reader's REJECT verdict is justified. I do not think the argument survives as ACCEPT or CONDITIONAL because the central equation already fails to realize the claimed causal structure, and the unspecified losses cannot repair Eq. 2's lack of a causal mask. A comparison against NarrativeBridge would also be useful, but the internal Eq. 2 issue is more fundamental. Verdict should remain unchanged: REJECT with high correctness risk.","tokens_in":8000,"tokens_out":6064,"duration_ms":56359,"concrete_test":"Analytical check: instantiate Eq. 2 with any two distinct frame embeddings and inspect Ac[i,j] for j>i; because the softmax denominator includes future keys, these entries are strictly positive, confirming that no causal mask is enforced. Experimental check: independently implement Section III-B without the Eq. 7 auxiliary terms (set lambda1=lambda2=0), using the same backbone and data split, and compare MSVD CIDEr with Table I's 18.20. If the score does not change materially, the 'causal dynamics encoder' is not the source of the reported gains. If the authors claim the undefined losses are essential, the exact formulas for Lcausal and Ltemporal must be supplied and the experiment rerun with them; until then, Eq. 7 cannot be evaluated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CTRM's Causal Dynamics Encoder and Temporal Relational Learner explicitly model causal and temporal structure, producing the Table I improvements. The load-bearing mechanism is Eq. 2, which is named 'causal attention.' But Eq. 2 is softmax(QcKc^T/sqrt(dk)) with no causal mask; the softmax is over all T key positions, so future frames receive positive attention weight, and Hc at any time step depends on future frames. The matrix is not causal. Eq. 4 is an ordinary transformer encoder over position-encoded features, so CTRM as defined is a generic self-attention plus transformer block. The only place causal/temporal supervision could enter is Lcausal and Ltemporal in Eq. 7, but these loss terms are never defined, the curated causal-temporal dataset in Section III-D2 is not described or released, and no code is provided. The Table I gains (18.20 vs. 17.88 CIDEr on MSVD; 17.90 vs. 17.44 on MSR-VTT) therefore cannot be attributed to any causal-temporal mechanism; they are equally consistent with added transformer capacity and undisclosed auxiliary training. This is an internal mismatch between the stated mechanism and the equations, not merely a missing artifact. Eq. 9 also sums Lpre, Lfine, and Lcontrast into one objective, contradicting the claimed three-stage strategy. The closest prior work, NarrativeBridge [4], is cited but omitted from the baseline table, leaving the state-of-the-art comparison incomplete.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Causal-Temporal Reasoning Module (CTRM) for video captioning, consisting of a Causal Dynamics Encoder (CDE) and a Temporal Relational Learner (TRL), integrated into a large vision-language model. The training strategy is described as three-stage: pre-training on video-text data, fine-tuning on a curated dataset with causal-temporal annotations, and contrastive alignment. Experiments on MSVD and MSR-VTT report CIDEr, BLEU-4, and ROUGE-L scores, supplemented by ablations, human evaluations, and analyses of causality and temporal consistency.","tokens_in":8370,"tokens_out":3431,"duration_ms":31098,"significance":"If the proposed mechanism were correctly specified and the results reproducible, the paper would offer a modest contribution by showing that an explicit causal-temporal module plus auxiliary losses improves narrative coherence in video captioning. However, the manuscript does not define the auxiliary losses, does not describe the curated dataset, reports single-run numbers without variance or significance tests, and the central equation named \"causal attention\" is not causally masked. As a result, the paper's significance cannot be assessed from the information provided, and the current claims are not supported by the evidence.","major_comments":[{"comment":"The \"causal attention\" in Eq. (2) is an unmasked softmax over all T key positions, so H_c at any time step aggregates information from future frames and the matrix is not causal. The name \"causal\" is therefore not justified by the equations; if causality is introduced only through L_causal and L_temporal, those terms must be defined (see next comment).","section":"§III-B1, Eq. (2)-(3)"},{"comment":"L_causal and L_temporal are never defined; the weights λ1 and λ2 and the temperature τ in Eq. (8) are also unspecified. Without definitions of these losses and of the annotations used to compute them, the fine-tuning stage has no describable content, and the improvements in Table I cannot be attributed to causal-temporal supervision.","section":"§III-D2, Eq. (7)"},{"comment":"The final loss L = L_pre + L_fine + L_contrast is presented as a single objective, which contradicts the claimed three-stage training strategy (pre-training, fine-tuning, contrastive alignment). It is unclear whether these losses are optimized jointly or sequentially, and the reappearance of L_pre in the final loss suggests that pre-training and fine-tuning are not actually separated as stages.","section":"§III-D, Eq. (9)"},{"comment":"The quantitative results are reported as single numbers with no standard deviations, no number of runs, no significance tests, and no dataset split details. Moreover, the CIDEr values (18.20 on MSVD, 17.90 on MSR-VTT) are far below typical published values on these benchmarks, and the paper does not comment on this discrepancy. The comparison is also incomplete: NarrativeBridge [4], the closest prior work cited in the introduction, is omitted from the baseline table.","section":"Table I and §IV-A"},{"comment":"The curated dataset enriched with causal and temporal annotations is not described: its size, source, annotation protocol, and train/test split are all absent. Similarly, the manual annotation used for the causality and temporal consistency analyses in Tables IV and V is not described (number of annotators, inter-annotator agreement, sample size). The human evaluation in Table III also lacks these details, so the reliability of these analyses cannot be assessed.","section":"§III-D2 and §IV-E"},{"comment":"The large vision-language model backbone is never specified. The text says \"such as Flamingo or BLIP-2\" but does not state which model is actually used, how frame features are extracted, what resolution or frame rate is used, or the parameter count. This makes the experimental setup irreproducible even in principle.","section":"§III-A and §IV-A"}],"minor_comments":[{"comment":"There are numerous typographical and spacing errors, e.g., \"descripti ve\" in the abstract and \"fine-tuning\" with a broken ligature; these should be corrected in revision.","section":"Abstract and running text"},{"comment":"The text attributes a live video captioning framework to \"Blanco-Fern´ andez et al.\" but the cited reference [17] is by W. Choi and J. Yoon; the citation does not match the description.","section":"§II-B, reference [17]"},{"comment":"The \"Without CTRM\" row reproduces the same numbers as the \"Transformer\" baseline in Table I; this should be explicitly acknowledged or clarified.","section":"Table II"},{"comment":"The caption decoder conditions on H_t, but the paper does not describe how the decoder attends to H_t (e.g., cross-attention layer, number of layers, fusion method); a specification is needed.","section":"§III-C, Eq. (5)"},{"comment":"The metrics \"Captions with Causality (%)\" and \"Consistent Captions (%)\" are not defined; the numerator, denominator, and annotation criteria should be stated.","section":"§IV-E, Tables IV and V"}],"recommendation":"reject","confidential_remarks":"The manuscript is not ready for the review process. The central mechanism is not actually causal as defined, the auxiliary losses and dataset that could support the claim are absent, and the experimental reporting is insufficient for verification. The omission of the closest prior work (NarrativeBridge) from the comparison table is also a concern. A resubmission with a properly specified method, released code and data, and a rigorous experimental protocol (error bars, significance tests, described splits) would be needed before the claims can be evaluated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know about arXiv:2412.10720. First, the 'causal attention' in Eq. 2 is unmasked full self-attention; there is no causal mask, so future frames influence the encoding. Second, the auxiliary losses and curated dataset that could inject causal supervision are never defined or released. That means the reported CIDEr gains cannot be traced to any causal-temporal mechanism; they are equally explained by added transformer capacity.\n\nThe paper is not without merit. The problem is real and the paper is clearly written. The ablation study and human evaluation are a good instinct. But the numbers are a red flag: CIDEr around 18 on MSVD is far below typical published results, which are usually 80+. The authors don't report error bars or significance tests, and the baseline table omits NarrativeBridge [4], the closest prior work they themselves cite.\n\nThe soft spots are load-bearing. Eq. 7 defines Lfine as a sum including Lcausal and Ltemporal, but these terms are never defined. The curated dataset in Section III-D2 is described in one sentence and never released. Eq. 9 sums Lpre, Lfine, and Lcontrast into one objective, which contradicts the claimed three-stage sequential strategy. And the stress-test is right: Eq. 2's softmax is over all key positions, so the matrix is not causal. Calling it 'causal attention' is an internal mismatch between the equations and the stated mechanism.\n\nI wouldn't cite this. It doesn't deserve referee time as it stands. If the authors release code, define the auxiliary losses, use a genuinely causal mask, and benchmark against NarrativeBridge with realistic numbers, it might be worth a second look. As submitted, it's a no.","headline":"The paper's causal mechanism is a mislabeled self-attention layer; undefined losses and a missing baseline leave the reported gains unexplained.","tokens_in":8874,"tokens_out":2230,"would_cite":false,"duration_ms":19882,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Inserting a Causal-Temporal Reasoning Module into large vision-language models improves the causal and temporal coherence of generated video captions, raising CIDEr to 18.20 on MSVD and 17.90 on MSR-VTT.","keywords":["video captioning","large vision-language models","causal reasoning","temporal reasoning","Causal-Temporal Reasoning Module","self-attention","MSVD","MSR-VTT"],"falsifier":"Re-run the same three-stage training while deleting the two auxiliary training terms, or replacing the specially annotated dataset with an ordinary captioning dataset, and see whether the score advantages on MSVD and MSR-VTT remain; if they disappear, the causal-temporal mechanism is what carries the result, and if they stay, it is not.","tokens_in":7806,"feed_emoji":"🎬","tokens_out":9530,"duration_ms":76351,"temperature":0.7,"pith_summary":"The paper argues that large vision-language models generate captions that are descriptively accurate but often miss the causal chain and temporal order of events, and that a bolt-on reasoning module can fix that. The module, called the Causal-Temporal Reasoning Module (CTRM), uses a self-attention Causal Dynamics Encoder to expose cause-effect links between frames and a transformer-based Temporal Relational Learner to preserve order; captions are then generated from the resulting temporal embeddings. The training recipe has three stages: pre-training on general video-text pairs, fine-tuning on a curated dataset with causal-temporal annotations, and a contrastive loss that aligns video embeddings with text embeddings. The reported numbers on MSVD and MSR-VTT put the full model ahead of every baseline on CIDEr, BLEU-4, and ROUGE-L, and the human evaluation rates it higher on fluency, coherence, and relevance.","feed_headline":"New causal-temporal module beats baselines in video captioning","feed_subtitle":"CIDEr rises to 18.20 on MSVD and 17.90 on MSR-VTT, with better human-rated coherence.","key_machinery":"The load-bearing object is the Causal-Temporal Reasoning Module (CTRM), built from the Causal Dynamics Encoder (CDE) and the Temporal Relational Learner (TRL). The CDE forms a causal attention matrix $A_c = \\mathrm{softmax}(Q_cK_c^\\top / \\sqrt{d_k})$ over frame features and produces causal embeddings $H_c = A_cV_c$; the TRL then applies a transformer encoder with positional encodings, $H_t = \\mathrm{Transformer}(H_c + P)$, to keep the events in order. These temporal embeddings, rather than the original frame features, are what the autoregressive caption decoder conditions on, and the same embeddings are aligned to text with a contrastive loss. This two-stage bottleneck is the mechanism claimed to carry the causal and temporal information into the generated caption.","core_discovery":"The central result is that causal and temporal structure can be pulled out of raw frame features by a two-part module, and that conditioning the caption decoder on that structure, rather than on raw frame features, is what makes generated narratives coherent. The CDE computes a causal self-attention matrix over frames; the TRL passes the output through a transformer with positional encodings; and the decoder samples each word conditioned on the resulting temporal embeddings. The authors attribute the benchmark gains to this mechanism, and their ablation supports the attribution: removing either submodule drops the MSVD CIDEr score from 18.20 to 17.10 or 16.80, and removing the whole CTRM drops it to 15.74, the same as the plain Transformer baseline. Human ratings follow the same pattern, with the full model scoring 4.4 on fluency, 4.5 on coherence, and 4.6 on relevance out of 5.","pith_inferences":["A natural test the paper does not run is to replace the curated causal-temporal fine-tuning data with ordinary captioning data while keeping CTRM; if the gains persist, the module alone, not the annotations, is doing the work.","The same two-part design could transfer to other tasks where event ordering matters, such as story ending generation or temporal grounding of narrated instructions.","Because the auxiliary losses $L_\\text{causal}$ and $L_\\text{temporal}$ are only named, not specified, a reader cannot currently tell whether the contrastive alignment loss alone would reproduce the reported gains."],"forward_implications":["If the CTRM result is correct, video captioning can gain narrative coherence without replacing the underlying vision-language backbone, because the module is added on top of existing frame features.","The ablation on MSVD implies that both submodules pull their weight: removing the CDE costs 1.10 CIDEr, removing the TRL costs 1.40 CIDEr, and removing the whole module loses 2.46 CIDEr.","The human evaluation implies that the improvement is visible to people, not just to automatic metrics, since annotators rate the full model above every baseline on fluency, coherence, and relevance.","The timing table implies that the extra reasoning comes at a small cost, with training time only slightly above the plain Transformer and inference time still 0.20 seconds per sample."],"supporting_citations":[{"why":"This reference supplies the large-scale pretrained visual-language backbone for dense video captioning that the proposed method augments.","marker":"[1]"},{"why":"This reference defines the causal-temporal narrative baseline, reported as CEN, against which the paper's MSVD and MSR-VTT scores are compared.","marker":"[4]"},{"why":"This reference establishes the generalist LVLM architecture that the paper extends with its reasoning module.","marker":"[8]"},{"why":"This reference supports the long-context modeling the paper relies on for coherent video narratives.","marker":"[14]"},{"why":"This reference provides a graph-based spatio-temporal captioning approach whose results the paper uses as a baseline.","marker":"[20]"}],"fun_headline_variants":["Causal-temporal module boosts video captioning","Causal and temporal reasoning for better video captions","New module reads cause and time in video","Video captioning improved by causal-temporal module","Causal-temporal attention lifts video narrative quality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains depend on a curated dataset annotated with causal and temporal relationships and on two auxiliary training terms that the paper names but never defines, releases, or quantifies; if that dataset and those terms are not actually usable, the fine-tuning stage credited for the improvement has no describable content.","fun_headline_variants_meta":{"raw":{"variants":["Causal-temporal module boosts video captioning","Causal and temporal reasoning for better video captions","New module reads cause and time in video","Video captioning improved by causal-temporal module","Causal-temporal attention lifts video narrative quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000493,"raw_usage":{"total_tokens":2427,"prompt_tokens":955,"completion_tokens":1472,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":1404}},"tokens_in":571,"tokens_out":1472,"duration_ms":10466,"temperature":1.0,"reasoning_tokens":1404,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:39:44.717087+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same three-stage training while deleting the two auxiliary training terms, or replacing the specially annotated dataset with an ordinary captioning dataset, and see whether the score advantages on MSVD and MSR-VTT remain; if they disappear, the causal-temporal mechanism is what carries the result, and if they stay, it is not.","supporting_citations":[{"cited_title":"Available: https://doi.org/10.48550/ar Xiv.2405.07046","cited_arxiv_id":null,"evidence_quote":"This reference provides a graph-based spatio-temporal captioning approach whose results the paper uses as a baseline."}],"review_version":1}