{"id":"e2a44bd0-dc39-49b3-b047-17c54ad8b22f","arxiv_id":"2607.20389","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Explicitly generating object trajectories and event timestamps before the final caption improves detailed video captioning on multiple benchmarks, though the improvement is largely driven by an external perception oracle.","lead":"PercepCap makes a video captioning model first write out a structured 'perception trace' (object locations over time and event timestamps) before it writes the final caption, and trains the model in two stages with this evidence. The paper reports consistent caption-quality gains over a strong baseline on five benchmarks, but most of the gain appears to come from an external model's perception rather than from the proposed training itself.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PercepCap is never evaluated at inference without its perception trace; without that cell, the stated claim that the trace itself improves captioning is not actually tested.","rationale":"The reader's weakest assumption focuses on the accuracy/independence of the caption-anchored perception ground truth. That is a real concern, and the paper's Section 5.5 acknowledges it. But the most load-bearing gap for the stated central claim is causal attribution: the paper never isolates the effect of the perception trace at inference. The claimed 'perceive-describe generation chain' implies that generating a trace before the caption is what improves output. Yet the ablation design always changes training and inference together, so we cannot tell whether the trace is doing causal work or is an epiphenomenon of training on a serialized target. A simple inference-time toggling experiment would settle this directly. This concern is complementary to the reader's: even if the perception ground truth were perfect, the central claim would remain untested without this control. The reader's verdict of CONDITIONAL is appropriate; no verdict change is needed, but this concrete missing experiment should be added before the claim can be accepted.","tokens_in":15589,"tokens_out":5793,"duration_ms":52799,"concrete_test":"Run the PD-SFT checkpoint (#5) and the final PercepCap model on DREAM-1K using the caption-only inference prompt (Figure 10), suppressing the perceive-describe chain. Compare AutoDQ F1 against (a) the same checkpoints with the perception-aware prompt (Figure 11), (b) #3 caption-only SFT, and (c) #2 baseline + external Gemini trace. Use at least 3 decoding passes and report mean±std. If the caption-only-inference F1 is within noise of the trace-inference F1, the trace is not load-bearing at inference; if it drops by >1.5 F1, the trace matters. Also log caption word counts to check verbosity as a confound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that structured perception traces improve captioning beyond direct caption generation. But the experiments compare models with different training histories and different inference prompts, never holding the model fixed while toggling the trace. In Table 3, #5 (PD-SFT, trace inference) vs #3 (caption-only SFT, no trace) differ in both training objective and inference prompt, so the +0.9 F1 cannot be attributed to the trace. #2 shows a zero-shot external Gemini trace yields 33.5, close to the final 33.9; this is consistent with the trace being a helpful scaffold, but it does not show the trained model's self-generated trace is load-bearing at inference. The necessary cell—running the PD-SFT or full PercepCap checkpoint with the caption-only prompt (Figure 10)—is absent. If the model still achieves ~33 F1 without generating a trace, the gain comes from training on the serialized target, not from the inference-time perception chain, undermining the claimed mechanism. If F1 drops sharply, the trace is doing real work. This missing control is the most direct way to falsify or support the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents PercepCap, a video captioning framework that inserts a structured spatio-temporal perception trace (object trajectories with bounding boxes and temporally grounded events) before final caption generation. Training uses two stages: Perceive-then-Describe SFT (PD-SFT) and Perception-Grounded RL (PG-RL) with a composite reward over object localization, event timestamps, event semantics, format, and caption coverage. To obtain paired trace/caption supervision, the authors propose Caption-Anchored Perception Data Construction, in which Gemini 3 Flash generates a caption-only anchor, the objects/events in that caption are extracted, and Gemini 3.1 Pro grounds them back into the video with boxes and timestamps. Experiments on DREAM-1K, CaReBench, ShortVidBench, MotionBench, and VidCapBench-AE show consistent improvements over the Qwen3-VL-8B caption-only baseline; ablations isolate several training choices.","tokens_in":15847,"tokens_out":4263,"duration_ms":41993,"significance":"If the mechanism holds, PercepCap is a useful step toward auditable, evidence-grounded video captioning and provides a practical recipe for converting caption-only MLLMs into perceive-describe chains. The paper's strengths are its consistent multi-benchmark gains, architecture-agnostic training recipe, public backbone, and an unusually candid limitations section. The main conceptual claim — that the explicit perception trace itself, not merely additional training on a serialized target or an external model's opinion, is what drives improvement — is currently not isolated by the experiments; the missing control is inexpensive to run and would substantially raise confidence.","major_comments":[{"comment":"The central claim that the perception trace improves captioning is not tested at inference. #5 vs #3 and #6 vs #4 differ in both the training objective and the inference prompt, so the gains cannot be attributed to the trace itself. #2 conditions on an external Gemini trace, not on the trained model's self-generated trace. The necessary control — running the PD-SFT or full PercepCap checkpoint with the caption-only prompt (Figure 10), or otherwise suppressing trace generation at inference — is absent. Without that cell, the +0.9 F1 from #5 to #7 and the gap vs #3 could come from training on the serialized caption target rather than from the inference-time perceive-describe chain. This is directly testable and should be added.","section":"§5.4, Table 3; §G"},{"comment":"The perception supervision is circular with respect to the caption: Caption-Anchored construction first generates the anchor caption, extracts its objects/events, and then grounds them with the same external model family (Gemini). The paper acknowledges error propagation in §5.5, but the claimed mechanism — perception evidence improves captioning — is not validated by independent perception ground truth. Table 4's perception probe prompts both variants to produce a trace under the same structured prompt, so it measures post-hoc perception ability, not the contribution of the trace inside PercepCap's actual inference chain. I recommend adding a human-verified or independently annotated perception-evaluation subset, and/or a manipulation experiment in which trace quality is perturbed and caption quality is measured.","section":"§4, §5.5, Table 4"},{"comment":"Tables 1 and 2 report single-point comparisons without variance or significance testing. Given that several margins over the strongest open-source baselines are small (e.g., ShortVidBench 40.4 vs 40.3 for Gemini 3 Flash and 39.7 for GLM; VidCapBench-AE Overall 19.0 vs 17.8), reporting standard errors over multiple evaluation runs or per-benchmark bootstrap intervals would clarify whether the improvements are robust rather than noise-level. This is not a blocker but is load-bearing for the claim of 'leading caption quality.'","section":"§5.3 and Table 1"}],"minor_comments":[{"comment":"Typos and formatting issues: 'Univerisity' in the affiliation, 'serveral' in §1, 'model, outputs' in §4 Step 2, 'eρ i' rendering in Eq. (3), and inconsistent F1/R/P ordering in Table 1 headers (the text defines AutoDQ as F1/Precision/Recall, but the headers list F1/R/P).","section":"Throughout"},{"comment":"The figure captions appear swapped: Figure 7 shows the object-extraction prompt but is captioned 'Prompt Template for Event Extraction of Training Data', while Figure 8 shows the event-extraction prompt but is captioned for object extraction. Please correct.","section":"Figures 7 and 8"},{"comment":"Abbreviations such as 'R_cap only RL' are not defined in the table caption; define them in the text. Also specify whether #2 uses the same 32-frame sampling as the trained variants, and whether the external Gemini trace is produced with the same perception prompt as the model's own trace.","section":"Table 3"},{"comment":"The reward coefficients in Eq. (5) are fixed to α=β=δ=0.5 and γ=λ=1.0, but no sensitivity analysis is reported. Even a small grid would strengthen the claim that the composite reward is robust and that the perception rewards matter beyond the caption reward.","section":"§5.1 and §C"}],"recommendation":"major_revision","confidential_remarks":"The missing inference-prompt control is the main risk: without running the PercepCap checkpoint with the caption-only prompt, the paper cannot support its central mechanistic claim. I do not see this as fatal; the ablations are otherwise careful and the limitation is acknowledged. I would ask for the control and the independent perception-validation evidence before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: PercepCap is a real attempt to make video captioning evidence-grounded, and it shows consistent gains across five benchmarks over a strong Qwen3-VL baseline. But the central claim—that the perception trace itself is load-bearing at inference—is not actually tested. The ablations never run the trained PercepCap checkpoint with a caption-only prompt, so the +F1 could come from training on the serialized target rather than from the trace generated at inference.\n\nWhat is genuinely good: the caption-anchored data construction is a sensible way to get caption-aligned perception annotations from a strong external model, and it is honestly described. The structured trace (object trajectories plus temporally grounded events) is more inspectable than free-form reasoning chains, and the two-stage training with decomposed rewards is a reasonable design. The ablation includes matched caption-only variants (#3 vs. #5, #4 vs. #6), which helps isolate the training paradigm. The zero-shot external-trace condition (#2) is a nice probe: it jumps to 33.5 F1, close to the final 33.9. The limitations section is candid about error propagation and inference cost. On its face, the empirical work is solid: the gains are consistent and the benchmarks are external with human ground truth.\n\nNow the soft spots, in proportion. The missing control is the critical one. The stress-test note is right: #5 vs. #3 differs in both training objective and inference prompt, so you cannot attribute the +0.9 F1 to the trace. The necessary cell is simple—run the PD-SFT or full PercepCap checkpoint with the caption-only prompt. If F1 stays near 33, the gain is mostly from training on the serialized chain, not from the trace at inference. If it drops sharply, the trace is doing real work. That single experiment would validate or falsify the mechanism. Second, the perception trace is derived from the caption itself (extract objects/events, then ground them), so it is not independent evidence for the caption; the paper acknowledges this in Section 5.5 but does not quantify how trace noise propagates into the rewards. Third, reproducibility is limited: no code, data, or checkpoints, and the construction relies on proprietary Gemini APIs. These are addressable concerns, not fatal flaws.\n\nWho this is for: video captioning and video MLLM researchers, especially those working on process supervision and structured intermediate representations. It deserves a serious referee; the missing control should be requested as a prerequisite for acceptance. I would cite it once the control is run, and I would bring it to a reading group to discuss the right way to attribute gains to inference-time scaffolding.\n\nRecommendation: engage with it, ask for the missing experiment, and treat this as a conditional accept rather than a reject.","headline":"A coherent, well-audited pipeline for perception-aware video captioning with consistent gains, but the central mechanism claim rests on a missing inference-time control.","tokens_in":16353,"tokens_out":2221,"would_cite":false,"duration_ms":21250,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that making a video captioner first emit a structured spatiotemporal perception trace—object trajectories with bounding boxes and events with time spans—and only then write the caption produces more detailed, better-grounde","keywords":["video captioning","spatio-temporal perception","multimodal large language models","perception trace","reinforcement learning","object tracking","temporal grounding","detailed video description"],"falsifier":"At inference, randomize the order of object ids and events in the perception trace before the caption is generated; if final caption quality does not drop relative to the intact trace, the model is not actually using the trace. A complementary check: replace the automatic trace with human-verified boxes and timestamps and see whether caption scores improve—if they do not, the paper's training signal, not its architecture, is the weak point.","tokens_in":15488,"feed_emoji":"🎬","tokens_out":8511,"duration_ms":75172,"temperature":0.7,"pith_summary":"PercepCap tries to establish that detailed video captioning improves when the model is forced to expose its perceptual evidence before writing the caption. Instead of mapping the video directly to text, it generates a structured spatiotemporal perception trace—objects identified and tracked across frames with bounding boxes, and events localized to start/end times—and only then produces the final description conditioned on that trace. The paper trains this two-stage behavior with a supervised fine-tuning step followed by reinforcement learning whose rewards separately score object localization, event timing, event semantics, and caption coverage. Across several captioning and caption-to-QA benchmarks, it reports consistent gains over the caption-only base model, and its ablations attribute the gain to the explicit perception interface rather than to extra caption supervision.","feed_headline":"Perception-first captions improve on caption-only models","feed_subtitle":"Splitting video captioning into a perception trace followed by the caption improves quality and makes evidence auditable.","key_machinery":"The load-bearing object is the structured spatiotemporal perception trace, a machine-readable JSON block the model must emit before the caption. It has two parts: objects, each with a temporally consistent id, a semantic label, and a localization list of timestamped 2D bounding boxes; and events, each with start/end timestamps and a natural-language description whose subjects are meant to match the tracked objects. The trace is the interface that makes the paper's claims work: it converts a single undifferentiated captioning task into two supervised stages, gives RL a place to attach separate perception- and caption-level rewards, and provides a human-auditable record of what the model perce","core_discovery":"The central claim is that the perceive–describe generation chain itself is what improves captioning, not just better caption data or more RL. By making object trajectories and temporally grounded events a required intermediate output, PercepCap turns caption generation into evidence-grounded generation: the final caption is a verbalization of a trace the model has already committed to. The training data for this chain is built by caption-anchored construction—first writing the caption, extracting the objects and events it mentions, then grounding those mentions back in the video with boxes and timestamps—so the trace and caption are guaranteed to refer to the same content. PercepCap then opt","pith_inferences":["A decisive test the paper does not run: swap in independently human-verified object tracks and event boundaries at inference. If captions do not improve beyond the caption-only baseline, the benefit would be more about conditioning on extra text than about genuine perception.","The same caption-anchored data construction could be applied to domain-specific captions (sports, egocentric video, surveillance) to bootstrap perception traces for new settings without manual annotation—though the paper does not demonstrate this.","If the trace is essentially a structured paraphrase of the caption, then the method's improvement may partly stem from inference-time chain-of-thought decoding rather than new perceptual capability; varying trace granularity would separate these effects.","The machine-readable trace (boxes plus timestamps) suggests a natural extension to retrieval or grounded QA, where the evidence can be consumed directly rather than re-extracted from text."],"forward_implications":["Caption-only datasets can be repurposed as perception-supervision data: write the caption, extract its objects and events, ground them with boxes and timestamps, and use the result for supervised and reinforcement training.","Caption errors become diagnosable: a failure can be traced to whether the model missed an object, placed an event at the wrong time, or verbalized correct evidence poorly.","Reinforcement learning gets finer credit assignment: separate rewards for spatial, temporal, semantic, and caption components can push up each dimension instead of optimizing only final text.","Captions produced this way retain more of the information needed for downstream tasks, since caption-to-QA accuracy also improves.","Because the trace is explicit, the pipeline is auditable: intermediate perception can be inspected or corrected before the caption is trusted."],"fun_headline_variants":["Perceive-then-describe chain improves video captions","Explicit spatio-temporal trace before caption","PercepCap: perception trace then caption","Captioning with an intermediate perception trace","Grounding captions in object trajectories and events"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The perception ground truth used for training comes from an automatic model that first writes the caption and then re-grounds that caption's own mentions in the video, with no human verification; if those automatically produced boxes and timestamps are wrong or merely paraphrase the caption, the perceived gain could come from conditioning on an external model's opinion rather than from better perception.","fun_headline_variants_meta":{"raw":{"variants":["Perceive-then-describe chain improves video captions","Explicit spatio-temporal trace before caption","PercepCap: perception trace then caption","Captioning with an intermediate perception trace","Grounding captions in object trajectories and events"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1404,"prompt_tokens":816,"completion_tokens":588,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":519}},"tokens_in":560,"tokens_out":588,"duration_ms":5733,"temperature":1.0,"reasoning_tokens":519,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T09:56:55.059403+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"At inference, randomize the order of object ids and events in the perception trace before the caption is generated; if final caption quality does not drop relative to the intact trace, the model is not actually using the trace. A complementary check: replace the automatic trace with human-verified boxes and timestamps and see whether caption scores improve—if they do not, the paper's training signal, not its architecture, is the weak point.","supporting_citations":[],"review_version":1}