{"id":"6ae26cdc-2851-4e11-8fbe-eeb0a8fda6c9","arxiv_id":"2412.09283","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"InstanceCap generates instance-level structured video captions by isolating objects with auxiliary models and prompting MLLMs with class-specific hints, improving text-to-video generation fidelity when used for finetuning.","lead":"This paper introduces InstanceCap, a method that breaks videos into individual objects and writes structured captions describing each object's appearance, motion, position, plus background and camera movement. The authors show that using these captions to finetune a text-to-video model improves fidelity of generated object details compared with prior caption styles.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's finetuning gain is confounded: InstanceCap gets an extra inference-time prompt enhancement/alignment that baselines do not, so the 28.63→37.88 gap may not come from InstanceVid finetuning.","rationale":"The reader's weakest-assumption analysis focused on the 3DV AE metric in Table 1, and that concern is valid: the validation in Table S1 only covers pixel-level distortions, not caption-semantic differences, and no error bars are reported. However, the paper has independent qualitative support for caption fidelity (Figures 1, 6) and a human evaluation (Figure 8), so the first contribution may survive even if 3DV AE is unreliable. The more load-bearing issue is the internal validity of Table 2's finetuning claim. InstanceEnhancer and the Aligning Prompt are explicitly part of the proposed inference pipeline, but no control is run to separate the effect of prompt enhancement from the effect of finetuning on InstanceVid. This is an omitted control rather than an internal inconsistency, so it does not warrant rejection; it warrants a conditional acceptance with a specific additional experiment. I therefore keep the reader's CONDITIONAL verdict but identify a different, sharper weakest point. The paper should be credited for its consistent qualitative trends and for the supplementary detail on the pipeline, but the central T2V-generation claim needs the matched-prompt control before it can be accepted at face value.","tokens_in":880,"tokens_out":1086,"duration_ms":119478,"concrete_test":"Run Table 2 on the Inseval set in two matched conditions. Condition A: feed the exact output of InstanceEnhancer plus the Figure S11 Aligning Prompt to both base Open-Sora v1.2 and Open-Sora+InstanceCap. Condition B: feed the raw Inseval prompt to both models. If base Open-Sora with the enhanced prompt reaches near 37.88%, or if Open-Sora+InstanceCap with raw prompts drops to near 28.63%, the Table 2 gain is due to the enhancement/alignment step rather than to InstanceVid finetuning. Report per-dimension scores with fixed seeds and error bars.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is the causal attribution in the T2V-generation experiment. Section 3.3 introduces InstanceEnhancer, and Figure S11 adds an LLM-based Aligning Prompt that converts InstanceCap's structured JSON into a ~180-word continuous paragraph before feeding it to Open-Sora. The paper does not state that the same enhancement/alignment is applied to the base Open-Sora row or to the Panda/ShareGPT4Video/LLaVA-finetuned rows in Table 2. Since Inseval prompts are short single sentences, the proposed model may receive far longer and more detailed conditioning than the comparators. Prompt length and specificity alone can improve instance-level success rates, so the claimed 9.25-point average gain over base Open-Sora cannot be attributed to finetuning on InstanceVid without a matched-prompt control. This directly threatens the paper's second central contribution, 'T2V models fine-tuned on our InstanceVid further achieve more precise generation on instance details and motion actions.' The reconstruction evidence (Table 1) and human caption evaluation (Figure 8) are less affected by this confound, so the caption-fidelity claim may still hold; the T2V finetuning claim, however, is not yet established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes InstanceCap, a structured captioning framework for text-to-video training data. It uses an auxiliary model cluster (object detection, SAM2 video segmentation, camera-motion detection) to isolate instances, then prompts MLLMs with a chain-of-thought procedure and human-designed class hints to produce a JSON caption containing a global description, per-instance appearance/actions/position, background detail, and camera movement. The authors curate a 22K-video InstanceVid dataset from OpenVid-1M and introduce InstanceEnhancer, a two-stage inference-time prompt enhancement and alignment pipeline. They evaluate by (i) reconstruction-via-recaption with CogVideoX-5b using a 3DV AE distance, CLIP SenbySen, and a user study, and (ii) finetuning Open-Sora on InstanceVid and measuring instance-level success on a new benchmark Inseval. The central claims are that InstanceCap captions are more faithful and less hallucinated than short, dense, or coarse-structured captions, and that finetuning on InstanceVid improves instance-level T2V generation.","tokens_in":22060,"tokens_out":7682,"duration_ms":76352,"significance":"The problem is well chosen: existing recaption paradigms trade off brevity, detail, and structure, and instance-level fidelity is increasingly important for controllable video generation. The instance-decomposition idea with blurred backgrounds and structured JSON output is a sensible and novel way to force MLLMs to attend to individual objects, and the detailed prompts, lexicon, and evaluation protocols are given in the supplement, which is a strength for reproducibility. If the claims hold, the 22K InstanceVid dataset and the Inseval benchmark would be useful resources. The current evidence is suggestive but not conclusive: the reconstruction metric is validated only for pixel-level distortions, and the finetuning result is potentially confounded by the inference-time enhancement pipeline.","major_comments":[{"comment":"The reported gain from finetuning on InstanceVid is confounded with the InstanceEnhancer and Aligning Prompt. The paper does not specify that the base Open-Sora row or the Panda/ShareGPT4Video/LLaVA rows receive the same two-stage expansion and 180-word alignment. Because Inseval prompts are short single sentences, the proposed system may receive much longer and more instance-detailed conditioning, which alone could improve all Inseval dimensions. Please add a matched-prompt control: run InstanceEnhancer/Aligning Prompt on the base model and on each finetuned baseline, or feed the raw Inseval prompt to all rows, and report both settings.","section":"§3.3, Table 2, Figure S11"},{"comment":"The 3DV AEscore is validated only against pixel-level distortions (blur, noise, compression, shifts), not against caption-content differences. The main Table 1 claim requires that this latent distance responds to caption fidelity. Please validate by recaptioning the same videos with captions that are systematically wrong in instance attributes (e.g., swapped colors or actions) and show that 3DV AE separates correct from incorrect captions; also report per-video variance and confidence intervals.","section":"Supplementary Section 5, Eq. (1), Table S1"},{"comment":"All reported numbers are point estimates without error bars, significance tests, or the number of independent seeds. The differences in Table 1 are small (3DV AE 134.25 vs. 137.50; CLIP SenbySen 0.2133 vs. 0.2156), and Table 2 reports percentages without stating the number of prompts per cell. Without uncertainty quantification, the claim of significant improvement is not supported. The Figure 8 user study also lacks participant count, video sample size, and inter-annotator agreement.","section":"§4.1, Tables 1 and 2"},{"comment":"Multiple Shape and Multiple Detail are omitted because all models performed poorly, so the reported Average is not an overall average and should be explicitly labeled as the mean over the reported subset. In addition, the Inseval evaluation is an LLM-based judge whose agreement with human preferences is not demonstrated; a small human validation of the QA judge would strengthen the central generation claim.","section":"Supplementary Section 6, Table 2"},{"comment":"The Open-Sora finetuning setup is not described: learning rate, batch size, number of steps, text-encoder updates, video resolution, and compute are all missing. Since the T2V-generation claim is a major contribution, this missing information prevents replication and makes it difficult to assess whether the gain is due to the InstanceVid data or to other training choices.","section":"§4.3, Table 2"}],"minor_comments":[{"comment":"The abstract's claim of significant improvement over previous models is stronger than the evidence: in Table 1, InstanceCap's CLIP SenbySen (0.2133) is below MiraData (0.2156), so the dominance is not across all metrics.","section":"Abstract, Table 1"},{"comment":"The Hallucination Scores are shown with higher bars, but the text does not state whether higher is better; please specify the direction and the scale.","section":"Figure 8"},{"comment":"The layer-specific weight matrix w_l is not defined; please clarify whether it is a learned weight matrix or a fixed weighting factor.","section":"Supplementary Section 5, Eq. (1)"},{"comment":"Please specify how the 100 reconstruction videos were selected and whether any of them overlap with the source set used to construct InstanceVid.","section":"§4.1"},{"comment":"Use consistent model names ('LLaVA-Video' vs. 'LLaVA') and describe how each '+captioner' row was constructed, including which captions were used for finetuning and which prompt text was used at inference.","section":"Table 2"},{"comment":"The claim that InstanceVid is curated from 'high-aesthetic, high-consistency' videos should cite or describe the filtering criteria, since the selection directly affects the generality of the downstream claims.","section":"§3.2"},{"comment":"The statement that full class hints 'will be provided in the code we plan to release later' should be replaced by including the full JSON in the supplement for immediate reproducibility.","section":"Supplementary Section 2"}],"recommendation":"major_revision","confidential_remarks":"The T2V finetuning claim is the weakest link: if the authors cannot provide a matched-prompt control, the claim should be substantially weakened. The evaluation set is drawn from OpenVid-1M, the same source as InstanceVid; this is not fatal, but the overlap should be disclosed in the main text. The paper appropriately cites the authors' own OpenVid-1M work given the dataset lineage, but the InstanceVid selection criteria should be made explicit. The GitHub repository is referenced, but code is not yet available; the release status should be clarified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this if you care about captioning for T2V. The central idea—isolate each instance with detection and SAM2, blur non-instance regions, and run MLLMs through a structured CoT with class-specific hints—is a sensible integration that does seem to produce more faithful, less hallucinated captions than short, dense, or coarse-structured baselines. The reconstruction and human-eval results (Table 1, Figure 8) are consistent with that claim, and the ablations on blur background vs. red screens and weak prompts are genuinely informative. Credit where due: the authors also state limitations honestly (detection dependence, instance-free scenes, small dataset).\n\nThe soft spots are real but not fatal. The biggest one is the T2V finetuning claim. In Table 2, InstanceCap uses InstanceEnhancer plus an LLM \"Aligning Prompt\" that expands short Inseval prompts into ~180-word structured paragraphs before they enter Open-Sora. The paper never says the same expansion is applied to the base Open-Sora row or the Panda/ShareGPT4Video/LLaVA-finetuned rows. If it isn't, then the 28.63→37.88 gap is contaminated: longer, more specific conditioning alone can lift instance-level scores. That confound needs a matched-prompt control before the finetuning claim is credible. The caption-fidelity claim in Table 1 is less affected, but the 3DV AE metric is validated only on pixel distortions, not caption differences, and the CLIP SenbySen score is actually slightly below MiraData. There are also no error bars or significance tests anywhere. Inseval is self-designed, and two Multiple-category rows are omitted because all models fail; that's defensible but should be pre-registered or at least reported with the failure rates. Mild self-citation (OpenVid-1M used both for InstanceVid and the reconstruction test set) is worth noting but not disqualifying—it's their dataset and they are transparent about it.\n\nWho is this for? Anyone working on video recaptioning or data curation for T2V. The reconstruction experiment is a useful recipe even if the generation experiment needs redoing. I'd send it to peer review because the core method is promising and the issues are addressable, but I would not cite the finetuning result until the confound is resolved and code/data are out.","headline":"InstanceCap has a plausible recipe and good reconstruction evidence, but its headline finetuning gain is confounded by an unmatched inference-time prompt enhancer.","tokens_in":22634,"tokens_out":2055,"would_cite":false,"duration_ms":20372,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"InstanceCap claims that decomposing video captions into per-instance structured descriptions improves text-to-video fidelity and reduces hallucinations, supported by a 22K dataset and an inference-time prompt enhancer.","keywords":["text-to-video generation","video captioning","instance-aware caption","structured caption","hallucination reduction","video-text dataset","prompt engineering","diffusion transformer"],"falsifier":"Run a perceptual study on the same 100 reconstruction videos where human raters, blind to caption method, pick which regenerated video is closer to the ground truth; if MiraData or dense captions are chosen at least as often as InstanceCap, the 3DV AE advantage would not translate into perceived fidelity. A second check: re-run the Table 2 fine-tuning with InstanceVid captions stripped of their structured JSON into plain paragraphs, and see whether the Inseval gains disappear, which would isolate the structure as the cause.","tokens_in":21592,"feed_emoji":"🎬","tokens_out":2629,"duration_ms":27490,"temperature":0.7,"pith_summary":"The paper tries to establish that video captions organized around individual instances, with separate fields for each object's class, appearance, actions, motion, and position, plus background and camera movement, let text-to-video models generate frames closer to the original video and follow fine-grained prompts more accurately. It argues that short captions miss details, dense captions hallucinate, and coarse structured captions stay too vague, so none keep the caption faithful to the video. The authors build a pipeline that isolates instances with an auxiliary model cluster, blurs out non-instance regions, and guides a multimodal LLM with per-class hints and a positive/negative lexicon to produce concise structured phrases. They also curate InstanceVid, 22K videos with these captions, fine-tune Open-Sora on it, and add InstanceEnhancer to turn short user prompts into the same structure at inference. If the claim holds, instance-level structured captions become a practical recipe for higher-fidelity text-to-video training and inference.","feed_headline":"Instance-aware captions sharpen text-to-video fidelity","feed_subtitle":"Decomposing captions per instance cuts hallucinations and lifts Open-Sora's instance-level accuracy by nine points.","key_machinery":"The central machinery is the auxiliary model cluster (AMC) combined with a structured caption schema. AMC uses object detection, video instance segmentation with SAM2, and camera motion detection to transform a global video into local instance videos with blurred backgrounds, so the MLLM annotates each instance in isolation while still receiving the global description as context. The schema fixes a JSON format with 'Global Description' and 'Structured Description' subfields for 'Main Objects' (each with Class, Appearance, Actions and Motion, Position), 'Background Detail', and 'Camera Movement'; the improved Chain-of-Thought prompts, class-specific hints for roughly eighty categories, and a positive/negative lexicon steer the MLLM toward concise, accurate phrases.","core_discovery":"The central claim is that instance-aware structured captions, which decompose a video into a global description plus a structured description of main objects, background, and camera movement, produce higher fidelity to the source video and fewer hallucinations than short captions, dense captions, or coarse structured captions. Quantitatively, the paper reports the best 3DV AE score in reconstruction-via-recaption (134.25 versus 137.50 for MiraData) and higher human-rated instance detail and hallucination scores, while fine-tuning Open-Sora on InstanceVid raises average Inseval accuracy from 28.63% to 37.88%. The authors argue this happens because isolating each instance with blurred backgrounds lets the multimodal LLM describe one object at a time, and because the human-designed class hints and lexical constraints keep those descriptions precise rather than inventive.","pith_inferences":["The blur-background trick, which the paper shows beats red-screen or weak-visual-prompt alternatives for multi-instance scenes, could likely transfer to other MLLM annotation tasks where attribute blending across objects is a known failure mode.","If the structured caption format is what drives the gains, one testable extension is to apply InstanceCap-style captions to longer videos or interactive multi-camera footage, where the separation of camera movement from instance action should matter even more.","The paper's own stated limitations, that precision depends on object detection and benefits fade in instance-free scenes, suggest the method is strongest for object-centric videos, so its advantage over dense captions may shrink on purely scenic or atmospheric footage.","The Inseval benchmark, with its QA-style CoT evaluation, could become a reusable instance-level probe for other T2V models, independent of the captioning method used to train them."],"forward_implications":["Fine-tuning a DiT-based T2V model on instance-aware structured captions improves instance-level instruction following, notably on detail and multi-object prompts, relative to the same base model trained on short, dense, or coarse-structured captions.","The 3DV AE metric offers a scalable automatic way to compare caption fidelity without human judges, since it measures latent-space distance between original and recaption-reconstructed videos.","The InstanceEnhancer shows that inference-time prompt restructuring can close the gap between short user prompts and long training captions, which is a known weakness of current T2V models.","The InstanceVid dataset and the structured caption schema can be reused directly by other T2V training pipelines, since the caption format is model-agnostic.","The human-designed class hints and positive/negative lexicon can be extended to new object categories, suggesting the pipeline generalizes beyond the roughly eighty classes tested."],"supporting_citations":[{"why":"MiraData supplies the coarse-level structured caption baseline that InstanceCap must beat in reconstruction fidelity and hallucination scores.","marker":"[9]"},{"why":"ShareGPT4Video represents the dense-caption approach whose hallucinations and redundancy InstanceCap claims to reduce.","marker":"[3]"},{"why":"Panda-70M provides the short-caption baseline that lacks sufficient video coverage.","marker":"[4]"},{"why":"CogVideoX-5b is the generation backbone used for reconstruction-via-recaption and also provides the 3D VAE whose latent distance defines the main fidelity metric.","marker":"[30]"},{"why":"SAM2 performs video instance segmentation, the key step that isolates instances from the global video with blurred backgrounds.","marker":"[20]"},{"why":"Open-Sora is the base T2V model fine-tuned on InstanceVid and the comparison point for all generation results.","marker":"[35]"},{"why":"OpenVid-1M is the source of the curated InstanceVid videos, so its quality and distribution shape the training set.","marker":"[16]"}],"fun_headline_variants":["Instance-aware captions cut AI video hallucinations","Per-instance descriptions sharpen video generation","Object-level captions boost text-to-video accuracy","Structured per-instance captions reduce video fakes","Decompose captions per instance for faithful video AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quantitative superiority over MiraData rests on the 3DV AE score, a distance between latent codes of original and regenerated videos whose validation only tested pixel-level distortions, not whether lower scores correspond to captions humans judge as more faithful; the approach also presumes object detection can find the instances that matter, which the paper itself notes fails in instance-free scenes.","fun_headline_variants_meta":{"raw":{"variants":["Instance-aware captions cut AI video hallucinations","Per-instance descriptions sharpen video generation","Object-level captions boost text-to-video accuracy","Structured per-instance captions reduce video fakes","Decompose captions per instance for faithful video AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000792,"raw_usage":{"total_tokens":3453,"prompt_tokens":875,"completion_tokens":2578,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":2505}},"tokens_in":491,"tokens_out":2578,"duration_ms":18011,"temperature":1.0,"reasoning_tokens":2505,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:05:08.424744+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a perceptual study on the same 100 reconstruction videos where human raters, blind to caption method, pick which regenerated video is closer to the ground truth; if MiraData or dense captions are chosen at least as often as InstanceCap, the 3DV AE advantage would not translate into perceived fidelity. A second check: re-run the Table 2 fine-tuning with InstanceVid captions stripped of their structured JSON into plain paragraphs, and see whether the Inseval gains disappear, which would isolate the structure as the cause.","supporting_citations":[{"cited_title":"Open-sora: Democratizing efficient video production for all, 2024","cited_arxiv_id":null,"evidence_quote":"Open-Sora is the base T2V model fine-tuned on InstanceVid and the comparison point for all generation results."}],"review_version":1}