{"id":"eaf81cbe-6f8b-4625-895b-b4e3bda1aa2c","arxiv_id":"2502.08639","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Users control a text-to-video model by arranging 3D bounding boxes and camera motion, and the system renders depth maps that steer the diffusion model to generate matching object and camera motion.","lead":"CineMaster is a system for generating videos from text where users place 3D boxes around objects, move them, and move the camera; the system renders depth maps and camera paths that guide a video diffusion model to match the plan. It adds a data pipeline that automatically extracts 3D boxes and camera trajectories from ordinary videos, and reports better control and quality than two existing methods.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing risk is that the Sec. 3.3 annotation pipeline is never validated against any ground-truth 3D data, and it composites depth, camera, and tracking estimates whose scales are never aligned.","rationale":"The reader identified the same core assumption: the automated annotation pipeline must be accurate enough to supervise a video diffusion model, and the paper never validates it against ground truth. I agree that this is the most load-bearing point. Without an independent check, the training signal is opaque, and the evaluation metrics do not provide a safeguard because they are defined against the user-specified control signals rather than against verified 3D geometry. The concern is concrete: Sec. 3.3 stitches together three different estimators with different scale conventions and no explicit alignment step, and the claimed 'ground-truth' dataset label is not supported. This does not by itself refute the method; a noisy-label model could still produce useful qualitative control, and the staged training on dense depth maps may mitigate the issue. However, it does mean the central quantitative claim should not be accepted without the proposed validation. I also note a smaller internal inconsistency: Table 2's numeric ordering implies the columns are mIoU, Traj-D, FID, FVD, CLIP-T, Depth-D, while Table 1 orders FVD before FID; this is not decisive but adds to the need for care. The paper also lacks code, data, and error bars, but those are secondary to the label-validity question. Overall, the conditional verdict stands: the concept is plausible, but the missing validation of the data-labeling pipeline is a genuine load-bearing gap.","tokens_in":12972,"tokens_out":6798,"duration_ms":69558,"concrete_test":"Render a held-out set of Blender videos (same engine as the interactive workflow) with known 3D object boxes, camera poses, and ground-truth depth, then run the Sec. 3.3 pipeline on them. Report per-object 3D box center error and IoU, camera trajectory error, and the scale ratio between DepthAnything V2 depth and MonST3R/SpatialTracker trajectories. If the projected 3D box IoU falls below roughly 0.7 or the mean 3D center error exceeds about 10% of object size, the training labels are too noisy to support the precise-control claim. Also recompute Table 2 with the column order used in Table 1 to confirm that the reported FVD/FID values are not swapped.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CineMaster learns precise 3D box and camera control from in-the-wild video. That requires the Sec. 3.3 pipeline to produce 3D boxes, object displacements, and camera poses in one consistent metric frame. The pipeline composes DepthAnything V2 metric depth (for inverse projection of the initial box), MonST3R camera poses, and SpatialTracker 3D point displacements (for moving the box across frames), with no stated coordinate/scale alignment and no validation against ground truth. Monocular estimators typically have different scale conventions, so a box computed in DepthAnything V2 coordinates and moved by SpatialTracker displacements may not project back onto the instance masks; more importantly, if the errors are systematic and correlated across the 156K-video training set, the Semantic Layout ControlNet and Camera Adapter learn a biased mapping from user-drawn boxes and rendered depth to videos. The evaluation cannot detect this bias because the 'ground-truth' boxes are the user-specified control signals rather than independently verified 3D annotations, and Depth-D is measured with DepthAnything V2 itself. The paper labels the pipeline outputs as ground truth in Sec. 3.3 ('datasets with both the ground-truth 3D bounding box and 3D camera trajectory annotations'), which overstates the reliability of an unvalidated automated estimator. Until the label source is checked, the precise-control claim and the quantitative superiority over MotionCtrl and Direct-A-Video are not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes CineMaster, a two-stage framework for 3D-aware controllable text-to-video generation. In the first stage, users construct control signals by placing 3D bounding boxes with class labels and defining camera trajectories in Blender; the system renders per-frame depth maps, camera poses, and class-label maps. In the second stage, a latent diffusion T2V model is fine-tuned with a Semantic Layout ControlNet (fusing depth latents with class-label embeddings placed via a scatter mask) and a Camera Adapter that injects 6-DoF camera poses into each DiT block. To obtain training data, the authors build an automated pipeline that extracts instance masks (Grounding DINO + SAM 2), metric depth (DepthAnything V2), camera poses (MonST3R), and per-object 3D boxes tracked by SpatialTracker displacements, producing 156K videos with 3D box and camera annotations. Experiments compare CineMaster to MotionCtrl and Direct-A-Video on mIoU, Traj-D, FVD, FID, CLIP-T, and an ablation study over training paradigms.","tokens_in":13258,"tokens_out":6418,"duration_ms":53178,"significance":"If the claims hold, CineMaster provides a practical 3D-native control interface for T2V generation and a scalable way to obtain 3D annotations from in-the-wild video. The paper's strengths include a clear system architecture, a substantial data pipeline, and qualitative demonstrations of joint object/camera control. The authors also make a good-faith attempt to evaluate with multiple metrics and baselines. However, the central quantitative claims rest on unvalidated labels and single-run numbers, so the paper's main contribution is currently not fully supported. The ideas are sufficiently novel and promising that the concerns can be addressed with additional validation and experiments.","major_comments":[{"comment":"The automatic labeling pipeline is the central source of training supervision, but it is never validated against ground-truth 3D data. The pipeline (Fig. 4) composites DepthAnything V2 metric depth, MonST3R camera poses, and SpatialTracker 3D displacements without a stated coordinate or scale alignment. These estimators are known to have different scale conventions; a box computed via inverse projection in the depth frame and moved with tracker displacements may not be consistent with the MonST3R camera frame. If the errors are systematic, the learned Semantic Layout ControlNet and Camera Adapter inherit the bias. In Section 3.3 and the Introduction, the outputs are called 'ground-truth 3D bounding box and 3D camera trajectory annotations,' which overstates an unvalidated estimator. Please compare the extracted boxes and trajectories against ground-truth annotations (e.g., synthetic sequences or scanned scenes) and report errors.","section":"3.3"},{"comment":"Depth-D is computed with SAM 2 and DepthAnything V2 on generated frames, and the reference depths are derived from the given 3D boxes. Since DepthAnything V2 is also the model used to generate the metric depth labels for training (Sec 3.3 and Sec 4.1), Depth-D effectively measures how well the model reproduces DepthAnything V2's output rather than true geometric accuracy. This circularity weakens the claim of improved depth control. Please either replace Depth-D with an independent depth source or explicitly acknowledge this limitation and supplement with a non-circular validation (e.g., on synthetic scenes with known depth).","section":"4.1, Evaluation Metrics"},{"comment":"The quantitative results are reported as single-run point estimates with no standard deviations, confidence intervals, or significance tests. The differences between CineMaster and Direct-A-Video on FID (175.9 vs 183.5) and CLIP-T (0.321 vs 0.273) are considerable, but without variance we cannot assess reliability. Additionally, only two baselines are used, and their conditions are converted from the 3D box condition (object trajectories for MotionCtrl, 2D boxes for Direct-A-Video), which may put them at a disadvantage. The abstract's claim of 'significantly outperforms' is not established. Please report statistics across multiple seeds, include more recent 3D-aware baselines (e.g., 3DTrajMaster, SynFMC, MotionCanvas), and use a consistent condition-conversion protocol with an analysis of its effect.","section":"Tables 1 and 2"},{"comment":"The ablation study varies training paradigms but does not include a 'without Camera Adapter' condition. The paper's contribution is joint object and camera control, and Section 3.2 argues that the camera adapter resolves ambiguity between object and camera motion. Without an ablation that removes or freezes the camera adapter, the benefit of joint training cannot be separated from the benefit of the Semantic Layout ControlNet alone. Please add such an ablation (and optionally a 'without depth maps' ablation) to support the architectural claims.","section":"4.3, Table 2"}],"minor_comments":[{"comment":"The label '3D Depth Box' in the figure is unclear; consider simplifying it to '3D Box' or 'Projected Depth Maps' to match the text.","section":"Figure 2"},{"comment":"The sentence 'We make a trade-off between object motion and camera motion by injecting semantic layout information and camera poses with 25 and 15 steps respectively' is vague; please specify whether these are training steps, inference steps, or guidance scales for the two adapters.","section":"4.1"},{"comment":"The claim of building 'the largest video datasets with both the ground-truth 3D bounding box and 3D camera trajectory annotations' is difficult to verify without a dataset release or comparison with existing annotated datasets (e.g., SynFMC); please clarify that the annotations are automatically estimated and not human-verified.","section":"1 and 3.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript would benefit from a clearer statement about the limitations of the automatic annotations in the main text rather than calling them ground truth. Also, the paper does not compare with recent 3D-aware methods despite citing them; this should be addressed if not in the main comparison. Given the current evidence, I would not accept without substantial revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a genuinely useful framework for 3D-native control of text-to-video generation, and the qualitative results are convincing enough to take the idea seriously. But the paper’s quantitative evidence is thinner than the claims, and the biggest unresolved risk is the unvalidated auto-labeling pipeline that produces the training data and is also used for evaluation.\n\nWhat's new: combining user-placed 3D bounding boxes, rendered depth maps, camera trajectories, and class labels as joint conditions in a DiT-based video diffusion model. The interactive Blender workflow is a practical contribution. The automatic pipeline for large-scale in-the-wild labels (instance segmentation + metric depth + MonST3R + SpatialTracker) is ambitious and could be valuable if validated. The ablations are well-designed and show that joint training and semantic injection matter. The paper is honest that orientation control is not supported.\n\nThe soft spots are real, but not fatal. Tables 1 and 2 report single runs with no error bars; only two baselines, both with conditions converted from their native formats. I don't think the conversion is malicious, but it makes “significantly outperforms” hard to assess. More importantly, the Sec. 3.3 pipeline is never checked against any ground-truth 3D annotations. DepthAnything V2, MonST3R, and SpatialTracker are composed without describing any coordinate/scale alignment, and the paper calls the outputs “ground-truth.” That overstates what an unvalidated monocular estimator mixture gives you. The Depth-D metric also uses DepthAnything V2, the same model that generated training depth labels, so it partially measures the model’s ability to imitate one depth estimator. If the pipeline has systematic biases, both the training signal and the metric inherit them.\n\nThat said, the central idea does not collapse. The qualitative comparisons show clear differences in object/camera decoupling, and the ablations are directionally consistent. The stress-test concern is valid, but it’s a validation gap, not an internal contradiction.\n\nWho this is for: researchers working on controllable video generation, especially those building training-data pipelines for 3D conditions. It deserves a serious referee, though it needs revision: error bars, more baselines, and an explicit ground-truth validation of the labeling pipeline (even on a small synthetic set or RealEstate10K poses) before the “ground-truth” language is used. I’d send it for review.","headline":"A promising 3D-box-and-camera control framework for text-to-video generation, with an ambitious but unvalidated auto-labeling pipeline and thin quantitative evidence; worth refereeing, not ready as is.","tokens_in":13812,"tokens_out":2372,"would_cite":true,"duration_ms":22736,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CineMaster turns 3D box layouts into camera-controlled video.","keywords":["text-to-video generation","3D-aware video generation","camera motion control","object motion control","diffusion transformer","3D bounding boxes","depth conditioning","video dataset annotation"],"falsifier":"Take synthetic or LiDAR-captured videos with known 3D object boxes and camera poses, run the labeling pipeline on them, and compare the extracted boxes and camera paths to the ground truth; if the errors are comparable to or larger than the motions the user wants to control, or if retraining with perfect labels fails to change the outputs, the central claim is disproved.","tokens_in":12761,"feed_emoji":"🎬","tokens_out":6945,"duration_ms":125578,"temperature":0.7,"pith_summary":"CineMaster claims that a text-to-video diffusion model can be steered like a film shoot by conditioning it on 3D bounding boxes, camera trajectories, per-frame projected depth maps, and object class labels. Users first stage objects and camera in a 3D scene editor, then the model renders the planned video from the exported control signals. Because real videos rarely carry 3D box and camera annotations, the paper adds an automated labeling pipeline that extracts those signals from in-the-wild footage, and it trains the layout controller and camera controller jointly so object motion and camera motion do not conflate. In direct comparisons, CineMaster reports better object-box alignment, lower trajectory deviation, and better video-quality and text-alignment scores than prior joint controllers. If the claim holds, this is a practical route from 3D shot planning to generated video for users who want director-level placement control.","feed_headline":"CineMaster turns 3D box layouts into camera-controlled video","feed_subtitle":"Depth maps, camera paths, and class labels let text-to-video models follow a director's shot plan.","key_machinery":"The load-bearing machinery is a Semantic Layout ControlNet paired with a Camera Adapter, built on a pretrained text-to-video transformer. The ControlNet copies half the transformer blocks; a semantic injector takes per-entity class-label embeddings and splices them into the hidden states at the positions given by downsampled entity masks, then concatenates the fused semantic map with depth latents and projects it with MLPs. That gives every subject an explicit 3D location. The Camera Adapter adds a sequence of 6-DoF camera poses (rotation plus translation) as residual features between the self-attention and temporal-attention layers of each block, so the model can tell apart 'the balloon rose' from 'the camera dropped.' The rendered depth maps serve as the common visual ground that ties the 3D scene layout to each output frame. Around this sits a data-labeling pipeline that turns raw video into the training conditions, plus a three-stage training schedule that first learns dense depth control, then adapts to 3D boxes, then jointly trains both controllers.","core_discovery":"The paper's central claim is that joint 3D object and camera control in text-to-video generation is achievable by combining three conditions: rendered depth maps that encode the desired 3D layout, camera trajectories that disambiguate camera motion from object motion, and per-entity class labels that bind each prompt subject to its 3D box. The authors show this through a two-stage design in which an interactive 3D workflow produces the conditions, and a fine-tuned diffusion transformer consumes them. They further claim that an automated pipeline—instance segmentation, metric depth, monocular camera pose, and 3D point tracking—can generate reliable 3D box and trajectory annotations at scale, yielding the largest training set of its kind and enabling joint training that outperforms existing 2D-trajectory and box-based controllers on every reported metric.","pith_inferences":["The method deliberately leaves out object orientation: rotating a 3D box to turn a person around is named as future work, so current control is translation-and-placement level rather than full 6-DoF object control.","Reported metrics measure alignment using automated detectors and monocular depth, so the true ceiling may be coupled to the same estimators that build the training labels; a human or synthetic-ground-truth evaluation would give a cleaner read.","If the monocular estimates in the labeling pipeline drift, the error is likely correlated across depth, pose, and tracking, which could bias the learned controller in ways that per-video examples do not reveal.","The same condition set could be transferred to image-to-video models or to user-provided 3D keyframe animations, since the control signals are renderer-agnostic."],"forward_implications":["Users can specify object placement and camera motion in 3D, and the generated video is claimed to follow both simultaneously without the object-camera coupling seen in prior controllers.","The automated labeling pipeline removes the need for synthetic or manually annotated 3D video datasets, so training data can be scaled up from in-the-wild footage.","Joint training of the layout controller and camera adapter is required: training them separately or freezing one while training the other degrades alignment, quality, and depth accuracy.","Because depth maps carry the layout, the method can use rendered depth from any 3D scene editor, not just footage-derived conditions."],"supporting_citations":[{"why":"Contributes the 3D bounding box as the scene representation and the line from depth conditioning to layout control.","marker":"[1]"},{"why":"Supplies metric depth maps used to build 3D point clouds and to pretrain the depth-based ControlNet.","marker":"[44]"},{"why":"Estimates camera poses from in-the-wild videos, providing camera trajectory labels for training.","marker":"[47]"},{"why":"Tracks 3D points from each entity's best frame to the rest, giving the inter-frame displacements that move each 3D box.","marker":"[41]"},{"why":"Detects open-set object boxes that seed instance segmentation for the labeling pipeline.","marker":"[24]"},{"why":"Produces video instance masks used to isolate entities and compute their 3D boxes.","marker":"[29]"},{"why":"Baseline that couples object and camera motion; the paper must beat its trajectory and quality numbers.","marker":"[40]"},{"why":"Baseline controlling object motion via 2D boxes and camera via training-free augmentation.","marker":"[45]"},{"why":"Supplies additional videos with wide camera motion for joint training.","marker":"[50]"}],"fun_headline_variants":["Direct your AI video: CineMaster adds 3D camera control","CineMaster: 3D boxes and camera paths steer text-to-video","From 3D layout to cinematic video: CineMaster's new controls","CineMaster lets you place objects and move camera in 3D","3D-aware text-to-video: CineMaster puts you in the director's chair"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the pipeline's estimated depths, camera positions, object masks, and 3D point tracks are accurate enough to teach the model what user-drawn 3D boxes should look like in video, and the paper does not verify these estimates against true 3D measurements.","fun_headline_variants_meta":{"raw":{"variants":["Direct your AI video: CineMaster adds 3D camera control","CineMaster: 3D boxes and camera paths steer text-to-video","From 3D layout to cinematic video: CineMaster's new controls","CineMaster lets you place objects and move camera in 3D","3D-aware text-to-video: CineMaster puts you in the director's chair"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000225,"raw_usage":{"total_tokens":1465,"prompt_tokens":946,"completion_tokens":519,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":416}},"tokens_in":562,"tokens_out":519,"duration_ms":4584,"temperature":1.0,"reasoning_tokens":416,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T23:57:18.464372+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take synthetic or LiDAR-captured videos with known 3D object boxes and camera poses, run the labeling pipeline on them, and compare the extracted boxes and camera paths to the ground truth; if the errors are comparable to or larger than the motions the user wants to control, or if retraining with perfect labels fails to change the outputs, the central claim is disproved.","supporting_citations":[{"cited_title":"Loosecontrol: Lifting controlnet for generalized depth conditioning","cited_arxiv_id":null,"evidence_quote":"Contributes the 3D bounding box as the scene representation and the line from depth conditioning to layout control."},{"cited_title":"Spa- tialtracker: Tracking any 2d pixels in 3d space","cited_arxiv_id":null,"evidence_quote":"Tracks 3D points from each entity's best frame to the rest, giving the inter-frame displacements that move each 3D box."},{"cited_title":"Sam 2: Segment anything in images and videos, 2024","cited_arxiv_id":null,"evidence_quote":"Produces video instance masks used to isolate entities and compute their 3D boxes."},{"cited_title":"Direct-a-video: Customized video generation with user-directed camera movement and object mo- tion","cited_arxiv_id":null,"evidence_quote":"Baseline controlling object motion via 2D boxes and camera via training-free augmentation."},{"cited_title":"Stereo magnifica- tion: Learning view synthesis using multiplane im- ages, 2018","cited_arxiv_id":null,"evidence_quote":"Supplies additional videos with wide camera motion for joint training."}],"review_version":1}