{"id":"64dd05a7-b32f-4082-a6b8-8045340f1f6e","arxiv_id":"2611.16417","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"VBot Exo2Ego Policy generates egocentric first-person video from exocentric third-person footage by adapting a pretrained video diffusion transformer with stochastic camera conditioning and auxiliary hand-object supervision.","lead":"This paper presents a video generation model that converts third-person videos into first-person observations of a selected actor, along with a lightweight data collection pipeline using mobile phones. A smart generalist should read this because it offers a practical way to produce first-person operation data from abundant real-world footage, potentially lowering the barrier for training manipulation skills in robotics and activity recognition.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Camera conditioning efficacy lacks controlled ablation on the main evaluation set; the only evidence (Table 7) uses a different data pool and is labeled descriptive.","rationale":"The reader identified the camera conditioning claim as the weakest assumption, noting the loss comparison is descriptive and uses a different data pool. My stress-test confirms this as the most load-bearing concern: if the ray conditioning does not actually contribute to the reported gains, then the paper's argument for a key design choice (stochastic injection with annealing) collapses. The reader's conditional verdict already calls for a more rigorous camera-conditioning ablation, which is exactly what the concrete test provides. I do not see a more fundamental flaw—other comparisons (semantic, hand supervision) are properly ablated on the main set, and the baseline comparisons are credible given identical clip sets and metrics. Therefore the verdict should remain unchanged: conditional acceptance hinges on the proposed controlled ablation, consistent with the reader's original condition.","tokens_in":19541,"tokens_out":3190,"duration_ms":25998,"concrete_test":"Train two versions of VBot Exo2Ego Policy on the same paired dataset (main 546-clip evaluation set) with identical training schedule, hyperparameters, and loss weights: one with the stochastic ray conditioning branch (Bernoulli gate and annealing per Eq. 6) and one with the ray branch entirely removed throughout training. Evaluate both on the fixed 546-clip panel (Table 3). If ΔPSNR < 0.5 dB and ΔFVD < 10% on the unseen subset, the ray conditioning contributes negligibly to the headline results and the descriptive claim is unsupported; larger differences would validate the conditioning approach.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that stochastic camera conditioning enables training with calibrated rays yet allows uncalibrated inference—is not validated on the same evaluation protocol used for the headline comparison (Table 3). The sole supporting experiment (Table 7, Appendix A.6) compares flow-MSE between a model trained with ray conditioning on 33,056 clips at native 256P and a historical RGB reference trained on a different data pool and with a different schedule. The paper itself states 'The historical RGB reference uses a different data pool and learning-rate schedule, so its loss comparison is descriptive' (Appendix A.2, sentence b338). This means the contribution of ray conditioning to the reported PSNR/FVD gains of 3.23–3.25 dB and 37.6–39.8% is not empirically separable from other components (architecture, semantic/hand supervision, training data). The baselines (EgoX, EgoX/H3) use target-view geometric priors while VBot Exo2Ego Policy does not, yet no controlled ablation removes the ray branch on the main 546-clip 448×448 evaluation set to measure its effect. Without this, the claim that camera conditioning is beneficial generalizing to uncalibrated inference is weakly supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper presents VBot Exo2Ego Policy, a system for generating egocentric (first-person) video from exocentric (third-person) footage. It introduces a lightweight data collection pipeline using paired iPhone captures and single-view crowdsourcing, and a video diffusion model built on MiniMax-H3 with stochastic camera conditioning, semantic mask supervision, and hand keypoint supervision. The model is evaluated on a collected dataset of 546 clips (475 seen, 71 unseen) and compared to the EgoX and an in-house EgoX/H3 baseline. The main claims are that VBot Exo2Ego Policy achieves better reconstruction (PSNR gains of 3.23–3.25 dB), perceptual similarity (LPIPS reduction), and video-distribution quality (FVD reduction of 37.6–39.8%) compared to these baselines. Auxiliary experiments show improvements in hand keypoint extraction and object IoU from generated videos.","tokens_in":19969,"tokens_out":3032,"duration_ms":24008,"significance":"If the results are reproducible, the work offers a practical pipeline for expanding egocentric manipulation data from more abundant exocentric sources, which is valuable for robotics, activity understanding, and AR/VR. The data collection standard and the model's ability to handle multi-person scenes and camera cuts address realistic deployment scenarios. The system-level gains over existing methods are substantial, though the specific contributions of each component need clearer validation.","major_comments":[{"comment":"The paper claims that stochastic camera conditioning enables training with calibrated rays while allowing uncalibrated inference (Section 4.2). However, the only direct evidence for this claim (Table 7) uses a different data pool and the paper itself states 'The historical RGB reference uses a different data pool and learning-rate schedule, so its loss comparison is descriptive' (Appendix A.2, sentence b338). No controlled ablation on the main 546-clip 448×448 evaluation set is performed. Without such an ablation, the contribution of ray conditioning to the headline PSNR/FVD gains (Table 3) is unknown. The authors should either run a proper ablation (e.g., removing the ray branch) on the main evaluation set, or clearly delimit the scope of the camera conditioning claim and acknowledge that its benefit has not been validated on the primary protocol.","section":"Section 6.4, Table 7, Appendix A.2","authors_acknowledge":"The historical RGB reference uses a different data pool and learning-rate schedule, so its loss comparison is descriptive.","authors_acknowledge_p":0.66},{"comment":"The baseline comparisons use different input frame counts (EgoX: 49 frames, EgoX/H3: 141 frames) and different base architectures (EgoX vs. MiniMax-H3). While a 49-frame version of VBot is reported, the gains may partially originate from the larger frame count or the stronger base model rather than the proposed components. A more controlled baseline—e.g., an identical base model trained without camera conditioning, semantic supervision, and hand supervision—would help attribute the improvements. The discussion should explicitly address the confounding factors.","section":"Table 3, comparison details","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper describes a lightweight data collection pipeline but provides no quantitative evaluation of its quality or efficiency: e.g., success rate of the common-start SOP and Sim(3) alignment, user effort, or comparison to existing datasets like Ego-Exo4D. Without such data, the 'lightweight' claim is unsupported. The authors should report statistics on alignment accuracy, data yield per recording session, and any quality control metrics.","section":"Section 3, data collection pipeline","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The common-start SOP and joint reconstruction are described, but the success rate of the Pi3/DA3 reconstruction and Sim(3) alignment is not reported. How often does the procedure produce usable geometric pairing?","section":"Section 3.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The annealing schedule for ray probability uses a piecewise linear decay. Was this schedule tuned? What motivated the initial probability of 0.5? A brief justification would improve clarity.","section":"Equation (6), Section 4.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The training loss curves in Figure 5 are informative, but the figure caption and labeling could be clearer: indicate which curve corresponds to stochastic rays and which to historical RGB, and note that the historical RGB uses a different data pool.","section":"Appendix A.2, Figure 5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The variants 'ACEaux' and 'MultiLayer' are used throughout Section 6.3 before being fully defined. Consider introducing them explicitly in Section 5.2 with a brief summary of their differences.","section":"Section 5.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Minor typos and formatting inconsistencies: e.g., 'Sectio' in a few places (check references), variable capitalization of 'Exo/Ego' in figure captions. Also, the paper mentions 'checkpoint sensitivity' (Appendix A.3) but does not quantify variability; a brief analysis of training seed variance would strengthen the reliability claims.","section":"General","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The paper builds on known components (MiniMax-H3, ACE, Qwen) and adds stochastic camera conditioning with annealing, semantic HOI supervision, and hand keypoint supervision. The practical data collection pipeline is a real contribution, and the ablations for hand and semantic supervision are thorough and credible. The quantitative gains over EgoX (3.2 dB PSNR, ~38 % FVD reduction) are convincing as a system-level result, even if built on an upgraded backbone.\n\nThe soft spot is the camera conditioning claim. The paper admits (sentence b338) that the ray-conditioning ablation (Table 7) compares a model trained on a different data pool with a different schedule—i.e., it is descriptive, not controlled. That means the contribution of ray conditioning to the headline numbers is not empirically separable from the architecture, semantic supervision, or data. The stress-test note is correct: Table 7 does not use the main 546-clip 448×448 evaluation set, and no ablation removes the ray branch on that set. The authors could easily run a controlled ablation (train without rays on the same data), and I think a reviewer should ask for it. For now, the paper overclaims on what the camera module achieves.\n\nOther concerns: reproducibility is low (no code/data released), and the checkpoint sensitivity noted in the appendix (FVD jumps at different steps) deserves acknowledgment in the main text. The semantic-mask and hand supervision results, however, stand on cleaner ablations.\n\nWho is this for? Researchers working on view synthesis for manipulation data or EXO-to-EGO generation. It moves the needle incrementally over EgoX and provides a useful reference system.\n\nRecommendation: Accept with major revision. The authors should add a controlled camera-conditioning ablation on the main evaluation set and release at least the model weights and a subset of paired data. The descriptive nature of Table 7 is a real weakness, but the rest of the paper is solid enough to warrant referee time.","headline":"Solid engineering with credible gains over EgoX, but the camera conditioning claim lacks a controlled ablation on the main evaluation set—the only evidence is descriptive.","tokens_in":20446,"tokens_out":782,"duration_ms":7051},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Video model turns third-person clips into first-person views of an operation, outperforming prior work by clear margins.","keywords":["exocentric-to-egocentric","video generation","first-person video","diffusion transformer","camera conditioning","hand-object interaction","data collection pipeline","actor reference"],"falsifier":"If one collects a set of exo videos from a moving camera with no common-start calibration and the model fails to maintain consistent actor orientation or hand placement compared to a version that always receives ground-truth rays, then the annealing generalization is weaker than claimed.","tokens_in":21081,"feed_emoji":"👀","tokens_out":3732,"duration_ms":29825,"temperature":0.7,"pith_summary":"The paper aims to establish that exocentric-to-egocentric video generation for manipulation tasks can be made practical through a lightweight data collection pipeline and a diffusion model with stochastic camera conditioning. If successful, anyone with a phone could record a task from a third-person view and automatically obtain a first-person video of the same action, vastly expanding training data for robotic or assistance systems. The model uses a pretrained video diffusion transformer (MiniMax-H3) trained with camera rays that are gradually removed, so at inference no calibration is needed. Auxiliary supervision on hand keypoints and object masks improves the fidelity of hand-object interactions. Quantitative evaluations show clear gains over prior work EgoX: PSNR +3.23 dB, FVD -37.6%.","feed_headline":"Exo-to-ego video gains 3.2 dB PSNR, cuts FVD 38%","feed_subtitle":"Outperforms EgoX with stochastic conditioning and hand supervision, and needs no calibration at inference.","key_machinery":"Stochastic camera conditioning adds Plücker rays to the EGO tokens during training via zero-initialized projections, with a Bernoulli gate that is annealed to zero so the final model does not depend on calibrated rays. Auxiliary supervision uses semantic masks for hand/object regions and ACE-Ego-Hand features plus 2D keypoints to shape the EGO representation for better hand articulation.","core_discovery":"VBot Exo2Ego Policy introduces a capture protocol that pairs two iPhones for synchronized ego/exo RGB-D video with ARKit poses, and a single-view crowdsourcing route for broader coverage. The generative model, built on a frozen video diffusion transformer, learns viewpoint conversion via stochastic Plücker-ray conditioning that is annealed away during training, allowing inference to proceed from just the exo video, text description, and an optional actor reference. Semantic hand/object masks and ACE-based hand feature supervision train the generator to produce videos from which hand joints and object masks can be reliably extracted by off-the-shelf models. The system outperforms EgoX on all evaluated metrics, with particular gains in hand localization accuracy.","pith_inferences":["The common-start SOP could eventually be replaced by automatic wide-baseline matching from feed-forward geometry models, making the data pipeline even lighter.","The annealing schedule might be replaced by consistency distillation or adversarial techniques to remove calibration requirements more directly.","This approach suggests that diffusion models can internalize 3D structure through stochastic geometric conditioning, opening the door to geometry-free inference for many view-conversion tasks.","If the model generalizes well, it could reduce the need for costly first-person data collection in dexterous manipulation learning across labs."],"forward_implications":["Large-scale generation of first-person operation videos from internet exo footage, enabling cheaper data expansion for robot learning or skill instruction.","The capture standard (paired iPhones plus common-start SOP) can be adopted by other labs to create high-quality ego-exo benchmarks for novel-view synthesis.","The stochastic conditioning method may generalize to other novel-view synthesis tasks where inference-time camera calibration is inconvenient or unavailable.","Hand supervision from ACE features demonstrates that diffusion generators can be trained to produce videos from which downstream perception modules recover more consistent joint structure."],"supporting_citations":[{"why":"Provides the primary baseline EgoX for exo-to-ego video generation; the paper compares against it directly.","marker":"[21]"},{"why":"Supplies synchronized ego-exo data (Ego-Exo4D) used for paired training and evaluation.","marker":"[23]"},{"why":"Supplies the ACE-Ego-Hand features and pseudo-labels used for hand-supervision.","marker":"[24]"},{"why":"Provides ground-truth hand annotations from HOT3D for calibrating the hand readout.","marker":"[25]"},{"why":"Supplies DUSt3R as a feed-forward geometry model considered for cross-view estimation.","marker":"[8]"}],"fun_headline_variants":["Exo-to-ego video without calibration beats EgoX by 3.2 dB PSNR","Lightweight dual-phone system generates exo-to-ego video without calibration","Hand-supervised exo-to-ego video cuts FVD 38%, needs no calibration","Stochastic conditioning enables inference-time calibration-free exo-to-ego","Exocentric to egocentric video with better hand localization and no calibration"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The paired capture pipeline assumes that the common-start SOP provides sufficient visual overlap for reconstructing metric camera geometry, and that this geometry, learned stochastically, generalizes to uncalibrated inference after annealing.","fun_headline_variants_meta":{"raw":{"variants":["Exo-to-ego video without calibration beats EgoX by 3.2 dB PSNR","Lightweight dual-phone system generates exo-to-ego video without calibration","Hand-supervised exo-to-ego video cuts FVD 38%, needs no calibration","Stochastic conditioning enables inference-time calibration-free exo-to-ego","Exocentric to egocentric video with better hand localization and no calibration"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000629,"raw_usage":{"cost":0.000628908,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000628908,"upstream_inference_prompt_cost":9.6068e-05,"upstream_inference_completions_cost":0.00053284},"total_tokens":2794,"prompt_tokens":891,"completion_tokens":1903,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":1796}},"tokens_in":891,"tokens_out":1903,"cost_source":"openrouter_usage","duration_ms":16045,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":1796,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T15:10:22.968528+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"If one collects a set of exo videos from a moving camera with no common-start calibration and the model fails to maintain consistent actor orientation or hand placement compared to a version that always receives ground-truth rays, then the annealing generalization is weaker than claimed.","supporting_citations":[{"cited_title":"EgoX: Egocentric Video Generation from a Single Exocentric Video","cited_arxiv_id":null,"evidence_quote":"Provides the primary baseline EgoX for exo-to-ego video generation; the paper compares against it directly."},{"cited_title":"Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives","cited_arxiv_id":null,"evidence_quote":"Supplies synchronized ego-exo data (Ego-Exo4D) used for paired training and evaluation."},{"cited_title":"ACE-Ego-Hand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery","cited_arxiv_id":"2608.20308","evidence_quote":"Supplies the ACE-Ego-Hand features and pseudo-labels used for hand-supervision."},{"cited_title":"HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos","cited_arxiv_id":"2411.19167","evidence_quote":"Provides ground-truth hand annotations from HOT3D for calibrating the hand readout."},{"cited_title":"DUSt3R: Geometric 3D Vision Made Easy","cited_arxiv_id":"2312.14132","evidence_quote":"Supplies DUSt3R as a feed-forward geometry model considered for cross-view estimation."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}