{"id":"a5e08f2b-50b2-4a8b-8b31-0c63694b6ea5","arxiv_id":"2506.03605","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new 28,497-sample dataset of 6DoF object manipulation trajectories is automatically extracted from egocentric video, and vision-language models are trained to generate these trajectories from action descriptions.","lead":"Researchers automatically extracted 28,497 text-described object movement trajectories from everyday first-person videos, then trained vision-language models to generate such trajectories from action descriptions. If the extracted trajectories are reliable, this could provide abundant training data for robots that follow spoken or written instructions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Rotation labels in EgoTraj are computed via SVD of tracked point clouds but never validated against ground truth; the flat GD across models and sampling suggests the model may be predicting near-constant rotations, so the claim of valid 6DoF trajectory generation is not yet supported.","rationale":"The reader's weakest_assumption identified exactly the SVD-based rotation extraction as unvalidated, and the evidence I examined reinforces this as the most load-bearing concern. The paper's headline claim is that the models 'successfully generate valid object trajectories'; validity includes the rotation component, which is measured by GD. The GD results are suspiciously invariant across models (0.541–0.543 in Table 2) and across sampling counts (0.541→0.540 in Table 4), which is a strong signal that the rotation supervision is not informative. The extraction pipeline combines monocular depth (scale-ambiguous, noisy), dense tracking of potentially occluded points, and a rigidity assumption; any of these failures would directly corrupt the SVD-based rotation labels. Since the training set itself has no ground truth, the only way to validate the rotation extraction is to run the same pipeline on a dataset with ground-truth 6DoF poses, such as HOT3D, and compare. The reader's CONDITIONAL verdict is appropriate: the paper is promising but the central empirical claim is not yet robustly supported. My stress-test does not change that verdict; it sharpens the specific condition (validate rotation extraction) that would settle the matter. I do not see an additional concern that would warrant escalation to REJECT, because the proposed check is feasible and the contribution would be valuable if the extraction error is small. I also agree with the reader that the best-of-n sampling and machine-generated test instructions are secondary concerns, but they are not as directly fatal to the dataset's core validity as the unvalidated rotation labels.","tokens_in":20666,"tokens_out":4202,"duration_ms":50704,"concrete_test":"Run the Sec. 3.1 extraction pipeline on a random subset of HOT3D clips using only the RGB frames and the same modules (GPT-4o temporal localization, Grounded SAM, SpaTracker, Depth Anything, point-cloud registration, and SVD rotation extraction), then compare the extracted relative 6DoF trajectories to the HOT3D ground-truth object poses. Report the median and mean per-trajectory geodesic rotation error and position error after aligning the initial frames. If the extraction rotation error is comparable to the model's GD (~0.54 rad) or larger, the training labels are too noisy to support the validity claim; if it is substantially smaller, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the extracted trajectories are valid and that models trained on them generate valid 6DoF object trajectories—rests on the correctness of the rotation labels in the automatically constructed training set. In Sec. 3.1 (Rotation Sequence Extraction), the rotation at each timestep is obtained by SVD alignment of the tracked object point cloud: H = UΣV^T, R = VU^T, under the assumptions of object rigidity, accurate 3D point tracking, and reliable monocular depth. None of these assumptions is quantitatively validated against any ground-truth 6DoF data. The paper's own Limitations section concedes that segmentation and point-cloud registration can fail, but the impact on rotation labels is never measured. This matters because the experimental evidence is consistent with a failure to learn meaningful rotations: the geodesic distance (GD) is nearly identical (~0.54 rad) across all backbone models and modalities in Table 2, and increasing the number of sampled trajectories from 1 to 10 in Table 4 changes GD from 0.541 to 0.540, an order-of-magnitude smaller improvement than for position metrics. Such flatness suggests the model is outputting a near-constant rotation, which would be expected if the training rotation supervision is corrupted. Since the dataset contribution and the model-evaluation claim both depend on the rotation component, the absence of any validation of the SVD-based rotation extraction is the most load-bearing weakness in the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an automated pipeline for extracting 6DoF object manipulation trajectories from egocentric videos in Ego-Exo4D, without using pre-recorded camera extrinsic parameters. The pipeline uses GPT-4o for temporal localization and action annotation, Grounded SAM for segmentation, monocular depth estimation and SpaTracker for 3D point tracking, point-cloud registration for camera alignment, and SVD alignment of tracked point clouds for rotation extraction. This yields EgoTraj, a dataset of 28,497 trajectories with action descriptions. The authors then train several VLM backbones (BLIP-2, VILA, PointLLM, MiniGPT-3D) with discretized trajectory tokens and evaluate on HOT3D, reporting ADE, FDE, and geodesic distance. They also show that conditioning captioning on trajectories improves verb similarity. The central claim is that the extracted trajectories are valid and that the trained models generate valid 6DoF object manipulation trajectories from action descriptions.","tokens_in":20975,"tokens_out":3999,"duration_ms":40902,"significance":"If the rotation labels were validated, the contribution would be significant: EgoTraj would be the first large-scale, text-annotated 6DoF manipulation trajectory dataset extracted from unconstrained egocentric video, with an automated and scalable pipeline. The evaluation on HOT3D uses external ground-truth tracking, so the main generation experiment is not circular, and the release of dataset and code is a concrete strength. The comparison against USST and the trajectory-conditioned captioning experiments also provide useful baselines. However, the paper's central claim hinges on the correctness of the automatically extracted rotation supervision, and that component is never validated. The near-constant geodesic distance across models and sampling sizes is a red flag that the rotation channel may be degenerate, so the significance of the work is currently conditional on resolving this issue.","major_comments":[{"comment":"The rotation labels in EgoTraj are obtained by SVD alignment of SpaTracker point clouds under the assumptions of object rigidity, accurate 3D point tracking, and reliable monocular depth, but no quantitative validation of these labels is provided. The paper's own Limitations section and Appendix Fig. 11 state that object segmentation and point-cloud registration can fail, and the Data Curation Methods discard such clips, yet the impact of these failures on the resulting rotation distribution is never measured. Because the central claim of 'valid 6DoF trajectories' depends directly on the rotation channel, I request a validation experiment: run the extraction pipeline on HOT3D videos (where ground-truth 6DoF trajectories exist) and report the rotation error between extracted and ground-truth rotations, along with the fraction of clips rejected at each stage. Without such a check, the training supervision for rotations is unverified.","section":"Sec. 3.1 (Rotation Sequence Extraction); Limitations; App. A.1 (Failure Cases)"},{"comment":"The rotation results are suspiciously flat. In Table 2, the geodesic distance is approximately 0.54 rad for nearly every model regardless of backbone or input modality, and Table 4 shows that increasing sampling from 1 to 10 improves GD only from 0.541 to 0.540 while ADE improves from 0.271 to 0.212. This pattern is consistent with the model emitting a near-constant rotation (for example, the identity or the training-set mean), and the text itself acknowledges 'the difficulty in generating accurate rotational movements.' To support the claim that the models generate valid 6DoF trajectories, the authors should compare against trivial baselines (identity rotation, mean rotation, constant-rotation Seq2Seq) and report the distribution of predicted rotations, such as per-axis variance and the fraction of predictions at bin boundaries. If the GD of these trivial baselines matches the reported values, the rotation-generation claim should be weakened accordingly.","section":"Table 2; Table 4; Sec. 5.2"},{"comment":"The test-time action descriptions for HOT3D are generated by GPT-4o using a prompt nearly identical to the prompt used to create training descriptions in EgoTraj, and the same model is used for temporal action localization in both. This introduces a style and vocabulary coupling between training and test text: the models may be tuned to reproduce GPT-4o's phrasing rather than to understand human-written instructions. This is not circularity in the 6DoF ground truth, but it is a confound for the claim that the model generalizes from action descriptions. I ask the authors to report whether evaluation results change when human-written or paraphrased descriptions are used, or at minimum to quantify the lexical overlap between training and test descriptions.","section":"Sec. 3.2 and App. A.2 (Action Description Generation); Sec. 5.1"}],"minor_comments":[{"comment":"The text contains a typo: 'pre-recored camera coordinates' should be 'pre-recorded camera coordinates.'","section":"Introduction"},{"comment":"The caption contains a typo: 'repored in [5]' should be 'reported in [5].'","section":"Table 1 caption"},{"comment":"The statement that a hand-object detection model [81] is used is confusing because reference [81] is 'Understanding human hands in contact at internet scale,' which is a contact-detection dataset paper, not an obvious hand-object detection method; please clarify the actual detection model used.","section":"App. A.1"},{"comment":"The introduction says the method works 'without pre-recorded camera coordinates or depth,' but the pipeline itself relies on monocular depth estimation. This is not an error, but the wording should be clarified to 'without pre-recorded camera coordinates or depth sensors' to avoid confusion.","section":"Sec. 3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is potentially interesting, but the flat geodesic distance and the absence of any validation of the SVD-based rotation extraction are serious concerns. I would not recommend rejection if the authors can demonstrate that trivial rotation baselines perform worse and that the extracted rotations are accurate on HOT3D; however, as currently written, the central claim that the models generate valid 6DoF trajectories is not supported by the evidence. The GPT-4o coupling between training and test descriptions is also a concern that should be addressed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe real contribution here is EgoTraj: 28,497 automatically extracted 6DoF trajectories from Ego-Exo4D, built without camera extrinsics. That is genuinely new and potentially useful as a diverse training source for text-conditioned manipulation generation, especially since the closest prior work (HOT3D) has only 986 trajectories. The pipeline—GPT-4o for action spans and object names, Grounded SAM for segmentation, SpaTracker for 3D points, point-cloud registration for camera motion, SVD for rotations—is clearly described and the paper ships code and data. The authors also do something right that many papers skip: they evaluate on an external dataset (HOT3D) with real tracking ground truth, so the main generation result is not circular, and they compare against a nontrivial baseline (USST) trained on their data. The captioning experiment showing verb similarity improves with trajectory input is a nice, if indirect, sanity check.\n\nThe soft spot is exactly where the stress-test note lands: the rotation labels in EgoTraj are never validated. The SVD alignment assumes rigidity, accurate point tracking, and reliable monocular depth, and the paper's own Limitations section admits segmentation and registration failures. None of that is quantified against any ground-truth 6DoF data. The evidence in Table 2 is consistent with the model learning a near-constant rotation: GD is about 0.54 rad across nearly all backbones and modalities, and going from 1 to 10 samples in Table 4 changes GD from 0.541 to 0.540, an order of magnitude smaller than the position-metric improvements. That flatness is suspicious. It might mean the task is genuinely hard, but it might mean the rotation supervision is corrupted. The paper does not give the reader a way to tell.\n\nThe other issues are real but smaller: the test-set action descriptions are generated by GPT-4o, so the model may be tuned to GPT-4o's text style; the best-of-n sampling in Table 4 inflates position numbers; and the average trajectory is short (15.39 frames after cropping), which limits how much can be claimed about long-horizon manipulation.\n\nWho is this for? People working on text-conditioned manipulation generation, egocentric video understanding, and scalable dataset construction. The dataset itself is worth having regardless of the model results. The paper deserves a serious referee—not a desk reject—but the referee should push hard for rotation-label validation, either by running the extraction pipeline on a subset of HOT3D or another ground-truth dataset, or by reporting rotation error on a held-out manually annotated sample. Without that, the central claim of valid 6DoF generation rests on an unmeasured assumption; the paper as it stands is a solid dataset paper with an unproven modeling claim on top.\n\nRecommendation: send it to review, require the rotation validation as a condition of acceptance, and treat the generation results as provisional until then.","headline":"Useful dataset contribution, but the rotation labels are unvalidated and the flat GD numbers suggest the rotation part of the claimed 6DoF generation may be trivial.","tokens_in":21510,"tokens_out":726,"would_cite":true,"duration_ms":9830,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Ordinary egocentric video can supply 28,497 text-labeled 6DoF manipulation trajectories, and models trained on them turn action descriptions into valid object motions.","keywords":["6DoF object manipulation","egocentric vision","trajectory generation","vision-language models","point clouds","action descriptions","dataset construction","singular value decomposition"],"falsifier":"Run the extraction pipeline on a motion-capture egocentric dataset whose true 6DoF object poses are known, compute the per-frame geodesic distance between the extracted and ground-truth rotations, and check whether the extraction error alone is already as large as the prediction errors the paper reports; if it is, the rotation labels are too corrupted to support the claimed trajectory validity.","tokens_in":20469,"feed_emoji":"🤖","tokens_out":9150,"duration_ms":91969,"temperature":0.7,"pith_summary":"The paper tries to establish that precise 6DoF object manipulation demonstrations can be extracted automatically from ordinary egocentric video, without pre-recorded camera coordinates or depth sensors, and that this extracted data can train models to generate object trajectories from an action description. The authors build EgoTraj, a dataset of 28,497 trajectories with associated action descriptions, by combining temporal action localization, open-vocabulary segmentation, dense 3D point tracking, monocular depth estimation, and point-cloud registration. They then train vision- and point-cloud-based language models to emit discretized trajectory tokens, and show on a separate egocentric benchmark with motion-capture ground truth that the models generate valid, action-dependent trajectories. If correct, this turns the large body of already-recorded egocentric video into a scalable source of training data for text-conditioned robot manipulation.","feed_headline":"Egocentric video yields 28,497 object manipulation trajectories","feed_subtitle":"No pre-recorded camera coordinates needed: the auto-extracted data trains models that turn action text into valid 6DoF motions.","key_machinery":"The carrying mechanism is the extraction pipeline's chain of geometry estimates. Temporal action localization fixes the action span and object name; open-vocabulary segmentation isolates the object in the first frame; a dense 3D point tracker followed by monocular depth estimation produces an object point-cloud sequence; and point-cloud registration (RANSAC feature matching refined by colored ICP) projects every frame into the first frame's camera coordinates. For rotation, the pipeline computes the covariance matrix $H$ between the initial and current object point clouds, factorizes it as $H = U\\Sigma V^T$, and sets the rotation matrix to $R = V U^T$, assuming the object is rigid. On the generation side, each of the six pose coordinates is quantized into 256 bins and mapped to special tokens, so a vision-language model emits the trajectory autoregressively.","core_discovery":"The central claim is that a scalable, fully automatic pipeline can recover 6DoF object manipulation trajectories from a single moving egocentric camera, and that these noisy but large-scale trajectories are sufficient supervision for text-conditioned trajectory generation. Concretely, the paper extracts 28,497 trajectories from the Ego-Exo4D videos, covering 228 action verbs and 4,158 objects, using a four-stage framework, and then trains several language-model backbones by discretizing each pose dimension into 256 tokens and treating generation as next-token prediction. On the HOT3D benchmark, which supplies high-precision optical-marker trajectories, the resulting models produce valid trajectories, and the point-cloud-based model achieves the lowest displacement error and qualitatively different motions for different action verbs such as 'transfer' and 'stir'. The paper also reports that adding trajectory information improves action-description generation, especially verb similarity, which it takes as evidence that the extracted motions reflect the described manipulation.","pith_inferences":["The paper leaves the rotation labels unvalidated; a natural extension is to compare SVD-extracted rotations against ground truth on motion-capture clips and, if they are noisy, to train position-only models to see how much of the reported validity comes from translation alone.","The rigid-object filter excludes deformable objects, so the framework's generality hinges on extending tracking and pose recovery to clothes, rope, and other non-rigid items; that is a testable next milestone rather than a claim of the paper.","Because the models are trained on one video collection and evaluated on another, an obvious stress test is to apply the same pipeline to additional egocentric corpora and measure whether the transfer gap widens or narrows with dataset scale.","A direct probe of whether the action description is driving generation, rather than the initial visual state, would be to fix the image and vary the text; the paper's qualitative examples suggest the text matters, but it does not quantify this."],"forward_implications":["Automatically extracted trajectories transfer to a motion-capture benchmark, so large collections of everyday egocentric video can substitute for expensive laboratory demonstration recordings.","Because the pipeline needs no pre-recorded camera coordinates or depth hardware, it can be applied to most existing egocentric video datasets, not only purpose-built manipulation collections.","Trajectory-conditioned models generate more accurate action descriptions, with verb similarity rising by roughly ten percent, indicating the extracted motion encodes the action's semantics.","Sampling from the language model yields multiple plausible trajectories for one action description, matching the fact that a manipulation can be performed in several ways.","Scaling the automatically extracted training set improves displacement errors, implying that the data-harvesting approach has continued returns as more egocentric video becomes available."],"supporting_citations":[{"why":"Source of the training videos and action descriptions; all 28,497 EgoTraj trajectories are extracted from it.","marker":"[30]"},{"why":"Provides the test set with precise optical-marker 6DoF trajectories used to evaluate trajectory validity.","marker":"[3]"},{"why":"Open-vocabulary segmentation model that isolates the manipulated object from the object name in the first frame.","marker":"[76]"},{"why":"Dense 3D point tracker that produces the object point-cloud sequence later used for position and rotation recovery.","marker":"[96]"},{"why":"Monocular depth estimator whose depth maps convert each frame into the RGB-D point clouds used for tracking, registration, and model input.","marker":"[100]"},{"why":"Used for temporal action localization, object naming, rigidity filtering, and action-description annotation on the evaluation split.","marker":"[1]"},{"why":"Point-cloud language model backbone that achieves the best trajectory-generation results in the evaluation.","marker":"[97]"},{"why":"Image-language backbone used for image- and depth-conditioned trajectory generation and for captioning experiments.","marker":"[50]"},{"why":"Previous work that discretizes robot actions into tokens, the scheme the paper extends to 6DoF object poses.","marker":"[9]"}],"fun_headline_variants":["28,497 trajectories auto-mined from egocentric video","Egocentric video yields 28,497 6DoF manipulation paths","Auto-extracted egocentric trajectories drive text-to-motion","Single camera video generates action-conditioned 6DoF motions","Point-cloud model turns action text into valid 6DoF trajectories"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach depends on the assumption that the automatically tracked point clouds are clean and the objects rigid enough for the computed rotations to be close to the true rotations, and this assumption is never checked against ground truth.","fun_headline_variants_meta":{"raw":{"variants":["28,497 trajectories auto-mined from egocentric video","Egocentric video yields 28,497 6DoF manipulation paths","Auto-extracted egocentric trajectories drive text-to-motion","Single camera video generates action-conditioned 6DoF motions","Point-cloud model turns action text into valid 6DoF trajectories"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000306,"raw_usage":{"total_tokens":1733,"prompt_tokens":905,"completion_tokens":828,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":741}},"tokens_in":521,"tokens_out":828,"duration_ms":8495,"temperature":1.0,"reasoning_tokens":741,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:58:47.707445+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the extraction pipeline on a motion-capture egocentric dataset whose true 6DoF object poses are known, compute the per-frame geodesic distance between the extracted and ground-truth rotations, and check whether the extraction error alone is already as large as the prediction errors the paper reports; if it is, the rotation labels are too corrupted to support the claimed trajectory validity.","supporting_citations":[{"cited_title":"Spatialtracker: Tracking any 2d pixels in 3d space","cited_arxiv_id":null,"evidence_quote":"Dense 3D point tracker that produces the object point-cloud sequence later used for position and rotation recovery."},{"cited_title":"BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models","cited_arxiv_id":null,"evidence_quote":"Image-language backbone used for image- and depth-conditioned trajectory generation and for captioning experiments."}],"review_version":1}