{"id":"9e13bcee-43d2-4259-8941-6ac3634dd82d","arxiv_id":"2604.28130","paper_version":3,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":8.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"MoCapAnything V2 presents the first end-to-end learnable Video-to-Pose and Pose-to-Rotation framework for monocular arbitrary-skeleton motion capture by conditioning on a reference pose-rotation pair.","lead":"The paper introduces the first fully end-to-end neural pipeline that predicts both 3D joint positions and joint rotations directly from monocular video for arbitrary skeletons. A smart generalist might care because it removes the non-differentiable inverse-kinematics bottleneck and claims 20x faster inference with lower rotation error on unseen skeletons.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Single reference pose-rotation pair may leave per-bone twist ambiguities unresolved for arbitrary skeletons","rationale":"The reader's weakest_assumption directly identifies the same ambiguity-resolution step as the load-bearing point; the concrete_test above would falsify or confirm whether that step holds for the arbitrary-skeleton case the paper targets.","tokens_in":1810,"tokens_out":346,"duration_ms":32484,"concrete_test":"Construct a minimal 3-bone kinematic chain with non-uniform bone lengths; supply rest pose + one reference (pose, rotation) pair; generate a target pose whose positions are reachable by at least two distinct rotation sets differing only in twist on the middle bone; feed both the reference and target positions into the described conditional network (or an analytical equivalent) and measure whether output rotations are unique up to <1° or exhibit >5° twist variance.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central technical claim rests on the assertion that one reference pose-rotation pair plus the rest pose fully anchors the coordinate system and converts the under-constrained pose-to-rotation map into a well-posed conditional regression problem. Section 3 (inferred from abstract) states this resolves missing axis conventions, yet provides no derivation showing uniqueness: given 3D joint positions, a bone direction constrains only two rotational degrees of freedom, leaving an axial twist that a single global reference may not pin down consistently across heterogeneous bone lengths and hierarchies. If residual twist freedom remains, the learned Pose-to-Rotation head can output rotations that satisfy positions but produce invalid animations, undermining both the ~10° error claim and the 6.54° unseen-skeleton result.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims to introduce the first fully end-to-end learnable framework for monocular motion capture on arbitrary skeletons. It replaces the standard factorized Video-to-Pose + analytical IK pipeline with jointly optimized learnable Video-to-Pose and Pose-to-Rotation stages. A reference pose-rotation pair from the target asset, supplied together with the rest pose, is used to anchor the rotation coordinate system and resolve pose-to-rotation ambiguities. Both stages share a skeleton-aware Global-Local Graph-guided Multi-Head Attention (GL-GMHA) module. Experiments on Truebones Zoo and Objaverse report rotation error reductions from ~17° to ~10° (6.54° on unseen skeletons) and ~20x faster inference than mesh-based methods.","tokens_in":2010,"tokens_out":684,"duration_ms":47642,"significance":"If the reference-pair conditioning truly converts the under-constrained rotation problem into a well-posed conditional regression task and the joint optimization yields the stated error reductions, the work would constitute a meaningful technical advance. It directly addresses the non-differentiability and ambiguity limitations of prior factorized pipelines, enables optimization for the final animation objective, and improves efficiency by avoiding mesh intermediates. The reported generalization to unseen skeletons and the GL-GMHA architecture for variable skeletons would be of practical interest to animation and computer vision communities.","major_comments":[{"comment":"Section 3 (method description): The central assertion that 'a reference pose-rotation pair from the target asset, which, together with the rest pose, not only anchors the mapping but also defines the underlying rotation coordinate system' and thereby turns rotation prediction into a 'well-constrained conditional problem' lacks any derivation or uniqueness argument. A 3D bone direction constrains only two rotational degrees of freedom, leaving an axial twist; it is not shown why a single global reference pair suffices to eliminate per-bone twist ambiguities consistently across heterogeneous bone lengths and hierarchies. This is load-bearing for both the claimed error reductions and the generalization result.","section":"Section 3"},{"comment":"Abstract / Experiments section: The reported quantitative improvements (~17° to ~10°, 6.54° on unseen skeletons) are presented without any description of error-bar methodology, number of evaluation runs, reference-pair selection protocol during training/testing, or basic dataset statistics. This makes it impossible to determine whether the joint end-to-end training, rather than the external reference input, is responsible for the gains.","section":"Abstract / Experiments"}],"minor_comments":[{"comment":"The abstract refers to 'Section 3' but the provided text contains no numbered sections; the full manuscript should ensure the method description is clearly labeled and cross-referenced.","section":"Abstract"},{"comment":"Notation for the reference pose-rotation pair, rest pose, and the conditioning mechanism should be introduced with explicit equations to support reproducibility.","section":"Method"}],"recommendation":"major_revision","confidential_remarks":"The reader's low soundness score is warranted by the absence of any derivation or experimental controls around the reference-pair mechanism. The manuscript appears to fit the journal scope, but the authors should be asked to clarify how the reference pair is obtained at inference time for truly arbitrary assets."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive review and for recognizing the potential technical contribution of the end-to-end framework. We address each major comment below and will revise the manuscript to incorporate the requested clarifications and derivations.","responses":[{"response":"We agree that the manuscript would be strengthened by a formal derivation. The reference pair supplies the complete rotation matrix (including twist) for a designated reference bone; combined with the rest pose, this fixes the local coordinate frame for that bone. Because the skeleton hierarchy and bone lengths are known, the fixed local axes propagate consistently to all descendant bones, removing per-bone twist freedom. In the revision we will add a new subsection in Section 3 containing (i) a coordinate-frame derivation showing uniqueness under the given conditioning and (ii) a brief discussion of how the same mechanism supports generalization to unseen skeletons whose hierarchies differ from the training set.","revision_made":"yes","referee_comment":"[Section 3] Section 3 (method description): The central assertion that 'a reference pose-rotation pair from the target asset, which, together with the rest pose, not only anchors the mapping but also defines the underlying rotation coordinate system' and thereby turns rotation prediction into a 'well-constrained conditional problem' lacks any derivation or uniqueness argument. A 3D bone direction constrains only two rotational degrees of freedom, leaving an axial twist; it is not shown why a single global reference pair suffices to eliminate per-bone twist ambiguities consistently across heterogeneous bone lengths and hierarchies. This is load-bearing for both the claimed error reductions and the generalization result."},{"response":"We acknowledge that the current manuscript omits these experimental details. In the revised version we will expand the Experiments section to report: error bars computed over five independent runs with different random seeds; the reference-pair selection protocol (a single fixed pair per skeleton, chosen once from the asset’s animation library and held constant for both training and testing); and basic dataset statistics (number of skeletons, total frames, average bone count, and train/test splits for Truebones Zoo and Objaverse). These additions will make the evaluation protocol transparent and allow readers to better assess the source of the observed gains.","revision_made":"yes","referee_comment":"[Abstract / Experiments] Abstract / Experiments section: The reported quantitative improvements (~17° to ~10°, 6.54° on unseen skeletons) are presented without any description of error-bar methodology, number of evaluation runs, reference-pair selection protocol during training/testing, or basic dataset statistics. This makes it impossible to determine whether the joint end-to-end training, rather than the external reference input, is responsible for the gains."}],"tokens_in":1633,"tokens_out":570,"duration_ms":46376,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point is that this work trains Video-to-Pose and Pose-to-Rotation end-to-end instead of stopping at joint positions and handing off to non-differentiable IK. They condition the rotation head on a rest pose plus one reference pose-rotation pair taken from the target asset, which they say supplies the missing coordinate conventions and turns the mapping into a well-posed conditional task. The shared GL-GMHA module handles local bone reasoning and global skeleton coordination, and they avoid mesh intermediates entirely.\n\nThat change produces the reported gains: rotation error drops from roughly 17° to 10°, reaches 6.54° on unseen skeletons, and inference runs about 20 times faster than mesh pipelines. Those numbers matter for anyone who needs to retarget to new rigs without per-asset IK tuning.\n\nThe soft spot is exactly the one flagged in the stress-test note. A 3D bone direction fixes only two rotational degrees of freedom; the axial twist around the bone remains free. One global reference pair may anchor the overall frame but does not automatically eliminate per-bone twist freedom when bone lengths and hierarchy vary. The abstract asserts that the reference plus rest pose defines the coordinate system, yet without an explicit uniqueness argument or ablation that isolates twist error, it is unclear whether the learned outputs stay animation-valid or simply satisfy position loss while allowing inconsistent twists. That gap directly affects how much weight to give the unseen-skeleton result.\n\nThe work is aimed at teams building production animation tools or real-time capture systems that currently rely on factorized pipelines. Readers who care about differentiable retargeting or skeleton-agnostic capture will find the conditioning idea and the graph attention design worth examining. The central departure from prior factorized methods is real, and the experiments use relevant datasets.\n\nI would send it to peer review. The joint-optimization claim is substantive enough to justify referee time, even if the ambiguity resolution needs tighter evidence in revision.","headline":"The paper replaces analytical IK with a learned, reference-conditioned Pose-to-Rotation stage for arbitrary skeletons, but the single-pair conditioning may leave axial twist under-constrained.","tokens_in":2505,"tokens_out":474,"would_cite":false,"duration_ms":35650,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"End-to-end learning resolves rotation ambiguities in arbitrary-skeleton motion capture from monocular video by conditioning on a reference pose-rotation pair.","keywords":["motion capture","end-to-end learning","arbitrary skeletons","monocular video","pose estimation","rotation prediction","inverse kinematics"],"falsifier":"Running the model on the same test sequences while deliberately omitting or mismatching the reference pair and measuring whether rotation error returns to the original ~17-degree baseline.","tokens_in":2729,"feed_emoji":"🎥","tokens_out":735,"duration_ms":26576,"temperature":0.7,"pith_summary":"Prior methods split the task into a video-to-pose network followed by an analytical inverse-kinematics step, but joint positions leave bone twists and other degrees of freedom undetermined. The paper replaces the non-differentiable IK stage with a learnable pose-to-rotation network whose input includes one reference pose-rotation pair taken from the target asset plus the rest pose. This single extra input anchors the local coordinate system and turns the otherwise ill-posed mapping into a supervised conditional task that can be trained jointly with the pose estimator. The resulting system predicts positions directly from video, skips mesh intermediates, and reports lower rotation error on both familiar and unseen skeletons together with substantially faster inference.","feed_headline":"End-to-end model learns rotations for any skeleton from video","feed_subtitle":"One reference pose-rotation pair resolves ambiguities that position-only pipelines left open, halving error and removing the mesh stage.","key_machinery":"A reference pose-rotation pair from the target asset, supplied together with the rest pose, that anchors the rotation coordinate system and converts the pose-to-rotation task into a well-constrained conditional prediction problem.","core_discovery":"The first fully end-to-end framework for arbitrary-skeleton motion capture jointly optimizes a Video-to-Pose stage and a Pose-to-Rotation stage; supplying one reference pose-rotation pair from the target asset together with the rest pose supplies the missing coordinate-system information and removes the ambiguities that positions alone cannot resolve.","pith_inferences":["The method suggests that any future monocular capture system for custom rigs will need an explicit mechanism to communicate the asset's local axis conventions rather than relying solely on joint locations.","Replacing the analytical IK solver with a learned conditional module opens the possibility of end-to-end fine-tuning on animation-specific losses such as foot sliding or contact preservation.","Because the reference pair is cheap to supply at inference time, the approach could be integrated into existing animation pipelines without retraining per character.","The reported twenty-fold speed-up relative to mesh-based pipelines indicates that skipping the surface representation stage may be the dominant factor for real-time deployment."],"forward_implications":["Joint optimization lets the network adapt to noisy pose predictions and directly minimize the final animation objective instead of an intermediate position loss.","Direct video-to-position prediction without mesh reconstruction removes a major source of latency and improves robustness to lighting or texture variation.","The same skeleton-aware attention module works across arbitrary bone counts and topologies because the reference pair normalizes the local frames.","Error on unseen skeletons drops to 6.54 degrees, indicating that the conditioning generalizes beyond the training distribution."],"fun_headline_variants":["Arbitrary-skeleton rotations learned end-to-end from monocular video","Reference pose-rotation pair eliminates rotation ambiguities from video","End-to-end framework skips mesh intermediates for skeleton capture","GL-GMHA module enables joint optimization of pose and rotation stages"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The ambiguity in the pose-to-rotation mapping can be resolved by supplying a single reference pose-rotation pair from the target asset together with the rest pose.","fun_headline_variants_meta":{"raw":{"variants":["Arbitrary-skeleton rotations learned end-to-end from monocular video","Reference pose-rotation pair eliminates rotation ambiguities from video","End-to-end framework skips mesh intermediates for skeleton capture","GL-GMHA module enables joint optimization of pose and rotation stages"]},"model":"grok-4.3","cost_usd":0.007725,"raw_usage":{"total_tokens":3493,"prompt_tokens":752,"num_sources_used":0,"completion_tokens":67,"cost_in_usd_ticks":77253000,"prompt_tokens_details":{"text_tokens":752,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2674,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":752,"tokens_out":67,"duration_ms":29786,"temperature":1.0,"reasoning_tokens":2674,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-01T08:27:01.747822+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Running the model on the same test sequences while deliberately omitting or mismatching the reference pair and measuring whether rotation error returns to the original ~17-degree baseline.","supporting_citations":[],"review_version":3}