{"id":"722b28f6-a2f6-4767-a747-b745071d163b","arxiv_id":"2508.02056","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"StarPose lifts 2D keypoints to 3D poses with an autoregressive diffusion process that conditions on historical pose predictions and physics-style constraints, reporting SOTA on Human3.6M and MPI-INF-3DHP.","lead":"StarPose is a new system that estimates 3D human poses from ordinary 2D video by using a diffusion model that builds on previously predicted poses. The authors report state-of-the-art accuracy and much smoother motion on two benchmarks, which could matter for animation, sports analysis, and autonomous driving.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training objective (Eq. 6) treats Dθ as a predictor of h_{k−1}, while inference (Eq. 8, Algorithm 1) uses the same Dθ as the noise predictor ε; without a reparameterization the described method cannot implement its own inference.","rationale":"The reader identified the same load-bearing weakness I would select: the dual role of Dθ as h_{k−1}-predictor in training (Eq. 6) and as ε-predictor in inference (Eq. 8, Algorithm 1) is never reconciled. This is not a matter of tuning or empirical disagreement; it is an internal inconsistency in the derivation. A direct re-derivation of Eq. 8 from Eq. 1 requires Dθ to equal the noise ε, while Eq. 6 requires Dθ to equal h_{k−1}. No mathematical bridge between these targets is supplied, and no separate output head is described. Because the reported SOTA numbers are the paper's central evidence, this flaw makes the method as written non-implementable. I also checked whether any other concern is more fundamental. The temporal-smoothness metrics are partly shaped by the explicit velocity and bone-length losses, so they are not independent evidence of learned temporal understanding, but this is secondary. The limitation section honestly notes failures under severe occlusion, which does not undermine the central argument. The paper has no machine-checked proofs or parameter-free derivations that would independently support the construction, so the internal inconsistency stands as the decisive issue. My recommendation is therefore to keep the reader's verdict unchanged: the manuscript should not be accepted in its current form.","tokens_in":26874,"tokens_out":4674,"duration_ms":57146,"concrete_test":"Clone the provided repository and inspect the training and inference code. First, check whether the training loss is implemented as MSE against h_{k−1} (Eq. 6) and whether inference is implemented as Algorithm 1. Then run a single-step numerical probe: sample a ground-truth h0, draw ε, compute hk via Eq. 1, and evaluate Dθ(hk, f2D, fhis, k). Compare the output to h_{k−1} (the training target) and to ε (the inference role). If the output matches ε and not h_{k−1}, the paper misreports the training objective; if it matches h_{k−1}, Eqs. 8 and Algorithm 1 are invalid; if it matches neither, both descriptions are inaccurate. If the released code instead contains a separate noise head or an explicit reparameterization, the reader's concern would be resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central issue is an internal inconsistency between training and inference, not a disagreement with the field. In Eq. 6, the loss is Ldiff = sum_k ||Dθ(hk, f2D, fhis, k) − h_{k−1}||², so the network is trained to output the previous noisy pose h_{k−1}. In Eq. 8, however, the clean pose is computed as h0|k = μ + (1/√α_k)(hk − μ − √(1−α_k) Dθ(...)). This formula is exactly the DDPM inversion h0 = (hk − √(1−α_k) ε)/√α_k if and only if Dθ is the noise predictor ε. Algorithm 1 then uses the same Dθ as ε again in line 4, h_{k−1} = √α_{k−1} h0|k + √(1−α_{k−1}) Dθ(...). Thus one network is asked to be both the h_{k−1} predictor (training) and the ε predictor (inference). These are different functions in the DDPM framework: h_{k−1} = μ + √α_{k−1}(h0−μ) + √(1−α_{k−1})ε is not recoverable from hk and k without knowing ε (or h0), while ε_θ is a different learned target. The paper gives no separate head, reparameterization, or derivation connecting the two. As written, the inference algorithm does not follow from the training objective, so the reported 29.9 mm MPJPE and 1.3 mm/s MPJVE cannot be attributed to the described method. This concern is load-bearing because the only evidence for the central claim is the experimental table, and the described procedure is not implementable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"StarPose proposes an autoregressive diffusion framework for monocular 3D human pose estimation. It conditions the denoising process on a Historical Pose Integration Module (HPIM) that fuses past 2D keypoints with previously predicted 3D poses, and it introduces a Spatial-Temporal Physical Guidance (STPG) energy function with four loss terms: 2D reprojection consistency, skeletal symmetry, bone length variance, and differential sequence variation. The method is evaluated on Human3.6M and MPI-INF-3DHP, reporting state-of-the-art MPJPE of 29.9 mm and 20.8 mm, respectively, together with very low MPJVE and ACC-ERR values. The paper claims that the autoregressive formulation and physical guidance jointly produce both higher accuracy and better temporal consistency than prior diffusion- and transformer-based methods.","tokens_in":27292,"tokens_out":9686,"duration_ms":103351,"significance":"If the method were implementable as described, the reported accuracy and temporal-smoothness numbers would constitute a substantial advance over recent diffusion-based pose estimators, and the proposed plug-and-play STPG module could be a useful addition to the literature. The paper contains extensive experiments, ablations, sensitivity analyses, and an efficiency comparison, and it provides a public code link. However, the central technical claim is undercut by a fundamental mismatch between the training objective and the inference update: the denoising network is trained to output the previous noisy pose h_{k-1} but is then used as the noise predictor epsilon at inference. Because this mismatch is in the core derivation, the reported results cannot currently be attributed to the described method.","major_comments":[{"comment":"The central derivation is internally inconsistent. Eq. (6) defines the training loss as Ldiff = sum_k ||Dtheta(hk, f2D, fhis, k) - h_{k-1}||^2, so Dtheta is trained to predict the previous noisy sample. However, Eq. (8) computes h0|k = mu + alpha_k^{-1/2}(hk - mu - (1-alpha_k)^{1/2}Dtheta(...)), which is the standard DDPM clean-pose estimator only if Dtheta is the noise epsilon. Algorithm 1 (lines 3-4) then reuses the same Dtheta as epsilon in the DDIM-style update h_{k-1} = alpha_{k-1}^{1/2} h0|k + (1-alpha_{k-1})^{1/2}Dtheta(...). These two parameterizations are not equivalent: with the forward process in Eq. (1), a network that outputs h_{k-1} is implicitly predicting a linear combination of hk and epsilon, not epsilon alone. The paper provides no separate head, no reparameterization, and no derivation connecting the two roles. Consequently, the inference procedure does not follow from the training objective, and the reported 29.9 mm MPJPE and 1.3 mm/s MPJVE cannot be attributed to the described method.","section":"III-B/III-C, Eqs. (6)/(8), Algorithm 1"},{"comment":"The temporal-consistency evidence is partly circular. The loss Ld in Eq. (15) directly penalizes squared inter-frame joint differences (h^t_{0|k} - h^{t-1}_{0|k})^2, and during inference this term is included in the energy function E (Eq. 16) whose gradient is used to update h_{k-1} (Algorithm 1 lines 5-7). The reported metrics MPJVE and ACC-ERR (Table III) measure exactly such inter-frame velocity and acceleration differences. The large reported gains (2.6 mm/s MPJVE and 3.7 mm/s^2 ACC-ERR) therefore reflect in part the explicit smoothing objective rather than a learned temporal model. The paper should include an ablation without Ld, and without STPG inference, to separate regularizer-driven smoothness from the autoregressive contribution.","section":"III-C, Eq. (15), Table III"},{"comment":"The ablation baseline appears weaker than the published method it is compared with. In Table VI, the authors' reproduction of DiffPose is 39.5 mm, whereas the original DiffPose paper reports 36.9 mm (as listed in Table II). Table VII then uses a 39.5 mm baseline to attribute a 9.6 mm total improvement to STPG and HPIM. If the baseline is intended to be DiffPose, this is not a faithful reproduction, and the component gains are inflated; if it is a different architecture, it should be described. Please clarify the baseline configuration and report the components on a baseline matching the published DiffPose setup.","section":"IV-F, Tables VI and VII"}],"minor_comments":[{"comment":"Algorithm 1 calls the denoising model with an undefined condition f_ST (lines 3-4), while the text and Eq. (8) use fhis. Please replace f_ST with fhis or define f_ST explicitly.","section":"Algorithm 1"},{"comment":"The step size rho_k, which is essential to the STPG inference update, is never specified in the implementation details or experiments. A concrete value or schedule should be reported for reproducibility.","section":"Algorithm 1 / Section IV-B"},{"comment":"Eq. (7) uses the same symbol h_{k-1} on both sides; please use a temporary variable such as tilde{h}_{k-1} for the uncorrected update before the energy-gradient correction.","section":"III-C, Eq. (7)"},{"comment":"The acronym 'SPTG' appears in the Introduction ('SPTG incorporates structural and kinematic constraints'); it should be 'STPG'.","section":"I, Introduction"},{"comment":"In Table V, StarPose uses an input length of f=81, whereas several compared methods use f=243; the influence of input sequence length on the comparison should be discussed, since longer contexts generally aid accuracy.","section":"Tables IV/V"}],"recommendation":"reject","confidential_remarks":"The central training/inference mismatch in Eqs. (6) and (8) is a load-bearing error: as written, the inference algorithm cannot be the inverse of the training objective, so the reported results are not attributable to the described method. Even if Eq. (6) were a typo and the model actually predicts noise, the manuscript would need a complete re-derivation, re-training, and re-validation; this is beyond a minor revision. I recommend rejection, with the possibility of resubmission if the parameterization is corrected and the experiments are re-run under a consistent formulation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick verdict: this paper has a genuinely new idea and very strong reported results, but the central diffusion derivation does not hold together. The training loss (Eq. 6) trains Dθ to output h_{k-1}, the previous noisy pose. The inference algorithm (Eq. 8 and Algorithm 1) uses the same Dθ as the noise predictor ε in the standard DDPM inversion h0|k = μ + (h_k − μ − √(1−α_k) Dθ)/√α_k. Those are different functions. No reparameterization or separate head is given to reconcile them. As written, you cannot reproduce the method from the description: the network is trained to do one thing and then used in inference to do another. This is a load-bearing flaw, not a quibble.\n\nThe paper also deserves credit where it's due. Autoregressive conditioning on historical predicted 3D poses inside a diffusion framework is a sensible direction, and the HPIM (GCN + attention over past 2D/3D poses) is a reasonable implementation. The experiments are extensive: Human3.6M and MPI-INF-3DHP, ablations of every component, a plug-and-play study, complexity analysis, and a code link. The reported numbers are excellent—2 mm MPJPE better than FinePose, and MPJVE/ACC-ERR that are roughly 3× better than prior diffusion methods.\n\nA second soft spot: the temporal smoothness gains are partly built into the objective. Ld (Eq. 15) directly penalizes inter-frame joint differences, and MPJVE/ACC-ERR measure exactly those differences. So those improvements are not independent evidence of temporal understanding; they are, to a degree, the loss itself. The ablation shows STPG-I (inference only) helps, which is interesting, but the circularity should be acknowledged.\n\nWho should read it? Anyone working on diffusion-based 3D HPE would benefit from its experimental protocol and the plug-and-play STPG study. But the central parameterization mismatch needs to be resolved before the results can be trusted.\n\nRecommendation: send it to peer review. The idea is novel and the experimental investment is substantial; a serious referee could push the authors to clarify the training/inference relationship. But in current form it should be rejected or returned for major revision, not accepted.\n\nMy take: would_cite false, reading_group maybe, serious_thinker no.","headline":"Novel autoregressive diffusion idea and strong results, but the training and inference parameterizations of Dθ are irreconcilable as written.","tokens_in":27880,"tokens_out":5419,"would_cite":false,"duration_ms":55855,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Autoregressive diffusion with physical guidance sets new 3D pose benchmarks","keywords":["3D human pose estimation","autoregressive diffusion","temporal consistency","physical guidance","historical pose integration","Human3.6M","MPI-INF-3DHP","spatial-temporal modeling"],"falsifier":"Train the denoising network exactly as described (Eq. 6) with a separate noise-prediction head for Eq. 8, or evaluate the clean-pose estimates $h_{0|k}$ produced by the shared network during training: if the reported MPJPE (29.9 mm on Human3.6M with CPN inputs) cannot be reproduced, or if the clean-pose estimates are so poor that the physical guidance terms are dominated by noise, then the inference procedure does not match the training objective as written.","tokens_in":26647,"feed_emoji":"🦴","tokens_out":5822,"duration_ms":53714,"temperature":0.7,"pith_summary":"StarPose claims that 3D human pose estimation from monocular 2D keypoints becomes more accurate and temporally consistent when the 2D-to-3D lifting is formulated as an autoregressive diffusion process: the denoising network receives, for each frame, the 2D pose context plus historical information built from previously predicted 3D poses, and its sampling steps are steered by spatial-temporal physical constraints (2D reprojection consistency, skeletal symmetry, bone-length stability, and differential motion smoothness). On the two standard benchmarks, the reported gains are an average MPJPE of 29.9 mm and P-MPJPE of 24.6 mm on Human3.6M with CPN 2D inputs, and 20.8 mm MPJPE, 98.9% PCK, and 82.5% AUC on MPI-INF-3DHP. The result matters because depth ambiguity and occlusion make single-frame lifting ill-posed, and most existing diffusion or transformer lifters treat each frame independently, ignoring both the temporal continuity of human motion and skeletal plausibility.","feed_headline":"StarPose: past 3D poses guide diffusion to record pose accuracy","feed_subtitle":"On Human3.6M, MPJPE drops to 29.9 mm and motion smoothness improves several-fold over prior diffusion lifters.","key_machinery":"The central object is the autoregressive reverse diffusion process of AutoPCD, defined by $h_{k-1}=D_\\theta(h_k, f_{2D}, f_{his}, k)$, where $f_{his}$ is built by HPIM from the past L predicted 3D poses and their 2D counterparts via a skeleton integration graph with GCN (short-distance) plus attention (long-distance) branches. The second machinery is the energy guidance of STPG: at each reverse step, a clean-pose estimate $h_{0|k}$ is formed from Eq. (8), four physical loss terms are evaluated on it, and the gradient $\\nabla_{h_k}E(c,h_{0|k})$ is subtracted from the updated sample (Algorithm 1). This guidance is what enforces anatomical and kinematic plausibility during both training and inference.","core_discovery":"The paper's central claim is that an autoregressive diffusion framework, which conditions every denoising step on the pose predicted for the preceding frame together with the 2D pose history, resolves depth ambiguity better than per-frame diffusion and deterministic transformers. The discovery has two parts. First, a Historical Pose Integration Module builds a skeleton graph over the past L 2D and 3D poses and combines GCN and attention outputs to produce a historical embedding $f_{his}$ that is concatenated with 2D context features as the denoising condition. Second, a Spatial-Temporal Physical Guidance (STPG) term is added both to training and to the inference-time reverse process as an energy gradient; it penalizes deviations from the 2D reprojection ray, left-right bone length asymmetry, bone length variance across a sliding window, and weighted joint velocity differences between adjacent frames. Together these components drive the reported improvements: 29.9 mm MPJPE / 24.6 mm P-MPJPE on Human3.6M (CPN inputs), 20.8 mm MPJPE on MPI-INF-3DHP, and 1.3 mm/s MPJVE / 1.6 mm/s² ACC-ERR.","pith_inferences":["The strong MPJVE/ACC-ERR numbers come largely from the differential sequence variation term, which smooths joint velocities by construction; a testable extension is to check how much of the accuracy gain would survive if the same term were added to a deterministic (non-diffusion) lifter.","Because the model feeds its own predictions back in, it is vulnerable to error accumulation; the paper's failure-case figure (severe occlusion) shows this, and an explicit test would be to measure how performance degrades as inference proceeds over long unedited video.","The reliance on camera intrinsics for reprojection consistency restricts deployment; the paper's learned-intrinsics variant ('Pred. Intr.') quantifies the drop, and a further extension would be to replace the reprojection term with a learned scale-invariant depth prior to remove the camera requirement entirely.","The shared-network assumption for $D_\\theta$ (predicting $h_{k-1}$ and serving as $\\epsilon$ in Eq. 8) is not standard; a simple ablation adding a separate noise head would show whether the reported gains depend on that conflation or are robust to it."],"forward_implications":["On Human3.6M with CPN 2D inputs, StarPose reports 29.9 mm MPJPE and 24.6 mm P-MPJPE, surpassing the previous diffusion SOTA (FinePose) by 2.0 mm; with ground-truth 2D inputs it reports 15.5 mm MPJPE.","On MPI-INF-3DHP it reports 20.8 mm MPJPE, 98.9% PCK, and 82.5% AUC, exceeding FinePose by 2.3 mm MPJPE.","Temporal smoothness improves sharply: 1.3 mm/s MPJVE and 1.6 mm/s² ACC-ERR, versus 3.9 mm/s and 5.3 mm/s² for the next-best diffusion method reported (KTPFormer).","STPG is training-free and plug-and-play: attaching it to DiffPose and D3DP lowers their Human3.6M MPJPE by 8.1% and 10.0% respectively, with FPS dropping less than 6%.","Ablations in the paper attribute 6.1 mm of the total 9.6 mm gain to HPIM and 3.2 mm to STPG (training plus inference), indicating the two components are complementary."],"supporting_citations":[{"why":"Supplies the GMM-based noisy pose distribution and the overall diffusion 2D-to-3D lifting formulation that StarPose extends with autoregressive conditioning.","marker":"[14]"},{"why":"Sets the previous state-of-the-art diffusion baseline (FinePose) that StarPose compares against and outperforms on both benchmarks.","marker":"[11]"},{"why":"Provides the multi-hypothesis aggregation and 2D reprojection baseline (D3DP) that StarPose improves upon and uses to test STPG as a plug-and-play module.","marker":"[15]"},{"why":"Defines the Human3.6M benchmark, training subjects, and evaluation protocol used for the main MPJPE and P-MPJPE results.","marker":"[29]"},{"why":"Defines the MPI-INF-3DHP benchmark used for the generalization results with ground-truth 2D inputs.","marker":"[30]"},{"why":"Provides the pre-trained MixSTE network used as the frozen 2D context encoder producing $f_{2D}$ for the denoising model.","marker":"[34]"},{"why":"Supplies the GraFormer GCN-attention backbone that the denoising model $D_\\theta$ is built from.","marker":"[61]"},{"why":"Provides the DDIM accelerated sampling that reduces the reverse diffusion steps from 50 to 5 during inference.","marker":"[63]"},{"why":"Supplies the CPN 2D keypoint detections used as inputs for the main Human3.6M evaluation.","marker":"[10]"}],"fun_headline_variants":["StarPose: past 3D poses guide diffusion to 29.9 mm MPJPE","History-aware diffusion cuts 3D pose error to 29.9 mm","Autoregressive diffusion remembers past poses for smoother 3D motion","StarPose: historical pose guidance steers diffusion for consistent lifting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The denoising network is assumed to act simultaneously as the predictor of the previous noisy sample $h_{k-1}$ (which is what its training loss optimizes) and as the noise predictor $\\epsilon$ in the clean-pose estimator $h_{0|k}$ used for energy guidance and sampling; in standard diffusion these are different roles, and the paper provides no separate head or reparameterization to reconcile them.","fun_headline_variants_meta":{"raw":{"variants":["StarPose: past 3D poses guide diffusion to 29.9 mm MPJPE","History-aware diffusion cuts 3D pose error to 29.9 mm","Autoregressive diffusion remembers past poses for smoother 3D motion","StarPose: historical pose guidance steers diffusion for consistent lifting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001787,"raw_usage":{"total_tokens":7119,"prompt_tokens":1095,"completion_tokens":6024,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":711,"completion_tokens_details":{"reasoning_tokens":5940}},"tokens_in":711,"tokens_out":6024,"duration_ms":42252,"temperature":1.0,"reasoning_tokens":5940,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T05:11:53.471338+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the denoising network exactly as described (Eq. 6) with a separate noise-prediction head for Eq. 8, or evaluate the clean-pose estimates $h_{0|k}$ produced by the shared network during training: if the reported MPJPE (29.9 mm on Human3.6M with CPN inputs) cannot be reproduced, or if the clean-pose estimates are so poor that the physical guidance terms are dominated by noise, then the inference procedure does not match the training objective as written.","supporting_citations":[{"cited_title":"Diff- pose: Toward more reliable 3d pose estimation,","cited_arxiv_id":null,"evidence_quote":"Supplies the GMM-based noisy pose distribution and the overall diffusion 2D-to-3D lifting formulation that StarPose extends with autoregressive conditioning."},{"cited_title":"Finepose: Fine-grained prompt-driven 3d human pose estimation via diffusion models,","cited_arxiv_id":null,"evidence_quote":"Sets the previous state-of-the-art diffusion baseline (FinePose) that StarPose compares against and outperforms on both benchmarks."},{"cited_title":"Diffusion-based 3d human pose estimation with multi- hypothesis aggregation,","cited_arxiv_id":null,"evidence_quote":"Provides the multi-hypothesis aggregation and 2D reprojection baseline (D3DP) that StarPose improves upon and uses to test STPG as a plug-and-play module."},{"cited_title":"Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments,","cited_arxiv_id":null,"evidence_quote":"Defines the Human3.6M benchmark, training subjects, and evaluation protocol used for the main MPJPE and P-MPJPE results."},{"cited_title":"Monocular 3d human pose estimation in the wild using IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 14 improved cnn supervision,","cited_arxiv_id":null,"evidence_quote":"Defines the MPI-INF-3DHP benchmark used for the generalization results with ground-truth 2D inputs."},{"cited_title":"Mixste: Seq2seq mixed spatio-temporal encoder for 3d human pose estimation in video,","cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained MixSTE network used as the frozen 2D context encoder producing $f_{2D}$ for the denoising model."},{"cited_title":"Graformer: Graph-oriented transformer for 3d pose estimation,","cited_arxiv_id":null,"evidence_quote":"Supplies the GraFormer GCN-attention backbone that the denoising model $D_\\theta$ is built from."},{"cited_title":"Denoising diffusion implicit models,","cited_arxiv_id":null,"evidence_quote":"Provides the DDIM accelerated sampling that reduces the reverse diffusion steps from 50 to 5 during inference."},{"cited_title":"Cascaded pyramid network for multi-person pose estimation,","cited_arxiv_id":null,"evidence_quote":"Supplies the CPN 2D keypoint detections used as inputs for the main Human3.6M evaluation."}],"review_version":1}