{"id":"7ff45889-1aaa-4a71-b274-04221ce4dd61","arxiv_id":"1908.06377","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using only 2D landmark labels, a student pose network trained with a soft NRSfM-derived loss achieves lower depth error than its NRSfM teacher and than prior weakly supervised methods.","lead":"This paper trains a 3D pose estimator using only 2D keypoint annotations, distilling a soft geometric loss from a Non-Rigid Structure from Motion teacher. On Human3.6M the student beats both its teacher and prior weakly supervised methods, matching a fully supervised baseline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing premise is the unvalidated 'oracle and fixed' camera assumption in Sec. 4.1; if Deep-NRSfM cameras are biased, the Eq. 13 subspace projection and the Eq. 14 loss evaluate the wrong depth geometry.","rationale":"I agree with the reader's identified weakest assumption. The derivation chain in Eqs. 6-14 is a sequence of affine transformations built on the teacher's camera matrices; if those cameras are inaccurate, every downstream quantity, including the depth subspace in Eq. 8, the projection in Eq. 13, and the knowledge-distilling loss in Eq. 14, is corrupted. The paper's only support is the assertion in Section 4.1 that the cameras are accurate, with no accompanying error analysis. This is not an attack on the method but a request for a sensitivity check that is inexpensive on H3.6M because ground-truth 3D poses are available for evaluation. Other concerns, such as missing code, omitted teacher details, and the exact match with the supervised baseline, are real but secondary; they do not directly challenge the internal logic of the loss construction. If the proposed camera-sensitivity test shows that replacing the teacher's cameras with ground-truth cameras leaves the results essentially unchanged, the central argument would be substantially supported. The verdict should remain conditional pending this test, so I do not propose a change to the reader's verdict.","tokens_in":13551,"tokens_out":6564,"duration_ms":69920,"concrete_test":"Compute ground-truth weak-perspective camera matrices M_gt(i) for the 35k-frame H3.6M training subset by fitting M to the known 3D poses and 2D projections, and report rotation/scale error of Deep-NRSfM's M(i) against M_gt(i). Then retrain the student with Eq. 14 using M_gt(i) in place of M(i), keeping the same dictionary, the same phi_nrsfm, and the same Eq. 15 regularizer weight, and compare training-set depth error and P-MPJPE with Table 1. If the results shift by more than a few mm or if camera errors are large on the ambiguous poses, the oracle/fixed assumption is the weak point; if the results are stable, the assumption is vindicated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, that Eq. 15 trains a 3D pose estimator from 2D labels alone and improves on the NRSfM teacher, depends on the teacher's camera matrices being correct. Section 4.1 states that 'the camera matrices from our modified Deep-NRSfM are accurate, thus we treat them as oracle and fixed,' but no error analysis supports this. The rotated dictionary B^(i) (Eq. 6), the xy/z split (Eq. 7), and the depth subspace S^(i)(z') = {phi : B_z^(i) phi = z'} (Eq. 8) all inherit the teacher's M^(i). If M^(i) is wrong, the z-coordinate in B_z^(i) is not the true image depth, the subspace is misplaced, and the projection in Eq. 13 plus the distillation loss in Eq. 14 reward or penalize depth hypotheses based on the teacher's camera error rather than on actual depth quality. This is especially dangerous because camera errors are likely correlated with precisely the ambiguous poses, such as a leg stretching toward or away from the camera, that the loss is meant to correct. The paper's training-set improvement over the teacher (Table 1) cannot be interpreted as the student correcting teacher depth unless the teacher's cameras are first shown to be reliable. This is a load-bearing, empirically testable assumption, not a mere implementation detail.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a weakly supervised 3D human pose estimation method that, given images and 2D landmark annotations only, trains a pose regression network without 3D ground truth, multi-view footage, or object-specific priors. The authors fit a modified Deep-NRSfM teacher on the 2D training set, obtaining a dictionary D, per-image camera matrices M^(i), and codes φ_(nrsfm)^(i). Treating D and M^(i) as fixed, they define, for a predicted depth z′, the affine subspace of codes whose reconstructed depth equals z′ (Eq. 8), define the distillation loss as the minimum NRSfM cost within that subspace (Eq. 10), and approximate it by the cost at the projection of the teacher code onto the subspace (Eqs. 12–14), claiming a convex upper bound. The student, an integral regression network, is trained with a 2D reprojection loss plus this distillation loss (Eq. 15). On Human3.6M, the method is reported to improve depth and pose error over the Deep-NRSfM teacher and over a weak supervision baseline, to outperform other weakly supervised methods while using less supervision, and to match a fully supervised baseline in MPJPE.","tokens_in":13878,"tokens_out":8854,"duration_ms":86280,"significance":"If the results hold, this is a useful contribution: it converts an NRSfM-style cost into a soft, differentiable supervisory signal for a CNN without hard depth labels, while retaining a geometric objective. The derivation of the subspace projection and the convex upper bound (Eqs. 8–14) is mathematically clean, and the comparison against the Weaksup-bs baseline isolates the effect of the proposed loss from the teacher and student architectures. The claim to use less supervision than prior weakly supervised methods is explicit and falsifiable. The main caveats are that the method's correctness depends on the teacher camera matrices being accurate, a premise that is asserted but not tested, and that the reported margins lack error bars and use a validation-set-selected hyperparameter. These issues are addressable and do not invalidate the core idea.","major_comments":[{"comment":"The premise that the Deep-NRSfM camera matrices are \"oracle and fixed\" is load-bearing but unvalidated. Because B_z^(i) is formed by rotating the dictionary with the teacher's M^(i) (Eq. 6), and the subspace S^(i)(z′) is defined as {φ : B_z^(i) φ = z′} (Eq. 8), any error in M^(i) changes which depth values are deemed consistent with a code. The projection in Eq. (13) and the distillation loss in Eq. (14) then evaluate depth hypotheses against the teacher's camera geometry rather than against true image depth. The paper's central evidence—that the student improves over the teacher (Tables 1 and 2)—cannot be read as the student correcting teacher depth unless camera accuracy is established. H3.6M provides ground-truth camera parameters; please report per-camera errors of the modified Deep-NRSfM against them, and add an ablation in which the loss is computed with ground-truth cameras (or with cameras from an alternative NRSfM method) to test sensitivity to this assumption.","section":"Sec. 4.1, Eqs. (6)–(8), (13)–(14)"},{"comment":"The headline numbers are single runs with the regularizer weight (0.3) selected on the same validation set used for reporting. Table 4 shows that the choice of weight matters for depth error: 0.01 gives 79.0 mm and 0.5 gives 76.7 mm, versus 73.1 mm for 0.3, against a 78.0 mm Weaksup-bs baseline. Thus the reported improvement is not fully robust to this selection for all metrics. Please report mean and standard deviation over multiple training runs, and either fix hyperparameters on a separate split or demonstrate that the conclusions hold across the reasonable range for depth error and MPJPE as well as PA-MPJPE.","section":"Sec. 5.4, Tables 2 and 4"},{"comment":"The teacher is described as a \"modified\" Deep-NRSfM with \"significant difference\" from the original (Table 1 footnote), and the camera and code estimation networks q_M and q_φ are deferred to the supplementary material. As the teacher defines the entire distillation target, this prevents the reader from assessing whether the reported gains are tied to the specific teacher modifications or to the proposed loss. Please include a precise description (or release code) of the modified Deep-NRSfM, and clarify how the teacher's hyperparameters (dictionary sizes, sparsity weight λ) were chosen. Without this, the comparison against other NRSfM methods in Table 1 is difficult to interpret.","section":"Sec. 5.3 and supplementary"}],"minor_comments":[{"comment":"There is a typo: \"vecorized\" should be \"vectorized\". Also, the norm in Eq. (9) is left unspecified, but Section 4.4 states that L2 is used for the reprojection error; please make this consistent earlier.","section":"Eq. (2)"},{"comment":"The acronym \"MUS\" is used in the text, but the cited reference [1] is titled \"Image collection pop-up\"; please provide the correct citation for MUS or remove the acronym.","section":"Sec. 5.3, references"},{"comment":"Table 1 is evaluated on the training set, Table 2 on the validation set, and Table 4 also reports validation-set numbers; the text should state this explicitly near each table to avoid confusion, especially because the regularizer weight is selected on the same validation set.","section":"Tables 1, 2, and 4"},{"comment":"The side note that an unlimited-capacity network would converge to the same depth as NRSfM is important: the method's improvement relies on the network's inductive bias. Please state this caveat more prominently, since it frames the interpretation of the experimental gains.","section":"Sec. 4.2"},{"comment":"Figure 2 is informative, but the caption and labels are dense; consider enlarging the figure or adding a legend for the orange lines, red dots, and yellow squares to make the geometry easier to follow.","section":"Fig. 2"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical claims hinge on a heavily modified teacher described only in the supplementary material and on validation-set tuning. I recommend requesting the implementation details or code as part of the revision. The teacher is from the same group as the prior Deep-NRSfM work; this is not a problem by itself, but the extent and effect of the modifications should be disclosed explicitly so the novelty of the distillation loss can be separated from the strength of the teacher. The lack of error bars is a further concern for the stated state-of-the-art comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The distillation loss in Eq. 14 is a real novelty: instead of regressing the teacher's hard depth labels, you tie the student's depth hypothesis to the NRSfM cost via a subspace of codes, and you get a tractable convex upper bound by evaluating the cost at the projection of the teacher code onto that subspace. That's clever, and the derivation (Eqs. 8-14) is clean. The ablation against Weaksup-bs isolates the gain, and Table 4 shows the L1 weight is not a knife-edge choice. Solid work.\n\nThe soft spots are real but not fatal. Section 4.1 asserts the modified Deep-NRSfM camera matrices are 'accurate' and fixes them as oracle, with no error analysis. The rotated dictionary B_z and the subspace projection in Eq. 13 inherit any camera bias. If the cameras are off in exactly the ambiguous poses that matter most—leg stretching toward or away from the camera—the loss may be rewarding the wrong depth geometry. This is testable: report camera error against ground-truth orientation, or show sensitivity to small camera perturbations. The paper doesn't. Second, reproducibility: the teacher architecture is only in the supplementary (which appears missing), no code, no error bars. Third, the abstract's claim of being 'more accurate than NRSfM methods' is supported only on the training set (Table 1); validation comparisons are against weak-supervision methods, not NRSfM. That's fine, but it's a softer claim than advertised.\n\nWho is this for? Anyone working on weakly supervised 3D pose, or distilling geometric cost functions into feedforward networks. The loss is general enough to apply to any dictionary-based NRSfM. It deserves a serious referee; the camera assumption needs empirical attention, and the supplementary must be restored, but the core idea is worth engaging.\n\nMy recommendation: send it to review. If it comes back with camera error analysis and reproducibility details, it could be a solid conference paper.","headline":"A genuinely new distillation loss for weakly supervised 3D pose, with a clean convex derivation, but the unvalidated 'oracle camera' assumption and missing reproducibility details keep it just short of fully convincing.","tokens_in":14434,"tokens_out":2710,"would_cite":true,"duration_ms":26438,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A pose estimator trained only on 2D landmarks can learn depth by distilling the NRSfM cost function instead of the teacher's depth output.","keywords":["weakly supervised 3D pose estimation","non-rigid structure from motion","knowledge distillation","2D landmark annotations","dictionary learning","deep image prior","Human3.6M","depth estimation"],"falsifier":"Keep the same training pipeline but perturb or randomize the teacher's camera matrices while holding the dictionary fixed; if the student's pose-aligned mean per-joint error on Human3.6M validation stays near 62.8 mm, the camera-fixity assumption is not load-bearing, while a large degradation would confirm it is.","tokens_in":13349,"feed_emoji":"🦴","tokens_out":9006,"duration_ms":73313,"temperature":0.7,"pith_summary":"The paper argues that a 3D pose estimator can be trained from images using only 2D landmark annotations, with no 3D labels, multi-view footage, or object-specific priors. The route is to distill knowledge from a non-rigid structure-from-motion (NRSfM) solver, not by regressing the teacher's depth predictions, but by using the teacher's own reconstruction cost as a soft loss: a depth hypothesis is scored by the minimum NRSfM cost over all codes consistent with that depth. Because a convolutional network's structure acts as a deep image prior, the student can correct the teacher's errors in ambiguous poses by associating image features. On Human3.6M the student reduces depth error from the teacher's 76.5 mm to 71.2 mm and reaches a validation mean per-joint position error (MPJPE) of 86.4 mm, the same as a fully supervised baseline, while using less supervision than other weakly supervised methods.","feed_headline":"With only 2D labels, pose network beats its 3D teacher","feed_subtitle":"Distilling the teacher's cost function lets image features fix depth errors and match fully supervised accuracy.","key_machinery":"The load-bearing construction is the split of the camera-rotated dictionary into $B_{xy}^{(i)}$ (2D projection) and $B_z^{(i)}$ (depth), together with the subspace $\\mathcal{S}^{(i)}(z') = \\{\\phi : B_z^{(i)}\\phi = z'\\}$ of codes consistent with a predicted depth $z'$. The loss is $\\tilde{L}^{(i)}(z') = \\|B_{xy}^{(i)}\\tilde{\\phi}(z') - w^{(i)}\\| + h(\\tilde{\\phi}(z'))$, where $\\tilde{\\phi}(z') = \\phi_{\\mathrm{nrsfm}}^{(i)} + (B_z^{(i)})^{\\dagger}(z' - B_z^{(i)}\\phi_{\\mathrm{nrsfm}}^{(i)})$ projects the NRSfM solution onto that subspace. This closed-form differentiable operator maps any predicted depth to the nearest code in the teacher's dictionary, so gradients from the teacher's cost reach the depth branch while the student remains free to choose depths supported by image features.","core_discovery":"The central claim is that the obstacle to using NRSfM as a teacher is the teacher's raw depth output, so the paper replaces hard depth regression with a loss that evaluates depth through the NRSfM objective itself. Given a network depth prediction $z'$, all dictionary codes reproducing that depth form a subspace $\\mathcal{S}^{(i)}(z')$, and the quality of a depth hypothesis is the minimum of reprojection error plus sparsity regularizer over that subspace. Because solving that constrained problem at every SGD step is impractical, the paper projects the NRSfM solution onto the subspace, obtaining a closed-form differentiable operator and a convex upper bound that still reflects 2D projection ambiguity. Training the integral-regression network with a 2D landmark loss for $(x,y)$ and this distilled loss for $z$ yields a student that reconstructs the training set more accurately than its NRSfM teacher (pose-aligned mean per-joint error 56.4 mm vs 73.2 mm; depth error 71.2 vs 76.5 mm), outperforms other weakly supervised methods, and matches the fully supervised baseline on validation mean per-joint position error (86.4 mm).","pith_inferences":["The method's reliance on fixed camera matrices suggests a natural extension the paper does not test: refining or marginalizing uncertain cameras instead of treating them as oracle.","If the deep image prior is the active ingredient, the advantage over the teacher should grow with image texture; ablating on textureless silhouettes would isolate that effect.","The same subspace-projection idea should transfer to other geometric self-supervised losses where a teacher is defined by an optimization cost rather than by network outputs.","The absence of object-specific priors implies transfer to non-human keypoint categories such as quadrupeds and hands, which the paper states as a motivation but does not demonstrate."],"forward_implications":["The method needs only 2D landmark annotations, so it removes the 3D-data bottleneck for training pose estimators on new object categories.","On Human3.6M, the distilled student reaches the same validation mean per-joint position error as a fully supervised network using the same pose-estimation architecture, while using no 3D labels.","The student improves on the teacher's training-set depth error, showing that the soft loss transfers NRSfM knowledge without copying its mistakes.","Because the loss is defined at the level of the NRSfM cost, the distillation recipe applies to any dictionary-learning NRSfM method and to any student network that outputs landmark depth."],"supporting_citations":[{"why":"Supplies the Deep-NRSfM teacher: the hierarchical sparse-coding dictionary, per-image codes, and camera matrices the distilled loss is built on.","marker":"[21]"},{"why":"Provides the integral-regression network used as the student pose estimator and the fully supervised baseline accuracy.","marker":"[36]"},{"why":"Weakly supervised baseline that learns from 2D projections but still uses 3D-generated synthetic poses; sets the comparison for supervision cost.","marker":"[12]"},{"why":"Weakly supervised 3D-interpreter baseline that requires external 3D supervision; comparison point for using less supervision.","marker":"[43]"},{"why":"Adversarial inverse-graphics baseline that requires 3D training data; comparison point.","marker":"[13]"},{"why":"Lifting-from-the-deep baseline that requires external 3D human models; comparison point.","marker":"[37]"},{"why":"Deep-image-prior argument used to explain why the convolutional student can correct the NRSfM teacher's ambiguous depths.","marker":"[39]"}],"fun_headline_variants":["2D landmarks only: pose net beats NRSfM teacher","Distill NRSfM's cost, not its depth, for better 3D pose","Weakly supervised 3D pose learned from NRSfM loss","No 3D data or multi-view: pose net matches full supervision","Pose net learns from NRSfM's cost, beats its teacher"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on assuming the NRSfM camera matrices are accurate and fixed, because the subspace and the distilled loss are built on them; if a camera is wrong, the loss evaluates depth against a misplaced projection.","fun_headline_variants_meta":{"raw":{"variants":["2D landmarks only: pose net beats NRSfM teacher","Distill NRSfM's cost, not its depth, for better 3D pose","Weakly supervised 3D pose learned from NRSfM loss","No 3D data or multi-view: pose net matches full supervision","Pose net learns from NRSfM's cost, beats its teacher"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001031,"raw_usage":{"total_tokens":4352,"prompt_tokens":966,"completion_tokens":3386,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":3299}},"tokens_in":582,"tokens_out":3386,"duration_ms":23765,"temperature":1.0,"reasoning_tokens":3299,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:47:41.942536+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Keep the same training pipeline but perturb or randomize the teacher's camera matrices while holding the dictionary fixed; if the student's pose-aligned mean per-joint error on Human3.6M validation stays near 62.8 mm, the camera-fixity assumption is not load-bearing, while a large degradation would confirm it is.","supporting_citations":[{"cited_title":"Deep Interpretable Non-Rigid Structure from Motion","cited_arxiv_id":"1902.10840","evidence_quote":"Supplies the Deep-NRSfM teacher: the hierarchical sparse-coding dictionary, per-image codes, and camera matrices the distilled loss is built on."},{"cited_title":"Integral human pose regression","cited_arxiv_id":null,"evidence_quote":"Provides the integral-regression network used as the student pose estimator and the fully supervised baseline accuracy."},{"cited_title":"Can 3d pose be learned from 2d projections alone? In Proceedings of the European Conference on Computer Vision (ECCV) , pages 0–0, 2018","cited_arxiv_id":null,"evidence_quote":"Weakly supervised baseline that learns from 2D projections but still uses 3D-generated synthetic poses; sets the comparison for supervision cost."},{"cited_title":"Single image 3d interpreter network","cited_arxiv_id":null,"evidence_quote":"Weakly supervised 3D-interpreter baseline that requires external 3D supervision; comparison point for using less supervision."},{"cited_title":"Harley, William Seto, and Katerina Fragkiadaki","cited_arxiv_id":null,"evidence_quote":"Adversarial inverse-graphics baseline that requires 3D training data; comparison point."},{"cited_title":"Lifting from the deep: Convolutional 3d pose estimation from a sin- gle image","cited_arxiv_id":null,"evidence_quote":"Lifting-from-the-deep baseline that requires external 3D human models; comparison point."},{"cited_title":"Deep image prior","cited_arxiv_id":null,"evidence_quote":"Deep-image-prior argument used to explain why the convolutional student can correct the NRSfM teacher's ambiguous depths."}],"review_version":1}