{"id":"b74f83cf-a879-4f0b-b617-1d6ba62b1960","arxiv_id":"2504.19227","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A generic MLP-Mixer trained with local low-rank subset and occlusion losses reconstructs 3D non-rigid shapes from 2D keypoints, cutting S-Up3D reconstruction error by over 70%.","lead":"This paper shows that a generic MLP-Mixer network, trained with two unsupervised losses, can lift partially occluded 2D keypoints of non-rigid objects into 3D shapes. On the S-Up3D benchmark it reports reconstruction error of 0.0163, over 70% lower than the previous state of the art.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3's 70% margin may be an evaluation-protocol artifact: the paper aligns ALLRAP with a one-DOF depth shift but does not show the baselines are aligned identically; a shared-protocol re-evaluation is needed.","rationale":"The reader's weakest_assumption is about the low-rank validity of local keypoint subsets, which is an interesting scientific premise. However, the most load-bearing element of the paper is the quantitative comparison in Table 3: the abstract and introduction rest on reducing SOTA error by over 70%. If that comparison is not apples-to-apples, the central claim fails regardless of whether the local low-rank prior is sensible. The paper itself flags a protocol difference in the metric paragraph by saying ALLRAP is evaluated 'up to one degree of freedom' and that, unlike prior work, it does not search over the z sign. It does not say how the baseline numbers were obtained under the same conditions. A large margin is inherently suspicious when the metric definition differs between the proposed method and the baselines, especially when code is not yet released. I therefore recommend keeping the CONDITIONAL verdict, with the condition being a shared-protocol re-evaluation or code release that reproduces the baseline numbers under the same alignment. This is a concrete, checkable condition rather than a speculative objection to the loss formulation.","tokens_in":10525,"tokens_out":10262,"duration_ms":119588,"concrete_test":"Obtain or re-run predictions from C3DPO, PAUL, DeepNRSfM++, and e2eMulti on the S-Up3D test split, then apply exactly the same evaluation protocol used for ALLRAP: per-sample mean-depth (z) offset removal, the same set of included keypoints including occluded ones, and no z-sign search. In parallel, evaluate ALLRAP under the baselines' likely protocol (full Procrustes alignment and best-of-two z signs). If ALLRAP retains a >50% margin over the best baseline under both protocols, the concern is resolved; if the margin drops below, say, 30% in either direction, the quantitative central claim is not robust and the paper should be revised or re-verified.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central quantitative claim is the S-Up3D result in Table 3: ALLRAP at MPJPE 0.0163 versus 0.057 for the next best method, a >70% reduction. For this claim to be meaningful, all methods must be evaluated under the same alignment protocol and on the same test keypoints. The paper states in Section 5: 'On S-Up3D we calculate the MPJPE up to one degree of freedom, as the z values can only be reconstructed up to a constant. Unlike [20, 28, 29], we don’t calculate MPJPE twice, once with z and once with -z...' This explicitly signals a protocol difference, but the paper does not state whether the baseline numbers in Table 3 were recomputed under the same one-DOF protocol or quoted from the original publications with their own alignments. If the baselines used different alignment degrees of freedom (for example, full Procrustes alignment, or no depth-offset removal, or a z-sign search), the reported gap could be substantially inflated. The 'one degree of freedom' removal is necessary for orthographic depth, but it must be applied identically to every method, and occluded keypoints must also be handled identically. Without code or a shared-protocol re-evaluation, the headline 70% improvement is not yet established as a method effect rather than a metric effect.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ALLRAP, an unsupervised 2D-3D lifting method for non-rigid objects. It trains a generic MLP-Mixer network with two unsupervised losses: a subset loss that encourages local neighborhoods of keypoints to be approximately low-rank after rigid alignment, and an occlusion loss that weakly enforces a negative correlation between visibility and predicted depth. On the S-Up3D benchmark the method reports an MPJPE of 0.0163, more than 70% lower than the next best baseline, with a standard deviation of 0.0014 over five seeds. The paper also reports one-shot reconstruction results on single sequences from DeformingThings4D and ZJU-MoCap, with lower errors than C3DPO, PAUL, and DeepNRSfM++, and it includes ablations on network architecture and subset selection.","tokens_in":10814,"tokens_out":9073,"duration_ms":90358,"significance":"If the S-Up3D result holds under a shared evaluation protocol, the contribution is significant: it demonstrates that a generic high-capacity network can outperform specialized low-rank architectures when the geometric prior is encoded in the loss, and the local low-rank prior is a compelling alternative to global low-rank constraints. The method is fully unsupervised, handles occluded keypoints, and the S-Up3D result is averaged over five seeds with a small standard deviation. The ablation tables are informative and the framing of the method as matrix inpainting is clear. However, the headline improvement rests on an alignment protocol that is not shown to be identical across methods, and the one-shot comparisons are under-specified; these issues must be resolved before the central claims are established.","major_comments":[{"comment":"The claim of a greater-than-70% reduction on S-Up3D is not yet supported because the text states that MPJPE is computed 'up to one degree of freedom' and that, unlike [20,28,29], the authors do not test both z and -z, but it does not state whether the baseline numbers in Table 3 were recomputed under this same protocol or quoted from the original papers with their own alignment conventions. If the baselines were aligned with full Procrustes, a z-sign search, or no depth-offset removal, the reported gap could be substantially inflated. Please recompute all baselines under the identical one-DOF z-offset alignment, on the same test keypoints, with the same handling of occluded points, and report the alignment details for every method.","section":"Section 5, Dataset Metrics; Table 3"},{"comment":"The one-shot perspective experiments are underspecified. Section 3 says the perspective case is handled 'along the camera rays', but no equations define how the network input, the predicted coordinates, the subset-loss scaling, or the occlusion loss are expressed in ray coordinates. It is also not stated whether the baselines C3DPO, PAUL, and DeepNRSfM++ were given exactly the same preprocessed inputs (keypoints with visibility above 30%, farthest-point downsampled to 100 keypoints) and the same scale-alignment protocol (centering plus regression scale). Table 4 reports a single run per sequence with no variance; please provide multiple seeds, standard deviations, and a detailed description of the identical protocol applied to all methods.","section":"Section 5, DeformingThings4D and ZJU-MoCap; Table 4"},{"comment":"The nearest-neighbor subset selection bootstraps the grouping of keypoints from the network's own 3D predictions, so the low-rank loss is applied to neighborhoods that the model itself chooses. The paper validates this empirically in Table 2, but it does not check whether the selected neighborhoods correspond to semantically coherent parts or to artifactually tight clusters created by the loss. Please provide an analysis such as the overlap of selected neighborhoods with ground-truth body parts on S-Up3D, or a comparison with fixed part-based neighborhoods, and discuss the behavior when the network's initial predictions are poor.","section":"Section 3.3.1, Subset selection"}],"minor_comments":[{"comment":"The inpainting rule 'we will only ever use the parts of the outputs that are not already available in the input' is not explicit; please state precisely that visible (x,y) coordinates are taken from the input while occluded (x,y) coordinates and all z coordinates are taken from the network output.","section":"Section 3.1"},{"comment":"The symbol n is used without a formal definition; please specify that the sum runs over the nonzero singular values of the reshaped residual matrix.","section":"Equation (8)"},{"comment":"The cosine similarity is undefined when either v_c or z_c has zero norm; please add an epsilon or a fallback for degenerate batches.","section":"Equation (9)"},{"comment":"The notation in the first two columns such as '8 ·' and '· 8' is not self-explanatory; please define 'Random' and 'NN' in the caption and clarify the entry format.","section":"Table 2"},{"comment":"The caption contains a typo: 'Errors are show in using' should read 'Errors are shown using'.","section":"Figure 4"},{"comment":"Reference [3] is listed as 'Anon. ... Under submission'; please provide the actual citation if the work is published or identifiable.","section":"References"},{"comment":"The conclusion states that the implementation will be open sourced; please include a link or supplement in the revision to support reproducibility.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong candidate if the authors can confirm that the S-Up3D baselines were evaluated under the identical one-DOF alignment protocol. The evaluation protocol ambiguity is the main risk to the headline claim. Given the large reported margin, an independent re-evaluation of the baselines is advisable. The one-shot comparisons also need clarification of the ray-based formulation and preprocessing controls."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know about this paper because it makes a clean, surprising empirical claim: a generic MLP-Mixer trained with a loss that encourages low-rank structure only on local keypoint subsets beats global low-rank deep NRSfM methods by over 70% on S-Up3D. I read the math and the protocol details, and I'm fairly convinced. The stress-test note worried that the comparison is unfair because ALLRAP uses a one-DOF depth alignment while baselines used a more permissive mirror-flip search. That worry doesn't hold up: the paper explicitly says baselines computed MPJPE twice (z and -z) and ALLRAP does not. So ALLRAP is held to a stricter protocol; the gap is not inflated by this.\n\nWhat's genuinely new is the formulation of the prior as a training-time loss: select neighborhoods using the network's own predictions, align them, and minimize the log-Gramian determinant of the residual. The ablation in Table 2 is the key evidence: global low-rank gives 0.0308, while local neighborhoods of size 32 give 0.0191 on the same validation split. The occlusion loss is a nice touch and plausibly explains why they can avoid the flip search. The MLP-Mixer result is also a useful data point: 0.24M parameters beat 6.8M MLPs.\n\nThe soft spots are in the one-shot experiments. The perspective-camera extension is described in one sentence ('along the camera rays') and never specified. In DeformingThings4D and ZJU-MoCap, they filter keypoints and downsample to 100, but don't say how baselines were handled. Those results are single runs, no variance bars. This is not fatal, but it is underspecified. Also, code is promised but not available, which makes the S-Up3D result harder to fully trust. And the comparison misses several recent deep NRSfM papers (e.g., the 2024 sequence-based methods they cite but don't benchmark). The weakest conceptual assumption—that selected subsets are indeed locally low-rank—is supported by the ablations, so I don't see it as a flaw.\n\nMy verdict: this deserves a serious referee. The central claim is important and likely correct. The paper needs a clearer description of the perspective parameterization, a shared pre-processing protocol for the one-shot comparisons, and ideally a code release, before publication. I would bring it to reading group.\n\nRecommendation for the editor: send to peer review. If revisions address the above, this is a strong CVPR/ICCV-type paper.","headline":"The local-low-rank loss story is real and the S-Up3D improvement is not a metric artifact; the one-shot experiments and missing code are the soft spots.","tokens_in":11355,"tokens_out":4925,"would_cite":true,"duration_ms":53998,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Applying low-rank constraints to local keypoint neighborhoods cuts non-rigid 3D reconstruction error by over 70%.","keywords":["2D-3D lifting","non-rigid structure from motion","unsupervised learning","low-rank constraints","MLP-Mixer","occlusion","local shape priors","one-shot reconstruction"],"falsifier":"Train the same ALLRAP pipeline on a deformable object with locally independent motion — for example, a cloth or flag where nearby points move with many independent degrees of freedom even within small neighborhoods — and measure MPJPE against a global low-rank baseline. If the local subset loss does not degrade relative to baselines on such data, the claim that local low-rankness is the active prior would be falsified.","tokens_in":10320,"feed_emoji":"🧍","tokens_out":8947,"duration_ms":72666,"temperature":0.7,"pith_summary":"Non-rigid 3D reconstruction from 2D keypoints is ill-posed, especially under occlusion, and classical deep solutions embed global low-rank shape priors directly into the network architecture. This paper argues that a generic, high-capacity model trained with an unsupervised loss can outperform those specialized architectures, provided the low-rank prior is applied locally — to automatically selected neighborhoods of keypoints — rather than to the whole shape. The claimed result is a mean per-joint position error of 0.0163 on the S-Up3D benchmark, more than 70% below the best previous baseline, with a standard deviation of 0.0014 across five seeds. If true, the geometric knowledge needed for lifting lives in the loss function, not in a bespoke network design, which simplifies training and extends naturally to one-shot video reconstruction.","feed_headline":"Local low-rank losses cut 3D pose error by over 70%","feed_subtitle":"A generic MLP-Mixer trained with local low-rank and occlusion losses tops specialized 3D reconstruction networks.","key_machinery":"The load-bearing mechanism is the subset loss: for each batch, it selects local neighborhoods of keypoints (by k-nearest neighbors in the predicted 3D space, or randomly), removes translation, aligns them to a batchwise mean shape via SVD and an optimal rigid alignment, and then penalizes the log Gramian determinant of the residual error matrix. Minimizing this log-volume of non-rigid variation pushes the network to predict body parts whose deformation is nearly low-rank. A second loss, the occlusion loss, clamps the cosine similarity between centered visibility and depth vectors at -0.05, resolving the depth mirror flip so the model does not suffer from z-flip ambiguity. The network itself is a generic MLP-Mixer with no built-in low-rank layers; the geometric prior lives entirely in the loss.","core_discovery":"The central claim is that for non-rigid objects with a semi-dense set of keypoints, local shape coherence is a stronger and more trainable prior than global low-rankness. The paper proposes ALLRAP, a method that trains an MLP-Mixer — a transformer-like network without attention — to directly predict 3D coordinates from partially occluded 2D keypoints, using two unsupervised batch losses. The subset loss selects local neighborhoods of keypoints, aligns them across the batch by rigid rotations, and penalizes the log Gramian determinant of the residual error, i.e., the log-volume of non-rigid deformation left after alignment. The occlusion loss enforces a weak negative correlation between visibility and depth to fix the depth flip ambiguity. The paper reports that this combination reduces the state-of-the-art MPJPE on S-Up3D by over 70% and also produces strong one-shot reconstructions on animal and human motion sequences.","pith_inferences":["The same local low-rank loss could be applied to dense mesh vertices or point clouds rather than sparse keypoints, potentially lifting continuous surfaces instead of discrete joints; this is an extension beyond the paper's experiments.","Since the prior is entirely in the loss, the MLP-Mixer could be replaced by other high-capacity architectures, and the paper's reported gains would then be attributable to the loss, not to the mixer; the paper does not test this substitution.","If the local low-rank premise is category-agnostic, a model trained on human poses could be fine-tuned on animal data with few examples; the one-shot animal results suggest this may work, but cross-category transfer is not reported.","The batch requirement B>1 means the method cannot lift a single isolated observation at training time; distilling the batch-trained network into a single-frame student would be a natural way to deploy it, a step the paper leaves implicit."],"forward_implications":["On the S-Up3D benchmark, ALLRAP reaches MPJPE 0.0163, a reduction of more than 70% over the best previous baseline, averaged over five seeds with standard deviation 0.0014.","A deliberately small MLP-Mixer, Mini ALLRAP, still beats all prior methods, implying the loss function rather than model scale carries the geometric prior.","In one-shot settings on single video sequences from ZJU-Mocap and DeformingThings4D, ALLRAP outperforms the baselines without any sequence-specific modules.","The occlusion loss removes the depth mirror-flip ambiguity, so the error can be reported once rather than as the minimum over z and -z.","Because subset selection uses K-nearest neighbors in the predicted 3D space, no skeleton or semantic part annotation is needed; the method applies to arbitrary tracked keypoints."],"supporting_citations":[{"why":"provides the SVD factorization used to compute the batchwise mean shape and residual.","marker":"[24]"},{"why":"introduces the low-rank shape basis prior that the subset loss localizes to neighborhoods.","marker":"[4]"},{"why":"supplies the MLP-Mixer architecture that maps keypoint tokens to 3D coordinates.","marker":"[23]"},{"why":"canonical-pose baseline that the S-Up3D and one-shot experiments compare against.","marker":"[20]"},{"why":"deep non-rigid SfM baseline that must be beaten in the same benchmarks.","marker":"[29]"},{"why":"autoencoder-based unsupervised lifting baseline used in comparisons.","marker":"[28]"},{"why":"defines the S-Up3D dataset used for the main benchmark.","marker":"[16]"},{"why":"provides the DeformingThings4D sequences for one-shot reconstruction.","marker":"[18]"},{"why":"provides ZJU-Mocap sequences used for one-shot reconstruction.","marker":"[21]"}],"fun_headline_variants":["Local priors beat global: unsupervised 3D lifting down 70%","MLP-Mixer with local losses: 70% less 3D shape error","70% error cut: local constraints for unsupervised 3D","Local low-rank beats global: 70% better 3D from 2D","Unsupervised local priors slash non-rigid 3D error by 70%+"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The subset loss assumes that automatically selected local neighborhoods of keypoints correspond to coherent body parts whose 3D deformation is approximately low-rank after rigid alignment; if real deformations in a neighborhood have many independent degrees of freedom, the loss imposes a wrong prior.","fun_headline_variants_meta":{"raw":{"variants":["Local priors beat global: unsupervised 3D lifting down 70%","MLP-Mixer with local losses: 70% less 3D shape error","70% error cut: local constraints for unsupervised 3D","Local low-rank beats global: 70% better 3D from 2D","Unsupervised local priors slash non-rigid 3D error by 70%+"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001202,"raw_usage":{"total_tokens":4915,"prompt_tokens":871,"completion_tokens":4044,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":3937}},"tokens_in":487,"tokens_out":4044,"duration_ms":29343,"temperature":1.0,"reasoning_tokens":3937,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:58:33.425992+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same ALLRAP pipeline on a deformable object with locally independent motion — for example, a cloth or flag where nearby points move with many independent degrees of freedom even within small neighborhoods — and measure MPJPE against a global low-rank baseline. If the local subset loss does not degrade relative to baselines on such data, the claim that local low-rankness is the active prior would be falsified.","supporting_citations":[{"cited_title":"Shape and motion from image streams under orthography: a factorization method","cited_arxiv_id":null,"evidence_quote":"provides the SVD factorization used to compute the batchwise mean shape and residual."},{"cited_title":"Recovering non-rigid 3d shape from image streams","cited_arxiv_id":null,"evidence_quote":"introduces the low-rank shape basis prior that the subset loss localizes to neighborhoods."},{"cited_title":"C3DPO: Canonical 3d pose networks for non-rigid structure from motion","cited_arxiv_id":null,"evidence_quote":"canonical-pose baseline that the S-Up3D and one-shot experiments compare against."},{"cited_title":"Deep nrsfm++: Towards unsupervised 2d-3d lifting in the wild","cited_arxiv_id":null,"evidence_quote":"deep non-rigid SfM baseline that must be beaten in the same benchmarks."},{"cited_title":"Paul: Procrustean au- toencoder for unsupervised lifting","cited_arxiv_id":null,"evidence_quote":"autoencoder-based unsupervised lifting baseline used in comparisons."},{"cited_title":"Black, and Peter V","cited_arxiv_id":null,"evidence_quote":"defines the S-Up3D dataset used for the main benchmark."},{"cited_title":"4dcomplete: Non-rigid motion esti- mation beyond the observable surface","cited_arxiv_id":null,"evidence_quote":"provides the DeformingThings4D sequences for one-shot reconstruction."}],"review_version":1}