{"id":"54ad04c3-be77-4c5c-a810-c0333c574ddf","arxiv_id":"2412.00837","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A family-aware Transformer with supervised contrastive learning and a diffusion-generated synthetic dataset achieves state-of-the-art 3D animal pose and shape estimation.","lead":"AniMer introduces a Transformer-based model that estimates 3D pose and shape of quadruped animals from a single image, together with a new synthetic dataset made by a diffusion model. On Animal3D and the Animal Kingdom benchmark, the method reports clear improvements over prior CNN and transformer baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The CtrlAni3D 3D labels are verified only by 2D silhouette consistency, not by 3D joint/shape alignment; since the OOD Animal Kingdom benchmark reports no 3D metrics, the claimed pose-and-shape gains rest on an unvalidated synthetic supervision.","rationale":"The reader's weakest_assumption already identified the unverified 3D alignment of CtrlAni3D labels, and my analysis agrees that this is the most load-bearing point. I mark agreement as 'partial' because I sharpen the issue in one respect: the Animal Kingdom OOD benchmark reports only 2D metrics, so the claimed OOD improvement cannot validate 3D pose or shape accuracy at all. The concern is not that the authors are dishonest; it is that the evidence presented is insufficient to rule out a plausible failure mode. ControlNet is known to follow 2D structural conditions (depth/mask) while sometimes deviating from the precise 3D articulation of the conditioning mesh, and the paper's own filtering procedure explicitly checks 2D cycle consistency and manual silhouette alignment, not 3D joint or shape alignment. If the labels were systematically 3D-biased, training on CtrlAni3D could improve 2D projections (thereby boosting Animal Kingdom PCK/AUC) while teaching incorrect geometry; the CtrlAni3D test scores would be inflated by the shared generation bias, and the real-benchmark gains on Animal3D would be the only reliable evidence. The proposed multi-view consistency test would settle this directly: if the same SMAL mesh explains independently generated views of the same pose, the 3D labels are trustworthy; if not, the central claim is compromised. I did not select the AniMer-b definition inconsistency (main text Sec. 5.2 vs. Supplementary Table 9) as the primary concern because it affects reproducibility but does not directly threaten the central empirical claim. Likewise, the lack of error bars is a reporting weakness rather than a specific false assumption. The paper has genuine independent support: strong gains on the real Animal3D benchmark, consistent ablations, and comparison against HMR2.0 adapted to the same training protocol. These make the central claim plausible, but the unverified 3D alignment of the largest 3D supervision source is precisely the kind of assumption that should gate acceptance. The appropriate verdict remains CONDITIONAL: the work should be accepted only if the authors release the dataset and code, report the multi-view or otherwise independent 3D validation, and clarify the AniMer-b inconsistency.","tokens_in":17255,"tokens_out":5574,"duration_ms":53480,"concrete_test":"Generate a multi-view consistency test for 200 randomly selected CtrlAni3D poses: for each conditioning SMAL mesh, render depth and mask from two additional camera viewpoints, generate images with the same ControlNet pipeline and text prompts, and run both the trained AniMer and an independent SMAL-fitting optimizer (using, e.g., a SuperAnimal 2D keypoint detector not trained on CtrlAni3D) jointly over the two views to recover a shared SMAL mesh. Compare the recovered mesh and joints to the original conditioning SMAL parameters after Procrustes alignment. If the average PA-MPJPE/PA-MPVPE of this multi-view reconstruction exceeds the corresponding Animal3D test errors (80.4/85.7) by a meaningful margin, the CtrlAni3D labels are not 3D pixel-aligned and the reported synthetic-data gains are suspect. If the errors are comparable, this concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (Abstract; Sec. 5.2, Table 1) is that AniMer outperforms prior methods on 3D benchmarks (Animal3D, CtrlAni3D) and on out-of-distribution Animal Kingdom. The load-bearing assumption is that CtrlAni3D images are pixel-aligned to the SMAL meshes in 3D. The generation pipeline (Sec. 4, Fig. 3) conditions ControlNet only on rendered SMAL depth and mask images, and the filtering step uses SAM2 mask IoU > 0.95 plus manual review. This verifies 2D silhouette consistency, not that the generated image actually depicts the conditioning SMAL mesh's 3D pose and shape. ControlNet can satisfy mask/depth conditions while altering limb foreshortening, joint angles, or body shape, especially under the diverse poses, camera truncation, and text-prompt variations described in Sec. 4. If this occurs, the 8,277 CtrlAni3D training images carry biased 3D supervision. Because the CtrlAni3D test set is generated by the same pipeline, its reported PA-MPJPE/PA-MPVPE (44.1/47.6 in Table 1) are not independent evidence of 3D accuracy. Animal Kingdom provides only 2D metrics (AUC, PCK), so the OOD gains do not measure 3D pose/shape accuracy. The one independent 3D signal, the Animal3D improvement with CtrlAni3D (Supplementary Table 7: PA-MPJPE 82.6 to 80.4, PA-MPVPE 88.4 to 85.7), is encouraging but does not quantify the label bias. The manuscript's own Sec. 4 acknowledges filtering by human review and shows failure cases, but no 3D consistency check is described. Thus the assumption that CtrlAni3D labels are 3D pixel-aligned remains unsupported, and the headline claim is only as strong as that assumption.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents AniMer, a Transformer-based architecture for estimating SMAL pose and shape parameters of quadruped animals from a single image. The method uses a ViT encoder with a family-aware supervised contrastive loss on a class token, followed by a Transformer decoder that directly regresses SMAL parameters. The authors also introduce CtrlAni3D, a synthetic dataset of roughly 10k images generated with ControlNet conditioned on rendered SMAL depth and mask maps, with semi-automated filtering based on SAM2 silhouette IoU and manual review. Training aggregates existing 2D and 3D animal datasets plus CtrlAni3D (41.3k images total). The paper reports improvements over HMR, WLDO, and HMR2.0 on Animal3D, on a CtrlAni3D test split, and on the out-of-distribution Animal Kingdom benchmark, along with ablations for the dataset and the contrastive loss.","tokens_in":17711,"tokens_out":5706,"duration_ms":49243,"significance":"If the claims hold, the paper makes a meaningful step for animal mesh recovery: it demonstrates that a high-capacity Transformer backbone together with a large, multi-species training set yields substantial gains over prior CNN-based methods, and it contributes a scalable synthetic-data generation pipeline that is likely useful beyond this specific task. The CtrlAni3D dataset itself is a potentially valuable community resource, and the family-aware contrastive design is a sensible way to exploit taxonomic structure. The evidence is partly strong: the improvement on real Animal3D when CtrlAni3D is added to training (Supplementary Table 7) provides an independent signal that the synthetic data helps genuine 3D accuracy. However, the quantitative support for the headline 'state-of-the-art' claim is weakened by the lack of repeated runs/error bars, the uncertainty about the HMR2.0 baseline protocol, and the absence of a direct 3D-consistency validation of the CtrlAni3D labels.","major_comments":[{"comment":"The supplement states that AniMer's performance varies substantially across settings: PA-MPJPE on Animal3D ranges from 87 to 78 mm and PCK@0.15 on Animal Kingdom ranges from 0.5 to 0.6. All tables in the main text report a single rune without error bars. Given that the claimed improvement over HMR2.0 on Animal3D PA-MPJPE is 94.1 vs. 80.4 (Table 1), the reported variation of up to 9 mm is comparable in magnitude to that difference, and the ablation differences in Tables 2 and 4 (e.g., 82.7 vs. 82.9 AUC) are well within the observed variation. Please report results over multiple seeds (at least 3) with mean and standard deviation, and indicate whether the improvements over baselines are statistically significant.","section":"Supplementary Sec. D and Table 1"},{"comment":"The quality-control pipeline verifies only 2D consistency: SAM2 mask IoU > 0.95 compares the generated foreground silhouette with the rendered SMAL mask, and manual review checks visual plausibility. This does not confirm that the generated image actually depicts the conditioning SMAL mesh's 3D pose and shape; a hallucinated image can match the 2D silhouette while differing in limb foreshortening, joint angles, or body proportions. Because the CtrlAni3D test split is produced by the same pipeline, the PA-MPJPE/PA-MPVPE numbers on CtrlAni3D in Table 1 are not independent evidence of 3D accuracy. Moreover, if the training labels are biased, the model could learn that bias and still show gains on Animal3D. The authors should provide a direct 3D validation of CtrlAni3D labels, for example by manually fitting SMAL to a random subset of generated images, or by comparing CtrlAni3D-trained models against an independently annotated real dataset, and report the resulting label noise level.","section":"Sec. 4, CtrlAni3D pipeline"},{"comment":"The text says 'all above methods' (HMR, WLDO, AniMer-a, AniMer-b) are retrained on the full aggregated dataset with the same losses and two-stage strategy, but HMR2.0 is introduced with the phrase 'we compare to HMR2.0 to highlight our special design choices.' It is not specified whether HMR2.0 was also retrained on the same data or used with its original pretrained weights. Since HMR2.0 is designed for SMPL (humans), a fair comparison requires adapting its output head to SMAL and retraining it on the same aggregated data. If the reported HMR2.0 numbers come from an unmodified model, the comparison is confounded and the headline claim 'outperforms HMR2.0' is not supported. Please clarify the exact HMR2.0 training protocol, including data, loss functions, and number of epochs, and either retrain it under identical conditions or label the comparison as 'off-the-shelf'.","section":"Sec. 5.2, Table 1, comparison protocol"}],"minor_comments":[{"comment":"The sentence 'the weights of ViT encoder are pretrained using Xu et al. [47]' refers to ViTPose++, but [47] is a pose-estimation work, not a general ViT pretraining method. Please specify that the backbone is initialized with ViTPose++ pretrained weights, or use a more direct citation for the ViT backbone.","section":"Sec. 3.2"},{"comment":"The rendering position is described as 'uniformly sampled between [−0.5, −0.5, 4] and [0.5, 0.5, 8]'; it would be clearer to state that these are world-coordinate ranges and give the exact sampling distribution.","section":"Sec. 4"},{"comment":"The row for HMR2.0 on Animal3D reports PA-MPJPE 94.1 and PA-MPVPE 98.5, but on CtrlAni3D the same model reports 60.9 and 66.4; it would be helpful to explain this large discrepancy, especially if the model is trained on the same aggregated data.","section":"Sec. 5.2, Table 1"},{"comment":"The 'others' column aggregates Animal Pose, APT-36K, AwA-Pose, Stanford Extra, and Zebra synthetic, but the exact composition is not given in the table or the text. Please define this set explicitly, since the ablation conclusions depend on it.","section":"Sec. 5.3, Table 2"},{"comment":"The term 'family aware' is used without a hyphen in the abstract and title; please use a consistent hyphenation ('family-aware') throughout.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be useful to the community due to the CtrlAni3D dataset and the clear demonstration that scaling the backbone helps animal mesh recovery. However, the evaluation falls short of the rigor needed for the strong quantitative claims: single runs without error bars, an unclear HMR2.0 baseline protocol, and a synthetic dataset whose 3D labels are only verified in 2D. I recommend asking for a major revision that adds multi-seed statistics, clarifies the HMR2.0 re-training, and provides a 3D-consistency validation of CtrlAni3D. The central idea is sound and the direction of the improvement is probably correct, but the precise numbers in Table 1 should not be taken at face value until these points are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi,\n\nBottom line: this is a legitimately useful paper for the animal pose/shape community, and the central claim is probably true. The authors combine a ViT-based SMAL regressor with a family-supervised contrastive loss and a diffusion-generated synthetic dataset (CtrlAni3D). They retrain strong baselines on the same aggregated data and report consistent gains on Animal3D, their synthetic test set, and the out-of-distribution Animal Kingdom benchmark. If the results hold up, the method is a clear step forward.\n\nWhat's new: the architecture transfer from HMR2.0 to animals, with a family-aware contrastive scheme on the class token, and the CtrlAni3D pipeline. The contrastive loss is neat and the ablation shows it helps, especially for rare species. The dataset aggregation itself is valuable; the field needs this.\n\nWhere I'd push back:\n\n1. The synthetic label verification is the load-bearing assumption. The filtering checks 2D silhouette consistency (SAM2 IoU), and the manual review is 2D as well. ControlNet can satisfy a mask while changing foreshortening or joint angles, so the 3D pose/shape labels could be biased. The paper doesn't provide a 3D consistency check. I don't think this is fatal—the depth conditioning does constrain geometry, and the fact that adding CtrlAni3D improves real 3D metrics on Animal3D (Supplementary Table 7) is positive evidence—but it needs to be addressed. A small experiment comparing predicted meshes on generated images with the conditioning meshes would help.\n\n2. No error bars and a wide admitted range. The supplement says PA-MPJPE spans 87–78 and PCK@0.15 spans 0.5–0.6 under different settings. The reported numbers are the good end. Even the worst case beats HMR2.0, so the conclusion is robust, but the headline numbers are not nailed down. Report multiple seeds, or at least the variance.\n\n3. Minor but annoying: the AniMer-a / AniMer-b naming is inconsistent between the main text and Supplementary Table 9. In the main text, AniMer-a is ResNet152 and AniMer-b is no pretraining; the supplement calls ResNet152 AniMer-b. Fix it.\n\nNo code or data is released, which is a shame. The project page exists, but the community would benefit from the synthetic generation pipeline and the model weights.\n\nOverall: this deserves a serious referee. The core result is likely correct, and the synthetic data direction is worth engaging with. I'd send it to peer review with a request for error bars, a 3D-consistency check on CtrlAni3D, and a naming fix. It's a conditional accept, not a reject.","headline":"A solid, useful step for multi-species animal mesh recovery, but the synthetic label fidelity and missing uncertainty reporting keep it at conditional rather than accept.","tokens_in":18295,"tokens_out":3230,"would_cite":true,"duration_ms":27519,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Family-aware Transformer beats previous animal 3D reconstruction","keywords":["animal pose estimation","shape estimation","SMAL model","Transformer","supervised contrastive learning","synthetic data","diffusion model","quadruped"],"falsifier":"Render a random sample of CtrlAni3D images, then run an independent three-dimensional fitting optimizer that does not use the dataset's own labels to estimate SMAL parameters from each image, and compare those fits to the dataset labels; systematic divergence on pose or shape, beyond the expected fitting noise, would indicate the labels are biased by two-dimensional-only filtering.","tokens_in":17053,"feed_emoji":"🐾","tokens_out":5113,"duration_ms":42997,"temperature":0.7,"pith_summary":"This paper sets out to show that animal pose and shape estimation can be scaled the way human mesh recovery was: by replacing CNN backbones with a high-capacity Transformer and by training on far more data. To make that data exist, the authors build CtrlAni3D, a synthetic dataset of about 10,000 images with pixel-aligned SMAL labels, generated by a diffusion-based pipeline that conditions on rendered depth and mask maps. They also add an animal-family supervised contrastive loss that pulls images of the same quadruped family together in feature space. The resulting system, AniMer, reports better 3D pose and shape errors and better 2D keypoint accuracy than prior CNN-based methods on Animal3D, on its own synthetic set, and on the out-of-distribution Animal Kingdom benchmark. If correct, this shows that the human-mesh-recovery scaling recipe transfers to animals and that synthetic data generated from parametric models can substitute for manually annotated 3D images.","feed_headline":"Family-aware Transformer beats previous animal 3D reconstruction","feed_subtitle":"New synthetic data plus family-aware contrastive loss beats prior models on wild animal images.","key_machinery":"The load-bearing component is a family-aware Transformer: a ViT backbone with a learnable class token, followed by a SMAL Transformer decoder that regresses SMAL parameters through MLPs, together with an animal-family supervised contrastive loss (Lcon) applied to the class token. The class token acts as a compact representation of the animal family, and the contrastive loss makes the token's embedding cluster animals of the same family while separating different families, which improves the network's ability to predict family-appropriate body shapes. The other key mechanism is the CtrlAni3D generation pipeline, which turns SMAL mesh labels into realistic images via ControlNet conditioned on rendered depth and mask maps, then filters them with SAM2-based mask IoU and manual review.","core_discovery":"The paper's central claim is that a single network can reconstruct the 3D pose and shape of diverse quadruped species from one RGB image with greater accuracy than existing methods, provided the network is built on a Transformer backbone, a supervised contrastive loss on animal family labels, and a sufficiently large and diverse training set. Following the recipe of HMR2.0, the authors connect a ViT encoder to a Transformer decoder to regress SMAL shape, pose, and camera parameters directly rather than as residuals. The family-aware contrastive loss operates on a class token that interacts with image tokens, pulling same-family animals together and separating different families, which the paper argues is essential because animal shape distributions differ across families. The paper further claims that the new CtrlAni3D synthetic dataset, produced by prompting ControlNet with SMAL-rendered depth and mask maps, improves in-the-wild generalization compared to training without it and compared to traditional CG-rendered synthetic data.","pith_inferences":["Because the family-aware contrastive loss clusters same-family shapes, AniMer should be directly applicable to rare or unseen species within the families it has seen, potentially improving few-shot adaptation to new quadruped species if the SMAL shape space covers them.","The CtrlAni3D pipeline is not tied to SMAL: it could be reused to generate pixel-aligned training data for any parametric mesh model, including new animal models with more expressive shape spaces, as long as depth and mask renders can be produced.","If the paper's reported gains on Animal Kingdom hold in independent testing, a practical near-term consequence is that behavioral biologists could use off-the-shelf AniMer reconstructions for quantitative movement analysis in the wild, without per-species fine-tuning.","One implication the paper does not chase is whether the family contrastive token can be swapped for a continuous shape embedding to interpolate between families rather than classify them; that would be a natural test of how much of the gain comes from the contrastive objective versus the Transformer capacity."],"forward_implications":["Applying the same scaling recipe used for human mesh recovery to animals yields consistent gains across 3D and 2D benchmarks, not just on the training distribution.","A diffusion-based conditional image generator can be used as a data engine for parametric mesh labels, replacing expensive manual 3D annotation and providing pixel-aligned supervision at scale.","Animal-family supervised contrastive learning makes the model more shape-aware for families that are underrepresented in the dataset, such as boars and cats.","Training on synthetic images generated from SMAL improves accuracy on real in-the-wild images enough to make synthetic data a practical component of animal mesh recovery."],"supporting_citations":[{"why":"Supplies the SMAL parametric model whose shape and pose parameters AniMer regresses.","marker":"[53]"},{"why":"Provides the Animal3D dataset and the CNN-based HMR/WLDO baselines that AniMer must beat.","marker":"[46]"},{"why":"Supplies the ViT-plus-Transformer-decoder architecture for mesh recovery that AniMer adapts to animals.","marker":"[10]"},{"why":"Provides the ViT backbone and the class-token concept used in the network.","marker":"[9]"},{"why":"Provides the ControlNet conditional image generator that creates the CtrlAni3D synthetic images.","marker":"[52]"},{"why":"Supplies the supervised contrastive learning objective that the family-aware loss is based on.","marker":"[15]"},{"why":"Provides the SAM2 segmentation model used in the semi-automated filtering of CtrlAni3D.","marker":"[34]"},{"why":"Provides the WLDO baseline and dog-specific pose priors used in the pose-space sampling for CtrlAni3D.","marker":"[5]"}],"fun_headline_variants":["Single transformer maps any quadruped’s 3D pose and shape","New synthetic data helps family-aware AI read animal poses","One model, many species: Transformer plus contrastive loss wins","AniMer fuses family labels and diffusion data for animal 3D","Transformer beats prior art on animal 3D with family-aware learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole approach assumes that the images in CtrlAni3D genuinely match the 3D pose and shape of their SMAL labels, but the automated filter only checks 2D mask similarity, so images could lie about 3D geometry and still pass.","fun_headline_variants_meta":{"raw":{"variants":["Single transformer maps any quadruped’s 3D pose and shape","New synthetic data helps family-aware AI read animal poses","One model, many species: Transformer plus contrastive loss wins","AniMer fuses family labels and diffusion data for animal 3D","Transformer beats prior art on animal 3D with family-aware learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1412,"prompt_tokens":1036,"completion_tokens":376,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":652,"completion_tokens_details":{"reasoning_tokens":286}},"tokens_in":652,"tokens_out":376,"duration_ms":4216,"temperature":1.0,"reasoning_tokens":286,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:56:33.232271+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Render a random sample of CtrlAni3D images, then run an independent three-dimensional fitting optimizer that does not use the dataset's own labels to estimate SMAL parameters from each image, and compare those fits to the dataset labels; systematic divergence on pose or shape, beyond the expected fitting noise, would indicate the labels are biased by two-dimensional-only filtering.","supporting_citations":[{"cited_title":"Animal3d: A comprehensive dataset of 3d ani- mal pose and shape","cited_arxiv_id":null,"evidence_quote":"Provides the Animal3D dataset and the CNN-based HMR/WLDO baselines that AniMer must beat."},{"cited_title":"Humans in 4d: Recon- structing and tracking humans with transformers","cited_arxiv_id":null,"evidence_quote":"Supplies the ViT-plus-Transformer-decoder architecture for mesh recovery that AniMer adapts to animals."},{"cited_title":"Adding conditional control to text-to-image diffusion models","cited_arxiv_id":null,"evidence_quote":"Provides the ControlNet conditional image generator that creates the CtrlAni3D synthetic images."},{"cited_title":"Who left the dogs out?: 3D animal reconstruction with expectation maximization in the loop","cited_arxiv_id":null,"evidence_quote":"Provides the WLDO baseline and dog-specific pose priors used in the pose-space sampling for CtrlAni3D."}],"review_version":1}