VARPose: Flexible 2D Pose Densification via Visual Autoregressive Modeling for Enhanced 3D Lifting

Dan Zeng; Kaiyuan Pu; Tiantian Yang

VARPose: Flexible 2D Pose Densification via Visual Autoregressive Modeling for Enhanced 3D Lifting

T0 review · 2 major / 6 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VARPose establishes that 2D pose densification can be reframed as next-scale autoregressive prediction, using one granularity-agnostic codebook to generate arbitrary — even unseen — joint densities and measurably improving 3D pose lifting a

desk verdict Genuine next-scale pose densification with a unified codebook, but the headline 2D metric is confounded by unquantified root-duplicated pseudo-nodes and the downstream gains need error bars. read the letter →

arxiv 2608.02214 v1 pith:MW7AQYDG submitted 2026-08-03 cs.CV

Kaiyuan Pu , Tiantian Yang , Dan Zeng This is my paper · ORCID

classification cs.CV

keywords humanposeestimation2Ddensificationvisualautoregressivemodelingnext-scalepredictiongranularity-agnosticcodebook3Dliftingmeshrecoveryresidualquantization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VARPose argues that the 'next-scale prediction' idea that powers modern image generation applies directly to human skeletons if we read 'scale' as 'joint density.' To do this it introduces a Granularity-agnostic Pose Tokenizer (GPT), a single residual-quantized codebook shared by skeletons of very different joint counts, and UniSkelar, an autoregressive Transformer that generates the token sequence of each finer density from the sparsest pose. On Human3.6M the model densifies 17-joint inputs to 48/96-joint outputs at 7.8 px mean error, beating the previous hierarchical autoregressive approach (9.9 px), and it generalizes to novel granularities (including a COCO-style 17-joint skeleton and 192/384/768-joint skeletons) with a frozen codebook and retrained decoder. Feeding the densified 161-joint pose into off-the-shelf 3D lifting backbones improves MPJPE by 0.6–0.9 mm on Human3.6M and 0.3 mm on MPI-INF-3DHP, and improves human mesh recovery by 2.1 mm MPJPE-24 — evidence that the densification carries real anatomical information rather than just extra points.

What carries the argument

Two coupled components carry the argument. (1) GPT — Granularity-agnostic Pose Tokenizer: a single hybrid codebook shared by all pose densities, quantizing via multi-level residual error correction (each level removes the previous levels' quantized residuals) so different granularities land in one shared discrete space; a frozen codebook with retrained decoders handles novel granularities, decoupling representation from topology-specific projection. (2) UniSkelar — Unified Skeletal Autoregressive model: a decoder-only Transformer with rotary position embeddings that predicts the next density level's token sequence in parallel, conditioned on the sparsest 17-joint pose; block-wise causal atte

What would settle it

Take the coarsened skeletons the paper uses (48, 96, 161, 192, 384, 768 joints) and count how many nodes are HEM pseudo-nodes relocated to the pelvis root. Then recompute the root-aligned 2D mean error and the 3D lifting / HMR deltas either (a) excluding those duplicated root joints or (b) after retraining on a coarsening that avoids forced pseudo-nodes (e.g., direct vertex subsampling). If the 2D error rises substantially above the reported 7.8 px or the downstream MPJPE gains disappear, the 'anatomical densification' claim collapses to a trivial root-repeat artifact.

Watch

Extended reading notes

Core claim

VARPose claims that next-scale prediction from visual autoregressive modeling transfers to human skeletons when 'scale' is redefined as skeletal granularity. Its Granularity-agnostic Pose Tokenizer (GPT) learns one codebook (V=4096, D=128) shared across 17-, 48-, and 96-joint poses, using residual quantization with interpolation and an MLP-ResNet to strip away granularity-specific offsets. UniSkelar, a decoder-only Transformer, then predicts the next density level's token map in parallel, conditioned on the sparsest pose, replacing raster-scan next-token generation with global next-scale generation. The reported results: 7.8 px 2D mean error on Human3.6M densification (vs. 9.9 px for the hie

Load-bearing premise

The load-bearing premise is that the dense skeletons used for training and evaluation are genuinely anatomical: the HEM-coarsening pseudo-nodes, which the paper relocates to the pelvis root without reporting their count, are a small enough share of each 48–768 joint skeleton that the reported 2D errors and 3D lifting gains reflect real structure rather than duplicated root coordinates.

Editorial extensions

If this is right

Sparse 2D skeletons can be super-resolved to arbitrary joint densities with a single pretrained codebook; a retrained lightweight decoder — or even few-shot adaptation with 30% of data — handles unseen joint definitions (e.g., COCO-style 17-joint, 192/384/768-joint) without retraining the representation.
Densified 2D poses act as a plug-in prior for 3D lifting: modest but consistent MPJPE reductions across multiple backbones and datasets, with the largest relative gains under occlusion (3DPW-Occ, 3.2 mm MPJPE improvement).
Cross-granularity representation is granularity-invariant: codebook entries do not cluster by density, which explains why novel granularities decode without retraining the codebook and why densification error stays nearly flat from 48 to 768 joints.
Next-scale prediction outperforms next-token prediction for skeleton generation (7.81 vs. 9.61 px), indicating that global structural coherence matters more than local token order for human-pose synthesis.
The full pipeline runs near real-time (21.6 ms/frame) because parallel next-scale generation replaces sequential token-by-token decoding, despite higher FLOPs than the previous method.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the pseudo-node caveat is small in practice, the same granularity-agnostic codebook idea could transfer to other structured sparse-to-dense domains (hand/face landmarks, animal skeletons, even generic graph densification), since the core claim is that representation can be decoupled from projection.
The paper's downstream gains are modest in absolute terms (0.3–0.9 mm); a natural extension is to test densification where it should matter most — heavy occlusion or extreme poses — and quantify gains as a function of occlusion level, since the current evidence is strongest there.
The authors' own comparison with depth-based input enrichment (AugLift) suggests densification is orthogonal to other 2D cues; combining the two could yield additive gains in domain-shift scenarios like 3DPW, where depth cues generalize better.
A testable prediction of the representation–projection decoupling: a family of granularity-agnostic pose foundation models could be built where adding a new skeleton definition costs only a small projection head plus a few-shot data, as demonstrated by the 30%-data COCO adaptation.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: VARPose is a legitimate new application of VAR's next-scale idea to pose densification, with a nice unified codebook and some genuinely promising generalization results, but the headline 2D metric is confounded by an unquantified pseudo-node relocation trick, and the downstream gains are small enough that the lack of error bars matters.

The genuinely new thing here is the granularity-agnostic pose tokenizer with a single residual-quantization codebook, plus an autoregressive model that predicts poses at increasing joint density. That is a real step beyond HiPART, which uses separate codebooks and fixed granularities. The ablation evidence supports the design: unified codebook beats pose-specific codebooks, next-scale beats next-token, and the model's ability to decode unseen granularities (COCO, 192/384/768 joints) with a frozen codebook and tiny decoders is a nice empirical result. The few-shot adaptation numbers (0.27 px extra with 30% data) are also credible. The code release should let people verify this, which is worth credit.

The soft spots: First, the pseudo-node relocation. Appendix B says all pseudo-nodes from HEM coarsening are moved to the pelvis root coordinate. The paper never says how many pseudo-nodes are in the 48/96/161-joint skeletons. If a meaningful share of the "dense" joints are just copies of the root, then the 2D mean error numbers are inflated by trivial zero-error joints, and part of the downstream lifting gain might come from feeding the network repeated pelvis coordinates rather than genuinely richer anatomy.

Second, no error bars anywhere. The Human3.6M gains are 0.6-0.9 mm, MPI-INF-3DHP is 0.3 mm, CPN is 0.4 mm. Those are within typical run-to-run variance, and several per-action deltas in Table 2 are negative. That doesn't kill the paper's central claim, but it means "tangible gains" is overstated until we see variance.

Third, the encoding path for novel granularities is underspecified. The method says encoders stay frozen and only light decoders are retrained, but it's not clear how the frozen encoder, trained on 17/48/96, produces features for a 192-joint pose. The paper says interpolation is used, but the exact procedure is left to the reader.

Fourth, the SOTA densification claim rests on a single task-matched baseline (HiPART), reproduced rather than reported. Thin, but not fatal.

Who is this for? People working on pose generation, 3D lifting with richer 2D inputs, or applying VAR-style modeling to non-image structured data. I'd want the authors to fix the pseudo-node counting and add seeds before believing the headline. The core direction is sound and worth referee time.

Referee Report

2 major / 6 minor

Summary. The paper proposes VARPose, a two-stage framework that adapts visual autoregressive "next-scale" modeling to 2D human pose densification. A Granularity-agnostic Pose Tokenizer (GPT) with a shared residual codebook encodes 17/48/96-joint poses into multi-scale discrete token sequences; a unified autoregressive model (UniSkelar) generates denser token sequences in a coarse-to-fine manner. The resulting dense 2D poses are injected into off-the-shelf 3D lifting models via joint-wise concatenation or cross-attention fusion, and are further used for SMPL mesh recovery through SMPL-IKS. The paper reports state-of-the-art 2D densification accuracy (7.8 px vs. 9.9 px for HiPART on Human3.6M), downstream MPJPE improvements on Human3.6M, MPI-INF-3DHP, and 3DPW, and generalization to novel granularities including COCO skeletons. The authors release code and models.

Significance. If the reported results are robust, VARPose is a novel and useful bridge between generative visual modeling and 3D pose estimation. The unified codebook and the representation-projection decoupling are elegant ideas with broad potential, and the paper is commendably thorough: it includes ablations of the tokenizer, quantization densities, codebook size, token expansion, robustness to Gaussian noise, detection errors, motion blur, and cross-dataset transfer. The released code and models strengthen reproducibility. However, the headline 2D densification claim and the downstream attribution are currently weakened by an unquantified artifact in the data-generation process, and the empirical gains are reported without variance estimates. These issues are addressable and do not invalidate the framework's core design, but they must be resolved before the paper can be accepted.

major comments (2)

[Appendix B; Sec. 4.3; Tables 5 and 7] The HEM coarsening pipeline creates pseudo-nodes that, as stated in Appendix B, are "relocated to the exact 3D coordinate of the root joint (pelvis)." The manuscript never reports how many pseudo-nodes are present in the 48/96/161/768-joint skeletons. Under the root-aligned 2D Mean Error metric, a pseudo-node relocated to the pelvis has zero target error if the model emits the root coordinate, and a model can learn this trivial mapping. Consequently, the claimed SOTA 7.8 px (and the 9.61/7.81 ablation numbers) may be diluted by an unknown fraction of zero-error joints. The same duplicated pelvis coordinates appear in the 161-joint input to MixSTE/GFPose, so the 0.6--0.9 mm MPJPE gains (Tables 2, 5, 7) could partly reflect repeated uninformative input coordinates rather than genuinely richer anatomical structure. Please report the number of pseudo-nodes per granularity, recompute 2D error
[Tables 2, 3, 12; Tables 5--8] All benchmark numbers appear to come from a single run, with no error bars, multi-seed variance, or statistical significance tests. Many claimed gains are small in absolute terms (e.g., Table 3: MPJPE 30.2 to 29.9, PCK +0.1; Table 2: 35.6 to 35.0; Table 12: 40.2 to 39.8). Since GPT and UniSkelar involve stochastic training and sampling, these differences may lie within run-to-run noise. The central claim that densification yields "tangible performance gains" requires at least mean +/- std over several seeds for the main comparisons, or an equivalent significance analysis. This is especially important for the 0.3 mm gain on MPI-INF-3DHP and the 0.1--0.4 px differences in ablation tables.

minor comments (6)

[Sec. 4.3 and Fig. 5] The phrase "unseen granularities" is used to describe COCO and 192/384/768-joint skeletons, but for these novel granularities the lightweight decoder is retrained (Sec. 3.3, Sec. 4.2) and the COCO result uses 30% of the training data (Table 10). The text should explicitly state that the representation is frozen while the decoder is adapted, to avoid implying fully zero-shot generalization.
[Sec. 4.7] The text says "a single 48 GB RTX 4090"; the RTX 4090 has 24 GB of VRAM. Please correct this factual error.
[Table 6] The row "17-GT-3D" is not defined in the main text or table caption. Clarify how it differs from "17-GT" (presumably the 3D-lifted intermediate representation).
[Algorithm 1, line 6] The condition "if n = 1 or k > K_{n-1}" is ambiguous because K_0 is undefined for n = 1. Rewrite with a base-case condition.
[Table 1] The Domain column contains entries such as "Z +" and "R^{...}" that are not typeset consistently or defined; please clean up the notation.
[Sec. 3.5, Eq. (9)] The magnitudes of L_Recon and L_VQ are reported as about 6e-7 and 2e-4, respectively, in Appendix E, but these values are not given in the main text. A brief note in Sec. 3.5 would help readers interpret the loss-balancing choice.

Circularity Check

1 steps flagged · score 3.0 of 10

No derivation-chain circularity; one metric-construction confound (pseudo-node relocation to pelvis) partially affects the headline 2D densification error.

self definitional [Appendix B (Data Processing); Sec 4.1 (Evaluation Metrics); Sec 4.3 (SOTA in 2D Pose Densification)]
"we relocate all pseudo-nodes present to the exact 3D coordinate of the root joint (pelvis) of the corresponding 17-joint ground-truth pose. This ensures that the pseudo-nodes do not contribute erroneously to the geometric loss function or visual output... For 2D evaluation, we adopt root-aligned 2D Mean Error to focus on relative accuracy, as the root in the sparse skeleton is highly precise."

HEM coarsening synthesizes pseudo-nodes without physical locations; the paper defines their target as the pelvis root. Because the 2D metric is root-aligned, every pseudo-node target is (0,0) after alignment, and a model trained on the same relocated targets can output the root to score zero on those joints by construction. The paper never reports the pseudo-node fraction at 48/96/161/768 joints, so the 7.8 px headline (and the 9.61/7.81 ablations) include an unknown number of trivially zero-error terms. The same duplicated root coordinates enter the 161-joint lifting input, partially confounding the 0.6-0.9 mm MPJPE gains, though the bone-interpolation ablation (Table 11) provides some independent support.

full rationale

The paper's core densify-then-lift pipeline is empirical: GPT and UniSkelar are trained on Human3.6M/HEM-SMPL targets and evaluated on held-out test data, and the downstream HPE/HMR gains are measured against standard external benchmarks. There is no analytic derivation chain that reduces to its inputs, no load-bearing self-citation (the authors do not cite their own prior work to justify VARPose), and no uniqueness/ansatz smuggling. The one genuine circularity-adjacent issue is the pseudo-node relocation in Appendix B: by defining pseudo-node GT coordinates as the pelvis root and then evaluating with root-aligned 2D Mean Error, the metric contains zero-error terms by construction. Because the pseudo-node count is unreported, the magnitude of this effect is unknown, but it is a real construction that weakens the headline 2D densification comparison. The downstream and COCO-transfer results are independent empirical evidence, so this is a partial confound rather than a full circular collapse; hence a moderate score of 3.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The paper contributes a learned representation and an AR model, but the central evaluation depends on several data-construction choices: HEM-derived dense targets, pseudo-node relocation, hand-picked quantization densities, and hyperparameters tuned on the test benchmark. These are not free parameters in a mathematical derivation, but they are choices fitted to the data that affect the reported gains.

free parameters (5)

loss weight lambda = 200
Set empirically in Eq. 9; ablation in Appendix E shows 100 and 400 are worse. The value is tuned on the test benchmark rather than a held-out set.
quantization densities rho_k = [48, 102, 192, 288, 432, 576]
Anchor densities come from 6x training joint counts (17, 48, 96), with midpoints inserted. These densities define the token sequence length and all generated granularities.
codebook vocabulary size V = 4096
Selected via Table 17 as the best among 512/1024/2048/4096/8192 on reconstruction error.
embedding dimension D = 128
Selected via Table 17 as a trade-off between reconstruction error and compute; D=512 gives lower error but much higher cost.
joint-to-token expansion factor = 6
Chosen after ablations in Tables 15-16; 1-to-6 with balanced positional/adjacency tokens gives the best reconstruction error.

assumptions (4)

domain assumption Pseudo-nodes created by HEM coarsening can be relocated to the pelvis root without corrupting pose structure or losses.
Appendix B states pseudo-nodes are repositioned to the root joint 'to ensure that the pseudo-nodes do not contribute erroneously to the geometric loss function or visual output'. This is load-bearing for the dense-pose quality claim and is not quantified.
domain assumption SMPL mesh coarsening yields valid 2D dense ground-truth skeletons for 48/96/192/384/768 joints.
The entire training and evaluation protocol relies on HEM coarsening of pseudo-GT meshes from Pose2Mesh; no independent dense 2D annotations are used.
domain assumption A codebook trained on 17/48/96-joint poses remains a sufficient representation for novel granularities after retraining only the decoder.
The generalization claim in Sec. 3.3 and Sec. 4.2 depends on this transfer assumption; it is validated only empirically and the encoding path for novel granularities is underspecified.
standard math Standard VQ-VAE training (straight-through estimator, EMA codebook updates) behaves as expected for skeletal coordinates.
The tokenizer relies on standard neural discrete representation learning (VQ-VAE, Ref. [34]); no formal guarantees are provided.

invented entities (1)

Synthetic pseudo-nodes from HEM graph coarsening
purpose: Fill binary-tree coarsening structure; relocated to the pelvis root and treated as real joints in training and downstream inputs.
These are not anatomical joints and have no physical location. Their presence may artificially reduce 2D mean error and dilute downstream attention, so the empirical evidence does not independently validate them.

how reviews work

0 comments

Cite this review

Pith. "Pith review of VARPose: Flexible 2D Pose Densification via Visual Autoregressive Modeling for Enhanced 3D Lifting." pith.science (2026). https://pith.science/paper/MW7AQYDG

@misc{pith2026260802214,
  author       = {Pith},
  title        = {Pith review of: VARPose: Flexible 2D Pose Densification via Visual Autoregressive Modeling for Enhanced 3D Lifting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MW7AQYDG}},
  note         = {Machine review of arXiv:2608.02214}
}

read the original abstract

Visual AutoRegressive Modeling (VAR) has excelled in natural image generation via next-scale prediction, but its use on topology-structured data like human skeletons is still unexplored. VARPose is proposed to adaptively densify 2D sparse poses, thereby enriching the anatomical information available for 3D lifting models. Our core contributions are twofold. First, we introduce a Granularity-agnostic Pose Tokenizer (GPT), which employs a single hybrid codebook and a residual quantization strategy to encode poses of varying densities into a unified, multi-scale discrete representation. Our results demonstrate the strong generalizability of this representation. By decoupling the representation from the projection, we can successfully decode novel pose granularities using a frozen codebook with a retrained decoder. Second, we propose UniSkelar, a unified autoregressive model that treats "joint density" as "scale". UniSkelar learns to predict the token sequence for the next density level in a coarse-to-fine manner, conditioned on the sparsest pose. VARPose not only outperforms state-of-the-art methods and generalizes to unseen granularities, but also confers tangible performance gains on downstream tasks, such as 3D Pose Estimation and Human Mesh Recovery, through 2D pose densification. Our code and model are available at https://github.com/BRL-SYSU/VARPose.git.

Discussion (0). Continue with ORCID to comment.

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.