{"id":"c92ddc31-0c27-4dfb-96cf-d83f1971e612","arxiv_id":"2506.20983","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SP-Ctrl improves pose-guided text-to-image generation with sparse poses by learning keypoint embeddings and supervising keypoint attention maps, nearly matching dense depth-based control.","lead":"A new ControlNet variant uses learnable keypoint embeddings and keypoint-specific text tokens to make sparse OpenPose signals control poses as accurately as dense depth maps in text-to-image generation. This could make pose-guided image generation simpler and more flexible, enabling easy pose editing and cross-species transfer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pose mAP gains may reflect estimator bias, not anatomical alignment; independent evaluation is needed to support the 'matches dense signals' claim.","rationale":"The reader identified the evaluation metric as the weakest assumption, and I agree. The concern is load-bearing because every quantitative comparison supporting the abstract's headline claim passes through ViTPose++ mAP. The paper's own training objective makes the risk concrete: KCL supervises cross-attention maps to match ground-truth keypoint heatmaps, which can encourage the generator to synthesize local keypoint-like cues. ViTPose++, trained on the same datasets, may exploit exactly those cues, inflating SP-Ctrl's score relative to a sparse-pose ControlNet that lacks such explicit keypoint supervision. This does not imply fraud or even that the method is weak; it means the current evidence cannot distinguish true anatomical alignment from estimator exploitation. The lack of error bars compounds the problem: the 1.44 mAP gap to depth is presented as 'matching,' but without variance we cannot tell whether it is meaningful. An independent estimator or human keypoint annotation on a subset would settle the issue. If the improvement survives such a test, the central claim is substantially supported; if not, the claim should be softened to 'improves controllability under its own evaluation metric.' I therefore keep the reader's CONDITIONAL verdict, which appropriately requests additional validation before full acceptance.","tokens_in":20733,"tokens_out":5354,"duration_ms":60154,"concrete_test":"Run a blinded evaluation on a random subset of 500 generated images per method: (1) re-estimate keypoints with a pose estimator not trained on AP-10K/Human-Art—for humans use ViTPose or OpenPose trained on COCO; for animals use a detector trained on other animal pose data or manual human annotation—and recompute OKS mAP against the same input poses; (2) additionally train SP-Ctrl and ControlNet three times with different seeds and report mean +/- standard deviation of pose mAP on AP-10K and Human-Art. If the SP-Ctrl gain and the depth gap (about 1.44) persist with the independent estimator and error bars exclude chance overlap, the concern is resolved; otherwise the 'matches dense signals' claim must be tempered.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that sparse-pose SP-Ctrl matches dense-signal methods rests on pose mAP computed with ViTPose++ pretrained on the same AP-10K and Human-Art datasets used for training (Section 4.1, Metrics). This metric is not a neutral ruler here. The proposed Keypoint Concept Learning (Section 3.2) explicitly supervises cross-attention maps with ground-truth keypoint heatmaps, teaching the generator to place localized appearance cues exactly at keypoint positions. Since ViTPose++ was trained on the same annotation distribution, it may recognize these cues (e.g., local contrast, texture patches) as keypoints even when anatomical alignment is imperfect. The reported 55.63 versus 57.07 mAP gap to depth-based ControlNet (Table 3) could be artificially closed, and the 11.38-point improvement over ControlNet (Table 1) could reflect 'detectability' rather than true pose fidelity. The appendix explicitly admits a >25% pose gap versus real images (Section E), yet no independent estimator, no human study, and no error bars are provided. Thus the load-bearing assumption that ViTPose++ mAP is an unbiased proxy for pose alignment is insecure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SP-Ctrl, a ControlNet-based method for pose-guided text-to-image generation using sparse OpenPose signals. It introduces two main components: (1) a Spatial-Pose Representation (SPR) that replaces fixed RGB keypoint embeddings with learnable embeddings produced by an MLP, and (2) a Keypoint Concept Learning (KCL) strategy that adds new textual tokens for each keypoint and applies a heatmap loss (Eq. 3) on cross-attention maps to align them with ground-truth keypoint heatmaps. Experiments on AP-10K and Human-Art report pose mAP gains over ControlNet (11.38 points on AP-10K, 5.85 on Human-Art) and pose mAP close to depth-based ControlNet on AP-10K (55.63 vs 57.07). The paper also demonstrates qualitative benefits in image diversity, cross-species generalization, and pose editing, and includes extensive ablations of the proposed modules and hyperparameters.","tokens_in":20964,"tokens_out":5144,"duration_ms":52703,"significance":"If the results are reliable, the paper makes a strong empirical case that sparse pose signals can be competitive with dense signals (e.g., depth) for pose-guided generation, while retaining advantages in editability and cross-species generalization. The two proposed mechanisms—learnable spatial-pose embeddings and keypoint concept learning—are novel, well-motivated by cross-attention analysis, and thoroughly ablated. The paper also ships a clear architecture description and qualitative comparisons. However, the central 'matches dense signals' claim rests on a single pose-estimator metric that may be biased, and no error bars are provided; the significance is therefore provisional until the evaluation concern is addressed.","major_comments":[{"comment":"The pose mAP is computed with ViTPose++-H pretrained on AP-10K and Human-Art, the same datasets used to train SP-Ctrl. Since the proposed KCL loss (Eq. 3) explicitly trains cross-attention maps to match ground-truth keypoint heatmaps, the generator may learn to place locally detectable keypoint-like appearance cues that this specific estimator recognizes, potentially inflating the measured pose alignment. This does not make the training loss circular, but it makes the evaluation metric a non-neutral ruler for the central claim of matching dense signals. The authors should validate with an independent pose estimator not trained on these datasets (e.g., a detector trained on COCO or a different architecture), a human perceptual study, or at least report pose mAP from a second estimator and error bars over multiple runs.","section":"Section 4.1, Metrics paragraph"},{"comment":"The abstract's claim that SP-Ctrl 'even matches the performance of dense signal-based methods' is overstated. On AP-10K, Table 3 shows a 1.44 mAP gap to depth-based ControlNet (55.63 vs 57.07), and on Human-Art no dense baseline is reported. The wording should be revised to 'approaches' or 'is competitive with,' and the residual gap should be acknowledged quantitatively. The text in Section 4.5 says 'almost matching,' which is more accurate, but the abstract and Figure 1 use 'matches' and 'gap' inconsistently; Figure 1's labels 'Gap=12.82%' and '11.38%' are not directly traceable to Table 3.","section":"Abstract and Section 4.5, Table 3"},{"comment":"No error bars or statistical significance tests are reported for any metric. The authors generate three images per prompt (Section 4.2), but all numbers in Tables 1–3 and Figures 6–7 are point estimates. Several ablation differences are small (e.g., 0.86% in Fig. 6(b), and 'competitive' across embedding dimensions in Fig. 6(c)), and the headline 11.38-point gain over ControlNet could also be subject to variance. The authors should report standard deviations across at least three seeds or bootstrap confidence intervals for the key comparisons (Ours vs. ControlNet, Ours vs. Depth).","section":"Section 4.2 and Section 4.4"}],"minor_comments":[{"comment":"The CLIP-Score evaluation removes the newly introduced keypoint tokens from the prompt during evaluation, while generation uses them. This is explained in Section 4.3 and Fig. S17, but the comparison with baseline methods, which use standard prompts, is not apples-to-apples. It would be informative to also report CLIP-Score for all methods using prompts without keypoint tokens, and to explicitly state the CLIP-Score drop (e.g., 0.68 in Section 4.4) as a limitation of the method.","section":"Section 4.3 and Appendix C, CLIP-Score analysis"},{"comment":"The bars labeled 'Mask (ControlNet)', 'Depth (ControlNet)', 'Sparse Pose (ControlNet)', and 'Sparse Pose (Ours)' are inconsistent with Table 3, which reports mask mAP 26.74, depth 57.07, sparse 44.25, and ours 55.63. The figure's 'Gap=12.82%' and '11.38%' labels are unclear; specify which gaps are shown and how they are computed.","section":"Figure 1"},{"comment":"The distinction between 'randomly initialized vectors E0', 'fixed E0', and 'learnable E0' is confusing. Clarify that E0 is the input to the spatial-pose embedding module G, and that in the default setting E0 is fixed and only G's parameters φ are learned.","section":"Section 3.1, Eq. (1)"},{"comment":"The decision to detach the gradient from the noisy image query Q when computing Lht is mentioned but not ablated. If this choice is important for training stability, include an ablation; otherwise, remove the detail or justify it briefly.","section":"Section 3.2"},{"comment":"Appendix E states that a significant gap in pose accuracy (greater than 25%) remains between synthesized and real images. This limitation should be acknowledged in the main paper's conclusion or abstract to calibrate reader expectations.","section":"Appendix E"},{"comment":"In Table 1, the 'Real Image' row for Human-Art has an empty FID cell. Use an em dash and add a note explaining why FID is not computed or not applicable for real images.","section":"Table 1"},{"comment":"The table headers use 'Pose mAP↑', 'FID↓', 'CLIP-Score↑' in some tables and 'Pose mAP', 'FID', 'CLIP-Score' in others; ensure consistent arrow notation across Tables 1–3 and the main text.","section":"General notation"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical paper with clear writing and thorough ablations. The main concern is the evaluation metric: using a pose estimator trained on the same AP-10K and Human-Art datasets, combined with a loss that places keypoint cues exactly at keypoint locations, may inflate pose mAP. This is not an accusation of misconduct, but it directly affects the central claim of matching dense signals. The authors should be asked to add an independent evaluation (e.g., a different pose estimator, human study, or error bars) before acceptance. The abstract's 'matches' wording should also be softened. The paper is close to being acceptable but needs these revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a solid, well-executed empirical contribution. What's new is the combination: replacing the fixed RGB keypoint colors in OpenPose with learnable per-keypoint embeddings, and adding keypoint-specific text tokens whose cross-attention maps are supervised directly with ground-truth keypoint heatmaps. The ablations are thorough and honest, and the gains over ControlNet with sparse pose are large and consistent (11.38 mAP on AP-10K, 5.85 on Human-Art). The qualitative results show visibly better pose fidelity, and the cross-species and pose-editing demos are genuinely useful. I believe the central result — sparse pose control can be made much stronger with these two mechanisms — holds up.\n\nThe main soft spot is the headline claim of matching dense signals. On AP-10K there is still a 1.44 mAP gap to depth-based ControlNet, and on Human-Art there is no dense baseline at all. The phrase \"even matches the performance of dense signal-based methods\" overstates what the tables show. The deeper concern is the evaluation metric: pose mAP is computed with ViTPose++ pretrained on the same AP-10K and Human-Art datasets used for training. Given that the heatmap loss explicitly teaches the generator to place local appearance cues at keypoint positions, the estimator could be rewarding detectability rather than anatomical correctness. The paper reports no error bars and no independent human evaluation. That said, I read the stress-test concern as a valid caution rather than a fatal flaw. The method also improves over ControlNet on the same estimator, so the comparison to the sparse-pose baseline is fair; the risk is mostly to the \"matching dense\" claim, which would need an independent evaluator or human study to be secure.\n\nMinor issues: CLIP-Score is computed with the keypoint tokens removed, which they acknowledge and justify, but the reported number is not directly comparable to methods that use the full prompt. The appendix admits a >25% pose accuracy gap versus real images, which is not emphasized in the abstract. These are presentation problems, not evidence of sloppiness.\n\nWho is this for? Anyone working on controllable generation with pose or other sparse conditions. The method is simple enough to reproduce, and the released code will help. I would send it to review — the core claim is worth referee time even if the abstract needs tempering. I would not cite it in my own work within 12 months, but that is because my work is far from this subfield, not because of quality.","headline":"Solid empirical paper: sparse-pose ControlNet with learnable keypoint embeddings and keypoint-token attention supervision shows real gains, but the headline claim of matching dense signals rests on a pose estimator trained on the same data.","tokens_in":21503,"tokens_out":644,"would_cite":false,"duration_ms":8736,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sparse keypoint skeletons can be made as precise as depth maps for pose-guided image generation.","keywords":["sparse pose signals","pose-guided text-to-image generation","OpenPose","ControlNet","learnable keypoint embeddings","keypoint concept learning","cross-attention heatmap","diffusion models"],"falsifier":"Re-run the AP-10K and Human-Art comparisons using a pose estimator never trained on those datasets, or have human annotators mark keypoints on a sample of generated images, and check whether SP-Ctrl keeps its reported lead over sparse-pose ControlNet and its small gap to depth-conditioned ControlNet.","tokens_in":20519,"feed_emoji":"🐾","tokens_out":7459,"duration_ms":65357,"temperature":0.7,"pith_summary":"This paper argues that sparse pose signals—the simple keypoint-and-skeleton stick figures of OpenPose—have been prematurely abandoned in favor of dense guides such as depth maps for pose-controlled text-to-image generation. It proposes SP-Ctrl, a ControlNet-style adapter that makes sparse keypoints precise enough to rival dense signals by turning keypoint colors into learnable embeddings and by adding per-keypoint text tokens whose cross-attention maps are trained to land on the keypoints. On the AP-10K animal benchmark the method raises pose mAP from 44.25 (ControlNet with OpenPose) to 55.63, approaching depth-based ControlNet at 57.07, and on Human-Art it reaches 51.11 versus 45.26 for ControlNet. The payoff is that users keep the editability, cross-species transfer, and shape diversity of sparse skeletons without giving up alignment.","feed_headline":"Sparse keypoints match dense depth for pose-guided image generation","feed_subtitle":"Learnable keypoints plus attention heatmaps lift pose mAP from 44.25 to 55.63 on AP-10K, near depth's 57.07.","key_machinery":"The machinery is the combination of a Spatial-Pose Representation (SPR) and a Keypoint Concept Learning (KCL) strategy inside a ControlNet. SPR replaces the fixed RGB blob colors of OpenPose with learnable per-keypoint embeddings produced by a small MLP, so each of the 17 keypoints gets a distinctive vector optimized for the diffusion loss, while skeleton edges use an all-ones embedding. KCL injects new textual tokens for each visible keypoint into the prompt and, during training, forces the cross-attention maps of those tokens at the 3rd transformer block and timesteps 250–500 to match Gaussian heatmaps centered at the keypoint locations. The heatmap loss is what makes the keypoint tokens spatially grounded, and the learnable embeddings are what make the pose input expressive.","core_discovery":"The paper's central claim is that sparse OpenPose-style keypoints are sufficient for precise pose-guided text-to-image generation, provided the pose signal is re-encoded and the model is taught what each keypoint means. Two bottlenecks are identified: OpenPose's RGB rendering is designed for visualization and carries little learnable signal, and point-like keypoints are hard for the diffusion model to attend to. SP-Ctrl addresses both: a Spatial-Pose Representation learns a distinct embedding per keypoint through an MLP optimized by the denoising objective, and Keypoint Concept Learning adds new text tokens such as <left-eye> to the prompt and minimizes an L2 loss between the cross-attention maps of those tokens and keypoint-heatmap targets during training. With these two additions, the model matches the pose alignment of depth-conditioned ControlNet while keeping the flexibility of sparse signals, as demonstrated on animal and human benchmarks.","pith_inferences":["The cross-attention heatmap constraint could generalize to other sparse geometric guidance, such as edges or object landmarks, wherever a spatial target map can be defined.","An independent pose estimator or a human keypoint-rating study would settle whether the mAP gains reflect true anatomical alignment or exploitation of ViTPose++ trained on the same benchmarks.","Because the keypoint embeddings are learned from random vectors, the results suggest the diffusion objective alone can imbue purely spatial tokens with semantic identity, a property worth testing in other conditioning tasks.","The remaining >25% mAP gap to real images noted in the appendix points to pose diversity rather than representation as the next bottleneck; augmenting training with synthetic complex poses is a concrete next test."],"forward_implications":["Sparse-pose ControlNet trained this way can approach or match the pose alignment of depth-conditioned ControlNet (55.63 versus 57.07 mAP on AP-10K) without requiring depth maps.","Because the condition is still a sparse skeleton, users can edit, create, and transfer poses manually or across species, which dense signals constrain.","The method does not need pretrained pose estimators during training, unlike HumanSD or GRPose, which simplifies the training pipeline.","The slight CLIP-Score decrease (23.86 versus 24.77 on AP-10K) is attributed to the new keypoint tokens being absent from the CLIP vocabulary, an evaluation discrepancy rather than a text-alignment failure."],"supporting_citations":[{"why":"Defines the sparse keypoint-and-skeleton representation (17 keypoints and links) that the method extends into a learnable spatial pose.","marker":"[5]"},{"why":"Provides the ControlNet architecture and the primary baseline whose sparse-pose mAP is the comparison point throughout the paper.","marker":"[55]"},{"why":"Supplies the spatio-textual representation idea that motivates replacing fixed RGB keypoint colors with learnable embeddings.","marker":"[1]"},{"why":"Provides the textual-inversion technique of learning new textual tokens, which underlies the per-keypoint concept tokens.","marker":"[10]"},{"why":"Establishes that cross-attention maps in Stable Diffusion correlate nouns with spatial regions, the premise for keypoint concept learning.","marker":"[46]"},{"why":"Shows that cross-attention maps can be used for spatial control, supporting the choice to constrain them with the heatmap loss.","marker":"[12]"},{"why":"The ViTPose++ pose estimator used to compute the OKS-based pose mAP metric on generated images.","marker":"[49]"},{"why":"Provides the AP-10K animal benchmark with 17 shared keypoints across 54 mammal species for training and evaluation.","marker":"[53]"},{"why":"Provides the Human-Art dataset spanning real and virtual scenarios, used as the second benchmark.","marker":"[19]"}],"fun_headline_variants":["Sparse keypoints now match dense depth in pose-guided generation","Learnable keypoints rival dense depth for pose control in T2I","SP-Ctrl: sparse signals equal dense guidance via keypoint attention","Rethinking sparse pose signals: attention makes them as good as depth"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported pose mAP is computed with ViTPose++ pretrained on the same AP-10K and Human-Art datasets used for training, and the paper provides no error bars or independent human evaluation, so the headline gains assume the estimator rewards true keypoint alignment rather than spurious patterns in the generated images.","fun_headline_variants_meta":{"raw":{"variants":["Sparse keypoints now match dense depth in pose-guided generation","Learnable keypoints rival dense depth for pose control in T2I","SP-Ctrl: sparse signals equal dense guidance via keypoint attention","Rethinking sparse pose signals: attention makes them as good as depth"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000241,"raw_usage":{"total_tokens":1531,"prompt_tokens":963,"completion_tokens":568,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":505}},"tokens_in":579,"tokens_out":568,"duration_ms":5837,"temperature":1.0,"reasoning_tokens":505,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:36:35.059110+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the AP-10K and Human-Art comparisons using a pose estimator never trained on those datasets, or have human annotators mark keypoints on a sample of generated images, and check whether SP-Ctrl keeps its reported lead over sparse-pose ControlNet and its small gap to depth-conditioned ControlNet.","supporting_citations":[{"cited_title":"Openpose: Realtime multi-person 2d pose estimation using part affinity fields","cited_arxiv_id":null,"evidence_quote":"Defines the sparse keypoint-and-skeleton representation (17 keypoints and links) that the method extends into a learnable spatial pose."},{"cited_title":"Adding conditional control to text-to-image diffusion models","cited_arxiv_id":null,"evidence_quote":"Provides the ControlNet architecture and the primary baseline whose sparse-pose mAP is the comparison point throughout the paper."},{"cited_title":"Spatext: Spatio-textual representation for con- trollable image generation","cited_arxiv_id":null,"evidence_quote":"Supplies the spatio-textual representation idea that motivates replacing fixed RGB keypoint colors with learnable embeddings."},{"cited_title":"What the DAAM: Interpreting stable diffu- sion using cross attention","cited_arxiv_id":null,"evidence_quote":"Establishes that cross-attention maps in Stable Diffusion correlate nouns with spatial regions, the premise for keypoint concept learning."},{"cited_title":"Prompt-to-prompt image editing with cross-attention control","cited_arxiv_id":null,"evidence_quote":"Shows that cross-attention maps can be used for spatial control, supporting the choice to constrain them with the heatmap loss."},{"cited_title":"Vit- pose++: Vision transformer for generic body pose estima- tion","cited_arxiv_id":null,"evidence_quote":"The ViTPose++ pose estimator used to compute the OKS-based pose mAP metric on generated images."},{"cited_title":"Ap-10k: A benchmark for animal pose estima- tion in the wild","cited_arxiv_id":null,"evidence_quote":"Provides the AP-10K animal benchmark with 17 shared keypoints across 54 mammal species for training and evaluation."},{"cited_title":"Human-art: A versatile human-centric dataset bridg- ing natural and artificial scenes","cited_arxiv_id":null,"evidence_quote":"Provides the Human-Art dataset spanning real and virtual scenarios, used as the second benchmark."}],"review_version":1}