Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

UniMC: Taming Diffusion Transformer for Unified Keypoint-Guided Multi-Class Image Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read This paper claims that per-instance structured tokens—class name, bounding box, and keypoint coordinates—can replace skeleton images as the control signal, letting one diffusion transformer generate multi-class images of overlapping…

desk verdict A genuinely useful dataset and a sensible token-condition fix for multi-class keypoint generation, held back by circular pose metrics and an unproven shared-keypoint schema. read the letter →

arxiv 2507.02713 v2 pith:XPO73ZTM submitted 2025-07-03 cs.CV

classification cs.CV
keywords keypoint-guidedimagegenerationdiffusiontransformermulti-classhumanandanimalposecontrolinstance-levelconditioningHAIG-2.9Mdatasetcontrollabletext-to-imageocclusionhandling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

UniMC is an attempt to solve a specific control problem: given a text prompt plus a set of per-object keypoint poses for humans and animals, generate an image that places exactly those objects in those poses, even when they overlap. The paper's bet is that skeleton images are the wrong condition format. Instead of drawing keypoints on a shared canvas, it encodes each instance's class name, bounding box, and keypoint coordinates into compact tokens, then injects those tokens into a diffusion transformer through timestep-aware self-attention. On its newly built HAIG-2.9M dataset, the model reports class accuracy of 93.55% for humans and 91.71% for animals, with pose AP of 30.01 and 28.38, and it outperforms skeleton-image and coordinate baselines. If true, the result matters because it points toward a single controllable generator for non-rigid objects generally, not just humans.

What carries the argument

The load-bearing mechanism is the per-instance condition token: for each instance, the class name is encoded with a text encoder, and the keypoint coordinates and bounding box are Fourier-mapped and passed through separate MLPs, with a learnable mask token for any missing part; the resulting keypoint and box tokens are then concatenated. These tokens are injected into every block of a pretrained diffusion transformer through a timestep-aware keypoint modulator, a shared self-attention layer plus a shared timestep adapter that supplies scale and shift, with a zero-initialized residual projection to preserve the pretrained model's behavior. The unified keypoint encoder is what allows one model to handle different species without training separate encoders for each.

What would settle it

Train the same controller with per-class keypoint encoders instead of the shared 17-keypoint encoder; if per-class encoders improve pose accuracy on anatomically unusual classes such as rhino and hippo, the shared-template assumption is the limiting factor.

Watch

Extended reading notes

Core claim

The central claim is that per-instance structured conditions—class, bounding box, and keypoint coordinates—can replace skeleton images as the control signal for keypoint-guided generation, resolving what the paper calls class binding confusion and instance binding confusion. A unified keypoint encoder maps keypoints of different species into one shared representation by Fourier-embedding coordinates and fusing them with class-name embeddings; a timestep-aware keypoint modulator then lets the keypoint tokens interact with the diffusion transformer's backbone tokens through self-attention at every block. The authors report that this yields high-quality multi-class, multi-instance generation with accurate pose control for both humans and animals, including heavily occluded and overlapping cases. They further claim that the large-scale HAIG-2.9M dataset, with 786K images, 2.9M instances, and combined human-animal keypoint, box, and caption annotations, is what makes training such a unified model possible.

Load-bearing premise

The paper assumes the same 17 keypoints mean the same body parts for every one of the 31 classes, so a single shared encoder can handle a horse's joints and a monkey's joints alike; if that alignment is wrong for some species, the unified pose control collapses.

Editorial extensions

If this is right

  • A single UniMC model can generate multiple overlapping humans and animals in one image, a scenario where skeleton-image conditioning becomes ambiguous.
  • Keypoint-level control now extends to animals rather than only humans, reaching 91.71% animal class accuracy and 28.38 animal pose AP on the HAIG-2.9M test set.
  • The dataset is a necessary part of the result: training the same model on COCO plus APT36K yields FID 33.79 with weaker animal control, while training on HAIG-2.9M yields FID 23.63.
  • Because only the keypoint encoder and modulator are trained, controllability is added to a frozen large text-to-image diffusion transformer at modest cost, with 8K training steps at 1024x1024 resolution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The token recipe may transfer to articulated object classes beyond the 31 studied ones, provided a keypoint template can be defined; the class embedding is the natural place where such zero-shot generalization would have to emerge.
  • The shared 17-keypoint template is the most likely scaling bottleneck: adding classes with very different limb geometry could require hierarchical or per-part keypoint embeddings rather than one flat template.
  • The dataset's paired human-animal annotations could support applications beyond generation, such as layout-aware editing or mixed-scene pose estimation, though the paper does not demonstrate these.
  • A direct probe of the mechanism would be to ablate the class embedding from the keypoint tokens and check whether cross-species pose accuracy collapses, which would isolate how much of the unification comes from the token fusion.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes UNIMC, a Diffusion Transformer (DiT) framework for keypoint-guided, multi-class, multi-instance image generation covering humans and animals. Instead of rendering skeleton images as control signals, UNIMC encodes each instance's class name, bounding box, and keypoint coordinates into compact tokens via a unified keypoint encoder, and injects them into a frozen PixArt-α backbone through a timestep-aware keypoint modulator. The authors also introduce HAIG-2.9M, a dataset of 786K images with 2.9M instance-level annotations (bounding boxes, keypoints, captions) across 31 classes. Experiments compare UNIMC with PixArt-α, ControlNet, and GLIGEN, reporting large gains in class accuracy (human 93.55, animal 91.71) and pose AP (human 30.01, animal 28.38), while maintaining comparable FID/KID and CLIP scores. Ablations study four modulator configurations and different training datasets.

Significance. If the control metrics are taken at face value, the paper makes a useful step toward unified keypoint-guided generation for non-rigid objects: the token-based condition formulation directly addresses the class- and instance-binding problems of skeleton-image conditions, and the proposed HAIG-2.9M dataset is substantially larger and more diverse than existing human-only or animal-only keypoint-image datasets. The method is evaluated with FID/KID, CLIP, class accuracy, pose AP, and a human preference study, and the ablations are systematic. However, the headline control claims rest on evaluation metrics computed with the same detectors used to annotate the training data, and the cross-species keypoint alignment that underlies the unified encoder is asserted rather than demonstrated. These issues make the central claims plausible but not yet conclusive.

major comments (4)
  1. [Sec. 4.2 and Sec. 5 (Evaluation Metrics)] The class-accuracy metric uses YOLO-World and the pose metrics use DWPose and ViTPose++H, which are exactly the models selected in Sec. 4.2 to annotate HAIG-2.9M. Because the generator is trained on annotations produced by these detectors, the reported control metrics in Table 3 may partly reflect the model learning detector-specific biases rather than ground-truth-aligned control. This is load-bearing for the central claims of improved class and pose control. Please add evaluation with independent detectors (or a held-out human-verified subset), and report agreement between the annotation model and the evaluation model.
  2. [Sec. 4 and Sec. 3.2.1] The unified keypoint encoder assumes that the 17 keypoints are semantically aligned across all 31 classes, but the paper only states in Sec. 4 that 'Each category has 17 corresponding keypoints' without supporting evidence. The Related Work itself acknowledges that 'varied keypoint definitions across classes introduce semantic and structural gaps.' Since the shared encoder is a central contribution, the aggregate animal pose AP in Table 3 cannot reveal whether keypoint indices are conflated across species or whether rare classes receive class-typical rather than condition-faithful poses. Please report per-class pose accuracy, a keypoint-confusion analysis, or an ablation that perturbs keypoint indexing across classes.
  3. [Sec. 5 (Comparison Methods)] The baseline setup is underspecified. ControlNet and GLIGEN are originally UNet-based methods, whereas UNIMC is built on the DiT-based PixArt-α; the manuscript does not state whether the baselines are used with Stable Diffusion, adapted to PixArt-α, or re-trained from scratch for Table 3. Without this information, the FID/KID and CLIP comparisons in Table 3 conflate the control mechanism with the base T2I model and training protocol. Please specify the exact base model, adaptation procedure, and training data for each baseline.
  4. [Appendix A.2 and Sec. 5 (Table 3)] The long-tail distribution is acknowledged but not quantitatively tested. Classes such as Orangutan (126 instances), Rhino (178), and Black Bear (304) are very small, and 'bear' is used as a supplement for black bear and polar bear. The paper claims that classes with few samples can still be controlled via the T2I prior, but the evidence is qualitative only. Since the test set is built with 40 images per class, the aggregate animal metrics in Table 3 may be dominated by frequent classes. Please provide per-class results, or explicitly restrict the central claims to classes meeting a minimum instance count.
minor comments (5)
  1. [Sec. 5.3] The heading 'Abalation Study' contains a typo; it should read 'Ablation Study'.
  2. [Sec. 4.2] The text says a 5% random human check is performed, but also that 5K images are randomly sampled for model selection; clarify whether these are the same procedure or two different checks.
  3. [Sec. 3.2.1] In Eq. (1), the symbol g_i is used generically for both keypoint and bounding-box tokens; please use g_i^{kpt} and g_i^{box} consistently in the equation to avoid ambiguity.
  4. [Appendix D.3] The human preference study uses 10 randomly selected samples and 10 participants; this sample size is too small to support strong conclusions, and no confidence intervals are reported. Please present the results with uncertainty estimates or increase the evaluation scale.
  5. [Appendix A.2 and Table 6] The class list in Table 6 contains 'Pig' twice and omits the 'Howling Monkey' entry from the sorted list in the same table; the table should be deduplicated and checked for consistency.

Circularity Check

2 steps flagged · score 6.0 of 10

Headline pose and class accuracy are measured with the same models used to annotate HAIG-2.9M, so those two metrics are partially self-referential; independent FID/KID, CLIP, and human-preference evidence keeps the paper from being wholly circular.

  1. fitted input called prediction [Sec. 5 (Evaluation Metrics); Sec. 4.2 (Data Annotation)]
    "Pose Accuracy. We use DWPose to extract human poses and ViTpose++H to extract animal poses from synthetic images and compare them with the input pose conditions. ... Human Keypoint Annotation. We evaluate three expert models: DWPose (Yang et al., 2023), RTMPose (Jiang et al., 2023), and OpenPose (Cao et al., 2017). We choose DWPose as our annotation model ... Animal Keypoint Annotation. We assess three expert models: ViTPose++H (Xu et al., 2022), X-Pose (Yang et al., 2025), and SuperAnimals (Ye et al., 2024). ViTPose++H is selected as our annotation model"

    The pose conditions in the test set are produced by DWPose (human) and ViTPose++H (animal), and the pose-AP evaluator is the same pair of models. The metric therefore compares the generator's output against the annotator's own keypoint estimates, rewarding agreement with annotator bias rather than with an independent pose label. The reported 30.01/28.38 AP is partially a self-consistency score, not an external validation of pose control.

  2. fitted input called prediction [Sec. 5 (Class Accuracy metric); Sec. 4.1-4.2 (dataset filtering and bounding-box annotation)]
    "Class Accuracy. YOLO-World detects each class and calculates the maximum IoU between detection boxes and the Ground Truth box. If the IoU is higher than 0.5, the class is considered correctly generated. ... Ultimately, we select YOLO-World as our annotation model ... we use two lightweight open-vocabulary object detection models, Grounding-Dino (Liu et al., 2023) and YOLO-World (Cheng et al., 2024), to check if the images contain the desired classes."

    Class accuracy uses YOLO-World as both the detector and, through the bounding-box annotation step, the source of the ground-truth boxes; YOLO-World also filtered the dataset. A generated instance counts as correct when YOLO-World's own detection overlaps YOLO-World's own target box, so the metric reduces to detector self-agreement and does not independently establish class-correct generation.

full rationale

The paper's central architectural claims (token-based instance/keypoint conditions, timestep-aware modulator) are not circular: they are supported by FID/KID, CLIP, human preference, ablations, and qualitative comparisons that do not depend on the annotation models. However, two headline metrics are circular by construction. The pose-AP metric feeds the generated image to DWPose/ViTPose++H and compares the extracted keypoints with the input keypoint conditions, but those input conditions were generated by the very same DWPose/ViTPose++H models selected in Sec. 4.2. Likewise, class accuracy uses YOLO-World as detector, while YOLO-World supplied the ground-truth boxes and was used in filtering. These metrics therefore partly measure the generator's ability to reproduce the annotators' biases, not independent pose/class correctness. The 17-keypoint cross-species alignment is asserted rather than demonstrated, but that is a validity/correctness concern, not a circularity under the defined patterns. Because the independent metrics remain non-circular, the paper is only partially circular, not globally forced; score 6.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced; the ledger captures the shared-keypoint schema, automated annotation quality, frozen backbone priors, and the standard diffusion objective that the central claim rests on.

free parameters (3)
  • condition dropout rates = bbox 0.5, keypoint 0.15, prompt 0.10
    Hand-chosen training augmentation rates in Sec 5 (Implementation Details); no sensitivity study provided.
  • learning rate = 2e-5
    Fixed learning rate chosen by hand for fine-tuning; no schedule or search reported.
  • aesthetic and resolution filters = aesthetic >= 5.0, resolution >= 512x512
    Dataset curation thresholds in Sec 4.1 that shape HAIG-2.9M and therefore the trained model.
assumptions (4)
  • domain assumption All 31 classes share a common 17-keypoint template from APT36K, and keypoint semantics are aligned across species after encoding with a class embedding.
    Sec 4 states 'Each category has 17 corresponding keypoints'; the unified keypoint encoder (Sec 3.2.1) relies on this alignment to share one encoder across species.
  • domain assumption Automated annotation models (YOLO-World, DWPose, ViTPose++H, GPT4o/CogVLM2) provide ground truth of sufficient quality, with only a 5% manual spot check.
    Sec 4.2 describes annotation by selected models after 5K-image voting; the dataset and all evaluations inherit these labels.
  • domain assumption PIXART-α contains adequate priors for both human and animal appearance, so only the new modules need training.
    Sec 5 uses PIXART-α-1024px as frozen backbone; if its animal priors were weak, the control modules could not produce the reported quality.
  • standard math The standard diffusion denoising objective (Eq. 4) is a valid training target for conditional control.
    Sec C.1 restates the conventional epsilon-matching loss; this is the standard assumption for diffusion fine-tuning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UniMC: Taming Diffusion Transformer for Unified Keypoint-Guided Multi-Class Image Generation." pith.science (2026). https://pith.science/paper/XPO73ZTM

@misc{pith2026250702713,
  author       = {Pith},
  title        = {Pith review of: UniMC: Taming Diffusion Transformer for Unified Keypoint-Guided Multi-Class Image Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XPO73ZTM}},
  note         = {Machine review of arXiv:2507.02713}
}
read the original abstract

Although significant advancements have been achieved in the progress of keypoint-guided Text-to-Image diffusion models, existing mainstream keypoint-guided models encounter challenges in controlling the generation of more general non-rigid objects beyond humans (e.g., animals). Moreover, it is difficult to generate multiple overlapping humans and animals based on keypoint controls solely. These challenges arise from two main aspects: the inherent limitations of existing controllable methods and the lack of suitable datasets. First, we design a DiT-based framework, named UniMC, to explore unifying controllable multi-class image generation. UniMC integrates instance- and keypoint-level conditions into compact tokens, incorporating attributes such as class, bounding box, and keypoint coordinates. This approach overcomes the limitations of previous methods that struggled to distinguish instances and classes due to their reliance on skeleton images as conditions. Second, we propose HAIG-2.9M, a large-scale, high-quality, and diverse dataset designed for keypoint-guided human and animal image generation. HAIG-2.9M includes 786K images with 2.9M instances. This dataset features extensive annotations such as keypoints, bounding boxes, and fine-grained captions for both humans and animals, along with rigorous manual inspection to ensure annotation accuracy. Extensive experiments demonstrate the high quality of HAIG-2.9M and the effectiveness of UniMC, particularly in heavy occlusions and multi-class scenarios.

Figures

Figures reproduced from arXiv: 2507.02713 by the authors.

Figure 1
Figure 1. Top: We establish HAIG-2.9M, a large-scale, high-quality, and highly diverse dataset with joint keypoint-level, instance-level, and densely semantic annotations for both humans and animals. Bottom: Based on HAIG-2.9M, we design UNIMC, a controllable DiT-based framework for keypoint-guided image generation, especially for multi-class (e.g., (a), (b), (c)) and heavy occlusion scenarios (e.g., (a), (b)). The bottom par… view at source ↗
Figure 2
Figure 2. Skeleton image conditions face two main issues: (a) Class binding confusion: difficult to distinguish classes from skeleton images alone; (b) Instance binding confu￾sion: challenging to distinguish keypoints of overlapping instances under occlusions (e.g., parts in red box). 2. Related Work Keypoint-Guided Image Generation. Controllable image generation is a critical research direction, evolving from early Generativ… view at source ↗
Figure 3
Figure 3. Timestep-Aware Keypoint based Modulator. Left: The block of PIXART-α. Right: Four modulator variants are proposed to control DiT (detailed in Sec. 3.2.2). All variants leverage self-attention to model backbone and keypoint tokens: (a) and (c) use global-wise timestep modulation (w/o and w pretrained timestep encoder, respectively), (b) adopts GLIGEN-style gated self-attention (Li et al., 2023), and (d) applies block… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Dataset Collection and Annotation Pipeline. (a) We source over 16M images from high-quality datasets and websites. After rigorous filtering and processing, we retain 786K images. (b) Our data is annotated by multiple expert models and subjected to strict manual review.…
Figure 5
Figure 5. Figure 5: Dataset Visualization. (a) HAIG-2.9M contains images of various styles. (b) shows an example from our annotated dataset. For visualization, we render the bounding box and keypoint as images overlaid on the original image. The top right corner displays our detailed capt…
Figure 6
Figure 6. Figure 6: Qualitative Comparisons. We provide all methods with required condition formats. From left to right: (a): Input condition, (b): UNIMC trained on HAIG-2.9M, (c): UNIMC trained on COCO, (d): UNIMC trained on APT36K, (e): UNIMC trained on COCO+APT36K, (f): ControlNet, (g)…
Figure 7
Figure 7. Figure 7: Additional Qualitative Comparisons (I). From left to right: (a): Input condition, (b): UNIMC trained on HAIG-2.9M, (c): UNIMC trained on COCO, (d): UNIMC trained on APT36K, (e): UNIMC trained on COCO+APT36K, (f): ControlNet, (g): GLIGEN. 16 [PITH_FULL_IMAGE:figures/fu…
Figure 8
Figure 8. Figure 8: Additional Qualitative Comparisons (II). From left to right: (a): Input condition, (b): UNIMC trained on HAIG-2.9M, (c): UNIMC trained on COCO, (d): UNIMC trained on APT36K, (e): UNIMC trained on COCO+APT36K, (f): ControlNet, (g): GLIGEN. 17 [PITH_FULL_IMAGE:figures/f…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. TerraDiT-$\Omega$: Unified Spatial Control for Satellite Image Synthesis with Any Geospatial Primitive

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    TerraDiT-Ω generates satellite imagery from native geospatial primitives via Geometry-Aware Local Attention and outperforms dense and sparse control baselines while boosting downstream GeoAI tasks.

Reference graph

Works this paper leans on

34 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    L., Kiros, J

    Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450,

  2. [5]

    Yolo-world: Real-time open-vocabulary object detection

    Cheng, T., Song, L., Ge, Y ., Liu, W., Wang, X., and Shan, Y . Yolo-world: Real-time open-vocabulary object detection. arXiv preprint arXiv:2401.17270,

  3. [6]

    Other works (Esser et al., 2024; Chen et al., 2023; OpenAI, 2024a; Gao et al., 2024; Li et al., 2024b; Xie et al., 2023; Nair et al.,

    Class Counts in the HAIG-2.9M, Sorted Alphabetically Class Count Class Count Class Count Class Count Antelope 2755 Deer 20701 Monkey 5658 Pig 4018 Bear 22833 Dog 114502 Orangutan 126 Polar Bear 5625 Black Bear 304 Elephant 49457 Panda 1521 Rabbit 623 Buffalo 1219 Fox 7536 Person 2159681 Raccoon 1962 Cat 36396 Giraffe 8508 Pig 4018 Rhino 178 Cheetah 2249 G...

  4. [8]

    Orange kittenwith bright blue eyes on rocky ground

    Comparison of human evaluations across different methods, supplementing Table 4 in the main paper. Methods Generation Quality Class and Keypoint Control Ability Pixart 47 0 ControlNet 11 9 GLIGEN 2 4 UniHA (ours) 40 87 E. Limitations Similar to AP10K (Yu et al., 2021), our dataset also faces a long-tail distribution problem. Additionally, our dataset has ...

  5. [9]

    Prompt-to-prompt im- age editing with cross attention control

    Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y ., and Cohen-Or, D. Prompt-to-prompt im- age editing with cross attention control. arXiv preprint arXiv:2208.01626,

  6. [11]

    Rtmpose: Real-time multi-person pose estimation based on mmpose

    Jiang, T., Lu, P., Zhang, L., Ma, N., Han, R., Lyu, C., Li, Y ., and Chen, K. Rtmpose: Real-time multi-person pose estimation based on mmpose. arXiv preprint arXiv:2303.07399,

  7. [13]

    Hunyuan- dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding

    Li, Z., Zhang, J., Lin, Q., Xiong, J., Long, Y ., Deng, X., Zhang, Y ., Liu, X., Huang, M., Xiao, Z., et al. Hunyuan- dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding. arXiv preprint arXiv:2405.08748, 2024b. Lin, T.-Y ., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Doll´ar, P., and Zitnick, C....

  8. [15]

    Fit: Flexible vision transformer for diffusion model

    Lu, Z., Wang, Z., Huang, D., Wu, C., Liu, X., Ouyang, W., and Bai, L. Fit: Flexible vision transformer for diffusion model. arXiv preprint arXiv:2402.12376,

Show all 34 references
  1. [16]

    S., Boffi, N

    Ma, N., Goldstein, M., Albergo, M. S., Boffi, N. M., Vanden- Eijnden, E., and Xie, S. Sit: Exploring flow and diffusion- based generative models with scalable interpolant trans- formers. arXiv preprint arXiv:2401.08740, 2024b. Midjourney. Midjourney,

  2. [17]

    G., Valanarasu, J

    Nair, N. G., Valanarasu, J. M. J., and Patel, V . M. Diff- scaler: Enhancing the generative prowess of diffusion transformers. arXiv preprint arXiv:2404.09976,

  3. [18]

    Hierarchical text-conditional image generation with clip latents

    Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3,

  4. [20]

    U-dits: Downsample tokens in u-shaped diffusion trans- formers

    Tian, Y ., Tu, Z., Chen, H., Hu, J., Xu, C., and Wang, Y . U-dits: Downsample tokens in u-shaped diffusion trans- formers. arXiv preprint arXiv:2405.02730,

  5. [21]

    Plug- and-play diffusion features for text-driven image-to- image translation

    Tumanyan, N., Geyer, M., Bagon, S., and Dekel, T. Plug- and-play diffusion features for text-driven image-to- image translation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 1921–1930,

  6. [22]

    Cogvlm: Visual expert for pretrained language models

    Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y ., Ji, J., Yang, Z., Zhao, L., Song, X., et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079,

  7. [23]

    S., Girdhar, R., and Misra, I

    Wang, X., Darrell, T., Rambhatla, S. S., Girdhar, R., and Misra, I. Instancediffusion: Instance-level control for im- age generation. arXiv preprint arXiv:2402.03290,

  8. [24]

    Magicpose4d: Crafting articulated models with appearance and motion control

    Zhang, H., Chang, D., Li, F., Soleymani, M., and Ahuja, N. Magicpose4d: Crafting articulated models with appearance and motion control. arXiv preprint arXiv:2405.14017,

  9. [25]

    Migc: Multi- instance generation controller for text-to-image synthesis

    Zhou, D., Li, Y ., Ma, F., Yang, Z., and Yang, Y . Migc: Multi- instance generation controller for text-to-image synthesis. arXiv preprint arXiv:2402.05408,

  10. [26]

    Dataset A.1

    12 UNIMC: Taming Diffusion Transformer for Unified Keypoint-Guided Multi-Class Image Generation A. Dataset A.1. Licenses Image Websites: • Pexels 1 (Pexels, 2024): Creative Commons CC0 license. • Pixabay 2 (Pixabay, 2024): Creative Commons CC0 license. • Stocksnap 3 (stocksnap...

  11. [27]

    pioneer the Transformer-based diffusion model, with subsequent works refining the architecture (Hatamizadeh et al., 2023; Ma et al., 2024b; Lu et al., 2024; Tian et al.,

  12. [28]

    or improving training efficiency (Gao et al., 2023; Zheng et al., 2024). For Text-to-Image (T2I) synthesis, the 1https://www.pexels.com/ 2https://pixabay.com/ 3https://stocksnap.io/ 4https://unsplash.com/ 5https://unsplash.com/license 6https://journeydb.github.io/ 7https://jou...

  13. [30]

    In this work, we use PIXART-α as our backbone, which is a variant of DiT (Peebles & Xie, 2023)

    have also proven the efficiency, potential and scalability of DiT. In this work, we use PIXART-α as our backbone, which is a variant of DiT (Peebles & Xie, 2023). We explore various DiT control variants to achieve unified keypoint-level control. C. Method C.1. Preliminaries Di...

  14. [31]

    During the sampling process, the model can transform Gaussian noise of normal distribution to real samples step-by-step

    define a forward diffusion process to gradually convert the sample x from a real data distribution pdata(x) into a noisy distribution, and learn the reverse process in an iterative denoising way (Sohl-Dickstein et al., 2015; Ho et al., 2020). During the sampling process, the m...

  15. [32]

    Latent Diffusion Model & PIXART-α

    to minimize the mean-squared error: min θ Ex,c,ϵ,t h ||ϵ − ϵθ( p ˆαtx + p 1 − ˆαtϵ, c, t)||2 2 i , (4) where x, c ∼ pdata are the sample-condition pairs from the training distribution; ϵ ∼ N(0, I) is the ground-truth noise; t ∼ U[1, T] is the time-step and T is the predefined ...

  16. [33]

    At the inference stage, we can reconstruct the generated image through the decoder ˆx = D(ˆz)

    first encodes the image x to latent space z = E(x) for training. At the inference stage, we can reconstruct the generated image through the decoder ˆx = D(ˆz). In this work, we use PIXART-α (Chen et al., 2024c) as our backbone, which is a basic T2I model based on the Latent Di...

  17. [2014]

    Liu, H., Li, C., Wu, Q., and Lee, Y . J. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024a. Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., et al. Grounding dino: Marrying dino with grounded pr...

  18. [2015]

    11 UNIMC: Taming Diffusion Transformer for Unified Keypoint-Guided Multi-Class Image Generation Laion-aesthetics v2, 2024a

    Schuhmann, C., Beaumont, R., Vencu, R., Gor- don, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. 11 UNIMC: Taming Diffusion Transformer for Unified Keypoint-Guided Multi-Class Image Generation Laion-aesthetics v2, 2024a. URL https: //lai...

  19. [2016]

    Qwen-vl: A frontier large vision- language model with versatile abilities

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A frontier large vision- language model with versatile abilities. arXiv preprint arXiv:2308.12966,

  20. [2017]

    Pixart-\sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation

    Chen, J., Ge, C., Xie, E., Wu, Y ., Yao, L., Ren, X., Wang, Z., Luo, P., Lu, H., and Li, Z. Pixart-\sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint arXiv:2403.04692, 2024a. Chen, J., Wu, Y ., Luo, S., Xie, E., Paul, S., L...

  21. [2018]

    Diffit: Diffusion vision transformers for image generation

    Hatamizadeh, A., Song, J., Liu, G., Kautz, J., and Vahdat, A. Diffit: Diffusion vision transformers for image generation. arXiv preprint arXiv:2312.02139,

  22. [2020]

    Animate anyone: Consistent and controllable image-to- video synthesis for character animation

    Hu, L., Gao, X., Zhang, P., Sun, K., Zhang, B., and Bo, L. Animate anyone: Consistent and controllable image-to- video synthesis for character animation. arXiv preprint arXiv:2311.17117,

  23. [2021]

    Scaling rectified flow transformers for high-resolution image synthesis

    Esser, P., Kulal, S., Blattmann, A., Entezari, R., M¨uller, J., Saini, H., Levi, Y ., Lorenz, D., Sauer, A., Boesel, F., et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206,

  24. [2022]

    Lumina-t2x: Transform- ing text into any modality, resolution, and duration via flow-based large diffusion transformers

    9 UNIMC: Taming Diffusion Transformer for Unified Keypoint-Guided Multi-Class Image Generation Gao, P., Zhuo, L., Lin, Z., Liu, C., Chen, J., Du, R., Xie, E., Luo, X., Qiu, L., Zhang, Y ., et al. Lumina-t2x: Transform- ing text into any modality, resolution, and duration via f...

  25. [2023]

    J., Arbel, M., and Gret- ton, A

    Bi´nkowski, M., Sutherland, D. J., Arbel, M., and Gret- ton, A. Demystifying mmd gans. arXiv preprint arXiv:1801.01401,

  26. [2024]

    Li, S., Fu, J., Liu, K., Wang, W., Lin, K.-Y ., and Wu, W

    URL https://github.com/GantMan/nsfw_model. Li, S., Fu, J., Liu, K., Wang, W., Lin, K.-Y ., and Wu, W. Cos- micman: A text-to-image foundation model for humans. arXiv preprint arXiv:2404.01294, 2024a. Li, Y ., Liu, H., Wu, Q., Mu, F., Yang, J., Gao, J., Li, C., and Lee, Y . J. ...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.